From afa at codespeak.net Fri Oct 1 00:18:01 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 00:18:01 +0200 (CEST) Subject: [pypy-svn] r77512 - pypy/branch/fast-forward/lib-python/modified-2.7.0/test Message-ID: <20100930221801.84632282BDB@codespeak.net> Author: afa Date: Fri Oct 1 00:17:59 2010 New Revision: 77512 Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py Log: Make test_support.findfile search files in 2.7.0, not in modified-2.7.0. Exact copy of change r58845. Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py ============================================================================== --- pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py Fri Oct 1 00:17:59 2010 @@ -430,16 +430,20 @@ rmtree(name) -def findfile(file, here=__file__, subdir=None): +def findfile(file, here=None, subdir=None): """Try to find a file on sys.path and the working directory. If it is not found the argument passed to the function is returned (this does not necessarily signal failure; could still be the legitimate path).""" + import test if os.path.isabs(file): return file if subdir is not None: file = os.path.join(subdir, file) path = sys.path - path = [os.path.dirname(here)] + path + if here is None: + path = test.__path__ + path + else: + path = [os.path.dirname(here)] + path for dn in path: fn = os.path.join(dn, file) if os.path.exists(fn): return fn From afa at codespeak.net Fri Oct 1 00:30:45 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 00:30:45 +0200 (CEST) Subject: [pypy-svn] r77513 - pypy/branch/fast-forward/lib-python Message-ID: <20100930223045.56FD5282BDB@codespeak.net> Author: afa Date: Fri Oct 1 00:30:43 2010 New Revision: 77513 Modified: pypy/branch/fast-forward/lib-python/conftest.py Log: Reduce timeout to 100 seconds per test script. Today 7 of them reached the 1000s limit, and the test lasted more than 3 hours. Modified: pypy/branch/fast-forward/lib-python/conftest.py ============================================================================== --- pypy/branch/fast-forward/lib-python/conftest.py (original) +++ pypy/branch/fast-forward/lib-python/conftest.py Fri Oct 1 00:30:43 2010 @@ -30,7 +30,7 @@ def pytest_addoption(parser): group = parser.getgroup("complicance testing options") group.addoption('-T', '--timeout', action="store", type="string", - default="1000", dest="timeout", + default="100", dest="timeout", help="fail a test module after the given timeout. " "specify in seconds or 'NUMmp' aka Mega-Pystones") group.addoption('--pypy', action="store", type="string", From agaynor at codespeak.net Fri Oct 1 00:41:43 2010 From: agaynor at codespeak.net (agaynor at codespeak.net) Date: Fri, 1 Oct 2010 00:41:43 +0200 (CEST) Subject: [pypy-svn] r77514 - in pypy/branch/fast-forward/pypy/objspace/std: . test Message-ID: <20100930224143.4FAD5282BDB@codespeak.net> Author: agaynor Date: Fri Oct 1 00:41:41 2010 New Revision: 77514 Modified: pypy/branch/fast-forward/pypy/objspace/std/inttype.py pypy/branch/fast-forward/pypy/objspace/std/test/test_intobject.py Log: Added int.bit_length. Modified: pypy/branch/fast-forward/pypy/objspace/std/inttype.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/inttype.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/inttype.py Fri Oct 1 00:41:41 2010 @@ -15,6 +15,18 @@ def int_conjugate__ANY(space, w_int): return space.pos(w_int) +int_bit_length = SMM("bit_length", 1, doc="int.bit_length() -> int\n\nNumber of bits necessary to represent self in binary.\n>>> bin(37)\n'0b100101'\n>>> (37).bit_length()\n6") + +def int_bit_length__ANY(space, w_int): + val = space.int_w(w_int) + if val < 0: + val = -val + bits = 0 + while val: + bits += 1 + val >>= 1 + return space.wrap(bits) + register_all(vars(), globals()) Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_intobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/test/test_intobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/test/test_intobject.py Fri Oct 1 00:41:41 2010 @@ -437,6 +437,18 @@ # __eq__ & the others. assert 1 .__cmp__ assert int .__cmp__ + + def test_bit_length(self): + for val, bits in [ + (0, 0), + (1, 1), + (10, 4), + (150, 8), + (-1, 1), + (-10, 4), + (-150, 8), + ]: + assert val.bit_length() == bits class AppTestIntOptimizedAdd(AppTestInt): From afa at codespeak.net Fri Oct 1 00:48:30 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 00:48:30 +0200 (CEST) Subject: [pypy-svn] r77515 - in pypy/trunk/pypy/interpreter: . test Message-ID: <20100930224830.270BC282BDB@codespeak.net> Author: afa Date: Fri Oct 1 00:48:28 2010 New Revision: 77515 Modified: pypy/trunk/pypy/interpreter/function.py pypy/trunk/pypy/interpreter/test/test_function.py Log: issue560 resolved: new.instancemethod should complain when both self and class are None. Merge of r75709 from branch/fast-forwad Modified: pypy/trunk/pypy/interpreter/function.py ============================================================================== --- pypy/trunk/pypy/interpreter/function.py (original) +++ pypy/trunk/pypy/interpreter/function.py Fri Oct 1 00:48:28 2010 @@ -430,8 +430,11 @@ self.w_class = w_class # possibly space.w_None def descr_method__new__(space, w_subtype, w_function, w_instance, w_class=None): - if space.is_w( w_instance, space.w_None ): + if space.is_w(w_instance, space.w_None): w_instance = None + if w_instance is None and space.is_w(w_class, space.w_None): + raise OperationError(space.w_TypeError, + space.wrap("unbound methods must have class")) method = space.allocate_instance(Method, w_subtype) Method.__init__(method, space, w_function, w_instance, w_class) return space.wrap(method) Modified: pypy/trunk/pypy/interpreter/test/test_function.py ============================================================================== --- pypy/trunk/pypy/interpreter/test/test_function.py (original) +++ pypy/trunk/pypy/interpreter/test/test_function.py Fri Oct 1 00:48:28 2010 @@ -482,6 +482,11 @@ raises(TypeError, m, MyInst(None)) raises(TypeError, m, MyInst(42)) + def test_invalid_creation(self): + import new + def f(): pass + raises(TypeError, new.instancemethod, f, None) + class TestMethod: def setup_method(self, method): From agaynor at codespeak.net Fri Oct 1 03:45:23 2010 From: agaynor at codespeak.net (agaynor at codespeak.net) Date: Fri, 1 Oct 2010 03:45:23 +0200 (CEST) Subject: [pypy-svn] r77516 - pypy/branch/fast-forward/pypy/objspace/std Message-ID: <20101001014523.7CCBC282BDB@codespeak.net> Author: agaynor Date: Fri Oct 1 03:45:20 2010 New Revision: 77516 Modified: pypy/branch/fast-forward/pypy/objspace/std/complexobject.py Log: Don't let an RPython exception escape from abs(complex). Modified: pypy/branch/fast-forward/pypy/objspace/std/complexobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/complexobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/complexobject.py Fri Oct 1 03:45:20 2010 @@ -216,7 +216,10 @@ return W_ComplexObject(w_complex.realval, w_complex.imagval) def abs__Complex(space, w_complex): - return space.newfloat(math.hypot(w_complex.realval, w_complex.imagval)) + try: + return space.newfloat(math.hypot(w_complex.realval, w_complex.imagval)) + except OverflowError, e: + raise OperationError(space.w_OverflowError, space.wrap(str(e))) def eq__Complex_Complex(space, w_complex1, w_complex2): return space.newbool((w_complex1.realval == w_complex2.realval) and From arigo at codespeak.net Fri Oct 1 11:31:30 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 11:31:30 +0200 (CEST) Subject: [pypy-svn] r77517 - pypy/extradoc/sprintinfo/ddorf2010 Message-ID: <20101001093130.A5413282BDB@codespeak.net> Author: arigo Date: Fri Oct 1 11:31:28 2010 New Revision: 77517 Added: pypy/extradoc/sprintinfo/ddorf2010/ - copied from r77448, pypy/extradoc/sprintinfo/ddorf2009/ Removed: pypy/extradoc/sprintinfo/ddorf2010/planning.txt Modified: pypy/extradoc/sprintinfo/ddorf2010/announce.txt pypy/extradoc/sprintinfo/ddorf2010/people.txt Log: The next sprint! Modified: pypy/extradoc/sprintinfo/ddorf2010/announce.txt ============================================================================== --- pypy/extradoc/sprintinfo/ddorf2009/announce.txt (original) +++ pypy/extradoc/sprintinfo/ddorf2010/announce.txt Fri Oct 1 11:31:28 2010 @@ -1,39 +1,15 @@ -D?sseldorf PyPy sprint November 6 - November 13 2009 +D?sseldorf PyPy sprint October 25th-31st 2010 ===================================================== The next PyPy sprint will be held in the Computer Science department of -Heinrich-Heine Universit?t D?sseldorf from the 6th to the 13th of -November 2009. This is a fully public sprint, everyone is welcome to +Heinrich-Heine Universit?t D?sseldorf from the 25th to the 31st of +October 2010. This is a fully public sprint, everyone is welcome to join us. Topics and goals ---------------- -At the sprint we intend to work on the JIT generator in PyPy and on -applying it to PyPy Python interpreter. - -The precise work that will be done is not fixed, as we don't know in -which state the JIT will be in November. However, possible areas of -work might include: - -- tweaking the interpreter/objspace to be more JIT-friendly, e.g. - instance implementation code, call code -- if there is interest starting non x86-32 JIT backends -- trying out existing software to find features where the optimizations - of the JIT could be improved -- improving our benchmarking infrastructure - -We will give special priority to topics that "non-core" people find -interesting (as long as they are somehow JIT-related). - -For an introduction of how our JIT-generation process works, please -refer to our blog: - -http://morepypy.blogspot.com/2009/03/jit-bit-of-look-inside.html - -There is also a more dense academic paper about the subject: - -http://codespeak.net/svn/pypy/extradoc/talk/icooolps2009/bolz-tracing-jit-final.pdf +Open. (to be defined more precisely) Location -------- @@ -54,4 +30,4 @@ you have codespeak commit rights). .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint -.. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/ddorf2009/people.txt +.. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/ddorf2010/people.txt Modified: pypy/extradoc/sprintinfo/ddorf2010/people.txt ============================================================================== --- pypy/extradoc/sprintinfo/ddorf2009/people.txt (original) +++ pypy/extradoc/sprintinfo/ddorf2010/people.txt Fri Oct 1 11:31:28 2010 @@ -1,4 +1,4 @@ -People coming to the Duesseldorf sprint November 2009 +People coming to the Duesseldorf sprint October 2010 ==================================================== People who have a ``?`` in their arrive/depart or accomodation @@ -9,11 +9,8 @@ Name Arrive/Depart Accomodation ==================== ============== ===================== Carl Friedrich Bolz always there private -Antonio Cuni 6/11 - 13/11 Hotel Haus Hillesheim -Armin Rigo 6/11 - 13/11 Hotel Haus Hillesheim -Holger Krekel 9/11 - 13/11 ? -Samuele Pedroni 5/11 - 13/11 Hotel Blaettler -==================== ============== ===================== +Antonio Cuni leaves the 30 private +Armin Rigo always there private ==================== ============== ===================== @@ -21,6 +18,8 @@ ==================== ============== ===================== Name Arrive/Depart Accomodation ==================== ============== ===================== +Holger Krekel +Samuele Pedroni Armin Rigo Michael Hudson Anders Chrigstroem @@ -44,9 +43,7 @@ Laura Creighton ? ? Beatrice Duering ? ? Johan Hahn ? ? -Holger Krekel ? ? Anders Chrigstroem ? ? -Samuele Pedroni ? ? Anders Lehmann ? ? Niklaus Haldimann ? ? ==================== ============== ===================== From arigo at codespeak.net Fri Oct 1 11:41:31 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 11:41:31 +0200 (CEST) Subject: [pypy-svn] r77518 - in pypy/trunk/pypy: jit/backend jit/backend/test jit/backend/x86 jit/metainterp jit/metainterp/optimizeopt jit/tl translator translator/goal Message-ID: <20101001094131.30A83282BDB@codespeak.net> Author: arigo Date: Fri Oct 1 11:41:29 2010 New Revision: 77518 Added: pypy/trunk/pypy/jit/backend/conftest.py - copied unchanged from r77517, pypy/branch/jit-str-fix/pypy/jit/backend/conftest.py pypy/trunk/pypy/jit/tl/jittest.py - copied unchanged from r77517, pypy/branch/jit-str-fix/pypy/jit/tl/jittest.py Removed: pypy/trunk/pypy/jit/backend/test/conftest.py Modified: pypy/trunk/pypy/jit/backend/test/test_ll_random.py pypy/trunk/pypy/jit/backend/test/test_random.py pypy/trunk/pypy/jit/backend/x86/regalloc.py pypy/trunk/pypy/jit/backend/x86/runner.py pypy/trunk/pypy/jit/metainterp/executor.py pypy/trunk/pypy/jit/metainterp/optimizeopt/__init__.py pypy/trunk/pypy/jit/metainterp/warmspot.py pypy/trunk/pypy/translator/driver.py pypy/trunk/pypy/translator/goal/translate.py Log: Merge branch/jit-str-fix, fixing the x86 backend's production of COPYSTRCONTENT and adding the --jittest option to translate.py. Modified: pypy/trunk/pypy/jit/backend/test/test_ll_random.py ============================================================================== --- pypy/trunk/pypy/jit/backend/test/test_ll_random.py (original) +++ pypy/trunk/pypy/jit/backend/test/test_ll_random.py Fri Oct 1 11:41:29 2010 @@ -386,6 +386,20 @@ v_string = self.get_string(builder, r) builder.do(self.opnum, [v_string]) +class AbstractCopyContentOperation(AbstractStringOperation): + def produce_into(self, builder, r): + v_srcstring = self.get_string(builder, r) + v_dststring = self.get_string(builder, r) + if v_srcstring.value == v_dststring.value: # because it's not a + raise test_random.CannotProduceOperation # memmove(), but memcpy() + srclen = len(v_srcstring.getref(self.ptr).chars) + dstlen = len(v_dststring.getref(self.ptr).chars) + v_length = builder.get_index(min(srclen, dstlen), r) + v_srcstart = builder.get_index(srclen - v_length.value + 1, r) + v_dststart = builder.get_index(dstlen - v_length.value + 1, r) + builder.do(self.opnum, [v_srcstring, v_dststring, + v_srcstart, v_dststart, v_length]) + class StrGetItemOperation(AbstractGetItemOperation, _StrOperation): pass @@ -404,6 +418,13 @@ class UnicodeLenOperation(AbstractStringLenOperation, _UnicodeOperation): pass +class CopyStrContentOperation(AbstractCopyContentOperation, _StrOperation): + pass + +class CopyUnicodeContentOperation(AbstractCopyContentOperation, + _UnicodeOperation): + pass + # there are five options in total: # 1. non raising call and guard_no_exception @@ -577,6 +598,8 @@ OPERATIONS.append(UnicodeSetItemOperation(rop.UNICODESETITEM)) OPERATIONS.append(StrLenOperation(rop.STRLEN)) OPERATIONS.append(UnicodeLenOperation(rop.UNICODELEN)) + OPERATIONS.append(CopyStrContentOperation(rop.COPYSTRCONTENT)) + #OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT)) for i in range(2): OPERATIONS.append(GuardClassOperation(rop.GUARD_CLASS)) Modified: pypy/trunk/pypy/jit/backend/test/test_random.py ============================================================================== --- pypy/trunk/pypy/jit/backend/test/test_random.py (original) +++ pypy/trunk/pypy/jit/backend/test/test_random.py Fri Oct 1 11:41:29 2010 @@ -1,7 +1,7 @@ import py, sys from pypy.rlib.rarithmetic import intmask, LONG_BIT from pypy.rpython.lltypesystem import llmemory -from pypy.jit.backend.test import conftest as demo_conftest +from pypy.jit.backend import conftest as demo_conftest from pypy.jit.metainterp.history import BasicFailDescr, TreeLoop from pypy.jit.metainterp.history import BoxInt, ConstInt, LoopToken from pypy.jit.metainterp.history import BoxPtr, ConstPtr @@ -102,7 +102,7 @@ elif isinstance(v, ConstFloat): args.append('ConstFloat(%r)' % v.value) elif isinstance(v, ConstInt): - args.append('ConstInt(%d)' % v.value) + args.append('ConstInt(%s)' % v.value) else: raise NotImplementedError(v) if op.getdescr() is None: @@ -113,7 +113,7 @@ except AttributeError: descrstr = ', descr=...' print >>s, ' ResOperation(rop.%s, [%s], %s%s),' % ( - opname[op.opnum], ', '.join(args), names[op.result], descrstr) + opname[op.getopnum()], ', '.join(args), names[op.result], descrstr) #if getattr(op, 'suboperations', None) is not None: # subops.append(op) @@ -189,7 +189,7 @@ v.value) print >>s, ' op = cpu.execute_token(looptoken)' if self.should_fail_by is None: - fail_args = self.loop.operations[-1].args + fail_args = self.loop.operations[-1].getarglist() else: fail_args = self.should_fail_by.getfailargs() for i, v in enumerate(fail_args): Modified: pypy/trunk/pypy/jit/backend/x86/regalloc.py ============================================================================== --- pypy/trunk/pypy/jit/backend/x86/regalloc.py (original) +++ pypy/trunk/pypy/jit/backend/x86/regalloc.py Fri Oct 1 11:41:29 2010 @@ -959,18 +959,23 @@ args = op.getarglist() base_loc = self.rm.make_sure_var_in_reg(args[0], args) ofs_loc = self.rm.make_sure_var_in_reg(args[2], args) + assert args[0] is not args[1] # forbidden case of aliasing self.rm.possibly_free_var(args[0]) - self.rm.possibly_free_var(args[2]) + if args[3] is not args[2] is not args[4]: # MESS MESS MESS: don't free + self.rm.possibly_free_var(args[2]) # it if ==args[3] or args[4] srcaddr_box = TempBox() - srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box) + forbidden_vars = [args[1], args[3], args[4], srcaddr_box] + srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box, forbidden_vars) self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc) # compute the destination address - base_loc = self.rm.make_sure_var_in_reg(args[1], args) - ofs_loc = self.rm.make_sure_var_in_reg(args[3], args) + base_loc = self.rm.make_sure_var_in_reg(args[1], forbidden_vars) + ofs_loc = self.rm.make_sure_var_in_reg(args[3], forbidden_vars) self.rm.possibly_free_var(args[1]) - self.rm.possibly_free_var(args[3]) + if args[3] is not args[4]: # more of the MESS described above + self.rm.possibly_free_var(args[3]) + forbidden_vars = [args[4], srcaddr_box] dstaddr_box = TempBox() - dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box) + dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box, forbidden_vars) self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc) # call memcpy() length_loc = self.loc(args[4]) Modified: pypy/trunk/pypy/jit/backend/x86/runner.py ============================================================================== --- pypy/trunk/pypy/jit/backend/x86/runner.py (original) +++ pypy/trunk/pypy/jit/backend/x86/runner.py Fri Oct 1 11:41:29 2010 @@ -87,7 +87,9 @@ def execute_token(self, executable_token): addr = executable_token._x86_bootstrap_code + #llop.debug_print(lltype.Void, ">>>> Entering", addr) func = rffi.cast(lltype.Ptr(self.BOOTSTRAP_TP), addr) + #llop.debug_print(lltype.Void, "<<<< Back") fail_index = self._execute_call(func) return self.get_fail_descr_from_number(fail_index) @@ -99,10 +101,7 @@ LLInterpreter.current_interpreter = self.debug_ll_interpreter res = 0 try: - #llop.debug_print(lltype.Void, ">>>> Entering", - # rffi.cast(lltype.Signed, func)) res = func() - #llop.debug_print(lltype.Void, "<<<< Back") finally: if not self.translate_support_code: LLInterpreter.current_interpreter = prev_interpreter Modified: pypy/trunk/pypy/jit/metainterp/executor.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/executor.py (original) +++ pypy/trunk/pypy/jit/metainterp/executor.py Fri Oct 1 11:41:29 2010 @@ -205,8 +205,8 @@ def do_copystrcontent(cpu, _, srcbox, dstbox, srcstartbox, dststartbox, lengthbox): - src = srcbox.getptr(lltype.Ptr(rstr.STR)) - dst = dstbox.getptr(lltype.Ptr(rstr.STR)) + src = srcbox.getref(lltype.Ptr(rstr.STR)) + dst = dstbox.getref(lltype.Ptr(rstr.STR)) srcstart = srcstartbox.getint() dststart = dststartbox.getint() length = lengthbox.getint() @@ -214,8 +214,8 @@ def do_copyunicodecontent(cpu, _, srcbox, dstbox, srcstartbox, dststartbox, lengthbox): - src = srcbox.getptr(lltype.Ptr(rstr.UNICODE)) - dst = dstbox.getptr(lltype.Ptr(rstr.UNICODE)) + src = srcbox.getref(lltype.Ptr(rstr.UNICODE)) + dst = dstbox.getref(lltype.Ptr(rstr.UNICODE)) srcstart = srcstartbox.getint() dststart = dststartbox.getint() length = lengthbox.getint() @@ -428,6 +428,10 @@ if arity == 3: func = get_execute_funclist(3, False)[opnum] return func(cpu, metainterp, argboxes[0], argboxes[1], argboxes[2]) + if arity == 5: # copystrcontent, copyunicodecontent + func = get_execute_funclist(5, False)[opnum] + return func(cpu, metainterp, argboxes[0], argboxes[1], + argboxes[2], argboxes[3], argboxes[4]) raise NotImplementedError Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/__init__.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/optimizeopt/__init__.py (original) +++ pypy/trunk/pypy/jit/metainterp/optimizeopt/__init__.py Fri Oct 1 11:41:29 2010 @@ -14,7 +14,7 @@ optimizations = [OptIntBounds(), OptRewrite(), OptVirtualize(), -# OptString(), + OptString(), OptHeap(), ] optimizer = Optimizer(metainterp_sd, loop, optimizations, virtuals) Modified: pypy/trunk/pypy/jit/metainterp/warmspot.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/warmspot.py (original) +++ pypy/trunk/pypy/jit/metainterp/warmspot.py Fri Oct 1 11:41:29 2010 @@ -67,9 +67,16 @@ def jittify_and_run(interp, graph, args, repeat=1, backendopt=False, trace_limit=sys.maxint, debug_level=DEBUG_STEPS, inline=False, **kwds): + from pypy.config.config import ConfigError translator = interp.typer.annotator.translator - translator.config.translation.gc = "boehm" - translator.config.translation.list_comprehension_operations = True + try: + translator.config.translation.gc = "boehm" + except ConfigError: + pass + try: + translator.config.translation.list_comprehension_operations = True + except ConfigError: + pass warmrunnerdesc = WarmRunnerDesc(translator, backendopt=backendopt, **kwds) for jd in warmrunnerdesc.jitdrivers_sd: jd.warmstate.set_param_threshold(3) # for tests Modified: pypy/trunk/pypy/translator/driver.py ============================================================================== --- pypy/trunk/pypy/translator/driver.py (original) +++ pypy/trunk/pypy/translator/driver.py Fri Oct 1 11:41:29 2010 @@ -426,6 +426,22 @@ [OOTYPE], "JIT compiler generation") + def task_jittest_lltype(self): + """ Run with the JIT on top of the llgraph backend + """ + # parent process loop: spawn a child, wait for the child to finish, + # print a message, and restart + from pypy.translator.goal import unixcheckpoint + unixcheckpoint.restartable_point(auto='run') + # load the module pypy/jit/tl/jittest.py, which you can hack at + # and restart without needing to restart the whole translation process + from pypy.jit.tl import jittest + jittest.jittest(self) + # + task_jittest_lltype = taskdef(task_jittest_lltype, + [RTYPE], + "test of the JIT on the llgraph backend") + def task_backendopt_lltype(self): """ Run all backend optimizations - lltype version """ @@ -433,7 +449,8 @@ backend_optimizations(self.translator) # task_backendopt_lltype = taskdef(task_backendopt_lltype, - [RTYPE, '??pyjitpl_lltype'], + [RTYPE, '??pyjitpl_lltype', + '??jittest_lltype'], "lltype back-end optimisations") BACKENDOPT = 'backendopt_lltype' Modified: pypy/trunk/pypy/translator/goal/translate.py ============================================================================== --- pypy/trunk/pypy/translator/goal/translate.py (original) +++ pypy/trunk/pypy/translator/goal/translate.py Fri Oct 1 11:41:29 2010 @@ -27,6 +27,7 @@ ("annotate", "do type inference", "-a --annotate", ""), ("rtype", "do rtyping", "-t --rtype", ""), ("pyjitpl", "JIT generation step", "--pyjitpl", ""), + ("jittest", "JIT test with llgraph backend", "--jittest", ""), ("backendopt", "do backend optimizations", "--backendopt", ""), ("source", "create source", "-s --source", ""), ("compile", "compile", "-c --compile", " (default goal)"), From arigo at codespeak.net Fri Oct 1 11:41:44 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 11:41:44 +0200 (CEST) Subject: [pypy-svn] r77519 - pypy/branch/jit-str-fix Message-ID: <20101001094144.21CF8282BDB@codespeak.net> Author: arigo Date: Fri Oct 1 11:41:42 2010 New Revision: 77519 Removed: pypy/branch/jit-str-fix/ Log: Remove merged branch. From afa at codespeak.net Fri Oct 1 11:54:07 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 11:54:07 +0200 (CEST) Subject: [pypy-svn] r77520 - in pypy/branch/fast-forward/pypy/module/rctime: . test Message-ID: <20101001095407.1BDF7282BDB@codespeak.net> Author: afa Date: Fri Oct 1 11:54:05 2010 New Revision: 77520 Modified: pypy/branch/fast-forward/pypy/module/rctime/app_time.py pypy/branch/fast-forward/pypy/module/rctime/interp_time.py pypy/branch/fast-forward/pypy/module/rctime/test/test_rctime.py Log: Fixes in time module: - unsupported formats in strftime must not pass silently - the python version of strptime is now called _strptime._strptime(). Modified: pypy/branch/fast-forward/pypy/module/rctime/app_time.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/rctime/app_time.py (original) +++ pypy/branch/fast-forward/pypy/module/rctime/app_time.py Fri Oct 1 11:54:05 2010 @@ -5,6 +5,7 @@ class struct_time: __metaclass__ = structseqtype __module__ = 'time' + name = 'time.struct_time' tm_year = structseqfield(0) tm_mon = structseqfield(1) @@ -24,7 +25,7 @@ (same as strftime()).""" import _strptime # from the CPython standard library - return _strptime.strptime(string, format) + return _strptime._strptime(string, format) __doc__ = """This module provides various functions to manipulate time values. Modified: pypy/branch/fast-forward/pypy/module/rctime/interp_time.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/rctime/interp_time.py (original) +++ pypy/branch/fast-forward/pypy/module/rctime/interp_time.py Fri Oct 1 11:54:05 2010 @@ -450,6 +450,21 @@ raise OperationError(space.w_ValueError, space.wrap("daylight savings flag out of range")) + if _WIN: + # check that the format string contains only valid directives + length = len(format) + i = 0 + while i < length: + if format[i] == '%': + i += 1 + if i < length and format[i] == '#': + # not documented by python + i += 1 + if i >= length or format[i] not in "aAbBcdfHIjmMpSUwWxXyYzZ%": + raise OperationError(space.w_ValueError, + space.wrap("invalid format string")) + i += 1 + i = 1024 while True: outbuf = lltype.malloc(rffi.CCHARP.TO, i, flavor='raw') Modified: pypy/branch/fast-forward/pypy/module/rctime/test/test_rctime.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/rctime/test/test_rctime.py (original) +++ pypy/branch/fast-forward/pypy/module/rctime/test/test_rctime.py Fri Oct 1 11:54:05 2010 @@ -135,7 +135,7 @@ raises(TypeError, rctime.struct_time, (1, 2, 3)) tup = (1, 2, 3, 4, 5, 6, 7, 8, 9) st_time = rctime.struct_time(tup) - assert str(st_time) == str(tup) + assert str(st_time).startswith('time.struct_time(tm_year=1, ') assert len(st_time) == len(tup) def test_tzset(self): @@ -224,8 +224,11 @@ import time as rctime tt = rctime.gmtime() - result = rctime.strftime('%D', tt) - if result != '': # else format not supported and we got '' + try: + result = rctime.strftime('%D', tt) + except ValueError: + pass + else: assert result == rctime.strftime('%m/%d/%y', tt) def test_strftime_bounds_checking(self): From antocuni at codespeak.net Fri Oct 1 12:05:18 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 1 Oct 2010 12:05:18 +0200 (CEST) Subject: [pypy-svn] r77521 - in pypy/branch/jitffi/pypy: config jit/backend/test module/_rawffi module/_rawffi/test rlib rlib/test translator/c/test Message-ID: <20101001100518.45EA2282BDB@codespeak.net> Author: antocuni Date: Fri Oct 1 12:05:16 2010 New Revision: 77521 Added: pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py - copied, changed from r77301, pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Removed: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Modified: pypy/branch/jitffi/pypy/config/pypyoption.py pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py pypy/branch/jitffi/pypy/module/_rawffi/__init__.py pypy/branch/jitffi/pypy/module/_rawffi/callback.py pypy/branch/jitffi/pypy/module/_rawffi/interp_rawffi.py pypy/branch/jitffi/pypy/module/_rawffi/structure.py pypy/branch/jitffi/pypy/module/_rawffi/test/test__rawffi.py pypy/branch/jitffi/pypy/rlib/libffi.py pypy/branch/jitffi/pypy/rlib/test/test_rdynload.py pypy/branch/jitffi/pypy/translator/c/test/test_newgc.py Log: remove the "import *" from clibffi at the top of libffi.py. Fix (hopefully :-)) all the places that are using the old interface to import from clibffi instead of libffi. Move clibffi specific tests into their own file. Modified: pypy/branch/jitffi/pypy/config/pypyoption.py ============================================================================== --- pypy/branch/jitffi/pypy/config/pypyoption.py (original) +++ pypy/branch/jitffi/pypy/config/pypyoption.py Fri Oct 1 12:05:16 2010 @@ -73,9 +73,9 @@ } module_import_dependencies = { - # no _rawffi if importing pypy.rlib.libffi raises ImportError + # no _rawffi if importing pypy.rlib.clibffi raises ImportError # or CompilationError - "_rawffi" : ["pypy.rlib.libffi"], + "_rawffi" : ["pypy.rlib.clibffi"], "zlib" : ["pypy.rlib.rzlib"], "bz2" : ["pypy.module.bz2.interp_bz2"], Modified: pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py Fri Oct 1 12:05:16 2010 @@ -517,7 +517,8 @@ assert res.value == func_ints(*args) def test_call_to_c_function(self): - from pypy.rlib.libffi import CDLL, ffi_type_uchar, ffi_type_sint + # XXX: fix this to use libffi instead of clibffi + from pypy.rlib.clibffi import CDLL, ffi_type_uchar, ffi_type_sint libc = CDLL('libc.so.6') c_tolower = libc.getpointer('tolower', [ffi_type_uchar], ffi_type_sint) c_tolower.push_arg('A') Modified: pypy/branch/jitffi/pypy/module/_rawffi/__init__.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_rawffi/__init__.py (original) +++ pypy/branch/jitffi/pypy/module/_rawffi/__init__.py Fri Oct 1 12:05:16 2010 @@ -39,11 +39,11 @@ if hasattr(interp_rawffi, 'check_HRESULT'): Module.interpleveldefs['check_HRESULT'] = 'interp_rawffi.check_HRESULT' - from pypy.rlib import libffi + from pypy.rlib import clibffi for name in ['FUNCFLAG_STDCALL', 'FUNCFLAG_CDECL', 'FUNCFLAG_PYTHONAPI', ]: - if hasattr(libffi, name): - Module.interpleveldefs[name] = "space.wrap(%r)" % getattr(libffi, name) + if hasattr(clibffi, name): + Module.interpleveldefs[name] = "space.wrap(%r)" % getattr(clibffi, name) super(Module, cls).buildloaders() buildloaders = classmethod(buildloaders) Modified: pypy/branch/jitffi/pypy/module/_rawffi/callback.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_rawffi/callback.py (original) +++ pypy/branch/jitffi/pypy/module/_rawffi/callback.py Fri Oct 1 12:05:16 2010 @@ -8,8 +8,8 @@ from pypy.module._rawffi.array import get_elem, push_elem from pypy.module._rawffi.interp_rawffi import W_DataInstance, letter2tp, \ wrap_value, unwrap_value, unwrap_truncate_int -from pypy.rlib.libffi import USERDATA_P, CallbackFuncPtr, FUNCFLAG_CDECL -from pypy.rlib.libffi import ffi_type_void +from pypy.rlib.clibffi import USERDATA_P, CallbackFuncPtr, FUNCFLAG_CDECL +from pypy.rlib.clibffi import ffi_type_void from pypy.module._rawffi.tracker import tracker from pypy.interpreter.error import OperationError from pypy.interpreter import gateway Modified: pypy/branch/jitffi/pypy/module/_rawffi/interp_rawffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_rawffi/interp_rawffi.py (original) +++ pypy/branch/jitffi/pypy/module/_rawffi/interp_rawffi.py Fri Oct 1 12:05:16 2010 @@ -5,7 +5,7 @@ from pypy.interpreter.gateway import interp2app, NoneNotWrapped from pypy.interpreter.typedef import TypeDef, GetSetProperty -from pypy.rlib.libffi import * +from pypy.rlib.clibffi import * from pypy.rpython.lltypesystem import lltype, rffi from pypy.rlib.unroll import unrolling_iterable Modified: pypy/branch/jitffi/pypy/module/_rawffi/structure.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_rawffi/structure.py (original) +++ pypy/branch/jitffi/pypy/module/_rawffi/structure.py Fri Oct 1 12:05:16 2010 @@ -15,7 +15,7 @@ from pypy.module._rawffi.interp_rawffi import wrap_value, unwrap_value from pypy.module._rawffi.interp_rawffi import unpack_shape_with_length from pypy.module._rawffi.interp_rawffi import size_alignment -from pypy.rlib import libffi +from pypy.rlib import clibffi from pypy.rlib.rarithmetic import intmask, r_uint def unpack_fields(space, w_fields): @@ -104,14 +104,14 @@ descr_fieldoffset.unwrap_spec = ['self', ObjSpace, str] # get the corresponding ffi_type - ffi_struct = lltype.nullptr(libffi.FFI_STRUCT_P.TO) + ffi_struct = lltype.nullptr(clibffi.FFI_STRUCT_P.TO) def get_basic_ffi_type(self): if not self.ffi_struct: # Repeated fields are delicate. Consider for example # struct { int a[5]; } # or struct { struct {int x;} a[5]; } - # Seeing no corresponding doc in libffi, let's just repeat + # Seeing no corresponding doc in clibffi, let's just repeat # the field 5 times... fieldtypes = [] for name, tp in self.fields: @@ -122,7 +122,7 @@ while count + basic_size <= total_size: fieldtypes.append(basic_ffi_type) count += basic_size - self.ffi_struct = libffi.make_struct_ffitype_e(self.size, + self.ffi_struct = clibffi.make_struct_ffitype_e(self.size, self.alignment, fieldtypes) return self.ffi_struct.ffistruct Modified: pypy/branch/jitffi/pypy/module/_rawffi/test/test__rawffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_rawffi/test/test__rawffi.py (original) +++ pypy/branch/jitffi/pypy/module/_rawffi/test/test__rawffi.py Fri Oct 1 12:05:16 2010 @@ -191,7 +191,7 @@ prepare_c_example = staticmethod(prepare_c_example) def setup_class(cls): - from pypy.rlib.libffi import get_libc_name + from pypy.rlib.clibffi import get_libc_name space = gettestobjspace(usemodules=('_rawffi', 'struct')) cls.space = space cls.w_lib_name = space.wrap(cls.prepare_c_example()) Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Fri Oct 1 12:05:16 2010 @@ -1,7 +1,16 @@ -from pypy.rlib.clibffi import * +from pypy.rlib import clibffi +from pypy.rlib.clibffi import get_libc_name from pypy.rlib.objectmodel import specialize from pypy.rlib import jit +def import_types(): + g = globals() + for key, value in clibffi.__dict__.iteritems(): + if key.startswith('ffi_type_'): + g[key] = value +import_types() +del import_types + class AbstractArg(object): next = None @@ -46,8 +55,8 @@ self.funcptr.push_arg(value) # XXX this is bad, fix it somehow in the future, but specialize:argtype # doesn't work correctly with mixing non-negative and normal integers - _push_arg._annenforceargs_ = [None, int] - #push_arg._annspecialcase_ = 'specialize:argtype(1)' + #_push_arg._annenforceargs_ = [None, int] + _push_arg._annspecialcase_ = 'specialize:argtype(1)' _push_arg.oopspec = 'libffi_push_arg(self, value)' def _do_call(self, funcsym, RESULT): Copied: pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py (from r77301, pypy/branch/jitffi/pypy/rlib/test/test_libffi.py) ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py Fri Oct 1 12:05:16 2010 @@ -3,7 +3,7 @@ """ from pypy.translator.c.test.test_genc import compile -from pypy.rlib.libffi import * +from pypy.rlib.clibffi import * from pypy.rlib.objectmodel import keepalive_until_here from pypy.rpython.lltypesystem.ll2ctypes import ALLOCATED from pypy.rpython.lltypesystem import rffi, lltype @@ -36,7 +36,6 @@ class TestCLibffi(BaseFfiTest): def setup_method(self, meth): - py.test.skip("broken during the refactoring, FIXME") ALLOCATED.clear() def test_library_open(self): @@ -423,19 +422,3 @@ print hex(handle) assert handle != 0 assert handle % 0x1000 == 0 - - -class TestLibffi(BaseFfiTest): - """ - Test the new JIT-friendly interface to libffi - """ - - def test_call_argchain(self): - libm = self.get_libm() - pow_ptr = libm.getpointer('pow', [ffi_type_double, ffi_type_double], - ffi_type_double) - pow = Func(pow_ptr) - argchain = FloatArg(2.0) - argchain.next = FloatArg(3.0) - res = pow.call(argchain, rffi.DOUBLE) - assert res == 8.0 Modified: pypy/branch/jitffi/pypy/rlib/test/test_rdynload.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_rdynload.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_rdynload.py Fri Oct 1 12:05:16 2010 @@ -1,5 +1,5 @@ from pypy.rlib.rdynload import * -from pypy.rlib.libffi import get_libc_name +from pypy.rlib.clibffi import get_libc_name from pypy.rpython.lltypesystem import rffi, lltype import py Modified: pypy/branch/jitffi/pypy/translator/c/test/test_newgc.py ============================================================================== --- pypy/branch/jitffi/pypy/translator/c/test/test_newgc.py (original) +++ pypy/branch/jitffi/pypy/translator/c/test/test_newgc.py Fri Oct 1 12:05:16 2010 @@ -624,13 +624,13 @@ os.unlink(self.filename) def define_callback_with_collect(cls): - from pypy.rlib.libffi import ffi_type_pointer, cast_type_to_ffitype,\ + from pypy.rlib.clibffi import ffi_type_pointer, cast_type_to_ffitype,\ CDLL, ffi_type_void, CallbackFuncPtr, ffi_type_sint from pypy.rpython.lltypesystem import rffi, ll2ctypes import gc ffi_size_t = cast_type_to_ffitype(rffi.SIZE_T) - from pypy.rlib.libffi import get_libc_name + from pypy.rlib.clibffi import get_libc_name def callback(ll_args, ll_res, stuff): gc.collect() From afa at codespeak.net Fri Oct 1 14:31:43 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 14:31:43 +0200 (CEST) Subject: [pypy-svn] r77522 - pypy/branch/fast-forward/lib_pypy/pypy_test Message-ID: <20101001123143.24DD7282BDB@codespeak.net> Author: afa Date: Fri Oct 1 14:31:41 2010 New Revision: 77522 Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py pypy/branch/fast-forward/lib_pypy/pypy_test/test_defaultdict.py pypy/branch/fast-forward/lib_pypy/pypy_test/test_deque_extra.py Log: Pypy's own implementation of collections was moved to _collections.py Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Fri Oct 1 14:31:41 2010 @@ -1,5 +1,5 @@ from __future__ import absolute_import -from .. import collections +from .. import _collections as collections import py def test_deque_remove_empty(): Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_defaultdict.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/pypy_test/test_defaultdict.py (original) +++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_defaultdict.py Fri Oct 1 14:31:41 2010 @@ -10,7 +10,7 @@ import copy -from ..collections import defaultdict +from .._collections import defaultdict def foobar(): return list Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_deque_extra.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/pypy_test/test_deque_extra.py (original) +++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_deque_extra.py Fri Oct 1 14:31:41 2010 @@ -7,7 +7,7 @@ class Test_deque: def setup_method(self,method): - from ..collections import deque + from .._collections import deque self.deque = deque self.d = deque(range(n)) From afa at codespeak.net Fri Oct 1 15:03:10 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 15:03:10 +0200 (CEST) Subject: [pypy-svn] r77523 - in pypy/branch/fast-forward: lib-python/modified-2.7.0/test lib_pypy/pypy_test Message-ID: <20101001130310.8029A282BDC@codespeak.net> Author: afa Date: Fri Oct 1 15:03:09 2010 New Revision: 77523 Added: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_functools.py - copied, changed from r77516, pypy/branch/fast-forward/lib-python/2.7.0/test/test_functools.py Removed: pypy/branch/fast-forward/lib_pypy/pypy_test/test_functools.py Log: Don't duplicate test_functools.py in pypy_test, use a slightly modified version of the official test suite. Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_functools.py (from r77516, pypy/branch/fast-forward/lib-python/2.7.0/test/test_functools.py) ============================================================================== --- pypy/branch/fast-forward/lib-python/2.7.0/test/test_functools.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_functools.py Fri Oct 1 15:03:09 2010 @@ -136,6 +136,7 @@ p = proxy(f) self.assertEqual(f.func, p.func) f = None + test_support.gc_collect() self.assertRaises(ReferenceError, getattr, p, 'func') def test_with_bound_and_unbound_methods(self): @@ -172,7 +173,7 @@ updated=functools.WRAPPER_UPDATES): # Check attributes were assigned for name in assigned: - self.assertTrue(getattr(wrapper, name) is getattr(wrapped, name)) + self.assertTrue(getattr(wrapper, name) == getattr(wrapped, name), name) # Check attributes were updated for name in updated: wrapper_attr = getattr(wrapper, name) From hakanardo at codespeak.net Fri Oct 1 15:46:27 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Fri, 1 Oct 2010 15:46:27 +0200 (CEST) Subject: [pypy-svn] r77524 - pypy/extradoc/sprintinfo/ddorf2010 Message-ID: <20101001134627.DA6C6282BDB@codespeak.net> Author: hakanardo Date: Fri Oct 1 15:46:25 2010 New Revision: 77524 Modified: pypy/extradoc/sprintinfo/ddorf2010/people.txt Log: registration Modified: pypy/extradoc/sprintinfo/ddorf2010/people.txt ============================================================================== --- pypy/extradoc/sprintinfo/ddorf2010/people.txt (original) +++ pypy/extradoc/sprintinfo/ddorf2010/people.txt Fri Oct 1 15:46:25 2010 @@ -11,6 +11,7 @@ Carl Friedrich Bolz always there private Antonio Cuni leaves the 30 private Armin Rigo always there private +Hakan Ardo 28/10 - 31/10 staying with antocuni ==================== ============== ===================== From arigo at codespeak.net Fri Oct 1 15:46:29 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 15:46:29 +0200 (CEST) Subject: [pypy-svn] r77525 - pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests Message-ID: <20101001134629.9676D282BDB@codespeak.net> Author: arigo Date: Fri Oct 1 15:46:28 2010 New Revision: 77525 Modified: pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py Log: Add a failing test. Modified: pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py ============================================================================== --- pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py (original) +++ pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py Fri Oct 1 15:46:28 2010 @@ -29,6 +29,7 @@ class RECT(Structure): _fields_ = [("left", c_int), ("top", c_int), ("right", c_int), ("bottom", c_int)] + class TestFunctions(BaseCTypesTestChecker): def test_mro(self): @@ -392,6 +393,18 @@ result = f("abcd", ord("b")) assert result == "bcd" + def test_caching_bug_1(self): + # the same test as test_call_some_args, with two extra lines + # in the middle that trigger caching in f._ptr, which then + # makes the last two lines fail + f = dll.my_strchr + f.argtypes = [c_char_p, c_int] + f.restype = c_char_p + result = f("abcd", ord("b")) + assert result == "bcd" + result = f("abcd", ord("b"), 42) + assert result == "bcd" + def test_sf1651235(self): py.test.skip("we are less strict in checking callback parameters") # see http://www.python.org/sf/1651235 From arigo at codespeak.net Fri Oct 1 16:18:53 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 16:18:53 +0200 (CEST) Subject: [pypy-svn] r77526 - pypy/trunk/lib_pypy/_ctypes Message-ID: <20101001141853.0765B282BDB@codespeak.net> Author: arigo Date: Fri Oct 1 16:18:51 2010 New Revision: 77526 Modified: pypy/trunk/lib_pypy/_ctypes/function.py Log: Minimal fix. Modified: pypy/trunk/lib_pypy/_ctypes/function.py ============================================================================== --- pypy/trunk/lib_pypy/_ctypes/function.py (original) +++ pypy/trunk/lib_pypy/_ctypes/function.py Fri Oct 1 16:18:51 2010 @@ -171,7 +171,7 @@ return self._build_result(restype, resbuffer, argtypes, argsandobjs) def _getfuncptr(self, argtypes, restype, thisarg=None): - if self._ptr is not None: + if self._ptr is not None and argtypes is self._argtypes_: return self._ptr if restype is None or not isinstance(restype, _CDataMeta): import ctypes From arigo at codespeak.net Fri Oct 1 16:24:15 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 16:24:15 +0200 (CEST) Subject: [pypy-svn] r77527 - pypy/branch/larger-writebarrier Message-ID: <20101001142415.40AC7282BDB@codespeak.net> Author: arigo Date: Fri Oct 1 16:24:13 2010 New Revision: 77527 Added: pypy/branch/larger-writebarrier/ - copied from r77526, pypy/trunk/ Log: It seems that "smaller-writebarrier" may have had a performance impact on a few benchmarks. In this branch I am going to try a mixed solution: using the same two-args write barrier for regular setfields, and the smaller (2 args instead of 3) version for setarrayitem. From afa at codespeak.net Fri Oct 1 16:43:33 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 16:43:33 +0200 (CEST) Subject: [pypy-svn] r77529 - in pypy/trunk/pypy/module/imp: . test Message-ID: <20101001144333.2EC47282BDC@codespeak.net> Author: afa Date: Fri Oct 1 16:43:31 2010 New Revision: 77529 Modified: pypy/trunk/pypy/module/imp/importing.py pypy/trunk/pypy/module/imp/test/test_import.py Log: Avoid a crash when "from .. import x" is used at toplevel. Modified: pypy/trunk/pypy/module/imp/importing.py ============================================================================== --- pypy/trunk/pypy/module/imp/importing.py (original) +++ pypy/trunk/pypy/module/imp/importing.py Fri Oct 1 16:43:31 2010 @@ -122,7 +122,7 @@ n = len(ctxt_name_prefix_parts)-level+1 assert n>=0 ctxt_name_prefix_parts = ctxt_name_prefix_parts[:n] - if ctxt_w_path is None: # plain module + if ctxt_name_prefix_parts and ctxt_w_path is None: # plain module ctxt_name_prefix_parts.pop() if ctxt_name_prefix_parts: rel_modulename = '.'.join(ctxt_name_prefix_parts) Modified: pypy/trunk/pypy/module/imp/test/test_import.py ============================================================================== --- pypy/trunk/pypy/module/imp/test/test_import.py (original) +++ pypy/trunk/pypy/module/imp/test/test_import.py Fri Oct 1 16:43:31 2010 @@ -360,6 +360,12 @@ """.rstrip() raises(ValueError, imp) + def test_future_relative_import_error_when_in_non_package2(self): + exec """def imp(): + from .. import inpackage + """.rstrip() + raises(ValueError, imp) + def test_relative_import_with___name__(self): import sys mydict = {'__name__': 'sys.foo'} From arigo at codespeak.net Fri Oct 1 16:55:32 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 16:55:32 +0200 (CEST) Subject: [pypy-svn] r77530 - in pypy/branch/larger-writebarrier/pypy: jit/backend/llgraph jit/backend/llsupport jit/backend/llsupport/test jit/backend/test jit/backend/x86 jit/metainterp rpython/memory rpython/memory/gc rpython/memory/gc/test rpython/memory/gctransform Message-ID: <20101001145532.DC842282BDC@codespeak.net> Author: arigo Date: Fri Oct 1 16:55:30 2010 New Revision: 77530 Modified: pypy/branch/larger-writebarrier/pypy/jit/backend/llgraph/llimpl.py pypy/branch/larger-writebarrier/pypy/jit/backend/llsupport/gc.py pypy/branch/larger-writebarrier/pypy/jit/backend/llsupport/test/test_gc.py pypy/branch/larger-writebarrier/pypy/jit/backend/test/runner_test.py pypy/branch/larger-writebarrier/pypy/jit/backend/x86/assembler.py pypy/branch/larger-writebarrier/pypy/jit/backend/x86/regalloc.py pypy/branch/larger-writebarrier/pypy/jit/metainterp/resoperation.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/base.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/generation.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gctransform/framework.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gcwrapper.py Log: In-progress: partial revert of the merge of branch/smaller-writebarrier (revision 77403). Modified: pypy/branch/larger-writebarrier/pypy/jit/backend/llgraph/llimpl.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/jit/backend/llgraph/llimpl.py (original) +++ pypy/branch/larger-writebarrier/pypy/jit/backend/llgraph/llimpl.py Fri Oct 1 16:55:30 2010 @@ -129,7 +129,7 @@ 'arraylen_gc' : (('ref',), 'int'), 'call' : (('ref', 'varargs'), 'intorptr'), 'call_assembler' : (('varargs',), 'intorptr'), - 'cond_call_gc_wb' : (('ptr',), None), + 'cond_call_gc_wb' : (('ptr', 'ptr'), None), 'oosend' : (('varargs',), 'intorptr'), 'oosend_pure' : (('varargs',), 'intorptr'), 'guard_true' : (('bool',), None), @@ -810,7 +810,7 @@ FLOAT: 0.0} return d[calldescr.typeinfo] - def op_cond_call_gc_wb(self, descr, a): + def op_cond_call_gc_wb(self, descr, a, b): py.test.skip("cond_call_gc_wb not supported") def op_oosend(self, descr, obj, *args): Modified: pypy/branch/larger-writebarrier/pypy/jit/backend/llsupport/gc.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/jit/backend/llsupport/gc.py (original) +++ pypy/branch/larger-writebarrier/pypy/jit/backend/llsupport/gc.py Fri Oct 1 16:55:30 2010 @@ -404,7 +404,7 @@ self.GC_MALLOC_BASIC = lltype.Ptr(lltype.FuncType( [lltype.Signed, lltype.Signed], llmemory.GCREF)) self.WB_FUNCPTR = lltype.Ptr(lltype.FuncType( - [llmemory.Address], lltype.Void)) + [llmemory.Address, llmemory.Address], lltype.Void)) self.write_barrier_descr = WriteBarrierDescr(self) # def malloc_array(itemsize, tid, num_elem): @@ -550,7 +550,8 @@ # the GC, and call it immediately llop1 = self.llop1 funcptr = llop1.get_write_barrier_failing_case(self.WB_FUNCPTR) - funcptr(llmemory.cast_ptr_to_adr(gcref_struct)) + funcptr(llmemory.cast_ptr_to_adr(gcref_struct), + llmemory.cast_ptr_to_adr(gcref_newptr)) def rewrite_assembler(self, cpu, operations): # Perform two kinds of rewrites in parallel: @@ -589,7 +590,7 @@ v = op.getarg(1) if isinstance(v, BoxPtr) or (isinstance(v, ConstPtr) and bool(v.value)): # store a non-NULL - self._gen_write_barrier(newops, op.getarg(0)) + self._gen_write_barrier(newops, op.getarg(0), v) op = op.copy_and_change(rop.SETFIELD_RAW) # ---------- write barrier for SETARRAYITEM_GC ---------- if op.getopnum() == rop.SETARRAYITEM_GC: @@ -598,15 +599,15 @@ bool(v.value)): # store a non-NULL # XXX detect when we should produce a # write_barrier_from_array - self._gen_write_barrier(newops, op.getarg(0)) + self._gen_write_barrier(newops, op.getarg(0), v) op = op.copy_and_change(rop.SETARRAYITEM_RAW) # ---------- newops.append(op) del operations[:] operations.extend(newops) - def _gen_write_barrier(self, newops, v_base): - args = [v_base] + def _gen_write_barrier(self, newops, v_base, v_value): + args = [v_base, v_value] newops.append(ResOperation(rop.COND_CALL_GC_WB, args, None, descr=self.write_barrier_descr)) Modified: pypy/branch/larger-writebarrier/pypy/jit/backend/llsupport/test/test_gc.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/jit/backend/llsupport/test/test_gc.py (original) +++ pypy/branch/larger-writebarrier/pypy/jit/backend/llsupport/test/test_gc.py Fri Oct 1 16:55:30 2010 @@ -141,8 +141,8 @@ repr(offset_to_length), p)) return p - def _write_barrier_failing_case(self, adr_struct): - self.record.append(('barrier', adr_struct)) + def _write_barrier_failing_case(self, adr_struct, adr_newptr): + self.record.append(('barrier', adr_struct, adr_newptr)) def get_write_barrier_failing_case(self, FPTRTYPE): return llhelper(FPTRTYPE, self._write_barrier_failing_case) @@ -239,6 +239,7 @@ s_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, s) r_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, r) s_adr = llmemory.cast_ptr_to_adr(s) + r_adr = llmemory.cast_ptr_to_adr(r) # s_hdr.tid &= ~gc_ll_descr.GCClass.JIT_WB_IF_FLAG gc_ll_descr.do_write_barrier(s_gcref, r_gcref) @@ -246,7 +247,7 @@ # s_hdr.tid |= gc_ll_descr.GCClass.JIT_WB_IF_FLAG gc_ll_descr.do_write_barrier(s_gcref, r_gcref) - assert self.llop1.record == [('barrier', s_adr)] + assert self.llop1.record == [('barrier', s_adr, r_adr)] def test_gen_write_barrier(self): gc_ll_descr = self.gc_ll_descr @@ -254,11 +255,13 @@ # newops = [] v_base = BoxPtr() - gc_ll_descr._gen_write_barrier(newops, v_base) + v_value = BoxPtr() + gc_ll_descr._gen_write_barrier(newops, v_base, v_value) assert llop1.record == [] assert len(newops) == 1 assert newops[0].getopnum() == rop.COND_CALL_GC_WB assert newops[0].getarg(0) == v_base + assert newops[0].getarg(1) == v_value assert newops[0].result is None wbdescr = newops[0].getdescr() assert isinstance(wbdescr.jit_wb_if_flag, int) @@ -358,6 +361,7 @@ # assert operations[0].getopnum() == rop.COND_CALL_GC_WB assert operations[0].getarg(0) == v_base + assert operations[0].getarg(1) == v_value assert operations[0].result is None # assert operations[1].getopnum() == rop.SETFIELD_RAW @@ -381,6 +385,7 @@ # assert operations[0].getopnum() == rop.COND_CALL_GC_WB assert operations[0].getarg(0) == v_base + assert operations[0].getarg(1) == v_value assert operations[0].result is None # assert operations[1].getopnum() == rop.SETARRAYITEM_RAW Modified: pypy/branch/larger-writebarrier/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/jit/backend/test/runner_test.py (original) +++ pypy/branch/larger-writebarrier/pypy/jit/backend/test/runner_test.py Fri Oct 1 16:55:30 2010 @@ -1427,12 +1427,12 @@ assert not excvalue def test_cond_call_gc_wb(self): - def func_void(a): - record.append(a) + def func_void(a, b): + record.append((a, b)) record = [] # S = lltype.GcStruct('S', ('tid', lltype.Signed)) - FUNC = self.FuncType([lltype.Ptr(S)], lltype.Void) + FUNC = self.FuncType([lltype.Ptr(S), lltype.Signed], lltype.Void) func_ptr = llhelper(lltype.Ptr(FUNC), func_void) funcbox = self.get_funcbox(self.cpu, func_ptr) class WriteBarrierDescr(AbstractDescr): @@ -1453,10 +1453,10 @@ sgcref = lltype.cast_opaque_ptr(llmemory.GCREF, s) del record[:] self.execute_operation(rop.COND_CALL_GC_WB, - [BoxPtr(sgcref)], + [BoxPtr(sgcref), ConstInt(-2121)], 'void', descr=WriteBarrierDescr()) if cond: - assert record == [s] + assert record == [(s, -2121)] else: assert record == [] Modified: pypy/branch/larger-writebarrier/pypy/jit/backend/x86/assembler.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/jit/backend/x86/assembler.py (original) +++ pypy/branch/larger-writebarrier/pypy/jit/backend/x86/assembler.py Fri Oct 1 16:55:30 2010 @@ -1780,11 +1780,12 @@ self.mc.PUSH_i32(loc.getint()) if IS_X86_64: - # We clobber this register to pass the arguments, but that's + # We clobber these registers to pass the arguments, but that's # okay, because consider_cond_call_gc_wb makes sure that any # caller-save registers with values in them are present in arglocs, # so they are saved on the stack above and restored below self.mc.MOV_rs(edi.value, 0) + self.mc.MOV_rs(esi.value, 8) # misaligned stack in the call, but it's ok because the write barrier # is not going to call anything more. Also, this assumes that the Modified: pypy/branch/larger-writebarrier/pypy/jit/backend/x86/regalloc.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/jit/backend/x86/regalloc.py (original) +++ pypy/branch/larger-writebarrier/pypy/jit/backend/x86/regalloc.py Fri Oct 1 16:55:30 2010 @@ -696,9 +696,13 @@ def consider_cond_call_gc_wb(self, op): assert op.result is None args = op.getarglist() + loc_newvalue = self.rm.make_sure_var_in_reg(op.getarg(1), args) + # ^^^ we force loc_newvalue in a reg (unless it's a Const), + # because it will be needed anyway by the following setfield_gc. + # It avoids loading it twice from the memory. loc_base = self.rm.make_sure_var_in_reg(op.getarg(0), args, imm_fine=False) - arglocs = [loc_base] + arglocs = [loc_base, loc_newvalue] # add eax, ecx and edx as extra "arguments" to ensure they are # saved and restored. Fish in self.rm to know which of these # registers really need to be saved (a bit of a hack). Moreover, Modified: pypy/branch/larger-writebarrier/pypy/jit/metainterp/resoperation.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/jit/metainterp/resoperation.py (original) +++ pypy/branch/larger-writebarrier/pypy/jit/metainterp/resoperation.py Fri Oct 1 16:55:30 2010 @@ -455,7 +455,7 @@ 'UNICODESETITEM/3', 'NEWUNICODE/1', #'RUNTIMENEW/1', # ootype operation - 'COND_CALL_GC_WB/1d', # [objptr] (for the write barrier) + 'COND_CALL_GC_WB/2d', # [objptr, newvalue] (for the write barrier) 'DEBUG_MERGE_POINT/1', # debugging only 'VIRTUAL_REF_FINISH/2', # removed before it's passed to the backend 'COPYSTRCONTENT/5', # src, dst, srcstart, dststart, length Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/base.py Fri Oct 1 16:55:30 2010 @@ -79,7 +79,7 @@ def set_root_walker(self, root_walker): self.root_walker = root_walker - def write_barrier(self, addr_struct): + def write_barrier(self, newvalue, addr_struct): pass def statistics(self, index): Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/generation.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/generation.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/generation.py Fri Oct 1 16:55:30 2010 @@ -157,6 +157,11 @@ "odd-valued (i.e. tagged) pointer unexpected here") return self.nursery <= addr < self.nursery_top + def appears_to_be_in_nursery(self, addr): + # same as is_in_nursery(), but may return True accidentally if + # 'addr' is a tagged pointer with just the wrong value. + return self.nursery <= addr < self.nursery_top + def malloc_fixedsize_clear(self, typeid, size, can_collect, has_finalizer=False, contains_weakptr=False): if (has_finalizer or not can_collect or @@ -326,7 +331,7 @@ addr = pointer.address[0] newaddr = self.copy(addr) pointer.address[0] = newaddr - self.write_into_last_generation_obj(obj) + self.write_into_last_generation_obj(obj, newaddr) # ____________________________________________________________ # Implementation of nursery-only collections @@ -457,9 +462,9 @@ # "if addr_struct.int0 & JIT_WB_IF_FLAG: remember_young_pointer()") JIT_WB_IF_FLAG = GCFLAG_NO_YOUNG_PTRS - def write_barrier(self, addr_struct): - if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: - self.remember_young_pointer(addr_struct) + def write_barrier(self, newvalue, addr_struct): + if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: + self.remember_young_pointer(addr_struct, newvalue) def _setup_wb(self): DEBUG = self.DEBUG @@ -470,23 +475,33 @@ # For x86, there is also an extra requirement: when the JIT calls # remember_young_pointer(), it assumes that it will not touch the SSE # registers, so it does not save and restore them (that's a *hack*!). - def remember_young_pointer(addr_struct): + def remember_young_pointer(addr_struct, addr): #llop.debug_print(lltype.Void, "\tremember_young_pointer", # addr_struct, "<-", addr) if DEBUG: ll_assert(not self.is_in_nursery(addr_struct), "nursery object with GCFLAG_NO_YOUNG_PTRS") - self.old_objects_pointing_to_young.append(addr_struct) - self.header(addr_struct).tid &= ~GCFLAG_NO_YOUNG_PTRS - self.write_into_last_generation_obj(addr_struct) + # + # What is important in this function is that it *must* + # clear the flag GCFLAG_NO_YOUNG_PTRS from 'addr_struct' + # if 'addr' is in the nursery. It is ok if, accidentally, + # it also clears the flag in some more rare cases, like + # 'addr' being a tagged pointer whose value happens to be + # a large integer that fools is_in_nursery(). + if self.appears_to_be_in_nursery(addr): + self.old_objects_pointing_to_young.append(addr_struct) + self.header(addr_struct).tid &= ~GCFLAG_NO_YOUNG_PTRS + self.write_into_last_generation_obj(addr_struct, addr) remember_young_pointer._dont_inline_ = True self.remember_young_pointer = remember_young_pointer - def write_into_last_generation_obj(self, addr_struct): + def write_into_last_generation_obj(self, addr_struct, addr): objhdr = self.header(addr_struct) if objhdr.tid & GCFLAG_NO_HEAP_PTRS: - objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS - self.last_generation_root_objects.append(addr_struct) + if (self.is_valid_gc_object(addr) and + not self.is_last_generation(addr)): + objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS + self.last_generation_root_objects.append(addr_struct) write_into_last_generation_obj._always_inline_ = True def assume_young_pointers(self, addr_struct): Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py Fri Oct 1 16:55:30 2010 @@ -726,16 +726,16 @@ def JIT_max_size_of_young_obj(cls): return cls.TRANSLATION_PARAMS['large_object'] - def write_barrier(self, addr_struct): + def write_barrier(self, newvalue, addr_struct): if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: - self.remember_young_pointer(addr_struct) + self.remember_young_pointer(addr_struct, newvalue) - def write_barrier_from_array(self, addr_array, index): + def write_barrier_from_array(self, newvalue, addr_array, index): if self.header(addr_array).tid & GCFLAG_NO_YOUNG_PTRS: if self.card_page_indices > 0: # <- constant-folded self.remember_young_pointer_from_array(addr_array, index) else: - self.remember_young_pointer(addr_array) + self.remember_young_pointer(addr_array, newvalue) def _init_writebarrier_logic(self): DEBUG = self.DEBUG @@ -746,7 +746,7 @@ # For x86, there is also an extra requirement: when the JIT calls # remember_young_pointer(), it assumes that it will not touch the SSE # registers, so it does not save and restore them (that's a *hack*!). - def remember_young_pointer(addr_struct): + def remember_young_pointer(addr_struct, newvalue): # 'addr_struct' is the address of the object in which we write. if DEBUG: ll_assert(not self.is_in_nursery(addr_struct), @@ -754,7 +754,9 @@ # # We assume that what we are writing is a pointer to the nursery # (and don't care for the fact that this new pointer may not - # actually point to the nursery, which seems ok). What we need is + # actually point to the nursery, which seems ok + # XXXXXX XXX wrong !! fix me + # ). What we need is # to remove the flag GCFLAG_NO_YOUNG_PTRS and add the old object # to the list 'old_objects_pointing_to_young'. We know that # 'addr_struct' cannot be in the nursery, because nursery objects @@ -784,12 +786,19 @@ # 'addr_array' is the address of the object in which we write, # which must have an array part; 'index' is the index of the # item that is (or contains) the pointer that we write. + if DEBUG: + ll_assert(not self.is_in_nursery(addr_array), + "nursery array with GCFLAG_NO_YOUNG_PTRS") objhdr = self.header(addr_array) if objhdr.tid & GCFLAG_HAS_CARDS == 0: # - # no cards, use default logic. The 'nocard_logic()' is just - # 'remember_young_pointer()', but forced to be inlined here. - nocard_logic(addr_array) + # no cards, use default logic. Mostly copied from above. + self.old_objects_pointing_to_young.append(addr_array) + objhdr = self.header(addr_array) + objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS + if objhdr.tid & GCFLAG_NO_HEAP_PTRS: + objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS + self.prebuilt_root_objects.append(addr_array) return # # 'addr_array' is a raw_malloc'ed array with card markers @@ -807,17 +816,15 @@ return # # We set the flag (even if the newly written address does not - # actually point to the nursery -- like remember_young_pointer()). + # actually point to the nursery, which seems to be ok -- actually + # it seems more important that remember_young_pointer_from_array() + # does not take 3 arguments). addr_byte.char[0] = chr(byte | bitmask) # if objhdr.tid & GCFLAG_CARDS_SET == 0: self.old_objects_with_cards_set.append(addr_array) objhdr.tid |= GCFLAG_CARDS_SET - nocard_logic = func_with_new_name(self.remember_young_pointer, - 'remember_young_pointer_nocard') - del nocard_logic._dont_inline_ - nocard_logic._always_inline_ = True remember_young_pointer_from_array._dont_inline_ = True self.remember_young_pointer_from_array = ( remember_young_pointer_from_array) Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py Fri Oct 1 16:55:30 2010 @@ -86,17 +86,19 @@ def write(self, p, fieldname, newvalue): if self.gc.needs_write_barrier: + newaddr = llmemory.cast_ptr_to_adr(newvalue) addr_struct = llmemory.cast_ptr_to_adr(p) - self.gc.write_barrier(addr_struct) + self.gc.write_barrier(newaddr, addr_struct) setattr(p, fieldname, newvalue) def writearray(self, p, index, newvalue): if self.gc.needs_write_barrier: + newaddr = llmemory.cast_ptr_to_adr(newvalue) addr_struct = llmemory.cast_ptr_to_adr(p) if hasattr(self.gc, 'write_barrier_from_array'): - self.gc.write_barrier_from_array(addr_struct, index) + self.gc.write_barrier_from_array(newvalue, addr_struct, index) else: - self.gc.write_barrier(addr_struct) + self.gc.write_barrier(newaddr, addr_struct) p[index] = newvalue def malloc(self, TYPE, n=None): Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gctransform/framework.py Fri Oct 1 16:55:30 2010 @@ -426,6 +426,7 @@ if GCClass.needs_write_barrier: self.write_barrier_ptr = getfn(GCClass.write_barrier.im_func, [s_gc, + annmodel.SomeAddress(), annmodel.SomeAddress()], annmodel.s_None, inline=True) @@ -434,7 +435,8 @@ # func should not be a bound method, but a real function assert isinstance(func, types.FunctionType) self.write_barrier_failing_case_ptr = getfn(func, - [annmodel.SomeAddress()], + [annmodel.SomeAddress(), + annmodel.SomeAddress()], annmodel.s_None) func = getattr(GCClass, 'write_barrier_from_array', None) if func is not None: @@ -1034,8 +1036,11 @@ v_index]) else: self.write_barrier_calls += 1 + v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], + resulttype = llmemory.Address) hop.genop("direct_call", [self.write_barrier_ptr, self.c_const_gc, + v_newvalue, v_structaddr]) hop.rename('bare_' + opname) Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gcwrapper.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gcwrapper.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gcwrapper.py Fri Oct 1 16:55:30 2010 @@ -94,6 +94,7 @@ assert (type(index) is int # <- fast path or lltype.typeOf(index) == lltype.Signed) self.gc.write_barrier_from_array( + llmemory.cast_ptr_to_adr(newvalue), llmemory.cast_ptr_to_adr(toplevelcontainer), index) wb = False @@ -101,6 +102,7 @@ # if wb: self.gc.write_barrier( + llmemory.cast_ptr_to_adr(newvalue), llmemory.cast_ptr_to_adr(toplevelcontainer)) llheap.setinterior(toplevelcontainer, inneraddr, INNERTYPE, newvalue) From arigo at codespeak.net Fri Oct 1 17:00:25 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 17:00:25 +0200 (CEST) Subject: [pypy-svn] r77531 - pypy/branch/larger-writebarrier/pypy/rpython/memory/gc Message-ID: <20101001150025.6ED33282BDC@codespeak.net> Author: arigo Date: Fri Oct 1 17:00:23 2010 New Revision: 77531 Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py Log: Add appears_to_be_in_nursery() in minimark.py too. Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py Fri Oct 1 17:00:23 2010 @@ -636,6 +636,11 @@ "odd-valued (i.e. tagged) pointer unexpected here") return self.nursery <= addr < self.nursery_top + def appears_to_be_in_nursery(self, addr): + # same as is_in_nursery(), but may return True accidentally if + # 'addr' is a tagged pointer with just the wrong value. + return self.nursery <= addr < self.nursery_top + def is_forwarded(self, obj): """Returns True if the nursery obj is marked as forwarded. Implemented a bit obscurely by checking an unrelated flag @@ -752,23 +757,21 @@ ll_assert(not self.is_in_nursery(addr_struct), "nursery object with GCFLAG_NO_YOUNG_PTRS") # - # We assume that what we are writing is a pointer to the nursery - # (and don't care for the fact that this new pointer may not - # actually point to the nursery, which seems ok - # XXXXXX XXX wrong !! fix me - # ). What we need is + # If it seems that what we are writing is a pointer to the nursery + # (as checked with appears_to_be_in_nursery()), then we need # to remove the flag GCFLAG_NO_YOUNG_PTRS and add the old object # to the list 'old_objects_pointing_to_young'. We know that # 'addr_struct' cannot be in the nursery, because nursery objects # never have the flag GCFLAG_NO_YOUNG_PTRS to start with. - self.old_objects_pointing_to_young.append(addr_struct) - objhdr = self.header(addr_struct) - objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS + if self.appears_to_be_in_nursery(newvalue): + self.old_objects_pointing_to_young.append(addr_struct) + objhdr = self.header(addr_struct) + objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS # # Second part: if 'addr_struct' is actually a prebuilt GC # object and it's the first time we see a write to it, we # add it to the list 'prebuilt_root_objects'. Note that we - # do it even in the (rare?) case of 'addr' being another + # do it even in the (rare?) case of 'addr' being NULL or another # prebuilt object, to simplify code. if objhdr.tid & GCFLAG_NO_HEAP_PTRS: objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS From arigo at codespeak.net Fri Oct 1 17:06:33 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 17:06:33 +0200 (CEST) Subject: [pypy-svn] r77532 - in pypy/branch/larger-writebarrier/pypy/rpython/memory/gc: . test Message-ID: <20101001150633.60755282BDB@codespeak.net> Author: arigo Date: Fri Oct 1 17:06:32 2010 New Revision: 77532 Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py Log: Typos. Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py Fri Oct 1 17:06:32 2010 @@ -753,6 +753,7 @@ # registers, so it does not save and restore them (that's a *hack*!). def remember_young_pointer(addr_struct, newvalue): # 'addr_struct' is the address of the object in which we write. + # 'newvalue' is the address that we are going to write in there. if DEBUG: ll_assert(not self.is_in_nursery(addr_struct), "nursery object with GCFLAG_NO_YOUNG_PTRS") @@ -763,9 +764,9 @@ # to the list 'old_objects_pointing_to_young'. We know that # 'addr_struct' cannot be in the nursery, because nursery objects # never have the flag GCFLAG_NO_YOUNG_PTRS to start with. + objhdr = self.header(addr_struct) if self.appears_to_be_in_nursery(newvalue): self.old_objects_pointing_to_young.append(addr_struct) - objhdr = self.header(addr_struct) objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS # # Second part: if 'addr_struct' is actually a prebuilt GC @@ -785,6 +786,7 @@ def _init_writebarrier_with_card_marker(self): + DEBUG = self.DEBUG def remember_young_pointer_from_array(addr_array, index): # 'addr_array' is the address of the object in which we write, # which must have an array part; 'index' is the index of the Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py Fri Oct 1 17:06:32 2010 @@ -96,7 +96,7 @@ newaddr = llmemory.cast_ptr_to_adr(newvalue) addr_struct = llmemory.cast_ptr_to_adr(p) if hasattr(self.gc, 'write_barrier_from_array'): - self.gc.write_barrier_from_array(newvalue, addr_struct, index) + self.gc.write_barrier_from_array(newaddr, addr_struct, index) else: self.gc.write_barrier(newaddr, addr_struct) p[index] = newvalue From antocuni at codespeak.net Fri Oct 1 17:15:59 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 1 Oct 2010 17:15:59 +0200 (CEST) Subject: [pypy-svn] r77533 - in pypy/branch/jitffi/pypy/rlib: . test Message-ID: <20101001151559.65F0B282BDB@codespeak.net> Author: antocuni Date: Fri Oct 1 17:15:57 2010 New Revision: 77533 Modified: pypy/branch/jitffi/pypy/rlib/libffi.py pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py Log: remove the dependency on clibffi.FuncPtr: now the new libffi.Func does the work by itself. This temporarly breaks some jit test, will fix them later Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Fri Oct 1 17:15:57 2010 @@ -1,7 +1,11 @@ -from pypy.rlib import clibffi -from pypy.rlib.clibffi import get_libc_name +from pypy.rpython.lltypesystem import rffi, lltype from pypy.rlib.objectmodel import specialize +from pypy.rlib.rarithmetic import intmask, r_uint from pypy.rlib import jit +from pypy.rlib import clibffi +from pypy.rlib.clibffi import get_libc_name, FUNCFLAG_CDECL, AbstractFuncPtr, \ + push_arg_as_ffiptr, c_ffi_call +from pypy.rlib.rdynload import dlopen, dlclose, dlsym, dlsym_byordinal def import_types(): g = globals() @@ -12,6 +16,8 @@ del import_types +# ---------------------------------------------------------------------- + class AbstractArg(object): next = None @@ -22,8 +28,8 @@ def __init__(self, intval): self.intval = intval - def push(self, func): - func._push_arg(self.intval) + def push(self, func, ll_args, i): + func._push_arg(self.intval, ll_args, i) class FloatArg(AbstractArg): """ An argument holding a float @@ -32,45 +38,111 @@ def __init__(self, floatval): self.floatval = floatval - def push(self, func): - func._push_arg(self.floatval) + def push(self, func, ll_args, i): + func._push_arg(self.floatval, ll_args, i) + -class Func(object): +class Func(AbstractFuncPtr): - _immutable_fields_ = ['funcptr', 'funcsym', 'argtypes', 'restype'] + _immutable_fields_ = ['funcsym', 'argtypes', 'restype'] - def __init__(self, funcptr): - # XXX: for now, this is just a wrapper around clibffi.FuncPtr, but in - # the future it will replace it completely - self.funcptr = funcptr - self.funcsym = funcptr.funcsym - self.argtypes = funcptr.argtypes - self.restype = funcptr.restype + def __init__(self, name, argtypes, restype, funcsym, flags=FUNCFLAG_CDECL, + keepalive=None): + AbstractFuncPtr.__init__(self, name, argtypes, restype, flags) + self.keepalive = keepalive + self.funcsym = funcsym def _prepare(self): - pass + ll_args = lltype.malloc(rffi.VOIDPP.TO, len(self.argtypes), flavor='raw') + return ll_args _prepare.oopspec = 'libffi_prepare_call(self)' - def _push_arg(self, value): - self.funcptr.push_arg(value) + def _push_arg(self, value, ll_args, i): + # XXX: check the type is not translated? + argtype = self.argtypes[i] + c_size = intmask(argtype.c_size) + ll_buf = lltype.malloc(rffi.CCHARP.TO, c_size, flavor='raw') + push_arg_as_ffiptr(argtype, value, ll_buf) + ll_args[i] = ll_buf # XXX this is bad, fix it somehow in the future, but specialize:argtype # doesn't work correctly with mixing non-negative and normal integers #_push_arg._annenforceargs_ = [None, int] _push_arg._annspecialcase_ = 'specialize:argtype(1)' - _push_arg.oopspec = 'libffi_push_arg(self, value)' + _push_arg.oopspec = 'libffi_push_arg(self, value, ll_args, i)' - def _do_call(self, funcsym, RESULT): - return self.funcptr.call(RESULT) + def _do_call(self, funcsym, ll_args, RESULT): + # XXX: check len(args)? + ll_result = lltype.nullptr(rffi.CCHARP.TO) + if self.restype != ffi_type_void: + ll_result = lltype.malloc(rffi.CCHARP.TO, + intmask(self.restype.c_size), + flavor='raw') + ffires = c_ffi_call(self.ll_cif, + self.funcsym, + rffi.cast(rffi.VOIDP, ll_result), + rffi.cast(rffi.VOIDPP, ll_args)) + if RESULT is not lltype.Void: + TP = lltype.Ptr(rffi.CArray(RESULT)) + res = rffi.cast(TP, ll_result)[0] + else: + res = None + self._free_buffers(ll_result, ll_args) + #check_fficall_result(ffires, self.flags) + return res _do_call._annspecialcase_ = 'specialize:arg(2)' _do_call.oopspec = 'libffi_call(self, funcsym, RESULT)' + def _free_buffers(self, ll_result, ll_args): + lltype.free(ll_result, flavor='raw') + for i in range(len(self.argtypes)): + lltype.free(ll_args[i], flavor='raw') + lltype.free(ll_args, flavor='raw') + @jit.unroll_safe @specialize.arg(2) def call(self, argchain, RESULT): - self._prepare() + # WARNING! This code is written carefully in a way that the JIT + # optimizer will see a sequence of calls like the following: + # + # libffi_prepare_call + # libffi_push_arg + # libffi_push_arg + # ... + # libffi_call + # + # It is important that there is no other operation in the middle, else + # the optimizer will fail to recognize the pattern and won't turn it + # into a fast CALL. Note that "arg = arg.next" is optimized away, + # assuming that archain is completely virtual. + ll_args = self._prepare() + i = 0 arg = argchain while arg: - arg.push(self) + arg.push(self, ll_args, i) + i += 1 arg = arg.next - #return self.funcptr.call(RESULT) - return self._do_call(self.funcsym, RESULT) + result = self._do_call(self.funcsym, ll_args, RESULT) + return result + +# ---------------------------------------------------------------------- + + +# XXX: it partially duplicate the code in clibffi.py +class CDLL(object): + def __init__(self, libname): + """Load the library, or raises DLOpenError.""" + self.lib = lltype.nullptr(rffi.CCHARP.TO) + ll_libname = rffi.str2charp(libname) + try: + self.lib = dlopen(ll_libname) + finally: + lltype.free(ll_libname, flavor='raw') + + def __del__(self): + if self.lib: + dlclose(self.lib) + self.lib = lltype.nullptr(rffi.CCHARP.TO) + + def getpointer(self, name, argtypes, restype, flags=FUNCFLAG_CDECL): + return Func(name, argtypes, restype, dlsym(self.lib, name), + flags=flags, keepalive=self) Modified: pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py Fri Oct 1 17:15:57 2010 @@ -11,13 +11,6 @@ import sys import time -def setup_module(mod): - for name in type_names: - # XXX force this to be seen by ll2ctypes - # so that ALLOCATED.clear() clears it - ffistruct = globals()[name] - rffi.cast(rffi.VOIDP, ffistruct) - def get_libm_name(platform): if platform == 'win32': return 'msvcrt.dll' @@ -27,17 +20,26 @@ return 'libm.so' class BaseFfiTest(object): + + def setup_class(cls): + for name in type_names: + # XXX force this to be seen by ll2ctypes + # so that ALLOCATED.clear() clears it + ffistruct = globals()[name] + rffi.cast(rffi.VOIDP, ffistruct) + + def setup_method(self, meth): + ALLOCATED.clear() + + +class TestCLibffi(BaseFfiTest): + def get_libc(self): return CDLL(get_libc_name()) def get_libm(self): return CDLL(get_libm_name(sys.platform)) - -class TestCLibffi(BaseFfiTest): - def setup_method(self, meth): - ALLOCATED.clear() - def test_library_open(self): lib = self.get_libc() del lib @@ -411,10 +413,11 @@ assert not ALLOCATED -class TestWin32Handles: +class TestWin32Handles(BaseFfiTest): def setup_class(cls): if sys.platform != 'win32': py.test.skip("Handle to libc library, Win-only test") + BaseFfiTest.setup_class(cls) def test_get_libc_handle(self): handle = get_libc_handle() From antocuni at codespeak.net Fri Oct 1 17:22:59 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 1 Oct 2010 17:22:59 +0200 (CEST) Subject: [pypy-svn] r77534 - in pypy/branch/jitffi/pypy: jit/codewriter jit/metainterp/test rlib Message-ID: <20101001152259.37DCA282BDB@codespeak.net> Author: antocuni Date: Fri Oct 1 17:22:57 2010 New Revision: 77534 Modified: pypy/branch/jitffi/pypy/jit/codewriter/support.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: fix the jit and the test, after the refactoring of libffi.py Modified: pypy/branch/jitffi/pypy/jit/codewriter/support.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/support.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/support.py Fri Oct 1 17:22:57 2010 @@ -222,18 +222,20 @@ # libffi support # -------------- -def _ll_1_libffi_prepare_call(func): - pass +def _ll_1_libffi_prepare_call(llfunc): + from pypy.rlib.libffi import Func + func = cast_base_ptr_to_instance(Func, llfunc) + return func._prepare() -def _ll_2_libffi_push_arg(llfunc, value): +def _ll_4_libffi_push_arg(llfunc, value, ll_args, i): from pypy.rlib.libffi import Func func = cast_base_ptr_to_instance(Func, llfunc) - return func._push_arg(value) + return func._push_arg(value, ll_args, i) -def _ll_3_libffi_call(llfunc, funcsym, RESULT): +def _ll_4_libffi_call(llfunc, funcsym, ll_args, RESULT): from pypy.rlib.libffi import Func func = cast_base_ptr_to_instance(Func, llfunc) - return func._do_call(funcsym, lltype.Signed) + return func._do_call(funcsym, ll_args, lltype.Signed) # XXX: should be RESULT, but it doesn't work Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py Fri Oct 1 17:22:57 2010 @@ -2,8 +2,7 @@ import py from pypy.rlib.jit import JitDriver, hint from pypy.jit.metainterp.test.test_basic import LLJitMixin -from pypy.rlib.clibffi import FuncPtr, CDLL, ffi_type_sint -from pypy.rlib.libffi import IntArg, Func +from pypy.rlib.libffi import CDLL, ffi_type_sint, IntArg, Func from pypy.tool.udir import udir from pypy.translator.tool.cbuild import ExternalCompilationInfo from pypy.translator.platform import platform @@ -29,9 +28,8 @@ def f(n): cdll = CDLL(self.lib_name) - fn = cdll.getpointer('sum_xy', [ffi_type_sint, ffi_type_sint], - ffi_type_sint) - func = Func(fn) + func = cdll.getpointer('sum_xy', [ffi_type_sint, ffi_type_sint], + ffi_type_sint) while n < 10: driver.jit_merge_point(n=n, func=func) driver.can_enter_jit(n=n, func=func) Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Fri Oct 1 17:22:57 2010 @@ -66,8 +66,8 @@ ll_args[i] = ll_buf # XXX this is bad, fix it somehow in the future, but specialize:argtype # doesn't work correctly with mixing non-negative and normal integers - #_push_arg._annenforceargs_ = [None, int] - _push_arg._annspecialcase_ = 'specialize:argtype(1)' + _push_arg._annenforceargs_ = [None, int, None, int] + #_push_arg._annspecialcase_ = 'specialize:argtype(1)' _push_arg.oopspec = 'libffi_push_arg(self, value, ll_args, i)' def _do_call(self, funcsym, ll_args, RESULT): @@ -89,8 +89,8 @@ self._free_buffers(ll_result, ll_args) #check_fficall_result(ffires, self.flags) return res - _do_call._annspecialcase_ = 'specialize:arg(2)' - _do_call.oopspec = 'libffi_call(self, funcsym, RESULT)' + _do_call._annspecialcase_ = 'specialize:arg(3)' + _do_call.oopspec = 'libffi_call(self, funcsym, ll_args, RESULT)' def _free_buffers(self, ll_result, ll_args): lltype.free(ll_result, flavor='raw') From arigo at codespeak.net Fri Oct 1 17:23:52 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 17:23:52 +0200 (CEST) Subject: [pypy-svn] r77535 - pypy/branch/larger-writebarrier/pypy/rpython/memory/gctransform Message-ID: <20101001152352.67787282BDB@codespeak.net> Author: arigo Date: Fri Oct 1 17:23:50 2010 New Revision: 77535 Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gctransform/framework.py Log: Fix: write_barrier_from_array() always takes 4 arguments now (but is inlined and may call remember_young_pointer_from_array() with less arguments). Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gctransform/framework.py Fri Oct 1 17:23:50 2010 @@ -443,6 +443,7 @@ self.write_barrier_from_array_ptr = getfn(func.im_func, [s_gc, annmodel.SomeAddress(), + annmodel.SomeAddress(), annmodel.SomeInteger()], annmodel.s_None, inline=True) @@ -1023,6 +1024,8 @@ and not isinstance(v_newvalue, Constant) and v_struct.concretetype.TO._gckind == "gc" and hop.spaceop not in self.clean_sets): + v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], + resulttype = llmemory.Address) v_structaddr = hop.genop("cast_ptr_to_adr", [v_struct], resulttype = llmemory.Address) if (self.write_barrier_from_array_ptr is not None and @@ -1032,12 +1035,11 @@ assert v_index.concretetype == lltype.Signed hop.genop("direct_call", [self.write_barrier_from_array_ptr, self.c_const_gc, + v_newvalue, v_structaddr, v_index]) else: self.write_barrier_calls += 1 - v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], - resulttype = llmemory.Address) hop.genop("direct_call", [self.write_barrier_ptr, self.c_const_gc, v_newvalue, From arigo at codespeak.net Fri Oct 1 17:54:06 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 17:54:06 +0200 (CEST) Subject: [pypy-svn] r77536 - in pypy/branch/larger-writebarrier/pypy/rpython/memory: . gc gc/test test Message-ID: <20101001155406.A04FB282BDB@codespeak.net> Author: arigo Date: Fri Oct 1 17:54:04 2010 New Revision: 77536 Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/base.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/generation.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/markcompact.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/marksweep.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/semispace.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py pypy/branch/larger-writebarrier/pypy/rpython/memory/gcwrapper.py pypy/branch/larger-writebarrier/pypy/rpython/memory/test/test_transformed_gc.py Log: Fix. This really requires doing something different if we are translated to C or not. Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/base.py Fri Oct 1 17:54:04 2010 @@ -20,12 +20,15 @@ prebuilt_gc_objects_are_static_roots = True object_minimal_size = 0 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE): + def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + translated_to_c=True): self.gcheaderbuilder = GCHeaderBuilder(self.HDR) self.AddressStack = get_address_stack(chunk_size) self.AddressDeque = get_address_deque(chunk_size) self.AddressDict = AddressDict self.config = config + assert isinstance(translated_to_c, bool) + self.translated_to_c = translated_to_c def setup(self): # all runtime mutable values' setup should happen here Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/generation.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/generation.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/generation.py Fri Oct 1 17:54:04 2010 @@ -5,7 +5,6 @@ from pypy.rpython.memory.gc.base import read_from_env from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage from pypy.rpython.lltypesystem import lltype, llmemory, llarena -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rlib.objectmodel import free_non_gc_object from pypy.rlib.debug import ll_assert from pypy.rlib.debug import debug_print, debug_start, debug_stop @@ -49,15 +48,17 @@ nursery_hash_base = -1 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + def __init__(self, config, nursery_size=32*WORD, min_nursery_size=32*WORD, auto_nursery_size=False, space_size=1024*WORD, - max_space_size=sys.maxint//2+1): - SemiSpaceGC.__init__(self, config, chunk_size = chunk_size, + max_space_size=sys.maxint//2+1, + **kwds): + SemiSpaceGC.__init__(self, config, space_size = space_size, - max_space_size = max_space_size) + max_space_size = max_space_size, + **kwds) assert min_nursery_size <= nursery_size <= space_size // 2 self.initial_nursery_size = nursery_size self.auto_nursery_size = auto_nursery_size @@ -160,6 +161,9 @@ def appears_to_be_in_nursery(self, addr): # same as is_in_nursery(), but may return True accidentally if # 'addr' is a tagged pointer with just the wrong value. + if not self.translated_to_c: + if not self.is_valid_gc_object(addr): + return False return self.nursery <= addr < self.nursery_top def malloc_fixedsize_clear(self, typeid, size, can_collect, Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/markcompact.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/markcompact.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/markcompact.py Fri Oct 1 17:54:04 2010 @@ -2,7 +2,6 @@ from pypy.rpython.memory.gc.base import MovingGCBase, read_from_env from pypy.rlib.debug import ll_assert, have_debug_prints from pypy.rlib.debug import debug_print, debug_start, debug_stop -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack, get_address_deque from pypy.rpython.memory.support import AddressDict from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage @@ -86,9 +85,9 @@ free = NULL next_collect_after = -1 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, space_size=4096, - min_next_collect_after=128): - MovingGCBase.__init__(self, config, chunk_size) + def __init__(self, config, space_size=4096, + min_next_collect_after=128, **kwds): + MovingGCBase.__init__(self, config, **kwds) self.space_size = space_size self.min_next_collect_after = min_next_collect_after Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/marksweep.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/marksweep.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/marksweep.py Fri Oct 1 17:54:04 2010 @@ -1,7 +1,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc, raw_free from pypy.rpython.lltypesystem.llmemory import raw_memcopy, raw_memclear from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack from pypy.rpython.memory.gcheader import GCHeaderBuilder from pypy.rpython.lltypesystem import lltype, llmemory, rffi, llgroup @@ -48,9 +47,9 @@ # translating to a real backend. TRANSLATION_PARAMS = {'start_heap_size': 8*1024*1024} # XXX adjust - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, start_heap_size=4096): + def __init__(self, config, start_heap_size=4096, **kwds): self.param_start_heap_size = start_heap_size - GCBase.__init__(self, config, chunk_size) + GCBase.__init__(self, config, **kwds) def setup(self): GCBase.setup(self) @@ -714,8 +713,8 @@ _alloc_flavor_ = "raw" COLLECT_EVERY = 2000 - def __init__(self, chunk_size=DEFAULT_CHUNK_SIZE, start_heap_size=4096): - MarkSweepGC.__init__(self, chunk_size, start_heap_size) + def __init__(self, config, **kwds): + MarkSweepGC.__init__(self, config, **kwds) self.count_mallocs = 0 def maybe_collect(self): Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/minimark.py Fri Oct 1 17:54:04 2010 @@ -4,7 +4,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc_usage from pypy.rpython.memory.gc.base import GCBase, MovingGCBase from pypy.rpython.memory.gc import minimarkpage, base, generation -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rlib.rarithmetic import ovfcheck, LONG_BIT, intmask, r_uint from pypy.rlib.rarithmetic import LONG_BIT_SHIFT from pypy.rlib.debug import ll_assert, debug_print, debug_start, debug_stop @@ -140,7 +139,7 @@ "large_object_gcptrs": 8250*WORD, } - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + def __init__(self, config, read_from_env=False, nursery_size=32*WORD, page_size=16*WORD, @@ -150,8 +149,9 @@ card_page_indices=0, large_object=8*WORD, large_object_gcptrs=10*WORD, - ArenaCollectionClass=None): - MovingGCBase.__init__(self, config, chunk_size) + ArenaCollectionClass=None, + **kwds): + MovingGCBase.__init__(self, config, **kwds) assert small_request_threshold % WORD == 0 self.read_from_env = read_from_env self.nursery_size = nursery_size @@ -639,6 +639,9 @@ def appears_to_be_in_nursery(self, addr): # same as is_in_nursery(), but may return True accidentally if # 'addr' is a tagged pointer with just the wrong value. + if not self.translated_to_c: + if not self.is_valid_gc_object(addr): + return False return self.nursery <= addr < self.nursery_top def is_forwarded(self, obj): Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/semispace.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/semispace.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/semispace.py Fri Oct 1 17:54:04 2010 @@ -1,7 +1,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc, raw_free from pypy.rpython.lltypesystem.llmemory import raw_memcopy, raw_memclear from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack, get_address_deque from pypy.rpython.memory.support import AddressDict from pypy.rpython.lltypesystem import lltype, llmemory, llarena, rffi, llgroup @@ -59,11 +58,11 @@ # translating to a real backend. TRANSLATION_PARAMS = {'space_size': 8*1024*1024} # XXX adjust - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, space_size=4096, - max_space_size=sys.maxint//2+1): + def __init__(self, config, space_size=4096, max_space_size=sys.maxint//2+1, + **kwds): self.param_space_size = space_size self.param_max_space_size = max_space_size - MovingGCBase.__init__(self, config, chunk_size) + MovingGCBase.__init__(self, config, **kwds) def setup(self): #self.total_collection_time = 0.0 Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gc/test/test_direct.py Fri Oct 1 17:54:04 2010 @@ -70,6 +70,7 @@ GC_PARAMS = self.GC_PARAMS.copy() if hasattr(meth, 'GC_PARAMS'): GC_PARAMS.update(meth.GC_PARAMS) + GC_PARAMS['translated_to_c'] = False self.gc = self.GCClass(config, **GC_PARAMS) self.gc.DEBUG = True self.rootwalker = DirectRootWalker(self) Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/gcwrapper.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/gcwrapper.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/gcwrapper.py Fri Oct 1 17:54:04 2010 @@ -9,7 +9,10 @@ def __init__(self, llinterp, flowgraphs, gc_class, GC_PARAMS={}): translator = llinterp.typer.annotator.translator config = translator.config.translation - self.gc = gc_class(config, chunk_size = 10, **GC_PARAMS) + self.gc = gc_class(config, + chunk_size = 10, + translated_to_c = False, + **GC_PARAMS) self.gc.set_root_walker(LLInterpRootWalker(self)) self.gc.DEBUG = True self.llinterp = llinterp Modified: pypy/branch/larger-writebarrier/pypy/rpython/memory/test/test_transformed_gc.py ============================================================================== --- pypy/branch/larger-writebarrier/pypy/rpython/memory/test/test_transformed_gc.py (original) +++ pypy/branch/larger-writebarrier/pypy/rpython/memory/test/test_transformed_gc.py Fri Oct 1 17:54:04 2010 @@ -906,7 +906,8 @@ gcname = "marksweep" class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 @@ -1144,7 +1145,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.marksweep import PrintingMarkSweepGC as GCClass - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestSemiSpaceGC(GenericMovingGCTests): @@ -1154,7 +1156,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.semispace import SemiSpaceGC as GCClass - GC_PARAMS = {'space_size': 512*WORD} + GC_PARAMS = {'space_size': 512*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestMarkCompactGC(GenericMovingGCTests): @@ -1163,7 +1166,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.markcompact import MarkCompactGC as GCClass - GC_PARAMS = {'space_size': 4096*WORD} + GC_PARAMS = {'space_size': 4096*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestGenerationGC(GenericMovingGCTests): @@ -1175,7 +1179,8 @@ from pypy.rpython.memory.gc.generation import GenerationGC as \ GCClass GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 32*WORD} + 'nursery_size': 32*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_weakref_across_minor_collection(cls): @@ -1372,7 +1377,8 @@ GenerationGC._teardown(self) GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 128*WORD} + 'nursery_size': 128*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_working_nursery(cls): @@ -1404,7 +1410,8 @@ from pypy.rpython.memory.gc.hybrid import HybridGC as GCClass GC_PARAMS = {'space_size': 512*WORD, 'nursery_size': 32*WORD, - 'large_object': 8*WORD} + 'large_object': 8*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_ref_from_rawmalloced_to_regular(cls): @@ -1477,6 +1484,7 @@ 'large_object': 8*WORD, 'large_object_gcptrs': 10*WORD, 'card_page_indices': 4, + 'translated_to_c': False, } root_stack_depth = 200 @@ -1585,7 +1593,8 @@ gcname = "marksweep" class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestHybridTaggedPointerGC(TaggedPointerGCTests): @@ -1596,7 +1605,8 @@ from pypy.rpython.memory.gc.generation import GenerationGC as \ GCClass GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 32*WORD} + 'nursery_size': 32*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestMarkCompactTaggedpointerGC(TaggedPointerGCTests): @@ -1605,5 +1615,6 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.markcompact import MarkCompactGC as GCClass - GC_PARAMS = {'space_size': 4096*WORD} + GC_PARAMS = {'space_size': 4096*WORD, + 'translated_to_c': False} root_stack_depth = 200 From arigo at codespeak.net Fri Oct 1 17:56:04 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 1 Oct 2010 17:56:04 +0200 (CEST) Subject: [pypy-svn] r77537 - in pypy/trunk/pypy: jit/backend/llgraph jit/backend/llsupport jit/backend/llsupport/test jit/backend/test jit/backend/x86 jit/metainterp rpython/memory rpython/memory/gc rpython/memory/gc/test rpython/memory/gctransform rpython/memory/test Message-ID: <20101001155604.74639282BDB@codespeak.net> Author: arigo Date: Fri Oct 1 17:56:02 2010 New Revision: 77537 Modified: pypy/trunk/pypy/jit/backend/llgraph/llimpl.py pypy/trunk/pypy/jit/backend/llsupport/gc.py pypy/trunk/pypy/jit/backend/llsupport/test/test_gc.py pypy/trunk/pypy/jit/backend/test/runner_test.py pypy/trunk/pypy/jit/backend/x86/assembler.py pypy/trunk/pypy/jit/backend/x86/regalloc.py pypy/trunk/pypy/jit/metainterp/resoperation.py pypy/trunk/pypy/rpython/memory/gc/base.py pypy/trunk/pypy/rpython/memory/gc/generation.py pypy/trunk/pypy/rpython/memory/gc/markcompact.py pypy/trunk/pypy/rpython/memory/gc/marksweep.py pypy/trunk/pypy/rpython/memory/gc/minimark.py pypy/trunk/pypy/rpython/memory/gc/semispace.py pypy/trunk/pypy/rpython/memory/gc/test/test_direct.py pypy/trunk/pypy/rpython/memory/gctransform/framework.py pypy/trunk/pypy/rpython/memory/gcwrapper.py pypy/trunk/pypy/rpython/memory/test/test_transformed_gc.py Log: Merge branch/larger-writebarrier: It seems that "smaller-writebarrier" may have had a performance impact on a few benchmarks. This is a mixed solution: using the old two-args write barrier for regular setfields, and the smaller (2 args instead of 3) version for setarrayitem. Modified: pypy/trunk/pypy/jit/backend/llgraph/llimpl.py ============================================================================== --- pypy/trunk/pypy/jit/backend/llgraph/llimpl.py (original) +++ pypy/trunk/pypy/jit/backend/llgraph/llimpl.py Fri Oct 1 17:56:02 2010 @@ -129,7 +129,7 @@ 'arraylen_gc' : (('ref',), 'int'), 'call' : (('ref', 'varargs'), 'intorptr'), 'call_assembler' : (('varargs',), 'intorptr'), - 'cond_call_gc_wb' : (('ptr',), None), + 'cond_call_gc_wb' : (('ptr', 'ptr'), None), 'oosend' : (('varargs',), 'intorptr'), 'oosend_pure' : (('varargs',), 'intorptr'), 'guard_true' : (('bool',), None), @@ -810,7 +810,7 @@ FLOAT: 0.0} return d[calldescr.typeinfo] - def op_cond_call_gc_wb(self, descr, a): + def op_cond_call_gc_wb(self, descr, a, b): py.test.skip("cond_call_gc_wb not supported") def op_oosend(self, descr, obj, *args): Modified: pypy/trunk/pypy/jit/backend/llsupport/gc.py ============================================================================== --- pypy/trunk/pypy/jit/backend/llsupport/gc.py (original) +++ pypy/trunk/pypy/jit/backend/llsupport/gc.py Fri Oct 1 17:56:02 2010 @@ -404,7 +404,7 @@ self.GC_MALLOC_BASIC = lltype.Ptr(lltype.FuncType( [lltype.Signed, lltype.Signed], llmemory.GCREF)) self.WB_FUNCPTR = lltype.Ptr(lltype.FuncType( - [llmemory.Address], lltype.Void)) + [llmemory.Address, llmemory.Address], lltype.Void)) self.write_barrier_descr = WriteBarrierDescr(self) # def malloc_array(itemsize, tid, num_elem): @@ -550,7 +550,8 @@ # the GC, and call it immediately llop1 = self.llop1 funcptr = llop1.get_write_barrier_failing_case(self.WB_FUNCPTR) - funcptr(llmemory.cast_ptr_to_adr(gcref_struct)) + funcptr(llmemory.cast_ptr_to_adr(gcref_struct), + llmemory.cast_ptr_to_adr(gcref_newptr)) def rewrite_assembler(self, cpu, operations): # Perform two kinds of rewrites in parallel: @@ -589,7 +590,7 @@ v = op.getarg(1) if isinstance(v, BoxPtr) or (isinstance(v, ConstPtr) and bool(v.value)): # store a non-NULL - self._gen_write_barrier(newops, op.getarg(0)) + self._gen_write_barrier(newops, op.getarg(0), v) op = op.copy_and_change(rop.SETFIELD_RAW) # ---------- write barrier for SETARRAYITEM_GC ---------- if op.getopnum() == rop.SETARRAYITEM_GC: @@ -598,15 +599,15 @@ bool(v.value)): # store a non-NULL # XXX detect when we should produce a # write_barrier_from_array - self._gen_write_barrier(newops, op.getarg(0)) + self._gen_write_barrier(newops, op.getarg(0), v) op = op.copy_and_change(rop.SETARRAYITEM_RAW) # ---------- newops.append(op) del operations[:] operations.extend(newops) - def _gen_write_barrier(self, newops, v_base): - args = [v_base] + def _gen_write_barrier(self, newops, v_base, v_value): + args = [v_base, v_value] newops.append(ResOperation(rop.COND_CALL_GC_WB, args, None, descr=self.write_barrier_descr)) Modified: pypy/trunk/pypy/jit/backend/llsupport/test/test_gc.py ============================================================================== --- pypy/trunk/pypy/jit/backend/llsupport/test/test_gc.py (original) +++ pypy/trunk/pypy/jit/backend/llsupport/test/test_gc.py Fri Oct 1 17:56:02 2010 @@ -141,8 +141,8 @@ repr(offset_to_length), p)) return p - def _write_barrier_failing_case(self, adr_struct): - self.record.append(('barrier', adr_struct)) + def _write_barrier_failing_case(self, adr_struct, adr_newptr): + self.record.append(('barrier', adr_struct, adr_newptr)) def get_write_barrier_failing_case(self, FPTRTYPE): return llhelper(FPTRTYPE, self._write_barrier_failing_case) @@ -239,6 +239,7 @@ s_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, s) r_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, r) s_adr = llmemory.cast_ptr_to_adr(s) + r_adr = llmemory.cast_ptr_to_adr(r) # s_hdr.tid &= ~gc_ll_descr.GCClass.JIT_WB_IF_FLAG gc_ll_descr.do_write_barrier(s_gcref, r_gcref) @@ -246,7 +247,7 @@ # s_hdr.tid |= gc_ll_descr.GCClass.JIT_WB_IF_FLAG gc_ll_descr.do_write_barrier(s_gcref, r_gcref) - assert self.llop1.record == [('barrier', s_adr)] + assert self.llop1.record == [('barrier', s_adr, r_adr)] def test_gen_write_barrier(self): gc_ll_descr = self.gc_ll_descr @@ -254,11 +255,13 @@ # newops = [] v_base = BoxPtr() - gc_ll_descr._gen_write_barrier(newops, v_base) + v_value = BoxPtr() + gc_ll_descr._gen_write_barrier(newops, v_base, v_value) assert llop1.record == [] assert len(newops) == 1 assert newops[0].getopnum() == rop.COND_CALL_GC_WB assert newops[0].getarg(0) == v_base + assert newops[0].getarg(1) == v_value assert newops[0].result is None wbdescr = newops[0].getdescr() assert isinstance(wbdescr.jit_wb_if_flag, int) @@ -358,6 +361,7 @@ # assert operations[0].getopnum() == rop.COND_CALL_GC_WB assert operations[0].getarg(0) == v_base + assert operations[0].getarg(1) == v_value assert operations[0].result is None # assert operations[1].getopnum() == rop.SETFIELD_RAW @@ -381,6 +385,7 @@ # assert operations[0].getopnum() == rop.COND_CALL_GC_WB assert operations[0].getarg(0) == v_base + assert operations[0].getarg(1) == v_value assert operations[0].result is None # assert operations[1].getopnum() == rop.SETARRAYITEM_RAW Modified: pypy/trunk/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/trunk/pypy/jit/backend/test/runner_test.py (original) +++ pypy/trunk/pypy/jit/backend/test/runner_test.py Fri Oct 1 17:56:02 2010 @@ -1427,12 +1427,12 @@ assert not excvalue def test_cond_call_gc_wb(self): - def func_void(a): - record.append(a) + def func_void(a, b): + record.append((a, b)) record = [] # S = lltype.GcStruct('S', ('tid', lltype.Signed)) - FUNC = self.FuncType([lltype.Ptr(S)], lltype.Void) + FUNC = self.FuncType([lltype.Ptr(S), lltype.Signed], lltype.Void) func_ptr = llhelper(lltype.Ptr(FUNC), func_void) funcbox = self.get_funcbox(self.cpu, func_ptr) class WriteBarrierDescr(AbstractDescr): @@ -1453,10 +1453,10 @@ sgcref = lltype.cast_opaque_ptr(llmemory.GCREF, s) del record[:] self.execute_operation(rop.COND_CALL_GC_WB, - [BoxPtr(sgcref)], + [BoxPtr(sgcref), ConstInt(-2121)], 'void', descr=WriteBarrierDescr()) if cond: - assert record == [s] + assert record == [(s, -2121)] else: assert record == [] Modified: pypy/trunk/pypy/jit/backend/x86/assembler.py ============================================================================== --- pypy/trunk/pypy/jit/backend/x86/assembler.py (original) +++ pypy/trunk/pypy/jit/backend/x86/assembler.py Fri Oct 1 17:56:02 2010 @@ -1780,11 +1780,12 @@ self.mc.PUSH_i32(loc.getint()) if IS_X86_64: - # We clobber this register to pass the arguments, but that's + # We clobber these registers to pass the arguments, but that's # okay, because consider_cond_call_gc_wb makes sure that any # caller-save registers with values in them are present in arglocs, # so they are saved on the stack above and restored below self.mc.MOV_rs(edi.value, 0) + self.mc.MOV_rs(esi.value, 8) # misaligned stack in the call, but it's ok because the write barrier # is not going to call anything more. Also, this assumes that the Modified: pypy/trunk/pypy/jit/backend/x86/regalloc.py ============================================================================== --- pypy/trunk/pypy/jit/backend/x86/regalloc.py (original) +++ pypy/trunk/pypy/jit/backend/x86/regalloc.py Fri Oct 1 17:56:02 2010 @@ -696,9 +696,13 @@ def consider_cond_call_gc_wb(self, op): assert op.result is None args = op.getarglist() + loc_newvalue = self.rm.make_sure_var_in_reg(op.getarg(1), args) + # ^^^ we force loc_newvalue in a reg (unless it's a Const), + # because it will be needed anyway by the following setfield_gc. + # It avoids loading it twice from the memory. loc_base = self.rm.make_sure_var_in_reg(op.getarg(0), args, imm_fine=False) - arglocs = [loc_base] + arglocs = [loc_base, loc_newvalue] # add eax, ecx and edx as extra "arguments" to ensure they are # saved and restored. Fish in self.rm to know which of these # registers really need to be saved (a bit of a hack). Moreover, Modified: pypy/trunk/pypy/jit/metainterp/resoperation.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/resoperation.py (original) +++ pypy/trunk/pypy/jit/metainterp/resoperation.py Fri Oct 1 17:56:02 2010 @@ -455,7 +455,7 @@ 'UNICODESETITEM/3', 'NEWUNICODE/1', #'RUNTIMENEW/1', # ootype operation - 'COND_CALL_GC_WB/1d', # [objptr] (for the write barrier) + 'COND_CALL_GC_WB/2d', # [objptr, newvalue] (for the write barrier) 'DEBUG_MERGE_POINT/1', # debugging only 'VIRTUAL_REF_FINISH/2', # removed before it's passed to the backend 'COPYSTRCONTENT/5', # src, dst, srcstart, dststart, length Modified: pypy/trunk/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/trunk/pypy/rpython/memory/gc/base.py (original) +++ pypy/trunk/pypy/rpython/memory/gc/base.py Fri Oct 1 17:56:02 2010 @@ -20,12 +20,15 @@ prebuilt_gc_objects_are_static_roots = True object_minimal_size = 0 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE): + def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + translated_to_c=True): self.gcheaderbuilder = GCHeaderBuilder(self.HDR) self.AddressStack = get_address_stack(chunk_size) self.AddressDeque = get_address_deque(chunk_size) self.AddressDict = AddressDict self.config = config + assert isinstance(translated_to_c, bool) + self.translated_to_c = translated_to_c def setup(self): # all runtime mutable values' setup should happen here @@ -79,7 +82,7 @@ def set_root_walker(self, root_walker): self.root_walker = root_walker - def write_barrier(self, addr_struct): + def write_barrier(self, newvalue, addr_struct): pass def statistics(self, index): Modified: pypy/trunk/pypy/rpython/memory/gc/generation.py ============================================================================== --- pypy/trunk/pypy/rpython/memory/gc/generation.py (original) +++ pypy/trunk/pypy/rpython/memory/gc/generation.py Fri Oct 1 17:56:02 2010 @@ -5,7 +5,6 @@ from pypy.rpython.memory.gc.base import read_from_env from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage from pypy.rpython.lltypesystem import lltype, llmemory, llarena -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rlib.objectmodel import free_non_gc_object from pypy.rlib.debug import ll_assert from pypy.rlib.debug import debug_print, debug_start, debug_stop @@ -49,15 +48,17 @@ nursery_hash_base = -1 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + def __init__(self, config, nursery_size=32*WORD, min_nursery_size=32*WORD, auto_nursery_size=False, space_size=1024*WORD, - max_space_size=sys.maxint//2+1): - SemiSpaceGC.__init__(self, config, chunk_size = chunk_size, + max_space_size=sys.maxint//2+1, + **kwds): + SemiSpaceGC.__init__(self, config, space_size = space_size, - max_space_size = max_space_size) + max_space_size = max_space_size, + **kwds) assert min_nursery_size <= nursery_size <= space_size // 2 self.initial_nursery_size = nursery_size self.auto_nursery_size = auto_nursery_size @@ -157,6 +158,14 @@ "odd-valued (i.e. tagged) pointer unexpected here") return self.nursery <= addr < self.nursery_top + def appears_to_be_in_nursery(self, addr): + # same as is_in_nursery(), but may return True accidentally if + # 'addr' is a tagged pointer with just the wrong value. + if not self.translated_to_c: + if not self.is_valid_gc_object(addr): + return False + return self.nursery <= addr < self.nursery_top + def malloc_fixedsize_clear(self, typeid, size, can_collect, has_finalizer=False, contains_weakptr=False): if (has_finalizer or not can_collect or @@ -326,7 +335,7 @@ addr = pointer.address[0] newaddr = self.copy(addr) pointer.address[0] = newaddr - self.write_into_last_generation_obj(obj) + self.write_into_last_generation_obj(obj, newaddr) # ____________________________________________________________ # Implementation of nursery-only collections @@ -457,9 +466,9 @@ # "if addr_struct.int0 & JIT_WB_IF_FLAG: remember_young_pointer()") JIT_WB_IF_FLAG = GCFLAG_NO_YOUNG_PTRS - def write_barrier(self, addr_struct): - if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: - self.remember_young_pointer(addr_struct) + def write_barrier(self, newvalue, addr_struct): + if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: + self.remember_young_pointer(addr_struct, newvalue) def _setup_wb(self): DEBUG = self.DEBUG @@ -470,23 +479,33 @@ # For x86, there is also an extra requirement: when the JIT calls # remember_young_pointer(), it assumes that it will not touch the SSE # registers, so it does not save and restore them (that's a *hack*!). - def remember_young_pointer(addr_struct): + def remember_young_pointer(addr_struct, addr): #llop.debug_print(lltype.Void, "\tremember_young_pointer", # addr_struct, "<-", addr) if DEBUG: ll_assert(not self.is_in_nursery(addr_struct), "nursery object with GCFLAG_NO_YOUNG_PTRS") - self.old_objects_pointing_to_young.append(addr_struct) - self.header(addr_struct).tid &= ~GCFLAG_NO_YOUNG_PTRS - self.write_into_last_generation_obj(addr_struct) + # + # What is important in this function is that it *must* + # clear the flag GCFLAG_NO_YOUNG_PTRS from 'addr_struct' + # if 'addr' is in the nursery. It is ok if, accidentally, + # it also clears the flag in some more rare cases, like + # 'addr' being a tagged pointer whose value happens to be + # a large integer that fools is_in_nursery(). + if self.appears_to_be_in_nursery(addr): + self.old_objects_pointing_to_young.append(addr_struct) + self.header(addr_struct).tid &= ~GCFLAG_NO_YOUNG_PTRS + self.write_into_last_generation_obj(addr_struct, addr) remember_young_pointer._dont_inline_ = True self.remember_young_pointer = remember_young_pointer - def write_into_last_generation_obj(self, addr_struct): + def write_into_last_generation_obj(self, addr_struct, addr): objhdr = self.header(addr_struct) if objhdr.tid & GCFLAG_NO_HEAP_PTRS: - objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS - self.last_generation_root_objects.append(addr_struct) + if (self.is_valid_gc_object(addr) and + not self.is_last_generation(addr)): + objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS + self.last_generation_root_objects.append(addr_struct) write_into_last_generation_obj._always_inline_ = True def assume_young_pointers(self, addr_struct): Modified: pypy/trunk/pypy/rpython/memory/gc/markcompact.py ============================================================================== --- pypy/trunk/pypy/rpython/memory/gc/markcompact.py (original) +++ pypy/trunk/pypy/rpython/memory/gc/markcompact.py Fri Oct 1 17:56:02 2010 @@ -2,7 +2,6 @@ from pypy.rpython.memory.gc.base import MovingGCBase, read_from_env from pypy.rlib.debug import ll_assert, have_debug_prints from pypy.rlib.debug import debug_print, debug_start, debug_stop -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack, get_address_deque from pypy.rpython.memory.support import AddressDict from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage @@ -86,9 +85,9 @@ free = NULL next_collect_after = -1 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, space_size=4096, - min_next_collect_after=128): - MovingGCBase.__init__(self, config, chunk_size) + def __init__(self, config, space_size=4096, + min_next_collect_after=128, **kwds): + MovingGCBase.__init__(self, config, **kwds) self.space_size = space_size self.min_next_collect_after = min_next_collect_after Modified: pypy/trunk/pypy/rpython/memory/gc/marksweep.py ============================================================================== --- pypy/trunk/pypy/rpython/memory/gc/marksweep.py (original) +++ pypy/trunk/pypy/rpython/memory/gc/marksweep.py Fri Oct 1 17:56:02 2010 @@ -1,7 +1,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc, raw_free from pypy.rpython.lltypesystem.llmemory import raw_memcopy, raw_memclear from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack from pypy.rpython.memory.gcheader import GCHeaderBuilder from pypy.rpython.lltypesystem import lltype, llmemory, rffi, llgroup @@ -48,9 +47,9 @@ # translating to a real backend. TRANSLATION_PARAMS = {'start_heap_size': 8*1024*1024} # XXX adjust - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, start_heap_size=4096): + def __init__(self, config, start_heap_size=4096, **kwds): self.param_start_heap_size = start_heap_size - GCBase.__init__(self, config, chunk_size) + GCBase.__init__(self, config, **kwds) def setup(self): GCBase.setup(self) @@ -714,8 +713,8 @@ _alloc_flavor_ = "raw" COLLECT_EVERY = 2000 - def __init__(self, chunk_size=DEFAULT_CHUNK_SIZE, start_heap_size=4096): - MarkSweepGC.__init__(self, chunk_size, start_heap_size) + def __init__(self, config, **kwds): + MarkSweepGC.__init__(self, config, **kwds) self.count_mallocs = 0 def maybe_collect(self): Modified: pypy/trunk/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/trunk/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/trunk/pypy/rpython/memory/gc/minimark.py Fri Oct 1 17:56:02 2010 @@ -4,7 +4,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc_usage from pypy.rpython.memory.gc.base import GCBase, MovingGCBase from pypy.rpython.memory.gc import minimarkpage, base, generation -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rlib.rarithmetic import ovfcheck, LONG_BIT, intmask, r_uint from pypy.rlib.rarithmetic import LONG_BIT_SHIFT from pypy.rlib.debug import ll_assert, debug_print, debug_start, debug_stop @@ -140,7 +139,7 @@ "large_object_gcptrs": 8250*WORD, } - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + def __init__(self, config, read_from_env=False, nursery_size=32*WORD, page_size=16*WORD, @@ -150,8 +149,9 @@ card_page_indices=0, large_object=8*WORD, large_object_gcptrs=10*WORD, - ArenaCollectionClass=None): - MovingGCBase.__init__(self, config, chunk_size) + ArenaCollectionClass=None, + **kwds): + MovingGCBase.__init__(self, config, **kwds) assert small_request_threshold % WORD == 0 self.read_from_env = read_from_env self.nursery_size = nursery_size @@ -636,6 +636,14 @@ "odd-valued (i.e. tagged) pointer unexpected here") return self.nursery <= addr < self.nursery_top + def appears_to_be_in_nursery(self, addr): + # same as is_in_nursery(), but may return True accidentally if + # 'addr' is a tagged pointer with just the wrong value. + if not self.translated_to_c: + if not self.is_valid_gc_object(addr): + return False + return self.nursery <= addr < self.nursery_top + def is_forwarded(self, obj): """Returns True if the nursery obj is marked as forwarded. Implemented a bit obscurely by checking an unrelated flag @@ -726,16 +734,16 @@ def JIT_max_size_of_young_obj(cls): return cls.TRANSLATION_PARAMS['large_object'] - def write_barrier(self, addr_struct): + def write_barrier(self, newvalue, addr_struct): if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: - self.remember_young_pointer(addr_struct) + self.remember_young_pointer(addr_struct, newvalue) - def write_barrier_from_array(self, addr_array, index): + def write_barrier_from_array(self, newvalue, addr_array, index): if self.header(addr_array).tid & GCFLAG_NO_YOUNG_PTRS: if self.card_page_indices > 0: # <- constant-folded self.remember_young_pointer_from_array(addr_array, index) else: - self.remember_young_pointer(addr_array) + self.remember_young_pointer(addr_array, newvalue) def _init_writebarrier_logic(self): DEBUG = self.DEBUG @@ -746,27 +754,28 @@ # For x86, there is also an extra requirement: when the JIT calls # remember_young_pointer(), it assumes that it will not touch the SSE # registers, so it does not save and restore them (that's a *hack*!). - def remember_young_pointer(addr_struct): + def remember_young_pointer(addr_struct, newvalue): # 'addr_struct' is the address of the object in which we write. + # 'newvalue' is the address that we are going to write in there. if DEBUG: ll_assert(not self.is_in_nursery(addr_struct), "nursery object with GCFLAG_NO_YOUNG_PTRS") # - # We assume that what we are writing is a pointer to the nursery - # (and don't care for the fact that this new pointer may not - # actually point to the nursery, which seems ok). What we need is + # If it seems that what we are writing is a pointer to the nursery + # (as checked with appears_to_be_in_nursery()), then we need # to remove the flag GCFLAG_NO_YOUNG_PTRS and add the old object # to the list 'old_objects_pointing_to_young'. We know that # 'addr_struct' cannot be in the nursery, because nursery objects # never have the flag GCFLAG_NO_YOUNG_PTRS to start with. - self.old_objects_pointing_to_young.append(addr_struct) objhdr = self.header(addr_struct) - objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS + if self.appears_to_be_in_nursery(newvalue): + self.old_objects_pointing_to_young.append(addr_struct) + objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS # # Second part: if 'addr_struct' is actually a prebuilt GC # object and it's the first time we see a write to it, we # add it to the list 'prebuilt_root_objects'. Note that we - # do it even in the (rare?) case of 'addr' being another + # do it even in the (rare?) case of 'addr' being NULL or another # prebuilt object, to simplify code. if objhdr.tid & GCFLAG_NO_HEAP_PTRS: objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS @@ -780,16 +789,24 @@ def _init_writebarrier_with_card_marker(self): + DEBUG = self.DEBUG def remember_young_pointer_from_array(addr_array, index): # 'addr_array' is the address of the object in which we write, # which must have an array part; 'index' is the index of the # item that is (or contains) the pointer that we write. + if DEBUG: + ll_assert(not self.is_in_nursery(addr_array), + "nursery array with GCFLAG_NO_YOUNG_PTRS") objhdr = self.header(addr_array) if objhdr.tid & GCFLAG_HAS_CARDS == 0: # - # no cards, use default logic. The 'nocard_logic()' is just - # 'remember_young_pointer()', but forced to be inlined here. - nocard_logic(addr_array) + # no cards, use default logic. Mostly copied from above. + self.old_objects_pointing_to_young.append(addr_array) + objhdr = self.header(addr_array) + objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS + if objhdr.tid & GCFLAG_NO_HEAP_PTRS: + objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS + self.prebuilt_root_objects.append(addr_array) return # # 'addr_array' is a raw_malloc'ed array with card markers @@ -807,17 +824,15 @@ return # # We set the flag (even if the newly written address does not - # actually point to the nursery -- like remember_young_pointer()). + # actually point to the nursery, which seems to be ok -- actually + # it seems more important that remember_young_pointer_from_array() + # does not take 3 arguments). addr_byte.char[0] = chr(byte | bitmask) # if objhdr.tid & GCFLAG_CARDS_SET == 0: self.old_objects_with_cards_set.append(addr_array) objhdr.tid |= GCFLAG_CARDS_SET - nocard_logic = func_with_new_name(self.remember_young_pointer, - 'remember_young_pointer_nocard') - del nocard_logic._dont_inline_ - nocard_logic._always_inline_ = True remember_young_pointer_from_array._dont_inline_ = True self.remember_young_pointer_from_array = ( remember_young_pointer_from_array) Modified: pypy/trunk/pypy/rpython/memory/gc/semispace.py ============================================================================== --- pypy/trunk/pypy/rpython/memory/gc/semispace.py (original) +++ pypy/trunk/pypy/rpython/memory/gc/semispace.py Fri Oct 1 17:56:02 2010 @@ -1,7 +1,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc, raw_free from pypy.rpython.lltypesystem.llmemory import raw_memcopy, raw_memclear from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack, get_address_deque from pypy.rpython.memory.support import AddressDict from pypy.rpython.lltypesystem import lltype, llmemory, llarena, rffi, llgroup @@ -59,11 +58,11 @@ # translating to a real backend. TRANSLATION_PARAMS = {'space_size': 8*1024*1024} # XXX adjust - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, space_size=4096, - max_space_size=sys.maxint//2+1): + def __init__(self, config, space_size=4096, max_space_size=sys.maxint//2+1, + **kwds): self.param_space_size = space_size self.param_max_space_size = max_space_size - MovingGCBase.__init__(self, config, chunk_size) + MovingGCBase.__init__(self, config, **kwds) def setup(self): #self.total_collection_time = 0.0 Modified: pypy/trunk/pypy/rpython/memory/gc/test/test_direct.py ============================================================================== --- pypy/trunk/pypy/rpython/memory/gc/test/test_direct.py (original) +++ pypy/trunk/pypy/rpython/memory/gc/test/test_direct.py Fri Oct 1 17:56:02 2010 @@ -70,6 +70,7 @@ GC_PARAMS = self.GC_PARAMS.copy() if hasattr(meth, 'GC_PARAMS'): GC_PARAMS.update(meth.GC_PARAMS) + GC_PARAMS['translated_to_c'] = False self.gc = self.GCClass(config, **GC_PARAMS) self.gc.DEBUG = True self.rootwalker = DirectRootWalker(self) @@ -86,17 +87,19 @@ def write(self, p, fieldname, newvalue): if self.gc.needs_write_barrier: + newaddr = llmemory.cast_ptr_to_adr(newvalue) addr_struct = llmemory.cast_ptr_to_adr(p) - self.gc.write_barrier(addr_struct) + self.gc.write_barrier(newaddr, addr_struct) setattr(p, fieldname, newvalue) def writearray(self, p, index, newvalue): if self.gc.needs_write_barrier: + newaddr = llmemory.cast_ptr_to_adr(newvalue) addr_struct = llmemory.cast_ptr_to_adr(p) if hasattr(self.gc, 'write_barrier_from_array'): - self.gc.write_barrier_from_array(addr_struct, index) + self.gc.write_barrier_from_array(newaddr, addr_struct, index) else: - self.gc.write_barrier(addr_struct) + self.gc.write_barrier(newaddr, addr_struct) p[index] = newvalue def malloc(self, TYPE, n=None): Modified: pypy/trunk/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/trunk/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/trunk/pypy/rpython/memory/gctransform/framework.py Fri Oct 1 17:56:02 2010 @@ -426,6 +426,7 @@ if GCClass.needs_write_barrier: self.write_barrier_ptr = getfn(GCClass.write_barrier.im_func, [s_gc, + annmodel.SomeAddress(), annmodel.SomeAddress()], annmodel.s_None, inline=True) @@ -434,13 +435,15 @@ # func should not be a bound method, but a real function assert isinstance(func, types.FunctionType) self.write_barrier_failing_case_ptr = getfn(func, - [annmodel.SomeAddress()], + [annmodel.SomeAddress(), + annmodel.SomeAddress()], annmodel.s_None) func = getattr(GCClass, 'write_barrier_from_array', None) if func is not None: self.write_barrier_from_array_ptr = getfn(func.im_func, [s_gc, annmodel.SomeAddress(), + annmodel.SomeAddress(), annmodel.SomeInteger()], annmodel.s_None, inline=True) @@ -1021,6 +1024,8 @@ and not isinstance(v_newvalue, Constant) and v_struct.concretetype.TO._gckind == "gc" and hop.spaceop not in self.clean_sets): + v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], + resulttype = llmemory.Address) v_structaddr = hop.genop("cast_ptr_to_adr", [v_struct], resulttype = llmemory.Address) if (self.write_barrier_from_array_ptr is not None and @@ -1030,12 +1035,14 @@ assert v_index.concretetype == lltype.Signed hop.genop("direct_call", [self.write_barrier_from_array_ptr, self.c_const_gc, + v_newvalue, v_structaddr, v_index]) else: self.write_barrier_calls += 1 hop.genop("direct_call", [self.write_barrier_ptr, self.c_const_gc, + v_newvalue, v_structaddr]) hop.rename('bare_' + opname) Modified: pypy/trunk/pypy/rpython/memory/gcwrapper.py ============================================================================== --- pypy/trunk/pypy/rpython/memory/gcwrapper.py (original) +++ pypy/trunk/pypy/rpython/memory/gcwrapper.py Fri Oct 1 17:56:02 2010 @@ -9,7 +9,10 @@ def __init__(self, llinterp, flowgraphs, gc_class, GC_PARAMS={}): translator = llinterp.typer.annotator.translator config = translator.config.translation - self.gc = gc_class(config, chunk_size = 10, **GC_PARAMS) + self.gc = gc_class(config, + chunk_size = 10, + translated_to_c = False, + **GC_PARAMS) self.gc.set_root_walker(LLInterpRootWalker(self)) self.gc.DEBUG = True self.llinterp = llinterp @@ -94,6 +97,7 @@ assert (type(index) is int # <- fast path or lltype.typeOf(index) == lltype.Signed) self.gc.write_barrier_from_array( + llmemory.cast_ptr_to_adr(newvalue), llmemory.cast_ptr_to_adr(toplevelcontainer), index) wb = False @@ -101,6 +105,7 @@ # if wb: self.gc.write_barrier( + llmemory.cast_ptr_to_adr(newvalue), llmemory.cast_ptr_to_adr(toplevelcontainer)) llheap.setinterior(toplevelcontainer, inneraddr, INNERTYPE, newvalue) Modified: pypy/trunk/pypy/rpython/memory/test/test_transformed_gc.py ============================================================================== --- pypy/trunk/pypy/rpython/memory/test/test_transformed_gc.py (original) +++ pypy/trunk/pypy/rpython/memory/test/test_transformed_gc.py Fri Oct 1 17:56:02 2010 @@ -906,7 +906,8 @@ gcname = "marksweep" class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 @@ -1144,7 +1145,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.marksweep import PrintingMarkSweepGC as GCClass - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestSemiSpaceGC(GenericMovingGCTests): @@ -1154,7 +1156,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.semispace import SemiSpaceGC as GCClass - GC_PARAMS = {'space_size': 512*WORD} + GC_PARAMS = {'space_size': 512*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestMarkCompactGC(GenericMovingGCTests): @@ -1163,7 +1166,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.markcompact import MarkCompactGC as GCClass - GC_PARAMS = {'space_size': 4096*WORD} + GC_PARAMS = {'space_size': 4096*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestGenerationGC(GenericMovingGCTests): @@ -1175,7 +1179,8 @@ from pypy.rpython.memory.gc.generation import GenerationGC as \ GCClass GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 32*WORD} + 'nursery_size': 32*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_weakref_across_minor_collection(cls): @@ -1372,7 +1377,8 @@ GenerationGC._teardown(self) GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 128*WORD} + 'nursery_size': 128*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_working_nursery(cls): @@ -1404,7 +1410,8 @@ from pypy.rpython.memory.gc.hybrid import HybridGC as GCClass GC_PARAMS = {'space_size': 512*WORD, 'nursery_size': 32*WORD, - 'large_object': 8*WORD} + 'large_object': 8*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_ref_from_rawmalloced_to_regular(cls): @@ -1477,6 +1484,7 @@ 'large_object': 8*WORD, 'large_object_gcptrs': 10*WORD, 'card_page_indices': 4, + 'translated_to_c': False, } root_stack_depth = 200 @@ -1585,7 +1593,8 @@ gcname = "marksweep" class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestHybridTaggedPointerGC(TaggedPointerGCTests): @@ -1596,7 +1605,8 @@ from pypy.rpython.memory.gc.generation import GenerationGC as \ GCClass GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 32*WORD} + 'nursery_size': 32*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestMarkCompactTaggedpointerGC(TaggedPointerGCTests): @@ -1605,5 +1615,6 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.markcompact import MarkCompactGC as GCClass - GC_PARAMS = {'space_size': 4096*WORD} + GC_PARAMS = {'space_size': 4096*WORD, + 'translated_to_c': False} root_stack_depth = 200 From afa at codespeak.net Fri Oct 1 18:26:13 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 18:26:13 +0200 (CEST) Subject: [pypy-svn] r77538 - in pypy/branch/fast-forward/pypy: interpreter interpreter/test objspace objspace/std Message-ID: <20101001162613.809B6282BDB@codespeak.net> Author: afa Date: Fri Oct 1 18:26:11 2010 New Revision: 77538 Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_typedef.py pypy/branch/fast-forward/pypy/interpreter/typedef.py pypy/branch/fast-forward/pypy/objspace/descroperation.py pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py pypy/branch/fast-forward/pypy/objspace/std/dicttype.py pypy/branch/fast-forward/pypy/objspace/std/listtype.py pypy/branch/fast-forward/pypy/objspace/std/objecttype.py pypy/branch/fast-forward/pypy/objspace/std/settype.py pypy/branch/fast-forward/pypy/objspace/std/slicetype.py pypy/branch/fast-forward/pypy/objspace/std/stdtypedef.py Log: Set list.__hash__ is now None, instead of a function that raises TypeError. no_hash_descr() is gone! This fixes isinstance([], _abcoll.Hashable) Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_typedef.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/test/test_typedef.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/test/test_typedef.py Fri Oct 1 18:26:11 2010 @@ -145,6 +145,19 @@ w_obj = self.space.wrap(W_SomeType()) assert self.space.getattr(w_obj, self.space.wrap('x')) is self.space.w_None + def test_unhashable(self): + class W_SomeType(Wrappable): + pass + W_SomeType.typedef = typedef.TypeDef( + 'some_type', + __hash__ = None) + w_obj = self.space.wrap(W_SomeType()) + self.space.appexec([w_obj], """(obj): + assert type(obj).__hash__ is None + err = raises(TypeError, hash, obj) + assert err.value.message == "'some_type' objects are unhashable" + """) + class AppTestTypeDef: Modified: pypy/branch/fast-forward/pypy/interpreter/typedef.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/typedef.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/typedef.py Fri Oct 1 18:26:11 2010 @@ -51,13 +51,6 @@ def default_identity_hash(space, w_obj): return space.wrap(compute_identity_hash(w_obj)) -def descr__hash__unhashable(space, w_obj): - typename = space.type(w_obj).getname(space, '?') - raise operationerrfmt(space.w_TypeError, - "'%s' objects are unhashable", typename) - -no_hash_descr = interp2app(descr__hash__unhashable) - # ____________________________________________________________ # # For each built-in app-level type Xxx that can be subclassed at @@ -885,7 +878,7 @@ __eq__ = interp2app(Cell.descr__eq__, unwrap_spec=['self', ObjSpace, W_Root]), __ne__ = descr_generic_ne, - __hash__ = no_hash_descr, + __hash__ = None, __reduce__ = interp2app(Cell.descr__reduce__, unwrap_spec=['self', ObjSpace]), __setstate__ = interp2app(Cell.descr__setstate__, Modified: pypy/branch/fast-forward/pypy/objspace/descroperation.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/descroperation.py (original) +++ pypy/branch/fast-forward/pypy/objspace/descroperation.py Fri Oct 1 18:26:11 2010 @@ -392,6 +392,10 @@ # default __hash__. This path should only be taken under very # obscure circumstances. return default_identity_hash(space, w_obj) + if space.is_w(w_hash, space.w_None): + typename = space.type(w_obj).getname(space, '?') + raise operationerrfmt(space.w_TypeError, + "'%s' objects are unhashable", typename) # XXX CPython has a special case for types with "__hash__ = None" # to produce a nicer error message, namely "unhashable type: 'X'". w_result = space.get_and_call_function(w_hash, w_obj) Modified: pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py Fri Oct 1 18:26:11 2010 @@ -2,7 +2,7 @@ from pypy.interpreter import gateway from pypy.interpreter.baseobjspace import ObjSpace, W_Root from pypy.objspace.std.register_all import register_all -from pypy.objspace.std.stdtypedef import StdTypeDef, SMM, no_hash_descr +from pypy.objspace.std.stdtypedef import StdTypeDef, SMM from pypy.objspace.std.stringtype import ( str_count, str_index, str_rindex, str_find, str_rfind, str_replace, @@ -31,6 +31,6 @@ If the argument is a bytearray, the return value is the same object.''', __new__ = gateway.interp2app(descr__new__), - __hash__ = no_hash_descr, + __hash__ = None, ) bytearray_typedef.registermethods(globals()) Modified: pypy/branch/fast-forward/pypy/objspace/std/dicttype.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/dicttype.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/dicttype.py Fri Oct 1 18:26:11 2010 @@ -2,7 +2,7 @@ from pypy.interpreter.error import OperationError from pypy.interpreter.mixedmodule import MixedModule from pypy.interpreter import gateway -from pypy.objspace.std.stdtypedef import StdTypeDef, SMM, no_hash_descr +from pypy.objspace.std.stdtypedef import StdTypeDef, SMM from pypy.objspace.std.register_all import register_all dict_copy = SMM('copy', 1, @@ -179,7 +179,7 @@ unwrap_spec= [gateway.ObjSpace, gateway.W_Root,gateway.Arguments]), - __hash__ = no_hash_descr, + __hash__ = None, fromkeys = gateway.interp2app(descr_fromkeys, as_classmethod=True), ) dict_typedef.registermethods(globals()) Modified: pypy/branch/fast-forward/pypy/objspace/std/listtype.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/listtype.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/listtype.py Fri Oct 1 18:26:11 2010 @@ -1,6 +1,6 @@ from pypy.interpreter import gateway from pypy.interpreter.error import OperationError -from pypy.objspace.std.stdtypedef import StdTypeDef, SMM, no_hash_descr +from pypy.objspace.std.stdtypedef import StdTypeDef, SMM from pypy.objspace.std.register_all import register_all from sys import maxint @@ -54,7 +54,7 @@ __new__ = gateway.interp2app(descr__new__, unwrap_spec=[gateway.ObjSpace, gateway.W_Root, gateway.Arguments]), - __hash__ = no_hash_descr, + __hash__ = None, ) list_typedef.registermethods(globals()) Modified: pypy/branch/fast-forward/pypy/objspace/std/objecttype.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/objecttype.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/objecttype.py Fri Oct 1 18:26:11 2010 @@ -4,7 +4,7 @@ from pypy.interpreter.argument import Arguments from pypy.interpreter.baseobjspace import ObjSpace from pypy.objspace.descroperation import Object -from pypy.objspace.std.stdtypedef import StdTypeDef, no_hash_descr +from pypy.objspace.std.stdtypedef import StdTypeDef from pypy.objspace.std.register_all import register_all Modified: pypy/branch/fast-forward/pypy/objspace/std/settype.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/settype.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/settype.py Fri Oct 1 18:26:11 2010 @@ -1,7 +1,7 @@ from pypy.interpreter.error import OperationError from pypy.interpreter import gateway from pypy.objspace.std.register_all import register_all -from pypy.objspace.std.stdtypedef import StdTypeDef, no_hash_descr, SMM +from pypy.objspace.std.stdtypedef import StdTypeDef, SMM set_add = SMM('add', 2, doc='Add an element to a set.\n\nThis' @@ -78,7 +78,7 @@ __new__ = gateway.interp2app(descr__new__, unwrap_spec=[gateway.ObjSpace, gateway.W_Root, gateway.Arguments]), - __hash__ = no_hash_descr, + __hash__ = None, ) set_typedef.registermethods(globals()) Modified: pypy/branch/fast-forward/pypy/objspace/std/slicetype.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/slicetype.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/slicetype.py Fri Oct 1 18:26:11 2010 @@ -1,6 +1,6 @@ from pypy.interpreter import baseobjspace, gateway from pypy.interpreter.typedef import GetSetProperty -from pypy.objspace.std.stdtypedef import StdTypeDef, SMM, no_hash_descr +from pypy.objspace.std.stdtypedef import StdTypeDef, SMM from pypy.objspace.std.register_all import register_all from pypy.interpreter.error import OperationError @@ -89,7 +89,7 @@ Create a slice object. This is used for extended slicing (e.g. a[0:10:2]).''', __new__ = gateway.interp2app(descr__new__), - __hash__ = no_hash_descr, + __hash__ = None, start = slicewprop('w_start'), stop = slicewprop('w_stop'), step = slicewprop('w_step'), Modified: pypy/branch/fast-forward/pypy/objspace/std/stdtypedef.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/stdtypedef.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/stdtypedef.py Fri Oct 1 18:26:11 2010 @@ -2,7 +2,7 @@ from pypy.interpreter.error import OperationError, operationerrfmt from pypy.interpreter.typedef import TypeDef, GetSetProperty, Member from pypy.interpreter.typedef import descr_get_dict, descr_set_dict -from pypy.interpreter.typedef import no_hash_descr, descr_del_dict +from pypy.interpreter.typedef import descr_del_dict from pypy.interpreter.baseobjspace import SpaceCache from pypy.objspace.std import model from pypy.objspace.std.model import StdObjSpaceMultiMethod @@ -10,7 +10,7 @@ from pypy.rlib import jit from pypy.tool.sourcetools import compile2 -__all__ = ['StdTypeDef', 'SMM', 'no_hash_descr'] +__all__ = ['StdTypeDef', 'SMM'] SMM = StdObjSpaceMultiMethod From afa at codespeak.net Fri Oct 1 18:38:41 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 18:38:41 +0200 (CEST) Subject: [pypy-svn] r77539 - pypy/branch/fast-forward/pypy/objspace Message-ID: <20101001163841.718ED282BDB@codespeak.net> Author: afa Date: Fri Oct 1 18:38:39 2010 New Revision: 77539 Modified: pypy/branch/fast-forward/pypy/objspace/descroperation.py Log: Kill a XXX fixed by the last change Modified: pypy/branch/fast-forward/pypy/objspace/descroperation.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/descroperation.py (original) +++ pypy/branch/fast-forward/pypy/objspace/descroperation.py Fri Oct 1 18:38:39 2010 @@ -396,8 +396,6 @@ typename = space.type(w_obj).getname(space, '?') raise operationerrfmt(space.w_TypeError, "'%s' objects are unhashable", typename) - # XXX CPython has a special case for types with "__hash__ = None" - # to produce a nicer error message, namely "unhashable type: 'X'". w_result = space.get_and_call_function(w_hash, w_obj) w_resulttype = space.type(w_result) if space.is_w(w_resulttype, space.w_int): From hakanardo at codespeak.net Fri Oct 1 19:36:21 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Fri, 1 Oct 2010 19:36:21 +0200 (CEST) Subject: [pypy-svn] r77540 - in pypy/branch/jit-unroll-loops/pypy: jit/metainterp jit/metainterp/optimizeopt jit/metainterp/test rlib Message-ID: <20101001173621.AF05C282BDB@codespeak.net> Author: hakanardo Date: Fri Oct 1 19:36:19 2010 New Revision: 77540 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py pypy/branch/jit-unroll-loops/pypy/rlib/jit.py Log: guards working, test_loop_invariant_mul1 passing Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Fri Oct 1 19:36:19 2010 @@ -14,15 +14,14 @@ if not self.enabled: self.emit_operation(op) return - + if op.getopnum() == rop.JUMP: loop = self.optimizer.loop loop.preamble.operations = self.optimizer.newoperations self.optimizer.newoperations = [] jump_args = op.getarglist() inputargs = self.inline(loop.preamble.operations + [op], - loop.inputargs, jump_args) - print "IN: ", inputargs + loop.inputargs, jump_args) loop.inputargs = inputargs jmp = ResOperation(rop.JUMP, loop.inputargs[:], None) jmp.setdescr(loop.token) @@ -36,16 +35,11 @@ for i in range(len(loop_args)): argmap[loop_args[i]] = jump_args[i] - print - print - print argmap - for v in self.optimizer.values.values(): v.fromstart = True inputargs = jump_args[:] for op in loop_operations: - print "I:", op newop = op.clone() for i in range(op.numargs()): a = op.getarg(i) @@ -63,18 +57,25 @@ args = newop.getarglist() newop.initarglist(args + inputargs[len(args):]) # FIXME: Assumes no virtuals - print "N:", newop current = len(self.optimizer.newoperations) self.emit_operation(newop) + for op in self.optimizer.newoperations[current:]: - print "E:", op + if op.is_guard(): + args = [] + for a in op._fail_args: + try: + a = argmap[a] + except KeyError: + pass + args.append(a) + op.setfailargs(args) + for a in op.getarglist(): if not isinstance(a, Const) and a in self.optimizer.values: v = self.getvalue(a) - print " testing ", a if v.fromstart and a not in inputargs: - print " ", a inputargs.append(a) return inputargs Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py Fri Oct 1 19:36:19 2010 @@ -298,7 +298,7 @@ self.liveboxes_from_env = liveboxes_from_env self.liveboxes = {} storage.rd_numb = numb - storage.rd_snapshot = None + #storage.rd_snapshot = None FIXME!! # collect liveboxes and virtuals n = len(liveboxes_from_env) - v Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Fri Oct 1 19:36:19 2010 @@ -316,8 +316,8 @@ res = self.meta_interp(f, [6, 7]) assert res == 252 self.check_loop_count(2) - self.check_loops({'guard_true': 1, - 'int_add': 1, 'int_sub': 1, 'int_gt': 1, + self.check_loops({'guard_true': 2, + 'int_add': 2, 'int_sub': 2, 'int_gt': 2, 'int_mul': 1, 'jump': 2}) Modified: pypy/branch/jit-unroll-loops/pypy/rlib/jit.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/rlib/jit.py (original) +++ pypy/branch/jit-unroll-loops/pypy/rlib/jit.py Fri Oct 1 19:36:19 2010 @@ -236,7 +236,8 @@ 'trace_eagerness': 200, 'trace_limit': 10000, 'inlining': False, - 'optimizer': OPTIMIZER_FULL, + #'optimizer': OPTIMIZER_FULL, + 'optimizer': OPTIMIZER_NO_PERFECTSPEC, 'debug' : DEBUG_STEPS, } unroll_parameters = unrolling_iterable(PARAMETERS.keys()) From afa at codespeak.net Fri Oct 1 22:45:28 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 22:45:28 +0200 (CEST) Subject: [pypy-svn] r77542 - in pypy/branch/fast-forward/pypy/interpreter: . test Message-ID: <20101001204528.99700282BDB@codespeak.net> Author: afa Date: Fri Oct 1 22:45:27 2010 New Revision: 77542 Modified: pypy/branch/fast-forward/pypy/interpreter/argument.py pypy/branch/fast-forward/pypy/interpreter/test/test_function.py Log: **kwargs now accepts any mapping Modified: pypy/branch/fast-forward/pypy/interpreter/argument.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/argument.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/argument.py Fri Oct 1 22:45:27 2010 @@ -149,14 +149,25 @@ # unpack the ** arguments if w_starstararg is not None: space = self.space - if not space.is_true(space.isinstance(w_starstararg, space.w_dict)): - raise OperationError(space.w_TypeError, - space.wrap("argument after ** must be " - "a dictionary")) - keywords_w = [None] * space.int_w(space.len(w_starstararg)) - keywords = [None] * space.int_w(space.len(w_starstararg)) + if space.isinstance_w(w_starstararg, space.w_dict): + keys_w = space.unpackiterable(w_starstararg) + else: + try: + w_keys = space.call_method(w_starstararg, "keys") + except OperationError, e: + if e.match(space, space.w_AttributeError): + w_type = space.type(w_starstararg) + typename = w_type.getname(space, '?') + raise OperationError( + space.w_TypeError, + space.wrap("argument after ** must be " + "a mapping, not %s" % (typename,))) + raise + keys_w = space.unpackiterable(w_keys) + keywords_w = [None] * len(keys_w) + keywords = [None] * len(keys_w) i = 0 - for w_key in space.unpackiterable(w_starstararg): + for w_key in keys_w: try: key = space.str_w(w_key) except OperationError, e: Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_function.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/test/test_function.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/test/test_function.py Fri Oct 1 22:45:27 2010 @@ -164,6 +164,21 @@ raises( TypeError, func, 42, {'arg1': 23}) + def test_kwargs_nondict_mapping(self): + class Mapping: + def keys(self): + return ('a', 'b') + def __getitem__(self, key): + return key + def func(arg1, **kw): + return arg1, kw + res = func(23, **Mapping()) + assert res[0] == 23 + assert res[1] == {'a': 'a', 'b': 'b'} + error = raises(TypeError, lambda: func(42, **[])) + assert error.value.message == ('argument after ** must be a mapping, ' + 'not list') + def test_default_arg(self): def func(arg1,arg2=42): return arg1, arg2 From afa at codespeak.net Fri Oct 1 22:53:06 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 1 Oct 2010 22:53:06 +0200 (CEST) Subject: [pypy-svn] r77543 - in pypy/branch/fast-forward: . lib_pypy/_ctypes pypy/jit/backend pypy/jit/backend/llgraph pypy/jit/backend/llsupport pypy/jit/backend/llsupport/test pypy/jit/backend/test pypy/jit/backend/x86 pypy/jit/metainterp pypy/jit/tl pypy/module/imp pypy/module/imp/test pypy/module/test_lib_pypy/ctypes_tests pypy/rpython/lltypesystem pypy/rpython/memory pypy/rpython/memory/gc pypy/rpython/memory/gc/test pypy/rpython/memory/gctransform pypy/rpython/memory/test pypy/translator pypy/translator/goal Message-ID: <20101001205306.1F56A282BDB@codespeak.net> Author: afa Date: Fri Oct 1 22:53:02 2010 New Revision: 77543 Added: pypy/branch/fast-forward/pypy/jit/backend/conftest.py - copied unchanged from r77542, pypy/trunk/pypy/jit/backend/conftest.py pypy/branch/fast-forward/pypy/jit/tl/jittest.py - copied unchanged from r77542, pypy/trunk/pypy/jit/tl/jittest.py Removed: pypy/branch/fast-forward/pypy/jit/backend/test/conftest.py Modified: pypy/branch/fast-forward/ (props changed) pypy/branch/fast-forward/lib_pypy/_ctypes/function.py pypy/branch/fast-forward/pypy/jit/backend/llgraph/llimpl.py pypy/branch/fast-forward/pypy/jit/backend/llsupport/gc.py pypy/branch/fast-forward/pypy/jit/backend/llsupport/test/test_gc.py pypy/branch/fast-forward/pypy/jit/backend/test/runner_test.py pypy/branch/fast-forward/pypy/jit/backend/test/test_ll_random.py pypy/branch/fast-forward/pypy/jit/backend/test/test_random.py pypy/branch/fast-forward/pypy/jit/backend/x86/assembler.py pypy/branch/fast-forward/pypy/jit/backend/x86/regalloc.py pypy/branch/fast-forward/pypy/jit/backend/x86/runner.py pypy/branch/fast-forward/pypy/jit/metainterp/executor.py pypy/branch/fast-forward/pypy/jit/metainterp/resoperation.py pypy/branch/fast-forward/pypy/jit/metainterp/warmspot.py pypy/branch/fast-forward/pypy/module/imp/importing.py pypy/branch/fast-forward/pypy/module/imp/test/test_import.py pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py pypy/branch/fast-forward/pypy/rpython/lltypesystem/llarena.py pypy/branch/fast-forward/pypy/rpython/memory/gc/base.py pypy/branch/fast-forward/pypy/rpython/memory/gc/generation.py pypy/branch/fast-forward/pypy/rpython/memory/gc/markcompact.py pypy/branch/fast-forward/pypy/rpython/memory/gc/marksweep.py pypy/branch/fast-forward/pypy/rpython/memory/gc/minimark.py pypy/branch/fast-forward/pypy/rpython/memory/gc/minimarkpage.py pypy/branch/fast-forward/pypy/rpython/memory/gc/semispace.py pypy/branch/fast-forward/pypy/rpython/memory/gc/test/test_direct.py pypy/branch/fast-forward/pypy/rpython/memory/gc/test/test_minimarkpage.py pypy/branch/fast-forward/pypy/rpython/memory/gctransform/framework.py pypy/branch/fast-forward/pypy/rpython/memory/gcwrapper.py pypy/branch/fast-forward/pypy/rpython/memory/test/test_transformed_gc.py pypy/branch/fast-forward/pypy/translator/driver.py pypy/branch/fast-forward/pypy/translator/goal/translate.py Log: Merge from trunk Modified: pypy/branch/fast-forward/lib_pypy/_ctypes/function.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_ctypes/function.py (original) +++ pypy/branch/fast-forward/lib_pypy/_ctypes/function.py Fri Oct 1 22:53:02 2010 @@ -171,7 +171,7 @@ return self._build_result(restype, resbuffer, argtypes, argsandobjs) def _getfuncptr(self, argtypes, restype, thisarg=None): - if self._ptr is not None: + if self._ptr is not None and argtypes is self._argtypes_: return self._ptr if restype is None or not isinstance(restype, _CDataMeta): import ctypes Modified: pypy/branch/fast-forward/pypy/jit/backend/llgraph/llimpl.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/backend/llgraph/llimpl.py (original) +++ pypy/branch/fast-forward/pypy/jit/backend/llgraph/llimpl.py Fri Oct 1 22:53:02 2010 @@ -129,7 +129,7 @@ 'arraylen_gc' : (('ref',), 'int'), 'call' : (('ref', 'varargs'), 'intorptr'), 'call_assembler' : (('varargs',), 'intorptr'), - 'cond_call_gc_wb' : (('ptr',), None), + 'cond_call_gc_wb' : (('ptr', 'ptr'), None), 'oosend' : (('varargs',), 'intorptr'), 'oosend_pure' : (('varargs',), 'intorptr'), 'guard_true' : (('bool',), None), @@ -810,7 +810,7 @@ FLOAT: 0.0} return d[calldescr.typeinfo] - def op_cond_call_gc_wb(self, descr, a): + def op_cond_call_gc_wb(self, descr, a, b): py.test.skip("cond_call_gc_wb not supported") def op_oosend(self, descr, obj, *args): Modified: pypy/branch/fast-forward/pypy/jit/backend/llsupport/gc.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/backend/llsupport/gc.py (original) +++ pypy/branch/fast-forward/pypy/jit/backend/llsupport/gc.py Fri Oct 1 22:53:02 2010 @@ -404,7 +404,7 @@ self.GC_MALLOC_BASIC = lltype.Ptr(lltype.FuncType( [lltype.Signed, lltype.Signed], llmemory.GCREF)) self.WB_FUNCPTR = lltype.Ptr(lltype.FuncType( - [llmemory.Address], lltype.Void)) + [llmemory.Address, llmemory.Address], lltype.Void)) self.write_barrier_descr = WriteBarrierDescr(self) # def malloc_array(itemsize, tid, num_elem): @@ -550,7 +550,8 @@ # the GC, and call it immediately llop1 = self.llop1 funcptr = llop1.get_write_barrier_failing_case(self.WB_FUNCPTR) - funcptr(llmemory.cast_ptr_to_adr(gcref_struct)) + funcptr(llmemory.cast_ptr_to_adr(gcref_struct), + llmemory.cast_ptr_to_adr(gcref_newptr)) def rewrite_assembler(self, cpu, operations): # Perform two kinds of rewrites in parallel: @@ -589,7 +590,7 @@ v = op.getarg(1) if isinstance(v, BoxPtr) or (isinstance(v, ConstPtr) and bool(v.value)): # store a non-NULL - self._gen_write_barrier(newops, op.getarg(0)) + self._gen_write_barrier(newops, op.getarg(0), v) op = op.copy_and_change(rop.SETFIELD_RAW) # ---------- write barrier for SETARRAYITEM_GC ---------- if op.getopnum() == rop.SETARRAYITEM_GC: @@ -598,15 +599,15 @@ bool(v.value)): # store a non-NULL # XXX detect when we should produce a # write_barrier_from_array - self._gen_write_barrier(newops, op.getarg(0)) + self._gen_write_barrier(newops, op.getarg(0), v) op = op.copy_and_change(rop.SETARRAYITEM_RAW) # ---------- newops.append(op) del operations[:] operations.extend(newops) - def _gen_write_barrier(self, newops, v_base): - args = [v_base] + def _gen_write_barrier(self, newops, v_base, v_value): + args = [v_base, v_value] newops.append(ResOperation(rop.COND_CALL_GC_WB, args, None, descr=self.write_barrier_descr)) Modified: pypy/branch/fast-forward/pypy/jit/backend/llsupport/test/test_gc.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/backend/llsupport/test/test_gc.py (original) +++ pypy/branch/fast-forward/pypy/jit/backend/llsupport/test/test_gc.py Fri Oct 1 22:53:02 2010 @@ -141,8 +141,8 @@ repr(offset_to_length), p)) return p - def _write_barrier_failing_case(self, adr_struct): - self.record.append(('barrier', adr_struct)) + def _write_barrier_failing_case(self, adr_struct, adr_newptr): + self.record.append(('barrier', adr_struct, adr_newptr)) def get_write_barrier_failing_case(self, FPTRTYPE): return llhelper(FPTRTYPE, self._write_barrier_failing_case) @@ -239,6 +239,7 @@ s_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, s) r_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, r) s_adr = llmemory.cast_ptr_to_adr(s) + r_adr = llmemory.cast_ptr_to_adr(r) # s_hdr.tid &= ~gc_ll_descr.GCClass.JIT_WB_IF_FLAG gc_ll_descr.do_write_barrier(s_gcref, r_gcref) @@ -246,7 +247,7 @@ # s_hdr.tid |= gc_ll_descr.GCClass.JIT_WB_IF_FLAG gc_ll_descr.do_write_barrier(s_gcref, r_gcref) - assert self.llop1.record == [('barrier', s_adr)] + assert self.llop1.record == [('barrier', s_adr, r_adr)] def test_gen_write_barrier(self): gc_ll_descr = self.gc_ll_descr @@ -254,11 +255,13 @@ # newops = [] v_base = BoxPtr() - gc_ll_descr._gen_write_barrier(newops, v_base) + v_value = BoxPtr() + gc_ll_descr._gen_write_barrier(newops, v_base, v_value) assert llop1.record == [] assert len(newops) == 1 assert newops[0].getopnum() == rop.COND_CALL_GC_WB assert newops[0].getarg(0) == v_base + assert newops[0].getarg(1) == v_value assert newops[0].result is None wbdescr = newops[0].getdescr() assert isinstance(wbdescr.jit_wb_if_flag, int) @@ -358,6 +361,7 @@ # assert operations[0].getopnum() == rop.COND_CALL_GC_WB assert operations[0].getarg(0) == v_base + assert operations[0].getarg(1) == v_value assert operations[0].result is None # assert operations[1].getopnum() == rop.SETFIELD_RAW @@ -381,6 +385,7 @@ # assert operations[0].getopnum() == rop.COND_CALL_GC_WB assert operations[0].getarg(0) == v_base + assert operations[0].getarg(1) == v_value assert operations[0].result is None # assert operations[1].getopnum() == rop.SETARRAYITEM_RAW Modified: pypy/branch/fast-forward/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/backend/test/runner_test.py (original) +++ pypy/branch/fast-forward/pypy/jit/backend/test/runner_test.py Fri Oct 1 22:53:02 2010 @@ -1427,12 +1427,12 @@ assert not excvalue def test_cond_call_gc_wb(self): - def func_void(a): - record.append(a) + def func_void(a, b): + record.append((a, b)) record = [] # S = lltype.GcStruct('S', ('tid', lltype.Signed)) - FUNC = self.FuncType([lltype.Ptr(S)], lltype.Void) + FUNC = self.FuncType([lltype.Ptr(S), lltype.Signed], lltype.Void) func_ptr = llhelper(lltype.Ptr(FUNC), func_void) funcbox = self.get_funcbox(self.cpu, func_ptr) class WriteBarrierDescr(AbstractDescr): @@ -1453,10 +1453,10 @@ sgcref = lltype.cast_opaque_ptr(llmemory.GCREF, s) del record[:] self.execute_operation(rop.COND_CALL_GC_WB, - [BoxPtr(sgcref)], + [BoxPtr(sgcref), ConstInt(-2121)], 'void', descr=WriteBarrierDescr()) if cond: - assert record == [s] + assert record == [(s, -2121)] else: assert record == [] Modified: pypy/branch/fast-forward/pypy/jit/backend/test/test_ll_random.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/backend/test/test_ll_random.py (original) +++ pypy/branch/fast-forward/pypy/jit/backend/test/test_ll_random.py Fri Oct 1 22:53:02 2010 @@ -386,6 +386,20 @@ v_string = self.get_string(builder, r) builder.do(self.opnum, [v_string]) +class AbstractCopyContentOperation(AbstractStringOperation): + def produce_into(self, builder, r): + v_srcstring = self.get_string(builder, r) + v_dststring = self.get_string(builder, r) + if v_srcstring.value == v_dststring.value: # because it's not a + raise test_random.CannotProduceOperation # memmove(), but memcpy() + srclen = len(v_srcstring.getref(self.ptr).chars) + dstlen = len(v_dststring.getref(self.ptr).chars) + v_length = builder.get_index(min(srclen, dstlen), r) + v_srcstart = builder.get_index(srclen - v_length.value + 1, r) + v_dststart = builder.get_index(dstlen - v_length.value + 1, r) + builder.do(self.opnum, [v_srcstring, v_dststring, + v_srcstart, v_dststart, v_length]) + class StrGetItemOperation(AbstractGetItemOperation, _StrOperation): pass @@ -404,6 +418,13 @@ class UnicodeLenOperation(AbstractStringLenOperation, _UnicodeOperation): pass +class CopyStrContentOperation(AbstractCopyContentOperation, _StrOperation): + pass + +class CopyUnicodeContentOperation(AbstractCopyContentOperation, + _UnicodeOperation): + pass + # there are five options in total: # 1. non raising call and guard_no_exception @@ -577,6 +598,8 @@ OPERATIONS.append(UnicodeSetItemOperation(rop.UNICODESETITEM)) OPERATIONS.append(StrLenOperation(rop.STRLEN)) OPERATIONS.append(UnicodeLenOperation(rop.UNICODELEN)) + OPERATIONS.append(CopyStrContentOperation(rop.COPYSTRCONTENT)) + #OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT)) for i in range(2): OPERATIONS.append(GuardClassOperation(rop.GUARD_CLASS)) Modified: pypy/branch/fast-forward/pypy/jit/backend/test/test_random.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/backend/test/test_random.py (original) +++ pypy/branch/fast-forward/pypy/jit/backend/test/test_random.py Fri Oct 1 22:53:02 2010 @@ -1,7 +1,7 @@ import py, sys from pypy.rlib.rarithmetic import intmask, LONG_BIT from pypy.rpython.lltypesystem import llmemory -from pypy.jit.backend.test import conftest as demo_conftest +from pypy.jit.backend import conftest as demo_conftest from pypy.jit.metainterp.history import BasicFailDescr, TreeLoop from pypy.jit.metainterp.history import BoxInt, ConstInt, LoopToken from pypy.jit.metainterp.history import BoxPtr, ConstPtr @@ -102,7 +102,7 @@ elif isinstance(v, ConstFloat): args.append('ConstFloat(%r)' % v.value) elif isinstance(v, ConstInt): - args.append('ConstInt(%d)' % v.value) + args.append('ConstInt(%s)' % v.value) else: raise NotImplementedError(v) if op.getdescr() is None: @@ -113,7 +113,7 @@ except AttributeError: descrstr = ', descr=...' print >>s, ' ResOperation(rop.%s, [%s], %s%s),' % ( - opname[op.opnum], ', '.join(args), names[op.result], descrstr) + opname[op.getopnum()], ', '.join(args), names[op.result], descrstr) #if getattr(op, 'suboperations', None) is not None: # subops.append(op) @@ -189,7 +189,7 @@ v.value) print >>s, ' op = cpu.execute_token(looptoken)' if self.should_fail_by is None: - fail_args = self.loop.operations[-1].args + fail_args = self.loop.operations[-1].getarglist() else: fail_args = self.should_fail_by.getfailargs() for i, v in enumerate(fail_args): Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/assembler.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/backend/x86/assembler.py (original) +++ pypy/branch/fast-forward/pypy/jit/backend/x86/assembler.py Fri Oct 1 22:53:02 2010 @@ -1780,11 +1780,12 @@ self.mc.PUSH_i32(loc.getint()) if IS_X86_64: - # We clobber this register to pass the arguments, but that's + # We clobber these registers to pass the arguments, but that's # okay, because consider_cond_call_gc_wb makes sure that any # caller-save registers with values in them are present in arglocs, # so they are saved on the stack above and restored below self.mc.MOV_rs(edi.value, 0) + self.mc.MOV_rs(esi.value, 8) # misaligned stack in the call, but it's ok because the write barrier # is not going to call anything more. Also, this assumes that the Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/regalloc.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/backend/x86/regalloc.py (original) +++ pypy/branch/fast-forward/pypy/jit/backend/x86/regalloc.py Fri Oct 1 22:53:02 2010 @@ -696,9 +696,13 @@ def consider_cond_call_gc_wb(self, op): assert op.result is None args = op.getarglist() + loc_newvalue = self.rm.make_sure_var_in_reg(op.getarg(1), args) + # ^^^ we force loc_newvalue in a reg (unless it's a Const), + # because it will be needed anyway by the following setfield_gc. + # It avoids loading it twice from the memory. loc_base = self.rm.make_sure_var_in_reg(op.getarg(0), args, imm_fine=False) - arglocs = [loc_base] + arglocs = [loc_base, loc_newvalue] # add eax, ecx and edx as extra "arguments" to ensure they are # saved and restored. Fish in self.rm to know which of these # registers really need to be saved (a bit of a hack). Moreover, @@ -959,18 +963,23 @@ args = op.getarglist() base_loc = self.rm.make_sure_var_in_reg(args[0], args) ofs_loc = self.rm.make_sure_var_in_reg(args[2], args) + assert args[0] is not args[1] # forbidden case of aliasing self.rm.possibly_free_var(args[0]) - self.rm.possibly_free_var(args[2]) + if args[3] is not args[2] is not args[4]: # MESS MESS MESS: don't free + self.rm.possibly_free_var(args[2]) # it if ==args[3] or args[4] srcaddr_box = TempBox() - srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box) + forbidden_vars = [args[1], args[3], args[4], srcaddr_box] + srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box, forbidden_vars) self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc) # compute the destination address - base_loc = self.rm.make_sure_var_in_reg(args[1], args) - ofs_loc = self.rm.make_sure_var_in_reg(args[3], args) + base_loc = self.rm.make_sure_var_in_reg(args[1], forbidden_vars) + ofs_loc = self.rm.make_sure_var_in_reg(args[3], forbidden_vars) self.rm.possibly_free_var(args[1]) - self.rm.possibly_free_var(args[3]) + if args[3] is not args[4]: # more of the MESS described above + self.rm.possibly_free_var(args[3]) + forbidden_vars = [args[4], srcaddr_box] dstaddr_box = TempBox() - dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box) + dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box, forbidden_vars) self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc) # call memcpy() length_loc = self.loc(args[4]) Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/runner.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/backend/x86/runner.py (original) +++ pypy/branch/fast-forward/pypy/jit/backend/x86/runner.py Fri Oct 1 22:53:02 2010 @@ -87,7 +87,9 @@ def execute_token(self, executable_token): addr = executable_token._x86_bootstrap_code + #llop.debug_print(lltype.Void, ">>>> Entering", addr) func = rffi.cast(lltype.Ptr(self.BOOTSTRAP_TP), addr) + #llop.debug_print(lltype.Void, "<<<< Back") fail_index = self._execute_call(func) return self.get_fail_descr_from_number(fail_index) @@ -99,10 +101,7 @@ LLInterpreter.current_interpreter = self.debug_ll_interpreter res = 0 try: - #llop.debug_print(lltype.Void, ">>>> Entering", - # rffi.cast(lltype.Signed, func)) res = func() - #llop.debug_print(lltype.Void, "<<<< Back") finally: if not self.translate_support_code: LLInterpreter.current_interpreter = prev_interpreter Modified: pypy/branch/fast-forward/pypy/jit/metainterp/executor.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/metainterp/executor.py (original) +++ pypy/branch/fast-forward/pypy/jit/metainterp/executor.py Fri Oct 1 22:53:02 2010 @@ -205,8 +205,8 @@ def do_copystrcontent(cpu, _, srcbox, dstbox, srcstartbox, dststartbox, lengthbox): - src = srcbox.getptr(lltype.Ptr(rstr.STR)) - dst = dstbox.getptr(lltype.Ptr(rstr.STR)) + src = srcbox.getref(lltype.Ptr(rstr.STR)) + dst = dstbox.getref(lltype.Ptr(rstr.STR)) srcstart = srcstartbox.getint() dststart = dststartbox.getint() length = lengthbox.getint() @@ -214,8 +214,8 @@ def do_copyunicodecontent(cpu, _, srcbox, dstbox, srcstartbox, dststartbox, lengthbox): - src = srcbox.getptr(lltype.Ptr(rstr.UNICODE)) - dst = dstbox.getptr(lltype.Ptr(rstr.UNICODE)) + src = srcbox.getref(lltype.Ptr(rstr.UNICODE)) + dst = dstbox.getref(lltype.Ptr(rstr.UNICODE)) srcstart = srcstartbox.getint() dststart = dststartbox.getint() length = lengthbox.getint() @@ -428,6 +428,10 @@ if arity == 3: func = get_execute_funclist(3, False)[opnum] return func(cpu, metainterp, argboxes[0], argboxes[1], argboxes[2]) + if arity == 5: # copystrcontent, copyunicodecontent + func = get_execute_funclist(5, False)[opnum] + return func(cpu, metainterp, argboxes[0], argboxes[1], + argboxes[2], argboxes[3], argboxes[4]) raise NotImplementedError Modified: pypy/branch/fast-forward/pypy/jit/metainterp/resoperation.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/metainterp/resoperation.py (original) +++ pypy/branch/fast-forward/pypy/jit/metainterp/resoperation.py Fri Oct 1 22:53:02 2010 @@ -455,7 +455,7 @@ 'UNICODESETITEM/3', 'NEWUNICODE/1', #'RUNTIMENEW/1', # ootype operation - 'COND_CALL_GC_WB/1d', # [objptr] (for the write barrier) + 'COND_CALL_GC_WB/2d', # [objptr, newvalue] (for the write barrier) 'DEBUG_MERGE_POINT/1', # debugging only 'VIRTUAL_REF_FINISH/2', # removed before it's passed to the backend 'COPYSTRCONTENT/5', # src, dst, srcstart, dststart, length Modified: pypy/branch/fast-forward/pypy/jit/metainterp/warmspot.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/metainterp/warmspot.py (original) +++ pypy/branch/fast-forward/pypy/jit/metainterp/warmspot.py Fri Oct 1 22:53:02 2010 @@ -67,9 +67,16 @@ def jittify_and_run(interp, graph, args, repeat=1, backendopt=False, trace_limit=sys.maxint, debug_level=DEBUG_STEPS, inline=False, **kwds): + from pypy.config.config import ConfigError translator = interp.typer.annotator.translator - translator.config.translation.gc = "boehm" - translator.config.translation.list_comprehension_operations = True + try: + translator.config.translation.gc = "boehm" + except ConfigError: + pass + try: + translator.config.translation.list_comprehension_operations = True + except ConfigError: + pass warmrunnerdesc = WarmRunnerDesc(translator, backendopt=backendopt, **kwds) for jd in warmrunnerdesc.jitdrivers_sd: jd.warmstate.set_param_threshold(3) # for tests Modified: pypy/branch/fast-forward/pypy/module/imp/importing.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/imp/importing.py (original) +++ pypy/branch/fast-forward/pypy/module/imp/importing.py Fri Oct 1 22:53:02 2010 @@ -122,7 +122,7 @@ n = len(ctxt_name_prefix_parts)-level+1 assert n>=0 ctxt_name_prefix_parts = ctxt_name_prefix_parts[:n] - if ctxt_w_path is None: # plain module + if ctxt_name_prefix_parts and ctxt_w_path is None: # plain module ctxt_name_prefix_parts.pop() if ctxt_name_prefix_parts: rel_modulename = '.'.join(ctxt_name_prefix_parts) Modified: pypy/branch/fast-forward/pypy/module/imp/test/test_import.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/imp/test/test_import.py (original) +++ pypy/branch/fast-forward/pypy/module/imp/test/test_import.py Fri Oct 1 22:53:02 2010 @@ -360,6 +360,12 @@ """.rstrip() raises(ValueError, imp) + def test_future_relative_import_error_when_in_non_package2(self): + exec """def imp(): + from .. import inpackage + """.rstrip() + raises(ValueError, imp) + def test_relative_import_with___name__(self): import sys mydict = {'__name__': 'sys.foo'} Modified: pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py (original) +++ pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py Fri Oct 1 22:53:02 2010 @@ -29,6 +29,7 @@ class RECT(Structure): _fields_ = [("left", c_int), ("top", c_int), ("right", c_int), ("bottom", c_int)] + class TestFunctions(BaseCTypesTestChecker): def test_mro(self): @@ -392,6 +393,18 @@ result = f("abcd", ord("b")) assert result == "bcd" + def test_caching_bug_1(self): + # the same test as test_call_some_args, with two extra lines + # in the middle that trigger caching in f._ptr, which then + # makes the last two lines fail + f = dll.my_strchr + f.argtypes = [c_char_p, c_int] + f.restype = c_char_p + result = f("abcd", ord("b")) + assert result == "bcd" + result = f("abcd", ord("b"), 42) + assert result == "bcd" + def test_sf1651235(self): py.test.skip("we are less strict in checking callback parameters") # see http://www.python.org/sf/1651235 Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/llarena.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/lltypesystem/llarena.py (original) +++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/llarena.py Fri Oct 1 22:53:02 2010 @@ -124,6 +124,9 @@ assert self.usagemap[i] == 'x' self.usagemap[i] = '#' + def mark_freed(self): + self.freed = True # this method is a hook for tests + class fakearenaaddress(llmemory.fakeaddress): def __init__(self, arena, offset): @@ -314,7 +317,7 @@ assert arena_addr.offset == 0 arena_addr.arena.reset(False) assert not arena_addr.arena.objectptrs - arena_addr.arena.freed = True + arena_addr.arena.mark_freed() def arena_reset(arena_addr, size, zero): """Free all objects in the arena, which can then be reused. Modified: pypy/branch/fast-forward/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gc/base.py Fri Oct 1 22:53:02 2010 @@ -20,12 +20,15 @@ prebuilt_gc_objects_are_static_roots = True object_minimal_size = 0 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE): + def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + translated_to_c=True): self.gcheaderbuilder = GCHeaderBuilder(self.HDR) self.AddressStack = get_address_stack(chunk_size) self.AddressDeque = get_address_deque(chunk_size) self.AddressDict = AddressDict self.config = config + assert isinstance(translated_to_c, bool) + self.translated_to_c = translated_to_c def setup(self): # all runtime mutable values' setup should happen here @@ -79,7 +82,7 @@ def set_root_walker(self, root_walker): self.root_walker = root_walker - def write_barrier(self, addr_struct): + def write_barrier(self, newvalue, addr_struct): pass def statistics(self, index): Modified: pypy/branch/fast-forward/pypy/rpython/memory/gc/generation.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gc/generation.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gc/generation.py Fri Oct 1 22:53:02 2010 @@ -5,7 +5,6 @@ from pypy.rpython.memory.gc.base import read_from_env from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage from pypy.rpython.lltypesystem import lltype, llmemory, llarena -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rlib.objectmodel import free_non_gc_object from pypy.rlib.debug import ll_assert from pypy.rlib.debug import debug_print, debug_start, debug_stop @@ -49,15 +48,17 @@ nursery_hash_base = -1 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + def __init__(self, config, nursery_size=32*WORD, min_nursery_size=32*WORD, auto_nursery_size=False, space_size=1024*WORD, - max_space_size=sys.maxint//2+1): - SemiSpaceGC.__init__(self, config, chunk_size = chunk_size, + max_space_size=sys.maxint//2+1, + **kwds): + SemiSpaceGC.__init__(self, config, space_size = space_size, - max_space_size = max_space_size) + max_space_size = max_space_size, + **kwds) assert min_nursery_size <= nursery_size <= space_size // 2 self.initial_nursery_size = nursery_size self.auto_nursery_size = auto_nursery_size @@ -157,6 +158,14 @@ "odd-valued (i.e. tagged) pointer unexpected here") return self.nursery <= addr < self.nursery_top + def appears_to_be_in_nursery(self, addr): + # same as is_in_nursery(), but may return True accidentally if + # 'addr' is a tagged pointer with just the wrong value. + if not self.translated_to_c: + if not self.is_valid_gc_object(addr): + return False + return self.nursery <= addr < self.nursery_top + def malloc_fixedsize_clear(self, typeid, size, can_collect, has_finalizer=False, contains_weakptr=False): if (has_finalizer or not can_collect or @@ -326,7 +335,7 @@ addr = pointer.address[0] newaddr = self.copy(addr) pointer.address[0] = newaddr - self.write_into_last_generation_obj(obj) + self.write_into_last_generation_obj(obj, newaddr) # ____________________________________________________________ # Implementation of nursery-only collections @@ -457,9 +466,9 @@ # "if addr_struct.int0 & JIT_WB_IF_FLAG: remember_young_pointer()") JIT_WB_IF_FLAG = GCFLAG_NO_YOUNG_PTRS - def write_barrier(self, addr_struct): - if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: - self.remember_young_pointer(addr_struct) + def write_barrier(self, newvalue, addr_struct): + if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: + self.remember_young_pointer(addr_struct, newvalue) def _setup_wb(self): DEBUG = self.DEBUG @@ -470,23 +479,33 @@ # For x86, there is also an extra requirement: when the JIT calls # remember_young_pointer(), it assumes that it will not touch the SSE # registers, so it does not save and restore them (that's a *hack*!). - def remember_young_pointer(addr_struct): + def remember_young_pointer(addr_struct, addr): #llop.debug_print(lltype.Void, "\tremember_young_pointer", # addr_struct, "<-", addr) if DEBUG: ll_assert(not self.is_in_nursery(addr_struct), "nursery object with GCFLAG_NO_YOUNG_PTRS") - self.old_objects_pointing_to_young.append(addr_struct) - self.header(addr_struct).tid &= ~GCFLAG_NO_YOUNG_PTRS - self.write_into_last_generation_obj(addr_struct) + # + # What is important in this function is that it *must* + # clear the flag GCFLAG_NO_YOUNG_PTRS from 'addr_struct' + # if 'addr' is in the nursery. It is ok if, accidentally, + # it also clears the flag in some more rare cases, like + # 'addr' being a tagged pointer whose value happens to be + # a large integer that fools is_in_nursery(). + if self.appears_to_be_in_nursery(addr): + self.old_objects_pointing_to_young.append(addr_struct) + self.header(addr_struct).tid &= ~GCFLAG_NO_YOUNG_PTRS + self.write_into_last_generation_obj(addr_struct, addr) remember_young_pointer._dont_inline_ = True self.remember_young_pointer = remember_young_pointer - def write_into_last_generation_obj(self, addr_struct): + def write_into_last_generation_obj(self, addr_struct, addr): objhdr = self.header(addr_struct) if objhdr.tid & GCFLAG_NO_HEAP_PTRS: - objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS - self.last_generation_root_objects.append(addr_struct) + if (self.is_valid_gc_object(addr) and + not self.is_last_generation(addr)): + objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS + self.last_generation_root_objects.append(addr_struct) write_into_last_generation_obj._always_inline_ = True def assume_young_pointers(self, addr_struct): Modified: pypy/branch/fast-forward/pypy/rpython/memory/gc/markcompact.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gc/markcompact.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gc/markcompact.py Fri Oct 1 22:53:02 2010 @@ -2,7 +2,6 @@ from pypy.rpython.memory.gc.base import MovingGCBase, read_from_env from pypy.rlib.debug import ll_assert, have_debug_prints from pypy.rlib.debug import debug_print, debug_start, debug_stop -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack, get_address_deque from pypy.rpython.memory.support import AddressDict from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage @@ -86,9 +85,9 @@ free = NULL next_collect_after = -1 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, space_size=4096, - min_next_collect_after=128): - MovingGCBase.__init__(self, config, chunk_size) + def __init__(self, config, space_size=4096, + min_next_collect_after=128, **kwds): + MovingGCBase.__init__(self, config, **kwds) self.space_size = space_size self.min_next_collect_after = min_next_collect_after Modified: pypy/branch/fast-forward/pypy/rpython/memory/gc/marksweep.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gc/marksweep.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gc/marksweep.py Fri Oct 1 22:53:02 2010 @@ -1,7 +1,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc, raw_free from pypy.rpython.lltypesystem.llmemory import raw_memcopy, raw_memclear from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack from pypy.rpython.memory.gcheader import GCHeaderBuilder from pypy.rpython.lltypesystem import lltype, llmemory, rffi, llgroup @@ -48,9 +47,9 @@ # translating to a real backend. TRANSLATION_PARAMS = {'start_heap_size': 8*1024*1024} # XXX adjust - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, start_heap_size=4096): + def __init__(self, config, start_heap_size=4096, **kwds): self.param_start_heap_size = start_heap_size - GCBase.__init__(self, config, chunk_size) + GCBase.__init__(self, config, **kwds) def setup(self): GCBase.setup(self) @@ -714,8 +713,8 @@ _alloc_flavor_ = "raw" COLLECT_EVERY = 2000 - def __init__(self, chunk_size=DEFAULT_CHUNK_SIZE, start_heap_size=4096): - MarkSweepGC.__init__(self, chunk_size, start_heap_size) + def __init__(self, config, **kwds): + MarkSweepGC.__init__(self, config, **kwds) self.count_mallocs = 0 def maybe_collect(self): Modified: pypy/branch/fast-forward/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gc/minimark.py Fri Oct 1 22:53:02 2010 @@ -4,7 +4,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc_usage from pypy.rpython.memory.gc.base import GCBase, MovingGCBase from pypy.rpython.memory.gc import minimarkpage, base, generation -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rlib.rarithmetic import ovfcheck, LONG_BIT, intmask, r_uint from pypy.rlib.rarithmetic import LONG_BIT_SHIFT from pypy.rlib.debug import ll_assert, debug_print, debug_start, debug_stop @@ -140,7 +139,7 @@ "large_object_gcptrs": 8250*WORD, } - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + def __init__(self, config, read_from_env=False, nursery_size=32*WORD, page_size=16*WORD, @@ -150,8 +149,9 @@ card_page_indices=0, large_object=8*WORD, large_object_gcptrs=10*WORD, - ArenaCollectionClass=None): - MovingGCBase.__init__(self, config, chunk_size) + ArenaCollectionClass=None, + **kwds): + MovingGCBase.__init__(self, config, **kwds) assert small_request_threshold % WORD == 0 self.read_from_env = read_from_env self.nursery_size = nursery_size @@ -636,6 +636,14 @@ "odd-valued (i.e. tagged) pointer unexpected here") return self.nursery <= addr < self.nursery_top + def appears_to_be_in_nursery(self, addr): + # same as is_in_nursery(), but may return True accidentally if + # 'addr' is a tagged pointer with just the wrong value. + if not self.translated_to_c: + if not self.is_valid_gc_object(addr): + return False + return self.nursery <= addr < self.nursery_top + def is_forwarded(self, obj): """Returns True if the nursery obj is marked as forwarded. Implemented a bit obscurely by checking an unrelated flag @@ -726,16 +734,16 @@ def JIT_max_size_of_young_obj(cls): return cls.TRANSLATION_PARAMS['large_object'] - def write_barrier(self, addr_struct): + def write_barrier(self, newvalue, addr_struct): if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: - self.remember_young_pointer(addr_struct) + self.remember_young_pointer(addr_struct, newvalue) - def write_barrier_from_array(self, addr_array, index): + def write_barrier_from_array(self, newvalue, addr_array, index): if self.header(addr_array).tid & GCFLAG_NO_YOUNG_PTRS: if self.card_page_indices > 0: # <- constant-folded self.remember_young_pointer_from_array(addr_array, index) else: - self.remember_young_pointer(addr_array) + self.remember_young_pointer(addr_array, newvalue) def _init_writebarrier_logic(self): DEBUG = self.DEBUG @@ -746,27 +754,28 @@ # For x86, there is also an extra requirement: when the JIT calls # remember_young_pointer(), it assumes that it will not touch the SSE # registers, so it does not save and restore them (that's a *hack*!). - def remember_young_pointer(addr_struct): + def remember_young_pointer(addr_struct, newvalue): # 'addr_struct' is the address of the object in which we write. + # 'newvalue' is the address that we are going to write in there. if DEBUG: ll_assert(not self.is_in_nursery(addr_struct), "nursery object with GCFLAG_NO_YOUNG_PTRS") # - # We assume that what we are writing is a pointer to the nursery - # (and don't care for the fact that this new pointer may not - # actually point to the nursery, which seems ok). What we need is + # If it seems that what we are writing is a pointer to the nursery + # (as checked with appears_to_be_in_nursery()), then we need # to remove the flag GCFLAG_NO_YOUNG_PTRS and add the old object # to the list 'old_objects_pointing_to_young'. We know that # 'addr_struct' cannot be in the nursery, because nursery objects # never have the flag GCFLAG_NO_YOUNG_PTRS to start with. - self.old_objects_pointing_to_young.append(addr_struct) objhdr = self.header(addr_struct) - objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS + if self.appears_to_be_in_nursery(newvalue): + self.old_objects_pointing_to_young.append(addr_struct) + objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS # # Second part: if 'addr_struct' is actually a prebuilt GC # object and it's the first time we see a write to it, we # add it to the list 'prebuilt_root_objects'. Note that we - # do it even in the (rare?) case of 'addr' being another + # do it even in the (rare?) case of 'addr' being NULL or another # prebuilt object, to simplify code. if objhdr.tid & GCFLAG_NO_HEAP_PTRS: objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS @@ -780,16 +789,24 @@ def _init_writebarrier_with_card_marker(self): + DEBUG = self.DEBUG def remember_young_pointer_from_array(addr_array, index): # 'addr_array' is the address of the object in which we write, # which must have an array part; 'index' is the index of the # item that is (or contains) the pointer that we write. + if DEBUG: + ll_assert(not self.is_in_nursery(addr_array), + "nursery array with GCFLAG_NO_YOUNG_PTRS") objhdr = self.header(addr_array) if objhdr.tid & GCFLAG_HAS_CARDS == 0: # - # no cards, use default logic. The 'nocard_logic()' is just - # 'remember_young_pointer()', but forced to be inlined here. - nocard_logic(addr_array) + # no cards, use default logic. Mostly copied from above. + self.old_objects_pointing_to_young.append(addr_array) + objhdr = self.header(addr_array) + objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS + if objhdr.tid & GCFLAG_NO_HEAP_PTRS: + objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS + self.prebuilt_root_objects.append(addr_array) return # # 'addr_array' is a raw_malloc'ed array with card markers @@ -807,17 +824,15 @@ return # # We set the flag (even if the newly written address does not - # actually point to the nursery -- like remember_young_pointer()). + # actually point to the nursery, which seems to be ok -- actually + # it seems more important that remember_young_pointer_from_array() + # does not take 3 arguments). addr_byte.char[0] = chr(byte | bitmask) # if objhdr.tid & GCFLAG_CARDS_SET == 0: self.old_objects_with_cards_set.append(addr_array) objhdr.tid |= GCFLAG_CARDS_SET - nocard_logic = func_with_new_name(self.remember_young_pointer, - 'remember_young_pointer_nocard') - del nocard_logic._dont_inline_ - nocard_logic._always_inline_ = True remember_young_pointer_from_array._dont_inline_ = True self.remember_young_pointer_from_array = ( remember_young_pointer_from_array) Modified: pypy/branch/fast-forward/pypy/rpython/memory/gc/minimarkpage.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gc/minimarkpage.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gc/minimarkpage.py Fri Oct 1 22:53:02 2010 @@ -4,15 +4,45 @@ from pypy.rlib.debug import ll_assert WORD = LONG_BIT // 8 -WORD_POWER_2 = {32: 2, 64: 3}[LONG_BIT] NULL = llmemory.NULL +WORD_POWER_2 = {32: 2, 64: 3}[LONG_BIT] +assert 1 << WORD_POWER_2 == WORD -# Terminology: the memory is subdivided into "pages". +# Terminology: the memory is subdivided into "arenas" containing "pages". # A page contains a number of allocated objects, called "blocks". -# The actual allocation occurs in whole arenas, which are subdivided -# into pages. We don't keep track of the arenas. A page can be: +# The actual allocation occurs in whole arenas, which are then subdivided +# into pages. For each arena we allocate one of the following structures: + +ARENA_PTR = lltype.Ptr(lltype.ForwardReference()) +ARENA = lltype.Struct('ArenaReference', + # -- The address of the arena, as returned by malloc() + ('base', llmemory.Address), + # -- The number of free and the total number of pages in the arena + ('nfreepages', lltype.Signed), + ('totalpages', lltype.Signed), + # -- A chained list of free pages in the arena. Ends with NULL. + ('freepages', llmemory.Address), + # -- A linked list of arenas. See below. + ('nextarena', ARENA_PTR), + ) +ARENA_PTR.TO.become(ARENA) +ARENA_NULL = lltype.nullptr(ARENA) + +# The idea is that when we need a free page, we take it from the arena +# which currently has the *lowest* number of free pages. This allows +# arenas with a lot of free pages to eventually become entirely free, at +# which point they are returned to the OS. If an arena has a total of +# 64 pages, then we have 64 global lists, arenas_lists[0] to +# arenas_lists[63], such that arenas_lists[i] contains exactly those +# arenas that have 'nfreepages == i'. We allocate pages out of the +# arena in 'current_arena'; when it is exhausted we pick another arena +# with the smallest value for nfreepages (but > 0). + +# ____________________________________________________________ +# +# Each page in an arena can be: # # - uninitialized: never touched so far. # @@ -21,10 +51,11 @@ # room for objects of that size, unless it is completely full. # # - free: used to be partially full, and is now free again. The page is -# on the chained list of free pages. +# on the chained list of free pages 'freepages' from its arena. -# Similarily, each allocated page contains blocks of a given size, which can -# be either uninitialized, allocated or free. +# Each allocated page contains blocks of a given size, which can again be in +# one of three states: allocated, free, or uninitialized. The uninitialized +# blocks (initially all of them) are at the tail of the page. PAGE_PTR = lltype.Ptr(lltype.ForwardReference()) PAGE_HEADER = lltype.Struct('PageHeader', @@ -32,13 +63,16 @@ # pages, it is a chained list of pages having the same size class, # rooted in 'page_for_size[size_class]'. For full pages, it is a # different chained list rooted in 'full_page_for_size[size_class]'. + # For free pages, it is the list 'freepages' in the arena header. ('nextpage', PAGE_PTR), - # -- The number of free blocks, and the number of uninitialized blocks. - # The number of allocated blocks is the rest. - ('nuninitialized', lltype.Signed), + # -- The arena this page is part of. + ('arena', ARENA_PTR), + # -- The number of free blocks. The numbers of uninitialized and + # allocated blocks can be deduced from the context if needed. ('nfree', lltype.Signed), - # -- The chained list of free blocks. If there are none, points to the - # first uninitialized block. + # -- The chained list of free blocks. It ends as a pointer to the + # first uninitialized block (pointing to data that is uninitialized, + # or to the end of the page). ('freeblock', llmemory.Address), # -- The structure above is 4 words, which is a good value: # '(1024-4) % N' is zero or very small for various small N's, @@ -72,13 +106,35 @@ self.nblocks_for_size = lltype.malloc(rffi.CArray(lltype.Signed), length, flavor='raw') self.hdrsize = llmemory.raw_malloc_usage(llmemory.sizeof(PAGE_HEADER)) + assert page_size > self.hdrsize self.nblocks_for_size[0] = 0 # unused for i in range(1, length): self.nblocks_for_size[i] = (page_size - self.hdrsize) // (WORD * i) # - self.uninitialized_pages = NULL + self.max_pages_per_arena = arena_size // page_size + self.arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR), + self.max_pages_per_arena, + flavor='raw', zero=True) + # this is used in mass_free() only + self.old_arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR), + self.max_pages_per_arena, + flavor='raw', zero=True) + # + # the arena currently consumed; it must have at least one page + # available, or be NULL. The arena object that we point to is + # not in any 'arenas_lists'. We will consume all its pages before + # we choose a next arena, even if there is a major collection + # in-between. + self.current_arena = ARENA_NULL + # + # guarantee that 'arenas_lists[1:min_empty_nfreepages]' are all empty + self.min_empty_nfreepages = self.max_pages_per_arena + # + # part of current_arena might still contain uninitialized pages self.num_uninitialized_pages = 0 - self.free_pages = NULL + # + # the total memory used, counting every block in use, without + # the additional bookkeeping stuff. self.total_memory_used = r_uint(0) @@ -109,16 +165,12 @@ # else: # The 'result' is part of the uninitialized blocks. - ll_assert(page.nuninitialized > 0, - "fully allocated page found in the page_for_size list") - page.nuninitialized -= 1 - if page.nuninitialized > 0: - freeblock = result + nsize - else: - freeblock = NULL + freeblock = result + nsize # page.freeblock = freeblock - if freeblock == NULL: + # + pageaddr = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) + if freeblock - pageaddr > self.page_size - nsize: # This was the last free block, so unlink the page from the # chained list and put it in the 'full_page_for_size' list. self.page_for_size[size_class] = page.nextpage @@ -132,37 +184,88 @@ def allocate_new_page(self, size_class): """Allocate and return a new page for the given size_class.""" # - if self.free_pages != NULL: + # Allocate a new arena if needed. + if self.current_arena == ARENA_NULL: + self.allocate_new_arena() + # + # The result is simply 'current_arena.freepages'. + arena = self.current_arena + result = arena.freepages + if arena.nfreepages > 0: + # + # The 'result' was part of the chained list; read the next. + arena.nfreepages -= 1 + freepages = result.address[0] + llarena.arena_reset(result, + llmemory.sizeof(llmemory.Address), + 0) # - # Get the page from the chained list 'free_pages'. - page = self.free_pages - self.free_pages = page.address[0] - llarena.arena_reset(page, llmemory.sizeof(llmemory.Address), 0) else: - # Get the next free page from the uninitialized pages. - if self.num_uninitialized_pages == 0: - self.allocate_new_arena() # Out of memory. Get a new arena. - page = self.uninitialized_pages - self.uninitialized_pages += self.page_size + # The 'result' is part of the uninitialized pages. + ll_assert(self.num_uninitialized_pages > 0, + "fully allocated arena found in self.current_arena") self.num_uninitialized_pages -= 1 + if self.num_uninitialized_pages > 0: + freepages = result + self.page_size + else: + freepages = NULL # - # Initialize the fields of the resulting page - llarena.arena_reserve(page, llmemory.sizeof(PAGE_HEADER)) - result = llmemory.cast_adr_to_ptr(page, PAGE_PTR) + arena.freepages = freepages + if freepages == NULL: + # This was the last page, so put the arena away into + # arenas_lists[0]. + ll_assert(arena.nfreepages == 0, + "freepages == NULL but nfreepages > 0") + arena.nextarena = self.arenas_lists[0] + self.arenas_lists[0] = arena + self.current_arena = ARENA_NULL # - result.nuninitialized = self.nblocks_for_size[size_class] - result.nfree = 0 - result.freeblock = page + self.hdrsize - result.nextpage = PAGE_NULL + # Initialize the fields of the resulting page + llarena.arena_reserve(result, llmemory.sizeof(PAGE_HEADER)) + page = llmemory.cast_adr_to_ptr(result, PAGE_PTR) + page.arena = arena + page.nfree = 0 + page.freeblock = result + self.hdrsize + page.nextpage = PAGE_NULL ll_assert(self.page_for_size[size_class] == PAGE_NULL, "allocate_new_page() called but a page is already waiting") - self.page_for_size[size_class] = result - return result + self.page_for_size[size_class] = page + return page + + + def _all_arenas(self): + """For testing. Enumerates all arenas.""" + if self.current_arena: + yield self.current_arena + for arena in self.arenas_lists: + while arena: + yield arena + arena = arena.nextarena def allocate_new_arena(self): - ll_assert(self.num_uninitialized_pages == 0, - "some uninitialized pages are already waiting") + """Loads in self.current_arena the arena to allocate from next.""" + # + # Pick an arena from 'arenas_lists[i]', with i as small as possible + # but > 0. Use caching with 'min_empty_nfreepages', which guarantees + # that 'arenas_lists[1:min_empty_nfreepages]' are all empty. + i = self.min_empty_nfreepages + while i < self.max_pages_per_arena: + # + if self.arenas_lists[i] != ARENA_NULL: + # + # Found it. + self.current_arena = self.arenas_lists[i] + self.arenas_lists[i] = self.current_arena.nextarena + return + # + i += 1 + self.min_empty_nfreepages = i + # + # No more arena with any free page. We must allocate a new arena. + if not we_are_translated(): + for a in self._all_arenas(): + assert a.nfreepages == 0 # # 'arena_base' points to the start of malloced memory; it might not # be a page-aligned address @@ -177,13 +280,15 @@ # 'npages' is the number of full pages just allocated npages = (arena_end - firstpage) // self.page_size # - # add these pages to the list - self.uninitialized_pages = firstpage + # Allocate an ARENA object and initialize it + arena = lltype.malloc(ARENA, flavor='raw') + arena.base = arena_base + arena.nfreepages = 0 # they are all uninitialized pages + arena.totalpages = npages + arena.freepages = firstpage self.num_uninitialized_pages = npages + self.current_arena = arena # - # increase a bit arena_size for the next time - self.arena_size = (self.arena_size // 4 * 5) + (self.page_size - 1) - self.arena_size = (self.arena_size // self.page_size) * self.page_size allocate_new_arena._dont_inline_ = True @@ -199,16 +304,51 @@ # # Walk the pages in 'page_for_size[size_class]' and # 'full_page_for_size[size_class]' and free some objects. - # Pages completely freed are added to 'self.free_pages', and - # become available for reuse by any size class. Pages not - # completely freed are re-chained either in + # Pages completely freed are added to 'page.arena.freepages', + # and become available for reuse by any size class. Pages + # not completely freed are re-chained either in # 'full_page_for_size[]' or 'page_for_size[]'. - self.mass_free_in_page(size_class, ok_to_free_func) + self.mass_free_in_pages(size_class, ok_to_free_func) # size_class -= 1 + # + # Rehash arenas into the correct arenas_lists[i]. If + # 'self.current_arena' contains an arena too, it remains there. + (self.old_arenas_lists, self.arenas_lists) = ( + self.arenas_lists, self.old_arenas_lists) + # + i = 0 + while i < self.max_pages_per_arena: + self.arenas_lists[i] = ARENA_NULL + i += 1 + # + i = 0 + while i < self.max_pages_per_arena: + arena = self.old_arenas_lists[i] + while arena != ARENA_NULL: + nextarena = arena.nextarena + # + if arena.nfreepages == arena.totalpages: + # + # The whole arena is empty. Free it. + llarena.arena_free(arena.base) + lltype.free(arena, flavor='raw') + # + else: + # Insert 'arena' in the correct arenas_lists[n] + n = arena.nfreepages + ll_assert(n < self.max_pages_per_arena, + "totalpages != nfreepages >= max_pages_per_arena") + arena.nextarena = self.arenas_lists[n] + self.arenas_lists[n] = arena + # + arena = nextarena + i += 1 + # + self.min_empty_nfreepages = 1 - def mass_free_in_page(self, size_class, ok_to_free_func): + def mass_free_in_pages(self, size_class, ok_to_free_func): nblocks = self.nblocks_for_size[size_class] block_size = size_class * WORD remaining_partial_pages = PAGE_NULL @@ -224,8 +364,7 @@ while page != PAGE_NULL: # # Collect the page. - surviving = self.walk_page(page, block_size, - nblocks, ok_to_free_func) + surviving = self.walk_page(page, block_size, ok_to_free_func) nextpage = page.nextpage # if surviving == nblocks: @@ -259,19 +398,23 @@ def free_page(self, page): """Free a whole page.""" # - # Done by inserting it in the 'free_pages' list. + # Insert the freed page in the arena's 'freepages' list. + # If nfreepages == totalpages, then it will be freed at the + # end of mass_free(). + arena = page.arena + arena.nfreepages += 1 pageaddr = llmemory.cast_ptr_to_adr(page) pageaddr = llarena.getfakearenaaddress(pageaddr) llarena.arena_reset(pageaddr, self.page_size, 0) llarena.arena_reserve(pageaddr, llmemory.sizeof(llmemory.Address)) - pageaddr.address[0] = self.free_pages - self.free_pages = pageaddr + pageaddr.address[0] = arena.freepages + arena.freepages = pageaddr - def walk_page(self, page, block_size, nblocks, ok_to_free_func): + def walk_page(self, page, block_size, ok_to_free_func): """Walk over all objects in a page, and ask ok_to_free_func().""" # - # 'freeblock' is the next free block, or NULL if there isn't any more. + # 'freeblock' is the next free block freeblock = page.freeblock # # 'prevfreeblockat' is the address of where 'freeblock' was read from. @@ -281,22 +424,28 @@ obj = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) obj += self.hdrsize surviving = 0 # initially + skip_free_blocks = page.nfree # - nblocks -= page.nuninitialized - index = nblocks - while index > 0: + while True: # if obj == freeblock: # + if skip_free_blocks == 0: + # + # 'obj' points to the first uninitialized block, + # or to the end of the page if there are none. + break + # # 'obj' points to a free block. It means that # 'prevfreeblockat.address[0]' does not need to be updated. # Just read the next free block from 'obj.address[0]'. + skip_free_blocks -= 1 prevfreeblockat = obj freeblock = obj.address[0] # else: # 'obj' points to a valid object. - ll_assert(not freeblock or freeblock > obj, + ll_assert(freeblock > obj, "freeblocks are linked out of order") # if ok_to_free_func(obj): @@ -310,15 +459,14 @@ prevfreeblockat = obj obj.address[0] = freeblock # + # Update the number of free objects in the page. + page.nfree += 1 + # else: # The object survives. surviving += 1 # obj += block_size - index -= 1 - # - # Update the number of free objects in the page. - page.nfree = nblocks - surviving # # Update the global total size of objects. self.total_memory_used += surviving * block_size @@ -327,6 +475,20 @@ return surviving + def _nuninitialized(self, page, size_class): + # Helper for debugging: count the number of uninitialized blocks + freeblock = page.freeblock + for i in range(page.nfree): + freeblock = freeblock.address[0] + assert freeblock != NULL + pageaddr = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) + num_initialized_blocks, rem = divmod( + freeblock - pageaddr - self.hdrsize, size_class * WORD) + assert rem == 0, "page size_class misspecified?" + nblocks = self.nblocks_for_size[size_class] + return nblocks - num_initialized_blocks + + # ____________________________________________________________ # Helpers to go from a pointer to the start of its page Modified: pypy/branch/fast-forward/pypy/rpython/memory/gc/semispace.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gc/semispace.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gc/semispace.py Fri Oct 1 22:53:02 2010 @@ -1,7 +1,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc, raw_free from pypy.rpython.lltypesystem.llmemory import raw_memcopy, raw_memclear from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack, get_address_deque from pypy.rpython.memory.support import AddressDict from pypy.rpython.lltypesystem import lltype, llmemory, llarena, rffi, llgroup @@ -59,11 +58,11 @@ # translating to a real backend. TRANSLATION_PARAMS = {'space_size': 8*1024*1024} # XXX adjust - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, space_size=4096, - max_space_size=sys.maxint//2+1): + def __init__(self, config, space_size=4096, max_space_size=sys.maxint//2+1, + **kwds): self.param_space_size = space_size self.param_max_space_size = max_space_size - MovingGCBase.__init__(self, config, chunk_size) + MovingGCBase.__init__(self, config, **kwds) def setup(self): #self.total_collection_time = 0.0 Modified: pypy/branch/fast-forward/pypy/rpython/memory/gc/test/test_direct.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gc/test/test_direct.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gc/test/test_direct.py Fri Oct 1 22:53:02 2010 @@ -70,6 +70,7 @@ GC_PARAMS = self.GC_PARAMS.copy() if hasattr(meth, 'GC_PARAMS'): GC_PARAMS.update(meth.GC_PARAMS) + GC_PARAMS['translated_to_c'] = False self.gc = self.GCClass(config, **GC_PARAMS) self.gc.DEBUG = True self.rootwalker = DirectRootWalker(self) @@ -86,17 +87,19 @@ def write(self, p, fieldname, newvalue): if self.gc.needs_write_barrier: + newaddr = llmemory.cast_ptr_to_adr(newvalue) addr_struct = llmemory.cast_ptr_to_adr(p) - self.gc.write_barrier(addr_struct) + self.gc.write_barrier(newaddr, addr_struct) setattr(p, fieldname, newvalue) def writearray(self, p, index, newvalue): if self.gc.needs_write_barrier: + newaddr = llmemory.cast_ptr_to_adr(newvalue) addr_struct = llmemory.cast_ptr_to_adr(p) if hasattr(self.gc, 'write_barrier_from_array'): - self.gc.write_barrier_from_array(addr_struct, index) + self.gc.write_barrier_from_array(newaddr, addr_struct, index) else: - self.gc.write_barrier(addr_struct) + self.gc.write_barrier(newaddr, addr_struct) p[index] = newvalue def malloc(self, TYPE, n=None): Modified: pypy/branch/fast-forward/pypy/rpython/memory/gc/test/test_minimarkpage.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gc/test/test_minimarkpage.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gc/test/test_minimarkpage.py Fri Oct 1 22:53:02 2010 @@ -12,17 +12,19 @@ def test_allocate_arena(): - ac = ArenaCollection(SHIFT + 16*20, 16, 1) + ac = ArenaCollection(SHIFT + 64*20, 64, 1) ac.allocate_new_arena() assert ac.num_uninitialized_pages == 20 - ac.uninitialized_pages + 16*20 # does not raise - py.test.raises(llarena.ArenaError, "ac.uninitialized_pages + 16*20 + 1") + upages = ac.current_arena.freepages + upages + 64*20 # does not raise + py.test.raises(llarena.ArenaError, "upages + 64*20 + 1") # - ac = ArenaCollection(SHIFT + 16*20 + 7, 16, 1) + ac = ArenaCollection(SHIFT + 64*20 + 7, 64, 1) ac.allocate_new_arena() assert ac.num_uninitialized_pages == 20 - ac.uninitialized_pages + 16*20 + 7 # does not raise - py.test.raises(llarena.ArenaError, "ac.uninitialized_pages + 16*20 + 16") + upages = ac.current_arena.freepages + upages + 64*20 + 7 # does not raise + py.test.raises(llarena.ArenaError, "upages + 64*20 + 64") def test_allocate_new_page(): @@ -31,7 +33,8 @@ # def checknewpage(page, size_class): size = WORD * size_class - assert page.nuninitialized == (pagesize - hdrsize) // size + assert (ac._nuninitialized(page, size_class) == + (pagesize - hdrsize) // size) assert page.nfree == 0 page1 = page.freeblock - hdrsize assert llmemory.cast_ptr_to_adr(page) == page1 @@ -44,13 +47,13 @@ page = ac.allocate_new_page(5) checknewpage(page, 5) assert ac.num_uninitialized_pages == 2 - assert ac.uninitialized_pages - pagesize == cast_ptr_to_adr(page) + assert ac.current_arena.freepages - pagesize == cast_ptr_to_adr(page) assert ac.page_for_size[5] == page # page = ac.allocate_new_page(3) checknewpage(page, 3) assert ac.num_uninitialized_pages == 1 - assert ac.uninitialized_pages - pagesize == cast_ptr_to_adr(page) + assert ac.current_arena.freepages - pagesize == cast_ptr_to_adr(page) assert ac.page_for_size[3] == page # page = ac.allocate_new_page(4) @@ -71,17 +74,17 @@ page = llmemory.cast_adr_to_ptr(pageaddr, PAGE_PTR) if step == 1: page.nfree = 0 - page.nuninitialized = nblocks - nusedblocks + nuninitialized = nblocks - nusedblocks else: page.nfree = nusedblocks - page.nuninitialized = nblocks - 2*nusedblocks + nuninitialized = nblocks - 2*nusedblocks + page.freeblock = pageaddr + hdrsize + nusedblocks * size_block if nusedblocks < nblocks: - page.freeblock = pageaddr + hdrsize + nusedblocks * size_block chainedlists = ac.page_for_size else: - page.freeblock = NULL chainedlists = ac.full_page_for_size page.nextpage = chainedlists[size_class] + page.arena = ac.current_arena chainedlists[size_class] = page if fill_with_objects: for i in range(0, nusedblocks*step, step): @@ -98,11 +101,15 @@ prev = 'prevhole.address[0]' endaddr = pageaddr + hdrsize + 2*nusedblocks * size_block exec '%s = endaddr' % prev in globals(), locals() + assert ac._nuninitialized(page, size_class) == nuninitialized # ac.allocate_new_arena() num_initialized_pages = len(pagelayout.rstrip(" ")) - ac._startpageaddr = ac.uninitialized_pages - ac.uninitialized_pages += pagesize * num_initialized_pages + ac._startpageaddr = ac.current_arena.freepages + if pagelayout.endswith(" "): + ac.current_arena.freepages += pagesize * num_initialized_pages + else: + ac.current_arena.freepages = NULL ac.num_uninitialized_pages -= num_initialized_pages # for i in reversed(range(num_initialized_pages)): @@ -115,8 +122,9 @@ link(pageaddr, size_class, size_block, nblocks, nblocks-1) elif c == '.': # a free, but initialized, page llarena.arena_reserve(pageaddr, llmemory.sizeof(llmemory.Address)) - pageaddr.address[0] = ac.free_pages - ac.free_pages = pageaddr + pageaddr.address[0] = ac.current_arena.freepages + ac.current_arena.freepages = pageaddr + ac.current_arena.nfreepages += 1 elif c == '#': # a random full page, in the list 'full_pages' size_class = fill_with_objects or 1 size_block = WORD * size_class @@ -142,26 +150,29 @@ def checkpage(ac, page, expected_position): assert llmemory.cast_ptr_to_adr(page) == pagenum(ac, expected_position) +def freepages(ac): + return ac.current_arena.freepages + def test_simple_arena_collection(): pagesize = hdrsize + 16 ac = arena_collection_for_test(pagesize, "##....# ") # - assert ac.free_pages == pagenum(ac, 2) + assert freepages(ac) == pagenum(ac, 2) page = ac.allocate_new_page(1); checkpage(ac, page, 2) - assert ac.free_pages == pagenum(ac, 3) + assert freepages(ac) == pagenum(ac, 3) page = ac.allocate_new_page(2); checkpage(ac, page, 3) - assert ac.free_pages == pagenum(ac, 4) + assert freepages(ac) == pagenum(ac, 4) page = ac.allocate_new_page(3); checkpage(ac, page, 4) - assert ac.free_pages == pagenum(ac, 5) + assert freepages(ac) == pagenum(ac, 5) page = ac.allocate_new_page(4); checkpage(ac, page, 5) - assert ac.free_pages == NULL and ac.num_uninitialized_pages == 3 + assert freepages(ac) == pagenum(ac, 7) and ac.num_uninitialized_pages == 3 page = ac.allocate_new_page(5); checkpage(ac, page, 7) - assert ac.free_pages == NULL and ac.num_uninitialized_pages == 2 + assert freepages(ac) == pagenum(ac, 8) and ac.num_uninitialized_pages == 2 page = ac.allocate_new_page(6); checkpage(ac, page, 8) - assert ac.free_pages == NULL and ac.num_uninitialized_pages == 1 + assert freepages(ac) == pagenum(ac, 9) and ac.num_uninitialized_pages == 1 page = ac.allocate_new_page(7); checkpage(ac, page, 9) - assert ac.free_pages == NULL and ac.num_uninitialized_pages == 0 + assert not ac.current_arena and ac.num_uninitialized_pages == 0 def chkob(ac, num_page, pos_obj, obj): @@ -205,47 +216,47 @@ ac = arena_collection_for_test(pagesize, "/.", fill_with_objects=2) page = getpage(ac, 0) assert page.nfree == 3 - assert page.nuninitialized == 3 + assert ac._nuninitialized(page, 2) == 3 chkob(ac, 0, 2*WORD, page.freeblock) # obj = ac.malloc(2*WORD); chkob(ac, 0, 2*WORD, obj) obj = ac.malloc(2*WORD); chkob(ac, 0, 6*WORD, obj) assert page.nfree == 1 - assert page.nuninitialized == 3 + assert ac._nuninitialized(page, 2) == 3 chkob(ac, 0, 10*WORD, page.freeblock) # obj = ac.malloc(2*WORD); chkob(ac, 0, 10*WORD, obj) assert page.nfree == 0 - assert page.nuninitialized == 3 + assert ac._nuninitialized(page, 2) == 3 chkob(ac, 0, 12*WORD, page.freeblock) # obj = ac.malloc(2*WORD); chkob(ac, 0, 12*WORD, obj) - assert page.nuninitialized == 2 + assert ac._nuninitialized(page, 2) == 2 obj = ac.malloc(2*WORD); chkob(ac, 0, 14*WORD, obj) obj = ac.malloc(2*WORD); chkob(ac, 0, 16*WORD, obj) assert page.nfree == 0 - assert page.nuninitialized == 0 + assert ac._nuninitialized(page, 2) == 0 obj = ac.malloc(2*WORD); chkob(ac, 1, 0*WORD, obj) def test_malloc_new_arena(): pagesize = hdrsize + 7*WORD ac = arena_collection_for_test(pagesize, "### ") + arena_size = ac.arena_size obj = ac.malloc(2*WORD); chkob(ac, 3, 0*WORD, obj) # 3rd page -> size 2 # del ac.allocate_new_arena # restore the one from the class - arena_size = ac.arena_size obj = ac.malloc(3*WORD) # need a new arena assert ac.num_uninitialized_pages == (arena_size // ac.page_size - - 1 # for start_of_page() - 1 # the just-allocated page ) class OkToFree(object): - def __init__(self, ac, answer): + def __init__(self, ac, answer, multiarenas=False): assert callable(answer) or 0.0 <= answer <= 1.0 self.ac = ac self.answer = answer + self.multiarenas = multiarenas self.lastnum = 0.0 self.seen = {} @@ -257,7 +268,10 @@ ok_to_free = self.lastnum >= 1.0 if ok_to_free: self.lastnum -= 1.0 - key = addr - self.ac._startpageaddr + if self.multiarenas: + key = (addr.arena, addr.offset) + else: + key = addr - self.ac._startpageaddr assert key not in self.seen self.seen[key] = ok_to_free return ok_to_free @@ -272,10 +286,10 @@ page = getpage(ac, 0) assert page == ac.page_for_size[2] assert page.nextpage == PAGE_NULL - assert page.nuninitialized == 1 + assert ac._nuninitialized(page, 2) == 1 assert page.nfree == 0 chkob(ac, 0, 4*WORD, page.freeblock) - assert ac.free_pages == NULL + assert freepages(ac) == NULL def test_mass_free_emptied_page(): pagesize = hdrsize + 7*WORD @@ -285,7 +299,7 @@ assert ok_to_free.seen == {hdrsize + 0*WORD: True, hdrsize + 2*WORD: True} pageaddr = pagenum(ac, 0) - assert pageaddr == ac.free_pages + assert pageaddr == freepages(ac) assert pageaddr.address[0] == NULL assert ac.page_for_size[2] == PAGE_NULL @@ -300,10 +314,9 @@ page = getpage(ac, 0) assert page == ac.full_page_for_size[2] assert page.nextpage == PAGE_NULL - assert page.nuninitialized == 0 + assert ac._nuninitialized(page, 2) == 0 assert page.nfree == 0 - assert page.freeblock == NULL - assert ac.free_pages == NULL + assert freepages(ac) == NULL assert ac.page_for_size[2] == PAGE_NULL def test_mass_free_full_is_partially_emptied(): @@ -319,19 +332,19 @@ pageaddr = pagenum(ac, 0) assert page == ac.page_for_size[2] assert page.nextpage == PAGE_NULL - assert page.nuninitialized == 0 + assert ac._nuninitialized(page, 2) == 0 assert page.nfree == 2 assert page.freeblock == pageaddr + hdrsize + 2*WORD assert page.freeblock.address[0] == pageaddr + hdrsize + 6*WORD - assert page.freeblock.address[0].address[0] == NULL - assert ac.free_pages == NULL + assert page.freeblock.address[0].address[0] == pageaddr + hdrsize + 8*WORD + assert freepages(ac) == NULL assert ac.full_page_for_size[2] == PAGE_NULL def test_mass_free_half_page_remains(): pagesize = hdrsize + 24*WORD ac = arena_collection_for_test(pagesize, "/", fill_with_objects=2) page = getpage(ac, 0) - assert page.nuninitialized == 4 + assert ac._nuninitialized(page, 2) == 4 assert page.nfree == 4 # ok_to_free = OkToFree(ac, False) @@ -344,7 +357,7 @@ pageaddr = pagenum(ac, 0) assert page == ac.page_for_size[2] assert page.nextpage == PAGE_NULL - assert page.nuninitialized == 4 + assert ac._nuninitialized(page, 2) == 4 assert page.nfree == 4 assert page.freeblock == pageaddr + hdrsize + 2*WORD assert page.freeblock.address[0] == pageaddr + hdrsize + 6*WORD @@ -352,14 +365,14 @@ pageaddr + hdrsize + 10*WORD assert page.freeblock.address[0].address[0].address[0] == \ pageaddr + hdrsize + 14*WORD - assert ac.free_pages == NULL + assert freepages(ac) == NULL assert ac.full_page_for_size[2] == PAGE_NULL def test_mass_free_half_page_becomes_more_free(): pagesize = hdrsize + 24*WORD ac = arena_collection_for_test(pagesize, "/", fill_with_objects=2) page = getpage(ac, 0) - assert page.nuninitialized == 4 + assert ac._nuninitialized(page, 2) == 4 assert page.nfree == 4 # ok_to_free = OkToFree(ac, 0.5) @@ -372,7 +385,7 @@ pageaddr = pagenum(ac, 0) assert page == ac.page_for_size[2] assert page.nextpage == PAGE_NULL - assert page.nuninitialized == 4 + assert ac._nuninitialized(page, 2) == 4 assert page.nfree == 6 fb = page.freeblock assert fb == pageaddr + hdrsize + 2*WORD @@ -384,7 +397,7 @@ pageaddr + hdrsize + 12*WORD assert fb.address[0].address[0].address[0].address[0].address[0] == \ pageaddr + hdrsize + 14*WORD - assert ac.free_pages == NULL + assert freepages(ac) == NULL assert ac.full_page_for_size[2] == PAGE_NULL # ____________________________________________________________ @@ -392,17 +405,29 @@ def test_random(): import random pagesize = hdrsize + 24*WORD - num_pages = 28 + num_pages = 3 ac = arena_collection_for_test(pagesize, " " * num_pages) live_objects = {} # - # Run the test until ac.allocate_new_arena() is called. + # Run the test until three arenas are freed. This is a quick test + # that the arenas are really freed by the logic. class DoneTesting(Exception): - pass - def done_testing(): - raise DoneTesting - ac.allocate_new_arena = done_testing - # + counter = 0 + def my_allocate_new_arena(): + # the following output looks cool on a 112-character-wide terminal. + lst = sorted(ac._all_arenas(), key=lambda a: a.base.arena._arena_index) + for a in lst: + print a.base.arena, a.base.arena.usagemap + print '-' * 80 + ac.__class__.allocate_new_arena(ac) + a = ac.current_arena.base.arena + def my_mark_freed(): + a.freed = True + DoneTesting.counter += 1 + if DoneTesting.counter > 3: + raise DoneTesting + a.mark_freed = my_mark_freed + ac.allocate_new_arena = my_allocate_new_arena try: while True: # @@ -410,12 +435,13 @@ for i in range(random.randrange(50, 100)): size_class = random.randrange(1, 7) obj = ac.malloc(size_class * WORD) - at = obj - ac._startpageaddr + at = (obj.arena, obj.offset) assert at not in live_objects live_objects[at] = size_class * WORD # # Free half the objects, randomly - ok_to_free = OkToFree(ac, lambda obj: random.random() < 0.5) + ok_to_free = OkToFree(ac, lambda obj: random.random() < 0.5, + multiarenas=True) ac.mass_free(ok_to_free) # # Check that we have seen all objects @@ -428,5 +454,4 @@ surviving_total_size += live_objects[at] assert ac.total_memory_used == surviving_total_size except DoneTesting: - # the following output looks cool on a 112-character-wide terminal. - print ac._startpageaddr.arena.usagemap + pass Modified: pypy/branch/fast-forward/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gctransform/framework.py Fri Oct 1 22:53:02 2010 @@ -426,6 +426,7 @@ if GCClass.needs_write_barrier: self.write_barrier_ptr = getfn(GCClass.write_barrier.im_func, [s_gc, + annmodel.SomeAddress(), annmodel.SomeAddress()], annmodel.s_None, inline=True) @@ -434,13 +435,15 @@ # func should not be a bound method, but a real function assert isinstance(func, types.FunctionType) self.write_barrier_failing_case_ptr = getfn(func, - [annmodel.SomeAddress()], + [annmodel.SomeAddress(), + annmodel.SomeAddress()], annmodel.s_None) func = getattr(GCClass, 'write_barrier_from_array', None) if func is not None: self.write_barrier_from_array_ptr = getfn(func.im_func, [s_gc, annmodel.SomeAddress(), + annmodel.SomeAddress(), annmodel.SomeInteger()], annmodel.s_None, inline=True) @@ -1021,6 +1024,8 @@ and not isinstance(v_newvalue, Constant) and v_struct.concretetype.TO._gckind == "gc" and hop.spaceop not in self.clean_sets): + v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], + resulttype = llmemory.Address) v_structaddr = hop.genop("cast_ptr_to_adr", [v_struct], resulttype = llmemory.Address) if (self.write_barrier_from_array_ptr is not None and @@ -1030,12 +1035,14 @@ assert v_index.concretetype == lltype.Signed hop.genop("direct_call", [self.write_barrier_from_array_ptr, self.c_const_gc, + v_newvalue, v_structaddr, v_index]) else: self.write_barrier_calls += 1 hop.genop("direct_call", [self.write_barrier_ptr, self.c_const_gc, + v_newvalue, v_structaddr]) hop.rename('bare_' + opname) Modified: pypy/branch/fast-forward/pypy/rpython/memory/gcwrapper.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/gcwrapper.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/gcwrapper.py Fri Oct 1 22:53:02 2010 @@ -9,7 +9,10 @@ def __init__(self, llinterp, flowgraphs, gc_class, GC_PARAMS={}): translator = llinterp.typer.annotator.translator config = translator.config.translation - self.gc = gc_class(config, chunk_size = 10, **GC_PARAMS) + self.gc = gc_class(config, + chunk_size = 10, + translated_to_c = False, + **GC_PARAMS) self.gc.set_root_walker(LLInterpRootWalker(self)) self.gc.DEBUG = True self.llinterp = llinterp @@ -94,6 +97,7 @@ assert (type(index) is int # <- fast path or lltype.typeOf(index) == lltype.Signed) self.gc.write_barrier_from_array( + llmemory.cast_ptr_to_adr(newvalue), llmemory.cast_ptr_to_adr(toplevelcontainer), index) wb = False @@ -101,6 +105,7 @@ # if wb: self.gc.write_barrier( + llmemory.cast_ptr_to_adr(newvalue), llmemory.cast_ptr_to_adr(toplevelcontainer)) llheap.setinterior(toplevelcontainer, inneraddr, INNERTYPE, newvalue) Modified: pypy/branch/fast-forward/pypy/rpython/memory/test/test_transformed_gc.py ============================================================================== --- pypy/branch/fast-forward/pypy/rpython/memory/test/test_transformed_gc.py (original) +++ pypy/branch/fast-forward/pypy/rpython/memory/test/test_transformed_gc.py Fri Oct 1 22:53:02 2010 @@ -906,7 +906,8 @@ gcname = "marksweep" class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 @@ -1144,7 +1145,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.marksweep import PrintingMarkSweepGC as GCClass - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestSemiSpaceGC(GenericMovingGCTests): @@ -1154,7 +1156,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.semispace import SemiSpaceGC as GCClass - GC_PARAMS = {'space_size': 512*WORD} + GC_PARAMS = {'space_size': 512*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestMarkCompactGC(GenericMovingGCTests): @@ -1163,7 +1166,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.markcompact import MarkCompactGC as GCClass - GC_PARAMS = {'space_size': 4096*WORD} + GC_PARAMS = {'space_size': 4096*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestGenerationGC(GenericMovingGCTests): @@ -1175,7 +1179,8 @@ from pypy.rpython.memory.gc.generation import GenerationGC as \ GCClass GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 32*WORD} + 'nursery_size': 32*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_weakref_across_minor_collection(cls): @@ -1372,7 +1377,8 @@ GenerationGC._teardown(self) GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 128*WORD} + 'nursery_size': 128*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_working_nursery(cls): @@ -1404,7 +1410,8 @@ from pypy.rpython.memory.gc.hybrid import HybridGC as GCClass GC_PARAMS = {'space_size': 512*WORD, 'nursery_size': 32*WORD, - 'large_object': 8*WORD} + 'large_object': 8*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_ref_from_rawmalloced_to_regular(cls): @@ -1477,6 +1484,7 @@ 'large_object': 8*WORD, 'large_object_gcptrs': 10*WORD, 'card_page_indices': 4, + 'translated_to_c': False, } root_stack_depth = 200 @@ -1585,7 +1593,8 @@ gcname = "marksweep" class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestHybridTaggedPointerGC(TaggedPointerGCTests): @@ -1596,7 +1605,8 @@ from pypy.rpython.memory.gc.generation import GenerationGC as \ GCClass GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 32*WORD} + 'nursery_size': 32*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestMarkCompactTaggedpointerGC(TaggedPointerGCTests): @@ -1605,5 +1615,6 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.markcompact import MarkCompactGC as GCClass - GC_PARAMS = {'space_size': 4096*WORD} + GC_PARAMS = {'space_size': 4096*WORD, + 'translated_to_c': False} root_stack_depth = 200 Modified: pypy/branch/fast-forward/pypy/translator/driver.py ============================================================================== --- pypy/branch/fast-forward/pypy/translator/driver.py (original) +++ pypy/branch/fast-forward/pypy/translator/driver.py Fri Oct 1 22:53:02 2010 @@ -426,6 +426,22 @@ [OOTYPE], "JIT compiler generation") + def task_jittest_lltype(self): + """ Run with the JIT on top of the llgraph backend + """ + # parent process loop: spawn a child, wait for the child to finish, + # print a message, and restart + from pypy.translator.goal import unixcheckpoint + unixcheckpoint.restartable_point(auto='run') + # load the module pypy/jit/tl/jittest.py, which you can hack at + # and restart without needing to restart the whole translation process + from pypy.jit.tl import jittest + jittest.jittest(self) + # + task_jittest_lltype = taskdef(task_jittest_lltype, + [RTYPE], + "test of the JIT on the llgraph backend") + def task_backendopt_lltype(self): """ Run all backend optimizations - lltype version """ @@ -433,7 +449,8 @@ backend_optimizations(self.translator) # task_backendopt_lltype = taskdef(task_backendopt_lltype, - [RTYPE, '??pyjitpl_lltype'], + [RTYPE, '??pyjitpl_lltype', + '??jittest_lltype'], "lltype back-end optimisations") BACKENDOPT = 'backendopt_lltype' Modified: pypy/branch/fast-forward/pypy/translator/goal/translate.py ============================================================================== --- pypy/branch/fast-forward/pypy/translator/goal/translate.py (original) +++ pypy/branch/fast-forward/pypy/translator/goal/translate.py Fri Oct 1 22:53:02 2010 @@ -27,6 +27,7 @@ ("annotate", "do type inference", "-a --annotate", ""), ("rtype", "do rtyping", "-t --rtype", ""), ("pyjitpl", "JIT generation step", "--pyjitpl", ""), + ("jittest", "JIT test with llgraph backend", "--jittest", ""), ("backendopt", "do backend optimizations", "--backendopt", ""), ("source", "create source", "-s --source", ""), ("compile", "compile", "-c --compile", " (default goal)"), From hakanardo at codespeak.net Sat Oct 2 10:31:31 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Sat, 2 Oct 2010 10:31:31 +0200 (CEST) Subject: [pypy-svn] r77545 - in pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test Message-ID: <20101002083131.A8FFD282BDB@codespeak.net> Author: hakanardo Date: Sat Oct 2 10:31:29 2010 New Revision: 77545 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Log: Look ahead on future operations using optimizer.operations no longer suported. test_loop_invariant_mul_ovf passing. Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py Sat Oct 2 10:31:29 2010 @@ -9,14 +9,28 @@ """Keeps track of the bounds placed on integers by the guards and remove redundant guards""" + def setup(self, virtuals): + self.posponedop = None + self.nextop = None + def propagate_forward(self, op): + if op.is_ovf(): + self.posponedop = op + return + if self.posponedop: + self.nextop = op + op = self.posponedop + self.posponedop = None + opnum = op.getopnum() for value, func in optimize_ops: if opnum == value: func(self, op) break else: + assert not op.is_ovf() self.emit_operation(op) + def propagate_bounds_backward(self, box): # FIXME: This takes care of the instruction where box is the reuslt @@ -85,45 +99,45 @@ v2 = self.getvalue(op.getarg(1)) resbound = v1.intbound.add_bound(v2.intbound) if resbound.has_lower and resbound.has_upper and \ - self.nextop().getopnum() == rop.GUARD_NO_OVERFLOW: + self.nextop.getopnum() == rop.GUARD_NO_OVERFLOW: # Transform into INT_ADD and remove guard op = op.copy_and_change(rop.INT_ADD) - self.skip_nextop() self.optimize_INT_ADD(op) # emit the op else: self.emit_operation(op) r = self.getvalue(op.result) r.intbound.intersect(resbound) + self.emit_operation(self.nextop) def optimize_INT_SUB_OVF(self, op): v1 = self.getvalue(op.getarg(0)) v2 = self.getvalue(op.getarg(1)) resbound = v1.intbound.sub_bound(v2.intbound) if resbound.has_lower and resbound.has_upper and \ - self.nextop().getopnum() == rop.GUARD_NO_OVERFLOW: + self.nextop.getopnum() == rop.GUARD_NO_OVERFLOW: # Transform into INT_SUB and remove guard op = op.copy_and_change(rop.INT_SUB) - self.skip_nextop() self.optimize_INT_SUB(op) # emit the op else: self.emit_operation(op) r = self.getvalue(op.result) r.intbound.intersect(resbound) - + self.emit_operation(self.nextop) + def optimize_INT_MUL_OVF(self, op): v1 = self.getvalue(op.getarg(0)) v2 = self.getvalue(op.getarg(1)) resbound = v1.intbound.mul_bound(v2.intbound) if resbound.has_lower and resbound.has_upper and \ - self.nextop().getopnum() == rop.GUARD_NO_OVERFLOW: + self.nextop.getopnum() == rop.GUARD_NO_OVERFLOW: # Transform into INT_MUL and remove guard op = op.copy_and_change(rop.INT_MUL) - self.skip_nextop() self.optimize_INT_MUL(op) # emit the op else: self.emit_operation(op) r = self.getvalue(op.result) r.intbound.intersect(resbound) + self.emit_operation(self.nextop) def optimize_INT_LT(self, op): v1 = self.getvalue(op.getarg(0)) Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py Sat Oct 2 10:31:29 2010 @@ -182,12 +182,6 @@ op = ResOperation(opnum, args, result) self.optimizer.pure_operations[self.optimizer.make_args_key(op)] = op - def nextop(self): - return self.optimizer.loop.operations[self.optimizer.i + 1] - - def skip_nextop(self): - self.optimizer.i += 1 - def setup(self, virtuals): pass @@ -205,6 +199,7 @@ self.pure_operations = args_dict() self.producer = {} self.pendingfields = [] + self.posponedop = None if optimizations: self.first_optimization = optimizations[0] @@ -254,9 +249,9 @@ return constbox return None - def make_equal_to(self, box, value): + def make_equal_to(self, box, value, replace=False): assert isinstance(value, OptValue) - assert box not in self.values + assert replace or box not in self.values self.values[box] = value def make_constant(self, box, constbox): @@ -392,10 +387,17 @@ def optimize_default(self, op): canfold = op.is_always_pure() - is_ovf = op.is_ovf() - if is_ovf: - nextop = self.loop.operations[self.i + 1] + if op.is_ovf(): + self.posponedop = op + return + if self.posponedop: + nextop = op + op = self.posponedop + self.posponedop = None canfold = nextop.getopnum() == rop.GUARD_NO_OVERFLOW + else: + nextop = None + if canfold: for i in range(op.numargs()): if self.get_constant_box(op.getarg(i)) is None: @@ -406,9 +408,8 @@ for i in range(op.numargs())] resbox = execute_nonspec(self.cpu, None, op.getopnum(), argboxes, op.getdescr()) + # FIXME: Don't we need to check for an overflow here? self.make_constant(op.result, resbox.constbox()) - if is_ovf: - self.i += 1 # skip next operation, it is the unneeded guard return # did we do the exact same operation already? @@ -416,20 +417,22 @@ oldop = self.pure_operations.get(args, None) if oldop is not None and oldop.getdescr() is op.getdescr(): assert oldop.getopnum() == op.getopnum() - self.make_equal_to(op.result, self.getvalue(oldop.result)) - if is_ovf: - self.i += 1 # skip next operation, it is the unneeded guard + self.make_equal_to(op.result, self.getvalue(oldop.result), + True) return else: self.pure_operations[args] = op # otherwise, the operation remains self.emit_operation(op) + if nextop: + self.emit_operation(nextop) - def optimize_GUARD_NO_OVERFLOW(self, op): - # otherwise the default optimizer will clear fields, which is unwanted - # in this case - self.emit_operation(op) + #def optimize_GUARD_NO_OVERFLOW(self, op): + # # otherwise the default optimizer will clear fields, which is unwanted + # # in this case + # self.emit_operation(op) + # FIXME: Is this still needed? def optimize_DEBUG_MERGE_POINT(self, op): self.emit_operation(op) Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Sat Oct 2 10:31:29 2010 @@ -334,11 +334,11 @@ return res res = self.meta_interp(f, [6, 7]) assert res == 308 - self.check_loop_count(1) - self.check_loops({'guard_true': 1, 'guard_no_overflow': 1, - 'int_add': 2, 'int_sub': 1, 'int_gt': 1, - 'int_mul': 1, 'int_mul_ovf': 1, - 'jump': 1}) + self.check_loop_count(2) + self.check_loops({'guard_true': 2, 'guard_no_overflow': 1, + 'int_add': 4, 'int_sub': 2, 'int_gt': 2, + 'int_mul': 2, 'int_mul_ovf': 1, + 'jump': 2}) def test_loop_invariant_mul_bridge1(self): myjitdriver = JitDriver(greens = [], reds = ['y', 'res', 'x']) @@ -364,12 +364,12 @@ myjitdriver.can_enter_jit(x=x, y=y, res=res) myjitdriver.jit_merge_point(x=x, y=y, res=res) res += x * x - if y<8: + if y<16: res += 1 y -= 1 return res - res = self.meta_interp(f, [6, 16]) - assert res == 583 + res = self.meta_interp(f, [6, 32]) + assert res == 1167 self.check_loop_count(3) self.check_loops({'int_lt': 1, 'int_gt': 1, 'guard_false': 1, 'guard_true': 1, From afa at codespeak.net Sat Oct 2 12:15:33 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sat, 2 Oct 2010 12:15:33 +0200 (CEST) Subject: [pypy-svn] r77546 - pypy/branch/fast-forward/lib-python Message-ID: <20101002101533.8B0CB282BAD@codespeak.net> Author: afa Date: Sat Oct 2 12:15:32 2010 New Revision: 77546 Modified: pypy/branch/fast-forward/lib-python/TODO Log: More TODO tasks Modified: pypy/branch/fast-forward/lib-python/TODO ============================================================================== --- pypy/branch/fast-forward/lib-python/TODO (original) +++ pypy/branch/fast-forward/lib-python/TODO Sat Oct 2 12:15:32 2010 @@ -28,6 +28,8 @@ - (list|str|unicode|bytearray).(index|find) should accept None as indices +- Fix fcntl.fcntl(fd, fcntl.F_NOTIFY, fcntl.DN_MULTISHOT) on 32bit platform. + Medium tasks ------------ @@ -37,6 +39,8 @@ - "exceptions must be old-style classes or derived from BaseException, not str" in the 'raise' statement and generator.throw() +- missing builtin: memoryview + Longer tasks ------------ From getxsick at codespeak.net Sun Oct 3 16:45:23 2010 From: getxsick at codespeak.net (getxsick at codespeak.net) Date: Sun, 3 Oct 2010 16:45:23 +0200 (CEST) Subject: [pypy-svn] r77550 - pypy/extradoc/sprintinfo/ddorf2010 Message-ID: <20101003144523.12FEA282BAD@codespeak.net> Author: getxsick Date: Sun Oct 3 16:45:21 2010 New Revision: 77550 Modified: pypy/extradoc/sprintinfo/ddorf2010/announce.txt Log: fix url Modified: pypy/extradoc/sprintinfo/ddorf2010/announce.txt ============================================================================== --- pypy/extradoc/sprintinfo/ddorf2010/announce.txt (original) +++ pypy/extradoc/sprintinfo/ddorf2010/announce.txt Sun Oct 3 16:45:21 2010 @@ -30,4 +30,4 @@ you have codespeak commit rights). .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint -.. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/ddorf2010/people.txt +.. _`people`: http://codespeak.net/svn/pypy/extradoc/sprintinfo/ddorf2010/people.txt From afa at codespeak.net Sun Oct 3 22:28:01 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 3 Oct 2010 22:28:01 +0200 (CEST) Subject: [pypy-svn] r77551 - pypy/branch/fast-forward/pypy/interpreter/test Message-ID: <20101003202801.3D7F2282BAD@codespeak.net> Author: afa Date: Sun Oct 3 22:27:59 2010 New Revision: 77551 Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_argument.py Log: Add methods to the DummySpace to let the tests pass Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_argument.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/test/test_argument.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/test/test_argument.py Sun Oct 3 22:27:59 2010 @@ -98,12 +98,27 @@ def isinstance(self, obj, cls): return isinstance(obj, cls) + isinstance_w = isinstance def exception_match(self, w_type1, w_type2): return issubclass(w_type1, w_type2) + def call_method(self, obj, name, *args): + try: + method = getattr(obj, name) + except AttributeError: + raise OperationError(AttributeError, name) + return method(*args) + + def type(self, obj): + class Type: + def getname(self, space, default): + return type(obj).__name__ + return Type() + w_TypeError = TypeError + w_AttributeError = AttributeError w_dict = dict class TestArgumentsNormal(object): From afa at codespeak.net Sun Oct 3 22:42:56 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 3 Oct 2010 22:42:56 +0200 (CEST) Subject: [pypy-svn] r77552 - pypy/branch/fast-forward/pypy/module/_rawffi Message-ID: <20101003204256.754A3282BAD@codespeak.net> Author: afa Date: Sun Oct 3 22:42:55 2010 New Revision: 77552 Modified: pypy/branch/fast-forward/pypy/module/_rawffi/interp_rawffi.py Log: Typecode '?' is now used for the Bool type. Change the default to '\0' Modified: pypy/branch/fast-forward/pypy/module/_rawffi/interp_rawffi.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_rawffi/interp_rawffi.py (original) +++ pypy/branch/fast-forward/pypy/module/_rawffi/interp_rawffi.py Sun Oct 3 22:42:55 2010 @@ -243,7 +243,7 @@ _array_shapes = None size = 0 alignment = 0 - itemcode = '?' + itemcode = '\0' def allocate(self, space, length, autofree=False): raise NotImplementedError From afa at codespeak.net Sun Oct 3 23:04:23 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 3 Oct 2010 23:04:23 +0200 (CEST) Subject: [pypy-svn] r77553 - pypy/branch/fast-forward/pypy/interpreter/astcompiler/test Message-ID: <20101003210423.2A783282BAD@codespeak.net> Author: afa Date: Sun Oct 3 23:04:21 2010 New Revision: 77553 Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_compiler.py Log: Reverse the test, now that pypy uses 2.7 bytecodes Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_compiler.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_compiler.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_compiler.py Sun Oct 3 23:04:21 2010 @@ -27,8 +27,8 @@ space = self.space code = compile_with_astcompiler(source, 'exec', space) # 2.7 bytecode is too different, the standard `dis` module crashes - # when trying to display pypy (2.5-like) bytecode. - if sys.version_info < (2, 7): + # on older cpython versions + if sys.version_info >= (2, 7): print code.dump() w_dict = space.newdict() From afa at codespeak.net Mon Oct 4 08:24:07 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 08:24:07 +0200 (CEST) Subject: [pypy-svn] r77554 - pypy/branch/fast-forward/pypy/module/_rawffi/test Message-ID: <20101004062407.6D4F1282BEA@codespeak.net> Author: afa Date: Mon Oct 4 08:24:04 2010 New Revision: 77554 Modified: pypy/branch/fast-forward/pypy/module/_rawffi/test/test__rawffi.py Log: Fix test of repr() Modified: pypy/branch/fast-forward/pypy/module/_rawffi/test/test__rawffi.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_rawffi/test/test__rawffi.py (original) +++ pypy/branch/fast-forward/pypy/module/_rawffi/test/test__rawffi.py Mon Oct 4 08:24:04 2010 @@ -710,7 +710,7 @@ # fragile S = _rawffi.Structure([('x', 'c'), ('y', 'l')]) assert (repr(_rawffi.Array((S, 2))) == - "<_rawffi.Array '?' (%d, %d)>" % (4*lsize, lsize)) + "<_rawffi.Array '\0' (%d, %d)>" % (4*lsize, lsize)) assert (repr(_rawffi.Structure([('x', 'i'), ('yz', 'i')])) == "<_rawffi.Structure 'x' 'yz' (%d, %d)>" % (2*isize, isize)) From antocuni at codespeak.net Mon Oct 4 10:10:37 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 10:10:37 +0200 (CEST) Subject: [pypy-svn] r77555 - in pypy/branch/jitffi/pypy: jit/metainterp/test rlib rlib/test Message-ID: <20101004081037.23D95282BAD@codespeak.net> Author: antocuni Date: Mon Oct 4 10:10:35 2010 New Revision: 77555 Added: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py (contents, props changed) Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: add a more user-friendly interface to build argchains. Moreover, checkin test_libffi, which I forgot to svn add earlier :-( Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py Mon Oct 4 10:10:35 2010 @@ -2,7 +2,7 @@ import py from pypy.rlib.jit import JitDriver, hint from pypy.jit.metainterp.test.test_basic import LLJitMixin -from pypy.rlib.libffi import CDLL, ffi_type_sint, IntArg, Func +from pypy.rlib.libffi import CDLL, ffi_type_sint, ArgChain, Func from pypy.tool.udir import udir from pypy.translator.tool.cbuild import ExternalCompilationInfo from pypy.translator.platform import platform @@ -34,10 +34,9 @@ driver.jit_merge_point(n=n, func=func) driver.can_enter_jit(n=n, func=func) func = hint(func, promote=True) - arg0 = IntArg(n) - arg1 = IntArg(1) - arg0.next = arg1 - n = func.call(arg0, lltype.Signed) + argchain = ArgChain() + argchain.int(n).int(1) + n = func.call(argchain, lltype.Signed) return n res = self.meta_interp(f, [0]) Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Mon Oct 4 10:10:35 2010 @@ -18,6 +18,28 @@ # ---------------------------------------------------------------------- +class ArgChain(object): + first = None + last = None + numargs = 0 + + def int(self, intval): + self._append(IntArg(intval)) + return self + + def float(self, floatval): + self._append(FloatArg(floatval)) + return self + + def _append(self, arg): + if self.first is None: + self.first = self.last = arg + else: + self.last.next = arg + self.last = arg + self.numargs += 1 + + class AbstractArg(object): next = None @@ -116,7 +138,7 @@ # assuming that archain is completely virtual. ll_args = self._prepare() i = 0 - arg = argchain + arg = argchain.first while arg: arg.push(self, ll_args, i) i += 1 Added: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Mon Oct 4 10:10:35 2010 @@ -0,0 +1,57 @@ +import py +import sys +from pypy.rpython.lltypesystem import rffi, lltype +from pypy.rpython.lltypesystem.ll2ctypes import ALLOCATED +from pypy.rlib.test.test_clibffi import BaseFfiTest, get_libm_name +from pypy.rlib.libffi import CDLL, Func, get_libc_name, ArgChain +from pypy.rlib.libffi import ffi_type_double, ffi_type_void + + +class TestLibffi(BaseFfiTest): + """ + Test the new JIT-friendly interface to libffi + """ + + def get_libc(self): + return CDLL(get_libc_name()) + + def get_libm(self): + return CDLL(get_libm_name(sys.platform)) + + def test_argchain(self): + chain = ArgChain() + assert chain.numargs == 0 + chain2 = chain.int(42) + assert chain2 is chain + assert chain.numargs == 1 + intarg = chain.first + assert chain.last is intarg + assert intarg.intval == 42 + chain.float(123.45) + assert chain.numargs == 2 + assert chain.first is intarg + assert intarg.next is chain.last + floatarg = intarg.next + assert floatarg.floatval == 123.45 + + def test_library_open(self): + lib = self.get_libc() + del lib + assert not ALLOCATED + + def test_library_get_func(self): + lib = self.get_libc() + ptr = lib.getpointer('fopen', [], ffi_type_void) + py.test.raises(KeyError, lib.getpointer, 'xxxxxxxxxxxxxxx', [], ffi_type_void) + del ptr + del lib + assert not ALLOCATED + + def test_call_argchain(self): + libm = self.get_libm() + pow = libm.getpointer('pow', [ffi_type_double, ffi_type_double], + ffi_type_double) + argchain = ArgChain() + argchain.float(2.0).float(3.0) + res = pow.call(argchain, rffi.DOUBLE) + assert res == 8.0 From antocuni at codespeak.net Mon Oct 4 10:14:56 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 10:14:56 +0200 (CEST) Subject: [pypy-svn] r77556 - pypy/branch/jitffi/pypy/rlib Message-ID: <20101004081456.84117282BAD@codespeak.net> Author: antocuni Date: Mon Oct 4 10:14:55 2010 New Revision: 77556 Modified: pypy/branch/jitffi/pypy/rlib/libffi.py Log: move the public interface (well, just one method) to the top of the class Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Mon Oct 4 10:14:55 2010 @@ -74,6 +74,37 @@ self.keepalive = keepalive self.funcsym = funcsym + @jit.unroll_safe + @specialize.arg(2) + def call(self, argchain, RESULT): + # WARNING! This code is written carefully in a way that the JIT + # optimizer will see a sequence of calls like the following: + # + # libffi_prepare_call + # libffi_push_arg + # libffi_push_arg + # ... + # libffi_call + # + # It is important that there is no other operation in the middle, else + # the optimizer will fail to recognize the pattern and won't turn it + # into a fast CALL. Note that "arg = arg.next" is optimized away, + # assuming that archain is completely virtual. + ll_args = self._prepare() + i = 0 + arg = argchain.first + while arg: + arg.push(self, ll_args, i) + i += 1 + arg = arg.next + result = self._do_call(self.funcsym, ll_args, RESULT) + return result + + # END OF THE PUBLIC INTERFACE + # ------------------------------------------------------------------------- + # the following methods are supposed to be seen opaquely by the JIT + # optimizer. Don't call them + def _prepare(self): ll_args = lltype.malloc(rffi.VOIDPP.TO, len(self.argtypes), flavor='raw') return ll_args @@ -120,31 +151,6 @@ lltype.free(ll_args[i], flavor='raw') lltype.free(ll_args, flavor='raw') - @jit.unroll_safe - @specialize.arg(2) - def call(self, argchain, RESULT): - # WARNING! This code is written carefully in a way that the JIT - # optimizer will see a sequence of calls like the following: - # - # libffi_prepare_call - # libffi_push_arg - # libffi_push_arg - # ... - # libffi_call - # - # It is important that there is no other operation in the middle, else - # the optimizer will fail to recognize the pattern and won't turn it - # into a fast CALL. Note that "arg = arg.next" is optimized away, - # assuming that archain is completely virtual. - ll_args = self._prepare() - i = 0 - arg = argchain.first - while arg: - arg.push(self, ll_args, i) - i += 1 - arg = arg.next - result = self._do_call(self.funcsym, ll_args, RESULT) - return result # ---------------------------------------------------------------------- From afa at codespeak.net Mon Oct 4 10:21:55 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 10:21:55 +0200 (CEST) Subject: [pypy-svn] r77557 - pypy/branch/fast-forward/lib-python/modified-2.7.0 Message-ID: <20101004082155.66493282BAD@codespeak.net> Author: afa Date: Mon Oct 4 10:21:54 2010 New Revision: 77557 Added: pypy/branch/fast-forward/lib-python/modified-2.7.0/pickle.py - copied, changed from r77554, pypy/branch/fast-forward/lib-python/2.7.0/pickle.py Log: Allow pickling of functions, this fixes greenlet tests. Same change as in modified-2.5.2 Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/pickle.py (from r77554, pypy/branch/fast-forward/lib-python/2.7.0/pickle.py) ============================================================================== --- pypy/branch/fast-forward/lib-python/2.7.0/pickle.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/pickle.py Mon Oct 4 10:21:54 2010 @@ -727,6 +727,29 @@ dispatch[InstanceType] = save_inst + def save_function(self, obj): + try: + return self.save_global(obj) + except PicklingError, e: + pass + # Check copy_reg.dispatch_table + reduce = dispatch_table.get(type(obj)) + if reduce: + rv = reduce(obj) + else: + # Check for a __reduce_ex__ method, fall back to __reduce__ + reduce = getattr(obj, "__reduce_ex__", None) + if reduce: + rv = reduce(self.proto) + else: + reduce = getattr(obj, "__reduce__", None) + if reduce: + rv = reduce() + else: + raise e + return self.save_reduce(obj=obj, *rv) + dispatch[FunctionType] = save_function + def save_global(self, obj, name=None, pack=struct.pack): write = self.write memo = self.memo @@ -768,7 +791,6 @@ self.memoize(obj) dispatch[ClassType] = save_global - dispatch[FunctionType] = save_global dispatch[BuiltinFunctionType] = save_global dispatch[TypeType] = save_global From afa at codespeak.net Mon Oct 4 10:40:39 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 10:40:39 +0200 (CEST) Subject: [pypy-svn] r77558 - pypy/trunk/pypy/module/parser/test Message-ID: <20101004084039.82D3B282BAD@codespeak.net> Author: afa Date: Mon Oct 4 10:40:38 2010 New Revision: 77558 Modified: pypy/trunk/pypy/module/parser/test/test_parser.py Log: Avoid hard-coded constant in test Modified: pypy/trunk/pypy/module/parser/test/test_parser.py ============================================================================== --- pypy/trunk/pypy/module/parser/test/test_parser.py (original) +++ pypy/trunk/pypy/module/parser/test/test_parser.py Mon Oct 4 10:40:38 2010 @@ -10,6 +10,9 @@ cls.w_m = space.appexec([], """(): import parser return parser""") + cls.w_symbol = space.appexec([], """(): + import symbol + return symbol""") class AppTestParser(ParserModuleTest): @@ -36,7 +39,7 @@ seq = getattr(s, meth)() assert isinstance(seq, tp) assert len(seq) == 4 - assert seq[0] == 286 + assert seq[0] == self.symbol.file_input assert len(seq[2]) == 2 assert len(seq[3]) == 2 assert seq[2][0] == 4 From afa at codespeak.net Mon Oct 4 10:42:02 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 10:42:02 +0200 (CEST) Subject: [pypy-svn] r77559 - pypy/branch/fast-forward/lib_pypy Message-ID: <20101004084202.DACBC282BAD@codespeak.net> Author: afa Date: Mon Oct 4 10:42:01 2010 New Revision: 77559 Modified: pypy/branch/fast-forward/lib_pypy/_subprocess.py Log: Add constant needed by the win32 implementation of subprocess. Modified: pypy/branch/fast-forward/lib_pypy/_subprocess.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_subprocess.py (original) +++ pypy/branch/fast-forward/lib_pypy/_subprocess.py Mon Oct 4 10:42:01 2010 @@ -162,7 +162,8 @@ STARTF_USESHOWWINDOW = 0x001 STARTF_USESTDHANDLES = 0x100 SW_HIDE = 0 -CREATE_NEW_CONSOLE = 0x010 +CREATE_NEW_CONSOLE = 0x010 +CREATE_NEW_PROCESS_GROUP = 0x200 def WaitForSingleObject(handle, milliseconds): res = _WaitForSingleObject(handle.handle, milliseconds) From antocuni at codespeak.net Mon Oct 4 11:24:15 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 11:24:15 +0200 (CEST) Subject: [pypy-svn] r77560 - in pypy/branch/jitffi/pypy: jit/codewriter jit/metainterp/optimizeopt rlib Message-ID: <20101004092415.5A163282BAD@codespeak.net> Author: antocuni Date: Mon Oct 4 11:24:13 2010 New Revision: 77560 Modified: pypy/branch/jitffi/pypy/jit/codewriter/support.py pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: use separate methods for pushing ints and floats. This is needed because oopspec and specialize don't really play well togheter Modified: pypy/branch/jitffi/pypy/jit/codewriter/support.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/support.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/support.py Mon Oct 4 11:24:13 2010 @@ -222,20 +222,21 @@ # libffi support # -------------- -def _ll_1_libffi_prepare_call(llfunc): +def func(llfunc): from pypy.rlib.libffi import Func - func = cast_base_ptr_to_instance(Func, llfunc) - return func._prepare() + return cast_base_ptr_to_instance(Func, llfunc) -def _ll_4_libffi_push_arg(llfunc, value, ll_args, i): - from pypy.rlib.libffi import Func - func = cast_base_ptr_to_instance(Func, llfunc) - return func._push_arg(value, ll_args, i) +def _ll_1_libffi_prepare_call(llfunc): + return func(llfunc)._prepare() + +def _ll_4_libffi_push_int(llfunc, value, ll_args, i): + return func(llfunc)._push_int(value, ll_args, i) + +def _ll_4_libffi_push_float(llfunc, value, ll_args, i): + return func(llfunc)._push_float(value, ll_args, i) def _ll_4_libffi_call(llfunc, funcsym, ll_args, RESULT): - from pypy.rlib.libffi import Func - func = cast_base_ptr_to_instance(Func, llfunc) - return func._do_call(funcsym, ll_args, lltype.Signed) + return func(llfunc)._do_call(funcsym, ll_args, lltype.Signed) # XXX: should be RESULT, but it doesn't work Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py Mon Oct 4 11:24:13 2010 @@ -63,7 +63,7 @@ funcname = str(funcval.box) if '_libffi_prepare_call' in funcname: return 'prepare_call' - elif '_libffi_push_arg' in funcname: + elif '_libffi_push_' in funcname: return 'push_arg' elif '_libffi_call' in funcname: return 'call' Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Mon Oct 4 11:24:13 2010 @@ -51,7 +51,7 @@ self.intval = intval def push(self, func, ll_args, i): - func._push_arg(self.intval, ll_args, i) + func._push_int(self.intval, ll_args, i) class FloatArg(AbstractArg): """ An argument holding a float @@ -61,7 +61,7 @@ self.floatval = floatval def push(self, func, ll_args, i): - func._push_arg(self.floatval, ll_args, i) + func._push_float(self.floatval, ll_args, i) class Func(AbstractFuncPtr): @@ -110,18 +110,23 @@ return ll_args _prepare.oopspec = 'libffi_prepare_call(self)' - def _push_arg(self, value, ll_args, i): + @specialize.arg(1) + def _push_arg(self, TYPE, value, ll_args, i): # XXX: check the type is not translated? argtype = self.argtypes[i] c_size = intmask(argtype.c_size) ll_buf = lltype.malloc(rffi.CCHARP.TO, c_size, flavor='raw') push_arg_as_ffiptr(argtype, value, ll_buf) ll_args[i] = ll_buf - # XXX this is bad, fix it somehow in the future, but specialize:argtype - # doesn't work correctly with mixing non-negative and normal integers - _push_arg._annenforceargs_ = [None, int, None, int] - #_push_arg._annspecialcase_ = 'specialize:argtype(1)' - _push_arg.oopspec = 'libffi_push_arg(self, value, ll_args, i)' + + def _push_int(self, value, ll_args, i): + self._push_arg(lltype.Signed, value, ll_args, i) + _push_int.oopspec = 'libffi_push_int(self, value, ll_args, i)' + _push_int._annenforceargs_ = [None, int, None, int] + + def _push_float(self, value, ll_args, i): + self._push_arg(lltype.Float, value, ll_args, i) + _push_float.oopspec = 'libffi_push_float(self, value, ll_args, i)' def _do_call(self, funcsym, ll_args, RESULT): # XXX: check len(args)? From antocuni at codespeak.net Mon Oct 4 11:29:01 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 11:29:01 +0200 (CEST) Subject: [pypy-svn] r77561 - in pypy/branch/jitffi/pypy/rlib: . test Message-ID: <20101004092901.D1C07282BAD@codespeak.net> Author: antocuni Date: Mon Oct 4 11:29:00 2010 New Revision: 77561 Modified: pypy/branch/jitffi/pypy/rlib/jit.py pypy/branch/jitffi/pypy/rlib/libffi.py pypy/branch/jitffi/pypy/rlib/test/test_jit.py Log: introduce a new decorator to set oopspec, and use it in libffi.py Modified: pypy/branch/jitffi/pypy/rlib/jit.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/jit.py (original) +++ pypy/branch/jitffi/pypy/rlib/jit.py Mon Oct 4 11:29:00 2010 @@ -77,6 +77,12 @@ return result return decorator +def oopspec(spec): + def decorator(func): + func.oopspec = spec + return func + return decorator + class Entry(ExtRegistryEntry): _about_ = hint Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Mon Oct 4 11:29:00 2010 @@ -105,10 +105,10 @@ # the following methods are supposed to be seen opaquely by the JIT # optimizer. Don't call them + @jit.oopspec('libffi_prepare_call(self)') def _prepare(self): ll_args = lltype.malloc(rffi.VOIDPP.TO, len(self.argtypes), flavor='raw') return ll_args - _prepare.oopspec = 'libffi_prepare_call(self)' @specialize.arg(1) def _push_arg(self, TYPE, value, ll_args, i): @@ -119,15 +119,17 @@ push_arg_as_ffiptr(argtype, value, ll_buf) ll_args[i] = ll_buf + @jit.oopspec('libffi_push_int(self, value, ll_args, i)') def _push_int(self, value, ll_args, i): self._push_arg(lltype.Signed, value, ll_args, i) - _push_int.oopspec = 'libffi_push_int(self, value, ll_args, i)' _push_int._annenforceargs_ = [None, int, None, int] + @jit.oopspec('libffi_push_float(self, value, ll_args, i)') def _push_float(self, value, ll_args, i): self._push_arg(lltype.Float, value, ll_args, i) - _push_float.oopspec = 'libffi_push_float(self, value, ll_args, i)' + @specialize.arg(3) + @jit.oopspec('libffi_call(self, funcsym, ll_args, RESULT)') def _do_call(self, funcsym, ll_args, RESULT): # XXX: check len(args)? ll_result = lltype.nullptr(rffi.CCHARP.TO) @@ -147,8 +149,6 @@ self._free_buffers(ll_result, ll_args) #check_fficall_result(ffires, self.flags) return res - _do_call._annspecialcase_ = 'specialize:arg(3)' - _do_call.oopspec = 'libffi_call(self, funcsym, ll_args, RESULT)' def _free_buffers(self, ll_result, ll_args): lltype.free(ll_result, flavor='raw') Modified: pypy/branch/jitffi/pypy/rlib/test/test_jit.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_jit.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_jit.py Mon Oct 4 11:29:00 2010 @@ -1,10 +1,16 @@ import py from pypy.rlib.jit import hint, we_are_jitted, JitDriver, purefunction_promote -from pypy.rlib.jit import JitHintError +from pypy.rlib.jit import JitHintError, oopspec from pypy.translator.translator import TranslationContext, graphof from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin from pypy.rpython.lltypesystem import lltype +def test_oopspec(): + @oopspec('foobar') + def fn(): + pass + assert fn.oopspec == 'foobar' + class BaseTestJIT(BaseRtypingTest): def test_hint(self): def f(): From antocuni at codespeak.net Mon Oct 4 11:38:57 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 11:38:57 +0200 (CEST) Subject: [pypy-svn] r77562 - in pypy/branch/jitffi/pypy: jit/metainterp/test rlib Message-ID: <20101004093857.C9F20282BAD@codespeak.net> Author: antocuni Date: Mon Oct 4 11:38:56 2010 New Revision: 77562 Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: actually test float args Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py Mon Oct 4 11:38:56 2010 @@ -2,7 +2,7 @@ import py from pypy.rlib.jit import JitDriver, hint from pypy.jit.metainterp.test.test_basic import LLJitMixin -from pypy.rlib.libffi import CDLL, ffi_type_sint, ArgChain, Func +from pypy.rlib.libffi import CDLL, ffi_type_sint, ffi_type_double, ArgChain, Func from pypy.tool.udir import udir from pypy.translator.tool.cbuild import ExternalCompilationInfo from pypy.translator.platform import platform @@ -14,9 +14,9 @@ # it via rlib.libffi c_file = udir.ensure("test_jit_direct_call", dir=1).join("xlib.c") c_file.write(py.code.Source(''' - int sum_xy(int x, int y) + int sum_xy(int x, double y) { - return (x + y); + return (x + (int)y); } ''')) eci = ExternalCompilationInfo(export_symbols=['sum_xy']) @@ -28,14 +28,14 @@ def f(n): cdll = CDLL(self.lib_name) - func = cdll.getpointer('sum_xy', [ffi_type_sint, ffi_type_sint], + func = cdll.getpointer('sum_xy', [ffi_type_sint, ffi_type_double], ffi_type_sint) while n < 10: driver.jit_merge_point(n=n, func=func) driver.can_enter_jit(n=n, func=func) func = hint(func, promote=True) argchain = ArgChain() - argchain.int(n).int(1) + argchain.int(n).float(1.2) n = func.call(argchain, lltype.Signed) return n Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Mon Oct 4 11:38:56 2010 @@ -1,5 +1,5 @@ from pypy.rpython.lltypesystem import rffi, lltype -from pypy.rlib.objectmodel import specialize +from pypy.rlib.objectmodel import specialize, enforceargs from pypy.rlib.rarithmetic import intmask, r_uint from pypy.rlib import jit from pypy.rlib import clibffi @@ -120,11 +120,12 @@ ll_args[i] = ll_buf @jit.oopspec('libffi_push_int(self, value, ll_args, i)') + @enforceargs( None, int, None, int) # fix the annotation for tests def _push_int(self, value, ll_args, i): self._push_arg(lltype.Signed, value, ll_args, i) - _push_int._annenforceargs_ = [None, int, None, int] @jit.oopspec('libffi_push_float(self, value, ll_args, i)') + @enforceargs( None, float, None, int) # fix the annotation for tests def _push_float(self, value, ll_args, i): self._push_arg(lltype.Float, value, ll_args, i) From afa at codespeak.net Mon Oct 4 11:40:31 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 11:40:31 +0200 (CEST) Subject: [pypy-svn] r77564 - pypy/branch/fast-forward/pypy/module/parser/test Message-ID: <20101004094031.38AF0282BAD@codespeak.net> Author: afa Date: Mon Oct 4 11:40:29 2010 New Revision: 77564 Modified: pypy/branch/fast-forward/pypy/module/parser/test/test_parser.py Log: Merge r77558 from trunk Modified: pypy/branch/fast-forward/pypy/module/parser/test/test_parser.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/parser/test/test_parser.py (original) +++ pypy/branch/fast-forward/pypy/module/parser/test/test_parser.py Mon Oct 4 11:40:29 2010 @@ -10,6 +10,9 @@ cls.w_m = space.appexec([], """(): import parser return parser""") + cls.w_symbol = space.appexec([], """(): + import symbol + return symbol""") class AppTestParser(ParserModuleTest): @@ -36,7 +39,7 @@ seq = getattr(s, meth)() assert isinstance(seq, tp) assert len(seq) == 4 - assert seq[0] == 286 + assert seq[0] == self.symbol.file_input assert len(seq[2]) == 2 assert len(seq[3]) == 2 assert seq[2][0] == 4 From antocuni at codespeak.net Mon Oct 4 13:48:30 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 13:48:30 +0200 (CEST) Subject: [pypy-svn] r77566 - in pypy/branch/jitffi/pypy: jit/codewriter rlib Message-ID: <20101004114830.A1DD0282BAD@codespeak.net> Author: antocuni Date: Mon Oct 4 13:48:29 2010 New Revision: 77566 Modified: pypy/branch/jitffi/pypy/jit/codewriter/support.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: manually specialize _do_call into _do_call_{int,float}. This is needed because oopspec does not play well with specialize() Modified: pypy/branch/jitffi/pypy/jit/codewriter/support.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/support.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/support.py Mon Oct 4 13:48:29 2010 @@ -235,9 +235,11 @@ def _ll_4_libffi_push_float(llfunc, value, ll_args, i): return func(llfunc)._push_float(value, ll_args, i) -def _ll_4_libffi_call(llfunc, funcsym, ll_args, RESULT): +def _ll_3_libffi_call_int(llfunc, funcsym, ll_args): return func(llfunc)._do_call(funcsym, ll_args, lltype.Signed) -# XXX: should be RESULT, but it doesn't work + +def _ll_3_libffi_call_float(llfunc, funcsym, ll_args): + return func(llfunc)._do_call(funcsym, ll_args, lltype.Float) Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Mon Oct 4 13:48:29 2010 @@ -74,6 +74,10 @@ self.keepalive = keepalive self.funcsym = funcsym + # ======================================================================== + # PUBLIC INTERFACE + # ======================================================================== + @jit.unroll_safe @specialize.arg(2) def call(self, argchain, RESULT): @@ -97,40 +101,59 @@ arg.push(self, ll_args, i) i += 1 arg = arg.next - result = self._do_call(self.funcsym, ll_args, RESULT) - return result + if RESULT is lltype.Signed: + return self._do_call_int(self.funcsym, ll_args) + elif RESULT is lltype.Float: + return self._do_call_float(self.funcsym, ll_args) + else: + raise TypeError, 'Unsupported result type: %s' % RESULT # END OF THE PUBLIC INTERFACE - # ------------------------------------------------------------------------- - # the following methods are supposed to be seen opaquely by the JIT - # optimizer. Don't call them + # ------------------------------------------------------------------------ + + # JIT friendly interface + # the following methods are supposed to be seen opaquely by the optimizer @jit.oopspec('libffi_prepare_call(self)') def _prepare(self): ll_args = lltype.malloc(rffi.VOIDPP.TO, len(self.argtypes), flavor='raw') return ll_args - @specialize.arg(1) - def _push_arg(self, TYPE, value, ll_args, i): - # XXX: check the type is not translated? - argtype = self.argtypes[i] - c_size = intmask(argtype.c_size) - ll_buf = lltype.malloc(rffi.CCHARP.TO, c_size, flavor='raw') - push_arg_as_ffiptr(argtype, value, ll_buf) - ll_args[i] = ll_buf + # _push_* and _do_call_* in theory could be automatically specialize()d by + # the annotator. However, specialization doesn't work well with oopspec, + # so we specialize them by hand @jit.oopspec('libffi_push_int(self, value, ll_args, i)') @enforceargs( None, int, None, int) # fix the annotation for tests def _push_int(self, value, ll_args, i): - self._push_arg(lltype.Signed, value, ll_args, i) + self._push_arg(value, ll_args, i) @jit.oopspec('libffi_push_float(self, value, ll_args, i)') @enforceargs( None, float, None, int) # fix the annotation for tests def _push_float(self, value, ll_args, i): - self._push_arg(lltype.Float, value, ll_args, i) + self._push_arg(value, ll_args, i) + + @jit.oopspec('libffi_call_int(self, funcsym, ll_args)') + def _do_call_int(self, funcsym, ll_args): + return self._do_call(funcsym, ll_args, lltype.Signed) + + @jit.oopspec('libffi_call_float(self, funcsym, ll_args)') + def _do_call_float(self, funcsym, ll_args): + return self._do_call(funcsym, ll_args, lltype.Float) + + # ------------------------------------------------------------------------ + # private methods + + @specialize.argtype(1) + def _push_arg(self, value, ll_args, i): + # XXX: check the type is not translated? + argtype = self.argtypes[i] + c_size = intmask(argtype.c_size) + ll_buf = lltype.malloc(rffi.CCHARP.TO, c_size, flavor='raw') + push_arg_as_ffiptr(argtype, value, ll_buf) + ll_args[i] = ll_buf @specialize.arg(3) - @jit.oopspec('libffi_call(self, funcsym, ll_args, RESULT)') def _do_call(self, funcsym, ll_args, RESULT): # XXX: check len(args)? ll_result = lltype.nullptr(rffi.CCHARP.TO) From antocuni at codespeak.net Mon Oct 4 13:56:49 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 13:56:49 +0200 (CEST) Subject: [pypy-svn] r77567 - pypy/branch/jitffi/pypy/jit/metainterp/test Message-ID: <20101004115649.00BCD282BAD@codespeak.net> Author: antocuni Date: Mon Oct 4 13:56:47 2010 New Revision: 77567 Added: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py - copied, changed from r77562, pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py Removed: pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py Log: rename test_direct_call into test_fficall, and add a test for returning floats Copied: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py (from r77562, pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py) ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_direct_call.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py Mon Oct 4 13:56:47 2010 @@ -8,7 +8,7 @@ from pypy.translator.platform import platform from pypy.rpython.lltypesystem import lltype, rffi -class TestDirectCall(LLJitMixin): +class TestFfiCall(LLJitMixin): def setup_class(cls): # prepare C code as an example, so we can load it and call # it via rlib.libffi @@ -16,14 +16,21 @@ c_file.write(py.code.Source(''' int sum_xy(int x, double y) { - return (x + (int)y); + return (x + (int)y); + } + + float abs(double x) + { + if (x<0) + return -x; + return x; } ''')) eci = ExternalCompilationInfo(export_symbols=['sum_xy']) cls.lib_name = str(platform.compile([c_file], eci, 'x', standalone=False)) - def test_one(self): + def test_simple(self): driver = JitDriver(reds = ['n', 'func'], greens = []) def f(n): @@ -48,3 +55,25 @@ 'guard_true': 1, 'jump': 1}) + + def test_float_result(self): + driver = JitDriver(reds = ['n', 'func', 'res'], greens = []) + + def f(n): + cdll = CDLL(self.lib_name) + func = cdll.getpointer('abs', [ffi_type_double], ffi_type_double) + res = 0.0 + while n < 10: + driver.jit_merge_point(n=n, func=func, res=res) + driver.can_enter_jit(n=n, func=func, res=res) + func = hint(func, promote=True) + argchain = ArgChain() + argchain.float(float(-n)) + res = func.call(argchain, lltype.Float) + n += 1 + return res + + res = self.meta_interp(f, [0]) + assert res == 9 + self.check_loops(call=1) + From antocuni at codespeak.net Mon Oct 4 14:20:51 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 14:20:51 +0200 (CEST) Subject: [pypy-svn] r77568 - in pypy/branch/jitffi/pypy: jit/codewriter rlib Message-ID: <20101004122051.F2650282BAD@codespeak.net> Author: antocuni Date: Mon Oct 4 14:20:50 2010 New Revision: 77568 Modified: pypy/branch/jitffi/pypy/jit/codewriter/support.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: use rffi.{LONG,DOUBLE} instead of lltype.{Signed,Float} (just for aesthetic reasons) Modified: pypy/branch/jitffi/pypy/jit/codewriter/support.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/support.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/support.py Mon Oct 4 14:20:50 2010 @@ -1,5 +1,5 @@ import sys -from pypy.rpython.lltypesystem import lltype, rclass +from pypy.rpython.lltypesystem import lltype, rclass, rffi from pypy.rpython.ootypesystem import ootype from pypy.rpython import rlist from pypy.rpython.lltypesystem import rstr as ll_rstr, rdict as ll_rdict @@ -236,10 +236,10 @@ return func(llfunc)._push_float(value, ll_args, i) def _ll_3_libffi_call_int(llfunc, funcsym, ll_args): - return func(llfunc)._do_call(funcsym, ll_args, lltype.Signed) + return func(llfunc)._do_call(funcsym, ll_args, rffi.LONG) def _ll_3_libffi_call_float(llfunc, funcsym, ll_args): - return func(llfunc)._do_call(funcsym, ll_args, lltype.Float) + return func(llfunc)._do_call(funcsym, ll_args, rffi.DOUBLE) Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Mon Oct 4 14:20:50 2010 @@ -101,9 +101,9 @@ arg.push(self, ll_args, i) i += 1 arg = arg.next - if RESULT is lltype.Signed: + if RESULT is rffi.LONG: return self._do_call_int(self.funcsym, ll_args) - elif RESULT is lltype.Float: + elif RESULT is rffi.DOUBLE: return self._do_call_float(self.funcsym, ll_args) else: raise TypeError, 'Unsupported result type: %s' % RESULT @@ -135,11 +135,11 @@ @jit.oopspec('libffi_call_int(self, funcsym, ll_args)') def _do_call_int(self, funcsym, ll_args): - return self._do_call(funcsym, ll_args, lltype.Signed) + return self._do_call(funcsym, ll_args, rffi.LONG) @jit.oopspec('libffi_call_float(self, funcsym, ll_args)') def _do_call_float(self, funcsym, ll_args): - return self._do_call(funcsym, ll_args, lltype.Float) + return self._do_call(funcsym, ll_args, rffi.DOUBLE) # ------------------------------------------------------------------------ # private methods From antocuni at codespeak.net Mon Oct 4 15:24:11 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 15:24:11 +0200 (CEST) Subject: [pypy-svn] r77569 - in pypy/branch/jitffi/pypy: jit/metainterp/test rlib Message-ID: <20101004132411.B8462282BAD@codespeak.net> Author: antocuni Date: Mon Oct 4 15:24:09 2010 New Revision: 77569 Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: cast the result to the correct type when calling a function Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py Mon Oct 4 15:24:09 2010 @@ -2,7 +2,7 @@ import py from pypy.rlib.jit import JitDriver, hint from pypy.jit.metainterp.test.test_basic import LLJitMixin -from pypy.rlib.libffi import CDLL, ffi_type_sint, ffi_type_double, ArgChain, Func +from pypy.rlib.libffi import CDLL, ffi_type_sint, ffi_type_double, ffi_type_uchar, ArgChain, Func from pypy.tool.udir import udir from pypy.translator.tool.cbuild import ExternalCompilationInfo from pypy.translator.platform import platform @@ -12,7 +12,7 @@ def setup_class(cls): # prepare C code as an example, so we can load it and call # it via rlib.libffi - c_file = udir.ensure("test_jit_direct_call", dir=1).join("xlib.c") + c_file = udir.ensure("test_jit_fficall", dir=1).join("xlib.c") c_file.write(py.code.Source(''' int sum_xy(int x, double y) { @@ -25,8 +25,13 @@ return -x; return x; } + + unsigned char cast_to_uchar(int x) + { + return 200+(unsigned char)x; + } ''')) - eci = ExternalCompilationInfo(export_symbols=['sum_xy']) + eci = ExternalCompilationInfo(export_symbols=[]) cls.lib_name = str(platform.compile([c_file], eci, 'x', standalone=False)) @@ -77,3 +82,23 @@ assert res == 9 self.check_loops(call=1) + def test_cast_result(self): + driver = JitDriver(reds = ['n', 'res', 'func'], greens = []) + + def f(n): + cdll = CDLL(self.lib_name) + func = cdll.getpointer('cast_to_uchar', [ffi_type_sint], + ffi_type_uchar) + res = 0 + while n < 10: + driver.jit_merge_point(n=n, func=func, res=res) + driver.can_enter_jit(n=n, func=func, res=res) + func = hint(func, promote=True) + argchain = ArgChain() + argchain.int(0) + res = func.call(argchain, rffi.UCHAR) + n += 1 + return res + + res = self.meta_interp(f, [0]) + assert res == 200 Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Mon Oct 4 15:24:09 2010 @@ -15,6 +15,13 @@ import_types() del import_types +def _fits_into_long(TYPE): + if not isinstance(TYPE, lltype.Number): + return False + if TYPE is rffi.FLOAT or TYPE is rffi.DOUBLE: + return False + sz = rffi.sizeof(TYPE) + return sz <= rffi.sizeof(rffi.LONG) # ---------------------------------------------------------------------- @@ -101,12 +108,15 @@ arg.push(self, ll_args, i) i += 1 arg = arg.next - if RESULT is rffi.LONG: - return self._do_call_int(self.funcsym, ll_args) + # + if _fits_into_long(RESULT): + res = self._do_call_int(self.funcsym, ll_args) elif RESULT is rffi.DOUBLE: return self._do_call_float(self.funcsym, ll_args) else: raise TypeError, 'Unsupported result type: %s' % RESULT + # + return rffi.cast(RESULT, res) # END OF THE PUBLIC INTERFACE # ------------------------------------------------------------------------ @@ -119,6 +129,7 @@ ll_args = lltype.malloc(rffi.VOIDPP.TO, len(self.argtypes), flavor='raw') return ll_args + # _push_* and _do_call_* in theory could be automatically specialize()d by # the annotator. However, specialization doesn't work well with oopspec, # so we specialize them by hand From antocuni at codespeak.net Mon Oct 4 15:28:44 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 15:28:44 +0200 (CEST) Subject: [pypy-svn] r77570 - pypy/branch/jitffi/pypy/jit/metainterp/test Message-ID: <20101004132844.416B4282BAD@codespeak.net> Author: antocuni Date: Mon Oct 4 15:28:42 2010 New Revision: 77570 Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py Log: use rffi.{LONG,DOUBLE} also here Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py Mon Oct 4 15:28:42 2010 @@ -48,7 +48,7 @@ func = hint(func, promote=True) argchain = ArgChain() argchain.int(n).float(1.2) - n = func.call(argchain, lltype.Signed) + n = func.call(argchain, rffi.LONG) return n res = self.meta_interp(f, [0]) @@ -74,7 +74,7 @@ func = hint(func, promote=True) argchain = ArgChain() argchain.float(float(-n)) - res = func.call(argchain, lltype.Float) + res = func.call(argchain, rffi.DOUBLE) n += 1 return res From arigo at codespeak.net Mon Oct 4 15:29:38 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 4 Oct 2010 15:29:38 +0200 (CEST) Subject: [pypy-svn] r77571 - pypy/branch/32ptr-on-64bit Message-ID: <20101004132938.6BD27282BAD@codespeak.net> Author: arigo Date: Mon Oct 4 15:29:36 2010 New Revision: 77571 Added: pypy/branch/32ptr-on-64bit/ - copied from r77569, pypy/trunk/ Log: A branch in which I play around with the idea of compressing 64-bit pointers into 32-bit integers. The "compression" is simply taking the pointer value and shifting it 3 bits, so it is limited to a 32GB heap. From arigo at codespeak.net Mon Oct 4 15:30:25 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 4 Oct 2010 15:30:25 +0200 (CEST) Subject: [pypy-svn] r77572 - in pypy/branch/32ptr-on-64bit/pypy: annotation config rpython rpython/lltypesystem rpython/memory rpython/memory/gc rpython/memory/test rpython/test translator/c translator/c/src translator/c/test Message-ID: <20101004133025.07E1A282BAD@codespeak.net> Author: arigo Date: Mon Oct 4 15:30:22 2010 New Revision: 77572 Added: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py Modified: pypy/branch/32ptr-on-64bit/pypy/annotation/model.py pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rclass.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rffi.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gctypelayout.py pypy/branch/32ptr-on-64bit/pypy/rpython/raddress.py pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py pypy/branch/32ptr-on-64bit/pypy/translator/c/database.py pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py pypy/branch/32ptr-on-64bit/pypy/translator/c/src/commondefs.h pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_lladdresses.py pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_newgc.py Log: In-progress. Modified: pypy/branch/32ptr-on-64bit/pypy/annotation/model.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/annotation/model.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/annotation/model.py Mon Oct 4 15:30:22 2010 @@ -518,6 +518,14 @@ def can_be_none(self): return False +# for compressed 32-bit "pointers" on 64-bit platforms + +class SomeHiddenGcRef32(SomeObject): + immutable = True + + def can_be_none(self): + return False + #____________________________________________________________ # annotation of low-level types @@ -582,6 +590,7 @@ (SomeChar(), lltype.Char), (SomeUnicodeCodePoint(), lltype.UniChar), (SomeAddress(), llmemory.Address), + (SomeHiddenGcRef32(), llmemory.HiddenGcRef32), ] def annotation_to_lltype(s_val, info=None): Modified: pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py Mon Oct 4 15:30:22 2010 @@ -193,6 +193,11 @@ "When true, enable the use of tagged pointers. " "If false, use normal boxing", default=False), + BoolOption("compressptr", "Compress pointers; limits the program to 32GB", + default=False, cmdline="--compressptr", + requires=[("translation.type_system", "lltype"), + ("translation.taggedpointers", False)] + + [("compressptr (64-bit only)", True)]*(not IS_64_BITS)), # options for ootype OptionDescription("ootype", "Object Oriented Typesystem options", [ Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py Mon Oct 4 15:30:22 2010 @@ -8,6 +8,7 @@ from pypy.rlib.objectmodel import Symbolic from pypy.rpython.lltypesystem import lltype from pypy.tool.uid import uid +from pypy.rlib.objectmodel import we_are_translated class AddressOffset(Symbolic): @@ -562,6 +563,61 @@ def array_item_type_match(T1, T2): return T1 == T2 or (T2 == GCREF and isinstance(T1, lltype.Ptr)) +# ____________________________________________________________ + +class _hiddengcref32(object): + """A 'hidden' compressed 32-bit value that represents a + full, 64-bit GC pointer.""" + + def __init__(self, adr64): + self.adr64 = adr64 + + def __repr__(self): + return '<_hiddengcref32>' + + def __str__(self): + return '<_hiddengcref32 %s>' % (self.adr64,) + +compressed_null_addr = _hiddengcref32(NULL) +HiddenGcRef32 = lltype.Primitive("HiddenGcRef32", compressed_null_addr) +_hiddengcref32._TYPE = HiddenGcRef32 + +class OddValueMarker(AddressOffset): + """This is the value '1'. Used as an odd-valued marker by the GC.""" + def __repr__(self): + return '< OddValueMarker 1 >' + def ref(self, ptr): + raise AssertionError("should not try to directly get the address" + " from a HiddenGcRef32") + +def has_odd_value_marker(addressofs): + if we_are_translated(): + return bool(addressofs & 1) + return _has_odd_value_marker(addressofs) + +def _has_odd_value_marker(addressofs): + while isinstance(addressofs, CompositeOffset): + addressofs = addressofs.offsets[-1] + return isinstance(addressofs, OddValueMarker) + +def remove_odd_value_marker(addressofs): + if we_are_translated(): + return addressofs - 1 + return _remove_odd_value_marker(addressofs) + +def _remove_odd_value_marker(addressofs): + if isinstance(addressofs, CompositeOffset): + offsets = list(addressofs.offsets) + offsets[-1] = _remove_odd_value_marker(offsets[-1]) + if offsets[-1] == 0: + del offsets[-1] + if len(offsets) == 1: + return offsets[0] + return CompositeOffset(*offsets) + assert isinstance(addressofs, OddValueMarker) + return 0 + +# ____________________________________________________________ class _fakeaccessor(object): def __init__(self, addr): @@ -597,6 +653,9 @@ class _char_fakeaccessor(_fakeaccessor): TYPE = lltype.Char +class _hiddengcref32_fakeaccessor(_fakeaccessor): + TYPE = HiddenGcRef32 + class _address_fakeaccessor(_fakeaccessor): TYPE = Address @@ -624,12 +683,14 @@ "char": lltype.Char, "address": Address, "float": lltype.Float, + "hiddengcref32": HiddenGcRef32, } fakeaddress.signed = property(_signed_fakeaccessor) fakeaddress.float = property(_float_fakeaccessor) fakeaddress.char = property(_char_fakeaccessor) fakeaddress.address = property(_address_fakeaccessor) +fakeaddress.hiddengcref32 = property(_hiddengcref32_fakeaccessor) fakeaddress._TYPE = Address # the obtained address will not keep the object alive. e.g. if the object is Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py Mon Oct 4 15:30:22 2010 @@ -426,6 +426,9 @@ 'gc_gettypeptr_group': LLOp(canfold=True), 'get_member_index': LLOp(canfold=True), + 'hide_into_adr32': LLOp(canrun=True), + 'show_from_adr32': LLOp(canrun=True), + # __________ used by the JIT ________ 'jit_marker': LLOp(), Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py Mon Oct 4 15:30:22 2010 @@ -526,6 +526,18 @@ def op_shrink_array(array, smallersize): return False +def op_hide_into_adr32(adr): + if lltype.typeOf(adr) != llmemory.Address: + adr = llmemory.cast_ptr_to_adr(adr) + return llmemory._hiddengcref32(adr) + +def op_show_from_adr32(RESTYPE, adr32): + if RESTYPE == llmemory.Address: + return adr32.adr64 + else: + return llmemory.cast_adr_to_ptr(adr32.adr64, RESTYPE) +op_show_from_adr32.need_result_type = True + # ____________________________________________________________ def get_op_impl(opname): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rclass.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rclass.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rclass.py Mon Oct 4 15:30:22 2010 @@ -4,6 +4,7 @@ from pypy.objspace.flow.model import Constant from pypy.rpython.error import TyperError from pypy.rpython.rmodel import Repr, inputconst, warning, mangle +from pypy.rpython.rmodel import externalvsinternalfield from pypy.rpython.rclass import AbstractClassRepr,\ AbstractInstanceRepr,\ MissingRTypeAttribute,\ @@ -326,7 +327,8 @@ fields = {} allinstancefields = {} if self.classdef is None: - fields['__class__'] = 'typeptr', get_type_repr(self.rtyper) + r = get_type_repr(self.rtyper) + fields['__class__'] = 'typeptr', r, r else: # instance attributes if llfields is None: @@ -337,8 +339,12 @@ if not attrdef.readonly: r = self.rtyper.getrepr(attrdef.s_value) mangled_name = 'inst_' + name - fields[name] = mangled_name, r - llfields.append((mangled_name, r.lowleveltype)) + if self.gcflavor == 'gc': + r, internal_r = externalvsinternalfield(self.rtyper, r) + else: + internal_r = r + fields[name] = mangled_name, r, internal_r + llfields.append((mangled_name, internal_r.lowleveltype)) self.rbase = getinstancerepr(self.rtyper, self.classdef.basedef, self.gcflavor) @@ -394,7 +400,7 @@ return getinstancerepr(self.rtyper, None, self.gcflavor) def _get_field(self, attr): - return self.fields[attr] + return self.fields[attr][:2] def null_instance(self): return nullptr(self.object_type) @@ -410,7 +416,7 @@ # recursively build the parent part of the instance self.rbase.initialize_prebuilt_data(value, classdef, result.super) # then add instance attributes from this level - for name, (mangled_name, r) in self.fields.items(): + for name, (mangled_name, _, r) in self.fields.items(): if r.lowleveltype is Void: llattrvalue = None else: @@ -440,7 +446,7 @@ def getfieldrepr(self, attr): """Return the repr used for the given attribute.""" if attr in self.fields: - mangled_name, r = self.fields[attr] + mangled_name, r, internal_r = self.fields[attr] return r else: if self.classdef is None: @@ -450,12 +456,13 @@ def getfield(self, vinst, attr, llops, force_cast=False, flags={}): """Read the given attribute (or __class__ for the type) of 'vinst'.""" if attr in self.fields: - mangled_name, r = self.fields[attr] + mangled_name, r, internal_r = self.fields[attr] cname = inputconst(Void, mangled_name) if force_cast: vinst = llops.genop('cast_pointer', [vinst], resulttype=self) self.hook_access_field(vinst, cname, llops, flags) - return llops.genop('getfield', [vinst, cname], resulttype=r) + v = llops.genop('getfield', [vinst, cname], resulttype=internal_r) + return llops.convertvar(v, internal_r, r) else: if self.classdef is None: raise MissingRTypeAttribute(attr) @@ -466,7 +473,8 @@ flags={}): """Write the given attribute (or __class__ for the type) of 'vinst'.""" if attr in self.fields: - mangled_name, r = self.fields[attr] + mangled_name, r, internal_r = self.fields[attr] + vvalue = llops.convertvar(vvalue, r, internal_r) cname = inputconst(Void, mangled_name) if force_cast: vinst = llops.genop('cast_pointer', [vinst], resulttype=self) @@ -499,7 +507,7 @@ for fldname in flds: if fldname == '__class__': continue - mangled_name, r = self.allinstancefields[fldname] + mangled_name, r, internal_r = self.allinstancefields[fldname] if r.lowleveltype is Void: continue value = self.classdef.classdesc.read_attribute(fldname, None) Added: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py ============================================================================== --- (empty file) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py Mon Oct 4 15:30:22 2010 @@ -0,0 +1,57 @@ +from pypy.tool.pairtype import pairtype +from pypy.rlib.objectmodel import we_are_translated +from pypy.config.translationoption import IS_64_BITS +from pypy.rpython.rmodel import Repr, inputconst +from pypy.rpython.lltypesystem import lltype, llmemory, rffi + + + +def get_compressed_gcref_repr(rtyper, baserepr): + try: + comprmgr = rtyper.compressed_gcref_manager + except AttributeError: + comprmgr = rtyper.compressed_gcref_manager = ComprGcRefManager(rtyper) + return comprmgr.get_compressed_gcref_repr(baserepr) + + +class ComprGcRefManager(object): + def __init__(self, rtyper): + assert IS_64_BITS # this is only for 64-bits + self.rtyper = rtyper + self.comprgcreprs = {} + + def get_compressed_gcref_repr(self, baserepr): + try: + return self.comprgcreprs[baserepr] + except KeyError: + comprgcrepr = CompressedGcRefRepr(self, baserepr) + self.comprgcreprs[baserepr] = comprgcrepr + return comprgcrepr + + +class CompressedGcRefRepr(Repr): + lowleveltype = llmemory.HiddenGcRef32 + + def __init__(self, mgr, baserepr): + self.mgr = mgr + self.baserepr = baserepr + self.BASETYPE = self.baserepr.lowleveltype + + def convert_const(self, value): + ptr = self.baserepr.convert_const(value) + T = lltype.typeOf(ptr) + assert T == self.BASETYPE + return llmemory._hiddengcref32(llmemory.cast_ptr_to_adr(ptr)) + + +class __extend__(pairtype(Repr, CompressedGcRefRepr)): + def convert_from_to((r_from, r_to), v, llops): + assert r_from.lowleveltype.TO._gckind == 'gc' + return llops.genop('hide_into_adr32', [v], + resulttype=llmemory.HiddenGcRef32) + +class __extend__(pairtype(CompressedGcRefRepr, Repr)): + def convert_from_to((r_from, r_to), v, llops): + assert r_to.lowleveltype.TO._gckind == 'gc' + return llops.genop('show_from_adr32', [v], + resulttype=r_to.lowleveltype) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rffi.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rffi.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rffi.py Mon Oct 4 15:30:22 2010 @@ -805,6 +805,8 @@ return 8 if tp is lltype.SingleFloat: return 4 + if tp is llmemory.HiddenGcRef32: + return 4 assert isinstance(tp, lltype.Number) if tp is lltype.Signed: return ULONG._type.BITS/8 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py Mon Oct 4 15:30:22 2010 @@ -1,4 +1,5 @@ from pypy.rpython.lltypesystem import lltype, llmemory, llarena +from pypy.rpython.lltypesystem.lloperation import llop from pypy.rlib.debug import ll_assert from pypy.rpython.memory.gcheader import GCHeaderBuilder from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE @@ -168,8 +169,10 @@ def trace(self, obj, callback, arg): """Enumerate the locations inside the given obj that can contain - GC pointers. For each such location, callback(pointer, arg) is - called, where 'pointer' is an address inside the object. + GC pointers. For each such GC pointer, callback(object, arg) is + called, where 'object' is the address of the stored object. + 'callback' can return a new address that replaces the old one in + 'obj', or it can return None. Typically, 'callback' is a bound method and 'arg' can be None. """ typeid = self.get_type_id(obj) @@ -179,16 +182,16 @@ item = obj + llmemory.gcarrayofptr_itemsoffset while length > 0: if self.points_to_valid_gc_object(item): - callback(item, arg) + newaddr = callback(item.address[0], arg) + if newaddr is not None: + item.address[0] = newaddr item += llmemory.gcarrayofptr_singleitemoffset length -= 1 return offsets = self.offsets_to_gc_pointers(typeid) i = 0 while i < len(offsets): - item = obj + offsets[i] - if self.points_to_valid_gc_object(item): - callback(item, arg) + self._trace_see(obj, offsets[i], callback, arg) i += 1 if self.has_gcptr_in_varsize(typeid): item = obj + self.varsize_offset_to_variable_part(typeid) @@ -198,14 +201,35 @@ while length > 0: j = 0 while j < len(offsets): - itemobj = item + offsets[j] - if self.points_to_valid_gc_object(itemobj): - callback(itemobj, arg) + self._trace_see(item, offsets[j], callback, arg) j += 1 item += itemlength length -= 1 trace._annspecialcase_ = 'specialize:arg(2)' + def _trace_see(self, obj, ofs, callback, arg): + if self.config.compressptr and llmemory.has_odd_value_marker(ofs): + # special handling of HiddenGcRef32 on 64-bit platforms + ofs = llmemory.remove_odd_value_marker(ofs) + item = obj + ofs + address = llop.show_from_adr32(llmemory.Address, + item.hiddengcref32[0]) + if self.is_valid_gc_object(address): + newaddr = callback(address, arg) + if newaddr is not None: + newaddr32 = llop.hide_into_adr32(llmemory.HiddenGcRef32, + newaddr) + item.hiddengcref32[0] = newaddr32 + else: + # common case + item = obj + ofs + if self.points_to_valid_gc_object(item): + newaddr = callback(item.address[0], arg) + if newaddr is not None: + item.address[0] = newaddr + _trace_see._annspecialcase_ = 'specialize:arg(3)' + _trace_see._always_inline_ = True + def trace_partial(self, obj, start, stop, callback, arg): """Like trace(), but only walk the array part, for indices in range(start, stop). Must only be called if has_gcptr_in_varsize(). @@ -218,7 +242,9 @@ item += llmemory.gcarrayofptr_singleitemoffset * start while length > 0: if self.points_to_valid_gc_object(item): - callback(item, arg) + newaddr = callback(item.address[0], arg) + if newaddr is not None: + item.address[0] = newaddr item += llmemory.gcarrayofptr_singleitemoffset length -= 1 return @@ -231,9 +257,7 @@ while length > 0: j = 0 while j < len(offsets): - itemobj = item + offsets[j] - if self.points_to_valid_gc_object(itemobj): - callback(itemobj, arg) + self._trace_see(item, offsets[j], callback, arg) j += 1 item += itemlength length -= 1 @@ -279,8 +303,7 @@ obj = root.address[0] ll_assert(bool(obj), "NULL address from walk_roots()") self._debug_record(obj) - def _debug_callback2(self, pointer, ignored): - obj = pointer.address[0] + def _debug_callback2(self, obj, ignored): ll_assert(bool(obj), "NULL address from self.trace()") self._debug_record(obj) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py Mon Oct 4 15:30:22 2010 @@ -1025,19 +1025,17 @@ def _trace_drag_out1(self, root): - self._trace_drag_out(root, None) + root.address[0] = self._trace_drag_out(root.address[0], None) - def _trace_drag_out(self, root, ignored): - obj = root.address[0] + def _trace_drag_out(self, obj, ignored): # # If 'obj' is not in the nursery, nothing to change. if not self.is_in_nursery(obj): - return + return obj # # If 'obj' was already forwarded, change it to its forwarding address. if self.is_forwarded(obj): - root.address[0] = self.get_forwarding_address(obj) - return + return self.get_forwarding_address(obj) # # First visit to 'obj': we must move it out of the nursery. size_gc_header = self.gcheaderbuilder.size_gc_header @@ -1075,14 +1073,14 @@ newobj = newhdr + size_gc_header llmemory.cast_adr_to_ptr(obj, FORWARDSTUBPTR).forw = newobj # - # Change the original pointer to this object. - root.address[0] = newobj - # # Add the newobj to the list 'old_objects_pointing_to_young', # because it can contain further pointers to other young objects. # We will fix such references to point to the copy of the young # objects when we walk 'old_objects_pointing_to_young'. self.old_objects_pointing_to_young.append(newobj) + # + # Change the original pointer to this newly built object. + return newobj def _malloc_out_of_nursery(self, totalsize): @@ -1279,8 +1277,8 @@ def _collect_ref(self, root): self.objects_to_trace.append(root.address[0]) - def _collect_ref_rec(self, root, ignored): - self.objects_to_trace.append(root.address[0]) + def _collect_ref_rec(self, obj, ignored): + self.objects_to_trace.append(obj) def visit_all_objects(self): pending = self.objects_to_trace @@ -1421,8 +1419,8 @@ self.objects_with_finalizers.delete() self.objects_with_finalizers = new_with_finalizer - def _append_if_nonnull(pointer, stack): - stack.append(pointer.address[0]) + def _append_if_nonnull(obj, stack): + stack.append(obj) _append_if_nonnull = staticmethod(_append_if_nonnull) def _finalization_state(self, obj): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py Mon Oct 4 15:30:22 2010 @@ -16,6 +16,8 @@ """ _alloc_flavor_ = 'raw' + # if config.translation.compressptr, the OFFSETS_TO_GC_PTR lists + # as odd integers the fields that are compressed pointers (UINT). OFFSETS_TO_GC_PTR = lltype.Array(lltype.Signed) ADDRESS_VOID_FUNC = lltype.FuncType([llmemory.Address], lltype.Void) FINALIZERTYPE = lltype.Ptr(ADDRESS_VOID_FUNC) @@ -389,6 +391,8 @@ # (adr + off) elif isinstance(TYPE, lltype.Ptr) and TYPE.TO._gckind == 'gc': offsets.append(0) + elif TYPE == llmemory.HiddenGcRef32: + offsets.append(llmemory.OddValueMarker()) return offsets def gc_pointers_inside(v, adr, mutable_only=False): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py Mon Oct 4 15:30:22 2010 @@ -88,7 +88,9 @@ def setinterior(self, toplevelcontainer, inneraddr, INNERTYPE, newvalue, offsets=()): if (lltype.typeOf(toplevelcontainer).TO._gckind == 'gc' and - isinstance(INNERTYPE, lltype.Ptr) and INNERTYPE.TO._gckind == 'gc'): + (isinstance(INNERTYPE, lltype.Ptr) and + INNERTYPE.TO._gckind == 'gc') + or INNERTYPE == llmemory.HiddenGcRef32): # wb = True if self.has_write_barrier_from_array: Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py Mon Oct 4 15:30:22 2010 @@ -77,6 +77,23 @@ #assert simulator.current_size - curr < 16000 * INT_SIZE / 4 #print "size before: %s, size after %s" % (curr, simulator.current_size) + def test_llinterp_instances(self): + class Cons(object): + def __init__(self, car, cdr): + self.car = car + self.cdr = cdr + def malloc_a_lot(): + i = 0 + while i < 10: + i += 1 + a = Cons(1, Cons(2, Cons(i, None))) + b = a + j = 0 + while j < 20: + j += 1 + b = Cons(j, b) + res = self.interpret(malloc_a_lot, []) + def test_global_list(self): lst = [] def append_to_list(i, j): @@ -776,3 +793,20 @@ class TestMiniMarkGCCardMarking(TestMiniMarkGC): GC_PARAMS = {'card_page_indices': 4} + +class TestMiniMarkGCCompressPtr(TestMiniMarkGC): + def setup_class(cls): + TestMiniMarkGC.setup_class.im_func(cls) + # + from pypy.config.translationoption import IS_64_BITS + if not IS_64_BITS: + py.test.skip("only for 64-bits") + from pypy.config.pypyoption import get_pypy_config + cls._config = get_pypy_config(translating=True) + cls._config.translation.compressptr = True + + def interpret(self, *args, **kwds): + if kwds.get('taggedpointers'): + py.test.skip("cannot have both taggedpointers and compressptr") + kwds['config'] = self._config + return super(TestMiniMarkGCCompressPtr, self).interpret(*args, **kwds) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gctypelayout.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gctypelayout.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gctypelayout.py Mon Oct 4 15:30:22 2010 @@ -37,6 +37,15 @@ for T, c in [(GC_S, 0), (GC_S2, 2), (GC_A, 0), (GC_A2, 0), (GC_S3, 2)]: assert len(offsets_to_gc_pointers(T)) == c +def test_hiddenptr32(): + from pypy.rpython.lltypesystem.llmemory import HIDDENPTR32 + from pypy.rpython.lltypesystem.llmemory import has_odd_value_marker + T = lltype.GcStruct('T', ('foo', lltype.Ptr(GC_S3)), ('bar', HIDDENPTR32)) + ofs = offsets_to_gc_pointers(T) + assert len(ofs) == 2 + assert not has_odd_value_marker(ofs[0]) + assert has_odd_value_marker(ofs[1]) + def test_layout_builder(lltype2vtable=None): # XXX a very minimal test layoutbuilder = TypeLayoutBuilder(FakeGC, lltype2vtable) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/raddress.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/raddress.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/raddress.py Mon Oct 4 15:30:22 2010 @@ -2,7 +2,7 @@ from pypy.tool.pairtype import pairtype from pypy.annotation import model as annmodel from pypy.rpython.lltypesystem.llmemory import NULL, Address, \ - cast_adr_to_int, fakeaddress + cast_adr_to_int, fakeaddress, HiddenGcRef32, _hiddengcref32 from pypy.rpython.rmodel import Repr, IntegerRepr from pypy.rpython.rptr import PtrRepr from pypy.rpython.lltypesystem import lltype @@ -24,6 +24,13 @@ def rtyper_makekey(self): return self.__class__, self.type +class __extend__(annmodel.SomeHiddenGcRef32): + def rtyper_makerepr(self, rtyper): + return hiddengcref32_repr + + def rtyper_makekey(self): + return self.__class__, + class AddressRepr(Repr): lowleveltype = Address @@ -139,4 +146,13 @@ def convert_from_to((r_ptr, r_addr), v, llops): return llops.genop('cast_ptr_to_adr', [v], resulttype=Address) +# ____________________________________________________________ + +class HiddenGcRef32Repr(Repr): + lowleveltype = HiddenGcRef32 + + def convert_const(self, value): + assert isinstance(value, _hiddengcref32) + return value +hiddengcref32_repr = HiddenGcRef32Repr() Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py Mon Oct 4 15:30:22 2010 @@ -438,9 +438,21 @@ from pypy.rpython import rclass if (isinstance(item_repr, rclass.AbstractInstanceRepr) and getattr(item_repr, 'gcflavor', 'gc') == 'gc'): + #if rtyper.annotator.translator.config.translation.compressptr: + # return externalvsinternalfield(rtyper, item_repr) return item_repr, rclass.getinstancerepr(rtyper, None) - else: - return item_repr, item_repr + return item_repr, item_repr + +def externalvsinternalfield(rtyper, field_repr): + # usually a no-op, except if config.translation.compressptr, in which case + # it tries hard to return a wrapping compressed_gcref_repr + if rtyper.annotator.translator.config.translation.compressptr: + if (isinstance(field_repr.lowleveltype, Ptr) and + field_repr.lowleveltype.TO._gckind == 'gc'): + from pypy.rpython.lltypesystem import rcompressed + return (field_repr, + rcompressed.get_compressed_gcref_repr(rtyper, field_repr)) + return field_repr, field_repr class DummyValueBuilder(object): Added: pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py ============================================================================== --- (empty file) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py Mon Oct 4 15:30:22 2010 @@ -0,0 +1,28 @@ +import py +from pypy.config.translationoption import IS_64_BITS +from pypy.rpython.test import test_rclass + + +def setup_module(mod): + if not IS_64_BITS: + py.test.skip("for 64-bits only") + + +class MixinCompressed64(object): + def _get_config(self): + from pypy.config.pypyoption import get_pypy_config + config = get_pypy_config(translating=True) + config.translation.compressptr = True + return config + + def interpret(self, *args, **kwds): + kwds['config'] = self._get_config() + return super(MixinCompressed64, self).interpret(*args, **kwds) + + def interpret_raises(self, *args, **kwds): + kwds['config'] = self._get_config() + return super(MixinCompressed64, self).interpret_raises(*args, **kwds) + + +class TestLLtype64(MixinCompressed64, test_rclass.TestLLtype): + pass Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/database.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/database.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/database.py Mon Oct 4 15:30:22 2010 @@ -55,6 +55,7 @@ # assign to a constant object is something C doesn't think is # constant self.late_initializations = [] + self.late_initializations_hiddengcref32 = [] self.namespace = CNameManager() if translator is None or translator.rtyper is None: Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py Mon Oct 4 15:30:22 2010 @@ -771,25 +771,33 @@ yield '}' def generic_initializationexpr(db, value, access_expr, decoration): - if isinstance(typeOf(value), ContainerType): + TYPE = typeOf(value) + if isinstance(TYPE, ContainerType): node = db.getcontainernode(value) lines = list(node.initializationexpr(decoration+'.')) lines[-1] += ',' return lines else: comma = ',' - if typeOf(value) == Ptr(PyObject) and value: + if TYPE == Ptr(PyObject) and value: # cannot just write 'gxxx' as a constant in a structure :-( node = db.getcontainernode(value._obj) expr = 'NULL /*%s*/' % node.name node.where_to_copy_me.append('&%s' % access_expr) - elif typeOf(value) == Float and (isinf(value) or isnan(value)): - db.late_initializations.append(('%s' % access_expr, db.get(value))) + elif TYPE == Float and (isinf(value) or isnan(value)): + db.late_initializations.append((access_expr, db.get(value))) expr = '0.0 /* patched later by %sinfinity */' % ( '-+'[value > 0]) + elif TYPE == llmemory.HiddenGcRef32: + if value.adr64: + db.late_initializations_hiddengcref32.append((access_expr, + value)) + expr = '0 /*HIDE_INTO_ADR32%s*/' % db.get(value.adr64.ptr) + else: + expr = '0' else: expr = db.get(value) - if typeOf(value) is Void: + if TYPE is Void: comma = '' expr += comma i = expr.find('\n') Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py Mon Oct 4 15:30:22 2010 @@ -7,7 +7,7 @@ from pypy.rpython.lltypesystem.llmemory import Address, \ AddressOffset, ItemOffset, ArrayItemsOffset, FieldOffset, \ CompositeOffset, ArrayLengthOffset, \ - GCHeaderOffset, GCREF, AddressAsInt + GCHeaderOffset, GCREF, AddressAsInt, HiddenGcRef32, OddValueMarker from pypy.rpython.lltypesystem.llarena import RoundedUpForAllocation from pypy.translator.c.support import cdecl, barebonearray @@ -63,6 +63,8 @@ return '(%s+%dL)' % (name, value.rest) elif isinstance(value, AddressAsInt): return '((long)%s)' % name_address(value.adr, db) + elif isinstance(value, OddValueMarker): + return '1 /*OddValueMarker*/' else: raise Exception("unimplemented symbolic %r"%value) if value is None: @@ -145,6 +147,12 @@ else: return 'NULL' +def name_hiddengcref32(value, db): + # The only prebuilt HiddenGcRef32 that should occur in a translated C + # program occur as fields or items of a GcStruct or GcArray. + db.get(value.adr64) + return 'HIDE_INTO_ADR32(???) /* see primitive.py, name_hiddengcref32() */' + def name_small_integer(value, db): """Works for integers of size at most INT or UINT.""" if isinstance(value, Symbolic): @@ -173,6 +181,7 @@ Void: name_void, Address: name_address, GCREF: name_gcref, + HiddenGcRef32: name_hiddengcref32, } PrimitiveType = { @@ -188,6 +197,7 @@ Void: 'void @', Address: 'void* @', GCREF: 'void* @', + HiddenGcRef32: 'hiddengcref32_t @', } def define_c_primitive(ll_type, c_name, suffix=''): Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/commondefs.h ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/src/commondefs.h (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/src/commondefs.h Mon Oct 4 15:30:22 2010 @@ -65,6 +65,15 @@ # define SIZEOF_LONG 8 # define SIZEOF_LONG_LONG 8 + /* HiddenGcRef32 support (on 64-bits only) */ + typedef unsigned int hiddengcref32_t; +# define uint32_t hiddengcref32_t +# define OP_SHOW_FROM_ADR32(x, r) r = (void*)(((unsigned long)(x)) << 3) +# define OP_HIDE_INTO_ADR32(x, r) \ + r = (hiddengcref32_t)(((unsigned long)(x)) >> 3); \ + RPyAssert((void*)(((unsigned long)(r)) << 3) == (x), \ + "pointer too big or misaligned!") + #endif /********************************************************/ Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_lladdresses.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_lladdresses.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_lladdresses.py Mon Oct 4 15:30:22 2010 @@ -232,3 +232,43 @@ assert res == 456 res = fc(77) assert res == 123 + + +##class TestHiddenGcRef32(StandaloneTests): + +## def setup_class(cls): +## from pypy.config.translationoption import IS_64_BITS +## if 0:# not IS_64_BITS: +## py.test.skip("only for 64-bits") + +## def test_hiddengcref32(self): +## from pypy.rpython.lltypesystem.lloperation import llop +## S = lltype.GcStruct('S', ('x', lltype.Signed), +## ('y', HiddenGcRef32)) +## prebuilt = lltype.malloc(S, immortal=True) +## prebuilt2 = lltype.malloc(S, immortal=True) +## prebuilt.x = 42 +## prebuilt2.x = 53 +## prebuilt.y = llop.hide_into_adr32(HiddenGcRef32, prebuilt2) +## prebuilt2.y = llop.hide_into_adr32(HiddenGcRef32, prebuilt) + +## def check(a, b): +## p = llop.show_from_adr32(lltype.Ptr(S), a.y) +## assert p == b + +## def f(argv): +## assert prebuilt.x == 42 +## assert prebuilt2.x == 53 +## check(prebuilt, prebuilt2) +## check(prebuilt2, prebuilt) +## p = lltype.malloc(S) +## p.y = llop.hide_into_adr32(HiddenGcRef32, prebuilt) +## prebuilt2.y = llop.hide_into_adr32(HiddenGcRef32, p) +## check(p, prebuilt) +## check(prebuilt2, p) +## check(prebuilt, prebuilt2) +## return 0 + +## fc = self.compile(f) +## res = fc([]) +## assert res == 0 Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_newgc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_newgc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_newgc.py Mon Oct 4 15:30:22 2010 @@ -18,6 +18,7 @@ should_be_moving = False removetypeptr = False taggedpointers = False + compressptr = False GC_CAN_MOVE = False GC_CAN_MALLOC_NONMOVABLE = True GC_CAN_SHRINK_ARRAY = False @@ -41,7 +42,8 @@ t = Translation(main, standalone=True, gc=cls.gcpolicy, policy=annpolicy.StrictAnnotatorPolicy(), taggedpointers=cls.taggedpointers, - gcremovetypeptr=cls.removetypeptr) + gcremovetypeptr=cls.removetypeptr, + compressptr=cls.compressptr) t.disable(['backendopt']) t.set_backend_extra_options(c_debug_defines=True) t.rtype() @@ -623,7 +625,7 @@ assert open(self.filename, 'r').read() == "hello world\n" os.unlink(self.filename) - def define_callback_with_collect(cls): + def XXXXXXXXXdefine_callback_with_collect(cls): from pypy.rlib.libffi import ffi_type_pointer, cast_type_to_ffitype,\ CDLL, ffi_type_void, CallbackFuncPtr, ffi_type_sint from pypy.rpython.lltypesystem import rffi, ll2ctypes @@ -1333,6 +1335,15 @@ def test_gc_heap_stats(self): py.test.skip("not implemented") +class TestMiniMarkGCCompressPtr(TestMiniMarkGC): + compressptr = True + + def setup_class(cls): + from pypy.config.translationoption import IS_64_BITS + if not IS_64_BITS: + py.test.skip("only for 64-bits") + TestMiniMarkGC.setup_class.im_func(cls) + # ____________________________________________________________________ class TaggedPointersTest(object): From antocuni at codespeak.net Mon Oct 4 15:52:29 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 15:52:29 +0200 (CEST) Subject: [pypy-svn] r77573 - in pypy/branch/jitffi/pypy: jit/backend/llsupport jit/backend/llsupport/test jit/backend/test jit/metainterp/optimizeopt jit/metainterp/test rlib rlib/test Message-ID: <20101004135229.EB15F282BE8@codespeak.net> Author: antocuni Date: Mon Oct 4 15:52:28 2010 New Revision: 77573 Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_ffisupport.py pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py pypy/branch/jitffi/pypy/rlib/libffi.py pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Log: introduce a new namespace to contain all the ffi types Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py Mon Oct 4 15:52:28 2010 @@ -4,7 +4,7 @@ def get_call_descr_dynamic(ffi_args, ffi_result, extrainfo=None): """Get a call descr: the types of result and args are represented by - rlib.libffi.ffi_type_*""" + rlib.libffi.types.*""" try: reskind = get_ffi_type_kind(ffi_result) argkinds = [get_ffi_type_kind(arg) for arg in ffi_args] @@ -23,41 +23,41 @@ # XXX: maybe we can turn this into a dictionary, but we need to do it at -# runtime as libffi.ffi_type_* are pointers +# runtime as libffi.types.* pointers def get_ffi_type_kind(ffi_type): - from pypy.rlib import libffi - if ffi_type is libffi.ffi_type_void: + from pypy.rlib.libffi import types + if ffi_type is types.void: return history.VOID - elif ffi_type is libffi.ffi_type_pointer: + elif ffi_type is types.pointer: return history.REF - elif ffi_type is libffi.ffi_type_double: + elif ffi_type is types.double: return history.FLOAT - elif ffi_type is libffi.ffi_type_uchar: + elif ffi_type is types.uchar: return history.INT - elif ffi_type is libffi.ffi_type_uint8: + elif ffi_type is types.uint8: return history.INT - elif ffi_type is libffi.ffi_type_schar: + elif ffi_type is types.schar: return history.INT - elif ffi_type is libffi.ffi_type_sint8: + elif ffi_type is types.sint8: return history.INT - elif ffi_type is libffi.ffi_type_uint16: + elif ffi_type is types.uint16: return history.INT - elif ffi_type is libffi.ffi_type_ushort: + elif ffi_type is types.ushort: return history.INT - elif ffi_type is libffi.ffi_type_sint16: + elif ffi_type is types.sint16: return history.INT - elif ffi_type is libffi.ffi_type_sshort: + elif ffi_type is types.sshort: return history.INT - elif ffi_type is libffi.ffi_type_uint: + elif ffi_type is types.uint: return history.INT - elif ffi_type is libffi.ffi_type_uint32: + elif ffi_type is types.uint32: return history.INT - elif ffi_type is libffi.ffi_type_sint: + elif ffi_type is types.sint: return history.INT - elif ffi_type is libffi.ffi_type_sint32: + elif ffi_type is types.sint32: return history.INT - ## elif ffi_type is libffi.ffi_type_uint64: + ## elif ffi_type is types.uint64: ## return history.INT - ## elif ffi_type is libffi.ffi_type_sint64: + ## elif ffi_type is types.sint64: ## return history.INT raise KeyError Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_ffisupport.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_ffisupport.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_ffisupport.py Mon Oct 4 15:52:28 2010 @@ -1,18 +1,18 @@ -from pypy.rlib import libffi +from pypy.rlib.libffi import types from pypy.jit.backend.llsupport.ffisupport import get_call_descr_dynamic, \ VoidCallDescr, DynamicIntCallDescr def test_call_descr_dynamic(): - args = [libffi.ffi_type_sint, libffi.ffi_type_double, libffi.ffi_type_pointer] - descr = get_call_descr_dynamic(args, libffi.ffi_type_void) + args = [types.sint, types.double, types.pointer] + descr = get_call_descr_dynamic(args, types.void) assert isinstance(descr, VoidCallDescr) assert descr.arg_classes == 'ifr' - descr = get_call_descr_dynamic([], libffi.ffi_type_sint8) + descr = get_call_descr_dynamic([], types.sint8) assert isinstance(descr, DynamicIntCallDescr) assert descr.get_result_size(False) == 1 - descr = get_call_descr_dynamic([], libffi.ffi_type_float) + descr = get_call_descr_dynamic([], types.float) assert descr is None # single floats are not supported so far Modified: pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py Mon Oct 4 15:52:28 2010 @@ -421,7 +421,7 @@ assert x == 3.5 - 42 def test_call(self): - from pypy.rlib.libffi import ffi_type_sint, ffi_type_uchar, ffi_type_sint16 + from pypy.rlib.libffi import types def func_int(a, b): return a + b @@ -429,9 +429,9 @@ return chr(ord(c) + ord(c1)) functions = [ - (func_int, lltype.Signed, ffi_type_sint, 655360), - (func_int, rffi.SHORT, ffi_type_sint16, 1213), - (func_char, lltype.Char, ffi_type_uchar, 12) + (func_int, lltype.Signed, types.sint, 655360), + (func_int, rffi.SHORT, types.sint16, 1213), + (func_char, lltype.Char, types.uchar, 12) ] for func, TP, ffi_type, num in functions: Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py Mon Oct 4 15:52:28 2010 @@ -15,7 +15,7 @@ def _get_signature(self, funcval): """ given the funcval, return a tuple (argtypes, restype), where the - actuall types are libffi.ffi_type_* + actuall types are libffi.types.* The implementation is tricky because we have three possible cases: Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py Mon Oct 4 15:52:28 2010 @@ -2,7 +2,7 @@ import py from pypy.rlib.jit import JitDriver, hint from pypy.jit.metainterp.test.test_basic import LLJitMixin -from pypy.rlib.libffi import CDLL, ffi_type_sint, ffi_type_double, ffi_type_uchar, ArgChain, Func +from pypy.rlib.libffi import CDLL, types, ArgChain, Func from pypy.tool.udir import udir from pypy.translator.tool.cbuild import ExternalCompilationInfo from pypy.translator.platform import platform @@ -40,8 +40,8 @@ def f(n): cdll = CDLL(self.lib_name) - func = cdll.getpointer('sum_xy', [ffi_type_sint, ffi_type_double], - ffi_type_sint) + func = cdll.getpointer('sum_xy', [types.sint, types.double], + types.sint) while n < 10: driver.jit_merge_point(n=n, func=func) driver.can_enter_jit(n=n, func=func) @@ -66,7 +66,7 @@ def f(n): cdll = CDLL(self.lib_name) - func = cdll.getpointer('abs', [ffi_type_double], ffi_type_double) + func = cdll.getpointer('abs', [types.double], types.double) res = 0.0 while n < 10: driver.jit_merge_point(n=n, func=func, res=res) @@ -87,8 +87,8 @@ def f(n): cdll = CDLL(self.lib_name) - func = cdll.getpointer('cast_to_uchar', [ffi_type_sint], - ffi_type_uchar) + func = cdll.getpointer('cast_to_uchar', [types.sint], + types.uchar) res = 0 while n < 10: driver.jit_merge_point(n=n, func=func, res=res) Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py Mon Oct 4 15:52:28 2010 @@ -3900,7 +3900,7 @@ # ------------------------------------------------ from pypy.rpython.lltypesystem import llmemory -from pypy.rlib.libffi import Func, ffi_type_sint, ffi_type_double +from pypy.rlib.libffi import Func, types from pypy.jit.metainterp.history import AbstractDescr class MyCallDescr(AbstractDescr): @@ -3928,7 +3928,7 @@ def _identityhash(self): return id(self) -class TestFfiCall(OptimizeOptTest, LLtypeMixin): +class TestFfiCall(BaseTestOptimizeOpt, LLtypeMixin): class namespace: cpu = LLtypeMixin.cpu @@ -3936,8 +3936,8 @@ int_float__int = MyCallDescr('if', 'i') funcptr = FakeLLObject() func = FakeLLObject(_fake_class=Func, - argtypes=[ffi_type_sint, ffi_type_double], - restype=ffi_type_sint) + argtypes=[types.sint, types.double], + restype=types.sint) namespace = namespace.__dict__ Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Mon Oct 4 15:52:28 2010 @@ -7,13 +7,31 @@ push_arg_as_ffiptr, c_ffi_call from pypy.rlib.rdynload import dlopen, dlclose, dlsym, dlsym_byordinal -def import_types(): - g = globals() - for key, value in clibffi.__dict__.iteritems(): - if key.startswith('ffi_type_'): - g[key] = value -import_types() -del import_types +class types(object): + """ + This namespace contains the primitive types you can use to declare the + signatures of the ffi functions. + + In general, the name of the types are closely related to the ones of the + C-level ffi_type_*: e.g, instead of ffi_type_sint you should use + libffi.types.sint. + + However, you should not rely on a perfect correspondence: in particular, + the exact meaning of ffi_type_{slong,ulong} changes a lot between libffi + versions, so types.slong could be different than ffi_type_slong. + """ + + @classmethod + def _import(cls): + prefix = 'ffi_type_' + for key, value in clibffi.__dict__.iteritems(): + if key.startswith(prefix): + name = key[len(prefix):] + setattr(cls, name, value) + cls.slong = clibffi.cast_type_to_ffitype(rffi.LONG) + cls.ulong = clibffi.cast_type_to_ffitype(rffi.ULONG) + +types._import() def _fits_into_long(TYPE): if not isinstance(TYPE, lltype.Number): @@ -23,7 +41,7 @@ sz = rffi.sizeof(TYPE) return sz <= rffi.sizeof(rffi.LONG) -# ---------------------------------------------------------------------- +# ====================================================================== class ArgChain(object): first = None @@ -71,6 +89,9 @@ func._push_float(self.floatval, ll_args, i) +# ====================================================================== + + class Func(AbstractFuncPtr): _immutable_fields_ = ['funcsym', 'argtypes', 'restype'] @@ -168,7 +189,7 @@ def _do_call(self, funcsym, ll_args, RESULT): # XXX: check len(args)? ll_result = lltype.nullptr(rffi.CCHARP.TO) - if self.restype != ffi_type_void: + if self.restype != types.void: ll_result = lltype.malloc(rffi.CCHARP.TO, intmask(self.restype.c_size), flavor='raw') @@ -192,8 +213,8 @@ lltype.free(ll_args, flavor='raw') -# ---------------------------------------------------------------------- - +# ====================================================================== + # XXX: it partially duplicate the code in clibffi.py class CDLL(object): Modified: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Mon Oct 4 15:52:28 2010 @@ -3,8 +3,7 @@ from pypy.rpython.lltypesystem import rffi, lltype from pypy.rpython.lltypesystem.ll2ctypes import ALLOCATED from pypy.rlib.test.test_clibffi import BaseFfiTest, get_libm_name -from pypy.rlib.libffi import CDLL, Func, get_libc_name, ArgChain -from pypy.rlib.libffi import ffi_type_double, ffi_type_void +from pypy.rlib.libffi import CDLL, Func, get_libc_name, ArgChain, types class TestLibffi(BaseFfiTest): @@ -41,16 +40,16 @@ def test_library_get_func(self): lib = self.get_libc() - ptr = lib.getpointer('fopen', [], ffi_type_void) - py.test.raises(KeyError, lib.getpointer, 'xxxxxxxxxxxxxxx', [], ffi_type_void) + ptr = lib.getpointer('fopen', [], types.void) + py.test.raises(KeyError, lib.getpointer, 'xxxxxxxxxxxxxxx', [], types.void) del ptr del lib assert not ALLOCATED def test_call_argchain(self): libm = self.get_libm() - pow = libm.getpointer('pow', [ffi_type_double, ffi_type_double], - ffi_type_double) + pow = libm.getpointer('pow', [types.double, types.double], + types.double) argchain = ArgChain() argchain.float(2.0).float(3.0) res = pow.call(argchain, rffi.DOUBLE) From afa at codespeak.net Mon Oct 4 16:19:43 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 16:19:43 +0200 (CEST) Subject: [pypy-svn] r77574 - pypy/trunk/pypy/jit/tl/spli Message-ID: <20101004141943.C601B282BE8@codespeak.net> Author: afa Date: Mon Oct 4 16:19:42 2010 New Revision: 77574 Modified: pypy/trunk/pypy/jit/tl/spli/interpreter.py Log: The "spli" interpreter uses the host CPython to create compiled files, be sure to use the same opcodes to interpret it. This fixes the tests when run on top of CPython 2.7. Modified: pypy/trunk/pypy/jit/tl/spli/interpreter.py ============================================================================== --- pypy/trunk/pypy/jit/tl/spli/interpreter.py (original) +++ pypy/trunk/pypy/jit/tl/spli/interpreter.py Mon Oct 4 16:19:42 2010 @@ -1,12 +1,14 @@ import os -from pypy.tool import stdlib_opcode as opcode +from pypy.tool import stdlib_opcode from pypy.jit.tl.spli import objects, pycode -from pypy.tool.stdlib_opcode import unrolling_opcode_descs -from pypy.tool.stdlib_opcode import opcode_method_names from pypy.rlib.unroll import unrolling_iterable from pypy.rlib.jit import JitDriver, hint, dont_look_inside from pypy.rlib.objectmodel import we_are_translated +opcode_method_names = stdlib_opcode.host_bytecode_spec.method_names +unrolling_opcode_descs = unrolling_iterable( + stdlib_opcode.host_bytecode_spec.ordered_opdescs) +HAVE_ARGUMENT = stdlib_opcode.host_HAVE_ARGUMENT compare_ops = [ "cmp_lt", # "<" @@ -79,7 +81,7 @@ self.stack_depth = hint(self.stack_depth, promote=True) op = ord(code[instr_index]) instr_index += 1 - if op >= opcode.HAVE_ARGUMENT: + if op >= HAVE_ARGUMENT: low = ord(code[instr_index]) hi = ord(code[instr_index + 1]) oparg = (hi << 8) | low @@ -183,6 +185,12 @@ next_instr += arg return next_instr + def POP_JUMP_IF_FALSE(self, arg, next_instr, code): + w_cond = self.pop() + if not w_cond.is_true(): + next_instr = arg + return next_instr + def JUMP_FORWARD(self, arg, next_instr, code): return next_instr + arg From afa at codespeak.net Mon Oct 4 16:23:11 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 16:23:11 +0200 (CEST) Subject: [pypy-svn] r77575 - pypy/branch/fast-forward/pypy/jit/tl/spli Message-ID: <20101004142311.9C8FD282BE8@codespeak.net> Author: afa Date: Mon Oct 4 16:23:10 2010 New Revision: 77575 Modified: pypy/branch/fast-forward/pypy/jit/tl/spli/interpreter.py Log: Merge r77574 from trunk Modified: pypy/branch/fast-forward/pypy/jit/tl/spli/interpreter.py ============================================================================== --- pypy/branch/fast-forward/pypy/jit/tl/spli/interpreter.py (original) +++ pypy/branch/fast-forward/pypy/jit/tl/spli/interpreter.py Mon Oct 4 16:23:10 2010 @@ -1,12 +1,14 @@ import os -from pypy.tool import stdlib_opcode as opcode +from pypy.tool import stdlib_opcode from pypy.jit.tl.spli import objects, pycode -from pypy.tool.stdlib_opcode import unrolling_opcode_descs -from pypy.tool.stdlib_opcode import opcode_method_names from pypy.rlib.unroll import unrolling_iterable from pypy.rlib.jit import JitDriver, hint, dont_look_inside from pypy.rlib.objectmodel import we_are_translated +opcode_method_names = stdlib_opcode.host_bytecode_spec.method_names +unrolling_opcode_descs = unrolling_iterable( + stdlib_opcode.host_bytecode_spec.ordered_opdescs) +HAVE_ARGUMENT = stdlib_opcode.host_HAVE_ARGUMENT compare_ops = [ "cmp_lt", # "<" @@ -79,7 +81,7 @@ self.stack_depth = hint(self.stack_depth, promote=True) op = ord(code[instr_index]) instr_index += 1 - if op >= opcode.HAVE_ARGUMENT: + if op >= HAVE_ARGUMENT: low = ord(code[instr_index]) hi = ord(code[instr_index + 1]) oparg = (hi << 8) | low @@ -183,6 +185,12 @@ next_instr += arg return next_instr + def POP_JUMP_IF_FALSE(self, arg, next_instr, code): + w_cond = self.pop() + if not w_cond.is_true(): + next_instr = arg + return next_instr + def JUMP_FORWARD(self, arg, next_instr, code): return next_instr + arg From antocuni at codespeak.net Mon Oct 4 16:50:07 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 4 Oct 2010 16:50:07 +0200 (CEST) Subject: [pypy-svn] r77576 - pypy/branch/jitffi/pypy/jit/backend/test Message-ID: <20101004145007.932BB282B90@codespeak.net> Author: antocuni Date: Mon Oct 4 16:50:05 2010 New Revision: 77576 Modified: pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py Log: fix the test to use the new libffi interface Modified: pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py Mon Oct 4 16:50:05 2010 @@ -517,16 +517,15 @@ assert res.value == func_ints(*args) def test_call_to_c_function(self): - # XXX: fix this to use libffi instead of clibffi - from pypy.rlib.clibffi import CDLL, ffi_type_uchar, ffi_type_sint + from pypy.rlib.libffi import CDLL, types, ArgChain libc = CDLL('libc.so.6') - c_tolower = libc.getpointer('tolower', [ffi_type_uchar], ffi_type_sint) - c_tolower.push_arg('A') - assert c_tolower.call(lltype.Signed) == ord('a') + c_tolower = libc.getpointer('tolower', [types.uchar], types.sint) + argchain = ArgChain().int(ord('A')) + assert c_tolower.call(argchain, rffi.INT) == ord('a') func_adr = llmemory.cast_ptr_to_adr(c_tolower.funcsym) funcbox = ConstInt(heaptracker.adr2int(func_adr)) - calldescr = self.cpu.calldescrof_dynamic([ffi_type_uchar], ffi_type_sint) + calldescr = self.cpu.calldescrof_dynamic([types.uchar], types.sint) res = self.execute_operation(rop.CALL, [funcbox, BoxInt(ord('A'))], 'int', From arigo at codespeak.net Mon Oct 4 16:51:18 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 4 Oct 2010 16:51:18 +0200 (CEST) Subject: [pypy-svn] r77577 - pypy/branch/minor-cleanup Message-ID: <20101004145118.E9B91282B90@codespeak.net> Author: arigo Date: Mon Oct 4 16:51:17 2010 New Revision: 77577 Added: pypy/branch/minor-cleanup/ - copied from r77576, pypy/trunk/ Log: Temporary branch. From arigo at codespeak.net Mon Oct 4 16:51:59 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 4 Oct 2010 16:51:59 +0200 (CEST) Subject: [pypy-svn] r77578 - in pypy/branch/minor-cleanup/pypy: annotation annotation/test module/sys objspace/std rpython rpython/test translator/goal Message-ID: <20101004145159.9584C282B90@codespeak.net> Author: arigo Date: Mon Oct 4 16:51:57 2010 New Revision: 77578 Removed: pypy/branch/minor-cleanup/pypy/rpython/rspecialcase.py pypy/branch/minor-cleanup/pypy/rpython/test/test_rspecialcase.py Modified: pypy/branch/minor-cleanup/pypy/annotation/policy.py pypy/branch/minor-cleanup/pypy/annotation/test/test_annrpython.py pypy/branch/minor-cleanup/pypy/module/sys/__init__.py pypy/branch/minor-cleanup/pypy/module/sys/state.py pypy/branch/minor-cleanup/pypy/objspace/std/fake.py pypy/branch/minor-cleanup/pypy/objspace/std/objspace.py pypy/branch/minor-cleanup/pypy/rpython/rtyper.py pypy/branch/minor-cleanup/pypy/translator/goal/ann_override.py Log: Kill an old way to do specialization at the annotator level. Modified: pypy/branch/minor-cleanup/pypy/annotation/policy.py ============================================================================== --- pypy/branch/minor-cleanup/pypy/annotation/policy.py (original) +++ pypy/branch/minor-cleanup/pypy/annotation/policy.py Mon Oct 4 16:51:57 2010 @@ -1,4 +1,4 @@ -# base annotation policy for overrides and specialization +# base annotation policy for specialization from pypy.annotation.specialize import default_specialize as default from pypy.annotation.specialize import specialize_argvalue, specialize_argtype, specialize_arglistitemtype from pypy.annotation.specialize import memo @@ -41,7 +41,7 @@ if directive is None: return pol.default_specialize - # specialize|override:name[(args)] + # specialize[(args)] directive_parts = directive.split('(', 1) if len(directive_parts) == 1: [name] = directive_parts @@ -60,14 +60,6 @@ except AttributeError: raise AttributeError("%r specialize tag not defined in annotation" "policy %s" % (name, pol)) - if directive.startswith('override:'): - # different signature: override__xyz(*args_s) - if parms: - raise Exception, "override:* specialisations don't support parameters" - def specialize_override(funcdesc, args_s): - funcdesc.overridden = True - return specializer(*args_s) - return specialize_override else: if not parms: return specializer @@ -92,9 +84,5 @@ from pypy.rpython.annlowlevel import LowLevelAnnotatorPolicy return LowLevelAnnotatorPolicy.specialize__ll_and_arg(*args) - def override__ignore(pol, *args): - bk = getbookkeeper() - return bk.immutablevalue(None) - class StrictAnnotatorPolicy(AnnotatorPolicy): allow_someobjects = False Modified: pypy/branch/minor-cleanup/pypy/annotation/test/test_annrpython.py ============================================================================== --- pypy/branch/minor-cleanup/pypy/annotation/test/test_annrpython.py (original) +++ pypy/branch/minor-cleanup/pypy/annotation/test/test_annrpython.py Mon Oct 4 16:51:57 2010 @@ -766,28 +766,6 @@ s = a.build_types(f, [list]) assert s.classdef is a.bookkeeper.getuniqueclassdef(IndexError) # KeyError ignored because l is a list - def test_overrides(self): - excs = [] - def record_exc(e): - """NOT_RPYTHON""" - excs.append(sys.exc_info) - record_exc._annspecialcase_ = "override:record_exc" - def g(): - pass - def f(): - try: - g() - except Exception, e: - record_exc(e) - class MyAnnotatorPolicy(policy.AnnotatorPolicy): - - def override__record_exc(pol, s_e): - return a.bookkeeper.immutablevalue(None) - - a = self.RPythonAnnotator(policy=MyAnnotatorPolicy()) - s = a.build_types(f, []) - assert s.const is None - def test_freeze_protocol(self): class Stuff: def __init__(self, flag): Modified: pypy/branch/minor-cleanup/pypy/module/sys/__init__.py ============================================================================== --- pypy/branch/minor-cleanup/pypy/module/sys/__init__.py (original) +++ pypy/branch/minor-cleanup/pypy/module/sys/__init__.py Mon Oct 4 16:51:57 2010 @@ -7,13 +7,15 @@ """Sys Builtin Module. """ def __init__(self, space, w_name): """NOT_RPYTHON""" # because parent __init__ isn't + if space.config.translating: + del self.__class__.interpleveldefs['pypy_getudir'] super(Module, self).__init__(space, w_name) self.checkinterval = 100 self.recursionlimit = 100 self.w_default_encoder = None self.defaultencoding = "ascii" self.filesystemencoding = None - + interpleveldefs = { '__name__' : '(space.wrap("sys"))', '__doc__' : '(space.wrap("PyPy sys module"))', @@ -37,7 +39,7 @@ 'argv' : 'state.get(space).w_argv', 'warnoptions' : 'state.get(space).w_warnoptions', 'builtin_module_names' : 'state.w_None', - 'pypy_getudir' : 'state.pypy_getudir', + 'pypy_getudir' : 'state.pypy_getudir', # not translated 'pypy_initial_path' : 'state.pypy_initial_path', '_getframe' : 'vm._getframe', Modified: pypy/branch/minor-cleanup/pypy/module/sys/state.py ============================================================================== --- pypy/branch/minor-cleanup/pypy/module/sys/state.py (original) +++ pypy/branch/minor-cleanup/pypy/module/sys/state.py Mon Oct 4 16:51:57 2010 @@ -95,15 +95,8 @@ def getio(space): return space.fromcache(IOState) -def _pypy_getudir(space): - """NOT_RPYTHON""" +def pypy_getudir(space): + """NOT_RPYTHON + (should be removed from interpleveldefs before translation)""" from pypy.tool.udir import udir return space.wrap(str(udir)) -_pypy_getudir._annspecialcase_ = "override:ignore" - -# we need the indirection because this function will live in a dictionary with other -# RPYTHON functions and share call sites with them. Better it not be a special-case -# directly. -def pypy_getudir(space): - return _pypy_getudir(space) - Modified: pypy/branch/minor-cleanup/pypy/objspace/std/fake.py ============================================================================== --- pypy/branch/minor-cleanup/pypy/objspace/std/fake.py (original) +++ pypy/branch/minor-cleanup/pypy/objspace/std/fake.py Mon Oct 4 16:51:57 2010 @@ -21,7 +21,6 @@ #debug_print("faking obj %s" % x) ft = fake_type(type(x)) return ft(space, x) -fake_object._annspecialcase_ = "override:fake_object" import sys @@ -47,7 +46,6 @@ w_exc = space.wrap(exc) w_value = space.wrap(value) raise OperationError, OperationError(w_exc, w_value), tb -wrap_exception._annspecialcase_ = "override:ignore" def fake_type(cpy_type): assert type(cpy_type) is type Modified: pypy/branch/minor-cleanup/pypy/objspace/std/objspace.py ============================================================================== --- pypy/branch/minor-cleanup/pypy/objspace/std/objspace.py (original) +++ pypy/branch/minor-cleanup/pypy/objspace/std/objspace.py Mon Oct 4 16:51:57 2010 @@ -242,7 +242,6 @@ w_result = getattr(self, 'w_' + x.__name__) return w_result return None - wrap_exception_cls._annspecialcase_ = "override:wrap_exception_cls" def unwrap(self, w_obj): if isinstance(w_obj, Wrappable): Modified: pypy/branch/minor-cleanup/pypy/rpython/rtyper.py ============================================================================== --- pypy/branch/minor-cleanup/pypy/rpython/rtyper.py (original) +++ pypy/branch/minor-cleanup/pypy/rpython/rtyper.py Mon Oct 4 16:51:57 2010 @@ -421,7 +421,7 @@ assert noexclink.exitcase is None if pos == "removed": # the exception cannot actually occur at all. - # See for example rspecialcase.rtype_call_specialcase(). + # This is set by calling exception_cannot_occur(). # We just remove all exception links. block.exitswitch = None block.exits = block.exits[:1] @@ -1019,7 +1019,7 @@ from pypy.rpython import rint, rbool, rfloat from pypy.rpython import rrange from pypy.rpython import rstr, rdict, rlist -from pypy.rpython import rclass, rbuiltin, rpbc, rspecialcase +from pypy.rpython import rclass, rbuiltin, rpbc from pypy.rpython import rexternalobj from pypy.rpython import rptr from pypy.rpython import rgeneric Modified: pypy/branch/minor-cleanup/pypy/translator/goal/ann_override.py ============================================================================== --- pypy/branch/minor-cleanup/pypy/translator/goal/ann_override.py (original) +++ pypy/branch/minor-cleanup/pypy/translator/goal/ann_override.py Mon Oct 4 16:51:57 2010 @@ -27,21 +27,6 @@ pol.pypytypes = {} pol.single_space = single_space - #def override__wrap_exception_cls(pol, space, x): - # import pypy.objspace.std.typeobject as typeobject - # clsdef = getbookkeeper().getuniqueclassdef(typeobject.W_TypeObject) - # return annmodel.SomeInstance(clsdef, can_be_None=True) - # - #def override__fake_object(pol, space, x): - # from pypy.interpreter import typedef - # clsdef = getbookkeeper().getuniqueclassdef(typedef.W_Root) - # return annmodel.SomeInstance(clsdef) - # - #def override__cpy_compile(pol, self, source, filename, mode, flags): - # from pypy.interpreter import pycode - # clsdef = getbookkeeper().getuniqueclassdef(pycode.PyCode) - # return annmodel.SomeInstance(clsdef) - def specialize__wrap(pol, funcdesc, args_s): from pypy.interpreter.baseobjspace import Wrappable from pypy.annotation.classdef import ClassDef From arigo at codespeak.net Mon Oct 4 16:54:04 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 4 Oct 2010 16:54:04 +0200 (CEST) Subject: [pypy-svn] r77579 - pypy/branch/larger-writebarrier Message-ID: <20101004145404.DB0E9282B90@codespeak.net> Author: arigo Date: Mon Oct 4 16:54:03 2010 New Revision: 77579 Removed: pypy/branch/larger-writebarrier/ Log: Remove merged branch. From afa at codespeak.net Mon Oct 4 19:01:36 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 19:01:36 +0200 (CEST) Subject: [pypy-svn] r77580 - pypy/branch/fast-forward/pypy/doc/config Message-ID: <20101004170136.28210282B90@codespeak.net> Author: afa Date: Mon Oct 4 19:01:35 2010 New Revision: 77580 Added: pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._multiprocessing.txt (contents, props changed) Log: Add missing doc file Added: pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._multiprocessing.txt ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._multiprocessing.txt Mon Oct 4 19:01:35 2010 @@ -0,0 +1,2 @@ +Use the '_multiprocessing' module. +Used by the 'multiprocessing standard lib module. This module is expected to be working and is included by default. From afa at codespeak.net Mon Oct 4 19:24:47 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 19:24:47 +0200 (CEST) Subject: [pypy-svn] r77581 - in pypy/branch/fast-forward/pypy: doc/config module/_io module/_io/test Message-ID: <20101004172447.4EAA9282B90@codespeak.net> Author: afa Date: Mon Oct 4 19:24:45 2010 New Revision: 77581 Added: pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._io.txt (contents, props changed) pypy/branch/fast-forward/pypy/module/_io/ (props changed) pypy/branch/fast-forward/pypy/module/_io/__init__.py (contents, props changed) pypy/branch/fast-forward/pypy/module/_io/interp_io.py (contents, props changed) pypy/branch/fast-forward/pypy/module/_io/test/ (props changed) pypy/branch/fast-forward/pypy/module/_io/test/test_io.py (contents, props changed) Log: Add minimal stub of a _io module, enough to "import io" though. Added: pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._io.txt ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._io.txt Mon Oct 4 19:24:45 2010 @@ -0,0 +1,2 @@ +Use the '_io module. +Used by the 'io' standard lib module. This module is expected to be working and is included by default. Added: pypy/branch/fast-forward/pypy/module/_io/__init__.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_io/__init__.py Mon Oct 4 19:24:45 2010 @@ -0,0 +1,30 @@ +from pypy.interpreter.mixedmodule import MixedModule +import sys + +class Module(MixedModule): + + appleveldefs = { + } + + interpleveldefs = { + 'DEFAULT_BUFFER_SIZE': 'space.wrap(interp_io.DEFAULT_BUFFER_SIZE)', + 'BlockingIOError': 'interp_io.W_BlockingIOError', + '_IOBase': 'interp_io.W_IOBase', + '_RawIOBase': 'interp_io.W_RawIOBase', + '_BufferedIOBase': 'interp_io.W_BufferedIOBase', + '_TextIOBase': 'interp_io.W_TextIOBase', + + 'FileIO': 'interp_io.W_FileIO', + 'BytesIO': 'interp_io.W_BytesIO', + 'StringIO': 'interp_io.W_StringIO', + 'BufferedReader': 'interp_io.W_BufferedReader', + 'BufferedWriter': 'interp_io.W_BufferedWriter', + 'BufferedRWPair': 'interp_io.W_BufferedRWPair', + 'BufferedRandom': 'interp_io.W_BufferedRandom', + 'TextIOWrapper': 'interp_io.W_TextIOWrapper', + } + + def startup(self, space): + for name in """UnsupportedOperation open IncrementalNewlineDecoder + """.split(): + space.setattr(self, space.wrap(name), space.w_None) Added: pypy/branch/fast-forward/pypy/module/_io/interp_io.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py Mon Oct 4 19:24:45 2010 @@ -0,0 +1,94 @@ +from pypy.interpreter.baseobjspace import ObjSpace, Wrappable +from pypy.interpreter.typedef import TypeDef, interp_attrproperty +from pypy.module.exceptions.interp_exceptions import W_IOError + +DEFAULT_BUFFER_SIZE = 8192 + + +class W_BlockingIOError(W_IOError): + def __init__(self, space): + W_IOError.__init__(self, space) + self.written = 0 + + def descr_init(self, space, w_errno, w_strerror, written=0): + W_IOError.descr_init(self, space, [w_errno, w_strerror]) + self.written = written + +W_BlockingIOError.typedef = TypeDef( + 'BlockingIOError', + __doc__ = ("Exception raised when I/O would block " + "on a non-blocking I/O stream"), + characters_written = interp_attrproperty('written', W_BlockingIOError), + ) + +class W_IOBase(Wrappable): + pass +W_IOBase.typedef = TypeDef( + '_IOBase', + ) + +class W_RawIOBase(W_IOBase): + pass +W_RawIOBase.typedef = TypeDef( + '_RawIOBase', W_IOBase.typedef, + ) + +class W_BufferedIOBase(W_IOBase): + pass +W_BufferedIOBase.typedef = TypeDef( + '_BufferedIOBase', W_IOBase.typedef, + ) + +class W_TextIOBase(W_IOBase): + pass +W_TextIOBase.typedef = TypeDef( + '_TextIOBase', W_IOBase.typedef, + ) + +class W_FileIO(W_RawIOBase): + pass +W_FileIO.typedef = TypeDef( + 'FileIO', W_RawIOBase.typedef, + ) + +class W_BytesIO(W_BufferedIOBase): + pass +W_BytesIO.typedef = TypeDef( + 'BytesIO', W_BufferedIOBase.typedef, + ) + +class W_StringIO(W_TextIOBase): + pass +W_StringIO.typedef = TypeDef( + 'StringIO', W_TextIOBase.typedef, + ) + +class W_BufferedReader(W_BufferedIOBase): + pass +W_BufferedReader.typedef = TypeDef( + 'BufferedReader', W_BufferedIOBase.typedef, + ) + +class W_BufferedWriter(W_BufferedIOBase): + pass +W_BufferedWriter.typedef = TypeDef( + 'BufferedWriter', W_BufferedIOBase.typedef, + ) + +class W_BufferedRWPair(W_BufferedIOBase): + pass +W_BufferedRWPair.typedef = TypeDef( + 'BufferedRWPair', W_BufferedIOBase.typedef, + ) + +class W_BufferedRandom(W_BufferedIOBase): + pass +W_BufferedRandom.typedef = TypeDef( + 'BufferedRandom', W_BufferedIOBase.typedef, + ) + +class W_TextIOWrapper(W_TextIOBase): + pass +W_TextIOWrapper.typedef = TypeDef( + 'TextIOWrapper', W_TextIOBase.typedef, + ) Added: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py Mon Oct 4 19:24:45 2010 @@ -0,0 +1,8 @@ +from pypy.conftest import gettestobjspace + +class AppTestIoModule: + def setup_class(cls): + cls.space = gettestobjspace(usemodules=['_io']) + + def test_import(self): + import io From afa at codespeak.net Mon Oct 4 19:27:35 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 19:27:35 +0200 (CEST) Subject: [pypy-svn] r77582 - pypy/branch/fast-forward/pypy/config Message-ID: <20101004172735.DE62136C225@codespeak.net> Author: afa Date: Mon Oct 4 19:27:34 2010 New Revision: 77582 Modified: pypy/branch/fast-forward/pypy/config/pypyoption.py Log: Translation should use the _io module by default Modified: pypy/branch/fast-forward/pypy/config/pypyoption.py ============================================================================== --- pypy/branch/fast-forward/pypy/config/pypyoption.py (original) +++ pypy/branch/fast-forward/pypy/config/pypyoption.py Mon Oct 4 19:27:34 2010 @@ -19,7 +19,7 @@ default_modules.update(dict.fromkeys( ["_codecs", "gc", "_weakref", "marshal", "errno", "imp", "math", "_sre", "_pickle_support", "operator", - "parser", "symbol", "token", "_ast", "_random", "__pypy__", + "parser", "symbol", "token", "_ast", "_io", "_random", "__pypy__", "_testing"])) From afa at codespeak.net Mon Oct 4 19:34:22 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 19:34:22 +0200 (CEST) Subject: [pypy-svn] r77583 - pypy/branch/fast-forward/pypy/module/_io Message-ID: <20101004173422.0B35B36C225@codespeak.net> Author: afa Date: Mon Oct 4 19:34:21 2010 New Revision: 77583 Modified: pypy/branch/fast-forward/pypy/module/_io/__init__.py Log: Try to fix translation Modified: pypy/branch/fast-forward/pypy/module/_io/__init__.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/__init__.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/__init__.py Mon Oct 4 19:34:21 2010 @@ -22,9 +22,8 @@ 'BufferedRWPair': 'interp_io.W_BufferedRWPair', 'BufferedRandom': 'interp_io.W_BufferedRandom', 'TextIOWrapper': 'interp_io.W_TextIOWrapper', - } - def startup(self, space): - for name in """UnsupportedOperation open IncrementalNewlineDecoder - """.split(): - space.setattr(self, space.wrap(name), space.w_None) + 'open': 'space.w_None', + 'UnsupportedOperation': 'space.w_None', + 'IncrementalNewlineDecoder': 'space.w_None', + } From afa at codespeak.net Mon Oct 4 20:02:10 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 20:02:10 +0200 (CEST) Subject: [pypy-svn] r77584 - pypy/branch/fast-forward/pypy/module/_io Message-ID: <20101004180210.B6552282B90@codespeak.net> Author: afa Date: Mon Oct 4 20:02:09 2010 New Revision: 77584 Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py Log: more translation fixes Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/interp_io.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py Mon Oct 4 20:02:09 2010 @@ -1,5 +1,6 @@ -from pypy.interpreter.baseobjspace import ObjSpace, Wrappable +from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root from pypy.interpreter.typedef import TypeDef, interp_attrproperty +from pypy.interpreter.gateway import interp2app, Arguments from pypy.module.exceptions.interp_exceptions import W_IOError DEFAULT_BUFFER_SIZE = 8192 @@ -10,14 +11,23 @@ W_IOError.__init__(self, space) self.written = 0 + def descr_new(space, w_subtype, __args__): + self = space.allocate_instance(W_BlockingIOError, w_subtype) + W_BlockingIOError.__init__(self, space) + return space.wrap(self) + descr_new.unwrap_spec = [ObjSpace, W_Root, Arguments] + def descr_init(self, space, w_errno, w_strerror, written=0): W_IOError.descr_init(self, space, [w_errno, w_strerror]) self.written = written + descr_init.unwrap_spec = ['self', ObjSpace, W_Root, W_Root, int] W_BlockingIOError.typedef = TypeDef( 'BlockingIOError', __doc__ = ("Exception raised when I/O would block " "on a non-blocking I/O stream"), + __new__ = interp2app(W_BlockingIOError.descr_new.im_func), + __init__ = interp2app(W_BlockingIOError.descr_init), characters_written = interp_attrproperty('written', W_BlockingIOError), ) From afa at codespeak.net Mon Oct 4 20:51:56 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 20:51:56 +0200 (CEST) Subject: [pypy-svn] r77585 - pypy/branch/fast-forward/pypy/rlib/test Message-ID: <20101004185156.DD046282B90@codespeak.net> Author: afa Date: Mon Oct 4 20:51:55 2010 New Revision: 77585 Modified: pypy/branch/fast-forward/pypy/rlib/test/test_rstring.py Log: Fix test now that UnicodeBuilder has been made stricter Modified: pypy/branch/fast-forward/pypy/rlib/test/test_rstring.py ============================================================================== --- pypy/branch/fast-forward/pypy/rlib/test/test_rstring.py (original) +++ pypy/branch/fast-forward/pypy/rlib/test/test_rstring.py Mon Oct 4 20:51:55 2010 @@ -39,7 +39,7 @@ s.append(u'a') s.append(u'abc') s.append_slice(u'abcdef', 1, 2) - s.append_multiple_char('d', 4) + s.append_multiple_char(u'd', 4) assert s.build() == 'aabcbdddd' assert isinstance(s.build(), unicode) From dan at codespeak.net Mon Oct 4 22:07:47 2010 From: dan at codespeak.net (dan at codespeak.net) Date: Mon, 4 Oct 2010 22:07:47 +0200 (CEST) Subject: [pypy-svn] r77586 - pypy/trunk/pypy/doc Message-ID: <20101004200747.BD1FD282B90@codespeak.net> Author: dan Date: Mon Oct 4 22:07:44 2010 New Revision: 77586 Modified: pypy/trunk/pypy/doc/docindex.txt Log: Fixed typo in docindex (Thanks ArneBab) Modified: pypy/trunk/pypy/doc/docindex.txt ============================================================================== --- pypy/trunk/pypy/doc/docindex.txt (original) +++ pypy/trunk/pypy/doc/docindex.txt Mon Oct 4 22:07:44 2010 @@ -84,7 +84,7 @@ PyPy's own tests `summary`_, daily updated, run through BuildBot infrastructure. You can also find CPython's compliance tests run with compiled ``pypy-c`` -exeuctables there. +executables there. information dating from early 2007: From afa at codespeak.net Mon Oct 4 23:11:00 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 23:11:00 +0200 (CEST) Subject: [pypy-svn] r77587 - in pypy/branch/fast-forward/pypy/module/_io: . test Message-ID: <20101004211100.C95DD282B90@codespeak.net> Author: afa Date: Mon Oct 4 23:10:58 2010 New Revision: 77587 Added: pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py pypy/branch/fast-forward/pypy/module/_io/test/test_stringio.py Modified: pypy/branch/fast-forward/pypy/module/_io/__init__.py pypy/branch/fast-forward/pypy/module/_io/interp_io.py Log: Start to implement io.StringIO, enough for py/_plugin/pytest_helpconfig.py to use it. Modified: pypy/branch/fast-forward/pypy/module/_io/__init__.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/__init__.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/__init__.py Mon Oct 4 23:10:58 2010 @@ -16,7 +16,7 @@ 'FileIO': 'interp_io.W_FileIO', 'BytesIO': 'interp_io.W_BytesIO', - 'StringIO': 'interp_io.W_StringIO', + 'StringIO': 'interp_stringio.W_StringIO', 'BufferedReader': 'interp_io.W_BufferedReader', 'BufferedWriter': 'interp_io.W_BufferedWriter', 'BufferedRWPair': 'interp_io.W_BufferedRWPair', Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/interp_io.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py Mon Oct 4 23:10:58 2010 @@ -67,12 +67,6 @@ 'BytesIO', W_BufferedIOBase.typedef, ) -class W_StringIO(W_TextIOBase): - pass -W_StringIO.typedef = TypeDef( - 'StringIO', W_TextIOBase.typedef, - ) - class W_BufferedReader(W_BufferedIOBase): pass W_BufferedReader.typedef = TypeDef( Added: pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py Mon Oct 4 23:10:58 2010 @@ -0,0 +1,68 @@ +from pypy.module._io.interp_io import W_TextIOBase +from pypy.interpreter.typedef import TypeDef +from pypy.interpreter.gateway import interp2app, unwrap_spec +from pypy.interpreter.error import operationerrfmt +from pypy.interpreter.baseobjspace import ObjSpace, W_Root + +class W_StringIO(W_TextIOBase): + def __init__(self): + self.buf = [] + self.pos = 0 + + def _check_closed(self): + pass + def _check_initialized(self): + pass + + @unwrap_spec(ObjSpace, W_Root) + def descr_new(space, w_subtype): + self = space.allocate_instance(W_StringIO, w_subtype) + W_StringIO.__init__(self) + return space.wrap(self) + + def resize_buffer(self, newlength): + if len(self.buf) > newlength: + self.buf = self.buf[:newlength] + if len(self.buf) < newlength: + self.buf.extend([u'\0'] * (newlength - len(self.buf))) + + def write(self, string): + # XXX self.decoder + decoded = string + # XXX writenl + + length = len(decoded) + if self.pos + length > len(self.buf): + self.resize_buffer(self.pos + length) + + for i in range(length): + self.buf[self.pos + i] = string[i] + self.pos += length + + @unwrap_spec('self', ObjSpace, W_Root) + def write_w(self, space, w_obj): + self._check_initialized() + if not space.isinstance_w(w_obj, space.w_unicode): + raise operationerrfmt(space.w_TypeError, + "string argument expected, got '%s'", + space.type(self).getname(space, '?')) + self._check_closed() + string = space.unicode_w(w_obj) + size = len(string) + if size: + self.write(string) + return space.wrap(size) + + @unwrap_spec('self', ObjSpace) + def getvalue_w(self, space): + self._check_initialized() + self._check_closed() + return space.wrap(u''.join(self.buf)) + +W_StringIO.typedef = TypeDef( + 'StringIO', W_TextIOBase.typedef, + __new__ = interp2app(W_StringIO.descr_new.im_func), + write=interp2app(W_StringIO.write_w), + getvalue=interp2app(W_StringIO.getvalue_w), + ) + Added: pypy/branch/fast-forward/pypy/module/_io/test/test_stringio.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_io/test/test_stringio.py Mon Oct 4 23:10:58 2010 @@ -0,0 +1,7 @@ +class AppTestStringIO: + def test_stringio(self): + import io + sio = io.StringIO() + sio.write(u'Hello ') + sio.write(u'world') + assert sio.getvalue() == u'Hello world' From afa at codespeak.net Mon Oct 4 23:11:50 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 4 Oct 2010 23:11:50 +0200 (CEST) Subject: [pypy-svn] r77588 - in pypy/branch/fast-forward/pypy/module: _io _io/test _multiprocessing/test Message-ID: <20101004211150.4813C282B90@codespeak.net> Author: afa Date: Mon Oct 4 23:11:48 2010 New Revision: 77588 Modified: pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py (contents, props changed) pypy/branch/fast-forward/pypy/module/_io/test/test_stringio.py (contents, props changed) pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py (contents, props changed) Log: fixeol Modified: pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py Mon Oct 4 23:11:48 2010 @@ -1,68 +1,68 @@ -from pypy.module._io.interp_io import W_TextIOBase -from pypy.interpreter.typedef import TypeDef -from pypy.interpreter.gateway import interp2app, unwrap_spec -from pypy.interpreter.error import operationerrfmt -from pypy.interpreter.baseobjspace import ObjSpace, W_Root - -class W_StringIO(W_TextIOBase): - def __init__(self): - self.buf = [] - self.pos = 0 - - def _check_closed(self): - pass - def _check_initialized(self): - pass - - @unwrap_spec(ObjSpace, W_Root) - def descr_new(space, w_subtype): - self = space.allocate_instance(W_StringIO, w_subtype) - W_StringIO.__init__(self) - return space.wrap(self) - - def resize_buffer(self, newlength): - if len(self.buf) > newlength: - self.buf = self.buf[:newlength] - if len(self.buf) < newlength: - self.buf.extend([u'\0'] * (newlength - len(self.buf))) - - def write(self, string): - # XXX self.decoder - decoded = string - # XXX writenl - - length = len(decoded) - if self.pos + length > len(self.buf): - self.resize_buffer(self.pos + length) - - for i in range(length): - self.buf[self.pos + i] = string[i] - self.pos += length - - @unwrap_spec('self', ObjSpace, W_Root) - def write_w(self, space, w_obj): - self._check_initialized() - if not space.isinstance_w(w_obj, space.w_unicode): - raise operationerrfmt(space.w_TypeError, - "string argument expected, got '%s'", - space.type(self).getname(space, '?')) - self._check_closed() - string = space.unicode_w(w_obj) - size = len(string) - if size: - self.write(string) - return space.wrap(size) - - @unwrap_spec('self', ObjSpace) - def getvalue_w(self, space): - self._check_initialized() - self._check_closed() - return space.wrap(u''.join(self.buf)) - -W_StringIO.typedef = TypeDef( - 'StringIO', W_TextIOBase.typedef, - __new__ = interp2app(W_StringIO.descr_new.im_func), - write=interp2app(W_StringIO.write_w), - getvalue=interp2app(W_StringIO.getvalue_w), - ) - +from pypy.module._io.interp_io import W_TextIOBase +from pypy.interpreter.typedef import TypeDef +from pypy.interpreter.gateway import interp2app, unwrap_spec +from pypy.interpreter.error import operationerrfmt +from pypy.interpreter.baseobjspace import ObjSpace, W_Root + +class W_StringIO(W_TextIOBase): + def __init__(self): + self.buf = [] + self.pos = 0 + + def _check_closed(self): + pass + def _check_initialized(self): + pass + + @unwrap_spec(ObjSpace, W_Root) + def descr_new(space, w_subtype): + self = space.allocate_instance(W_StringIO, w_subtype) + W_StringIO.__init__(self) + return space.wrap(self) + + def resize_buffer(self, newlength): + if len(self.buf) > newlength: + self.buf = self.buf[:newlength] + if len(self.buf) < newlength: + self.buf.extend([u'\0'] * (newlength - len(self.buf))) + + def write(self, string): + # XXX self.decoder + decoded = string + # XXX writenl + + length = len(decoded) + if self.pos + length > len(self.buf): + self.resize_buffer(self.pos + length) + + for i in range(length): + self.buf[self.pos + i] = string[i] + self.pos += length + + @unwrap_spec('self', ObjSpace, W_Root) + def write_w(self, space, w_obj): + self._check_initialized() + if not space.isinstance_w(w_obj, space.w_unicode): + raise operationerrfmt(space.w_TypeError, + "string argument expected, got '%s'", + space.type(self).getname(space, '?')) + self._check_closed() + string = space.unicode_w(w_obj) + size = len(string) + if size: + self.write(string) + return space.wrap(size) + + @unwrap_spec('self', ObjSpace) + def getvalue_w(self, space): + self._check_initialized() + self._check_closed() + return space.wrap(u''.join(self.buf)) + +W_StringIO.typedef = TypeDef( + 'StringIO', W_TextIOBase.typedef, + __new__ = interp2app(W_StringIO.descr_new.im_func), + write=interp2app(W_StringIO.write_w), + getvalue=interp2app(W_StringIO.getvalue_w), + ) + Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_stringio.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/test/test_stringio.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/test/test_stringio.py Mon Oct 4 23:11:48 2010 @@ -1,7 +1,7 @@ -class AppTestStringIO: - def test_stringio(self): - import io - sio = io.StringIO() - sio.write(u'Hello ') - sio.write(u'world') - assert sio.getvalue() == u'Hello world' +class AppTestStringIO: + def test_stringio(self): + import io + sio = io.StringIO() + sio.write(u'Hello ') + sio.write(u'world') + assert sio.getvalue() == u'Hello world' Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py Mon Oct 4 23:11:48 2010 @@ -1,48 +1,48 @@ -import py -import sys -from pypy.conftest import gettestobjspace - -class TestConnection: - def test_simple(self): - from pypy.module._multiprocessing import interp_connection - -class AppTestConnection: - def setup_class(cls): - space = gettestobjspace(usemodules=('_multiprocessing', 'thread')) - cls.space = space - if sys.platform == "win32": - # stubs for some modules, - # just for multiprocessing to import correctly. - w_modules = space.sys.get('modules') - space.setitem(w_modules, space.wrap('msvcrt'), space.sys) - space.setitem(w_modules, space.wrap('_subprocess'), space.sys) - - # import multiprocessing once - space.appexec([], """(): import multiprocessing""") - - def test_winpipe_connection(self): - import sys - if sys.platform != "win32": - skip("win32 only") - - import multiprocessing - rhandle, whandle = multiprocessing.Pipe() - - obj = [1, 2.0, "hello"] - whandle.send(obj) - obj2 = rhandle.recv() - assert obj == obj2 - - def test_ospipe_connection(self): - import _multiprocessing - import os - fd1, fd2 = os.pipe() - rhandle = _multiprocessing.Connection(fd1, writable=False) - whandle = _multiprocessing.Connection(fd2, readable=False) - - obj = [1, 2.0, "hello"] - whandle.send(obj) - obj2 = rhandle.recv() - assert obj == obj2 - - +import py +import sys +from pypy.conftest import gettestobjspace + +class TestConnection: + def test_simple(self): + from pypy.module._multiprocessing import interp_connection + +class AppTestConnection: + def setup_class(cls): + space = gettestobjspace(usemodules=('_multiprocessing', 'thread')) + cls.space = space + if sys.platform == "win32": + # stubs for some modules, + # just for multiprocessing to import correctly. + w_modules = space.sys.get('modules') + space.setitem(w_modules, space.wrap('msvcrt'), space.sys) + space.setitem(w_modules, space.wrap('_subprocess'), space.sys) + + # import multiprocessing once + space.appexec([], """(): import multiprocessing""") + + def test_winpipe_connection(self): + import sys + if sys.platform != "win32": + skip("win32 only") + + import multiprocessing + rhandle, whandle = multiprocessing.Pipe() + + obj = [1, 2.0, "hello"] + whandle.send(obj) + obj2 = rhandle.recv() + assert obj == obj2 + + def test_ospipe_connection(self): + import _multiprocessing + import os + fd1, fd2 = os.pipe() + rhandle = _multiprocessing.Connection(fd1, writable=False) + whandle = _multiprocessing.Connection(fd2, readable=False) + + obj = [1, 2.0, "hello"] + whandle.send(obj) + obj2 = rhandle.recv() + assert obj == obj2 + + From fijal at codespeak.net Mon Oct 4 23:32:32 2010 From: fijal at codespeak.net (fijal at codespeak.net) Date: Mon, 4 Oct 2010 23:32:32 +0200 (CEST) Subject: [pypy-svn] r77589 - pypy/trunk/pypy/jit/metainterp Message-ID: <20101004213232.2F5A1282B90@codespeak.net> Author: fijal Date: Mon Oct 4 23:32:30 2010 New Revision: 77589 Modified: pypy/trunk/pypy/jit/metainterp/simple_optimize.py Log: CALL_LOOPINVARIANT is just a CALL Modified: pypy/trunk/pypy/jit/metainterp/simple_optimize.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/simple_optimize.py (original) +++ pypy/trunk/pypy/jit/metainterp/simple_optimize.py Mon Oct 4 23:32:30 2010 @@ -9,9 +9,10 @@ def transform(op): from pypy.jit.metainterp.history import AbstractDescr - # Rename CALL_PURE to CALL. + # Rename CALL_PURE and CALL_INVARIANT to CALL. # Simplify the VIRTUAL_REF_* so that they don't show up in the backend. - if op.getopnum() == rop.CALL_PURE: + if (op.getopnum() == rop.CALL_PURE or + op.getopnum() == rop.CALL_LOOPINVARIANT): op = ResOperation(rop.CALL, op.getarglist()[1:], op.result, op.getdescr()) elif op.getopnum() == rop.VIRTUAL_REF: From afa at codespeak.net Tue Oct 5 00:35:39 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 00:35:39 +0200 (CEST) Subject: [pypy-svn] r77590 - pypy/branch/fast-forward/pypy/module/_ssl Message-ID: <20101004223539.958B5282B90@codespeak.net> Author: afa Date: Tue Oct 5 00:35:38 2010 New Revision: 77590 Modified: pypy/branch/fast-forward/pypy/module/_ssl/__init__.py pypy/branch/fast-forward/pypy/module/_ssl/interp_ssl.py Log: Probably a very incomplete fix, but at least the tests in module/_ssl pass Modified: pypy/branch/fast-forward/pypy/module/_ssl/__init__.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_ssl/__init__.py (original) +++ pypy/branch/fast-forward/pypy/module/_ssl/__init__.py Tue Oct 5 00:35:38 2010 @@ -2,7 +2,7 @@ class Module(MixedModule): interpleveldefs = { - 'ssl': 'interp_ssl.ssl', + 'sslwrap': 'interp_ssl.sslwrap', } appleveldefs = { Modified: pypy/branch/fast-forward/pypy/module/_ssl/interp_ssl.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_ssl/interp_ssl.py (original) +++ pypy/branch/fast-forward/pypy/module/_ssl/interp_ssl.py Tue Oct 5 00:35:38 2010 @@ -359,6 +359,10 @@ return self.space.wrap(result) read.unwrap_spec = ['self', int] + def do_handshake(self): + # XXX + pass + SSLObject.typedef = TypeDef("SSLObject", server = interp2app(SSLObject.server, @@ -367,11 +371,12 @@ unwrap_spec=SSLObject.issuer.unwrap_spec), write = interp2app(SSLObject.write, unwrap_spec=SSLObject.write.unwrap_spec), - read = interp2app(SSLObject.read, unwrap_spec=SSLObject.read.unwrap_spec) + read = interp2app(SSLObject.read, unwrap_spec=SSLObject.read.unwrap_spec), + do_handshake=interp2app(SSLObject.do_handshake, unwrap_spec=['self']), ) -def new_sslobject(space, w_sock, w_key_file, w_cert_file): +def new_sslobject(space, w_sock, side, w_key_file, w_cert_file): ss = SSLObject(space) sock_fd = space.int_w(space.call_method(w_sock, "fileno")) @@ -562,8 +567,12 @@ return errstr, errval -def ssl(space, w_socket, w_key_file=None, w_cert_file=None): - """ssl(socket, [keyfile, certfile]) -> sslobject""" - return space.wrap(new_sslobject(space, w_socket, w_key_file, w_cert_file)) -ssl.unwrap_spec = [ObjSpace, W_Root, W_Root, W_Root] +def sslwrap(space, w_socket, side, w_key_file=None, w_cert_file=None, + cert_mode=PY_SSL_CERT_NONE, protocol=PY_SSL_VERSION_SSL23, + w_cacerts_file=None, w_cipher=None): + """sslwrap(socket, side, [keyfile, certfile]) -> sslobject""" + return space.wrap(new_sslobject( + space, w_socket, side, w_key_file, w_cert_file)) +sslwrap.unwrap_spec = [ObjSpace, W_Root, int, W_Root, W_Root, + int, int, W_Root, W_Root] From afa at codespeak.net Tue Oct 5 01:34:44 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 01:34:44 +0200 (CEST) Subject: [pypy-svn] r77591 - pypy/branch/fast-forward/lib-python Message-ID: <20101004233444.70EEF282B90@codespeak.net> Author: afa Date: Tue Oct 5 01:34:42 2010 New Revision: 77591 Modified: pypy/branch/fast-forward/lib-python/TODO Log: One more task, in the "easy" level. Modified: pypy/branch/fast-forward/lib-python/TODO ============================================================================== --- pypy/branch/fast-forward/lib-python/TODO (original) +++ pypy/branch/fast-forward/lib-python/TODO Tue Oct 5 01:34:42 2010 @@ -30,6 +30,8 @@ - Fix fcntl.fcntl(fd, fcntl.F_NOTIFY, fcntl.DN_MULTISHOT) on 32bit platform. +- Missing module future_builtins + Medium tasks ------------ From afa at codespeak.net Tue Oct 5 08:58:01 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 08:58:01 +0200 (CEST) Subject: [pypy-svn] r77592 - pypy/branch/fast-forward/lib-python/modified-2.7.0 Message-ID: <20101005065801.80F3A282B90@codespeak.net> Author: afa Date: Tue Oct 5 08:57:59 2010 New Revision: 77592 Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/pickle.py Log: PyPy can save references to module dictionaries. Same code as in modified-2.5.2, except for a hack needed by previous versions of the py library. This fixes a test in pypy.interpreter.test.test_zzpickle_and_slow Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/pickle.py ============================================================================== --- pypy/branch/fast-forward/lib-python/modified-2.7.0/pickle.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/pickle.py Tue Oct 5 08:57:59 2010 @@ -638,6 +638,10 @@ # else tmp is empty, and we're done def save_dict(self, obj): + modict_saver = self._pickle_moduledict(obj) + if modict_saver is not None: + return self.save_reduce(*modict_saver) + write = self.write if self.bin: @@ -687,6 +691,29 @@ write(SETITEM) # else tmp is empty, and we're done + def _pickle_moduledict(self, obj): + # save module dictionary as "getattr(module, '__dict__')" + + # build index of module dictionaries + try: + modict = self.module_dict_ids + except AttributeError: + modict = {} + from sys import modules + for mod in modules.values(): + if isinstance(mod, ModuleType): + modict[id(mod.__dict__)] = mod + self.module_dict_ids = modict + + thisid = id(obj) + try: + themodule = modict[thisid] + except KeyError: + return None + from __builtin__ import getattr + return getattr, (themodule, '__dict__') + + def save_inst(self, obj): cls = obj.__class__ From antocuni at codespeak.net Tue Oct 5 10:02:46 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 5 Oct 2010 10:02:46 +0200 (CEST) Subject: [pypy-svn] r77593 - in pypy/branch/jitffi/pypy: jit/metainterp/test rlib rlib/test Message-ID: <20101005080246.2338A36C228@codespeak.net> Author: antocuni Date: Tue Oct 5 10:02:44 2010 New Revision: 77593 Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py pypy/branch/jitffi/pypy/rlib/libffi.py pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Log: unify the float() and int() methods of ArgChain into arg(), which is specialized by the type of the argument Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py Tue Oct 5 10:02:44 2010 @@ -47,7 +47,7 @@ driver.can_enter_jit(n=n, func=func) func = hint(func, promote=True) argchain = ArgChain() - argchain.int(n).float(1.2) + argchain.arg(n).arg(1.2) n = func.call(argchain, rffi.LONG) return n @@ -73,7 +73,7 @@ driver.can_enter_jit(n=n, func=func, res=res) func = hint(func, promote=True) argchain = ArgChain() - argchain.float(float(-n)) + argchain.arg(float(-n)) res = func.call(argchain, rffi.DOUBLE) n += 1 return res @@ -95,7 +95,7 @@ driver.can_enter_jit(n=n, func=func, res=res) func = hint(func, promote=True) argchain = ArgChain() - argchain.int(0) + argchain.arg(0) res = func.call(argchain, rffi.UCHAR) n += 1 return res Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Tue Oct 5 10:02:44 2010 @@ -48,12 +48,16 @@ last = None numargs = 0 - def int(self, intval): - self._append(IntArg(intval)) - return self - - def float(self, floatval): - self._append(FloatArg(floatval)) + @specialize.argtype(1) + def arg(self, val): + TYPE = lltype.typeOf(val) + if TYPE is rffi.LONG: + cls = IntArg + elif TYPE is rffi.DOUBLE: + cls = FloatArg + else: + raise TypeError, 'Unsupported argument type: %s' % TYPE + self._append(cls(val)) return self def _append(self, arg): Modified: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Tue Oct 5 10:02:44 2010 @@ -20,13 +20,13 @@ def test_argchain(self): chain = ArgChain() assert chain.numargs == 0 - chain2 = chain.int(42) + chain2 = chain.arg(42) assert chain2 is chain assert chain.numargs == 1 intarg = chain.first assert chain.last is intarg assert intarg.intval == 42 - chain.float(123.45) + chain.arg(123.45) assert chain.numargs == 2 assert chain.first is intarg assert intarg.next is chain.last @@ -51,6 +51,6 @@ pow = libm.getpointer('pow', [types.double, types.double], types.double) argchain = ArgChain() - argchain.float(2.0).float(3.0) + argchain.arg(2.0).arg(3.0) res = pow.call(argchain, rffi.DOUBLE) assert res == 8.0 From antocuni at codespeak.net Tue Oct 5 11:21:41 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 5 Oct 2010 11:21:41 +0200 (CEST) Subject: [pypy-svn] r77594 - in pypy/branch/jitffi/pypy: jit/metainterp/test rlib/test Message-ID: <20101005092141.08E47282B90@codespeak.net> Author: antocuni Date: Tue Oct 5 11:21:39 2010 New Revision: 77594 Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Log: refactor the test to share most of the between test_libffi and test_fficall. The reason is that ffi calls now really follow two different paths when jitted or non jitted, so we need to always test both. Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py Tue Oct 5 11:21:39 2010 @@ -1,104 +1,43 @@ import py from pypy.rlib.jit import JitDriver, hint -from pypy.jit.metainterp.test.test_basic import LLJitMixin -from pypy.rlib.libffi import CDLL, types, ArgChain, Func -from pypy.tool.udir import udir -from pypy.translator.tool.cbuild import ExternalCompilationInfo -from pypy.translator.platform import platform +from pypy.rlib.unroll import unrolling_iterable +from pypy.rlib.libffi import ArgChain +from pypy.rlib.test.test_libffi import TestLibffiCall as _TestLibffiCall from pypy.rpython.lltypesystem import lltype, rffi - -class TestFfiCall(LLJitMixin): - def setup_class(cls): - # prepare C code as an example, so we can load it and call - # it via rlib.libffi - c_file = udir.ensure("test_jit_fficall", dir=1).join("xlib.c") - c_file.write(py.code.Source(''' - int sum_xy(int x, double y) - { - return (x + (int)y); - } - - float abs(double x) - { - if (x<0) - return -x; - return x; - } - - unsigned char cast_to_uchar(int x) - { - return 200+(unsigned char)x; - } - ''')) - eci = ExternalCompilationInfo(export_symbols=[]) - cls.lib_name = str(platform.compile([c_file], eci, 'x', - standalone=False)) - - def test_simple(self): - driver = JitDriver(reds = ['n', 'func'], greens = []) - - def f(n): - cdll = CDLL(self.lib_name) - func = cdll.getpointer('sum_xy', [types.sint, types.double], - types.sint) - while n < 10: - driver.jit_merge_point(n=n, func=func) - driver.can_enter_jit(n=n, func=func) - func = hint(func, promote=True) - argchain = ArgChain() - argchain.arg(n).arg(1.2) - n = func.call(argchain, rffi.LONG) - return n - - res = self.meta_interp(f, [0]) - assert res == 10 - self.check_loops({ - 'call': 1, - 'guard_no_exception': 1, - 'int_lt': 1, - 'guard_true': 1, - 'jump': 1}) +from pypy.jit.metainterp.test.test_basic import LLJitMixin - def test_float_result(self): - driver = JitDriver(reds = ['n', 'func', 'res'], greens = []) +class TestFfiCall(LLJitMixin, _TestLibffiCall): + def call(self, funcspec, args, RESULT, init_result=0): + """ + Call the function specified by funcspec in a loop, and let the jit to + see and optimize it. + """ + # + lib, name, argtypes, restype = funcspec + args = unrolling_iterable(args) + # + reds = ['n', 'res', 'func'] + if type(init_result) is float: + reds = ['n', 'func', 'res'] # floats must be *after* refs + driver = JitDriver(reds=reds, greens=[]) + # def f(n): - cdll = CDLL(self.lib_name) - func = cdll.getpointer('abs', [types.double], types.double) - res = 0.0 + func = lib.getpointer(name, argtypes, restype) + res = init_result while n < 10: - driver.jit_merge_point(n=n, func=func, res=res) - driver.can_enter_jit(n=n, func=func, res=res) + driver.jit_merge_point(n=n, res=res, func=func) + driver.can_enter_jit(n=n, res=res, func=func) func = hint(func, promote=True) argchain = ArgChain() - argchain.arg(float(-n)) - res = func.call(argchain, rffi.DOUBLE) + for argval in args: # this loop is unrolled + argchain.arg(argval) + res = func.call(argchain, RESULT) n += 1 return res - + # res = self.meta_interp(f, [0]) - assert res == 9 - self.check_loops(call=1) - - def test_cast_result(self): - driver = JitDriver(reds = ['n', 'res', 'func'], greens = []) + return res - def f(n): - cdll = CDLL(self.lib_name) - func = cdll.getpointer('cast_to_uchar', [types.sint], - types.uchar) - res = 0 - while n < 10: - driver.jit_merge_point(n=n, func=func, res=res) - driver.can_enter_jit(n=n, func=func, res=res) - func = hint(func, promote=True) - argchain = ArgChain() - argchain.arg(0) - res = func.call(argchain, rffi.UCHAR) - n += 1 - return res - - res = self.meta_interp(f, [0]) - assert res == 200 Modified: pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py Tue Oct 5 11:21:39 2010 @@ -20,7 +20,10 @@ return 'libm.so' class BaseFfiTest(object): - + + CDLL = None # overridden by subclasses + + @classmethod def setup_class(cls): for name in type_names: # XXX force this to be seen by ll2ctypes @@ -31,14 +34,17 @@ def setup_method(self, meth): ALLOCATED.clear() - -class TestCLibffi(BaseFfiTest): - def get_libc(self): - return CDLL(get_libc_name()) + return self.CDLL(get_libc_name()) def get_libm(self): - return CDLL(get_libm_name(sys.platform)) + return self.CDLL(get_libm_name(sys.platform)) + + + +class TestCLibffi(BaseFfiTest): + + CDLL = CDLL def test_library_open(self): lib = self.get_libc() Modified: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Tue Oct 5 11:21:39 2010 @@ -5,17 +5,9 @@ from pypy.rlib.test.test_clibffi import BaseFfiTest, get_libm_name from pypy.rlib.libffi import CDLL, Func, get_libc_name, ArgChain, types +class TestLibffiMisc(BaseFfiTest): -class TestLibffi(BaseFfiTest): - """ - Test the new JIT-friendly interface to libffi - """ - - def get_libc(self): - return CDLL(get_libc_name()) - - def get_libm(self): - return CDLL(get_libm_name(sys.platform)) + CDLL = CDLL def test_argchain(self): chain = ArgChain() @@ -46,11 +38,100 @@ del lib assert not ALLOCATED - def test_call_argchain(self): + +class TestLibffiCall(BaseFfiTest): + """ + Test various kind of calls through libffi. + + The peculiarity of these tests is that they are run both directly (going + really through libffi) and by jit/metainterp/test/test_fficall.py, which + tests the call when JITted. + + If you need to test a behaviour than it's not affected by JITing (e.g., + typechecking), you should put your test in TestLibffiMisc. + """ + + CDLL = CDLL + + @classmethod + def setup_class(cls): + from pypy.tool.udir import udir + from pypy.translator.tool.cbuild import ExternalCompilationInfo + from pypy.translator.platform import platform + + BaseFfiTest.setup_class() + # prepare C code as an example, so we can load it and call + # it via rlib.libffi + c_file = udir.ensure("test_libffi", dir=1).join("foolib.c") + c_file.write(py.code.Source(''' + int sum_xy(int x, double y) + { + return (x + (int)y); + } + + unsigned char cast_to_uchar_and_ovf(int x) + { + return 200+(unsigned char)x; + } + ''')) + eci = ExternalCompilationInfo(export_symbols=[]) + cls.libfoo_name = str(platform.compile([c_file], eci, 'x', + standalone=False)) + + def get_libfoo(self): + return self.CDLL(self.libfoo_name) + + def call(self, funcspec, args, RESULT, init_result=0): + """ + Call the specified function after constructing and ArgChain with the + arguments in ``args``. + + The function is specified with ``funcspec``, which is a tuple of the + form (lib, name, argtypes, restype). + + This method is overridden by metainterp/test/test_fficall.py in + order to do the call in a loop and JIT it. The optional arguments are + used only by that overridden method. + + """ + lib, name, argtypes, restype = funcspec + func = lib.getpointer(name, argtypes, restype) + chain = ArgChain() + for arg in args: + chain.arg(arg) + return func.call(chain, RESULT) + + def check_loops(self, *args, **kwds): + """ + Ignored here, but does something in the JIT tests + """ + pass + + # ------------------------------------------------------------------------ + + def test_simple(self): + libfoo = self.get_libfoo() + func = (libfoo, 'sum_xy', [types.sint, types.double], types.sint) + res = self.call(func, [38, 4.2], rffi.LONG) + assert res == 42 + self.check_loops({ + 'call': 1, + 'guard_no_exception': 1, + 'int_add': 1, + 'int_lt': 1, + 'guard_true': 1, + 'jump': 1}) + + def test_float_result(self): libm = self.get_libm() - pow = libm.getpointer('pow', [types.double, types.double], - types.double) - argchain = ArgChain() - argchain.arg(2.0).arg(3.0) - res = pow.call(argchain, rffi.DOUBLE) + func = (libm, 'pow', [types.double, types.double], types.double) + res = self.call(func, [2.0, 3.0], rffi.DOUBLE, init_result=0.0) assert res == 8.0 + self.check_loops(call=1) + + def test_cast_result(self): + libfoo = self.get_libfoo() + func = (libfoo, 'cast_to_uchar_and_ovf', [types.sint], types.uchar) + res = self.call(func, [0], rffi.UCHAR) + assert res == 200 + self.check_loops(call=1) From fijal at codespeak.net Tue Oct 5 11:47:37 2010 From: fijal at codespeak.net (fijal at codespeak.net) Date: Tue, 5 Oct 2010 11:47:37 +0200 (CEST) Subject: [pypy-svn] r77595 - pypy/trunk/pypy/jit/metainterp/optimizeopt Message-ID: <20101005094737.1290F36C228@codespeak.net> Author: fijal Date: Tue Oct 5 11:47:36 2010 New Revision: 77595 Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py Log: Write down a comment that this code is dead Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py (original) +++ pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py Tue Oct 5 11:47:36 2010 @@ -245,6 +245,9 @@ def optimize_CALL_LOOPINVARIANT(self, op): funcvalue = self.getvalue(op.getarg(0)) if not funcvalue.is_constant(): + # XXX this code path is never executed in tests nor in production. + # in fact, it can't even happen since residual_call in codewriter + # expects a compile-time constant self.emit_operation(op) return key = make_hashable_int(op.getarg(0).getint()) From fijal at codespeak.net Tue Oct 5 11:47:56 2010 From: fijal at codespeak.net (fijal at codespeak.net) Date: Tue, 5 Oct 2010 11:47:56 +0200 (CEST) Subject: [pypy-svn] r77596 - pypy/trunk/pypy/jit/metainterp Message-ID: <20101005094756.7D19B36C228@codespeak.net> Author: fijal Date: Tue Oct 5 11:47:55 2010 New Revision: 77596 Modified: pypy/trunk/pypy/jit/metainterp/simple_optimize.py Log: CALL_LOOPINVARIANT is a normal call (unlike CALL_PURE) Modified: pypy/trunk/pypy/jit/metainterp/simple_optimize.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/simple_optimize.py (original) +++ pypy/trunk/pypy/jit/metainterp/simple_optimize.py Tue Oct 5 11:47:55 2010 @@ -9,12 +9,13 @@ def transform(op): from pypy.jit.metainterp.history import AbstractDescr - # Rename CALL_PURE and CALL_INVARIANT to CALL. + # Rename CALL_PURE and CALL_LOOPINVARIANT to CALL. # Simplify the VIRTUAL_REF_* so that they don't show up in the backend. - if (op.getopnum() == rop.CALL_PURE or - op.getopnum() == rop.CALL_LOOPINVARIANT): + if op.getopnum() == rop.CALL_PURE: op = ResOperation(rop.CALL, op.getarglist()[1:], op.result, op.getdescr()) + elif op.getopnum() == rop.CALL_LOOPINVARIANT: + op = op.copy_and_change(rop.CALL) elif op.getopnum() == rop.VIRTUAL_REF: op = ResOperation(rop.SAME_AS, [op.getarg(0)], op.result) elif op.getopnum() == rop.VIRTUAL_REF_FINISH: From arigo at codespeak.net Tue Oct 5 13:09:42 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 13:09:42 +0200 (CEST) Subject: [pypy-svn] r77597 - in pypy/branch/32ptr-on-64bit/pypy: config rlib rpython/lltypesystem rpython/memory rpython/memory/gc rpython/memory/gc/test rpython/memory/gctransform translator/c Message-ID: <20101005110942.5451B282B90@codespeak.net> Author: arigo Date: Tue Oct 5 13:09:40 2010 New Revision: 77597 Added: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py - copied, changed from r77571, pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/test/test_minimarkpage2.py - copied, changed from r77571, pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/test/test_minimarkpage.py Modified: pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py Log: Random progress. Modified: pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py Tue Oct 5 13:09:40 2010 @@ -196,7 +196,8 @@ BoolOption("compressptr", "Compress pointers; limits the program to 32GB", default=False, cmdline="--compressptr", requires=[("translation.type_system", "lltype"), - ("translation.taggedpointers", False)] + ("translation.taggedpointers", False), + ("translation.gc", "minimark")] # for now + [("compressptr (64-bit only)", True)]*(not IS_64_BITS)), # options for ootype Modified: pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py Tue Oct 5 13:09:40 2010 @@ -50,7 +50,7 @@ constant_names = ['MAP_SHARED', 'MAP_PRIVATE', 'PROT_READ', 'PROT_WRITE', 'MS_SYNC'] - opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS', + opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS', 'MAP_FIXED', 'PROT_EXEC', 'MAP_DENYWRITE', 'MAP_EXECUTABLE'] for name in constant_names: Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py Tue Oct 5 13:09:40 2010 @@ -1154,16 +1154,14 @@ return hop.genop('direct_ptradd', [v_ptr, v_n], resulttype = v_ptr.concretetype) -class _lladdress(long): +class _lladdress(object): _TYPE = llmemory.Address - def __new__(cls, void_p): + def __init__(self, void_p): if isinstance(void_p, (int, long)): void_p = ctypes.c_void_p(void_p) - self = long.__new__(cls, void_p.value) self.void_p = void_p self.intval = intmask(void_p.value) - return self def _cast_to_ptr(self, TP): return force_cast(TP, self.intval) @@ -1179,6 +1177,12 @@ def __ne__(self, other): return not self == other + def __add__(self, other): + return _lladdress(self.intval + other) + + def __sub__(self, other): + return _lladdress(self.intval - other) + class _llgcopaque(lltype._container): _TYPE = llmemory.GCREF.TO _name = "_llgcopaque" Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py Tue Oct 5 13:09:40 2010 @@ -7,6 +7,13 @@ def get_compressed_gcref_repr(rtyper, baserepr): + # Return either the original baserepr, or another repr standing for + # a HiddenGcRef32. The idea is that we only get a HiddenGcRef32 for + # fixed-sized structures (XXX that are not too big); thus this is only + # for structures that gets allocated by the minimarkpage2 mmap()- + # within-32GB-of-RAM. + if baserepr.lowleveltype.TO._is_varsize(): + return baserepr try: comprmgr = rtyper.compressed_gcref_manager except AttributeError: Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py Tue Oct 5 13:09:40 2010 @@ -50,7 +50,7 @@ # collection. It is automatically set to True by test_gc.py. The # checking logic is translatable, so the flag can be set to True # here before translation. - DEBUG = False + DEBUG = True def set_query_functions(self, is_varsize, has_gcptr_in_varsize, is_gcarrayofgcptr, Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py Tue Oct 5 13:09:40 2010 @@ -3,7 +3,7 @@ from pypy.rpython.lltypesystem.lloperation import llop from pypy.rpython.lltypesystem.llmemory import raw_malloc_usage from pypy.rpython.memory.gc.base import GCBase, MovingGCBase -from pypy.rpython.memory.gc import minimarkpage, base, generation +from pypy.rpython.memory.gc import base, generation from pypy.rlib.rarithmetic import ovfcheck, LONG_BIT, intmask, r_uint from pypy.rlib.rarithmetic import LONG_BIT_SHIFT from pypy.rlib.debug import ll_assert, debug_print, debug_start, debug_stop @@ -102,11 +102,12 @@ # The system page size. Like obmalloc.c, we assume that it is 4K # for 32-bit systems; unlike obmalloc.c, we assume that it is 8K - # for 64-bit systems, for consistent results. + # for 64-bit systems, for consistent results. (ignored if we + # use minimarkpage2.py) "page_size": 1024*WORD, # The size of an arena. Arenas are groups of pages allocated - # together. + # together. (ignored if we use minimarkpage2.py) "arena_size": 65536*WORD, # The maximum size of an object allocated compactly. All objects @@ -181,7 +182,14 @@ # # The ArenaCollection() handles the nonmovable objects allocation. if ArenaCollectionClass is None: - ArenaCollectionClass = minimarkpage.ArenaCollection + if self.translated_to_c and self.config.compressptr: + from pypy.rpython.memory.gc import minimarkpage2 + ArenaCollectionClass = minimarkpage2.ArenaCollection2 + arena_size = minimarkpage2.ARENA_SIZE + page_size = 4096 + else: + from pypy.rpython.memory.gc import minimarkpage + ArenaCollectionClass = minimarkpage.ArenaCollection self.ac = ArenaCollectionClass(arena_size, page_size, small_request_threshold) # @@ -267,7 +275,7 @@ self.max_heap_size = float(max_heap_size) # self.minor_collection() # to empty the nursery - llarena.arena_free(self.nursery) + self.ac.free_big_chunk(self.nursery) self.nursery_size = newsize self.allocate_nursery() @@ -280,9 +288,11 @@ # in malloc_fixedsize_clear(). The few extra pages are never used # anyway so it doesn't even count. extra = self.nonlarge_gcptrs_max + 1 - self.nursery = llarena.arena_malloc(self.nursery_size + extra, 2) + fullsize = self.nursery_size + extra + self.nursery = self.ac.allocate_big_chunk(fullsize) if not self.nursery: raise MemoryError("cannot allocate nursery") + llarena.arena_reset(self.nursery, fullsize, 2) # the current position in the nursery: self.nursery_free = self.nursery # the end of the nursery: @@ -1526,6 +1536,12 @@ self.all_objects = [] self.total_memory_used = 0 + def allocate_big_chunk(self, arena_size): + return llarena.arena_malloc(arena_size, False) + + def free_big_chunk(self, arena): + llarena.arena_free(arena) + def malloc(self, size): nsize = raw_malloc_usage(size) ll_assert(nsize > 0, "malloc: size is null or negative") Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py Tue Oct 5 13:09:40 2010 @@ -138,6 +138,13 @@ self.total_memory_used = r_uint(0) + def allocate_big_chunk(self, arena_size): + return llarena.arena_malloc(arena_size, False) + + def free_big_chunk(self, arena): + llarena.arena_free(arena) + + def malloc(self, size): """Allocate a block from a page in an arena.""" nsize = llmemory.raw_malloc_usage(size) Copied: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py (from r77571, pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py) ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py Tue Oct 5 13:09:40 2010 @@ -2,6 +2,7 @@ from pypy.rlib.rarithmetic import LONG_BIT, r_uint from pypy.rlib.objectmodel import we_are_translated from pypy.rlib.debug import ll_assert +from pypy.rlib import rmmap WORD = LONG_BIT // 8 NULL = llmemory.NULL @@ -13,32 +14,12 @@ # A page contains a number of allocated objects, called "blocks". # The actual allocation occurs in whole arenas, which are then subdivided -# into pages. For each arena we allocate one of the following structures: +# into pages. Arenas are allocated (after translation to C) as an mmap() +# at fixed addresses: -ARENA_PTR = lltype.Ptr(lltype.ForwardReference()) -ARENA = lltype.Struct('ArenaReference', - # -- The address of the arena, as returned by malloc() - ('base', llmemory.Address), - # -- The number of free and the total number of pages in the arena - ('nfreepages', lltype.Signed), - ('totalpages', lltype.Signed), - # -- A chained list of free pages in the arena. Ends with NULL. - ('freepages', llmemory.Address), - # -- A linked list of arenas. See below. - ('nextarena', ARENA_PTR), - ) -ARENA_PTR.TO.become(ARENA) -ARENA_NULL = lltype.nullptr(ARENA) - -# The idea is that when we need a free page, we take it from the arena -# which currently has the *lowest* number of free pages. This allows -# arenas with a lot of free pages to eventually become entirely free, at -# which point they are returned to the OS. If an arena has a total of -# 64 pages, then we have 64 global lists, arenas_lists[0] to -# arenas_lists[63], such that arenas_lists[i] contains exactly those -# arenas that have 'nfreepages == i'. We allocate pages out of the -# arena in 'current_arena'; when it is exhausted we pick another arena -# with the smallest value for nfreepages (but > 0). +ARENA_SIZE = 0x100000 # 1MB +ARENA_ADDR_START = 0x10000000 # 256MB (too low a number, segfault on linux) +ARENA_ADDR_STOP = 0x800000000 # 32GB # ____________________________________________________________ # @@ -63,28 +44,30 @@ # pages, it is a chained list of pages having the same size class, # rooted in 'page_for_size[size_class]'. For full pages, it is a # different chained list rooted in 'full_page_for_size[size_class]'. - # For free pages, it is the list 'freepages' in the arena header. + # For free pages, it is the list 'freepages'. ('nextpage', PAGE_PTR), - # -- The arena this page is part of. - ('arena', ARENA_PTR), # -- The number of free blocks. The numbers of uninitialized and # allocated blocks can be deduced from the context if needed. - ('nfree', lltype.Signed), - # -- The chained list of free blocks. It ends as a pointer to the + ('nfree', rffi.INT), + # -- The chained list of free blocks. It ends as a reference to the # first uninitialized block (pointing to data that is uninitialized, - # or to the end of the page). - ('freeblock', llmemory.Address), - # -- The structure above is 4 words, which is a good value: - # '(1024-4) % N' is zero or very small for various small N's, + # or to the end of the page). Each entry in the free list is encoded + # as an offset to the start of the page. + ('freeblock', rffi.INT), + # -- The structure above is 2 words, which is a good value: + # '(512-2) % N' is zero or very small for various small N's, # i.e. there is not much wasted space. ) PAGE_PTR.TO.become(PAGE_HEADER) PAGE_NULL = lltype.nullptr(PAGE_HEADER) +FREEBLOCK = lltype.Struct('FreeBlock', ('freeblock', rffi.INT)) +FREEBLOCK_PTR = lltype.Ptr(FREEBLOCK) + # ---------- -class ArenaCollection(object): +class ArenaCollection2(object): _alloc_flavor_ = "raw" def __init__(self, arena_size, page_size, small_request_threshold): @@ -111,31 +94,19 @@ for i in range(1, length): self.nblocks_for_size[i] = (page_size - self.hdrsize) // (WORD * i) # - self.max_pages_per_arena = arena_size // page_size - self.arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR), - self.max_pages_per_arena, - flavor='raw', zero=True) - # this is used in mass_free() only - self.old_arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR), - self.max_pages_per_arena, - flavor='raw', zero=True) - # - # the arena currently consumed; it must have at least one page - # available, or be NULL. The arena object that we point to is - # not in any 'arenas_lists'. We will consume all its pages before - # we choose a next arena, even if there is a major collection - # in-between. - self.current_arena = ARENA_NULL + # The next address to get an arena from + self.next_arena_addr = ARENA_ADDR_START # - # guarantee that 'arenas_lists[1:min_empty_nfreepages]' are all empty - self.min_empty_nfreepages = self.max_pages_per_arena - # - # part of current_arena might still contain uninitialized pages + # Uninitialized pages from the current arena + self.next_uninitialized_page = NULL self.num_uninitialized_pages = 0 # # the total memory used, counting every block in use, without # the additional bookkeeping stuff. self.total_memory_used = r_uint(0) + # + # Chained list of pages that used to contain stuff but are now free. + self.freepages = NULL def malloc(self, size): @@ -153,24 +124,28 @@ page = self.allocate_new_page(size_class) # # The result is simply 'page.freeblock' - result = page.freeblock - if page.nfree > 0: + pageaddr = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) + resultofs = rffi.getintfield(page, 'freeblock') + result = pageaddr + resultofs + page_nfree = rffi.getintfield(page, 'nfree') + if page_nfree > 0: # # The 'result' was part of the chained list; read the next. - page.nfree -= 1 - freeblock = result.address[0] + page_nfree -= 1 + rffi.setintfield(page, 'nfree', page_nfree) + freeblockptr = llmemory.cast_adr_to_ptr(result, FREEBLOCK_PTR) + freeblock = rffi.getintfield(freeblockptr, 'freeblock') llarena.arena_reset(result, - llmemory.sizeof(llmemory.Address), + llmemory.sizeof(FREEBLOCK), 0) # else: # The 'result' is part of the uninitialized blocks. - freeblock = result + nsize + freeblock = resultofs + nsize # - page.freeblock = freeblock + rffi.setintfield(page, 'freeblock', freeblock) # - pageaddr = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) - if freeblock - pageaddr > self.page_size - nsize: + if freeblock > self.page_size - nsize: # This was the last free block, so unlink the page from the # chained list and put it in the 'full_page_for_size' list. self.page_for_size[size_class] = page.nextpage @@ -184,48 +159,38 @@ def allocate_new_page(self, size_class): """Allocate and return a new page for the given size_class.""" # - # Allocate a new arena if needed. - if self.current_arena == ARENA_NULL: - self.allocate_new_arena() - # - # The result is simply 'current_arena.freepages'. - arena = self.current_arena - result = arena.freepages - if arena.nfreepages > 0: - # - # The 'result' was part of the chained list; read the next. - arena.nfreepages -= 1 - freepages = result.address[0] + # If available, return the next page in self.freepages + if self.freepages != NULL: + result = self.freepages + self.freepages = result.address[0] llarena.arena_reset(result, llmemory.sizeof(llmemory.Address), 0) # else: - # The 'result' is part of the uninitialized pages. + # + # No more free page. Allocate a new arena if needed. + if self.next_uninitialized_page == NULL: + self.allocate_new_arena() + # + # The result is simply 'self.next_uninitialized_page'. + result = self.next_uninitialized_page + # ll_assert(self.num_uninitialized_pages > 0, - "fully allocated arena found in self.current_arena") + "fully allocated arena found in next_uninitialized_page") self.num_uninitialized_pages -= 1 if self.num_uninitialized_pages > 0: freepages = result + self.page_size else: freepages = NULL - # - arena.freepages = freepages - if freepages == NULL: - # This was the last page, so put the arena away into - # arenas_lists[0]. - ll_assert(arena.nfreepages == 0, - "freepages == NULL but nfreepages > 0") - arena.nextarena = self.arenas_lists[0] - self.arenas_lists[0] = arena - self.current_arena = ARENA_NULL + # + self.next_uninitialized_page = freepages # # Initialize the fields of the resulting page llarena.arena_reserve(result, llmemory.sizeof(PAGE_HEADER)) page = llmemory.cast_adr_to_ptr(result, PAGE_PTR) - page.arena = arena - page.nfree = 0 - page.freeblock = result + self.hdrsize + rffi.setintfield(page, 'nfree', 0) + rffi.setintfield(page, 'freeblock', self.hdrsize) page.nextpage = PAGE_NULL ll_assert(self.page_for_size[size_class] == PAGE_NULL, "allocate_new_page() called but a page is already waiting") @@ -233,63 +198,52 @@ return page - def _all_arenas(self): - """For testing. Enumerates all arenas.""" - if self.current_arena: - yield self.current_arena - for arena in self.arenas_lists: - while arena: - yield arena - arena = arena.nextarena + def allocate_new_arena(self): + """Allocates an arena and load it in self.next_uninitialized_page.""" + arena_base = self.allocate_big_chunk(self.arena_size) + self.next_uninitialized_page = arena_base + self.num_uninitialized_pages = self.arena_size // self.page_size + allocate_new_arena._dont_inline_ = True + def allocate_big_chunk(self, arena_size): + if we_are_translated(): + return self._allocate_new_arena_mmap(arena_size) + else: + return llarena.arena_malloc(arena_size, False) - def allocate_new_arena(self): - """Loads in self.current_arena the arena to allocate from next.""" + def free_big_chunk(self, arena): + if we_are_translated(): + pass # good enough + else: + llarena.arena_free(arena) + + def _allocate_new_arena_mmap(self, arena_size): # - # Pick an arena from 'arenas_lists[i]', with i as small as possible - # but > 0. Use caching with 'min_empty_nfreepages', which guarantees - # that 'arenas_lists[1:min_empty_nfreepages]' are all empty. - i = self.min_empty_nfreepages - while i < self.max_pages_per_arena: - # - if self.arenas_lists[i] != ARENA_NULL: - # - # Found it. - self.current_arena = self.arenas_lists[i] - self.arenas_lists[i] = self.current_arena.nextarena - return - # - i += 1 - self.min_empty_nfreepages = i - # - # No more arena with any free page. We must allocate a new arena. - if not we_are_translated(): - for a in self._all_arenas(): - assert a.nfreepages == 0 - # - # 'arena_base' points to the start of malloced memory; it might not - # be a page-aligned address - arena_base = llarena.arena_malloc(self.arena_size, False) - if not arena_base: - raise MemoryError("couldn't allocate the next arena") - arena_end = arena_base + self.arena_size - # - # 'firstpage' points to the first unused page - firstpage = start_of_page(arena_base + self.page_size - 1, - self.page_size) - # 'npages' is the number of full pages just allocated - npages = (arena_end - firstpage) // self.page_size - # - # Allocate an ARENA object and initialize it - arena = lltype.malloc(ARENA, flavor='raw') - arena.base = arena_base - arena.nfreepages = 0 # they are all uninitialized pages - arena.totalpages = npages - arena.freepages = firstpage - self.num_uninitialized_pages = npages - self.current_arena = arena + # Round up the number in arena_size. + arena_size = (arena_size + ARENA_SIZE - 1) & ~(ARENA_SIZE-1) # - allocate_new_arena._dont_inline_ = True + # Try to mmap() at a MAP_FIXED address, in a 'while' loop until it + # succeeds. The important part is that it must return an address + # that is in the lower 32GB of the addressable space. + while 1: + addr = self.next_arena_addr + if addr + arena_size > ARENA_ADDR_STOP: + raise MemoryError("exhausted the 32GB of memory") + self.next_arena_addr = addr + arena_size + flags = rmmap.MAP_PRIVATE | rmmap.MAP_ANONYMOUS | rmmap.MAP_FIXED + prot = rmmap.PROT_READ | rmmap.PROT_WRITE + arena_base = rmmap.c_mmap_safe(rffi.cast(rffi.CCHARP, addr), + arena_size, prot, flags, -1, 0) + if arena_base != rffi.cast(rffi.CCHARP, -1): + break + # + # 'arena_base' points to the start of mmap()ed memory. + # Sanity-check it. + if rffi.cast(lltype.Unsigned, arena_base) >= ARENA_ADDR_STOP: + raise MMapIgnoredFIXED("mmap() ignored the MAP_FIXED and returned" + " an address that is not in the first 32GB") + # + return rffi.cast(llmemory.Address, arena_base) def mass_free(self, ok_to_free_func): @@ -304,48 +258,13 @@ # # Walk the pages in 'page_for_size[size_class]' and # 'full_page_for_size[size_class]' and free some objects. - # Pages completely freed are added to 'page.arena.freepages', + # Pages completely freed are added to 'self.freepages', # and become available for reuse by any size class. Pages # not completely freed are re-chained either in # 'full_page_for_size[]' or 'page_for_size[]'. self.mass_free_in_pages(size_class, ok_to_free_func) # size_class -= 1 - # - # Rehash arenas into the correct arenas_lists[i]. If - # 'self.current_arena' contains an arena too, it remains there. - (self.old_arenas_lists, self.arenas_lists) = ( - self.arenas_lists, self.old_arenas_lists) - # - i = 0 - while i < self.max_pages_per_arena: - self.arenas_lists[i] = ARENA_NULL - i += 1 - # - i = 0 - while i < self.max_pages_per_arena: - arena = self.old_arenas_lists[i] - while arena != ARENA_NULL: - nextarena = arena.nextarena - # - if arena.nfreepages == arena.totalpages: - # - # The whole arena is empty. Free it. - llarena.arena_free(arena.base) - lltype.free(arena, flavor='raw') - # - else: - # Insert 'arena' in the correct arenas_lists[n] - n = arena.nfreepages - ll_assert(n < self.max_pages_per_arena, - "totalpages != nfreepages >= max_pages_per_arena") - arena.nextarena = self.arenas_lists[n] - self.arenas_lists[n] = arena - # - arena = nextarena - i += 1 - # - self.min_empty_nfreepages = 1 def mass_free_in_pages(self, size_class, ok_to_free_func): @@ -398,33 +317,29 @@ def free_page(self, page): """Free a whole page.""" # - # Insert the freed page in the arena's 'freepages' list. - # If nfreepages == totalpages, then it will be freed at the - # end of mass_free(). - arena = page.arena - arena.nfreepages += 1 + # Insert the freed page in the 'freepages' list. pageaddr = llmemory.cast_ptr_to_adr(page) pageaddr = llarena.getfakearenaaddress(pageaddr) llarena.arena_reset(pageaddr, self.page_size, 0) llarena.arena_reserve(pageaddr, llmemory.sizeof(llmemory.Address)) - pageaddr.address[0] = arena.freepages - arena.freepages = pageaddr + pageaddr.address[0] = self.freepages + self.freepages = pageaddr def walk_page(self, page, block_size, ok_to_free_func): """Walk over all objects in a page, and ask ok_to_free_func().""" # + pageaddr = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) + # # 'freeblock' is the next free block - freeblock = page.freeblock + freeblock = pageaddr + rffi.getintfield(page, 'freeblock') # # 'prevfreeblockat' is the address of where 'freeblock' was read from. prevfreeblockat = lltype.direct_fieldptr(page, 'freeblock') - prevfreeblockat = llmemory.cast_ptr_to_adr(prevfreeblockat) # - obj = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) - obj += self.hdrsize + obj = pageaddr + self.hdrsize surviving = 0 # initially - skip_free_blocks = page.nfree + skip_free_blocks = rffi.getintfield(page, 'nfree') # while True: # @@ -437,11 +352,14 @@ break # # 'obj' points to a free block. It means that - # 'prevfreeblockat.address[0]' does not need to be updated. + # 'prevfreeblockat[0]' does not need to be updated. # Just read the next free block from 'obj.address[0]'. skip_free_blocks -= 1 - prevfreeblockat = obj - freeblock = obj.address[0] + prevfreeblockat = llmemory.cast_adr_to_ptr(obj, FREEBLOCK_PTR) + freeblock = pageaddr + rffi.getintfield(prevfreeblockat, + 'freeblock') + prevfreeblockat = lltype.direct_fieldptr(prevfreeblockat, + 'freeblock') # else: # 'obj' points to a valid object. @@ -452,15 +370,20 @@ # # The object should die. llarena.arena_reset(obj, _dummy_size(block_size), 0) - llarena.arena_reserve(obj, - llmemory.sizeof(llmemory.Address)) + llarena.arena_reserve(obj, llmemory.sizeof(FREEBLOCK)) # Insert 'obj' in the linked list of free blocks. - prevfreeblockat.address[0] = obj - prevfreeblockat = obj - obj.address[0] = freeblock + prevfreeblockat[0] = rffi.cast(rffi.INT, obj - pageaddr) + prevfreeblockat = llmemory.cast_adr_to_ptr(obj, + FREEBLOCK_PTR) + prevfreeblockat.freeblock = rffi.cast(rffi.INT, + freeblock - pageaddr) + prevfreeblockat = lltype.direct_fieldptr(prevfreeblockat, + 'freeblock') # # Update the number of free objects in the page. - page.nfree += 1 + page_nfree = rffi.getintfield(page, 'nfree') + page_nfree += 1 + rffi.setintfield(page, 'nfree', page_nfree) # else: # The object survives. @@ -477,35 +400,26 @@ def _nuninitialized(self, page, size_class): # Helper for debugging: count the number of uninitialized blocks - freeblock = page.freeblock + freeblock = rffi.getintfield(page, 'freeblock') + pageaddr = llmemory.cast_ptr_to_adr(page) + pageaddr = llarena.getfakearenaaddress(pageaddr) for i in range(page.nfree): - freeblock = freeblock.address[0] - assert freeblock != NULL - pageaddr = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) + freeblockaddr = pageaddr + freeblock + freeblockptr = llmemory.cast_adr_to_ptr(freeblockaddr, + FREEBLOCK_PTR) + freeblock = rffi.getintfield(freeblockptr, 'freeblock') + assert freeblock != 0 num_initialized_blocks, rem = divmod( - freeblock - pageaddr - self.hdrsize, size_class * WORD) + freeblock - self.hdrsize, size_class * WORD) assert rem == 0, "page size_class misspecified?" nblocks = self.nblocks_for_size[size_class] return nblocks - num_initialized_blocks # ____________________________________________________________ -# Helpers to go from a pointer to the start of its page -def start_of_page(addr, page_size): - """Return the address of the start of the page that contains 'addr'.""" - if we_are_translated(): - offset = llmemory.cast_adr_to_int(addr) % page_size - return addr - offset - else: - return _start_of_page_untranslated(addr, page_size) - -def _start_of_page_untranslated(addr, page_size): - assert isinstance(addr, llarena.fakearenaaddress) - shift = WORD # for testing, we assume that the whole arena is not - # on a page boundary - ofs = ((addr.offset - shift) // page_size) * page_size + shift - return llarena.fakearenaaddress(addr.arena, ofs) +class MMapIgnoredFIXED(Exception): + pass def _dummy_size(size): if we_are_translated(): Copied: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/test/test_minimarkpage2.py (from r77571, pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/test/test_minimarkpage.py) ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/test/test_minimarkpage.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/test/test_minimarkpage2.py Tue Oct 5 13:09:40 2010 @@ -1,30 +1,23 @@ import py -from pypy.rpython.memory.gc.minimarkpage import ArenaCollection -from pypy.rpython.memory.gc.minimarkpage import PAGE_HEADER, PAGE_PTR -from pypy.rpython.memory.gc.minimarkpage import PAGE_NULL, WORD -from pypy.rpython.memory.gc.minimarkpage import _dummy_size -from pypy.rpython.lltypesystem import lltype, llmemory, llarena +from pypy.rpython.memory.gc.minimarkpage2 import ArenaCollection2 +from pypy.rpython.memory.gc.minimarkpage2 import PAGE_HEADER, PAGE_PTR +from pypy.rpython.memory.gc.minimarkpage2 import PAGE_NULL, WORD +from pypy.rpython.memory.gc.minimarkpage2 import FREEBLOCK, FREEBLOCK_PTR +from pypy.rpython.memory.gc.minimarkpage2 import _dummy_size +from pypy.rpython.lltypesystem import lltype, llmemory, llarena, rffi from pypy.rpython.lltypesystem.llmemory import cast_ptr_to_adr NULL = llmemory.NULL -SHIFT = WORD hdrsize = llmemory.raw_malloc_usage(llmemory.sizeof(PAGE_HEADER)) def test_allocate_arena(): - ac = ArenaCollection(SHIFT + 64*20, 64, 1) + ac = ArenaCollection2(64*20, 64, 1) ac.allocate_new_arena() assert ac.num_uninitialized_pages == 20 - upages = ac.current_arena.freepages + upages = ac.next_uninitialized_page upages + 64*20 # does not raise py.test.raises(llarena.ArenaError, "upages + 64*20 + 1") - # - ac = ArenaCollection(SHIFT + 64*20 + 7, 64, 1) - ac.allocate_new_arena() - assert ac.num_uninitialized_pages == 20 - upages = ac.current_arena.freepages - upages + 64*20 + 7 # does not raise - py.test.raises(llarena.ArenaError, "upages + 64*20 + 64") def test_allocate_new_page(): @@ -36,24 +29,24 @@ assert (ac._nuninitialized(page, size_class) == (pagesize - hdrsize) // size) assert page.nfree == 0 - page1 = page.freeblock - hdrsize - assert llmemory.cast_ptr_to_adr(page) == page1 + assert page.freeblock == hdrsize assert page.nextpage == PAGE_NULL # - ac = ArenaCollection(arenasize, pagesize, 99) + ac = ArenaCollection2(arenasize, pagesize, 99) assert ac.num_uninitialized_pages == 0 assert ac.total_memory_used == 0 # page = ac.allocate_new_page(5) checknewpage(page, 5) assert ac.num_uninitialized_pages == 2 - assert ac.current_arena.freepages - pagesize == cast_ptr_to_adr(page) + assert lltype.typeOf(ac.next_uninitialized_page) == llmemory.Address + assert ac.next_uninitialized_page - pagesize == cast_ptr_to_adr(page) assert ac.page_for_size[5] == page # page = ac.allocate_new_page(3) checknewpage(page, 3) assert ac.num_uninitialized_pages == 1 - assert ac.current_arena.freepages - pagesize == cast_ptr_to_adr(page) + assert ac.next_uninitialized_page - pagesize == cast_ptr_to_adr(page) assert ac.page_for_size[3] == page # page = ac.allocate_new_page(4) @@ -66,25 +59,24 @@ assert " " not in pagelayout.rstrip(" ") nb_pages = len(pagelayout) arenasize = pagesize * (nb_pages + 1) - 1 - ac = ArenaCollection(arenasize, pagesize, 9*WORD) + ac = ArenaCollection2(arenasize, pagesize, 9*WORD) # def link(pageaddr, size_class, size_block, nblocks, nusedblocks, step=1): assert step in (1, 2) llarena.arena_reserve(pageaddr, llmemory.sizeof(PAGE_HEADER)) page = llmemory.cast_adr_to_ptr(pageaddr, PAGE_PTR) if step == 1: - page.nfree = 0 + page.nfree = rffi.cast(rffi.INT, 0) nuninitialized = nblocks - nusedblocks else: - page.nfree = nusedblocks + page.nfree = rffi.cast(rffi.INT, nusedblocks) nuninitialized = nblocks - 2*nusedblocks - page.freeblock = pageaddr + hdrsize + nusedblocks * size_block + page.freeblock = rffi.cast(rffi.INT, hdrsize + nusedblocks*size_block) if nusedblocks < nblocks: chainedlists = ac.page_for_size else: chainedlists = ac.full_page_for_size page.nextpage = chainedlists[size_class] - page.arena = ac.current_arena chainedlists[size_class] = page if fill_with_objects: for i in range(0, nusedblocks*step, step): @@ -93,23 +85,27 @@ if step == 2: prev = 'page.freeblock' for i in range(1, nusedblocks*step, step): - holeaddr = pageaddr + hdrsize + i * size_block - llarena.arena_reserve(holeaddr, - llmemory.sizeof(llmemory.Address)) - exec '%s = holeaddr' % prev in globals(), locals() - prevhole = holeaddr - prev = 'prevhole.address[0]' - endaddr = pageaddr + hdrsize + 2*nusedblocks * size_block - exec '%s = endaddr' % prev in globals(), locals() + holeofs = hdrsize + i * size_block + llarena.arena_reserve(pageaddr + holeofs, + llmemory.sizeof(FREEBLOCK)) + exec '%s = rffi.cast(rffi.INT, holeofs)' % prev \ + in globals(), locals() + prevhole = pageaddr + holeofs + prevhole = llmemory.cast_adr_to_ptr(prevhole, + FREEBLOCK_PTR) + prev = 'prevhole.freeblock' + endofs = hdrsize + 2*nusedblocks * size_block + exec '%s = rffi.cast(rffi.INT, endofs)' % prev \ + in globals(), locals() assert ac._nuninitialized(page, size_class) == nuninitialized # ac.allocate_new_arena() num_initialized_pages = len(pagelayout.rstrip(" ")) - ac._startpageaddr = ac.current_arena.freepages + ac._startpageaddr = ac.next_uninitialized_page if pagelayout.endswith(" "): - ac.current_arena.freepages += pagesize * num_initialized_pages + ac.next_uninitialized_page += pagesize * num_initialized_pages else: - ac.current_arena.freepages = NULL + ac.next_uninitialized_page = NULL ac.num_uninitialized_pages -= num_initialized_pages # for i in reversed(range(num_initialized_pages)): @@ -122,9 +118,8 @@ link(pageaddr, size_class, size_block, nblocks, nblocks-1) elif c == '.': # a free, but initialized, page llarena.arena_reserve(pageaddr, llmemory.sizeof(llmemory.Address)) - pageaddr.address[0] = ac.current_arena.freepages - ac.current_arena.freepages = pageaddr - ac.current_arena.nfreepages += 1 + pageaddr.address[0] = ac.freepages + ac.freepages = pageaddr elif c == '#': # a random full page, in the list 'full_pages' size_class = fill_with_objects or 1 size_block = WORD * size_class @@ -151,7 +146,7 @@ assert llmemory.cast_ptr_to_adr(page) == pagenum(ac, expected_position) def freepages(ac): - return ac.current_arena.freepages + return ac.freepages or ac.next_uninitialized_page def test_simple_arena_collection(): @@ -172,7 +167,7 @@ page = ac.allocate_new_page(6); checkpage(ac, page, 8) assert freepages(ac) == pagenum(ac, 9) and ac.num_uninitialized_pages == 1 page = ac.allocate_new_page(7); checkpage(ac, page, 9) - assert not ac.current_arena and ac.num_uninitialized_pages == 0 + assert freepages(ac) == NULL and ac.num_uninitialized_pages == 0 def chkob(ac, num_page, pos_obj, obj): @@ -217,18 +212,18 @@ page = getpage(ac, 0) assert page.nfree == 3 assert ac._nuninitialized(page, 2) == 3 - chkob(ac, 0, 2*WORD, page.freeblock) + assert page.freeblock == hdrsize + 2*WORD # obj = ac.malloc(2*WORD); chkob(ac, 0, 2*WORD, obj) obj = ac.malloc(2*WORD); chkob(ac, 0, 6*WORD, obj) assert page.nfree == 1 assert ac._nuninitialized(page, 2) == 3 - chkob(ac, 0, 10*WORD, page.freeblock) + assert page.freeblock == hdrsize + 10*WORD # obj = ac.malloc(2*WORD); chkob(ac, 0, 10*WORD, obj) assert page.nfree == 0 assert ac._nuninitialized(page, 2) == 3 - chkob(ac, 0, 12*WORD, page.freeblock) + assert page.freeblock == hdrsize + 12*WORD # obj = ac.malloc(2*WORD); chkob(ac, 0, 12*WORD, obj) assert ac._nuninitialized(page, 2) == 2 @@ -288,7 +283,7 @@ assert page.nextpage == PAGE_NULL assert ac._nuninitialized(page, 2) == 1 assert page.nfree == 0 - chkob(ac, 0, 4*WORD, page.freeblock) + assert page.freeblock == hdrsize + 4*WORD assert freepages(ac) == NULL def test_mass_free_emptied_page(): @@ -319,6 +314,14 @@ assert freepages(ac) == NULL assert ac.page_for_size[2] == PAGE_NULL +def deref(page, freeblock, repeat=1): + for i in range(repeat): + pageaddr = llmemory.cast_ptr_to_adr(page) + pageaddr = llarena.getfakearenaaddress(pageaddr) + obj = llmemory.cast_adr_to_ptr(pageaddr + freeblock, FREEBLOCK_PTR) + freeblock = obj.freeblock + return freeblock + def test_mass_free_full_is_partially_emptied(): pagesize = hdrsize + 9*WORD ac = arena_collection_for_test(pagesize, "#", fill_with_objects=2) @@ -334,9 +337,9 @@ assert page.nextpage == PAGE_NULL assert ac._nuninitialized(page, 2) == 0 assert page.nfree == 2 - assert page.freeblock == pageaddr + hdrsize + 2*WORD - assert page.freeblock.address[0] == pageaddr + hdrsize + 6*WORD - assert page.freeblock.address[0].address[0] == pageaddr + hdrsize + 8*WORD + assert page.freeblock == hdrsize + 2*WORD + assert deref(page, page.freeblock) == hdrsize + 6*WORD + assert deref(page, page.freeblock, 2) == hdrsize + 8*WORD assert freepages(ac) == NULL assert ac.full_page_for_size[2] == PAGE_NULL @@ -359,12 +362,10 @@ assert page.nextpage == PAGE_NULL assert ac._nuninitialized(page, 2) == 4 assert page.nfree == 4 - assert page.freeblock == pageaddr + hdrsize + 2*WORD - assert page.freeblock.address[0] == pageaddr + hdrsize + 6*WORD - assert page.freeblock.address[0].address[0] == \ - pageaddr + hdrsize + 10*WORD - assert page.freeblock.address[0].address[0].address[0] == \ - pageaddr + hdrsize + 14*WORD + assert deref(page, page.freeblock, 0) == hdrsize + 2*WORD + assert deref(page, page.freeblock, 1) == hdrsize + 6*WORD + assert deref(page, page.freeblock, 2) == hdrsize + 10*WORD + assert deref(page, page.freeblock, 3) == hdrsize + 14*WORD assert freepages(ac) == NULL assert ac.full_page_for_size[2] == PAGE_NULL @@ -387,16 +388,12 @@ assert page.nextpage == PAGE_NULL assert ac._nuninitialized(page, 2) == 4 assert page.nfree == 6 - fb = page.freeblock - assert fb == pageaddr + hdrsize + 2*WORD - assert fb.address[0] == pageaddr + hdrsize + 4*WORD - assert fb.address[0].address[0] == pageaddr + hdrsize + 6*WORD - assert fb.address[0].address[0].address[0] == \ - pageaddr + hdrsize + 10*WORD - assert fb.address[0].address[0].address[0].address[0] == \ - pageaddr + hdrsize + 12*WORD - assert fb.address[0].address[0].address[0].address[0].address[0] == \ - pageaddr + hdrsize + 14*WORD + assert deref(page, page.freeblock, 0) == hdrsize + 2*WORD + assert deref(page, page.freeblock, 1) == hdrsize + 4*WORD + assert deref(page, page.freeblock, 2) == hdrsize + 6*WORD + assert deref(page, page.freeblock, 3) == hdrsize + 10*WORD + assert deref(page, page.freeblock, 4) == hdrsize + 12*WORD + assert deref(page, page.freeblock, 5) == hdrsize + 14*WORD assert freepages(ac) == NULL assert ac.full_page_for_size[2] == PAGE_NULL @@ -408,25 +405,20 @@ num_pages = 3 ac = arena_collection_for_test(pagesize, " " * num_pages) live_objects = {} + all_arenas = [] # - # Run the test until three arenas are freed. This is a quick test - # that the arenas are really freed by the logic. class DoneTesting(Exception): counter = 0 def my_allocate_new_arena(): - # the following output looks cool on a 112-character-wide terminal. - lst = sorted(ac._all_arenas(), key=lambda a: a.base.arena._arena_index) - for a in lst: - print a.base.arena, a.base.arena.usagemap + # the following output looks cool on a 208-character-wide terminal. + DoneTesting.counter += 1 + if DoneTesting.counter > 9: + raise DoneTesting + for a in all_arenas: + print a.arena.usagemap print '-' * 80 ac.__class__.allocate_new_arena(ac) - a = ac.current_arena.base.arena - def my_mark_freed(): - a.freed = True - DoneTesting.counter += 1 - if DoneTesting.counter > 3: - raise DoneTesting - a.mark_freed = my_mark_freed + all_arenas.append(ac.next_uninitialized_page) ac.allocate_new_arena = my_allocate_new_arena try: while True: Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py Tue Oct 5 13:09:40 2010 @@ -10,12 +10,12 @@ from pypy.rlib import rstack, rgc from pypy.rlib.debug import ll_assert from pypy.translator.backendopt import graphanalyze -from pypy.translator.backendopt.support import var_needsgc from pypy.annotation import model as annmodel from pypy.rpython import annlowlevel from pypy.rpython.rbuiltin import gen_cast from pypy.rpython.memory.gctypelayout import ll_weakref_deref, WEAKREF from pypy.rpython.memory.gctypelayout import convert_weakref_to, WEAKREFPTR +from pypy.rpython.memory.gctypelayout import is_gc_pointer_or_hidden from pypy.rpython.memory.gctransform.log import log from pypy.tool.sourcetools import func_with_new_name from pypy.rpython.lltypesystem.lloperation import llop, LL_OPERATIONS @@ -68,9 +68,7 @@ mallocvars[op.result] = True elif op.opname in ("setfield", "setarrayitem", "setinteriorfield"): TYPE = op.args[-1].concretetype - if (op.args[0] in mallocvars and - isinstance(TYPE, lltype.Ptr) and - TYPE.TO._gckind == "gc"): + if op.args[0] in mallocvars and is_gc_pointer_or_hidden(TYPE): result.add(op) else: if collect_analyzer.analyze(op): @@ -1017,15 +1015,20 @@ v_struct = hop.spaceop.args[0] v_newvalue = hop.spaceop.args[-1] assert opname in ('setfield', 'setarrayitem', 'setinteriorfield') - assert isinstance(v_newvalue.concretetype, lltype.Ptr) + assert is_gc_pointer_or_hidden(v_newvalue.concretetype) # XXX for some GCs the skipping if the newvalue is a constant won't be # ok if (self.write_barrier_ptr is not None and not isinstance(v_newvalue, Constant) and v_struct.concretetype.TO._gckind == "gc" and hop.spaceop not in self.clean_sets): - v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], - resulttype = llmemory.Address) + if v_newvalue.concretetype == llmemory.HiddenGcRef32: + ... + v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], + resulttype = llmemory.Address) + else: + v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], + resulttype = llmemory.Address) v_structaddr = hop.genop("cast_ptr_to_adr", [v_struct], resulttype = llmemory.Address) if (self.write_barrier_from_array_ptr is not None and @@ -1089,7 +1092,7 @@ GCTransformer.gct_setfield(self, hop) def var_needs_set_transform(self, var): - return var_needsgc(var) + return is_gc_pointer_or_hidden(var.concretetype) def push_alive_nopyobj(self, var, llops): pass Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py Tue Oct 5 13:09:40 2010 @@ -366,6 +366,10 @@ # # Helpers to discover GC pointers inside structures +def is_gc_pointer_or_hidden(TYPE): + return ((isinstance(TYPE, lltype.Ptr) and TYPE.TO._gckind == 'gc') + or TYPE == llmemory.HiddenGcRef32) + def offsets_to_gc_pointers(TYPE): offsets = [] if isinstance(TYPE, lltype.Struct): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py Tue Oct 5 13:09:40 2010 @@ -1,4 +1,5 @@ from pypy.rpython.lltypesystem import lltype, llmemory, llheap +from pypy.rpython.lltypesystem.lloperation import llop from pypy.rpython import llinterp from pypy.rpython.annlowlevel import llhelper from pypy.rpython.memory import gctypelayout @@ -88,9 +89,12 @@ def setinterior(self, toplevelcontainer, inneraddr, INNERTYPE, newvalue, offsets=()): if (lltype.typeOf(toplevelcontainer).TO._gckind == 'gc' and - (isinstance(INNERTYPE, lltype.Ptr) and - INNERTYPE.TO._gckind == 'gc') - or INNERTYPE == llmemory.HiddenGcRef32): + is_gc_pointer_or_hidden(INNERTYPE)): + # + if INNERTYPE == llmemory.HiddenGcRef32: + newvalueaddr = llop.show_from_adr32(llmemory.Address, newvalue) + else: + newvalueaddr = llmemory.cast_ptr_to_adr(newvalue) # wb = True if self.has_write_barrier_from_array: @@ -99,7 +103,7 @@ assert (type(index) is int # <- fast path or lltype.typeOf(index) == lltype.Signed) self.gc.write_barrier_from_array( - llmemory.cast_ptr_to_adr(newvalue), + newvalueaddr, llmemory.cast_ptr_to_adr(toplevelcontainer), index) wb = False @@ -107,7 +111,7 @@ # if wb: self.gc.write_barrier( - llmemory.cast_ptr_to_adr(newvalue), + newvalueaddr, llmemory.cast_ptr_to_adr(toplevelcontainer)) llheap.setinterior(toplevelcontainer, inneraddr, INNERTYPE, newvalue) Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py Tue Oct 5 13:09:40 2010 @@ -881,15 +881,20 @@ def gen_startupcode(f, database): # generate the start-up code and put it into a function + if database.late_initializations_hiddengcref32: + gen_late_initializations_hiddengcref32(f, database) print >> f, 'char *RPython_StartupCode(void) {' print >> f, '\tchar *error = NULL;' - for line in database.gcpolicy.gc_startup_code(): - print >> f,"\t" + line # put float infinities in global constants, we should not have so many of them for now to make # a table+loop preferable for dest, value in database.late_initializations: print >> f, "\t%s = %s;" % (dest, value) + if database.late_initializations_hiddengcref32: + print >> f, "\tpypy_init_hiddengcref32();" + + for line in database.gcpolicy.gc_startup_code(): + print >> f,"\t" + line firsttime = True for node in database.containerlist: @@ -904,6 +909,25 @@ print >> f, '\treturn error;' print >> f, '}' +def gen_late_initializations_hiddengcref32(f, database): + print >> f, 'static void* pypy_hiddengcref32[] = {' + for access_expr, name in database.late_initializations_hiddengcref32: + print >> f, '\t&%s, %s,' % (access_expr, name) + print >> f, '''\tNULL /* sentinel */ +}; + +static void pypy_init_hiddengcref32(void) +{ +\tvoid** p; +\tfor (p = pypy_hiddengcref32; p[0] != NULL; p += 2) +\t{ +\t\thiddengcref32_t h; +\t\tOP_HIDE_INTO_ADR32((p[1]), h); +\t\t*(hiddengcref32_t**)(p[0]) = h; +\t} +} +''' + def commondefs(defines): from pypy.rlib.rarithmetic import LONG_BIT defines['PYPY_LONG_BIT'] = LONG_BIT Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py Tue Oct 5 13:09:40 2010 @@ -790,9 +790,12 @@ '-+'[value > 0]) elif TYPE == llmemory.HiddenGcRef32: if value.adr64: + name = db.get(value.adr64.ptr) db.late_initializations_hiddengcref32.append((access_expr, - value)) - expr = '0 /*HIDE_INTO_ADR32%s*/' % db.get(value.adr64.ptr) + name)) + if not name.startswith('('): + name = '(%s)' % name + expr = '0 /*HIDE_INTO_ADR32%s*/' % name else: expr = '0' else: From arigo at codespeak.net Tue Oct 5 13:33:41 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 13:33:41 +0200 (CEST) Subject: [pypy-svn] r77598 - in pypy/branch/32ptr-on-64bit/pypy: rpython/memory/gctransform translator/c translator/c/test Message-ID: <20101005113341.5032D282B90@codespeak.net> Author: arigo Date: Tue Oct 5 13:33:39 2010 New Revision: 77598 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_newgc.py Log: Finish the basic version. test_newgc starts to pass. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py Tue Oct 5 13:33:39 2010 @@ -11,6 +11,7 @@ from pypy.rlib.debug import ll_assert from pypy.translator.backendopt import graphanalyze from pypy.annotation import model as annmodel +from pypy.objspace.flow.model import Constant from pypy.rpython import annlowlevel from pypy.rpython.rbuiltin import gen_cast from pypy.rpython.memory.gctypelayout import ll_weakref_deref, WEAKREF @@ -1010,45 +1011,60 @@ return None def transform_generic_set(self, hop): - from pypy.objspace.flow.model import Constant opname = hop.spaceop.opname v_struct = hop.spaceop.args[0] v_newvalue = hop.spaceop.args[-1] assert opname in ('setfield', 'setarrayitem', 'setinteriorfield') assert is_gc_pointer_or_hidden(v_newvalue.concretetype) - # XXX for some GCs the skipping if the newvalue is a constant won't be - # ok if (self.write_barrier_ptr is not None - and not isinstance(v_newvalue, Constant) and v_struct.concretetype.TO._gckind == "gc" and hop.spaceop not in self.clean_sets): - if v_newvalue.concretetype == llmemory.HiddenGcRef32: - ... - v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], - resulttype = llmemory.Address) - else: - v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], - resulttype = llmemory.Address) - v_structaddr = hop.genop("cast_ptr_to_adr", [v_struct], - resulttype = llmemory.Address) - if (self.write_barrier_from_array_ptr is not None and - self._set_into_gc_array_part(hop.spaceop) is not None): - self.write_barrier_from_array_calls += 1 - v_index = self._set_into_gc_array_part(hop.spaceop) - assert v_index.concretetype == lltype.Signed - hop.genop("direct_call", [self.write_barrier_from_array_ptr, - self.c_const_gc, - v_newvalue, - v_structaddr, - v_index]) - else: - self.write_barrier_calls += 1 - hop.genop("direct_call", [self.write_barrier_ptr, - self.c_const_gc, - v_newvalue, - v_structaddr]) + self._add_write_barrier_call(hop, v_struct, v_newvalue) hop.rename('bare_' + opname) + def _add_write_barrier_call(self, hop, v_struct, v_newvalue): + if isinstance(v_newvalue, Constant): + # comes from a Constant -- skip + # XXX for some GCs the skipping if the newvalue is + # a constant won't be ok + return + if v_newvalue.concretetype == llmemory.HiddenGcRef32: + v_newvalue = self._fetch_unpacked_pointer(hop, v_newvalue) + if isinstance(v_newvalue, Constant): + # comes from a Constant -- skip + return + else: + v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue], + resulttype = llmemory.Address) + v_structaddr = hop.genop("cast_ptr_to_adr", [v_struct], + resulttype = llmemory.Address) + if (self.write_barrier_from_array_ptr is not None and + self._set_into_gc_array_part(hop.spaceop) is not None): + self.write_barrier_from_array_calls += 1 + v_index = self._set_into_gc_array_part(hop.spaceop) + assert v_index.concretetype == lltype.Signed + hop.genop("direct_call", [self.write_barrier_from_array_ptr, + self.c_const_gc, + v_newvalue, + v_structaddr, + v_index]) + else: + self.write_barrier_calls += 1 + hop.genop("direct_call", [self.write_barrier_ptr, + self.c_const_gc, + v_newvalue, + v_structaddr]) + + def _fetch_unpacked_pointer(self, hop, v_value): + # optimization for the common case where this setfield is preceded + # by hide_into_adr32() + for op in hop.llops[::-1]: + if op.opname == 'hide_into_adr32' and op.result == v_value: + return op.args[0] + else: + return hop.genop("show_from_adr32", [v_value], + resulttype = llmemory.Address) + def transform_getfield_typeptr(self, hop): # this would become quite a lot of operations, even if it compiles # to C code that is just as efficient as "obj->typeptr". To avoid Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py Tue Oct 5 13:33:39 2010 @@ -923,7 +923,7 @@ \t{ \t\thiddengcref32_t h; \t\tOP_HIDE_INTO_ADR32((p[1]), h); -\t\t*(hiddengcref32_t**)(p[0]) = h; +\t\t*(hiddengcref32_t*)(p[0]) = h; \t} } ''' Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_newgc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_newgc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_newgc.py Tue Oct 5 13:33:39 2010 @@ -673,7 +673,7 @@ return f - def test_callback_with_collect(self): + def XXXXXXXXXtest_callback_with_collect(self): assert self.run('callback_with_collect') def define_can_move(cls): From arigo at codespeak.net Tue Oct 5 14:09:29 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 14:09:29 +0200 (CEST) Subject: [pypy-svn] r77599 - pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc Message-ID: <20101005120929.B14E336C228@codespeak.net> Author: arigo Date: Tue Oct 5 14:09:28 2010 New Revision: 77599 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py Log: In case of compressptr, complain (crash with FixedSizeObjectTooLarge) if allocating a fixed-size object that is too large to fit in the ArenaCollection. The default limit should probably be increased quite a bit. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py Tue Oct 5 14:09:28 2010 @@ -514,6 +514,11 @@ cardheadersize = 0 extra_flags = 0 # + # We get here for large fixed-size objects. Complain if + # self.config.compressptr is set. + if self.config.compressptr and not self.is_varsize(typeid): + raise FixedSizeObjectTooLarge + # else: # Reserve N extra words containing card bits before the object. extra_words = self.card_marking_words_for_length(length) @@ -1564,3 +1569,7 @@ else: self.all_objects.append((rawobj, nsize)) self.total_memory_used += nsize + + +class FixedSizeObjectTooLarge(Exception): + pass From arigo at codespeak.net Tue Oct 5 14:27:23 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 14:27:23 +0200 (CEST) Subject: [pypy-svn] r77600 - in pypy/branch/32ptr-on-64bit/pypy: rpython rpython/lltypesystem translator/c Message-ID: <20101005122723.A07F4282B90@codespeak.net> Author: arigo Date: Tue Oct 5 14:27:21 2010 New Revision: 77600 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py Log: Check-in before I think about changing the approach. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py Tue Oct 5 14:27:21 2010 @@ -3,7 +3,8 @@ from pypy.config.translationoption import IS_64_BITS from pypy.rpython.rmodel import Repr, inputconst from pypy.rpython.lltypesystem import lltype, llmemory, rffi - +from pypy.rpython.lltypesystem.lloperation import llop +from pypy.rpython.error import TyperError def get_compressed_gcref_repr(rtyper, baserepr): @@ -38,6 +39,8 @@ class CompressedGcRefRepr(Repr): lowleveltype = llmemory.HiddenGcRef32 + ll_hash_function = None + ll_fasthash_function = None def __init__(self, mgr, baserepr): self.mgr = mgr @@ -50,6 +53,42 @@ assert T == self.BASETYPE return llmemory._hiddengcref32(llmemory.cast_ptr_to_adr(ptr)) + def get_ll_eq_function(self): + if self.baserepr.get_ll_eq_function() is not None: + raise TyperError("%r has an eq function" % (self.baserepr,)) + return None + + def get_ll_hash_function(self): + if self.ll_hash_function is None: + basefunc = self.baserepr.get_ll_hash_function() + BASETYPE = self.BASETYPE + # + def ll_hiddengcref32_hash(x): + x = llop.show_from_adr32(BASETYPE, x) + return basefunc(x) + # + return self.ll_hash_function + + def get_ll_fasthash_function(self): + if self.ll_fasthash_function is None: + basefunc = self.baserepr.get_ll_fasthash_function() + if basefunc is None: + return None + BASETYPE = self.BASETYPE + # + def ll_hiddengcref32_fasthash(x): + x = llop.show_from_adr32(BASETYPE, x) + return basefunc(x) + # + return self.ll_fasthash_function + + def get_ll_dummyval_obj(self, rtyper, s_value): + DummyVal() + + +class DummyVal(object): + ll_dummy_value = llop.hide_into_adr32xxx + class __extend__(pairtype(Repr, CompressedGcRefRepr)): def convert_from_to((r_from, r_to), v, llops): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py Tue Oct 5 14:27:21 2010 @@ -438,8 +438,11 @@ from pypy.rpython import rclass if (isinstance(item_repr, rclass.AbstractInstanceRepr) and getattr(item_repr, 'gcflavor', 'gc') == 'gc'): - #if rtyper.annotator.translator.config.translation.compressptr: - # return externalvsinternalfield(rtyper, item_repr) + if rtyper.annotator.translator.config.translation.compressptr: + item_repr, internal_item_repr = externalvsinternalfield(rtyper, + item_repr) + if internal_item_repr is not item_repr: + return item_repr, internal_item_repr return item_repr, rclass.getinstancerepr(rtyper, None) return item_repr, item_repr Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py Tue Oct 5 14:27:21 2010 @@ -789,7 +789,7 @@ expr = '0.0 /* patched later by %sinfinity */' % ( '-+'[value > 0]) elif TYPE == llmemory.HiddenGcRef32: - if value.adr64: + if value.special_value is None: name = db.get(value.adr64.ptr) db.late_initializations_hiddengcref32.append((access_expr, name)) @@ -797,7 +797,8 @@ name = '(%s)' % name expr = '0 /*HIDE_INTO_ADR32%s*/' % name else: - expr = '0' + assert isinstance(value.special_value, int) + expr = str(value.special_value) else: expr = db.get(value) if TYPE is Void: Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py Tue Oct 5 14:27:21 2010 @@ -150,7 +150,8 @@ def name_hiddengcref32(value, db): # The only prebuilt HiddenGcRef32 that should occur in a translated C # program occur as fields or items of a GcStruct or GcArray. - db.get(value.adr64) + if not value.special_value: + db.get(value.adr64) return 'HIDE_INTO_ADR32(???) /* see primitive.py, name_hiddengcref32() */' def name_small_integer(value, db): From fijal at codespeak.net Tue Oct 5 14:28:34 2010 From: fijal at codespeak.net (fijal at codespeak.net) Date: Tue, 5 Oct 2010 14:28:34 +0200 (CEST) Subject: [pypy-svn] r77601 - in pypy/benchmarks: . unladen_swallow Message-ID: <20101005122834.6007A36C228@codespeak.net> Author: fijal Date: Tue Oct 5 14:28:32 2010 New Revision: 77601 Modified: pypy/benchmarks/runner.py pypy/benchmarks/unladen_swallow/perf.py Log: An ability to store raw results Modified: pypy/benchmarks/runner.py ============================================================================== --- pypy/benchmarks/runner.py (original) +++ pypy/benchmarks/runner.py Tue Oct 5 14:28:32 2010 @@ -11,7 +11,8 @@ def run_and_store(benchmark_set, result_filename, pypy_c_path, revision=0, options='', branch='trunk', args='', upload=False, - force_host=None, fast=False, baseline=sys.executable): + force_host=None, fast=False, baseline=sys.executable, + full_store=False): funcs = perf.BENCH_FUNCS.copy() funcs.update(perf._FindAllBenchmarks(benchmarks.__dict__)) opts = ['-b', ','.join(benchmark_set), '--inherit_env=PATH', @@ -20,6 +21,8 @@ opts += ['--fast'] if args: opts += ['--args', args] + if full_store: + opts.append('--no_statistics') opts += [baseline, pypy_c_path] results = perf.main(opts, funcs) f = open(str(result_filename), "w") @@ -93,6 +96,8 @@ help="Force the hostname") parser.add_option("--fast", default=False, action="store_true", help="Run shorter benchmark runs") + parser.add_option("--full-store", default=False, action="store_true", + help="") options, args = parser.parse_args(argv) benchmarks = options.benchmarks.split(',') for benchmark in benchmarks: @@ -101,7 +106,7 @@ run_and_store(benchmarks, options.output_filename, options.pypy_c, options.revision, args=options.args, upload=options.upload, force_host=options.force_host, fast=options.fast, - baseline=options.baseline) + baseline=options.baseline, full_store=options.full_store) if __name__ == '__main__': main(sys.argv[1:]) Modified: pypy/benchmarks/unladen_swallow/perf.py ============================================================================== --- pypy/benchmarks/unladen_swallow/perf.py (original) +++ pypy/benchmarks/unladen_swallow/perf.py Tue Oct 5 14:28:32 2010 @@ -406,6 +406,14 @@ return ("%(base_time)f -> %(changed_time)f: %(time_delta)s" % self.__dict__) +class RawResult(object): + def __init__(self, base_times, changed_times): + self.base_times = base_times + self.changed_times = changed_times + + def string_representation(self): + return "Raw results: %s %s" % (self.base_times, self.changed_times) + def CompareMemoryUsage(base_usage, changed_usage, options): """Like CompareMultipleRuns, but for memory usage.""" max_base, max_changed = max(base_usage), max(changed_usage) @@ -660,6 +668,8 @@ human consumption. """ assert len(base_times) == len(changed_times) + if options.no_statistics: + return RawResult(base_times, changed_times) if len(base_times) == 1: # With only one data point, we can't do any of the interesting stats # below. @@ -1564,6 +1574,8 @@ parser.add_option("--no_charts", default=False, action="store_true", help=("Don't use google charts for displaying the" " graph outcome")) + parser.add_option("--no_statistics", default=False, action="store_true", + help=("Don't perform statistics - return raw data")) options, args = parser.parse_args(argv) if len(args) != 2: From arigo at codespeak.net Tue Oct 5 14:36:56 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 14:36:56 +0200 (CEST) Subject: [pypy-svn] r77602 - in pypy/branch/minor-cleanup/pypy/translator: jvm/test oosupport/test_template Message-ID: <20101005123656.D477B36C228@codespeak.net> Author: arigo Date: Tue Oct 5 14:36:55 2010 New Revision: 77602 Modified: pypy/branch/minor-cleanup/pypy/translator/jvm/test/test_class.py pypy/branch/minor-cleanup/pypy/translator/oosupport/test_template/class_.py Log: Kill stuff here. Modified: pypy/branch/minor-cleanup/pypy/translator/jvm/test/test_class.py ============================================================================== --- pypy/branch/minor-cleanup/pypy/translator/jvm/test/test_class.py (original) +++ pypy/branch/minor-cleanup/pypy/translator/jvm/test/test_class.py Tue Oct 5 14:36:55 2010 @@ -1,6 +1,6 @@ import py from pypy.translator.jvm.test.runtest import JvmTest -from pypy.translator.oosupport.test_template.class_ import BaseTestClass, BaseTestSpecialcase +from pypy.translator.oosupport.test_template.class_ import BaseTestClass class TestJvmClass(JvmTest, BaseTestClass): def test_overridden_classattr_as_defaults(self): @@ -26,6 +26,3 @@ def test_specialize_methods(self): py.test.skip('ABSTRACT METHOD FIX: RE-TEST AFTER MERGE') - -class TestJvmSpecialCase(JvmTest, BaseTestSpecialcase): - pass Modified: pypy/branch/minor-cleanup/pypy/translator/oosupport/test_template/class_.py ============================================================================== --- pypy/branch/minor-cleanup/pypy/translator/oosupport/test_template/class_.py (original) +++ pypy/branch/minor-cleanup/pypy/translator/oosupport/test_template/class_.py Tue Oct 5 14:36:55 2010 @@ -1,6 +1,5 @@ import py from pypy.rpython.test import test_rclass -from pypy.rpython.test.test_rspecialcase import BaseTestRspecialcase class BaseTestClass(test_rclass.TestOOtype): def test_abstract_method(self): @@ -66,6 +65,3 @@ def test_cast_object_mix_null(self): py.test.skip('cannot return ootype.NULL from translated functions') - -class BaseTestSpecialcase(BaseTestRspecialcase): - pass From arigo at codespeak.net Tue Oct 5 14:37:46 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 14:37:46 +0200 (CEST) Subject: [pypy-svn] r77603 - in pypy/trunk/pypy: annotation annotation/test module/sys objspace/std rpython rpython/test translator/goal translator/jvm/test translator/oosupport/test_template Message-ID: <20101005123746.6948536C224@codespeak.net> Author: arigo Date: Tue Oct 5 14:37:44 2010 New Revision: 77603 Removed: pypy/trunk/pypy/rpython/rspecialcase.py pypy/trunk/pypy/rpython/test/test_rspecialcase.py Modified: pypy/trunk/pypy/annotation/policy.py pypy/trunk/pypy/annotation/test/test_annrpython.py pypy/trunk/pypy/module/sys/__init__.py pypy/trunk/pypy/module/sys/state.py pypy/trunk/pypy/objspace/std/fake.py pypy/trunk/pypy/objspace/std/objspace.py pypy/trunk/pypy/rpython/rtyper.py pypy/trunk/pypy/translator/goal/ann_override.py pypy/trunk/pypy/translator/jvm/test/test_class.py pypy/trunk/pypy/translator/oosupport/test_template/class_.py Log: Merge branch/minor-cleanup: Kill an old way to do specialization at the annotator level. Modified: pypy/trunk/pypy/annotation/policy.py ============================================================================== --- pypy/trunk/pypy/annotation/policy.py (original) +++ pypy/trunk/pypy/annotation/policy.py Tue Oct 5 14:37:44 2010 @@ -1,4 +1,4 @@ -# base annotation policy for overrides and specialization +# base annotation policy for specialization from pypy.annotation.specialize import default_specialize as default from pypy.annotation.specialize import specialize_argvalue, specialize_argtype, specialize_arglistitemtype from pypy.annotation.specialize import memo @@ -41,7 +41,7 @@ if directive is None: return pol.default_specialize - # specialize|override:name[(args)] + # specialize[(args)] directive_parts = directive.split('(', 1) if len(directive_parts) == 1: [name] = directive_parts @@ -60,14 +60,6 @@ except AttributeError: raise AttributeError("%r specialize tag not defined in annotation" "policy %s" % (name, pol)) - if directive.startswith('override:'): - # different signature: override__xyz(*args_s) - if parms: - raise Exception, "override:* specialisations don't support parameters" - def specialize_override(funcdesc, args_s): - funcdesc.overridden = True - return specializer(*args_s) - return specialize_override else: if not parms: return specializer @@ -92,9 +84,5 @@ from pypy.rpython.annlowlevel import LowLevelAnnotatorPolicy return LowLevelAnnotatorPolicy.specialize__ll_and_arg(*args) - def override__ignore(pol, *args): - bk = getbookkeeper() - return bk.immutablevalue(None) - class StrictAnnotatorPolicy(AnnotatorPolicy): allow_someobjects = False Modified: pypy/trunk/pypy/annotation/test/test_annrpython.py ============================================================================== --- pypy/trunk/pypy/annotation/test/test_annrpython.py (original) +++ pypy/trunk/pypy/annotation/test/test_annrpython.py Tue Oct 5 14:37:44 2010 @@ -766,28 +766,6 @@ s = a.build_types(f, [list]) assert s.classdef is a.bookkeeper.getuniqueclassdef(IndexError) # KeyError ignored because l is a list - def test_overrides(self): - excs = [] - def record_exc(e): - """NOT_RPYTHON""" - excs.append(sys.exc_info) - record_exc._annspecialcase_ = "override:record_exc" - def g(): - pass - def f(): - try: - g() - except Exception, e: - record_exc(e) - class MyAnnotatorPolicy(policy.AnnotatorPolicy): - - def override__record_exc(pol, s_e): - return a.bookkeeper.immutablevalue(None) - - a = self.RPythonAnnotator(policy=MyAnnotatorPolicy()) - s = a.build_types(f, []) - assert s.const is None - def test_freeze_protocol(self): class Stuff: def __init__(self, flag): Modified: pypy/trunk/pypy/module/sys/__init__.py ============================================================================== --- pypy/trunk/pypy/module/sys/__init__.py (original) +++ pypy/trunk/pypy/module/sys/__init__.py Tue Oct 5 14:37:44 2010 @@ -7,13 +7,15 @@ """Sys Builtin Module. """ def __init__(self, space, w_name): """NOT_RPYTHON""" # because parent __init__ isn't + if space.config.translating: + del self.__class__.interpleveldefs['pypy_getudir'] super(Module, self).__init__(space, w_name) self.checkinterval = 100 self.recursionlimit = 100 self.w_default_encoder = None self.defaultencoding = "ascii" self.filesystemencoding = None - + interpleveldefs = { '__name__' : '(space.wrap("sys"))', '__doc__' : '(space.wrap("PyPy sys module"))', @@ -37,7 +39,7 @@ 'argv' : 'state.get(space).w_argv', 'warnoptions' : 'state.get(space).w_warnoptions', 'builtin_module_names' : 'state.w_None', - 'pypy_getudir' : 'state.pypy_getudir', + 'pypy_getudir' : 'state.pypy_getudir', # not translated 'pypy_initial_path' : 'state.pypy_initial_path', '_getframe' : 'vm._getframe', Modified: pypy/trunk/pypy/module/sys/state.py ============================================================================== --- pypy/trunk/pypy/module/sys/state.py (original) +++ pypy/trunk/pypy/module/sys/state.py Tue Oct 5 14:37:44 2010 @@ -95,15 +95,8 @@ def getio(space): return space.fromcache(IOState) -def _pypy_getudir(space): - """NOT_RPYTHON""" +def pypy_getudir(space): + """NOT_RPYTHON + (should be removed from interpleveldefs before translation)""" from pypy.tool.udir import udir return space.wrap(str(udir)) -_pypy_getudir._annspecialcase_ = "override:ignore" - -# we need the indirection because this function will live in a dictionary with other -# RPYTHON functions and share call sites with them. Better it not be a special-case -# directly. -def pypy_getudir(space): - return _pypy_getudir(space) - Modified: pypy/trunk/pypy/objspace/std/fake.py ============================================================================== --- pypy/trunk/pypy/objspace/std/fake.py (original) +++ pypy/trunk/pypy/objspace/std/fake.py Tue Oct 5 14:37:44 2010 @@ -21,7 +21,6 @@ #debug_print("faking obj %s" % x) ft = fake_type(type(x)) return ft(space, x) -fake_object._annspecialcase_ = "override:fake_object" import sys @@ -47,7 +46,6 @@ w_exc = space.wrap(exc) w_value = space.wrap(value) raise OperationError, OperationError(w_exc, w_value), tb -wrap_exception._annspecialcase_ = "override:ignore" def fake_type(cpy_type): assert type(cpy_type) is type Modified: pypy/trunk/pypy/objspace/std/objspace.py ============================================================================== --- pypy/trunk/pypy/objspace/std/objspace.py (original) +++ pypy/trunk/pypy/objspace/std/objspace.py Tue Oct 5 14:37:44 2010 @@ -242,7 +242,6 @@ w_result = getattr(self, 'w_' + x.__name__) return w_result return None - wrap_exception_cls._annspecialcase_ = "override:wrap_exception_cls" def unwrap(self, w_obj): if isinstance(w_obj, Wrappable): Modified: pypy/trunk/pypy/rpython/rtyper.py ============================================================================== --- pypy/trunk/pypy/rpython/rtyper.py (original) +++ pypy/trunk/pypy/rpython/rtyper.py Tue Oct 5 14:37:44 2010 @@ -421,7 +421,7 @@ assert noexclink.exitcase is None if pos == "removed": # the exception cannot actually occur at all. - # See for example rspecialcase.rtype_call_specialcase(). + # This is set by calling exception_cannot_occur(). # We just remove all exception links. block.exitswitch = None block.exits = block.exits[:1] @@ -1019,7 +1019,7 @@ from pypy.rpython import rint, rbool, rfloat from pypy.rpython import rrange from pypy.rpython import rstr, rdict, rlist -from pypy.rpython import rclass, rbuiltin, rpbc, rspecialcase +from pypy.rpython import rclass, rbuiltin, rpbc from pypy.rpython import rexternalobj from pypy.rpython import rptr from pypy.rpython import rgeneric Modified: pypy/trunk/pypy/translator/goal/ann_override.py ============================================================================== --- pypy/trunk/pypy/translator/goal/ann_override.py (original) +++ pypy/trunk/pypy/translator/goal/ann_override.py Tue Oct 5 14:37:44 2010 @@ -27,21 +27,6 @@ pol.pypytypes = {} pol.single_space = single_space - #def override__wrap_exception_cls(pol, space, x): - # import pypy.objspace.std.typeobject as typeobject - # clsdef = getbookkeeper().getuniqueclassdef(typeobject.W_TypeObject) - # return annmodel.SomeInstance(clsdef, can_be_None=True) - # - #def override__fake_object(pol, space, x): - # from pypy.interpreter import typedef - # clsdef = getbookkeeper().getuniqueclassdef(typedef.W_Root) - # return annmodel.SomeInstance(clsdef) - # - #def override__cpy_compile(pol, self, source, filename, mode, flags): - # from pypy.interpreter import pycode - # clsdef = getbookkeeper().getuniqueclassdef(pycode.PyCode) - # return annmodel.SomeInstance(clsdef) - def specialize__wrap(pol, funcdesc, args_s): from pypy.interpreter.baseobjspace import Wrappable from pypy.annotation.classdef import ClassDef Modified: pypy/trunk/pypy/translator/jvm/test/test_class.py ============================================================================== --- pypy/trunk/pypy/translator/jvm/test/test_class.py (original) +++ pypy/trunk/pypy/translator/jvm/test/test_class.py Tue Oct 5 14:37:44 2010 @@ -1,6 +1,6 @@ import py from pypy.translator.jvm.test.runtest import JvmTest -from pypy.translator.oosupport.test_template.class_ import BaseTestClass, BaseTestSpecialcase +from pypy.translator.oosupport.test_template.class_ import BaseTestClass class TestJvmClass(JvmTest, BaseTestClass): def test_overridden_classattr_as_defaults(self): @@ -26,6 +26,3 @@ def test_specialize_methods(self): py.test.skip('ABSTRACT METHOD FIX: RE-TEST AFTER MERGE') - -class TestJvmSpecialCase(JvmTest, BaseTestSpecialcase): - pass Modified: pypy/trunk/pypy/translator/oosupport/test_template/class_.py ============================================================================== --- pypy/trunk/pypy/translator/oosupport/test_template/class_.py (original) +++ pypy/trunk/pypy/translator/oosupport/test_template/class_.py Tue Oct 5 14:37:44 2010 @@ -1,6 +1,5 @@ import py from pypy.rpython.test import test_rclass -from pypy.rpython.test.test_rspecialcase import BaseTestRspecialcase class BaseTestClass(test_rclass.TestOOtype): def test_abstract_method(self): @@ -66,6 +65,3 @@ def test_cast_object_mix_null(self): py.test.skip('cannot return ootype.NULL from translated functions') - -class BaseTestSpecialcase(BaseTestRspecialcase): - pass From arigo at codespeak.net Tue Oct 5 14:38:03 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 14:38:03 +0200 (CEST) Subject: [pypy-svn] r77604 - pypy/branch/minor-cleanup Message-ID: <20101005123803.95F5F36C228@codespeak.net> Author: arigo Date: Tue Oct 5 14:38:01 2010 New Revision: 77604 Removed: pypy/branch/minor-cleanup/ Log: Branch merged. From antocuni at codespeak.net Tue Oct 5 15:36:57 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 5 Oct 2010 15:36:57 +0200 (CEST) Subject: [pypy-svn] r77606 - in pypy/branch/jitffi/pypy/annotation: . test Message-ID: <20101005133657.83F4D282B90@codespeak.net> Author: antocuni Date: Tue Oct 5 15:36:55 2010 New Revision: 77606 Modified: pypy/branch/jitffi/pypy/annotation/model.py pypy/branch/jitffi/pypy/annotation/test/test_model.py Log: add a failing test and fix it. The problem was that annotation_to_lltype tries to see if the annotation is contained in one of those inside annotation_to_ll_map, but for SomeSingleFloat whose .const!=None the .contains() raises TypeError when trying to compare the two consts. Moving it as the first solves the problem. Modified: pypy/branch/jitffi/pypy/annotation/model.py ============================================================================== --- pypy/branch/jitffi/pypy/annotation/model.py (original) +++ pypy/branch/jitffi/pypy/annotation/model.py Tue Oct 5 15:36:55 2010 @@ -574,11 +574,11 @@ NUMBER = object() annotation_to_ll_map = [ + (SomeSingleFloat(), lltype.SingleFloat), (s_None, lltype.Void), # also matches SomeImpossibleValue() (s_Bool, lltype.Bool), (SomeInteger(knowntype=r_ulonglong), NUMBER), (SomeFloat(), lltype.Float), - (SomeSingleFloat(), lltype.SingleFloat), (SomeChar(), lltype.Char), (SomeUnicodeCodePoint(), lltype.UniChar), (SomeAddress(), llmemory.Address), Modified: pypy/branch/jitffi/pypy/annotation/test/test_model.py ============================================================================== --- pypy/branch/jitffi/pypy/annotation/test/test_model.py (original) +++ pypy/branch/jitffi/pypy/annotation/test/test_model.py Tue Oct 5 15:36:55 2010 @@ -128,7 +128,7 @@ assert isinstance(s_p, SomeOOInstance) and s_p.ootype == C def test_annotation_to_lltype(): - from pypy.rlib.rarithmetic import r_uint + from pypy.rlib.rarithmetic import r_uint, r_singlefloat s_i = SomeInteger() s_pos = SomeInteger(nonneg=True) s_1 = SomeInteger(nonneg=True); s_1.const = 1 @@ -151,6 +151,9 @@ C = ootype.Instance('C', ROOT, {}) ref = SomeOOInstance(C) assert annotation_to_lltype(ref) == C + s_singlefloat = SomeSingleFloat() + s_singlefloat.const = r_singlefloat(0.0) + assert annotation_to_lltype(s_singlefloat) == lltype.SingleFloat def test_ll_union(): PS1 = lltype.Ptr(lltype.GcStruct('s')) From arigo at codespeak.net Tue Oct 5 15:41:31 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 15:41:31 +0200 (CEST) Subject: [pypy-svn] r77607 - in pypy/branch/32ptr-on-64bit/pypy: annotation rpython rpython/lltypesystem rpython/lltypesystem/test rpython/memory Message-ID: <20101005134131.B807F282B90@codespeak.net> Author: arigo Date: Tue Oct 5 15:41:29 2010 New Revision: 77607 Modified: pypy/branch/32ptr-on-64bit/pypy/annotation/model.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py pypy/branch/32ptr-on-64bit/pypy/rpython/raddress.py Log: In-progress: make HiddenGcRef32 a Ptr(GcOpaque), like GCREF. Modified: pypy/branch/32ptr-on-64bit/pypy/annotation/model.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/annotation/model.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/annotation/model.py Tue Oct 5 15:41:29 2010 @@ -518,14 +518,6 @@ def can_be_none(self): return False -# for compressed 32-bit "pointers" on 64-bit platforms - -class SomeHiddenGcRef32(SomeObject): - immutable = True - - def can_be_none(self): - return False - #____________________________________________________________ # annotation of low-level types @@ -590,7 +582,6 @@ (SomeChar(), lltype.Char), (SomeUnicodeCodePoint(), lltype.UniChar), (SomeAddress(), llmemory.Address), - (SomeHiddenGcRef32(), llmemory.HiddenGcRef32), ] def annotation_to_lltype(s_val, info=None): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py Tue Oct 5 15:41:29 2010 @@ -9,6 +9,7 @@ from pypy.rpython.lltypesystem import lltype from pypy.tool.uid import uid from pypy.rlib.objectmodel import we_are_translated +from pypy.rlib import rarithmetic class AddressOffset(Symbolic): @@ -565,22 +566,11 @@ # ____________________________________________________________ -class _hiddengcref32(object): - """A 'hidden' compressed 32-bit value that represents a - full, 64-bit GC pointer.""" - - def __init__(self, adr64): - self.adr64 = adr64 - - def __repr__(self): - return '<_hiddengcref32>' - - def __str__(self): - return '<_hiddengcref32 %s>' % (self.adr64,) - -compressed_null_addr = _hiddengcref32(NULL) -HiddenGcRef32 = lltype.Primitive("HiddenGcRef32", compressed_null_addr) -_hiddengcref32._TYPE = HiddenGcRef32 +# HiddenGcRef32 is only on 64-bits with 'compressptr' enabled. +# It is actually a 32-bit pointer, not a normal 64-bit pointer, +# but it's represented as a regular lltype.Ptr() in order to integrate +# better with the rest of the code. +HiddenGcRef32 = lltype.Ptr(lltype.GcOpaqueType('HIDDENGCREF32')) class OddValueMarker(AddressOffset): """This is the value '1'. Used as an odd-valued marker by the GC.""" @@ -653,8 +643,11 @@ class _char_fakeaccessor(_fakeaccessor): TYPE = lltype.Char -class _hiddengcref32_fakeaccessor(_fakeaccessor): - TYPE = HiddenGcRef32 +r_int32 = rarithmetic.build_int("r_int", True, 32) +INT32 = lltype.build_number("INT", r_int32) + +class _int32_fakeaccessor(_fakeaccessor): + TYPE = INT32 class _address_fakeaccessor(_fakeaccessor): TYPE = Address @@ -683,14 +676,14 @@ "char": lltype.Char, "address": Address, "float": lltype.Float, - "hiddengcref32": HiddenGcRef32, + "int32": INT32, } fakeaddress.signed = property(_signed_fakeaccessor) fakeaddress.float = property(_float_fakeaccessor) fakeaddress.char = property(_char_fakeaccessor) fakeaddress.address = property(_address_fakeaccessor) -fakeaddress.hiddengcref32 = property(_hiddengcref32_fakeaccessor) +fakeaddress.int32 = property(_int32_fakeaccessor) fakeaddress._TYPE = Address # the obtained address will not keep the object alive. e.g. if the object is Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py Tue Oct 5 15:41:29 2010 @@ -426,9 +426,6 @@ 'gc_gettypeptr_group': LLOp(canfold=True), 'get_member_index': LLOp(canfold=True), - 'hide_into_adr32': LLOp(canrun=True), - 'show_from_adr32': LLOp(canrun=True), - # __________ used by the JIT ________ 'jit_marker': LLOp(), Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py Tue Oct 5 15:41:29 2010 @@ -3,16 +3,14 @@ from pypy.config.translationoption import IS_64_BITS from pypy.rpython.rmodel import Repr, inputconst from pypy.rpython.lltypesystem import lltype, llmemory, rffi -from pypy.rpython.lltypesystem.lloperation import llop from pypy.rpython.error import TyperError def get_compressed_gcref_repr(rtyper, baserepr): # Return either the original baserepr, or another repr standing for # a HiddenGcRef32. The idea is that we only get a HiddenGcRef32 for - # fixed-sized structures (XXX that are not too big); thus this is only - # for structures that gets allocated by the minimarkpage2 mmap()- - # within-32GB-of-RAM. + # fixed-sized structures; they get allocated by the minimarkpage2 + # mmap()-within-32GB-of-RAM. if baserepr.lowleveltype.TO._is_varsize(): return baserepr try: @@ -51,7 +49,7 @@ ptr = self.baserepr.convert_const(value) T = lltype.typeOf(ptr) assert T == self.BASETYPE - return llmemory._hiddengcref32(llmemory.cast_ptr_to_adr(ptr)) + return lltype.cast_opaque_ptr(self.lowleveltype, ptr) def get_ll_eq_function(self): if self.baserepr.get_ll_eq_function() is not None: @@ -64,9 +62,10 @@ BASETYPE = self.BASETYPE # def ll_hiddengcref32_hash(x): - x = llop.show_from_adr32(BASETYPE, x) + x = lltype.cast_opaque_ptr(BASETYPE, x) return basefunc(x) # + self.ll_hash_function = ll_hiddengcref32_hash return self.ll_hash_function def get_ll_fasthash_function(self): @@ -77,27 +76,33 @@ BASETYPE = self.BASETYPE # def ll_hiddengcref32_fasthash(x): - x = llop.show_from_adr32(BASETYPE, x) + x = lltype.cast_opaque_ptr(BASETYPE, x) return basefunc(x) # + self.ll_fasthash_function = ll_hiddengcref32_hash return self.ll_fasthash_function def get_ll_dummyval_obj(self, rtyper, s_value): - DummyVal() + return DummyVal() class DummyVal(object): - ll_dummy_value = llop.hide_into_adr32xxx + TYPE = llmemory.HiddenGcRef32.TO + ll_dummy_value = lltype.opaqueptr(TYPE, "dummy_value", dummy_value=True) + def _freeze_(self): + return True class __extend__(pairtype(Repr, CompressedGcRefRepr)): def convert_from_to((r_from, r_to), v, llops): assert r_from.lowleveltype.TO._gckind == 'gc' - return llops.genop('hide_into_adr32', [v], + assert not isinstance(r_from.lowleveltype.TO, lltype.GcOpaqueType) + return llops.genop('cast_opaque_ptr', [v], resulttype=llmemory.HiddenGcRef32) class __extend__(pairtype(CompressedGcRefRepr, Repr)): def convert_from_to((r_from, r_to), v, llops): assert r_to.lowleveltype.TO._gckind == 'gc' - return llops.genop('show_from_adr32', [v], + assert not isinstance(r_to.lowleveltype.TO, lltype.GcOpaqueType) + return llops.genop('cast_opaque_ptr', [v], resulttype=r_to.lowleveltype) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py Tue Oct 5 15:41:29 2010 @@ -645,3 +645,13 @@ #assert cast_int_to_adr(i) == adr -- depends on ll2ctypes details i = cast_adr_to_int(NULL, mode="forced") assert type(i) is int and i == 0 + +def test_HiddenGcRef32(): + from pypy.config.translationoption import IS_64_BITS + if not IS_64_BITS: + py.test.skip("only on 64-bits") + S = lltype.GcStruct('S') + p = lltype.malloc(S) + q = lltype.cast_opaque_ptr(HiddenGcRef32, p) + r = lltype.cast_opaque_ptr(lltype.Ptr(S), q) + assert r == p Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py Tue Oct 5 15:41:29 2010 @@ -366,10 +366,6 @@ # # Helpers to discover GC pointers inside structures -def is_gc_pointer_or_hidden(TYPE): - return ((isinstance(TYPE, lltype.Ptr) and TYPE.TO._gckind == 'gc') - or TYPE == llmemory.HiddenGcRef32) - def offsets_to_gc_pointers(TYPE): offsets = [] if isinstance(TYPE, lltype.Struct): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/raddress.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/raddress.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/raddress.py Tue Oct 5 15:41:29 2010 @@ -2,7 +2,7 @@ from pypy.tool.pairtype import pairtype from pypy.annotation import model as annmodel from pypy.rpython.lltypesystem.llmemory import NULL, Address, \ - cast_adr_to_int, fakeaddress, HiddenGcRef32, _hiddengcref32 + cast_adr_to_int, fakeaddress from pypy.rpython.rmodel import Repr, IntegerRepr from pypy.rpython.rptr import PtrRepr from pypy.rpython.lltypesystem import lltype @@ -24,13 +24,6 @@ def rtyper_makekey(self): return self.__class__, self.type -class __extend__(annmodel.SomeHiddenGcRef32): - def rtyper_makerepr(self, rtyper): - return hiddengcref32_repr - - def rtyper_makekey(self): - return self.__class__, - class AddressRepr(Repr): lowleveltype = Address @@ -145,14 +138,3 @@ def convert_from_to((r_ptr, r_addr), v, llops): return llops.genop('cast_ptr_to_adr', [v], resulttype=Address) - -# ____________________________________________________________ - -class HiddenGcRef32Repr(Repr): - lowleveltype = HiddenGcRef32 - - def convert_const(self, value): - assert isinstance(value, _hiddengcref32) - return value - -hiddengcref32_repr = HiddenGcRef32Repr() From antocuni at codespeak.net Tue Oct 5 16:21:43 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 5 Oct 2010 16:21:43 +0200 (CEST) Subject: [pypy-svn] r77608 - in pypy/branch/jitffi/pypy: jit/metainterp/test rlib rlib/test Message-ID: <20101005142143.5C8B0282B90@codespeak.net> Author: antocuni Date: Tue Oct 5 16:21:41 2010 New Revision: 77608 Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py pypy/branch/jitffi/pypy/rlib/libffi.py pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Log: always cast the integer input arguments into a long Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py Tue Oct 5 16:21:41 2010 @@ -10,6 +10,8 @@ class TestFfiCall(LLJitMixin, _TestLibffiCall): + # ===> ../../../rlib/test/test_libffi.py + def call(self, funcspec, args, RESULT, init_result=0): """ Call the function specified by funcspec in a loop, and let the jit to Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Tue Oct 5 16:21:41 2010 @@ -33,10 +33,11 @@ types._import() + at specialize.arg(0) def _fits_into_long(TYPE): - if not isinstance(TYPE, lltype.Number): + if not isinstance(TYPE, lltype.Primitive): return False - if TYPE is rffi.FLOAT or TYPE is rffi.DOUBLE: + if TYPE is lltype.Void or TYPE is rffi.FLOAT or TYPE is rffi.DOUBLE: return False sz = rffi.sizeof(TYPE) return sz <= rffi.sizeof(rffi.LONG) @@ -51,8 +52,9 @@ @specialize.argtype(1) def arg(self, val): TYPE = lltype.typeOf(val) - if TYPE is rffi.LONG: + if _fits_into_long(TYPE): cls = IntArg + val = rffi.cast(rffi.LONG, val) elif TYPE is rffi.DOUBLE: cls = FloatArg else: Modified: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Tue Oct 5 16:21:41 2010 @@ -63,17 +63,17 @@ # prepare C code as an example, so we can load it and call # it via rlib.libffi c_file = udir.ensure("test_libffi", dir=1).join("foolib.c") - c_file.write(py.code.Source(''' - int sum_xy(int x, double y) - { - return (x + (int)y); - } - - unsigned char cast_to_uchar_and_ovf(int x) - { - return 200+(unsigned char)x; - } - ''')) + # automatically collect the C source from the docstrings of the tests + snippets = [] + for name in dir(cls): + if name.startswith('test_'): + meth = getattr(cls, name) + # the heuristic to determine it it's really C code could be + # improved: so far we just check that there is a '{' :-) + if meth.__doc__ is not None and '{' in meth.__doc__: + snippets.append(meth.__doc__) + # + c_file.write(py.code.Source('\n'.join(snippets))) eci = ExternalCompilationInfo(export_symbols=[]) cls.libfoo_name = str(platform.compile([c_file], eci, 'x', standalone=False)) @@ -110,6 +110,12 @@ # ------------------------------------------------------------------------ def test_simple(self): + """ + int sum_xy(int x, double y) + { + return (x + (int)y); + } + """ libfoo = self.get_libfoo() func = (libfoo, 'sum_xy', [types.sint, types.double], types.sint) res = self.call(func, [38, 4.2], rffi.LONG) @@ -130,8 +136,26 @@ self.check_loops(call=1) def test_cast_result(self): + """ + unsigned char cast_to_uchar_and_ovf(int x) + { + return 200+(unsigned char)x; + } + """ libfoo = self.get_libfoo() func = (libfoo, 'cast_to_uchar_and_ovf', [types.sint], types.uchar) res = self.call(func, [0], rffi.UCHAR) assert res == 200 self.check_loops(call=1) + + def test_cast_argument(self): + """ + int many_args(char a, int b) + { + return a+b; + } + """ + libfoo = self.get_libfoo() + func = (libfoo, 'many_args', [types.uchar, types.sint], types.sint) + res = self.call(func, [chr(20), 22], rffi.LONG) + assert res == 42 From arigo at codespeak.net Tue Oct 5 16:28:54 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 16:28:54 +0200 (CEST) Subject: [pypy-svn] r77609 - in pypy/branch/32ptr-on-64bit/pypy: rpython/lltypesystem rpython/memory rpython/memory/gc rpython/memory/gctransform translator/c translator/c/src translator/c/test Message-ID: <20101005142854.AF50C282B90@codespeak.net> Author: arigo Date: Tue Oct 5 16:28:52 2010 New Revision: 77609 Added: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py pypy/branch/32ptr-on-64bit/pypy/translator/c/database.py pypy/branch/32ptr-on-64bit/pypy/translator/c/funcgen.py pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py pypy/branch/32ptr-on-64bit/pypy/translator/c/src/commondefs.h pypy/branch/32ptr-on-64bit/pypy/translator/c/src/g_prerequisite.h pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_boehm.py pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_lladdresses.py Log: Still random progress. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py Tue Oct 5 16:28:52 2010 @@ -9,7 +9,6 @@ from pypy.rpython.lltypesystem import lltype from pypy.tool.uid import uid from pypy.rlib.objectmodel import we_are_translated -from pypy.rlib import rarithmetic class AddressOffset(Symbolic): @@ -643,12 +642,6 @@ class _char_fakeaccessor(_fakeaccessor): TYPE = lltype.Char -r_int32 = rarithmetic.build_int("r_int", True, 32) -INT32 = lltype.build_number("INT", r_int32) - -class _int32_fakeaccessor(_fakeaccessor): - TYPE = INT32 - class _address_fakeaccessor(_fakeaccessor): TYPE = Address @@ -671,19 +664,22 @@ raise TypeError(TARGETTYPE) ptr[0] = value +class _address32_fakeaccessor(_address_fakeaccessor): + pass + supported_access_types = {"signed": lltype.Signed, "unsigned": lltype.Unsigned, "char": lltype.Char, "address": Address, "float": lltype.Float, - "int32": INT32, + "address32": Address, } fakeaddress.signed = property(_signed_fakeaccessor) fakeaddress.float = property(_float_fakeaccessor) fakeaddress.char = property(_char_fakeaccessor) fakeaddress.address = property(_address_fakeaccessor) -fakeaddress.int32 = property(_int32_fakeaccessor) +fakeaddress.address32 = property(_address32_fakeaccessor) fakeaddress._TYPE = Address # the obtained address will not keep the object alive. e.g. if the object is Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py Tue Oct 5 16:28:52 2010 @@ -1,5 +1,4 @@ from pypy.rpython.lltypesystem import lltype, llmemory, llarena -from pypy.rpython.lltypesystem.lloperation import llop from pypy.rlib.debug import ll_assert from pypy.rpython.memory.gcheader import GCHeaderBuilder from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE @@ -212,14 +211,11 @@ # special handling of HiddenGcRef32 on 64-bit platforms ofs = llmemory.remove_odd_value_marker(ofs) item = obj + ofs - address = llop.show_from_adr32(llmemory.Address, - item.hiddengcref32[0]) + address = item.address32[0] if self.is_valid_gc_object(address): newaddr = callback(address, arg) if newaddr is not None: - newaddr32 = llop.hide_into_adr32(llmemory.HiddenGcRef32, - newaddr) - item.hiddengcref32[0] = newaddr32 + item.address32[0] = newaddr else: # common case item = obj + ofs Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py Tue Oct 5 16:28:52 2010 @@ -16,7 +16,6 @@ from pypy.rpython.rbuiltin import gen_cast from pypy.rpython.memory.gctypelayout import ll_weakref_deref, WEAKREF from pypy.rpython.memory.gctypelayout import convert_weakref_to, WEAKREFPTR -from pypy.rpython.memory.gctypelayout import is_gc_pointer_or_hidden from pypy.rpython.memory.gctransform.log import log from pypy.tool.sourcetools import func_with_new_name from pypy.rpython.lltypesystem.lloperation import llop, LL_OPERATIONS Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py Tue Oct 5 16:28:52 2010 @@ -389,10 +389,10 @@ #adr = llmemory.cast_ptr_to_adr(ex) #for off in offsets: # (adr + off) - elif isinstance(TYPE, lltype.Ptr) and TYPE.TO._gckind == 'gc': - offsets.append(0) elif TYPE == llmemory.HiddenGcRef32: offsets.append(llmemory.OddValueMarker()) + elif isinstance(TYPE, lltype.Ptr) and TYPE.TO._gckind == 'gc': + offsets.append(0) return offsets def gc_pointers_inside(v, adr, mutable_only=False): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gcwrapper.py Tue Oct 5 16:28:52 2010 @@ -89,12 +89,7 @@ def setinterior(self, toplevelcontainer, inneraddr, INNERTYPE, newvalue, offsets=()): if (lltype.typeOf(toplevelcontainer).TO._gckind == 'gc' and - is_gc_pointer_or_hidden(INNERTYPE)): - # - if INNERTYPE == llmemory.HiddenGcRef32: - newvalueaddr = llop.show_from_adr32(llmemory.Address, newvalue) - else: - newvalueaddr = llmemory.cast_ptr_to_adr(newvalue) + isinstance(INNERTYPE, lltype.Ptr) and INNERTYPE.TO._gckind == 'gc'): # wb = True if self.has_write_barrier_from_array: @@ -103,7 +98,7 @@ assert (type(index) is int # <- fast path or lltype.typeOf(index) == lltype.Signed) self.gc.write_barrier_from_array( - newvalueaddr, + llmemory.cast_ptr_to_adr(newvalue), llmemory.cast_ptr_to_adr(toplevelcontainer), index) wb = False @@ -111,7 +106,7 @@ # if wb: self.gc.write_barrier( - newvalueaddr, + llmemory.cast_ptr_to_adr(newvalue), llmemory.cast_ptr_to_adr(toplevelcontainer)) llheap.setinterior(toplevelcontainer, inneraddr, INNERTYPE, newvalue) Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/database.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/database.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/database.py Tue Oct 5 16:28:52 2010 @@ -4,6 +4,7 @@ ContainerType, OpaqueType, FixedSizeArray, _uninitialized from pypy.rpython.lltypesystem import lltype, rffi from pypy.rpython.lltypesystem.llmemory import WeakRef, _WeakRefType, GCREF +from pypy.rpython.lltypesystem.llmemory import HiddenGcRef32 from pypy.rpython.lltypesystem.rffi import CConstant from pypy.rpython.lltypesystem import llgroup from pypy.tool.sourcetools import valid_identifier @@ -99,7 +100,7 @@ return node def gettype(self, T, varlength=1, who_asks=None, argnames=[]): - if isinstance(T, Primitive) or T == GCREF: + if isinstance(T, Primitive) or T == GCREF or T == HiddenGcRef32: return PrimitiveType[T] elif isinstance(T, Ptr): if (isinstance(T.TO, OpaqueType) and @@ -181,7 +182,7 @@ if isinstance(obj, CConstant): return obj.c_name # without further checks T = typeOf(obj) - if isinstance(T, Primitive) or T == GCREF: + if isinstance(T, Primitive) or T == GCREF or T == HiddenGcRef32: return PrimitiveName[T](obj, self) elif isinstance(T, Ptr): if (isinstance(T.TO, OpaqueType) and Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/funcgen.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/funcgen.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/funcgen.py Tue Oct 5 16:28:52 2010 @@ -10,7 +10,7 @@ from pypy.rpython.lltypesystem.lltype import pyobjectptr, ContainerType from pypy.rpython.lltypesystem.lltype import Struct, Array, FixedSizeArray from pypy.rpython.lltypesystem.lltype import ForwardReference, FuncType -from pypy.rpython.lltypesystem.llmemory import Address +from pypy.rpython.lltypesystem.llmemory import Address, HiddenGcRef32 from pypy.translator.backendopt.ssa import SSI_to_SSA from pypy.translator.backendopt.innerloop import find_inner_loops from pypy.tool.identity_dict import identity_dict @@ -649,7 +649,15 @@ OP_CAST_PTR_TO_ADR = OP_CAST_POINTER OP_CAST_ADR_TO_PTR = OP_CAST_POINTER - OP_CAST_OPAQUE_PTR = OP_CAST_POINTER + + def OP_CAST_OPAQUE_PTR(self, op): + if op.result.concretetype == HiddenGcRef32: + return 'OP_HIDE_INTO_ADR32(%s, %s);' % (self.expr(op.args[0]), + self.expr(op.result)) + if op.args[0].concretetype == HiddenGcRef32: + return 'OP_SHOW_FROM_ADR32(%s, %s);' % (self.expr(op.args[0]), + self.expr(op.result)) + return self.OP_CAST_OPAQUE_PTR(op) def OP_CAST_INT_TO_PTR(self, op): TYPE = self.lltypemap(op.result) Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py Tue Oct 5 16:28:52 2010 @@ -910,6 +910,7 @@ print >> f, '}' def gen_late_initializations_hiddengcref32(f, database): + print >> f print >> f, 'static void* pypy_hiddengcref32[] = {' for access_expr, name in database.late_initializations_hiddengcref32: print >> f, '\t&%s, %s,' % (access_expr, name) @@ -922,7 +923,7 @@ \tfor (p = pypy_hiddengcref32; p[0] != NULL; p += 2) \t{ \t\thiddengcref32_t h; -\t\tOP_HIDE_INTO_ADR32((p[1]), h); +\t\tOP_HIDE_INTO_ADR32_CHECK((p[1]), h); \t\t*(hiddengcref32_t*)(p[0]) = h; \t} } Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py Tue Oct 5 16:28:52 2010 @@ -2,7 +2,8 @@ Struct, Array, FixedSizeArray, FuncType, PyObjectType, typeOf, \ GcStruct, GcArray, RttiStruct, ContainerType, \ parentlink, Ptr, PyObject, Void, OpaqueType, Float, \ - RuntimeTypeInfo, getRuntimeTypeInfo, Char, _subarray + RuntimeTypeInfo, getRuntimeTypeInfo, Char, _subarray, \ + cast_opaque_ptr from pypy.rpython.lltypesystem import llmemory, llgroup from pypy.translator.c.funcgen import FunctionCodeGenerator from pypy.translator.c.external import CExternalFunctionCodeGenerator @@ -788,17 +789,15 @@ db.late_initializations.append((access_expr, db.get(value))) expr = '0.0 /* patched later by %sinfinity */' % ( '-+'[value > 0]) - elif TYPE == llmemory.HiddenGcRef32: - if value.special_value is None: - name = db.get(value.adr64.ptr) - db.late_initializations_hiddengcref32.append((access_expr, - name)) - if not name.startswith('('): - name = '(%s)' % name - expr = '0 /*HIDE_INTO_ADR32%s*/' % name - else: - assert isinstance(value.special_value, int) - expr = str(value.special_value) + elif (TYPE == llmemory.HiddenGcRef32 + and value and not hasattr(value, 'dummy_value')): + realobj = value._obj.container + realvalue = cast_opaque_ptr(Ptr(typeOf(realobj)), value) + name = db.get(realvalue) + db.late_initializations_hiddengcref32.append((access_expr, name)) + if not name.startswith('('): + name = '(%s)' % name + expr = '0 /*HIDE_INTO_ADR32%s*/' % name else: expr = db.get(value) if TYPE is Void: Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py Tue Oct 5 16:28:52 2010 @@ -148,10 +148,16 @@ return 'NULL' def name_hiddengcref32(value, db): + if hasattr(value, 'dummy_value'): + return '1 /* dummy value */' + elif not value: + return '0' # NULL # The only prebuilt HiddenGcRef32 that should occur in a translated C - # program occur as fields or items of a GcStruct or GcArray. - if not value.special_value: - db.get(value.adr64) + # program occur as fields or items of a GcStruct or GcArray. They are + # handled by late_initializations_hiddengcref32. + realobj = value._obj.container + realvalue = cast_opaque_ptr(Ptr(typeOf(realobj)), value) + db.get(realvalue) # force following dependencies return 'HIDE_INTO_ADR32(???) /* see primitive.py, name_hiddengcref32() */' def name_small_integer(value, db): Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/commondefs.h ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/src/commondefs.h (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/src/commondefs.h Tue Oct 5 16:28:52 2010 @@ -65,15 +65,6 @@ # define SIZEOF_LONG 8 # define SIZEOF_LONG_LONG 8 - /* HiddenGcRef32 support (on 64-bits only) */ - typedef unsigned int hiddengcref32_t; -# define uint32_t hiddengcref32_t -# define OP_SHOW_FROM_ADR32(x, r) r = (void*)(((unsigned long)(x)) << 3) -# define OP_HIDE_INTO_ADR32(x, r) \ - r = (hiddengcref32_t)(((unsigned long)(x)) >> 3); \ - RPyAssert((void*)(((unsigned long)(r)) << 3) == (x), \ - "pointer too big or misaligned!") - #endif /********************************************************/ Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/g_prerequisite.h ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/src/g_prerequisite.h (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/src/g_prerequisite.h Tue Oct 5 16:28:52 2010 @@ -29,3 +29,8 @@ #include "src/align.h" + + +#if PYPY_LONG_BIT == 64 +# include "src/hiddengcref32.h" +#endif Added: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h ============================================================================== --- (empty file) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h Tue Oct 5 16:28:52 2010 @@ -0,0 +1,29 @@ + +/******************** 64-bit hiddengcref32 support ********************/ + +typedef unsigned int hiddengcref32_t; + + +void RPyPointerTooBig(void); + +#ifndef PYPY_NOT_MAIN_FILE +void RPyPointerTooBig(void) { + fprintf(stderr, "Fatal error: Pointer too big or misaligned. " + "This can occur if your C\n" + "compiler puts static data after the first 32GB " + "of virtual address space.\n"); + abort(); +} +#endif + + +#define OP_SHOW_FROM_ADR32(x, r) r = (void*)(((unsigned long)(x)) << 3) + +#define OP_HIDE_INTO_ADR32_CHECK(x, r) \ + r = (hiddengcref32_t)(((unsigned long)(x)) >> 3); \ + if ((void*)(((unsigned long)(r)) << 3) != (x)) \ + RPyPointerTooBig() + +#define OP_HIDE_INTO_ADR32(x, r) \ + RPyAssert(!(((long)(x)) & 0x7FFFFFFF8), "Pointer too big or misaligned"); \ + r = (hiddengcref32_t)(((unsigned long)(x)) >> 3) Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_boehm.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_boehm.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_boehm.py Tue Oct 5 16:28:52 2010 @@ -469,3 +469,41 @@ assert res[2] != compute_hash(c) # likely assert res[3] == compute_hash(d) assert res[4] == compute_hash(("Hi", None, (7.5, 2, d))) + + def test_hiddengcref32(self): + from pypy.config.translationoption import IS_64_BITS + if not IS_64_BITS: + py.test.skip("only for 64-bits") + py.test.skip("not a stand-alone executable: the static data is " + "after the limit of 32GB") + # + HiddenGcRef32 = llmemory.HiddenGcRef32 + S = lltype.GcStruct('S', ('x', lltype.Signed), + ('y', HiddenGcRef32)) + prebuilt = lltype.malloc(S, immortal=True) + prebuilt2 = lltype.malloc(S, immortal=True) + prebuilt.x = 42 + prebuilt2.x = 53 + prebuilt.y = lltype.cast_opaque_ptr(HiddenGcRef32, prebuilt2) + prebuilt2.y = lltype.cast_opaque_ptr(HiddenGcRef32, prebuilt) + + def check(a, b): + p = lltype.cast_opaque_ptr(lltype.Ptr(S), a.y) + assert p == b + + def fn(): + assert prebuilt.x == 42 + assert prebuilt2.x == 53 + check(prebuilt, prebuilt2) + check(prebuilt2, prebuilt) + p = lltype.malloc(S) + p.y = lltype.cast_opaque_ptr(HiddenGcRef32, prebuilt) + prebuilt2.y = lltype.cast_opaque_ptr(HiddenGcRef32, p) + check(p, prebuilt) + check(prebuilt2, p) + check(prebuilt, prebuilt2) + return 0 + + f = self.getcompiled(fn) + res = fn() + assert res == 0 Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_lladdresses.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_lladdresses.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/test/test_lladdresses.py Tue Oct 5 16:28:52 2010 @@ -232,43 +232,3 @@ assert res == 456 res = fc(77) assert res == 123 - - -##class TestHiddenGcRef32(StandaloneTests): - -## def setup_class(cls): -## from pypy.config.translationoption import IS_64_BITS -## if 0:# not IS_64_BITS: -## py.test.skip("only for 64-bits") - -## def test_hiddengcref32(self): -## from pypy.rpython.lltypesystem.lloperation import llop -## S = lltype.GcStruct('S', ('x', lltype.Signed), -## ('y', HiddenGcRef32)) -## prebuilt = lltype.malloc(S, immortal=True) -## prebuilt2 = lltype.malloc(S, immortal=True) -## prebuilt.x = 42 -## prebuilt2.x = 53 -## prebuilt.y = llop.hide_into_adr32(HiddenGcRef32, prebuilt2) -## prebuilt2.y = llop.hide_into_adr32(HiddenGcRef32, prebuilt) - -## def check(a, b): -## p = llop.show_from_adr32(lltype.Ptr(S), a.y) -## assert p == b - -## def f(argv): -## assert prebuilt.x == 42 -## assert prebuilt2.x == 53 -## check(prebuilt, prebuilt2) -## check(prebuilt2, prebuilt) -## p = lltype.malloc(S) -## p.y = llop.hide_into_adr32(HiddenGcRef32, prebuilt) -## prebuilt2.y = llop.hide_into_adr32(HiddenGcRef32, p) -## check(p, prebuilt) -## check(prebuilt2, p) -## check(prebuilt, prebuilt2) -## return 0 - -## fc = self.compile(f) -## res = fc([]) -## assert res == 0 From arigo at codespeak.net Tue Oct 5 16:34:18 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 16:34:18 +0200 (CEST) Subject: [pypy-svn] r77610 - pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc Message-ID: <20101005143418.112C8282B90@codespeak.net> Author: arigo Date: Tue Oct 5 16:34:16 2010 New Revision: 77610 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py Log: Fix test_gc. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py Tue Oct 5 16:34:16 2010 @@ -516,7 +516,8 @@ # # We get here for large fixed-size objects. Complain if # self.config.compressptr is set. - if self.config.compressptr and not self.is_varsize(typeid): + if (self.translated_to_c and self.config.compressptr + and not self.is_varsize(typeid)): raise FixedSizeObjectTooLarge # else: From arigo at codespeak.net Tue Oct 5 16:46:41 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 16:46:41 +0200 (CEST) Subject: [pypy-svn] r77611 - in pypy/branch/32ptr-on-64bit/pypy/rpython: lltypesystem memory/gctransform memory/test Message-ID: <20101005144641.CDC2C282B90@codespeak.net> Author: arigo Date: Tue Oct 5 16:46:40 2010 New Revision: 77611 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_transformed_gc.py Log: Whack whack whack until test_transformed_gc starts to run. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py Tue Oct 5 16:46:40 2010 @@ -5,10 +5,11 @@ # sizeof, offsetof import weakref -from pypy.rlib.objectmodel import Symbolic +from pypy.rlib.objectmodel import Symbolic, we_are_translated +from pypy.rlib.objectmodel import running_on_llinterp +from pypy.rlib.debug import llinterpcall from pypy.rpython.lltypesystem import lltype from pypy.tool.uid import uid -from pypy.rlib.objectmodel import we_are_translated class AddressOffset(Symbolic): @@ -580,9 +581,11 @@ " from a HiddenGcRef32") def has_odd_value_marker(addressofs): - if we_are_translated(): + if not we_are_translated(): + return _has_odd_value_marker(addressofs) + if not running_on_llinterp: return bool(addressofs & 1) - return _has_odd_value_marker(addressofs) + return llinterpcall(lltype.Bool, _has_odd_value_marker, addressofs) def _has_odd_value_marker(addressofs): while isinstance(addressofs, CompositeOffset): @@ -590,9 +593,11 @@ return isinstance(addressofs, OddValueMarker) def remove_odd_value_marker(addressofs): - if we_are_translated(): + if not we_are_translated(): + return _remove_odd_value_marker(addressofs) + if not running_on_llinterp: return addressofs - 1 - return _remove_odd_value_marker(addressofs) + return llinterpcall(lltype.Signed, _remove_odd_value_marker, addressofs) def _remove_odd_value_marker(addressofs): if isinstance(addressofs, CompositeOffset): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py Tue Oct 5 16:46:40 2010 @@ -10,6 +10,7 @@ from pypy.rlib import rstack, rgc from pypy.rlib.debug import ll_assert from pypy.translator.backendopt import graphanalyze +from pypy.translator.backendopt.support import var_needsgc from pypy.annotation import model as annmodel from pypy.objspace.flow.model import Constant from pypy.rpython import annlowlevel @@ -68,7 +69,9 @@ mallocvars[op.result] = True elif op.opname in ("setfield", "setarrayitem", "setinteriorfield"): TYPE = op.args[-1].concretetype - if op.args[0] in mallocvars and is_gc_pointer_or_hidden(TYPE): + if (op.args[0] in mallocvars and + isinstance(TYPE, lltype.Ptr) and + TYPE.TO._gckind == "gc"): result.add(op) else: if collect_analyzer.analyze(op): @@ -1014,7 +1017,7 @@ v_struct = hop.spaceop.args[0] v_newvalue = hop.spaceop.args[-1] assert opname in ('setfield', 'setarrayitem', 'setinteriorfield') - assert is_gc_pointer_or_hidden(v_newvalue.concretetype) + assert var_needsgc(v_newvalue) if (self.write_barrier_ptr is not None and v_struct.concretetype.TO._gckind == "gc" and hop.spaceop not in self.clean_sets): @@ -1056,12 +1059,12 @@ def _fetch_unpacked_pointer(self, hop, v_value): # optimization for the common case where this setfield is preceded - # by hide_into_adr32() + # by v_value = cast_opaque_ptr(v_normal_pointer) for op in hop.llops[::-1]: - if op.opname == 'hide_into_adr32' and op.result == v_value: + if op.opname == 'cast_opaque_ptr' and op.result == v_value: return op.args[0] else: - return hop.genop("show_from_adr32", [v_value], + return hop.genop("cast_opaque_ptr", [v_value], resulttype = llmemory.Address) def transform_getfield_typeptr(self, hop): @@ -1107,7 +1110,7 @@ GCTransformer.gct_setfield(self, hop) def var_needs_set_transform(self, var): - return is_gc_pointer_or_hidden(var.concretetype) + return var_needsgc(var) def push_alive_nopyobj(self, var, llops): pass Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_transformed_gc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_transformed_gc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_transformed_gc.py Tue Oct 5 16:46:40 2010 @@ -49,7 +49,8 @@ GC_CAN_MOVE = False GC_CAN_MALLOC_NONMOVABLE = True taggedpointers = False - + compressptr = False + def setup_class(cls): funcs0 = [] funcs2 = [] @@ -105,7 +106,8 @@ s_args = annmodel.SomePtr(lltype.Ptr(ARGS)) t = rtype(entrypoint, [s_args], gcname=cls.gcname, stacklessgc=cls.stacklessgc, - taggedpointers=cls.taggedpointers) + taggedpointers=cls.taggedpointers, + compressptr=cls.compressptr) for fixup in mixlevelstuff: if fixup: @@ -1513,6 +1515,15 @@ res = run([]) assert res == 123 +class TestMiniMarkGCCompressPtr(TestMiniMarkGC): + compressptr = True + + def setup_class(cls): + from pypy.config.translationoption import IS_64_BITS + if not IS_64_BITS: + py.test.skip("only for 64-bits") + TestMiniMarkGC.setup_class.im_func(cls) + # ________________________________________________________________ # tagged pointers From afa at codespeak.net Tue Oct 5 18:06:39 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 18:06:39 +0200 (CEST) Subject: [pypy-svn] r77613 - pypy/branch/fast-forward/pypy/doc/config Message-ID: <20101005160639.603C6282B90@codespeak.net> Author: afa Date: Tue Oct 5 18:06:37 2010 New Revision: 77613 Modified: pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._multiprocessing.txt Log: typo Modified: pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._multiprocessing.txt ============================================================================== --- pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._multiprocessing.txt (original) +++ pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._multiprocessing.txt Tue Oct 5 18:06:37 2010 @@ -1,2 +1,2 @@ Use the '_multiprocessing' module. -Used by the 'multiprocessing standard lib module. This module is expected to be working and is included by default. +Used by the 'multiprocessing' standard lib module. This module is expected to be working and is included by default. From arigo at codespeak.net Tue Oct 5 18:23:15 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 18:23:15 +0200 (CEST) Subject: [pypy-svn] r77614 - in pypy/branch/32ptr-on-64bit/pypy/rpython: . lltypesystem lltypesystem/test memory/gc memory/gctransform Message-ID: <20101005162315.B9DAF282B9C@codespeak.net> Author: arigo Date: Tue Oct 5 18:23:13 2010 New Revision: 77614 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py Log: Intermediate check-in. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py Tue Oct 5 18:23:13 2010 @@ -1025,14 +1025,22 @@ def op_raw_load(self, addr, typ, offset): checkadr(addr) - value = getattr(addr, str(typ).lower())[offset] + if typ == llmemory.HiddenGcRef32: + name = 'hiddengcref32' + else: + name = str(typ).lower() + value = getattr(addr, name)[offset] assert lltype.typeOf(value) == typ return value def op_raw_store(self, addr, typ, offset, value): checkadr(addr) + if typ == llmemory.HiddenGcRef32: + name = 'hiddengcref32' + else: + name = str(typ).lower() assert lltype.typeOf(value) == typ - getattr(addr, str(typ).lower())[offset] = value + getattr(addr, name)[offset] = value def op_stack_malloc(self, size): # mmh raise NotImplementedError("backend only") Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py Tue Oct 5 18:23:13 2010 @@ -669,22 +669,22 @@ raise TypeError(TARGETTYPE) ptr[0] = value -class _address32_fakeaccessor(_address_fakeaccessor): - pass +class _hiddengcref32_fakeaccessor(_fakeaccessor): + TYPE = HiddenGcRef32 supported_access_types = {"signed": lltype.Signed, "unsigned": lltype.Unsigned, "char": lltype.Char, "address": Address, "float": lltype.Float, - "address32": Address, + "hiddengcref32": HiddenGcRef32, } fakeaddress.signed = property(_signed_fakeaccessor) fakeaddress.float = property(_float_fakeaccessor) fakeaddress.char = property(_char_fakeaccessor) fakeaddress.address = property(_address_fakeaccessor) -fakeaddress.address32 = property(_address32_fakeaccessor) +fakeaddress.hiddengcref32 = property(_hiddengcref32_fakeaccessor) fakeaddress._TYPE = Address # the obtained address will not keep the object alive. e.g. if the object is Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py Tue Oct 5 18:23:13 2010 @@ -426,6 +426,9 @@ 'gc_gettypeptr_group': LLOp(canfold=True), 'get_member_index': LLOp(canfold=True), + 'hide_into_adr32': LLOp(canrun=True), + 'show_from_adr32': LLOp(canrun=True), + # __________ used by the JIT ________ 'jit_marker': LLOp(), Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py Tue Oct 5 18:23:13 2010 @@ -526,16 +526,21 @@ def op_shrink_array(array, smallersize): return False -def op_hide_into_adr32(adr): - if lltype.typeOf(adr) != llmemory.Address: - adr = llmemory.cast_ptr_to_adr(adr) - return llmemory._hiddengcref32(adr) +def op_hide_into_adr32(ptr): + if lltype.typeOf(ptr) == llmemory.Address: + if not ptr: + return lltype.nullptr(llmemory.HiddenGcRef32.TO) + ptr = ptr.ptr + return lltype.cast_opaque_ptr(llmemory.HiddenGcRef32, ptr) -def op_show_from_adr32(RESTYPE, adr32): +def op_show_from_adr32(RESTYPE, ptr32): if RESTYPE == llmemory.Address: - return adr32.adr64 - else: - return llmemory.cast_adr_to_ptr(adr32.adr64, RESTYPE) + if not ptr32: + return llmemory.NULL + PTRTYPE = lltype.Ptr(ptr32._obj.container._TYPE) + ptr = lltype.cast_opaque_ptr(PTRTYPE, ptr32) + return llmemory.cast_ptr_to_adr(ptr) + return lltype.cast_opaque_ptr(RESTYPE, ptr32) op_show_from_adr32.need_result_type = True # ____________________________________________________________ Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py Tue Oct 5 18:23:13 2010 @@ -3,6 +3,7 @@ from pypy.config.translationoption import IS_64_BITS from pypy.rpython.rmodel import Repr, inputconst from pypy.rpython.lltypesystem import lltype, llmemory, rffi +from pypy.rpython.lltypesystem.lloperation import llop from pypy.rpython.error import TyperError @@ -49,7 +50,7 @@ ptr = self.baserepr.convert_const(value) T = lltype.typeOf(ptr) assert T == self.BASETYPE - return lltype.cast_opaque_ptr(self.lowleveltype, ptr) + return llop.hide_into_adr32(self.lowleveltype, ptr) def get_ll_eq_function(self): if self.baserepr.get_ll_eq_function() is not None: @@ -62,7 +63,7 @@ BASETYPE = self.BASETYPE # def ll_hiddengcref32_hash(x): - x = lltype.cast_opaque_ptr(BASETYPE, x) + x = llop.show_from_adr32(BASETYPE, x) return basefunc(x) # self.ll_hash_function = ll_hiddengcref32_hash @@ -76,7 +77,7 @@ BASETYPE = self.BASETYPE # def ll_hiddengcref32_fasthash(x): - x = lltype.cast_opaque_ptr(BASETYPE, x) + x = llop.show_from_adr32(BASETYPE, x) return basefunc(x) # self.ll_fasthash_function = ll_hiddengcref32_hash @@ -97,12 +98,12 @@ def convert_from_to((r_from, r_to), v, llops): assert r_from.lowleveltype.TO._gckind == 'gc' assert not isinstance(r_from.lowleveltype.TO, lltype.GcOpaqueType) - return llops.genop('cast_opaque_ptr', [v], + return llops.genop('hide_into_adr32', [v], resulttype=llmemory.HiddenGcRef32) class __extend__(pairtype(CompressedGcRefRepr, Repr)): def convert_from_to((r_from, r_to), v, llops): assert r_to.lowleveltype.TO._gckind == 'gc' assert not isinstance(r_to.lowleveltype.TO, lltype.GcOpaqueType) - return llops.genop('cast_opaque_ptr', [v], + return llops.genop('show_from_adr32', [v], resulttype=r_to.lowleveltype) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py Tue Oct 5 18:23:13 2010 @@ -650,8 +650,10 @@ from pypy.config.translationoption import IS_64_BITS if not IS_64_BITS: py.test.skip("only on 64-bits") + # + from pypy.rpython.lltypesystem.lloperation import llop S = lltype.GcStruct('S') p = lltype.malloc(S) - q = lltype.cast_opaque_ptr(HiddenGcRef32, p) - r = lltype.cast_opaque_ptr(lltype.Ptr(S), q) + q = llop.hide_into_adr32(HiddenGcRef32, p) + r = llop.show_from_adr32(lltype.Ptr(S), q) assert r == p Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py Tue Oct 5 18:23:13 2010 @@ -1,4 +1,5 @@ from pypy.rpython.lltypesystem import lltype, llmemory, llarena +from pypy.rpython.lltypesystem.lloperation import llop from pypy.rlib.debug import ll_assert from pypy.rpython.memory.gcheader import GCHeaderBuilder from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE @@ -211,11 +212,13 @@ # special handling of HiddenGcRef32 on 64-bit platforms ofs = llmemory.remove_odd_value_marker(ofs) item = obj + ofs - address = item.address32[0] + address = llop.show_from_adr32( + llmemory.Address, item.hiddengcref32[0]) if self.is_valid_gc_object(address): newaddr = callback(address, arg) if newaddr is not None: - item.address32[0] = newaddr + item.hiddengcref32[0] = llop.hide_into_adr32( + llmemory.HiddenGcRef32, newaddr) else: # common case item = obj + ofs Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py Tue Oct 5 18:23:13 2010 @@ -1059,12 +1059,12 @@ def _fetch_unpacked_pointer(self, hop, v_value): # optimization for the common case where this setfield is preceded - # by v_value = cast_opaque_ptr(v_normal_pointer) + # by v_value = hide_into_adr32(v_normal_pointer) for op in hop.llops[::-1]: - if op.opname == 'cast_opaque_ptr' and op.result == v_value: + if op.opname == 'hide_into_adr32' and op.result == v_value: return op.args[0] else: - return hop.genop("cast_opaque_ptr", [v_value], + return hop.genop("show_from_adr32", [v_value], resulttype = llmemory.Address) def transform_getfield_typeptr(self, hop): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py Tue Oct 5 18:23:13 2010 @@ -100,7 +100,8 @@ return self.primitive_to_repr[lltype] except KeyError: pass - if isinstance(lltype, Primitive): + from pypy.rpython.lltypesystem import llmemory + if isinstance(lltype, Primitive) or lltype == llmemory.HiddenGcRef32: repr = self.primitive_to_repr[lltype] = self.getrepr(annmodel.lltype_to_annotation(lltype)) return repr raise TyperError('There is no primitive repr for %r'%(lltype,)) From arigo at codespeak.net Tue Oct 5 18:26:52 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 18:26:52 +0200 (CEST) Subject: [pypy-svn] r77615 - in pypy/branch/32ptr-on-64bit/pypy/rpython: . lltypesystem memory/gc Message-ID: <20101005162652.4B84F282B9C@codespeak.net> Author: arigo Date: Tue Oct 5 18:26:50 2010 New Revision: 77615 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py Log: Figured that I don't need "addr.hiddengcref32[0]" after all. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py Tue Oct 5 18:26:50 2010 @@ -1025,22 +1025,14 @@ def op_raw_load(self, addr, typ, offset): checkadr(addr) - if typ == llmemory.HiddenGcRef32: - name = 'hiddengcref32' - else: - name = str(typ).lower() - value = getattr(addr, name)[offset] + value = getattr(addr, str(typ).lower())[offset] assert lltype.typeOf(value) == typ return value def op_raw_store(self, addr, typ, offset, value): checkadr(addr) - if typ == llmemory.HiddenGcRef32: - name = 'hiddengcref32' - else: - name = str(typ).lower() assert lltype.typeOf(value) == typ - getattr(addr, name)[offset] = value + getattr(addr, str(typ).lower())[offset] = value def op_stack_malloc(self, size): # mmh raise NotImplementedError("backend only") Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/llmemory.py Tue Oct 5 18:26:50 2010 @@ -669,22 +669,17 @@ raise TypeError(TARGETTYPE) ptr[0] = value -class _hiddengcref32_fakeaccessor(_fakeaccessor): - TYPE = HiddenGcRef32 - supported_access_types = {"signed": lltype.Signed, "unsigned": lltype.Unsigned, "char": lltype.Char, "address": Address, "float": lltype.Float, - "hiddengcref32": HiddenGcRef32, } fakeaddress.signed = property(_signed_fakeaccessor) fakeaddress.float = property(_float_fakeaccessor) fakeaddress.char = property(_char_fakeaccessor) fakeaddress.address = property(_address_fakeaccessor) -fakeaddress.hiddengcref32 = property(_hiddengcref32_fakeaccessor) fakeaddress._TYPE = Address # the obtained address will not keep the object alive. e.g. if the object is Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py Tue Oct 5 18:26:50 2010 @@ -13,6 +13,8 @@ ('links', lltype.Array(lltype.Signed))) ARRAY_TYPEID_MAP = lltype.GcArray(lltype.Ptr(TYPEID_MAP)) +HIDDENGCREFFIELD = lltype.FixedSizeArray(llmemory.HiddenGcRef32, 1) + class GCBase(object): _alloc_flavor_ = "raw" moving_gc = False @@ -212,13 +214,13 @@ # special handling of HiddenGcRef32 on 64-bit platforms ofs = llmemory.remove_odd_value_marker(ofs) item = obj + ofs - address = llop.show_from_adr32( - llmemory.Address, item.hiddengcref32[0]) + item = llmemory.cast_adr_to_ptr(item, lltype.Ptr(HIDDENGCREFFIELD)) + address = llop.show_from_adr32(llmemory.Address, item[0]) if self.is_valid_gc_object(address): newaddr = callback(address, arg) if newaddr is not None: - item.hiddengcref32[0] = llop.hide_into_adr32( - llmemory.HiddenGcRef32, newaddr) + item[0] = llop.hide_into_adr32(llmemory.HiddenGcRef32, + newaddr) else: # common case item = obj + ofs Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py Tue Oct 5 18:26:50 2010 @@ -100,8 +100,7 @@ return self.primitive_to_repr[lltype] except KeyError: pass - from pypy.rpython.lltypesystem import llmemory - if isinstance(lltype, Primitive) or lltype == llmemory.HiddenGcRef32: + if isinstance(lltype, Primitive): repr = self.primitive_to_repr[lltype] = self.getrepr(annmodel.lltype_to_annotation(lltype)) return repr raise TyperError('There is no primitive repr for %r'%(lltype,)) From afa at codespeak.net Tue Oct 5 18:36:39 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 18:36:39 +0200 (CEST) Subject: [pypy-svn] r77616 - pypy/branch/fast-forward/pypy/interpreter/pyparser Message-ID: <20101005163639.9DF4B282B9C@codespeak.net> Author: afa Date: Tue Oct 5 18:36:38 2010 New Revision: 77616 Added: pypy/branch/fast-forward/pypy/interpreter/pyparser/autopath.py - copied unchanged from r77594, pypy/branch/fast-forward/pypy/bin/autopath.py pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py (contents, props changed) pypy/branch/fast-forward/pypy/interpreter/pyparser/pylexer.py (contents, props changed) Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py Log: I finally found how pytokenize.py is generated. Tweak the generator until the result looks similar enough to the previous arrays. the only notable difference is that state #2 and #3 have been swapped. Added: pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py Tue Oct 5 18:36:38 2010 @@ -0,0 +1,312 @@ +#! /usr/bin/env python +"""Module genPytokenize + +Generates finite state automata for recognizing Python tokens. These are hand +coded versions of the regular expressions originally appearing in Ping's +tokenize module in the Python standard library. + +When run from the command line, this should pretty print the DFA machinery. + +$Id: genPytokenize.py,v 1.1 2003/10/02 17:37:17 jriehl Exp $ +""" + +import autopath +from pypy.interpreter.pyparser.pylexer import * +from pypy.interpreter.pyparser.automata import NonGreedyDFA, DFA, DEFAULT + +def makePyPseudoDFA (): + import string + states = [] + # ____________________________________________________________ + def makeLineCont (): + return chain(states, + newArcPair(states, "\\"), + maybe(states, newArcPair(states, "\r")), + newArcPair(states, "\n")) + # ____________________________________________________________ + # Ignore stuff + def makeWhitespace (): + return any(states, groupStr(states, " \f\t")) + # ____________________________________________________________ + def makeComment (): + return chain(states, + newArcPair(states, "#"), + any(states, notGroupStr(states, "\r\n"))) + # ____________________________________________________________ + #ignore = chain(states, + # makeWhitespace(), + # any(states, chain(states, + # makeLineCont(), + # makeWhitespace())), + # maybe(states, makeComment())) + # ____________________________________________________________ + # Names + name = chain(states, + groupStr(states, string.letters + "_"), + any(states, groupStr(states, + string.letters + string.digits + "_"))) + # ____________________________________________________________ + # Digits + def makeDigits (): + return groupStr(states, "0123456789") + # ____________________________________________________________ + # Integer numbers + hexNumber = chain(states, + newArcPair(states, "0"), + groupStr(states, "xX"), + any(states, groupStr(states, "0123456789abcdefABCDEF")), + maybe(states, groupStr(states, "lL"))) + octNumber = chain(states, + newArcPair(states, "0"), + any(states, groupStr(states, "01234567")), + maybe(states, groupStr(states, "lL"))) + decNumber = chain(states, + groupStr(states, "123456789"), + any(states, makeDigits()), + maybe(states, groupStr(states, "lL"))) + intNumber = group(states, hexNumber, octNumber, decNumber) + # ____________________________________________________________ + # Exponents + def makeExp (): + return chain(states, + groupStr(states, "eE"), + maybe(states, groupStr(states, "+-")), + atleastonce(states, makeDigits())) + # ____________________________________________________________ + # Floating point numbers + def makeFloat (): + pointFloat = chain(states, + group(states, + chain(states, + atleastonce(states, makeDigits()), + newArcPair(states, "."), + any(states, makeDigits())), + chain(states, + newArcPair(states, "."), + atleastonce(states, makeDigits()))), + maybe(states, makeExp())) + expFloat = chain(states, + atleastonce(states, makeDigits()), + makeExp()) + return group(states, pointFloat, expFloat) + # ____________________________________________________________ + # Imaginary numbers + imagNumber = group(states, + chain(states, + atleastonce(states, makeDigits()), + groupStr(states, "jJ")), + chain(states, + makeFloat(), + groupStr(states, "jJ"))) + # ____________________________________________________________ + # Any old number. + number = group(states, imagNumber, makeFloat(), intNumber) + # ____________________________________________________________ + # Funny + operator = group(states, + chain(states, + chainStr(states, "**"), + maybe(states, newArcPair(states, "="))), + chain(states, + chainStr(states, ">>"), + maybe(states, newArcPair(states, "="))), + chain(states, + chainStr(states, "<<"), + maybe(states, newArcPair(states, "="))), + chainStr(states, "<>"), + chainStr(states, "!="), + chain(states, + chainStr(states, "//"), + maybe(states, newArcPair(states, "="))), + chain(states, + groupStr(states, "+-*/%&|^=<>"), + maybe(states, newArcPair(states, "="))), + newArcPair(states, "~")) + bracket = groupStr(states, "[](){}") + special = group(states, + chain(states, + maybe(states, newArcPair(states, "\r")), + newArcPair(states, "\n")), + groupStr(states, "@:;.,`")) + funny = group(states, operator, bracket, special) + # ____________________________________________________________ + def makeStrPrefix (): + return chain(states, + maybe(states, groupStr(states, "uUbB")), + maybe(states, groupStr(states, "rR"))) + # ____________________________________________________________ + contStr = group(states, + chain(states, + makeStrPrefix(), + newArcPair(states, "'"), + any(states, + notGroupStr(states, "\n'\\")), + any(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT), + any(states, + notGroupStr(states, "\n'\\")))), + group(states, + newArcPair(states, "'"), + makeLineCont())), + chain(states, + makeStrPrefix(), + newArcPair(states, '"'), + any(states, + notGroupStr(states, '\n"\\')), + any(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT), + any(states, + notGroupStr(states, '\n"\\')))), + group(states, + newArcPair(states, '"'), + makeLineCont()))) + triple = group(states, + makeStrPrefix(), + group(states, + chainStr(states, "'''"), + chainStr(states, '"""'))) + pseudoExtras = group(states, + makeLineCont(), + makeComment(), + triple) + pseudoToken = chain(states, + makeWhitespace(), + group(states, + pseudoExtras, number, funny, contStr, name)) + dfaStates, dfaAccepts = nfaToDfa(states, *pseudoToken) + return DFA(dfaStates, dfaAccepts) + +# ______________________________________________________________________ + +def makePyEndDFAMap (): + states = [] + single = chain(states, + any(states, notGroupStr(states, "'\\")), + any(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT), + any(states, notGroupStr(states, "'\\")))), + newArcPair(states, "'")) + singleDFA = DFA(*nfaToDfa(states, *single)) + states = [] + double = chain(states, + any(states, notGroupStr(states, '"\\')), + any(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT), + any(states, notGroupStr(states, '"\\')))), + newArcPair(states, '"')) + doubleDFA = DFA(*nfaToDfa(states, *double)) + states = [] + single3 = chain(states, + any(states, notGroupStr(states, "'\\")), + any(states, + chain(states, + group(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT)), + chain(states, + newArcPair(states, "'"), + notChainStr(states, "''"))), + any(states, notGroupStr(states, "'\\")))), + chainStr(states, "'''")) + single3DFA = NonGreedyDFA(*nfaToDfa(states, *single3)) + states = [] + double3 = chain(states, + any(states, notGroupStr(states, '"\\')), + any(states, + chain(states, + group(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT)), + chain(states, + newArcPair(states, '"'), + notChainStr(states, '""'))), + any(states, notGroupStr(states, '"\\')))), + chainStr(states, '"""')) + double3DFA = NonGreedyDFA(*nfaToDfa(states, *double3)) + map = {"'" : singleDFA, + '"' : doubleDFA, + "r" : None, + "R" : None, + "u" : None, + "U" : None, + "b" : None, + "B" : None} + for uniPrefix in ("", "u", "U", "b", "B", ): + for rawPrefix in ("", "r", "R"): + prefix = uniPrefix + rawPrefix + map[prefix + "'''"] = single3DFA + map[prefix + '"""'] = double3DFA + return map + +# ______________________________________________________________________ + +def output(name, dfa_class, dfa): + import textwrap + i = 0 + for line in textwrap.wrap(repr(dfa.accepts), width = 50): + if i == 0: + print "accepts =", line + else: + print " ", line + i += 1 + import StringIO + print "states = [" + for state in dfa.states: + s = StringIO.StringIO() + i = 0 + for k, v in sorted(state.items()): + i += 1 + if k == '\x00default': + k = "automata.DEFAULT" + else: + k = repr(k) + s.write(k) + s.write('::') + s.write(repr(v)) + if i < len(state): + s.write(', ') + s.write('},') + i = 0 + for line in textwrap.wrap(s.getvalue(), width=35): + line = line.replace('::', ': ') + if i == 0: + print ' {' + line + else: + print ' ' + line + i += 1 + print + print " ]" + print "%s = automata.%s(states, accepts)" % (name, dfa_class) + print + +def main (): + pseudoDFA = makePyPseudoDFA() + output("pseudoDFA", "DFA", pseudoDFA) + endDFAMap = makePyEndDFAMap() + output("double3DFA", "NonGreedyDFA", endDFAMap['"""']) + output("single3DFA", "NonGreedyDFA", endDFAMap["'''"]) + output("doubleDFA", "DFA", endDFAMap['"']) + output("singleDFA", "DFA", endDFAMap["'"]) + print "endDFAs = {\"'\" : singleDFA," + print " '\"' : doubleDFA," + print " 'r' : None," + print " 'R' : None," + print " 'u' : None," + print " 'U' : None," + print " 'b' : None," + print " 'B' : None}" + +# ______________________________________________________________________ + +if __name__ == "__main__": + main() Added: pypy/branch/fast-forward/pypy/interpreter/pyparser/pylexer.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pylexer.py Tue Oct 5 18:36:38 2010 @@ -0,0 +1,240 @@ +# Used by genpytokenize.py to generate the parser in pytokenize.py +from pypy.interpreter.pyparser.automata import DFA, DEFAULT + +class EMPTY: pass + +def newArcPair (states, transitionLabel): + s1Index = len(states) + s2Index = s1Index + 1 + states.append([(transitionLabel, s2Index)]) + states.append([]) + return s1Index, s2Index + +# ______________________________________________________________________ + +def chain (states, *stateIndexPairs): + if len(stateIndexPairs) > 1: + start, lastFinish = stateIndexPairs[0] + for nStart, nFinish in stateIndexPairs[1:]: + states[lastFinish].append((EMPTY, nStart)) + lastFinish = nFinish + return start, nFinish + else: + return stateIndexPairs[0] + + +# ______________________________________________________________________ + +def chainStr (states, str): + return chain(states, *map(lambda x : newArcPair(states, x), str)) + +# ______________________________________________________________________ + +def notChainStr (states, str): + """XXX I'm not sure this is how it should be done, but I'm going to + try it anyway. Note that for this case, I require only single character + arcs, since I would have to basically invert all accepting states and + non-accepting states of any sub-NFA's. + """ + assert len(str) > 0 + arcs = map(lambda x : newArcPair(states, x), str) + finish = len(states) + states.append([]) + start, lastFinish = arcs[0] + states[start].append((EMPTY, finish)) + for crntStart, crntFinish in arcs[1:]: + states[lastFinish].append((EMPTY, crntStart)) + states[crntStart].append((EMPTY, finish)) + return start, finish + +# ______________________________________________________________________ + +def group (states, *stateIndexPairs): + if len(stateIndexPairs) > 1: + start = len(states) + finish = start + 1 + startList = [] + states.append(startList) + states.append([]) + for eStart, eFinish in stateIndexPairs: + startList.append((EMPTY, eStart)) + states[eFinish].append((EMPTY, finish)) + return start, finish + else: + return stateIndexPairs[0] + +# ______________________________________________________________________ + +def groupStr (states, str): + return group(states, *map(lambda x : newArcPair(states, x), str)) + +# ______________________________________________________________________ + +def notGroup (states, *stateIndexPairs): + """Like group, but will add a DEFAULT transition to a new end state, + causing anything in the group to not match by going to a dead state. + XXX I think this is right... + """ + start, dead = group(states, *stateIndexPairs) + finish = len(states) + states.append([]) + states[start].append((DEFAULT, finish)) + return start, finish + +# ______________________________________________________________________ + +def notGroupStr (states, str): + return notGroup(states, *map(lambda x : newArcPair(states, x), str)) +# ______________________________________________________________________ + +def any (states, *stateIndexPairs): + start, finish = group(states, *stateIndexPairs) + states[finish].append((EMPTY, start)) + return start, start + +# ______________________________________________________________________ + +def maybe (states, *stateIndexPairs): + start, finish = group(states, *stateIndexPairs) + states[start].append((EMPTY, finish)) + return start, finish + +# ______________________________________________________________________ + +def atleastonce (states, *stateIndexPairs): + start, finish = group(states, *stateIndexPairs) + states[finish].append((EMPTY, start)) + return start, finish + +# ______________________________________________________________________ + +def closure (states, start, result = 0L): + if None == result: + result = 0L + if 0 == (result & (1L << start)): + result |= (1L << start) + for label, arrow in states[start]: + if label == EMPTY: + result |= closure(states, arrow, result) + return result + +# ______________________________________________________________________ + +def nfaToDfa (states, start, finish): + tempStates = [] + startClosure = closure(states, start) + crntTempState = [startClosure, [], 0 != (startClosure & (1L << finish))] + tempStates.append(crntTempState) + index = 0 + while index < len(tempStates): + crntTempState = tempStates[index] + crntClosure, crntArcs, crntAccept = crntTempState + for index2 in range(0, len(states)): + if 0 != (crntClosure & (1L << index2)): + for label, nfaArrow in states[index2]: + if label == EMPTY: + continue + foundTempArc = False + for tempArc in crntArcs: + if tempArc[0] == label: + foundTempArc = True + break + if not foundTempArc: + tempArc = [label, -1, 0L] + crntArcs.append(tempArc) + tempArc[2] = closure(states, nfaArrow, tempArc[2]) + for arcIndex in range(0, len(crntArcs)): + label, arrow, targetStates = crntArcs[arcIndex] + targetFound = False + arrow = 0 + for destTempState in tempStates: + if destTempState[0] == targetStates: + targetFound = True + break + arrow += 1 + if not targetFound: + assert arrow == len(tempStates) + newState = [targetStates, [], 0 != (targetStates & + (1L << finish))] + tempStates.append(newState) + crntArcs[arcIndex][1] = arrow + index += 1 + tempStates = simplifyTempDfa(tempStates) + states = finalizeTempDfa(tempStates) + return states + +# ______________________________________________________________________ + +def sameState (s1, s2): + """sameState(s1, s2) + Note: + state := [ nfaclosure : Long, [ arc ], accept : Boolean ] + arc := [ label, arrow : Int, nfaClosure : Long ] + """ + if (len(s1[1]) != len(s2[1])) or (s1[2] != s2[2]): + return False + for arcIndex in range(0, len(s1[1])): + arc1 = s1[1][arcIndex] + arc2 = s2[1][arcIndex] + if arc1[:-1] != arc2[:-1]: + return False + return True + +# ______________________________________________________________________ + +def simplifyTempDfa (tempStates): + """simplifyTempDfa (tempStates) + """ + changes = True + deletedStates = [] + while changes: + changes = False + for i in range(1, len(tempStates)): + if i in deletedStates: + continue + for j in range(0, i): + if j in deletedStates: + continue + if sameState(tempStates[i], tempStates[j]): + deletedStates.append(i) + for k in range(0, len(tempStates)): + if k in deletedStates: + continue + for arc in tempStates[k][1]: + if arc[1] == i: + arc[1] = j + changes = True + break + for stateIndex in deletedStates: + tempStates[stateIndex] = None + return tempStates +# ______________________________________________________________________ + +def finalizeTempDfa (tempStates): + """finalizeTempDfa (tempStates) + + Input domain: + tempState := [ nfaClosure : Long, [ tempArc ], accept : Boolean ] + tempArc := [ label, arrow, nfaClosure ] + + Output domain: + state := [ arcMap, accept : Boolean ] + """ + states = [] + accepts = [] + stateMap = {} + tempIndex = 0 + for tempIndex in range(0, len(tempStates)): + tempState = tempStates[tempIndex] + if None != tempState: + stateMap[tempIndex] = len(states) + states.append({}) + accepts.append(tempState[2]) + for tempIndex in stateMap.keys(): + stateBitset, tempArcs, accepting = tempStates[tempIndex] + newIndex = stateMap[tempIndex] + arcMap = states[newIndex] + for tempArc in tempArcs: + arcMap[tempArc[0]] = stateMap[tempArc[1]] + return states, accepts + Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py Tue Oct 5 18:36:38 2010 @@ -21,39 +21,40 @@ __all__ = [ "tokenize" ] # ______________________________________________________________________ -# Automatically generated DFA's (with one or two hand tweeks): -pseudoStatesAccepts = [True, True, True, True, True, True, True, True, - True, True, False, True, True, True, False, False, - False, False, True, False, False, True, True, False, - True, False, True, False, True, False, True, False, - False, False, True, False, False, False, True] - -pseudoStates = [ - {'\t': 0, '\n': 13, '\x0c': 0, '\r': 14, ' ': 0, '!': 10, - '"': 16, '#': 18, '%': 12, '&': 12, - "'": 15, '(': 13, ')': 13, '*': 7, - '+': 12, ',': 13, '-': 12, '.': 6, - '/': 11, '0': 4, '1': 5, '2': 5, - '3': 5, '4': 5, '5': 5, '6': 5, - '7': 5, '8': 5, '9': 5, ':': 13, - ';': 13, '<': 9, '=': 12, '>': 8, - '@': 13, 'A': 1, - 'B': 3, 'C': 1, 'D': 1, 'E': 1, - 'F': 1, 'G': 1, 'H': 1, 'I': 1, - 'J': 1, 'K': 1, 'L': 1, 'M': 1, - 'N': 1, 'O': 1, 'P': 1, 'Q': 1, - 'R': 2, 'S': 1, 'T': 1, 'U': 3, - 'V': 1, 'W': 1, 'X': 1, 'Y': 1, - 'Z': 1, '[': 13, '\\': 17, ']': 13, - '^': 12, '_': 1, '`': 13, 'a': 1, - 'b': 3, 'c': 1, 'd': 1, 'e': 1, - 'f': 1, 'g': 1, 'h': 1, 'i': 1, - 'j': 1, 'k': 1, 'l': 1, 'm': 1, - 'n': 1, 'o': 1, 'p': 1, 'q': 1, - 'r': 2, 's': 1, 't': 1, 'u': 3, - 'v': 1, 'w': 1, 'x': 1, 'y': 1, - 'z': 1, '{': 13, '|': 12, '}': 13, - '~': 13}, +# Automatically generated DFA's + +accepts = [True, True, True, True, True, True, True, True, + True, True, False, True, True, True, False, False, + False, False, True, False, False, True, True, + False, True, False, True, False, True, False, + True, False, False, False, True, False, False, + False, True] +states = [ + {'\t': 0, '\n': 13, '\x0c': 0, + '\r': 14, ' ': 0, '!': 10, '"': 16, + '#': 18, '%': 12, '&': 12, "'": 15, + '(': 13, ')': 13, '*': 7, '+': 12, + ',': 13, '-': 12, '.': 6, '/': 11, + '0': 4, '1': 5, '2': 5, '3': 5, + '4': 5, '5': 5, '6': 5, '7': 5, + '8': 5, '9': 5, ':': 13, ';': 13, + '<': 9, '=': 12, '>': 8, '@': 13, + 'A': 1, 'B': 2, 'C': 1, 'D': 1, + 'E': 1, 'F': 1, 'G': 1, 'H': 1, + 'I': 1, 'J': 1, 'K': 1, 'L': 1, + 'M': 1, 'N': 1, 'O': 1, 'P': 1, + 'Q': 1, 'R': 3, 'S': 1, 'T': 1, + 'U': 2, 'V': 1, 'W': 1, 'X': 1, + 'Y': 1, 'Z': 1, '[': 13, '\\': 17, + ']': 13, '^': 12, '_': 1, '`': 13, + 'a': 1, 'b': 2, 'c': 1, 'd': 1, + 'e': 1, 'f': 1, 'g': 1, 'h': 1, + 'i': 1, 'j': 1, 'k': 1, 'l': 1, + 'm': 1, 'n': 1, 'o': 1, 'p': 1, + 'q': 1, 'r': 3, 's': 1, 't': 1, + 'u': 2, 'v': 1, 'w': 1, 'x': 1, + 'y': 1, 'z': 1, '{': 13, '|': 12, + '}': 13, '~': 13}, {'0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, @@ -72,39 +73,39 @@ 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - {'"': 16, "'": 15, '0': 1, '1': 1, + {'"': 20, "'": 19, '0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, 'A': 1, 'B': 1, 'C': 1, 'D': 1, 'E': 1, 'F': 1, 'G': 1, 'H': 1, 'I': 1, 'J': 1, 'K': 1, 'L': 1, 'M': 1, 'N': 1, 'O': 1, 'P': 1, - 'Q': 1, 'R': 1, 'S': 1, 'T': 1, + 'Q': 1, 'R': 3, 'S': 1, 'T': 1, 'U': 1, 'V': 1, 'W': 1, 'X': 1, 'Y': 1, 'Z': 1, '_': 1, 'a': 1, 'b': 1, 'c': 1, 'd': 1, 'e': 1, 'f': 1, 'g': 1, 'h': 1, 'i': 1, 'j': 1, 'k': 1, 'l': 1, 'm': 1, 'n': 1, 'o': 1, 'p': 1, 'q': 1, - 'r': 1, 's': 1, 't': 1, 'u': 1, + 'r': 3, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - {'"': 16, "'": 15, '0': 1, '1': 1, + {'"': 20, "'": 19, '0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, 'A': 1, 'B': 1, 'C': 1, 'D': 1, 'E': 1, 'F': 1, 'G': 1, 'H': 1, 'I': 1, 'J': 1, 'K': 1, 'L': 1, 'M': 1, 'N': 1, 'O': 1, 'P': 1, - 'Q': 1, 'R': 2, 'S': 1, 'T': 1, + 'Q': 1, 'R': 1, 'S': 1, 'T': 1, 'U': 1, 'V': 1, 'W': 1, 'X': 1, 'Y': 1, 'Z': 1, '_': 1, 'a': 1, 'b': 1, 'c': 1, 'd': 1, 'e': 1, 'f': 1, 'g': 1, 'h': 1, 'i': 1, 'j': 1, 'k': 1, 'l': 1, 'm': 1, 'n': 1, 'o': 1, 'p': 1, 'q': 1, - 'r': 2, 's': 1, 't': 1, 'u': 1, + 'r': 1, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, @@ -128,11 +129,11 @@ {'=': 13, '>': 12}, - {'=': 13, '<': 12, '>': 13}, + {'<': 12, '=': 13, '>': 13}, {'=': 13}, - {'=': 13, '/': 12}, + {'/': 12, '=': 13}, {'=': 13}, @@ -140,77 +141,85 @@ {'\n': 13}, - {automata.DEFAULT: 19, '\n': 27, '\\': 29, "'": 28}, + {automata.DEFAULT: 19, '\n': 27, + "'": 28, '\\': 29}, - {automata.DEFAULT: 20, '"': 30, '\n': 27, '\\': 31}, + {automata.DEFAULT: 20, '\n': 27, + '"': 30, '\\': 31}, {'\n': 13, '\r': 14}, - {automata.DEFAULT: 18, '\n': 27, '\r': 27}, + {automata.DEFAULT: 18, '\n': 27, + '\r': 27}, + + {automata.DEFAULT: 19, '\n': 27, + "'": 13, '\\': 29}, - {automata.DEFAULT: 19, '\n': 27, '\\': 29, "'": 13}, + {automata.DEFAULT: 20, '\n': 27, + '"': 13, '\\': 31}, - {automata.DEFAULT: 20, '"': 13, '\n': 27, '\\': 31}, - {'0': 21, '1': 21, '2': 21, '3': 21, '4': 21, '5': 21, '6': 21, '7': 21, '8': 21, '9': 21, 'A': 21, 'B': 21, 'C': 21, 'D': 21, 'E': 21, 'F': 21, 'L': 13, 'a': 21, 'b': 21, 'c': 21, 'd': 21, 'e': 21, 'f': 21, 'l': 13}, - + {'.': 24, '0': 22, '1': 22, '2': 22, '3': 22, '4': 22, '5': 22, '6': 22, '7': 22, '8': 23, '9': 23, 'E': 25, 'J': 13, 'L': 13, 'e': 25, 'j': 13, 'l': 13}, - + {'.': 24, '0': 23, '1': 23, '2': 23, '3': 23, '4': 23, '5': 23, '6': 23, '7': 23, '8': 23, '9': 23, 'E': 25, 'J': 13, 'e': 25, 'j': 13}, - + {'0': 24, '1': 24, '2': 24, '3': 24, '4': 24, '5': 24, '6': 24, '7': 24, '8': 24, '9': 24, 'E': 32, 'J': 13, 'e': 32, 'j': 13}, - + {'+': 33, '-': 33, '0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34}, - + {'0': 26, '1': 26, '2': 26, '3': 26, '4': 26, '5': 26, '6': 26, '7': 26, '8': 26, '9': 26, 'E': 32, 'J': 13, 'e': 32, 'j': 13}, - + {}, {"'": 13}, - {automata.DEFAULT: 35, '\n': 13, '\r': 14}, + {automata.DEFAULT: 35, '\n': 13, + '\r': 14}, {'"': 13}, - {automata.DEFAULT: 36, '\n': 13, '\r': 14}, + {automata.DEFAULT: 36, '\n': 13, + '\r': 14}, {'+': 37, '-': 37, '0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38}, - - + {'0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34}, - + {'0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34, 'J': 13, 'j': 13}, - - {automata.DEFAULT: 35, '\n': 27, '\\': 29, "'": 13}, - - {automata.DEFAULT: 36, '"': 13, '\n': 27, '\\': 31}, - + + {automata.DEFAULT: 35, '\n': 27, + "'": 13, '\\': 29}, + + {automata.DEFAULT: 36, '\n': 27, + '"': 13, '\\': 31}, + {'0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38}, @@ -218,56 +227,95 @@ {'0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38, 'J': 13, 'j': 13}, + ] +pseudoDFA = automata.DFA(states, accepts) -pseudoDFA = automata.DFA(pseudoStates, pseudoStatesAccepts) +accepts = [False, False, False, False, False, True] +states = [ + {automata.DEFAULT: 0, '"': 1, + '\\': 2}, + + {automata.DEFAULT: 4, '"': 3, + '\\': 2}, -double3StatesAccepts = [False, False, False, False, False, True] -double3States = [ - {automata.DEFAULT: 0, '"': 1, '\\': 2}, - {automata.DEFAULT: 4, '"': 3, '\\': 2}, {automata.DEFAULT: 4}, - {automata.DEFAULT: 4, '"': 5, '\\': 2}, - {automata.DEFAULT: 4, '"': 1, '\\': 2}, - {automata.DEFAULT: 4, '"': 5, '\\': 2}, + + {automata.DEFAULT: 4, '"': 5, + '\\': 2}, + + {automata.DEFAULT: 4, '"': 1, + '\\': 2}, + + {automata.DEFAULT: 4, '"': 5, + '\\': 2}, + ] -double3DFA = automata.NonGreedyDFA(double3States, double3StatesAccepts) +double3DFA = automata.NonGreedyDFA(states, accepts) + +accepts = [False, False, False, False, False, True] +states = [ + {automata.DEFAULT: 0, "'": 1, + '\\': 2}, + + {automata.DEFAULT: 4, "'": 3, + '\\': 2}, -single3StatesAccepts = [False, False, False, False, False, True] -single3States = [ - {automata.DEFAULT: 0, '\\': 2, "'": 1}, - {automata.DEFAULT: 4, '\\': 2, "'": 3}, {automata.DEFAULT: 4}, - {automata.DEFAULT: 4, '\\': 2, "'": 5}, - {automata.DEFAULT: 4, '\\': 2, "'": 1}, - {automata.DEFAULT: 4, '\\': 2, "'": 5}, + + {automata.DEFAULT: 4, "'": 5, + '\\': 2}, + + {automata.DEFAULT: 4, "'": 1, + '\\': 2}, + + {automata.DEFAULT: 4, "'": 5, + '\\': 2}, + ] -single3DFA = automata.NonGreedyDFA(single3States, single3StatesAccepts) +single3DFA = automata.NonGreedyDFA(states, accepts) + +accepts = [False, True, False, False] +states = [ + {automata.DEFAULT: 0, '"': 1, + '\\': 2}, -singleStatesAccepts = [False, True, False] -singleStates = [ - {automata.DEFAULT: 0, '\\': 2, "'": 1}, {}, - {automata.DEFAULT: 0}, + + {automata.DEFAULT: 3}, + + {automata.DEFAULT: 3, '"': 1, + '\\': 2}, + ] -singleDFA = automata.DFA(singleStates, singleStatesAccepts) +doubleDFA = automata.DFA(states, accepts) + +accepts = [False, True, False, False] +states = [ + {automata.DEFAULT: 0, "'": 1, + '\\': 2}, -doubleStatesAccepts = [False, True, False] -doubleStates = [ - {automata.DEFAULT: 0, '"': 1, '\\': 2}, {}, - {automata.DEFAULT: 0}, + + {automata.DEFAULT: 3}, + + {automata.DEFAULT: 3, "'": 1, + '\\': 2}, + ] -doubleDFA = automata.DFA(doubleStates, doubleStatesAccepts) +singleDFA = automata.DFA(states, accepts) endDFAs = {"'" : singleDFA, '"' : doubleDFA, - "r" : None, - "R" : None, - "u" : None, - "U" : None, - "b" : None, - "B" : None} + 'r' : None, + 'R' : None, + 'u' : None, + 'U' : None, + 'b' : None, + 'B' : None} + +#_______________________________________________________________________ +# End of automatically generated DFA's for uniPrefix in ("", "u", "U", "b", "B"): for rawPrefix in ("", "r", "R"): From afa at codespeak.net Tue Oct 5 18:53:43 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 18:53:43 +0200 (CEST) Subject: [pypy-svn] r77617 - in pypy/branch/fast-forward/pypy/interpreter: astcompiler astcompiler/test pyparser pyparser/test Message-ID: <20101005165343.7CD09282B9C@codespeak.net> Author: afa Date: Tue Oct 5 18:53:41 2010 New Revision: 77617 Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py Log: Add new octal literal syntax 0o777 to the tokenizer. Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py Tue Oct 5 18:53:41 2010 @@ -1037,13 +1037,15 @@ if raw.startswith("0"): if len(raw) > 2 and raw[1] in "Xx": base = 16 + ## elif len(raw) > 2 and raw[1] in "Oo": # Fallback below is enough + ## base = 8 elif len(raw) > 1: base = 8 # strip leading characters i = 0 limit = len(raw) - 1 while i < limit: - if raw[i] not in "0xX": + if raw[i] not in "0xXoO": break i += 1 raw = raw[i:] Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py Tue Oct 5 18:53:41 2010 @@ -1137,6 +1137,8 @@ assert space.eq_w(get_num("-0xAAAAAAL"), space.wrap(-0xAAAAAAL)) n = get_num(str(-sys.maxint - 1)) assert space.is_true(space.isinstance(n, space.w_int)) + for num in ("0o53", "0O53", "0o0000053", "0O00053"): + assert space.eq_w(get_num(num), space.wrap(053)) def check_comprehension(self, brackets, ast_type): def brack(s): Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py Tue Oct 5 18:53:41 2010 @@ -58,6 +58,7 @@ maybe(states, groupStr(states, "lL"))) octNumber = chain(states, newArcPair(states, "0"), + maybe(states, groupStr(states, "oO")), any(states, groupStr(states, "01234567")), maybe(states, groupStr(states, "lL"))) decNumber = chain(states, Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py Tue Oct 5 18:53:41 2010 @@ -26,9 +26,9 @@ accepts = [True, True, True, True, True, True, True, True, True, True, False, True, True, True, False, False, False, False, True, False, False, True, True, - False, True, False, True, False, True, False, - True, False, False, False, True, False, False, - False, True] + True, False, True, False, True, False, True, + False, True, False, False, False, True, False, + False, False, True] states = [ {'\t': 0, '\n': 13, '\x0c': 0, '\r': 14, ' ': 0, '!': 10, '"': 16, @@ -109,21 +109,22 @@ 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - {'.': 24, '0': 22, '1': 22, '2': 22, - '3': 22, '4': 22, '5': 22, '6': 22, - '7': 22, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'L': 13, 'X': 21, 'e': 25, - 'j': 13, 'l': 13, 'x': 21}, + {'.': 25, '0': 23, '1': 23, '2': 23, + '3': 23, '4': 23, '5': 23, '6': 23, + '7': 23, '8': 24, '9': 24, 'E': 26, + 'J': 13, 'L': 13, 'O': 22, 'X': 21, + 'e': 26, 'j': 13, 'l': 13, 'o': 22, + 'x': 21}, - {'.': 24, '0': 5, '1': 5, '2': 5, + {'.': 25, '0': 5, '1': 5, '2': 5, '3': 5, '4': 5, '5': 5, '6': 5, - '7': 5, '8': 5, '9': 5, 'E': 25, - 'J': 13, 'L': 13, 'e': 25, 'j': 13, + '7': 5, '8': 5, '9': 5, 'E': 26, + 'J': 13, 'L': 13, 'e': 26, 'j': 13, 'l': 13}, - {'0': 26, '1': 26, '2': 26, '3': 26, - '4': 26, '5': 26, '6': 26, '7': 26, - '8': 26, '9': 26}, + {'0': 27, '1': 27, '2': 27, '3': 27, + '4': 27, '5': 27, '6': 27, '7': 27, + '8': 27, '9': 27}, {'*': 12, '=': 13}, @@ -141,92 +142,101 @@ {'\n': 13}, - {automata.DEFAULT: 19, '\n': 27, - "'": 28, '\\': 29}, + {automata.DEFAULT: 19, '\n': 28, + "'": 29, '\\': 30}, - {automata.DEFAULT: 20, '\n': 27, - '"': 30, '\\': 31}, + {automata.DEFAULT: 20, '\n': 28, + '"': 31, '\\': 32}, {'\n': 13, '\r': 14}, - {automata.DEFAULT: 18, '\n': 27, - '\r': 27}, + {automata.DEFAULT: 18, '\n': 28, + '\r': 28}, - {automata.DEFAULT: 19, '\n': 27, - "'": 13, '\\': 29}, + {automata.DEFAULT: 19, '\n': 28, + "'": 13, '\\': 30}, - {automata.DEFAULT: 20, '\n': 27, - '"': 13, '\\': 31}, + {automata.DEFAULT: 20, '\n': 28, + '"': 13, '\\': 32}, {'0': 21, '1': 21, '2': 21, '3': 21, '4': 21, '5': 21, '6': 21, '7': 21, '8': 21, '9': 21, 'A': 21, 'B': 21, 'C': 21, 'D': 21, 'E': 21, 'F': 21, 'L': 13, 'a': 21, 'b': 21, 'c': 21, - 'd': 21, 'e': 21, 'f': 21, 'l': 13}, - - {'.': 24, '0': 22, '1': 22, '2': 22, - '3': 22, '4': 22, '5': 22, '6': 22, - '7': 22, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'L': 13, 'e': 25, 'j': 13, + 'd': 21, 'e': 21, 'f': 21, 'l': 13}, - {'.': 24, '0': 23, '1': 23, '2': 23, + {'0': 22, '1': 22, '2': 22, '3': 22, + '4': 22, '5': 22, '6': 22, '7': 22, + 'L': 13, 'l': 13}, + + {'.': 25, '0': 23, '1': 23, '2': 23, '3': 23, '4': 23, '5': 23, '6': 23, - '7': 23, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'e': 25, 'j': 13}, + '7': 23, '8': 24, '9': 24, 'E': 26, + 'J': 13, 'L': 13, 'e': 26, 'j': 13, + 'l': 13}, - {'0': 24, '1': 24, '2': 24, '3': 24, - '4': 24, '5': 24, '6': 24, '7': 24, - '8': 24, '9': 24, 'E': 32, 'J': 13, - 'e': 32, 'j': 13}, - - {'+': 33, '-': 33, '0': 34, '1': 34, - '2': 34, '3': 34, '4': 34, '5': 34, - '6': 34, '7': 34, '8': 34, '9': 34}, - - {'0': 26, '1': 26, '2': 26, '3': 26, - '4': 26, '5': 26, '6': 26, '7': 26, - '8': 26, '9': 26, 'E': 32, 'J': 13, - 'e': 32, 'j': 13}, + {'.': 25, '0': 24, '1': 24, '2': 24, + '3': 24, '4': 24, '5': 24, '6': 24, + '7': 24, '8': 24, '9': 24, 'E': 26, + 'J': 13, 'e': 26, 'j': 13}, + + {'0': 25, '1': 25, '2': 25, '3': 25, + '4': 25, '5': 25, '6': 25, '7': 25, + '8': 25, '9': 25, 'E': 33, 'J': 13, + 'e': 33, 'j': 13}, + + {'+': 34, '-': 34, '0': 35, '1': 35, + '2': 35, '3': 35, '4': 35, '5': 35, + '6': 35, '7': 35, '8': 35, + '9': 35}, + + {'0': 27, '1': 27, '2': 27, '3': 27, + '4': 27, '5': 27, '6': 27, '7': 27, + '8': 27, '9': 27, 'E': 33, 'J': 13, + 'e': 33, 'j': 13}, {}, {"'": 13}, - {automata.DEFAULT: 35, '\n': 13, + {automata.DEFAULT: 36, '\n': 13, '\r': 14}, {'"': 13}, - {automata.DEFAULT: 36, '\n': 13, + {automata.DEFAULT: 37, '\n': 13, '\r': 14}, - {'+': 37, '-': 37, '0': 38, '1': 38, - '2': 38, '3': 38, '4': 38, '5': 38, - '6': 38, '7': 38, '8': 38, '9': 38}, - - {'0': 34, '1': 34, '2': 34, '3': 34, - '4': 34, '5': 34, '6': 34, '7': 34, - '8': 34, '9': 34}, - - {'0': 34, '1': 34, '2': 34, '3': 34, - '4': 34, '5': 34, '6': 34, '7': 34, - '8': 34, '9': 34, 'J': 13, 'j': 13}, - - {automata.DEFAULT: 35, '\n': 27, - "'": 13, '\\': 29}, - - {automata.DEFAULT: 36, '\n': 27, - '"': 13, '\\': 31}, - - {'0': 38, '1': 38, '2': 38, '3': 38, - '4': 38, '5': 38, '6': 38, '7': 38, - '8': 38, '9': 38}, - - {'0': 38, '1': 38, '2': 38, '3': 38, - '4': 38, '5': 38, '6': 38, '7': 38, - '8': 38, '9': 38, 'J': 13, 'j': 13}, + {'+': 38, '-': 38, '0': 39, '1': 39, + '2': 39, '3': 39, '4': 39, '5': 39, + '6': 39, '7': 39, '8': 39, + '9': 39}, + + {'0': 35, '1': 35, '2': 35, '3': 35, + '4': 35, '5': 35, '6': 35, '7': 35, + '8': 35, '9': 35}, + + {'0': 35, '1': 35, '2': 35, '3': 35, + '4': 35, '5': 35, '6': 35, '7': 35, + '8': 35, '9': 35, 'J': 13, + 'j': 13}, + + {automata.DEFAULT: 36, '\n': 28, + "'": 13, '\\': 30}, + + {automata.DEFAULT: 37, '\n': 28, + '"': 13, '\\': 32}, + + {'0': 39, '1': 39, '2': 39, '3': 39, + '4': 39, '5': 39, '6': 39, '7': 39, + '8': 39, '9': 39}, + + {'0': 39, '1': 39, '2': 39, '3': 39, + '4': 39, '5': 39, '6': 39, '7': 39, + '8': 39, '9': 39, 'J': 13, + 'j': 13}, ] pseudoDFA = automata.DFA(states, accepts) Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py Tue Oct 5 18:53:41 2010 @@ -108,3 +108,8 @@ self.parse("br'\\\n'") py.test.raises(SyntaxError, self.parse, "b'a\\n") + + def test_new_octal_literal(self): + self.parse('0777') + self.parse('0o777') + py.test.raises(SyntaxError, self.parse, "0o778") From arigo at codespeak.net Tue Oct 5 18:53:58 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 18:53:58 +0200 (CEST) Subject: [pypy-svn] r77618 - pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test Message-ID: <20101005165358.A78AF282B9C@codespeak.net> Author: arigo Date: Tue Oct 5 18:53:57 2010 New Revision: 77618 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_transformed_gc.py Log: Fix this test, now that we can (slowly) run test_transformed_gc with the gc flag DEBUG=True. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_transformed_gc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_transformed_gc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_transformed_gc.py Tue Oct 5 18:53:57 2010 @@ -1315,7 +1315,8 @@ def test_immutable_to_old_promotion(self): run, transformer = self.runner("immutable_to_old_promotion", transformer=True) run([1, 4]) - if not transformer.GCClass.prebuilt_gc_objects_are_static_roots: + if (not transformer.GCClass.prebuilt_gc_objects_are_static_roots + and not transformer.GCClass.DEBUG): assert len(transformer.layoutbuilder.addresses_of_static_ptrs) == 0 else: assert len(transformer.layoutbuilder.addresses_of_static_ptrs) >= 4 From arigo at codespeak.net Tue Oct 5 18:54:25 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 18:54:25 +0200 (CEST) Subject: [pypy-svn] r77619 - pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform Message-ID: <20101005165425.2A314282B9C@codespeak.net> Author: arigo Date: Tue Oct 5 18:54:23 2010 New Revision: 77619 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py Log: Fix: confusion between addresses and pointers. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py Tue Oct 5 18:54:23 2010 @@ -1031,7 +1031,8 @@ # a constant won't be ok return if v_newvalue.concretetype == llmemory.HiddenGcRef32: - v_newvalue = self._fetch_unpacked_pointer(hop, v_newvalue) + v_newvalue = self._fetch_unpacked_address(hop, v_newvalue) + assert v_newvalue.concretetype == llmemory.Address if isinstance(v_newvalue, Constant): # comes from a Constant -- skip return @@ -1057,12 +1058,16 @@ v_newvalue, v_structaddr]) - def _fetch_unpacked_pointer(self, hop, v_value): + def _fetch_unpacked_address(self, hop, v_value): # optimization for the common case where this setfield is preceded # by v_value = hide_into_adr32(v_normal_pointer) for op in hop.llops[::-1]: if op.opname == 'hide_into_adr32' and op.result == v_value: - return op.args[0] + v_value = op.args[0] + if v_value.concretetype != llmemory.Address: + v_value = hop.genop("cast_ptr_to_adr", [v_value], + resulttype = llmemory.Address) + return v_value else: return hop.genop("show_from_adr32", [v_value], resulttype = llmemory.Address) From arigo at codespeak.net Tue Oct 5 19:02:35 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 19:02:35 +0200 (CEST) Subject: [pypy-svn] r77620 - in pypy/branch/32ptr-on-64bit/pypy: rpython/lltypesystem rpython/lltypesystem/test rpython/memory/gc rpython/memory/gctransform translator/c translator/c/src Message-ID: <20101005170235.DCF62282B9C@codespeak.net> Author: arigo Date: Tue Oct 5 19:02:33 2010 New Revision: 77620 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py pypy/branch/32ptr-on-64bit/pypy/translator/c/funcgen.py pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h Log: Rename {hide_into,show_from}_adr32 into *_ptr32, which matches the current implementation better. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lloperation.py Tue Oct 5 19:02:33 2010 @@ -426,8 +426,8 @@ 'gc_gettypeptr_group': LLOp(canfold=True), 'get_member_index': LLOp(canfold=True), - 'hide_into_adr32': LLOp(canrun=True), - 'show_from_adr32': LLOp(canrun=True), + 'hide_into_ptr32': LLOp(canrun=True), + 'show_from_ptr32': LLOp(canrun=True), # __________ used by the JIT ________ Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/opimpl.py Tue Oct 5 19:02:33 2010 @@ -526,14 +526,14 @@ def op_shrink_array(array, smallersize): return False -def op_hide_into_adr32(ptr): +def op_hide_into_ptr32(ptr): if lltype.typeOf(ptr) == llmemory.Address: if not ptr: return lltype.nullptr(llmemory.HiddenGcRef32.TO) ptr = ptr.ptr return lltype.cast_opaque_ptr(llmemory.HiddenGcRef32, ptr) -def op_show_from_adr32(RESTYPE, ptr32): +def op_show_from_ptr32(RESTYPE, ptr32): if RESTYPE == llmemory.Address: if not ptr32: return llmemory.NULL @@ -541,7 +541,7 @@ ptr = lltype.cast_opaque_ptr(PTRTYPE, ptr32) return llmemory.cast_ptr_to_adr(ptr) return lltype.cast_opaque_ptr(RESTYPE, ptr32) -op_show_from_adr32.need_result_type = True +op_show_from_ptr32.need_result_type = True # ____________________________________________________________ Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py Tue Oct 5 19:02:33 2010 @@ -50,7 +50,7 @@ ptr = self.baserepr.convert_const(value) T = lltype.typeOf(ptr) assert T == self.BASETYPE - return llop.hide_into_adr32(self.lowleveltype, ptr) + return llop.hide_into_ptr32(self.lowleveltype, ptr) def get_ll_eq_function(self): if self.baserepr.get_ll_eq_function() is not None: @@ -63,7 +63,7 @@ BASETYPE = self.BASETYPE # def ll_hiddengcref32_hash(x): - x = llop.show_from_adr32(BASETYPE, x) + x = llop.show_from_ptr32(BASETYPE, x) return basefunc(x) # self.ll_hash_function = ll_hiddengcref32_hash @@ -77,7 +77,7 @@ BASETYPE = self.BASETYPE # def ll_hiddengcref32_fasthash(x): - x = llop.show_from_adr32(BASETYPE, x) + x = llop.show_from_ptr32(BASETYPE, x) return basefunc(x) # self.ll_fasthash_function = ll_hiddengcref32_hash @@ -98,12 +98,12 @@ def convert_from_to((r_from, r_to), v, llops): assert r_from.lowleveltype.TO._gckind == 'gc' assert not isinstance(r_from.lowleveltype.TO, lltype.GcOpaqueType) - return llops.genop('hide_into_adr32', [v], + return llops.genop('hide_into_ptr32', [v], resulttype=llmemory.HiddenGcRef32) class __extend__(pairtype(CompressedGcRefRepr, Repr)): def convert_from_to((r_from, r_to), v, llops): assert r_to.lowleveltype.TO._gckind == 'gc' assert not isinstance(r_to.lowleveltype.TO, lltype.GcOpaqueType) - return llops.genop('show_from_adr32', [v], + return llops.genop('show_from_ptr32', [v], resulttype=r_to.lowleveltype) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_llmemory.py Tue Oct 5 19:02:33 2010 @@ -654,6 +654,6 @@ from pypy.rpython.lltypesystem.lloperation import llop S = lltype.GcStruct('S') p = lltype.malloc(S) - q = llop.hide_into_adr32(HiddenGcRef32, p) - r = llop.show_from_adr32(lltype.Ptr(S), q) + q = llop.hide_into_ptr32(HiddenGcRef32, p) + r = llop.show_from_ptr32(lltype.Ptr(S), q) assert r == p Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py Tue Oct 5 19:02:33 2010 @@ -215,11 +215,11 @@ ofs = llmemory.remove_odd_value_marker(ofs) item = obj + ofs item = llmemory.cast_adr_to_ptr(item, lltype.Ptr(HIDDENGCREFFIELD)) - address = llop.show_from_adr32(llmemory.Address, item[0]) + address = llop.show_from_ptr32(llmemory.Address, item[0]) if self.is_valid_gc_object(address): newaddr = callback(address, arg) if newaddr is not None: - item[0] = llop.hide_into_adr32(llmemory.HiddenGcRef32, + item[0] = llop.hide_into_ptr32(llmemory.HiddenGcRef32, newaddr) else: # common case Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py Tue Oct 5 19:02:33 2010 @@ -1060,16 +1060,16 @@ def _fetch_unpacked_address(self, hop, v_value): # optimization for the common case where this setfield is preceded - # by v_value = hide_into_adr32(v_normal_pointer) + # by v_value = hide_into_ptr32(v_normal_pointer) for op in hop.llops[::-1]: - if op.opname == 'hide_into_adr32' and op.result == v_value: + if op.opname == 'hide_into_ptr32' and op.result == v_value: v_value = op.args[0] if v_value.concretetype != llmemory.Address: v_value = hop.genop("cast_ptr_to_adr", [v_value], resulttype = llmemory.Address) return v_value else: - return hop.genop("show_from_adr32", [v_value], + return hop.genop("show_from_ptr32", [v_value], resulttype = llmemory.Address) def transform_getfield_typeptr(self, hop): Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/funcgen.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/funcgen.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/funcgen.py Tue Oct 5 19:02:33 2010 @@ -647,17 +647,28 @@ self.expr(op.args[0]))) return '\t'.join(result) - OP_CAST_PTR_TO_ADR = OP_CAST_POINTER - OP_CAST_ADR_TO_PTR = OP_CAST_POINTER + def OP_CAST_PTR_TO_ADR(self, op): + # detect a case that is not supported and should be handled with + # show_from_ptr32. + if op.args[0].concretetype == HiddenGcRef32: + raise Exception("cast_ptr_to_adr(hiddengcref32): not supported") + return self.OP_CAST_POINTER(op) + + def OP_CAST_ADR_TO_PTR(self, op): + # detect a case that is not supported and should be handled with + # hide_into_ptr32. + if op.result.concretetype == HiddenGcRef32: + raise Exception("hiddengcref32 = cast_adr_to_ptr(): not supported") + return self.OP_CAST_POINTER(op) def OP_CAST_OPAQUE_PTR(self, op): + # detect cases that are not supported and should be handled with + # hide_into_ptr32 or show_from_ptr32. if op.result.concretetype == HiddenGcRef32: - return 'OP_HIDE_INTO_ADR32(%s, %s);' % (self.expr(op.args[0]), - self.expr(op.result)) + raise Exception("hiddengcref32 = cast_opaque_ptr(): not supported") if op.args[0].concretetype == HiddenGcRef32: - return 'OP_SHOW_FROM_ADR32(%s, %s);' % (self.expr(op.args[0]), - self.expr(op.result)) - return self.OP_CAST_OPAQUE_PTR(op) + raise Exception("cast_opaque_ptr(hiddengcref32): not supported") + return self.OP_CAST_POINTER(op) def OP_CAST_INT_TO_PTR(self, op): TYPE = self.lltypemap(op.result) Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py Tue Oct 5 19:02:33 2010 @@ -923,7 +923,7 @@ \tfor (p = pypy_hiddengcref32; p[0] != NULL; p += 2) \t{ \t\thiddengcref32_t h; -\t\tOP_HIDE_INTO_ADR32_CHECK((p[1]), h); +\t\tOP_HIDE_INTO_PTR32_CHECK((p[1]), h); \t\t*(hiddengcref32_t*)(p[0]) = h; \t} } Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py Tue Oct 5 19:02:33 2010 @@ -790,14 +790,14 @@ expr = '0.0 /* patched later by %sinfinity */' % ( '-+'[value > 0]) elif (TYPE == llmemory.HiddenGcRef32 - and value and not hasattr(value, 'dummy_value')): + and value and not hasattr(value._obj, 'dummy_value')): realobj = value._obj.container realvalue = cast_opaque_ptr(Ptr(typeOf(realobj)), value) name = db.get(realvalue) db.late_initializations_hiddengcref32.append((access_expr, name)) if not name.startswith('('): name = '(%s)' % name - expr = '0 /*HIDE_INTO_ADR32%s*/' % name + expr = '0 /*HIDE_INTO_PTR32%s*/' % name else: expr = db.get(value) if TYPE is Void: Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py Tue Oct 5 19:02:33 2010 @@ -148,17 +148,17 @@ return 'NULL' def name_hiddengcref32(value, db): - if hasattr(value, 'dummy_value'): - return '1 /* dummy value */' - elif not value: + if not value: return '0' # NULL + elif hasattr(value._obj, 'dummy_value'): + return '1 /* dummy value */' # The only prebuilt HiddenGcRef32 that should occur in a translated C # program occur as fields or items of a GcStruct or GcArray. They are # handled by late_initializations_hiddengcref32. realobj = value._obj.container realvalue = cast_opaque_ptr(Ptr(typeOf(realobj)), value) db.get(realvalue) # force following dependencies - return 'HIDE_INTO_ADR32(???) /* see primitive.py, name_hiddengcref32() */' + return 'HIDE_INTO_PTR32(???) /* see primitive.py, name_hiddengcref32() */' def name_small_integer(value, db): """Works for integers of size at most INT or UINT.""" Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h Tue Oct 5 19:02:33 2010 @@ -17,13 +17,13 @@ #endif -#define OP_SHOW_FROM_ADR32(x, r) r = (void*)(((unsigned long)(x)) << 3) +#define OP_SHOW_FROM_PTR32(x, r) r = (void*)(((unsigned long)(x)) << 3) -#define OP_HIDE_INTO_ADR32_CHECK(x, r) \ +#define OP_HIDE_INTO_PTR32_CHECK(x, r) \ r = (hiddengcref32_t)(((unsigned long)(x)) >> 3); \ if ((void*)(((unsigned long)(r)) << 3) != (x)) \ RPyPointerTooBig() -#define OP_HIDE_INTO_ADR32(x, r) \ +#define OP_HIDE_INTO_PTR32(x, r) \ RPyAssert(!(((long)(x)) & 0x7FFFFFFF8), "Pointer too big or misaligned"); \ r = (hiddengcref32_t)(((unsigned long)(x)) >> 3) From arigo at codespeak.net Tue Oct 5 19:10:30 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 19:10:30 +0200 (CEST) Subject: [pypy-svn] r77621 - pypy/branch/32ptr-on-64bit/pypy/translator/c/src Message-ID: <20101005171030.8EDE836C22E@codespeak.net> Author: arigo Date: Tue Oct 5 19:10:29 2010 New Revision: 77621 Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h Log: Bah. Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h Tue Oct 5 19:10:29 2010 @@ -25,5 +25,5 @@ RPyPointerTooBig() #define OP_HIDE_INTO_PTR32(x, r) \ - RPyAssert(!(((long)(x)) & 0x7FFFFFFF8), "Pointer too big or misaligned"); \ + RPyAssert(!(((long)(x)) & ~0x7FFFFFFF8), "Pointer too big or misaligned"); \ r = (hiddengcref32_t)(((unsigned long)(x)) >> 3) From arigo at codespeak.net Tue Oct 5 19:10:48 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 19:10:48 +0200 (CEST) Subject: [pypy-svn] r77622 - in pypy/branch/32ptr-on-64bit/pypy: config rpython rpython/memory/gctransform Message-ID: <20101005171048.39D8A282B9C@codespeak.net> Author: arigo Date: Tue Oct 5 19:10:46 2010 New Revision: 77622 Modified: pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py pypy/branch/32ptr-on-64bit/pypy/rpython/rbuiltin.py Log: Fix fix fix... Modified: pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py Tue Oct 5 19:10:46 2010 @@ -197,7 +197,8 @@ default=False, cmdline="--compressptr", requires=[("translation.type_system", "lltype"), ("translation.taggedpointers", False), - ("translation.gc", "minimark")] # for now + ("translation.gc", "minimark"), # for now + ("translation.gcrootfinder", "shadowstack")] # xxx + [("compressptr (64-bit only)", True)]*(not IS_64_BITS)), # options for ootype Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py Tue Oct 5 19:10:46 2010 @@ -1166,6 +1166,9 @@ c_k = rmodel.inputconst(lltype.Signed, k) v_newaddr = hop.genop("raw_load", [base_addr, c_type, c_k], resulttype=llmemory.Address) + if var.concretetype == llmemory.HiddenGcRef32: + v_newaddr = gen_cast(hop.llops, llmemory.HiddenGcRef32, + v_newaddr) hop.genop("gc_reload_possibly_moved", [v_newaddr, var]) def compute_borrowed_vars(self, graph): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/rbuiltin.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/rbuiltin.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/rbuiltin.py Tue Oct 5 19:10:46 2010 @@ -464,12 +464,20 @@ else: return llops.genop('cast_pointer', [v_value], resulttype = TGT) elif ORIG == llmemory.Address: - return llops.genop('cast_adr_to_ptr', [v_value], resulttype = TGT) + if TGT == llmemory.HiddenGcRef32: + return llops.genop('hide_into_ptr32', [v_value], + resulttype = TGT) + else: + return llops.genop('cast_adr_to_ptr', [v_value], + resulttype = TGT) elif isinstance(ORIG, lltype.Primitive): v_value = gen_cast(llops, lltype.Signed, v_value) return llops.genop('cast_int_to_ptr', [v_value], resulttype=TGT) elif TGT == llmemory.Address and isinstance(ORIG, lltype.Ptr): - return llops.genop('cast_ptr_to_adr', [v_value], resulttype = TGT) + if ORIG == llmemory.HiddenGcRef32: + return llops.genop('show_from_ptr32', [v_value], resulttype = TGT) + else: + return llops.genop('cast_ptr_to_adr', [v_value], resulttype = TGT) elif isinstance(TGT, lltype.Primitive): if isinstance(ORIG, lltype.Ptr): v_value = llops.genop('cast_ptr_to_int', [v_value], From afa at codespeak.net Tue Oct 5 19:31:41 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 19:31:41 +0200 (CEST) Subject: [pypy-svn] r77623 - in pypy/branch/fast-forward/pypy/objspace/std: . test Message-ID: <20101005173141.783B4282B90@codespeak.net> Author: afa Date: Tue Oct 5 19:31:39 2010 New Revision: 77623 Modified: pypy/branch/fast-forward/pypy/objspace/std/strutil.py pypy/branch/fast-forward/pypy/objspace/std/test/test_floatobject.py Log: Add float('infinity') Modified: pypy/branch/fast-forward/pypy/objspace/std/strutil.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/strutil.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/strutil.py Tue Oct 5 19:31:39 2010 @@ -178,10 +178,12 @@ low = s.lower() - if low == "-inf": + if low == "-inf" or low == "-infinity": return -INFINITY elif low == "inf" or low == "+inf": return INFINITY + elif low == "infinity" or low == "+infinity": + return INFINITY elif low == "nan" or low == "-nan" or low == "+nan": return NAN Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_floatobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/test/test_floatobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/test/test_floatobject.py Tue Oct 5 19:31:39 2010 @@ -98,6 +98,9 @@ assert float("-INf") == -inf assert str(inf) == "inf" assert str(-inf) == "-inf" + assert str(float("infinity")) == 'inf' + assert str(float("+infinity")) == 'inf' + assert str(float("-infinity")) == '-inf' assert str(float("nan")) == "nan" assert str(float("-nAn")) == "nan" assert repr(inf) == "inf" From arigo at codespeak.net Tue Oct 5 19:32:19 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 5 Oct 2010 19:32:19 +0200 (CEST) Subject: [pypy-svn] r77624 - in pypy/branch/32ptr-on-64bit/pypy/rpython/memory: . gc Message-ID: <20101005173219.83129282B90@codespeak.net> Author: arigo Date: Tue Oct 5 19:32:18 2010 New Revision: 77624 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py Log: Hack goes on. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py Tue Oct 5 19:32:18 2010 @@ -178,6 +178,7 @@ Typically, 'callback' is a bound method and 'arg' can be None. """ typeid = self.get_type_id(obj) + # XXX missing performance shortcut for GcArray(HiddenGcRef32) if self.is_gcarrayofgcptr(typeid): # a performance shortcut for GcArray(gcptr) length = (obj + llmemory.gcarrayofptr_lengthoffset).signed[0] Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py Tue Oct 5 19:32:18 2010 @@ -131,6 +131,7 @@ T_IS_GCARRAY_OF_GCPTR = 0x40000 T_IS_WEAKREF = 0x80000 T_IS_RPYTHON_INSTANCE = 0x100000 # the type is a subclass of OBJECT +T_IS_GCARRAY_OF_HIDDENGCREF32 = 0x200000 T_KEY_MASK = intmask(0xFF000000) T_KEY_VALUE = intmask(0x7A000000) # bug detection only @@ -175,7 +176,10 @@ ARRAY = TYPE if (isinstance(ARRAY.OF, lltype.Ptr) and ARRAY.OF.TO._gckind == 'gc'): - infobits |= T_IS_GCARRAY_OF_GCPTR + if ARRAY.OF == llmemory.HiddenGcRef32: + pass # infobits |= T_IS_GCARRAY_OF_HIDDENGCREF32 + else: + infobits |= T_IS_GCARRAY_OF_GCPTR varinfo.ofstolength = llmemory.ArrayLengthOffset(ARRAY) varinfo.ofstovar = llmemory.itemoffsetof(TYPE, 0) assert isinstance(ARRAY, lltype.Array) From afa at codespeak.net Tue Oct 5 19:54:44 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 19:54:44 +0200 (CEST) Subject: [pypy-svn] r77625 - in pypy/branch/fast-forward/pypy: objspace/std objspace/std/test rlib Message-ID: <20101005175444.8CDAB282B90@codespeak.net> Author: afa Date: Tue Oct 5 19:54:42 2010 New Revision: 77625 Modified: pypy/branch/fast-forward/pypy/objspace/std/floatobject.py pypy/branch/fast-forward/pypy/objspace/std/test/test_floatobject.py pypy/branch/fast-forward/pypy/rlib/rarithmetic.py pypy/branch/fast-forward/pypy/rlib/rbigint.py Log: Ensure that int(float('nan')) raises a ValueError Modified: pypy/branch/fast-forward/pypy/objspace/std/floatobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/floatobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/floatobject.py Tue Oct 5 19:54:42 2010 @@ -74,21 +74,20 @@ try: return W_LongObject.fromfloat(w_floatobj.floatval) except OverflowError: + if isnan(w_floatobj.floatval): + raise OperationError( + space.w_ValueError, + space.wrap("cannot convert float NaN to integer")) raise OperationError(space.w_OverflowError, space.wrap("cannot convert float infinity to long")) def trunc__Float(space, w_floatobj): whole = math.modf(w_floatobj.floatval)[1] try: - i = ovfcheck_float_to_int(whole) + value = ovfcheck_float_to_int(whole) except OverflowError: - pass + return long__Float(space, w_floatobj) else: - return space.wrap(i) - try: - return W_LongObject.fromfloat(w_floatobj.floatval) - except OverflowError: - raise OperationError(space.w_OverflowError, - space.wrap("cannot convert infinity to long")) + return space.newint(value) def float_w__Float(space, w_float): return w_float.floatval Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_floatobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/test/test_floatobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/test/test_floatobject.py Tue Oct 5 19:54:42 2010 @@ -89,6 +89,11 @@ assert int(42.1234) == 42 assert int(4e10) == 40000000000L + raises(OverflowError, int, float('inf')) + raises(OverflowError, long, float('inf')) + raises(ValueError, int, float('nan')) + raises(ValueError, long, float('nan')) + def test_float_string(self): assert 42 == float("42") assert 42.25 == float("42.25") Modified: pypy/branch/fast-forward/pypy/rlib/rarithmetic.py ============================================================================== --- pypy/branch/fast-forward/pypy/rlib/rarithmetic.py (original) +++ pypy/branch/fast-forward/pypy/rlib/rarithmetic.py Tue Oct 5 19:54:42 2010 @@ -217,6 +217,8 @@ # successfully be casted to an int. if sys.maxint == 2147483647: def ovfcheck_float_to_int(x): + if isnan(x): + raise OverflowError if -2147483649.0 < x < 2147483648.0: return int(x) raise OverflowError @@ -225,6 +227,8 @@ # Note the "<= x <" here, as opposed to "< x <" above. # This is justified by test_typed in translator/c/test. def ovfcheck_float_to_int(x): + if isnan(x): + raise OverflowError if -9223372036854776832.0 <= x < 9223372036854775296.0: return int(x) raise OverflowError Modified: pypy/branch/fast-forward/pypy/rlib/rbigint.py ============================================================================== --- pypy/branch/fast-forward/pypy/rlib/rbigint.py (original) +++ pypy/branch/fast-forward/pypy/rlib/rbigint.py Tue Oct 5 19:54:42 2010 @@ -1,5 +1,5 @@ from pypy.rlib.rarithmetic import LONG_BIT, intmask, r_uint, r_ulonglong -from pypy.rlib.rarithmetic import ovfcheck, r_longlong, widen +from pypy.rlib.rarithmetic import ovfcheck, r_longlong, widen, isinf, isnan from pypy.rlib.debug import make_sure_not_resized import math, sys @@ -109,7 +109,7 @@ def fromfloat(dval): """ Create a new bigint object from a float """ neg = 0 - if isinf(dval): + if isinf(dval) or isnan(dval): raise OverflowError if dval < 0.0: neg = 1 @@ -1259,9 +1259,6 @@ assert x > 0.0 return x * sign, exponent -def isinf(x): - return x != 0.0 and x / 2 == x - ##def ldexp(x, exp): ## assert type(x) is float ## lb1 = LONG_BIT - 1 From afa at codespeak.net Tue Oct 5 20:00:58 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 20:00:58 +0200 (CEST) Subject: [pypy-svn] r77626 - in pypy/branch/fast-forward/pypy/interpreter: astcompiler astcompiler/test pyparser pyparser/test Message-ID: <20101005180058.4F1BC282B90@codespeak.net> Author: afa Date: Tue Oct 5 20:00:56 2010 New Revision: 77626 Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py Log: Add Binary literals: 0b11101 Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py Tue Oct 5 20:00:56 2010 @@ -1037,6 +1037,8 @@ if raw.startswith("0"): if len(raw) > 2 and raw[1] in "Xx": base = 16 + elif len(raw) > 2 and raw[1] in "Bb": + base = 2 ## elif len(raw) > 2 and raw[1] in "Oo": # Fallback below is enough ## base = 8 elif len(raw) > 1: @@ -1045,7 +1047,7 @@ i = 0 limit = len(raw) - 1 while i < limit: - if raw[i] not in "0xXoO": + if raw[i] not in "0xXoOBb": break i += 1 raw = raw[i:] Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py Tue Oct 5 20:00:56 2010 @@ -1139,6 +1139,8 @@ assert space.is_true(space.isinstance(n, space.w_int)) for num in ("0o53", "0O53", "0o0000053", "0O00053"): assert space.eq_w(get_num(num), space.wrap(053)) + for num in ("0b00101", "0B00101", "0b101", "0B101"): + assert space.eq_w(get_num(num), space.wrap(5)) def check_comprehension(self, brackets, ast_type): def brack(s): Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py Tue Oct 5 20:00:56 2010 @@ -61,11 +61,16 @@ maybe(states, groupStr(states, "oO")), any(states, groupStr(states, "01234567")), maybe(states, groupStr(states, "lL"))) + binNumber = chain(states, + newArcPair(states, "0"), + groupStr(states, "bB"), + any(states, groupStr(states, "01")), + maybe(states, groupStr(states, "lL"))) decNumber = chain(states, groupStr(states, "123456789"), any(states, makeDigits()), maybe(states, groupStr(states, "lL"))) - intNumber = group(states, hexNumber, octNumber, decNumber) + intNumber = group(states, hexNumber, octNumber, binNumber, decNumber) # ____________________________________________________________ # Exponents def makeExp (): Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py Tue Oct 5 20:00:56 2010 @@ -26,7 +26,7 @@ accepts = [True, True, True, True, True, True, True, True, True, True, False, True, True, True, False, False, False, False, True, False, False, True, True, - True, False, True, False, True, False, True, + True, True, False, True, False, True, False, True, False, True, False, False, False, True, False, False, False, True] states = [ @@ -109,22 +109,22 @@ 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - {'.': 25, '0': 23, '1': 23, '2': 23, + {'.': 26, '0': 23, '1': 23, '2': 23, '3': 23, '4': 23, '5': 23, '6': 23, - '7': 23, '8': 24, '9': 24, 'E': 26, - 'J': 13, 'L': 13, 'O': 22, 'X': 21, - 'e': 26, 'j': 13, 'l': 13, 'o': 22, - 'x': 21}, + '7': 23, '8': 25, '9': 25, 'B': 24, + 'E': 27, 'J': 13, 'L': 13, 'O': 22, + 'X': 21, 'b': 24, 'e': 27, 'j': 13, + 'l': 13, 'o': 22, 'x': 21}, - {'.': 25, '0': 5, '1': 5, '2': 5, + {'.': 26, '0': 5, '1': 5, '2': 5, '3': 5, '4': 5, '5': 5, '6': 5, - '7': 5, '8': 5, '9': 5, 'E': 26, - 'J': 13, 'L': 13, 'e': 26, 'j': 13, + '7': 5, '8': 5, '9': 5, 'E': 27, + 'J': 13, 'L': 13, 'e': 27, 'j': 13, 'l': 13}, - {'0': 27, '1': 27, '2': 27, '3': 27, - '4': 27, '5': 27, '6': 27, '7': 27, - '8': 27, '9': 27}, + {'0': 28, '1': 28, '2': 28, '3': 28, + '4': 28, '5': 28, '6': 28, '7': 28, + '8': 28, '9': 28}, {'*': 12, '=': 13}, @@ -142,22 +142,22 @@ {'\n': 13}, - {automata.DEFAULT: 19, '\n': 28, - "'": 29, '\\': 30}, + {automata.DEFAULT: 19, '\n': 29, + "'": 30, '\\': 31}, - {automata.DEFAULT: 20, '\n': 28, - '"': 31, '\\': 32}, + {automata.DEFAULT: 20, '\n': 29, + '"': 32, '\\': 33}, {'\n': 13, '\r': 14}, - {automata.DEFAULT: 18, '\n': 28, - '\r': 28}, + {automata.DEFAULT: 18, '\n': 29, + '\r': 29}, - {automata.DEFAULT: 19, '\n': 28, - "'": 13, '\\': 30}, + {automata.DEFAULT: 19, '\n': 29, + "'": 13, '\\': 31}, - {automata.DEFAULT: 20, '\n': 28, - '"': 13, '\\': 32}, + {automata.DEFAULT: 20, '\n': 29, + '"': 13, '\\': 33}, {'0': 21, '1': 21, '2': 21, '3': 21, '4': 21, '5': 21, '6': 21, '7': 21, @@ -171,71 +171,74 @@ '4': 22, '5': 22, '6': 22, '7': 22, 'L': 13, 'l': 13}, - {'.': 25, '0': 23, '1': 23, '2': 23, + {'.': 26, '0': 23, '1': 23, '2': 23, '3': 23, '4': 23, '5': 23, '6': 23, - '7': 23, '8': 24, '9': 24, 'E': 26, - 'J': 13, 'L': 13, 'e': 26, 'j': 13, + '7': 23, '8': 25, '9': 25, 'E': 27, + 'J': 13, 'L': 13, 'e': 27, 'j': 13, 'l': 13}, - {'.': 25, '0': 24, '1': 24, '2': 24, - '3': 24, '4': 24, '5': 24, '6': 24, - '7': 24, '8': 24, '9': 24, 'E': 26, - 'J': 13, 'e': 26, 'j': 13}, - - {'0': 25, '1': 25, '2': 25, '3': 25, - '4': 25, '5': 25, '6': 25, '7': 25, - '8': 25, '9': 25, 'E': 33, 'J': 13, - 'e': 33, 'j': 13}, - - {'+': 34, '-': 34, '0': 35, '1': 35, - '2': 35, '3': 35, '4': 35, '5': 35, - '6': 35, '7': 35, '8': 35, - '9': 35}, - - {'0': 27, '1': 27, '2': 27, '3': 27, - '4': 27, '5': 27, '6': 27, '7': 27, - '8': 27, '9': 27, 'E': 33, 'J': 13, - 'e': 33, 'j': 13}, + {'0': 24, '1': 24, 'L': 13, + 'l': 13}, + + {'.': 26, '0': 25, '1': 25, '2': 25, + '3': 25, '4': 25, '5': 25, '6': 25, + '7': 25, '8': 25, '9': 25, 'E': 27, + 'J': 13, 'e': 27, 'j': 13}, + + {'0': 26, '1': 26, '2': 26, '3': 26, + '4': 26, '5': 26, '6': 26, '7': 26, + '8': 26, '9': 26, 'E': 34, 'J': 13, + 'e': 34, 'j': 13}, + + {'+': 35, '-': 35, '0': 36, '1': 36, + '2': 36, '3': 36, '4': 36, '5': 36, + '6': 36, '7': 36, '8': 36, + '9': 36}, + + {'0': 28, '1': 28, '2': 28, '3': 28, + '4': 28, '5': 28, '6': 28, '7': 28, + '8': 28, '9': 28, 'E': 34, 'J': 13, + 'e': 34, 'j': 13}, {}, {"'": 13}, - {automata.DEFAULT: 36, '\n': 13, + {automata.DEFAULT: 37, '\n': 13, '\r': 14}, {'"': 13}, - {automata.DEFAULT: 37, '\n': 13, + {automata.DEFAULT: 38, '\n': 13, '\r': 14}, - {'+': 38, '-': 38, '0': 39, '1': 39, - '2': 39, '3': 39, '4': 39, '5': 39, - '6': 39, '7': 39, '8': 39, - '9': 39}, - - {'0': 35, '1': 35, '2': 35, '3': 35, - '4': 35, '5': 35, '6': 35, '7': 35, - '8': 35, '9': 35}, - - {'0': 35, '1': 35, '2': 35, '3': 35, - '4': 35, '5': 35, '6': 35, '7': 35, - '8': 35, '9': 35, 'J': 13, + {'+': 39, '-': 39, '0': 40, '1': 40, + '2': 40, '3': 40, '4': 40, '5': 40, + '6': 40, '7': 40, '8': 40, + '9': 40}, + + {'0': 36, '1': 36, '2': 36, '3': 36, + '4': 36, '5': 36, '6': 36, '7': 36, + '8': 36, '9': 36}, + + {'0': 36, '1': 36, '2': 36, '3': 36, + '4': 36, '5': 36, '6': 36, '7': 36, + '8': 36, '9': 36, 'J': 13, 'j': 13}, - {automata.DEFAULT: 36, '\n': 28, - "'": 13, '\\': 30}, + {automata.DEFAULT: 37, '\n': 29, + "'": 13, '\\': 31}, - {automata.DEFAULT: 37, '\n': 28, - '"': 13, '\\': 32}, + {automata.DEFAULT: 38, '\n': 29, + '"': 13, '\\': 33}, - {'0': 39, '1': 39, '2': 39, '3': 39, - '4': 39, '5': 39, '6': 39, '7': 39, - '8': 39, '9': 39}, - - {'0': 39, '1': 39, '2': 39, '3': 39, - '4': 39, '5': 39, '6': 39, '7': 39, - '8': 39, '9': 39, 'J': 13, + {'0': 40, '1': 40, '2': 40, '3': 40, + '4': 40, '5': 40, '6': 40, '7': 40, + '8': 40, '9': 40}, + + {'0': 40, '1': 40, '2': 40, '3': 40, + '4': 40, '5': 40, '6': 40, '7': 40, + '8': 40, '9': 40, 'J': 13, 'j': 13}, ] Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py Tue Oct 5 20:00:56 2010 @@ -112,4 +112,10 @@ def test_new_octal_literal(self): self.parse('0777') self.parse('0o777') + self.parse('0o777L') py.test.raises(SyntaxError, self.parse, "0o778") + + def test_new_binary_literal(self): + self.parse('0b1101') + self.parse('0b0l') + py.test.raises(SyntaxError, self.parse, "0b112") From afa at codespeak.net Tue Oct 5 20:12:51 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 20:12:51 +0200 (CEST) Subject: [pypy-svn] r77627 - in pypy/branch/fast-forward/pypy/objspace/std: . test Message-ID: <20101005181251.B9D7F282B90@codespeak.net> Author: afa Date: Tue Oct 5 20:12:50 2010 New Revision: 77627 Modified: pypy/branch/fast-forward/pypy/objspace/std/strutil.py pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py Log: Fix int("0b10", 2) and long("0o77L", 8) Modified: pypy/branch/fast-forward/pypy/objspace/std/strutil.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/strutil.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/strutil.py Tue Oct 5 20:12:50 2010 @@ -52,7 +52,9 @@ if base == 0: if s.startswith('0x') or s.startswith('0X'): base = 16 - elif s.startswith('0'): + elif s.startswith('0b') or s.startswith('0B'): + base = 2 + elif s.startswith('0'): # also covers the '0o' case base = 8 else: base = 10 @@ -64,6 +66,10 @@ self.error() if base == 16 and (s.startswith('0x') or s.startswith('0X')): s = s[2:] + if base == 8 and (s.startswith('0o') or s.startswith('0O')): + s = s[2:] + if base == 2 and (s.startswith('0b') or s.startswith('0B')): + s = s[2:] self.s = s self.n = len(s) self.i = 0 Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py Tue Oct 5 20:12:50 2010 @@ -52,6 +52,9 @@ ('0X', 0, 0), # " " ('0x', 16, 0), # " " ('0X', 16, 0), # " " + ('0b11', 2, 3), + ('0B10', 2, 2), + ('0o77', 8, 63), ] for s, base, expected in cases: assert string_to_int(s, base) == expected @@ -65,6 +68,8 @@ space = self.space cases = ['0x123', # must use base 0 or 16 ' 0X12 ', + '0b01', + '0o01', '', '++12', '+-12', From afa at codespeak.net Tue Oct 5 20:13:29 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 20:13:29 +0200 (CEST) Subject: [pypy-svn] r77628 - pypy/branch/fast-forward/lib-python Message-ID: <20101005181329.07D05282B90@codespeak.net> Author: afa Date: Tue Oct 5 20:13:28 2010 New Revision: 77628 Modified: pypy/branch/fast-forward/lib-python/TODO Log: 2 tasks finished Modified: pypy/branch/fast-forward/lib-python/TODO ============================================================================== --- pypy/branch/fast-forward/lib-python/TODO (original) +++ pypy/branch/fast-forward/lib-python/TODO Tue Oct 5 20:13:28 2010 @@ -12,10 +12,6 @@ - Missing builtin: bytearray (possibly reuse module.__pypy__.bytebuffer) -- Octal literals: 0o777 - -- float('infinity'), float('nan') - - Seen in test_inspect, this has never worked in pypy:: assert eval('a', None, dict(a=42)) == 42 From agaynor at codespeak.net Tue Oct 5 20:35:34 2010 From: agaynor at codespeak.net (agaynor at codespeak.net) Date: Tue, 5 Oct 2010 20:35:34 +0200 (CEST) Subject: [pypy-svn] r77629 - pypy/branch/fast-forward/pypy/module/math Message-ID: <20101005183534.976E5282B90@codespeak.net> Author: agaynor Date: Tue Oct 5 20:35:33 2010 New Revision: 77629 Modified: pypy/branch/fast-forward/pypy/module/math/__init__.py pypy/branch/fast-forward/pypy/module/math/interp_math.py Log: Added exmp1 to math module. Modified: pypy/branch/fast-forward/pypy/module/math/__init__.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/math/__init__.py (original) +++ pypy/branch/fast-forward/pypy/module/math/__init__.py Tue Oct 5 20:35:33 2010 @@ -34,6 +34,7 @@ 'atan2' : 'interp_math.atan2', 'modf' : 'interp_math.modf', 'exp' : 'interp_math.exp', + 'expm1' : 'interp_math.expm1', 'acos' : 'interp_math.acos', 'isinf' : 'interp_math.isinf', 'isnan' : 'interp_math.isnan', Modified: pypy/branch/fast-forward/pypy/module/math/interp_math.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/math/interp_math.py (original) +++ pypy/branch/fast-forward/pypy/module/math/interp_math.py Tue Oct 5 20:35:33 2010 @@ -328,6 +328,10 @@ return math1(space, math.exp, w_x) exp.unwrap_spec = [ObjSpace, W_Root] +def expm1(space, w_x): + return math1(space, math.expm1, w_x) +expm1.unwrap_spec = [ObjSpace, W_Root] + def acos(space, w_x): """acos(x) From david at codespeak.net Tue Oct 5 21:03:51 2010 From: david at codespeak.net (david at codespeak.net) Date: Tue, 5 Oct 2010 21:03:51 +0200 (CEST) Subject: [pypy-svn] r77630 - in pypy/branch/arm-backend/pypy/jit/backend: llsupport x86 Message-ID: <20101005190351.2AF14282B90@codespeak.net> Author: david Date: Tue Oct 5 21:03:38 2010 New Revision: 77630 Modified: pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py pypy/branch/arm-backend/pypy/jit/backend/x86/regalloc.py Log: move _compute_vars_longevity to llsupport and rename it to compute_vars_longevity Modified: pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py Tue Oct 5 21:03:38 2010 @@ -44,7 +44,7 @@ all_regs = [] no_lower_byte_regs = [] save_around_call_regs = [] - + def __init__(self, longevity, frame_manager=None, assembler=None): self.free_regs = self.all_regs[:] self.longevity = longevity @@ -244,7 +244,7 @@ if isinstance(v, Const): return self.return_constant(v, forbidden_vars, selected_reg, imm_fine) - + prev_loc = self.loc(v) loc = self.force_allocate_reg(v, forbidden_vars, selected_reg, need_lower_byte=need_lower_byte) @@ -349,3 +349,37 @@ be stored by the cpu, according to the variable type """ raise NotImplementedError("Abstract") + +def compute_vars_longevity(inputargs, operations): + # compute a dictionary that maps variables to index in + # operations that is a "last-time-seen" + longevity = {} + start_live = {} + for inputarg in inputargs: + start_live[inputarg] = 0 + for i in range(len(operations)): + op = operations[i] + if op.result is not None: + start_live[op.result] = i + for j in range(op.numargs()): + arg = op.getarg(j) + if isinstance(arg, Box): + if arg not in start_live: + print "Bogus arg in operation %d at %d" % (op.getopnum(), i) + raise AssertionError + longevity[arg] = (start_live[arg], i) + if op.is_guard(): + for arg in op.getfailargs(): + if arg is None: # hole + continue + assert isinstance(arg, Box) + if arg not in start_live: + print "Bogus arg in guard %d at %d" % (op.getopnum(), i) + raise AssertionError + longevity[arg] = (start_live[arg], i) + for arg in inputargs: + if arg not in longevity: + longevity[arg] = (-1, -1) + for arg in longevity: + assert isinstance(arg, Box) + return longevity Modified: pypy/branch/arm-backend/pypy/jit/backend/x86/regalloc.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/x86/regalloc.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/x86/regalloc.py Tue Oct 5 21:03:38 2010 @@ -16,7 +16,7 @@ from pypy.jit.backend.llsupport.descr import BaseFieldDescr, BaseArrayDescr from pypy.jit.backend.llsupport.descr import BaseCallDescr, BaseSizeDescr from pypy.jit.backend.llsupport.regalloc import FrameManager, RegisterManager,\ - TempBox + TempBox, compute_vars_longevity from pypy.jit.backend.x86.arch import WORD, FRAME_FIXED_SIZE, IS_X86_32, IS_X86_64 class X86RegisterManager(RegisterManager): @@ -106,7 +106,7 @@ def convert_to_imm(self, c): const_id, adr = self.float_constants.record_float(c.getfloat()) return ConstFloatLoc(adr, const_id) - + def after_call(self, v): # the result is stored in st0, but we don't have this around, # so genop_call will move it to some frame location immediately @@ -152,7 +152,7 @@ cpu = self.assembler.cpu cpu.gc_ll_descr.rewrite_assembler(cpu, operations) # compute longevity of variables - longevity = self._compute_vars_longevity(inputargs, operations) + longevity = compute_vars_longevity(inputargs, operations) self.longevity = longevity # XXX if cpu.WORD == 4: @@ -163,7 +163,7 @@ xmm_reg_mgr_cls = X86_64_XMMRegisterManager else: raise AssertionError("Word size should be 4 or 8") - + self.rm = gpr_reg_mgr_cls(longevity, frame_manager = self.fm, assembler = self.assembler) @@ -173,7 +173,7 @@ def prepare_loop(self, inputargs, operations, looptoken): self._prepare(inputargs, operations) jump = operations[-1] - loop_consts = self._compute_loop_consts(inputargs, jump, looptoken) + loop_consts = compute_loop_consts(inputargs, jump, looptoken) self.loop_consts = loop_consts return self._process_inputargs(inputargs) @@ -340,11 +340,11 @@ arglocs)) else: self.assembler.dump('%s(%s)' % (guard_op, arglocs)) - current_depths = (self.fm.frame_depth, self.param_depth) + current_depths = (self.fm.frame_depth, self.param_depth) self.assembler.regalloc_perform_guard(guard_op, faillocs, arglocs, result_loc, current_depths) - self.possibly_free_vars(guard_op.getfailargs()) + self.possibly_free_vars(guard_op.getfailargs()) def PerformDiscard(self, op, arglocs): if not we_are_translated(): @@ -397,39 +397,6 @@ assert not self.rm.reg_bindings assert not self.xrm.reg_bindings - def _compute_vars_longevity(self, inputargs, operations): - # compute a dictionary that maps variables to index in - # operations that is a "last-time-seen" - longevity = {} - start_live = {} - for inputarg in inputargs: - start_live[inputarg] = 0 - for i in range(len(operations)): - op = operations[i] - if op.result is not None: - start_live[op.result] = i - for j in range(op.numargs()): - arg = op.getarg(j) - if isinstance(arg, Box): - if arg not in start_live: - print "Bogus arg in operation %d at %d" % (op.getopnum(), i) - raise AssertionError - longevity[arg] = (start_live[arg], i) - if op.is_guard(): - for arg in op.getfailargs(): - if arg is None: # hole - continue - assert isinstance(arg, Box) - if arg not in start_live: - print "Bogus arg in guard %d at %d" % (op.getopnum(), i) - raise AssertionError - longevity[arg] = (start_live[arg], i) - for arg in inputargs: - if arg not in longevity: - longevity[arg] = (-1, -1) - for arg in longevity: - assert isinstance(arg, Box) - return longevity def loc(self, v): if v is None: # xxx kludgy @@ -692,7 +659,7 @@ self._call(op, [imm(size), vable] + [self.loc(op.getarg(i)) for i in range(op.numargs())], guard_not_forced_op=guard_op) - + def consider_cond_call_gc_wb(self, op): assert op.result is None args = op.getarglist() @@ -1006,13 +973,13 @@ xmmtmploc = self.xrm.force_allocate_reg(box1, selected_reg=xmmtmp) # Part about non-floats # XXX we don't need a copy, we only just the original list - src_locations = [self.loc(op.getarg(i)) for i in range(op.numargs()) + src_locations = [self.loc(op.getarg(i)) for i in range(op.numargs()) if op.getarg(i).type != FLOAT] assert tmploc not in nonfloatlocs dst_locations = [loc for loc in nonfloatlocs if loc is not None] remap_frame_layout(assembler, src_locations, dst_locations, tmploc) # Part about floats - src_locations = [self.loc(op.getarg(i)) for i in range(op.numargs()) + src_locations = [self.loc(op.getarg(i)) for i in range(op.numargs()) if op.getarg(i).type == FLOAT] dst_locations = [loc for loc in floatlocs if loc is not None] remap_frame_layout(assembler, src_locations, dst_locations, xmmtmp) From david at codespeak.net Tue Oct 5 21:25:14 2010 From: david at codespeak.net (david at codespeak.net) Date: Tue, 5 Oct 2010 21:25:14 +0200 (CEST) Subject: [pypy-svn] r77631 - in pypy/branch/arm-backend/pypy/jit/backend/arm: . test Message-ID: <20101005192514.395BC282B9C@codespeak.net> Author: david Date: Tue Oct 5 21:25:08 2010 New Revision: 77631 Added: pypy/branch/arm-backend/pypy/jit/backend/arm/ pypy/branch/arm-backend/pypy/jit/backend/arm/__init__.py pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py pypy/branch/arm-backend/pypy/jit/backend/arm/conditions.py pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/ pypy/branch/arm-backend/pypy/jit/backend/arm/test/__init__.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Log: basic encoding for some arm instructions and infrastructure for testing Added: pypy/branch/arm-backend/pypy/jit/backend/arm/__init__.py ============================================================================== Added: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py ============================================================================== --- (empty file) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py Tue Oct 5 21:25:08 2010 @@ -0,0 +1,79 @@ +import conditions as cond +from pypy.rlib.rmmap import alloc +from pypy.rpython.lltypesystem import lltype, rffi + +class ARMv7Builder(object): + + def __init__(self): + self._data = alloc(1024) + self._pos = 0 + + def LDR_ri(self, rt, rn, imm=0, cond=cond.AL): + # XXX U and P bits are not encoded yet + self.write32(cond << 28 + | 5 << 24 + | 9 << 20 + | (rn & 0xF) << 16 + | (rt & 0xF) << 12 + | (imm & 0xFFF)) + + def ADD_ri(self, rt, rn, imm, cond=cond.AL): + # XXX S bit + self.write32(cond << 28 + | 2 << 24 + | 8 << 20 + | (rn & 0xF) << 16 + | (rt & 0xF) << 12 + | (imm & 0xFFF)) + def MOV_ri(self, rt, imm=0, cond=cond.AL): + # XXX Check the actual allowed size for imm + # XXX S bit + self.write32(cond << 28 + | 0x3 << 24 + | 0xA << 20 + #| 0x0 << 16 + | (rt & 0xF) << 12 + | (imm & 0xFFF)) + + def STR_ri(self, rt, rn, imm=0, cond=cond.AL): + self.write32(cond << 28 + | 0x5 << 24 + | 0x8 << 20 + | (rn & 0xF) << 16 + | (rt & 0xF) << 12 + | (imm & 0xFFF)) + + def ASR_ri(self, rd, rm, imm=0, cond=cond.AL, s=0): + self.write32(cond << 28 + | 0xD << 21 + | (s & 0x1) << 20 + | (rd & 0xF) << 12 + | (imm & 0x1F) << 7 + | 0x4 << 4 + | (rm & 0xF)) + + #XXX encode shifttype correctly + def ORR_rr(self, rd, rn, rm, imm=0, cond=cond.AL, s=0, shifttype=0): + self.write32(cond << 28 + | 0x3 << 23 + | (s & 0x1) << 20 + | (rn & 0xFF) << 16 + | (rd & 0xFF) << 12 + | (imm & 0x1F) << 7 + | (shifttype & 0x3) << 5 + | (rm & 0xFF)) + + def write32(self, word): + self.writechar(chr(word & 0xFF)) + self.writechar(chr((word >> 8) & 0xFF)) + self.writechar(chr((word >> 16) & 0xFF)) + self.writechar(chr((word >> 24) & 0xFF)) + + def writechar(self, char): + self._data[self._pos] = char + self._pos += 1 + + def baseaddr(self): + return rffi.cast(lltype.Signed, self._data) + + Added: pypy/branch/arm-backend/pypy/jit/backend/arm/conditions.py ============================================================================== --- (empty file) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/conditions.py Tue Oct 5 21:25:08 2010 @@ -0,0 +1,15 @@ +EQ = 0x0 +NE = 0x1 +CS = 0x2 +CC = 0x3 +MI = 0x4 +PL = 0x5 +VS = 0x6 +VC = 0x7 +HI = 0x8 +LS = 0x9 +GE = 0xA +LT = 0xB +GT = 0xC +LE = 0xD +AL = 0xE Added: pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py ============================================================================== --- (empty file) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py Tue Oct 5 21:25:08 2010 @@ -0,0 +1,6 @@ +r0, r1, r2, r3, r4, r5, r6, r7, r8, r9, r10, r11, r12, r13, r14, r15 = range(16) +# aliases for registers +ip = 12 +sp = 13 +lr = 14 +pc = 15 Added: pypy/branch/arm-backend/pypy/jit/backend/arm/test/__init__.py ============================================================================== Added: pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py ============================================================================== --- (empty file) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py Tue Oct 5 21:25:08 2010 @@ -0,0 +1,52 @@ +import os +import tempfile +class ASMInstruction(object): + + if os.uname()[0] == 'Darwin': + asm = '~/Code/arm-jit/android/android-ndk-r4b//build/prebuilt/darwin-x86/arm-eabi-4.4.0/arm-eabi/bin/as' + else: + asm = 'as' + asm_opts = '-mcpu=cortex-a8 -march=armv7' + body = """.section .text +.arm +.global main +main: + .ascii "START" + %s + .ascii "END" +""" + begin_tag = 'START' + end_tag = 'END' + + def __init__(self, instr): + self.instr = instr + self.file = tempfile.NamedTemporaryFile(mode='w') + self.name = self.file.name + self.tmpdir = os.path.dirname(self.name) + + def encode(self): + f = open("%s/a.out" % (self.tmpdir),'rb') + data = f.read() + f.close() + i = data.find(self.begin_tag) + assert i>=0 + j = data.find(self.end_tag, i) + assert j>=0 + as_code = data[i+len(self.begin_tag):j] + return as_code + + + + def assemble(self, *args): + res = self.body % (self.instr) + self.file.write(res) + self.file.flush() + os.system("%s %s %s -o %s/a.out" % (self.asm, self.asm_opts, self.name, self.tmpdir)) + + def __del__(self): + self.file.close() + +def assemble(instr): + a = ASMInstruction(instr) + a.assemble(instr) + return a.encode() Added: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py ============================================================================== --- (empty file) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Tue Oct 5 21:25:08 2010 @@ -0,0 +1,58 @@ +from pypy.jit.backend.arm import registers as r +from pypy.jit.backend.arm import codebuilder +from gen import assemble +import py +class CodeBuilder(codebuilder.ARMv7Builder): + def __init__(self): + self.buffer = [] + + def writechar(self, char): + self.buffer.append(char) + + def hexdump(self): + return ''.join(self.buffer) + +class TestInstrCodeBuilder(object): + def setup_method(self, ffuu_method): + self.cb = CodeBuilder() + + def test_ldr(self): + self.cb.LDR_ri(r.r0, r.r1) + self.assert_equal("LDR r0, [r1]") + + def test_add_ri(self): + self.cb.ADD_ri(r.r0, r.r1, 1) + self.assert_equal("ADD r0, r1, #1") + + def test_mov_ri(self): + self.cb.MOV_ri(r.r9, 123) + self.assert_equal("MOV r9, #123") + + def test_mov_ri2(self): + self.cb.MOV_ri(r.r9, 255) + self.assert_equal("MOV r9, #255") + + def test_mov_ri_max(self): + py.test.skip("Check the actual largest thing") + self.cb.MOV_ri(r.r9, 0xFFF) + self.assert_equal("MOV r9, #4095") + + def test_str_ri(self): + self.cb.STR_ri(r.r9, r.r14) + self.assert_equal("STR r9, [r14]") + + def test_asr_ri(self): + self.cb.ASR_ri(r.r7, r.r5, 24) + self.assert_equal('ASR r7, r5, #24') + + def test_orr_rr_no_shift(self): + self.cb.ORR_rr(r.r0, r.r7,r.r12) + self.assert_equal('ORR r0, r7, r12') + + def test_orr_rr_lsl_8(self): + self.cb.ORR_rr(r.r0, r.r7,r.r12, 8) + self.assert_equal('ORR r0, r7, r12, lsl #8') + + def assert_equal(self, asm): + assert self.cb.hexdump() == assemble(asm) + From david at codespeak.net Tue Oct 5 21:33:11 2010 From: david at codespeak.net (david at codespeak.net) Date: Tue, 5 Oct 2010 21:33:11 +0200 (CEST) Subject: [pypy-svn] r77632 - in pypy/branch/arm-backend/pypy/jit/backend/arm: . test Message-ID: <20101005193311.D3E16282B9C@codespeak.net> Author: david Date: Tue Oct 5 21:33:10 2010 New Revision: 77632 Added: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_runner.py Log: simple implementation of cpu and assembler to pass test_compile_linear_loop in runner_test Added: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- (empty file) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Tue Oct 5 21:33:10 2010 @@ -0,0 +1,52 @@ +from pypy.jit.backend.arm.codebuilder import ARMv7Builder +from pypy.jit.backend.arm import registers as r +from pypy.jit.backend.arm.regalloc import RegAlloc, ARMRegisterManager +from pypy.jit.metainterp.resoperation import rop +from pypy.rpython.lltypesystem import lltype +# XXX Move to llsupport +from pypy.jit.backend.x86.support import values_array + + +class AssemblerARM(object): + + def __init__(self, cpu, failargs_limit=1000): + self.mc = ARMv7Builder() + self.cpu = cpu + self.fail_boxes_int = values_array(lltype.Signed, failargs_limit) + + def assemble_loop(self, inputargs, operations, looptoken): + assert len(inputargs) == 1 + reg = 0 + self.gen_preamble() + addr = self.fail_boxes_int.get_addr_for_num(0) + self.gen_load_int(r.r3, addr) + self.mc.LDR_ri(r.r2, r.r3) + for op in operations: + if op.getopnum() == rop.INT_ADD: + self.mc.ADD_ri(r.r1, r.r2, op.getarg(1).getint()) + elif op.getopnum() == rop.FINISH: + n = self.cpu.get_fail_descr_number(op.getdescr()) + self.mc.MOV_ri(r.r0, n) + self.mc.STR_ri(r.r1, r.r3) + self.gen_out() + + def gen_out(self): + self.mc.write32(0xe50b3010) # str r3, [fp, #-16] + self.mc.write32(0xe51b3010) # ldr r3, [fp, #-16] + #self.mc.write32(0xe1a00003) # mov r0, r3 + self.mc.write32(0xe24bd00c) # sub sp, fp, #12 ; 0xc + self.mc.write32(0xe89da800) # ldm sp, {fp, sp, pc} + + def gen_preamble(self): + self.mc.write32(0xe1a0c00d) # mov ip, sp + self.mc.write32(0xe92dd800) #push {fp, ip, lr, pc} + self.mc.write32(0xe24cb004) # sub fp, ip, #4 ; 0x4 + self.mc.write32(0xe24dd008) #sub sp, sp, #8 ; 0x8 + self.mc.write32(0xe50b0014) # str r0, [fp, #-20] + + def gen_load_int(self, reg, value): + self.mc.MOV_ri(reg, (value & 0xFF)) + + for offset in range(8, 25, 8): + self.mc.MOV_ri(r.ip, (value >> offset) & 0xFF) + self.mc.ORR_rr(reg, reg, r.ip, offset) Added: pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py ============================================================================== --- (empty file) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py Tue Oct 5 21:33:10 2010 @@ -0,0 +1,43 @@ +from pypy.jit.backend.arm.assembler import AssemblerARM +from pypy.jit.backend.llsupport.llmodel import AbstractLLCPU +from pypy.rpython.llinterp import LLInterpreter +from pypy.rpython.lltypesystem import lltype, rffi + + +class ArmCPU(AbstractLLCPU): + + BOOTSTRAP_TP = lltype.FuncType([], lltype.Signed) + supports_floats = False + + def __init__(self, rtyper, stats, opts=None, translate_support_code=False, + gcdescr=None): + AbstractLLCPU.__init__(self, rtyper, stats, opts, + translate_support_code, gcdescr) + self.assembler = AssemblerARM(self) + + def compile_loop(self, inputargs, operations, looptoken): + self.assembler.assemble_loop(inputargs, operations, looptoken) + + def set_future_value_int(self, index, intvalue): + self.assembler.fail_boxes_int.setitem(index, intvalue) + + def get_latest_value_int(self, index): + return self.assembler.fail_boxes_int.getitem(index) + + def execute_token(self, executable_token): + addr = self.assembler.mc.baseaddr()#executable_token._arm_bootstrap_code + assert addr % 8 == 0 + func = rffi.cast(lltype.Ptr(self.BOOTSTRAP_TP), addr) + fail_index = self._execute_call(func) + return self.get_fail_descr_from_number(fail_index) + + def _execute_call(self, func): + #prev_interpreter = LLInterpreter.current_interpreter + #LLInterpreter.current_interpreter = self.debug_ll_interpreter + res = 0 + #try: + res = func() + #finally: + # LLInterpreter.current_interpreter = prev_interpreter + return res + Added: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_runner.py ============================================================================== --- (empty file) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_runner.py Tue Oct 5 21:33:10 2010 @@ -0,0 +1,13 @@ +from pypy.jit.backend.arm.runner import ArmCPU +from pypy.jit.backend.test.runner_test import LLtypeBackendTest + +class FakeStats(object): + pass + +class TestARM(LLtypeBackendTest): + + # for the individual tests see + # ====> ../../test/runner_test.py + + def setup_method(self, meth): + self.cpu = ArmCPU(rtyper=None, stats=FakeStats()) From david at codespeak.net Tue Oct 5 21:34:22 2010 From: david at codespeak.net (david at codespeak.net) Date: Tue, 5 Oct 2010 21:34:22 +0200 (CEST) Subject: [pypy-svn] r77633 - pypy/branch/arm-backend/pypy/jit/backend/arm/test Message-ID: <20101005193422.A84EE36C222@codespeak.net> Author: david Date: Tue Oct 5 21:34:21 2010 New Revision: 77633 Added: pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py Log: integration tests to check generated instructions on arm Added: pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py ============================================================================== --- (empty file) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py Tue Oct 5 21:34:21 2010 @@ -0,0 +1,21 @@ +from pypy.rpython.lltypesystem import lltype, rffi + +def run_asm(asm): + BOOTSTRAP_TP = lltype.FuncType([], lltype.Signed) + addr = asm.mc.baseaddr()#executable_token._arm_bootstrap_code + assert addr % 8 == 0 + func = rffi.cast(lltype.Ptr(BOOTSTRAP_TP), addr) + return func() + +def skip_unless_arm(f): + import os + def skip_it(*args): + import py + py.test.skip('only for arm') + + func = skip_it + try: + if os.uname()[4].index('arm') >= 0: + func=f + finally: + return func Added: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py ============================================================================== --- (empty file) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py Tue Oct 5 21:34:21 2010 @@ -0,0 +1,38 @@ +from pypy.jit.backend.arm import registers as r +from pypy.jit.backend.arm.assembler import AssemblerARM +from pypy.jit.backend.arm.test.support import skip_unless_arm, run_asm + +class TestRunningAssembler(): + def setup_method(self, method): + self.a = AssemblerARM(None) + + @skip_unless_arm + def test_load_small_int_to_reg(self): + self.a.gen_preamble() + self.a.gen_load_int(r.r0, 123) + self.a.gen_out() + assert run_asm(self.a) == 123 + + @skip_unless_arm + def test_load_medium_int_to_reg(self): + self.a.gen_preamble() + self.a.gen_load_int(r.r0, 0xBBD7) + self.a.gen_out() + assert run_asm(self.a) == 48087 + + @skip_unless_arm + def test_load_int_to_reg(self): + self.a.gen_preamble() + self.a.gen_load_int(r.r0, 0xFFFFFF85) + self.a.gen_out() + assert run_asm(self.a) == -123 + + + @skip_unless_arm + def test_or(self): + self.a.gen_preamble() + self.a.mc.MOV_ri(r.r1, 8) + self.a.mc.MOV_ri(r.r2, 8) + self.a.mc.ORR_rr(r.r0, r.r1, r.r2, 4) + self.a.gen_out() + assert run_asm(self.a) == 0x88 From afa at codespeak.net Tue Oct 5 21:44:50 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 21:44:50 +0200 (CEST) Subject: [pypy-svn] r77634 - pypy/branch/fast-forward/pypy/interpreter/pyparser/test Message-ID: <20101005194450.0C351282B90@codespeak.net> Author: afa Date: Tue Oct 5 21:44:49 2010 New Revision: 77634 Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py Log: Add a failing test Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py Tue Oct 5 21:44:49 2010 @@ -100,6 +100,10 @@ tree = self.parse("x = 43", "single") assert tree.type == syms.single_input + def test_multiline_string(self): + self.parse("''' \n '''") + self.parse("r''' \n '''") + def test_bytes_literal(self): self.parse('b" "') self.parse('br" "') From afa at codespeak.net Tue Oct 5 22:01:25 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 22:01:25 +0200 (CEST) Subject: [pypy-svn] r77635 - pypy/branch/fast-forward/pypy/interpreter/pyparser Message-ID: <20101005200125.4DD7D282B9C@codespeak.net> Author: afa Date: Tue Oct 5 22:01:23 2010 New Revision: 77635 Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py Log: Fix an error in the tokenizer's grammar: triple-quoted string with prefix were not correctly decoded r""" """ Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py Tue Oct 5 22:01:23 2010 @@ -170,7 +170,7 @@ group(states, newArcPair(states, '"'), makeLineCont()))) - triple = group(states, + triple = chain(states, makeStrPrefix(), group(states, chainStr(states, "'''"), Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py Tue Oct 5 22:01:23 2010 @@ -23,11 +23,11 @@ # ______________________________________________________________________ # Automatically generated DFA's -accepts = [True, True, True, True, True, True, True, True, +accepts = [False, True, True, True, True, True, True, True, True, True, False, True, True, True, False, False, - False, False, True, False, False, True, True, - True, True, False, True, False, True, False, True, - False, True, False, False, False, True, False, + False, False, True, True, True, True, True, False, + True, False, True, False, True, False, False, + True, False, False, False, False, True, False, False, False, True] states = [ {'\t': 0, '\n': 13, '\x0c': 0, @@ -73,7 +73,7 @@ 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - {'"': 20, "'": 19, '0': 1, '1': 1, + {'"': 16, "'": 15, '0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, 'A': 1, 'B': 1, 'C': 1, 'D': 1, @@ -91,7 +91,7 @@ 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - {'"': 20, "'": 19, '0': 1, '1': 1, + {'"': 16, "'": 15, '0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, 'A': 1, 'B': 1, 'C': 1, 'D': 1, @@ -109,22 +109,22 @@ 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - {'.': 26, '0': 23, '1': 23, '2': 23, - '3': 23, '4': 23, '5': 23, '6': 23, - '7': 23, '8': 25, '9': 25, 'B': 24, - 'E': 27, 'J': 13, 'L': 13, 'O': 22, - 'X': 21, 'b': 24, 'e': 27, 'j': 13, - 'l': 13, 'o': 22, 'x': 21}, + {'.': 24, '0': 21, '1': 21, '2': 21, + '3': 21, '4': 21, '5': 21, '6': 21, + '7': 21, '8': 23, '9': 23, 'B': 22, + 'E': 25, 'J': 13, 'L': 13, 'O': 20, + 'X': 19, 'b': 22, 'e': 25, 'j': 13, + 'l': 13, 'o': 20, 'x': 19}, - {'.': 26, '0': 5, '1': 5, '2': 5, + {'.': 24, '0': 5, '1': 5, '2': 5, '3': 5, '4': 5, '5': 5, '6': 5, - '7': 5, '8': 5, '9': 5, 'E': 27, - 'J': 13, 'L': 13, 'e': 27, 'j': 13, + '7': 5, '8': 5, '9': 5, 'E': 25, + 'J': 13, 'L': 13, 'e': 25, 'j': 13, 'l': 13}, - {'0': 28, '1': 28, '2': 28, '3': 28, - '4': 28, '5': 28, '6': 28, '7': 28, - '8': 28, '9': 28}, + {'0': 26, '1': 26, '2': 26, '3': 26, + '4': 26, '5': 26, '6': 26, '7': 26, + '8': 26, '9': 26}, {'*': 12, '=': 13}, @@ -142,52 +142,46 @@ {'\n': 13}, - {automata.DEFAULT: 19, '\n': 29, - "'": 30, '\\': 31}, + {automata.DEFAULT: 30, '\n': 27, + "'": 28, '\\': 29}, - {automata.DEFAULT: 20, '\n': 29, - '"': 32, '\\': 33}, + {automata.DEFAULT: 33, '\n': 27, + '"': 31, '\\': 32}, {'\n': 13, '\r': 14}, - {automata.DEFAULT: 18, '\n': 29, - '\r': 29}, - - {automata.DEFAULT: 19, '\n': 29, - "'": 13, '\\': 31}, + {automata.DEFAULT: 18, '\n': 27, + '\r': 27}, - {automata.DEFAULT: 20, '\n': 29, - '"': 13, '\\': 33}, - - {'0': 21, '1': 21, '2': 21, '3': 21, - '4': 21, '5': 21, '6': 21, '7': 21, - '8': 21, '9': 21, 'A': 21, 'B': 21, - 'C': 21, 'D': 21, 'E': 21, 'F': 21, - 'L': 13, 'a': 21, 'b': 21, 'c': 21, - 'd': 21, 'e': 21, 'f': 21, + {'0': 19, '1': 19, '2': 19, '3': 19, + '4': 19, '5': 19, '6': 19, '7': 19, + '8': 19, '9': 19, 'A': 19, 'B': 19, + 'C': 19, 'D': 19, 'E': 19, 'F': 19, + 'L': 13, 'a': 19, 'b': 19, 'c': 19, + 'd': 19, 'e': 19, 'f': 19, 'l': 13}, - {'0': 22, '1': 22, '2': 22, '3': 22, - '4': 22, '5': 22, '6': 22, '7': 22, + {'0': 20, '1': 20, '2': 20, '3': 20, + '4': 20, '5': 20, '6': 20, '7': 20, 'L': 13, 'l': 13}, - {'.': 26, '0': 23, '1': 23, '2': 23, - '3': 23, '4': 23, '5': 23, '6': 23, - '7': 23, '8': 25, '9': 25, 'E': 27, - 'J': 13, 'L': 13, 'e': 27, 'j': 13, + {'.': 24, '0': 21, '1': 21, '2': 21, + '3': 21, '4': 21, '5': 21, '6': 21, + '7': 21, '8': 23, '9': 23, 'E': 25, + 'J': 13, 'L': 13, 'e': 25, 'j': 13, 'l': 13}, - {'0': 24, '1': 24, 'L': 13, + {'0': 22, '1': 22, 'L': 13, 'l': 13}, - {'.': 26, '0': 25, '1': 25, '2': 25, - '3': 25, '4': 25, '5': 25, '6': 25, - '7': 25, '8': 25, '9': 25, 'E': 27, - 'J': 13, 'e': 27, 'j': 13}, + {'.': 24, '0': 23, '1': 23, '2': 23, + '3': 23, '4': 23, '5': 23, '6': 23, + '7': 23, '8': 23, '9': 23, 'E': 25, + 'J': 13, 'e': 25, 'j': 13}, - {'0': 26, '1': 26, '2': 26, '3': 26, - '4': 26, '5': 26, '6': 26, '7': 26, - '8': 26, '9': 26, 'E': 34, 'J': 13, + {'0': 24, '1': 24, '2': 24, '3': 24, + '4': 24, '5': 24, '6': 24, '7': 24, + '8': 24, '9': 24, 'E': 34, 'J': 13, 'e': 34, 'j': 13}, {'+': 35, '-': 35, '0': 36, '1': 36, @@ -195,9 +189,9 @@ '6': 36, '7': 36, '8': 36, '9': 36}, - {'0': 28, '1': 28, '2': 28, '3': 28, - '4': 28, '5': 28, '6': 28, '7': 28, - '8': 28, '9': 28, 'E': 34, 'J': 13, + {'0': 26, '1': 26, '2': 26, '3': 26, + '4': 26, '5': 26, '6': 26, '7': 26, + '8': 26, '9': 26, 'E': 34, 'J': 13, 'e': 34, 'j': 13}, {}, @@ -207,11 +201,17 @@ {automata.DEFAULT: 37, '\n': 13, '\r': 14}, + {automata.DEFAULT: 30, '\n': 27, + "'": 13, '\\': 29}, + {'"': 13}, {automata.DEFAULT: 38, '\n': 13, '\r': 14}, + {automata.DEFAULT: 33, '\n': 27, + '"': 13, '\\': 32}, + {'+': 39, '-': 39, '0': 40, '1': 40, '2': 40, '3': 40, '4': 40, '5': 40, '6': 40, '7': 40, '8': 40, @@ -226,11 +226,11 @@ '8': 36, '9': 36, 'J': 13, 'j': 13}, - {automata.DEFAULT: 37, '\n': 29, - "'": 13, '\\': 31}, + {automata.DEFAULT: 37, '\n': 27, + "'": 13, '\\': 29}, - {automata.DEFAULT: 38, '\n': 29, - '"': 13, '\\': 33}, + {automata.DEFAULT: 38, '\n': 27, + '"': 13, '\\': 32}, {'0': 40, '1': 40, '2': 40, '3': 40, '4': 40, '5': 40, '6': 40, '7': 40, From afa at codespeak.net Tue Oct 5 23:14:00 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 5 Oct 2010 23:14:00 +0200 (CEST) Subject: [pypy-svn] r77636 - in pypy/branch/fast-forward/pypy/interpreter/astcompiler: . test Message-ID: <20101005211400.0E17D282B90@codespeak.net> Author: afa Date: Tue Oct 5 23:13:59 2010 New Revision: 77636 Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py Log: Properly skip the number prefix: in base 16, don't strip the digit 'b'! Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py Tue Oct 5 23:13:59 2010 @@ -1047,7 +1047,11 @@ i = 0 limit = len(raw) - 1 while i < limit: - if raw[i] not in "0xXoOBb": + if base == 16 and raw[i] not in "0xX": + break + if base == 8 and raw[i] not in "0oO": + break + if base == 2 and raw[i] not in "0bB": break i += 1 raw = raw[i:] Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py Tue Oct 5 23:13:59 2010 @@ -1129,6 +1129,7 @@ assert space.eq_w(get_num("00053"), space.wrap(053)) for num in ("0x53", "0X53", "0x0000053", "0X00053"): assert space.eq_w(get_num(num), space.wrap(0x53)) + assert space.eq_w(get_num("0Xb0d2"), space.wrap(0xb0d2)) assert space.eq_w(get_num("0X53"), space.wrap(0x53)) assert space.eq_w(get_num("0"), space.wrap(0)) assert space.eq_w(get_num("00000"), space.wrap(0)) From afa at codespeak.net Wed Oct 6 00:13:40 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Wed, 6 Oct 2010 00:13:40 +0200 (CEST) Subject: [pypy-svn] r77637 - pypy/branch/fast-forward/lib_pypy Message-ID: <20101005221340.E87E0282B9C@codespeak.net> Author: afa Date: Wed Oct 6 00:13:39 2010 New Revision: 77637 Modified: pypy/branch/fast-forward/lib_pypy/hashlib.py Log: hashlib.algorithms should be a tuple Modified: pypy/branch/fast-forward/lib_pypy/hashlib.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/hashlib.py (original) +++ pypy/branch/fast-forward/lib_pypy/hashlib.py Wed Oct 6 00:13:39 2010 @@ -108,7 +108,7 @@ new = __hash_new -algorithms = ['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512'] +algorithms = ('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512') def __getfunc(name): def new(string=''): From antocuni at codespeak.net Wed Oct 6 11:19:42 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 6 Oct 2010 11:19:42 +0200 (CEST) Subject: [pypy-svn] r77642 - in pypy/branch/jitffi/pypy: jit/backend/llgraph rlib rlib/test rpython/lltypesystem Message-ID: <20101006091942.A7BAD282B90@codespeak.net> Author: antocuni Date: Wed Oct 6 11:19:40 2010 New Revision: 77642 Modified: pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py pypy/branch/jitffi/pypy/rlib/libffi.py pypy/branch/jitffi/pypy/rlib/test/test_libffi.py pypy/branch/jitffi/pypy/rpython/lltypesystem/lltype.py Log: add support for passing pointers as arguments to ffi calls Modified: pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py Wed Oct 6 11:19:40 2010 @@ -1018,6 +1018,9 @@ if isinstance(TYPE, lltype.Ptr): if isinstance(x, (int, long, llmemory.AddressAsInt)): x = llmemory.cast_int_to_adr(x) + if TYPE is rffi.VOIDP: + # assume that we want a "C-style" cast, without typechecking the value + return rffi.cast(TYPE, x) return llmemory.cast_adr_to_ptr(x, TYPE) elif TYPE == llmemory.Address: if isinstance(x, (int, long, llmemory.AddressAsInt)): Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Wed Oct 6 11:19:40 2010 @@ -1,5 +1,5 @@ from pypy.rpython.lltypesystem import rffi, lltype -from pypy.rlib.objectmodel import specialize, enforceargs +from pypy.rlib.objectmodel import specialize, enforceargs, we_are_translated from pypy.rlib.rarithmetic import intmask, r_uint from pypy.rlib import jit from pypy.rlib import clibffi @@ -35,6 +35,8 @@ @specialize.arg(0) def _fits_into_long(TYPE): + if isinstance(TYPE, lltype.Ptr): + return True # pointers always fits into longs if not isinstance(TYPE, lltype.Primitive): return False if TYPE is lltype.Void or TYPE is rffi.FLOAT or TYPE is rffi.DOUBLE: @@ -44,6 +46,17 @@ # ====================================================================== + at specialize.memo() +def _check_type(TYPE): + if isinstance(TYPE, lltype.Ptr): + if TYPE.TO._gckind != 'raw': + raise TypeError, "Can only push raw values to C, not 'gc'" + # XXX probably we should recursively check for struct fields here, + # lets just ignore that for now + if isinstance(TYPE.TO, lltype.Array) and 'nolength' not in TYPE.TO._hints: + raise TypeError, "Can only push to C arrays without length info" + + class ArgChain(object): first = None last = None @@ -52,6 +65,7 @@ @specialize.argtype(1) def arg(self, val): TYPE = lltype.typeOf(val) + _check_type(TYPE) if _fits_into_long(TYPE): cls = IntArg val = rffi.cast(rffi.LONG, val) Modified: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Wed Oct 6 11:19:40 2010 @@ -25,6 +25,18 @@ floatarg = intarg.next assert floatarg.floatval == 123.45 + def test_wrong_args(self): + # so far the test passes but for the wrong reason :-), i.e. because + # .arg() only supports integers and floats + chain = ArgChain() + x = lltype.malloc(lltype.GcStruct('xxx')) + y = lltype.malloc(lltype.GcArray(rffi.LONG), 3) + z = lltype.malloc(lltype.Array(rffi.LONG), 4, flavor='raw') + py.test.raises(TypeError, "chain.arg(x)") + py.test.raises(TypeError, "chain.arg(y)") + py.test.raises(TypeError, "chain.arg(z)") + lltype.free(z, flavor='raw') + def test_library_open(self): lib = self.get_libc() del lib @@ -159,3 +171,27 @@ func = (libfoo, 'many_args', [types.uchar, types.sint], types.sint) res = self.call(func, [chr(20), 22], rffi.LONG) assert res == 42 + + def test_call_time(self): + import time + libc = self.get_libc() + # XXX assume time_t is long + # XXX: on msvcr80 the name of the function is _time32, fix it in that case + func = (libc, 'time', [types.pointer], types.ulong) + LONGP = rffi.CArray(rffi.LONG) + null = lltype.nullptr(LONGP) + t0 = self.call(func, [null], rffi.LONG) + time.sleep(1) + t1 = self.call(func, [null], rffi.LONG) + assert t1 > t0 + # + ptr_result = lltype.malloc(LONGP, 1, flavor='raw') + t2 = self.call(func, [ptr_result], rffi.LONG) + assert ptr_result[0] == t2 + lltype.free(ptr_result, flavor='raw') + if self.__class__ is TestLibffiCall: + # the test does not make sense when run with the JIT through + # meta_interp, because the __del__ are not properly called (hence + # we "leak" memory) + del libc + assert not ALLOCATED Modified: pypy/branch/jitffi/pypy/rpython/lltypesystem/lltype.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/lltypesystem/lltype.py (original) +++ pypy/branch/jitffi/pypy/rpython/lltypesystem/lltype.py Wed Oct 6 11:19:40 2010 @@ -794,6 +794,8 @@ return llmemory.cast_adr_to_ptr(value, TGT) elif TGT == llmemory.Address and isinstance(ORIG, Ptr): return llmemory.cast_ptr_to_adr(value) + elif TGT == Signed and isinstance(ORIG, Ptr) and ORIG.TO._gckind == 'raw': + return llmemory.cast_adr_to_int(llmemory.cast_ptr_to_adr(value), 'symbolic') raise TypeError("don't know how to cast from %r to %r" % (ORIG, TGT)) From antocuni at codespeak.net Wed Oct 6 12:23:18 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 6 Oct 2010 12:23:18 +0200 (CEST) Subject: [pypy-svn] r77643 - in pypy/branch/jitffi/pypy: jit/backend/llsupport jit/metainterp rlib/test Message-ID: <20101006102318.6148D36C22E@codespeak.net> Author: antocuni Date: Wed Oct 6 12:23:16 2010 New Revision: 77643 Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py pypy/branch/jitffi/pypy/jit/metainterp/warmstate.py pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Log: add support for returning pointers Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py Wed Oct 6 12:23:16 2010 @@ -29,7 +29,7 @@ if ffi_type is types.void: return history.VOID elif ffi_type is types.pointer: - return history.REF + return history.INT elif ffi_type is types.double: return history.FLOAT elif ffi_type is types.uchar: Modified: pypy/branch/jitffi/pypy/jit/metainterp/warmstate.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/warmstate.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/warmstate.py Wed Oct 6 12:23:16 2010 @@ -1,5 +1,5 @@ import sys -from pypy.rpython.lltypesystem import lltype, llmemory, rstr +from pypy.rpython.lltypesystem import lltype, llmemory, rstr, rffi from pypy.rpython.ootypesystem import ootype from pypy.rpython.annlowlevel import hlstr, llstr, cast_base_ptr_to_instance from pypy.rpython.annlowlevel import cast_object_to_ptr @@ -23,7 +23,11 @@ """ INPUT = lltype.typeOf(x) if INPUT is lltype.Signed: - return lltype.cast_primitive(TYPE, x) # XXX missing: Ptr(non-gc) + if isinstance(TYPE, lltype.Ptr) and TYPE.TO._gckind == 'raw': + # non-gc pointer + return rffi.cast(TYPE, x) + else: + return lltype.cast_primitive(TYPE, x) elif INPUT is lltype.Float: assert TYPE is lltype.Float return x Modified: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Wed Oct 6 12:23:16 2010 @@ -195,3 +195,24 @@ # we "leak" memory) del libc assert not ALLOCATED + + def test_return_pointer(self): + """ + struct pair { + long a; + long b; + }; + + struct pair my_static_pair = {10, 20}; + + long* get_pointer_to_b() + { + return &my_static_pair.b; + } + """ + libfoo = self.get_libfoo() + func = (libfoo, 'get_pointer_to_b', [], types.pointer) + LONGP = lltype.Ptr(rffi.CArray(rffi.LONG)) + null = lltype.nullptr(LONGP.TO) + res = self.call(func, [], LONGP, init_result=null) + assert res[0] == 20 From david at codespeak.net Wed Oct 6 14:58:44 2010 From: david at codespeak.net (david at codespeak.net) Date: Wed, 6 Oct 2010 14:58:44 +0200 (CEST) Subject: [pypy-svn] r77645 - in pypy/branch/arm-backend/pypy/jit/backend: llsupport x86 Message-ID: <20101006125844.B527B36C22E@codespeak.net> Author: david Date: Wed Oct 6 14:58:42 2010 New Revision: 77645 Modified: pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py pypy/branch/arm-backend/pypy/jit/backend/x86/regalloc.py Log: finish moving compute_loop_consts to llsupport Modified: pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py Wed Oct 6 14:58:42 2010 @@ -383,3 +383,14 @@ for arg in longevity: assert isinstance(arg, Box) return longevity + +def compute_loop_consts(self, inputargs, jump, looptoken): + if jump.getopnum() != rop.JUMP or jump.getdescr() is not looptoken: + loop_consts = {} + else: + loop_consts = {} + for i in range(len(inputargs)): + if inputargs[i] is jump.getarg(i): + loop_consts[inputargs[i]] = i + return loop_consts + Modified: pypy/branch/arm-backend/pypy/jit/backend/x86/regalloc.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/x86/regalloc.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/x86/regalloc.py Wed Oct 6 14:58:42 2010 @@ -16,7 +16,7 @@ from pypy.jit.backend.llsupport.descr import BaseFieldDescr, BaseArrayDescr from pypy.jit.backend.llsupport.descr import BaseCallDescr, BaseSizeDescr from pypy.jit.backend.llsupport.regalloc import FrameManager, RegisterManager,\ - TempBox, compute_vars_longevity + TempBox, compute_vars_longevity, compute_loop_consts from pypy.jit.backend.x86.arch import WORD, FRAME_FIXED_SIZE, IS_X86_32, IS_X86_64 class X86RegisterManager(RegisterManager): @@ -267,16 +267,6 @@ return self.rm.force_allocate_reg(var, forbidden_vars, selected_reg, need_lower_byte) - def _compute_loop_consts(self, inputargs, jump, looptoken): - if jump.getopnum() != rop.JUMP or jump.getdescr() is not looptoken: - loop_consts = {} - else: - loop_consts = {} - for i in range(len(inputargs)): - if inputargs[i] is jump.getarg(i): - loop_consts[inputargs[i]] = i - return loop_consts - def _update_bindings(self, locs, inputargs): # XXX this should probably go to llsupport/regalloc.py used = {} From david at codespeak.net Wed Oct 6 15:07:34 2010 From: david at codespeak.net (david at codespeak.net) Date: Wed, 6 Oct 2010 15:07:34 +0200 (CEST) Subject: [pypy-svn] r77646 - pypy/branch/arm-backend/pypy/jit/backend/arm Message-ID: <20101006130734.D9CE436C22E@codespeak.net> Author: david Date: Wed Oct 6 15:07:33 2010 New Revision: 77646 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Log: remove currently unused import Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Wed Oct 6 15:07:33 2010 @@ -1,6 +1,6 @@ from pypy.jit.backend.arm.codebuilder import ARMv7Builder from pypy.jit.backend.arm import registers as r -from pypy.jit.backend.arm.regalloc import RegAlloc, ARMRegisterManager +#from pypy.jit.backend.arm.regalloc import RegAlloc, ARMRegisterManager from pypy.jit.metainterp.resoperation import rop from pypy.rpython.lltypesystem import lltype # XXX Move to llsupport From david at codespeak.net Wed Oct 6 15:07:59 2010 From: david at codespeak.net (david at codespeak.net) Date: Wed, 6 Oct 2010 15:07:59 +0200 (CEST) Subject: [pypy-svn] r77647 - pypy/branch/arm-backend/pypy/jit/backend/arm/test Message-ID: <20101006130759.CD2F136C22E@codespeak.net> Author: david Date: Wed Oct 6 15:07:58 2010 New Revision: 77647 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_runner.py Log: move platform based skipping to the module level Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py Wed Oct 6 15:07:58 2010 @@ -7,15 +7,15 @@ func = rffi.cast(lltype.Ptr(BOOTSTRAP_TP), addr) return func() -def skip_unless_arm(f): +def skip_unless_arm(): import os - def skip_it(*args): - import py - py.test.skip('only for arm') + import py - func = skip_it + skip = True try: if os.uname()[4].index('arm') >= 0: - func=f + skip = False finally: - return func + if skip: + py.test.skip('only for arm') + Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py Wed Oct 6 15:07:58 2010 @@ -2,25 +2,24 @@ from pypy.jit.backend.arm.assembler import AssemblerARM from pypy.jit.backend.arm.test.support import skip_unless_arm, run_asm +skip_unless_arm() + class TestRunningAssembler(): def setup_method(self, method): self.a = AssemblerARM(None) - @skip_unless_arm def test_load_small_int_to_reg(self): self.a.gen_preamble() self.a.gen_load_int(r.r0, 123) self.a.gen_out() assert run_asm(self.a) == 123 - @skip_unless_arm def test_load_medium_int_to_reg(self): self.a.gen_preamble() self.a.gen_load_int(r.r0, 0xBBD7) self.a.gen_out() assert run_asm(self.a) == 48087 - @skip_unless_arm def test_load_int_to_reg(self): self.a.gen_preamble() self.a.gen_load_int(r.r0, 0xFFFFFF85) @@ -28,7 +27,6 @@ assert run_asm(self.a) == -123 - @skip_unless_arm def test_or(self): self.a.gen_preamble() self.a.mc.MOV_ri(r.r1, 8) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_runner.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_runner.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_runner.py Wed Oct 6 15:07:58 2010 @@ -1,5 +1,8 @@ from pypy.jit.backend.arm.runner import ArmCPU from pypy.jit.backend.test.runner_test import LLtypeBackendTest +from pypy.jit.backend.arm.test.support import skip_unless_arm + +skip_unless_arm() class FakeStats(object): pass From antocuni at codespeak.net Wed Oct 6 15:13:20 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 6 Oct 2010 15:13:20 +0200 (CEST) Subject: [pypy-svn] r77648 - pypy/branch/jitffi/pypy/rlib/test Message-ID: <20101006131320.30C5736C22E@codespeak.net> Author: antocuni Date: Wed Oct 6 15:13:18 2010 New Revision: 77648 Modified: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Log: use an ad-hoc function instead of time: it tests the same behaviour, but at least we don't have to sleep() for 1 second waiting for the time to change Modified: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Wed Oct 6 15:13:18 2010 @@ -172,29 +172,43 @@ res = self.call(func, [chr(20), 22], rffi.LONG) assert res == 42 - def test_call_time(self): - import time - libc = self.get_libc() - # XXX assume time_t is long - # XXX: on msvcr80 the name of the function is _time32, fix it in that case - func = (libc, 'time', [types.pointer], types.ulong) - LONGP = rffi.CArray(rffi.LONG) - null = lltype.nullptr(LONGP) - t0 = self.call(func, [null], rffi.LONG) - time.sleep(1) - t1 = self.call(func, [null], rffi.LONG) - assert t1 > t0 + def test_pointer_as_argument(self): + """#include + long inc(long* x) + { + long oldval; + if (x == NULL) + return -1; + oldval = *x; + *x = oldval+1; + return oldval; + } + """ + libfoo = self.get_libfoo() + func = (libfoo, 'inc', [types.pointer], types.slong) + LONGP = lltype.Ptr(rffi.CArray(rffi.LONG)) + null = lltype.nullptr(LONGP.TO) + res = self.call(func, [null], rffi.LONG) + assert res == -1 # - ptr_result = lltype.malloc(LONGP, 1, flavor='raw') - t2 = self.call(func, [ptr_result], rffi.LONG) - assert ptr_result[0] == t2 - lltype.free(ptr_result, flavor='raw') + ptr_result = lltype.malloc(LONGP.TO, 1, flavor='raw') + ptr_result[0] = 41 + res = self.call(func, [ptr_result], rffi.LONG) if self.__class__ is TestLibffiCall: + # the function was called only once + assert res == 41 + assert ptr_result[0] == 42 + lltype.free(ptr_result, flavor='raw') # the test does not make sense when run with the JIT through # meta_interp, because the __del__ are not properly called (hence # we "leak" memory) - del libc + del libfoo assert not ALLOCATED + else: + # the function as been called 9 times + assert res == 50 + assert ptr_result[0] == 51 + lltype.free(ptr_result, flavor='raw') def test_return_pointer(self): """ @@ -216,3 +230,4 @@ null = lltype.nullptr(LONGP.TO) res = self.call(func, [], LONGP, init_result=null) assert res[0] == 20 + From afa at codespeak.net Wed Oct 6 15:19:53 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Wed, 6 Oct 2010 15:19:53 +0200 (CEST) Subject: [pypy-svn] r77649 - in pypy/trunk/pypy/interpreter: . astcompiler/test Message-ID: <20101006131953.D459536C22E@codespeak.net> Author: afa Date: Wed Oct 6 15:19:52 2010 New Revision: 77649 Modified: pypy/trunk/pypy/interpreter/astcompiler/test/test_compiler.py pypy/trunk/pypy/interpreter/pycode.py pypy/trunk/pypy/interpreter/pyframe.py Log: very partial merge of r75516 from branch/fast-forward: allow these tests to run on top of CPython 2.7 (better identification of pypy and cpython bytecodes) Modified: pypy/trunk/pypy/interpreter/astcompiler/test/test_compiler.py ============================================================================== --- pypy/trunk/pypy/interpreter/astcompiler/test/test_compiler.py (original) +++ pypy/trunk/pypy/interpreter/astcompiler/test/test_compiler.py Wed Oct 6 15:19:52 2010 @@ -41,7 +41,7 @@ co_expr = compile(evalexpr, '', 'eval') space = self.space pyco_expr = PyCode._from_code(space, co_expr) - w_res = pyco_expr.exec_code(space, w_dict, w_dict) + w_res = pyco_expr.exec_host_bytecode(space, w_dict, w_dict) res = space.str_w(space.repr(w_res)) if not isinstance(expected, float): assert res == repr(expected) Modified: pypy/trunk/pypy/interpreter/pycode.py ============================================================================== --- pypy/trunk/pypy/interpreter/pycode.py (original) +++ pypy/trunk/pypy/interpreter/pycode.py Wed Oct 6 15:19:52 2010 @@ -253,6 +253,12 @@ tuple(self.co_freevars), tuple(self.co_cellvars) ) + def exec_host_bytecode(self, w_dict, w_globals, w_locals): + from pypy.interpreter.pyframe import CPythonFrame + frame = CPythonFrame(self.space, self, w_globals, None) + frame.setdictscope(w_locals) + return frame.run() + def dump(self): """A dis.dis() dump of the code object.""" co = self._to_code() Modified: pypy/trunk/pypy/interpreter/pyframe.py ============================================================================== --- pypy/trunk/pypy/interpreter/pyframe.py (original) +++ pypy/trunk/pypy/interpreter/pyframe.py Wed Oct 6 15:19:52 2010 @@ -13,6 +13,7 @@ from pypy.rlib.rarithmetic import intmask from pypy.rlib import jit, rstack from pypy.tool import stdlib_opcode +from pypy.tool.stdlib_opcode import host_bytecode_spec # Define some opcodes used g = globals() @@ -140,7 +141,8 @@ # the following 'assert' is an annotation hint: it hides from # the annotator all methods that are defined in PyFrame but # overridden in the {,Host}FrameClass subclasses of PyFrame. - assert isinstance(self, self.space.FrameClass) + assert (isinstance(self, self.space.FrameClass) or + not self.space.config.translating) executioncontext = self.space.getexecutioncontext() executioncontext.enter(self) try: @@ -634,6 +636,18 @@ return space.wrap(self.builtin is not space.builtin) return space.w_False +class CPythonFrame(PyFrame): + """ + Execution of host (CPython) opcodes. + """ + + bytecode_spec = host_bytecode_spec + opcode_method_names = host_bytecode_spec.method_names + opcodedesc = host_bytecode_spec.opcodedesc + opdescmap = host_bytecode_spec.opdescmap + HAVE_ARGUMENT = host_bytecode_spec.HAVE_ARGUMENT + + # ____________________________________________________________ def get_block_class(opname): From afa at codespeak.net Wed Oct 6 15:37:50 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Wed, 6 Oct 2010 15:37:50 +0200 (CEST) Subject: [pypy-svn] r77650 - pypy/trunk/pypy/interpreter/pyparser Message-ID: <20101006133750.E8E51282B90@codespeak.net> Author: afa Date: Wed Oct 6 15:37:49 2010 New Revision: 77650 Added: pypy/trunk/pypy/interpreter/pyparser/autopath.py - copied unchanged from r77638, pypy/branch/fast-forward/pypy/interpreter/pyparser/autopath.py pypy/trunk/pypy/interpreter/pyparser/genpytokenize.py - copied, changed from r77638, pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py pypy/trunk/pypy/interpreter/pyparser/pylexer.py - copied unchanged from r77638, pypy/branch/fast-forward/pypy/interpreter/pyparser/pylexer.py Modified: pypy/trunk/pypy/interpreter/pyparser/pytokenize.py Log: Revive the source code of the pypy tokenizer. "genpytokenize.py" outputs a fragment to be copied to pytokenize.py, and makes it possible to modify the grammar. I carefully looked at the differences: - some states have changed numbers. not a real change. - accepts[0] used to be True, i.e. the empty string was an accepted token! It's no more the case, and doesn't seem to be an issue. This was probably due to a bug in the grammar: triplestring was group(maybe(prefix), group(''', """)) instead of chain(maybe(prefix), group(''', """)) and this used to match the empty string... Copied: pypy/trunk/pypy/interpreter/pyparser/genpytokenize.py (from r77638, pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py) ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py (original) +++ pypy/trunk/pypy/interpreter/pyparser/genpytokenize.py Wed Oct 6 15:37:49 2010 @@ -58,19 +58,13 @@ maybe(states, groupStr(states, "lL"))) octNumber = chain(states, newArcPair(states, "0"), - maybe(states, groupStr(states, "oO")), any(states, groupStr(states, "01234567")), maybe(states, groupStr(states, "lL"))) - binNumber = chain(states, - newArcPair(states, "0"), - groupStr(states, "bB"), - any(states, groupStr(states, "01")), - maybe(states, groupStr(states, "lL"))) decNumber = chain(states, groupStr(states, "123456789"), any(states, makeDigits()), maybe(states, groupStr(states, "lL"))) - intNumber = group(states, hexNumber, octNumber, binNumber, decNumber) + intNumber = group(states, hexNumber, octNumber, decNumber) # ____________________________________________________________ # Exponents def makeExp (): @@ -138,7 +132,7 @@ # ____________________________________________________________ def makeStrPrefix (): return chain(states, - maybe(states, groupStr(states, "uUbB")), + maybe(states, groupStr(states, "uU")), maybe(states, groupStr(states, "rR"))) # ____________________________________________________________ contStr = group(states, @@ -244,10 +238,8 @@ "r" : None, "R" : None, "u" : None, - "U" : None, - "b" : None, - "B" : None} - for uniPrefix in ("", "u", "U", "b", "B", ): + "U" : None} + for uniPrefix in ("", "u", "U", ): for rawPrefix in ("", "r", "R"): prefix = uniPrefix + rawPrefix map[prefix + "'''"] = single3DFA @@ -267,7 +259,8 @@ i += 1 import StringIO print "states = [" - for state in dfa.states: + for numstate, state in enumerate(dfa.states): + print " #", numstate s = StringIO.StringIO() i = 0 for k, v in sorted(state.items()): @@ -283,14 +276,17 @@ s.write(', ') s.write('},') i = 0 - for line in textwrap.wrap(s.getvalue(), width=35): + if len(state) <= 4: + text = [s.getvalue()] + else: + text = textwrap.wrap(s.getvalue(), width=36) + for line in text: line = line.replace('::', ': ') if i == 0: print ' {' + line else: print ' ' + line i += 1 - print print " ]" print "%s = automata.%s(states, accepts)" % (name, dfa_class) print @@ -301,16 +297,8 @@ endDFAMap = makePyEndDFAMap() output("double3DFA", "NonGreedyDFA", endDFAMap['"""']) output("single3DFA", "NonGreedyDFA", endDFAMap["'''"]) - output("doubleDFA", "DFA", endDFAMap['"']) output("singleDFA", "DFA", endDFAMap["'"]) - print "endDFAs = {\"'\" : singleDFA," - print " '\"' : doubleDFA," - print " 'r' : None," - print " 'R' : None," - print " 'u' : None," - print " 'U' : None," - print " 'b' : None," - print " 'B' : None}" + output("doubleDFA", "DFA", endDFAMap['"']) # ______________________________________________________________________ Modified: pypy/trunk/pypy/interpreter/pyparser/pytokenize.py ============================================================================== --- pypy/trunk/pypy/interpreter/pyparser/pytokenize.py (original) +++ pypy/trunk/pypy/interpreter/pyparser/pytokenize.py Wed Oct 6 15:37:49 2010 @@ -21,40 +21,42 @@ __all__ = [ "tokenize" ] # ______________________________________________________________________ -# Automatically generated DFA's (with one or two hand tweeks): -pseudoStatesAccepts = [True, True, True, True, True, True, True, True, - True, True, False, True, True, True, False, False, - False, False, True, False, False, True, True, False, - True, False, True, False, True, False, True, False, - False, False, True, False, False, False, True] - -pseudoStates = [ - {'\t': 0, '\n': 13, '\x0c': 0, '\r': 14, ' ': 0, '!': 10, - '"': 16, '#': 18, '%': 12, '&': 12, - "'": 15, '(': 13, ')': 13, '*': 7, - '+': 12, ',': 13, '-': 12, '.': 6, - '/': 11, '0': 4, '1': 5, '2': 5, - '3': 5, '4': 5, '5': 5, '6': 5, - '7': 5, '8': 5, '9': 5, ':': 13, - ';': 13, '<': 9, '=': 12, '>': 8, - '@': 13, 'A': 1, - 'B': 1, 'C': 1, 'D': 1, 'E': 1, - 'F': 1, 'G': 1, 'H': 1, 'I': 1, - 'J': 1, 'K': 1, 'L': 1, 'M': 1, - 'N': 1, 'O': 1, 'P': 1, 'Q': 1, - 'R': 2, 'S': 1, 'T': 1, 'U': 3, - 'V': 1, 'W': 1, 'X': 1, 'Y': 1, - 'Z': 1, '[': 13, '\\': 17, ']': 13, - '^': 12, '_': 1, '`': 13, 'a': 1, - 'b': 1, 'c': 1, 'd': 1, 'e': 1, - 'f': 1, 'g': 1, 'h': 1, 'i': 1, - 'j': 1, 'k': 1, 'l': 1, 'm': 1, - 'n': 1, 'o': 1, 'p': 1, 'q': 1, - 'r': 2, 's': 1, 't': 1, 'u': 3, - 'v': 1, 'w': 1, 'x': 1, 'y': 1, - 'z': 1, '{': 13, '|': 12, '}': 13, - '~': 13}, +# Automatically generated DFA's +accepts = [False, True, True, True, True, True, True, True, + True, True, False, True, True, True, False, False, + False, False, True, True, True, False, True, + False, True, False, True, False, False, True, + False, False, False, False, True, False, False, + False, True] +states = [ + # 0 + {'\t': 0, '\n': 13, '\x0c': 0, + '\r': 14, ' ': 0, '!': 10, '"': 16, + '#': 18, '%': 12, '&': 12, "'": 15, + '(': 13, ')': 13, '*': 7, '+': 12, + ',': 13, '-': 12, '.': 6, '/': 11, + '0': 4, '1': 5, '2': 5, '3': 5, + '4': 5, '5': 5, '6': 5, '7': 5, + '8': 5, '9': 5, ':': 13, ';': 13, + '<': 9, '=': 12, '>': 8, '@': 13, + 'A': 1, 'B': 1, 'C': 1, 'D': 1, + 'E': 1, 'F': 1, 'G': 1, 'H': 1, + 'I': 1, 'J': 1, 'K': 1, 'L': 1, + 'M': 1, 'N': 1, 'O': 1, 'P': 1, + 'Q': 1, 'R': 2, 'S': 1, 'T': 1, + 'U': 3, 'V': 1, 'W': 1, 'X': 1, + 'Y': 1, 'Z': 1, '[': 13, '\\': 17, + ']': 13, '^': 12, '_': 1, '`': 13, + 'a': 1, 'b': 1, 'c': 1, 'd': 1, + 'e': 1, 'f': 1, 'g': 1, 'h': 1, + 'i': 1, 'j': 1, 'k': 1, 'l': 1, + 'm': 1, 'n': 1, 'o': 1, 'p': 1, + 'q': 1, 'r': 2, 's': 1, 't': 1, + 'u': 3, 'v': 1, 'w': 1, 'x': 1, + 'y': 1, 'z': 1, '{': 13, '|': 12, + '}': 13, '~': 13}, + # 1 {'0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, 'A': 1, 'B': 1, @@ -71,7 +73,7 @@ 'p': 1, 'q': 1, 'r': 1, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - + # 2 {'"': 16, "'": 15, '0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, @@ -89,7 +91,7 @@ 'r': 1, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - + # 3 {'"': 16, "'": 15, '0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, @@ -107,158 +109,182 @@ 'r': 2, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - - {'.': 24, '0': 22, '1': 22, '2': 22, - '3': 22, '4': 22, '5': 22, '6': 22, - '7': 22, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'L': 13, 'X': 21, 'e': 25, - 'j': 13, 'l': 13, 'x': 21}, - - {'.': 24, '0': 5, '1': 5, '2': 5, + # 4 + {'.': 22, '0': 20, '1': 20, '2': 20, + '3': 20, '4': 20, '5': 20, '6': 20, + '7': 20, '8': 21, '9': 21, 'E': 23, + 'J': 13, 'L': 13, 'X': 19, 'e': 23, + 'j': 13, 'l': 13, 'x': 19}, + # 5 + {'.': 22, '0': 5, '1': 5, '2': 5, '3': 5, '4': 5, '5': 5, '6': 5, - '7': 5, '8': 5, '9': 5, 'E': 25, - 'J': 13, 'L': 13, 'e': 25, 'j': 13, + '7': 5, '8': 5, '9': 5, 'E': 23, + 'J': 13, 'L': 13, 'e': 23, 'j': 13, 'l': 13}, - - {'0': 26, '1': 26, '2': 26, '3': 26, - '4': 26, '5': 26, '6': 26, '7': 26, - '8': 26, '9': 26}, - + # 6 + {'0': 24, '1': 24, '2': 24, '3': 24, + '4': 24, '5': 24, '6': 24, '7': 24, + '8': 24, '9': 24}, + # 7 {'*': 12, '=': 13}, - + # 8 {'=': 13, '>': 12}, - - {'=': 13, '<': 12, '>': 13}, - + # 9 + {'<': 12, '=': 13, '>': 13}, + # 10 {'=': 13}, - - {'=': 13, '/': 12}, - + # 11 + {'/': 12, '=': 13}, + # 12 {'=': 13}, - + # 13 {}, - + # 14 {'\n': 13}, - - {automata.DEFAULT: 19, '\n': 27, '\\': 29, "'": 28}, - - {automata.DEFAULT: 20, '"': 30, '\n': 27, '\\': 31}, - + # 15 + {automata.DEFAULT: 28, '\n': 25, "'": 26, '\\': 27}, + # 16 + {automata.DEFAULT: 31, '\n': 25, '"': 29, '\\': 30}, + # 17 {'\n': 13, '\r': 14}, - - {automata.DEFAULT: 18, '\n': 27, '\r': 27}, - - {automata.DEFAULT: 19, '\n': 27, '\\': 29, "'": 13}, - - {automata.DEFAULT: 20, '"': 13, '\n': 27, '\\': 31}, - - {'0': 21, '1': 21, '2': 21, '3': 21, - '4': 21, '5': 21, '6': 21, '7': 21, - '8': 21, '9': 21, 'A': 21, 'B': 21, - 'C': 21, 'D': 21, 'E': 21, 'F': 21, - 'L': 13, 'a': 21, 'b': 21, 'c': 21, - 'd': 21, 'e': 21, 'f': 21, 'l': 13}, - - {'.': 24, '0': 22, '1': 22, '2': 22, - '3': 22, '4': 22, '5': 22, '6': 22, - '7': 22, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'L': 13, 'e': 25, 'j': 13, + # 18 + {automata.DEFAULT: 18, '\n': 25, '\r': 25}, + # 19 + {'0': 19, '1': 19, '2': 19, '3': 19, + '4': 19, '5': 19, '6': 19, '7': 19, + '8': 19, '9': 19, 'A': 19, 'B': 19, + 'C': 19, 'D': 19, 'E': 19, 'F': 19, + 'L': 13, 'a': 19, 'b': 19, 'c': 19, + 'd': 19, 'e': 19, 'f': 19, 'l': 13}, + # 20 + {'.': 22, '0': 20, '1': 20, '2': 20, + '3': 20, '4': 20, '5': 20, '6': 20, + '7': 20, '8': 21, '9': 21, 'E': 23, + 'J': 13, 'L': 13, 'e': 23, 'j': 13, 'l': 13}, - - {'.': 24, '0': 23, '1': 23, '2': 23, - '3': 23, '4': 23, '5': 23, '6': 23, - '7': 23, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'e': 25, 'j': 13}, - - {'0': 24, '1': 24, '2': 24, '3': 24, - '4': 24, '5': 24, '6': 24, '7': 24, - '8': 24, '9': 24, 'E': 32, 'J': 13, + # 21 + {'.': 22, '0': 21, '1': 21, '2': 21, + '3': 21, '4': 21, '5': 21, '6': 21, + '7': 21, '8': 21, '9': 21, 'E': 23, + 'J': 13, 'e': 23, 'j': 13}, + # 22 + {'0': 22, '1': 22, '2': 22, '3': 22, + '4': 22, '5': 22, '6': 22, '7': 22, + '8': 22, '9': 22, 'E': 32, 'J': 13, 'e': 32, 'j': 13}, - + # 23 {'+': 33, '-': 33, '0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34}, - - {'0': 26, '1': 26, '2': 26, '3': 26, - '4': 26, '5': 26, '6': 26, '7': 26, - '8': 26, '9': 26, 'E': 32, 'J': 13, + # 24 + {'0': 24, '1': 24, '2': 24, '3': 24, + '4': 24, '5': 24, '6': 24, '7': 24, + '8': 24, '9': 24, 'E': 32, 'J': 13, 'e': 32, 'j': 13}, - + # 25 {}, - + # 26 {"'": 13}, - + # 27 {automata.DEFAULT: 35, '\n': 13, '\r': 14}, - + # 28 + {automata.DEFAULT: 28, '\n': 25, "'": 13, '\\': 27}, + # 29 {'"': 13}, - + # 30 {automata.DEFAULT: 36, '\n': 13, '\r': 14}, - + # 31 + {automata.DEFAULT: 31, '\n': 25, '"': 13, '\\': 30}, + # 32 {'+': 37, '-': 37, '0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38}, - - + # 33 {'0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34}, - + # 34 {'0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34, 'J': 13, 'j': 13}, - - {automata.DEFAULT: 35, '\n': 27, '\\': 29, "'": 13}, - - {automata.DEFAULT: 36, '"': 13, '\n': 27, '\\': 31}, - + # 35 + {automata.DEFAULT: 35, '\n': 25, "'": 13, '\\': 27}, + # 36 + {automata.DEFAULT: 36, '\n': 25, '"': 13, '\\': 30}, + # 37 {'0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38}, - + # 38 {'0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38, 'J': 13, 'j': 13}, ] +pseudoDFA = automata.DFA(states, accepts) -pseudoDFA = automata.DFA(pseudoStates, pseudoStatesAccepts) - -double3StatesAccepts = [False, False, False, False, False, True] -double3States = [ +accepts = [False, False, False, False, False, True] +states = [ + # 0 {automata.DEFAULT: 0, '"': 1, '\\': 2}, + # 1 {automata.DEFAULT: 4, '"': 3, '\\': 2}, + # 2 {automata.DEFAULT: 4}, + # 3 {automata.DEFAULT: 4, '"': 5, '\\': 2}, + # 4 {automata.DEFAULT: 4, '"': 1, '\\': 2}, + # 5 {automata.DEFAULT: 4, '"': 5, '\\': 2}, ] -double3DFA = automata.NonGreedyDFA(double3States, double3StatesAccepts) +double3DFA = automata.NonGreedyDFA(states, accepts) -single3StatesAccepts = [False, False, False, False, False, True] -single3States = [ - {automata.DEFAULT: 0, '\\': 2, "'": 1}, - {automata.DEFAULT: 4, '\\': 2, "'": 3}, +accepts = [False, False, False, False, False, True] +states = [ + # 0 + {automata.DEFAULT: 0, "'": 1, '\\': 2}, + # 1 + {automata.DEFAULT: 4, "'": 3, '\\': 2}, + # 2 {automata.DEFAULT: 4}, - {automata.DEFAULT: 4, '\\': 2, "'": 5}, - {automata.DEFAULT: 4, '\\': 2, "'": 1}, - {automata.DEFAULT: 4, '\\': 2, "'": 5}, + # 3 + {automata.DEFAULT: 4, "'": 5, '\\': 2}, + # 4 + {automata.DEFAULT: 4, "'": 1, '\\': 2}, + # 5 + {automata.DEFAULT: 4, "'": 5, '\\': 2}, ] -single3DFA = automata.NonGreedyDFA(single3States, single3StatesAccepts) +single3DFA = automata.NonGreedyDFA(states, accepts) -singleStatesAccepts = [False, True, False] -singleStates = [ - {automata.DEFAULT: 0, '\\': 2, "'": 1}, +accepts = [False, True, False, False] +states = [ + # 0 + {automata.DEFAULT: 0, "'": 1, '\\': 2}, + # 1 {}, - {automata.DEFAULT: 0}, + # 2 + {automata.DEFAULT: 3}, + # 3 + {automata.DEFAULT: 3, "'": 1, '\\': 2}, ] -singleDFA = automata.DFA(singleStates, singleStatesAccepts) +singleDFA = automata.DFA(states, accepts) -doubleStatesAccepts = [False, True, False] -doubleStates = [ +accepts = [False, True, False, False] +states = [ + # 0 {automata.DEFAULT: 0, '"': 1, '\\': 2}, + # 1 {}, - {automata.DEFAULT: 0}, + # 2 + {automata.DEFAULT: 3}, + # 3 + {automata.DEFAULT: 3, '"': 1, '\\': 2}, ] -doubleDFA = automata.DFA(doubleStates, doubleStatesAccepts) +doubleDFA = automata.DFA(states, accepts) + + +#_______________________________________________________________________ +# End of automatically generated DFA's endDFAs = {"'" : singleDFA, '"' : doubleDFA, From david at codespeak.net Wed Oct 6 15:48:50 2010 From: david at codespeak.net (david at codespeak.net) Date: Wed, 6 Oct 2010 15:48:50 +0200 (CEST) Subject: [pypy-svn] r77651 - in pypy/branch/arm-backend/pypy/jit/backend/arm: . test Message-ID: <20101006134850.C678B36C22E@codespeak.net> Author: david Date: Wed Oct 6 15:48:49 2010 New Revision: 77651 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Log: Add encoding for MOV reg, reg Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Wed Oct 6 15:48:49 2010 @@ -38,7 +38,8 @@ self.mc.write32(0xe89da800) # ldm sp, {fp, sp, pc} def gen_preamble(self): - self.mc.write32(0xe1a0c00d) # mov ip, sp + self.mc.MOV_rr(r.ip, r.sp) + #self.mc.write32(0xe1a0c00d) # mov ip, sp self.mc.write32(0xe92dd800) #push {fp, ip, lr, pc} self.mc.write32(0xe24cb004) # sub fp, ip, #4 ; 0x4 self.mc.write32(0xe24dd008) #sub sp, sp, #8 ; 0x8 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py Wed Oct 6 15:48:49 2010 @@ -25,6 +25,13 @@ | (rn & 0xF) << 16 | (rt & 0xF) << 12 | (imm & 0xFFF)) + def MOV_rr(self, rd, rm, cond=cond.AL, s=0): + self.write32(cond << 28 + | 0xD << 21 + | (s & 0x1) << 20 + | (rd & 0xFF) << 12 + | (rm & 0xFF)) + def MOV_ri(self, rt, imm=0, cond=cond.AL): # XXX Check the actual allowed size for imm # XXX S bit Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Wed Oct 6 15:48:49 2010 @@ -24,6 +24,10 @@ self.cb.ADD_ri(r.r0, r.r1, 1) self.assert_equal("ADD r0, r1, #1") + def test_mov_rr(self): + self.cb.MOV_rr(r.r7, r.r12) + self.assert_equal("MOV r7, r12") + def test_mov_ri(self): self.cb.MOV_ri(r.r9, 123) self.assert_equal("MOV r9, #123") From cfbolz at codespeak.net Wed Oct 6 16:02:59 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 6 Oct 2010 16:02:59 +0200 (CEST) Subject: [pypy-svn] r77652 - pypy/extradoc/talk/pepm2011 Message-ID: <20101006140259.2427D282B90@codespeak.net> Author: cfbolz Date: Wed Oct 6 16:02:57 2010 New Revision: 77652 Added: pypy/extradoc/talk/pepm2011/math.lyx Log: add the formulas as a LyX document Added: pypy/extradoc/talk/pepm2011/math.lyx ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pepm2011/math.lyx Wed Oct 6 16:02:57 2010 @@ -0,0 +1,520 @@ +#LyX 1.6.7 created this file. For more info see http://www.lyx.org/ +\lyxformat 345 +\begin_document +\begin_header +\textclass article +\use_default_options true +\language english +\inputencoding auto +\font_roman default +\font_sans default +\font_typewriter default +\font_default_family default +\font_sc false +\font_osf false +\font_sf_scale 100 +\font_tt_scale 100 + +\graphics default +\paperfontsize default +\spacing single +\use_hyperref false +\papersize a4paper +\use_geometry false +\use_amsmath 1 +\use_esint 1 +\cite_engine basic +\use_bibtopic false +\paperorientation portrait +\secnumdepth 3 +\tocdepth 3 +\paragraph_separation skip +\defskip medskip +\quotes_language english +\papercolumns 1 +\papersides 1 +\paperpagestyle default +\tracking_changes false +\output_changes false +\author "" +\author "" +\end_header + +\begin_body + +\begin_layout Section* +Semantics +\end_layout + +\begin_layout Standard +\begin_inset Tabular + + + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\emph on +new +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{l\,\mathrm{fresh}}{v=\mathrm{new}(T),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto l\right],H\left[l\mapsto\left(T,\mathrm{null},\mathrm{null}\right)\right]}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\emph on +get +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(u)\neq\mathrm{null}}{v=\mathrm{get}(u,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto H\left(E\left(u\right)\right)_{F}\right],H}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle {\displaystyle \frac{E(u)=\mathrm{null}}{v=\mathrm{get}(u,F),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\emph on +set +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(v)\neq\mathrm{null}}{\mathrm{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H\left[E\left(v\right)\mapsto\left(H\left(E\left(v\right)\right)!_{F}E(u)\right)\right]}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(v)=\mathrm{null}}{\mathrm{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\emph on +guard +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(v)\neq\mathrm{null},\,\mathrm{type}(H(E(v))=T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(v)=\mathrm{null}\vee\mathrm{type}(H(E(v))\neq T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\end_inset + + +\begin_inset Newline newline +\end_inset + + +\begin_inset Newline newline +\end_inset + + +\end_layout + +\begin_layout Section* +Optimizations +\end_layout + +\begin_layout Standard +\begin_inset Tabular + + + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\emph on +new +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{v^{*}\,\mathrm{fresh}}{v=\mathrm{new}(T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto v^{*}\right],S\left[v^{*}\mapsto\left(T,\mathrm{null,null}\right)\right]}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\emph on +get +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(u)\in\mathrm{dom}(S)}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto S(E(u))_{F}\right],S}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(u)\notin\mathrm{dom}(S)\, v^{*}\,\mathrm{fresh}}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle v^{*}=\mathrm{get}(E(u),F)\right\rangle ,E\left[v\mapsto v^{*}\right],S}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\emph on +set +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S\left[E\left(v\right)\mapsto\left(S(E(v))!_{F}E(u)\right)\right]}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}\left(S\right),\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathrm{set}\left(E(v),F,E(u)\right)\right\rangle ,E,S^{\prime}}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\emph on +guard +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\emph on +lifting +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{v^{*}\notin\mathrm{dom}(S)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle \,\right\rangle ,S}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathrm{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset Formula ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathrm{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathrm{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$ +\end_inset + + +\end_layout + +\end_inset + + + + +\end_inset + + +\end_layout + +\end_body +\end_document From cfbolz at codespeak.net Wed Oct 6 16:04:27 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 6 Oct 2010 16:04:27 +0200 (CEST) Subject: [pypy-svn] r77653 - pypy/extradoc/talk/pepm2011 Message-ID: <20101006140427.7EFA3282B90@codespeak.net> Author: cfbolz Date: Wed Oct 6 16:04:26 2010 New Revision: 77653 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: change title, add michael as an author, refactor some small things Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 6 16:04:26 2010 @@ -52,9 +52,9 @@ \CopyrightYear{XXX} \copyrightdata{XXX} -\title{Escape Analysis and Specialization in a Tracing JIT} +\title{Allocation Removal by Partial Evaluation in a Tracing JIT} -\authorinfo{Carl Friedrich Bolz \and Antonio Cuni \and Maciej Fija?kowski \and Samuele Pedroni \and Armin Rigo} +\authorinfo{Carl Friedrich Bolz \and Antonio Cuni \and Maciej Fija?kowski \and Michael Leuschel \and Samuele Pedroni \and Armin Rigo} {Heinrich-Heine-Universit?t D?sseldorf, STUPS Group, Germany XXX} {cfbolz at gmx.de, anto.cuni at gmail.com, fijal at merlinux.eu, samuele.pedroni at gmail.com, arigo at tunes.org} @@ -488,7 +488,7 @@ The creation of objects in category 1 is removed by the optimization described in Section~\ref{sec:virtuals}. XXX -\section{Escape Analysis in a Tracing JIT} +\section{Escape Analysis in a Tracing JIT} % XXX imprecise title \label{sec:virtuals} @@ -505,23 +505,22 @@ This process is called \emph{escape analysis}. The escape analysis of our tracing JIT works by using \emph{virtual objects}: The trace is walked from beginning to end and whenever a \texttt{new} operation is seen, the operation is -removed and a virtual object\arigo{XXX what I have in mind when I talk -of ``virtual object'' is the run-time behavior -- i.e. a real object that -would exist at run-time, except that it has be virtual-ized. Here you seem -to mean rather ``virtual object description'' or something.} -is constructed. The virtual object summarizes the -shape of the object that is allocated at this position in the original trace, -and is used by the optimization to improve the trace. The shapes describe +removed and a shape description is constructed and associated with the variable +that would have stored the result of \texttt{new}. This result, a newly +allocated object, is called \emph{virtual}, because after the optimization is +finished, it does not need to be allocated at all. The shape description summarizes +the shape of this original object +and is used by the optimization to improve the trace. The shape describes where the values that would be stored in the fields of the allocated objects -come from. Whenever the optimizer sees a \texttt{setfield} that writes into a virtual -object, that shape summary is thus updated and the operation can be removed. -When the optimizer encounters a \texttt{getfield} from a virtual, the result is read -from the virtual object, and the operation is also removed. -Equivalently, a \texttt{guard\_class} on a virtual object can be removed as -well, because the virtual object has a fixed and known class. +come from. Whenever the optimizer sees a \texttt{setfield} that writes into such +an object, that shape description is updated and the operation can be removed. +When the optimizer encounters a \texttt{getfield} from such an object, the result is read +from the shape description, and the operation is also removed. +Equivalently, a \texttt{guard\_class} on a variable that has a shape description can be removed as +well, because the shape description stores the class. In the example from last section, the following operations would produce two -virtual objects, and be completely removed from the optimized trace: +shape descriptions, and be completely removed from the optimized trace: \texttt{ \begin{tabular}{l} @@ -532,9 +531,10 @@ \end{tabular} } -The virtual object stored in $p_{5}$ would know that it is an \texttt{BoxedInteger}, and that -the \texttt{intval} field contains $i_{4}$, the one stored in $p_{6}$ would know that -its \texttt{intval} field contains the constant -100. +The shape description associated with $p_{5}$ would know that it is an +\texttt{BoxedInteger}, and that the \texttt{intval} field contains $i_{4}$, the +associated with $p_{6}$ would know that its \texttt{intval} field contains the +constant -100. The following operations, that use $p_{5}$ and $p_{6}$ could then be optimized using that knowledge: @@ -569,7 +569,9 @@ object is used in any other operation, it cannot stay virtual. For example, when a virtual object is stored in a globally accessible place, the object needs to actually be -allocated, as it might live longer than one iteration of the loop. +allocated, as it might live longer than one iteration of the loop. The actual +allocation is inserted just before the operation that requires the object to be +non-virtual. This is what happens at the end of the trace in Figure~\ref{fig:unopt-trace}, when the \texttt{jump} operation is hit. The arguments of the jump are at this point virtual objects. Before the @@ -607,12 +609,12 @@ The optimized trace contains only two allocations, instead of the original five, and only three \texttt{guard\_class} operations, from the original seven. -\subsection{Algorithm} -\label{sub:Algorithm} +\section{Formal Description of the Algorithm} +\label{sec:Algorithm} XXX want some sort of pseudo-code -% subsection Algorithm (end) +% section Formal Description of the Algorithm (end) %___________________________________________________________________________ From afa at codespeak.net Wed Oct 6 16:41:04 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Wed, 6 Oct 2010 16:41:04 +0200 (CEST) Subject: [pypy-svn] r77654 - in pypy/branch/fast-forward/pypy/interpreter/astcompiler: . tools Message-ID: <20101006144104.120F436C225@codespeak.net> Author: afa Date: Wed Oct 6 16:41:01 2010 New Revision: 77654 Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/ast.py pypy/branch/fast-forward/pypy/interpreter/astcompiler/tools/asdl_py.py Log: Regenerated ast.py. The change probably means that code inside dict and set comprehension was not optimized. Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/ast.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/ast.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/ast.py Wed Oct 6 16:41:01 2010 @@ -1499,6 +1499,8 @@ def mutate_over(self, visitor): self.elt = self.elt.mutate_over(visitor) + if self.generators: + visitor._mutate_sequence(self.generators) return visitor.visit_SetComp(self) def sync_app_attrs(self, space): @@ -1540,6 +1542,8 @@ def mutate_over(self, visitor): self.key = self.key.mutate_over(visitor) self.value = self.value.mutate_over(visitor) + if self.generators: + visitor._mutate_sequence(self.generators) return visitor.visit_DictComp(self) def sync_app_attrs(self, space): Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/tools/asdl_py.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/tools/asdl_py.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/tools/asdl_py.py Wed Oct 6 16:41:01 2010 @@ -576,7 +576,7 @@ class _FieldsWrapper(Wrappable): - "Hack around the fact we can't store tuples on a TypeDef." + "Hack around the fact we can\'t store tuples on a TypeDef." def __init__(self, fields): self.fields = fields From david at codespeak.net Wed Oct 6 17:00:31 2010 From: david at codespeak.net (david at codespeak.net) Date: Wed, 6 Oct 2010 17:00:31 +0200 (CEST) Subject: [pypy-svn] r77655 - in pypy/branch/arm-backend/pypy/jit/backend/arm: . test Message-ID: <20101006150031.83B95282B90@codespeak.net> Author: david Date: Wed Oct 6 17:00:29 2010 New Revision: 77655 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Log: Encode and use PUSH instr to push a set of registers to the stack Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Wed Oct 6 17:00:29 2010 @@ -17,7 +17,7 @@ def assemble_loop(self, inputargs, operations, looptoken): assert len(inputargs) == 1 reg = 0 - self.gen_preamble() + self.gen_func_prolog() addr = self.fail_boxes_int.get_addr_for_num(0) self.gen_load_int(r.r3, addr) self.mc.LDR_ri(r.r2, r.r3) @@ -28,19 +28,18 @@ n = self.cpu.get_fail_descr_number(op.getdescr()) self.mc.MOV_ri(r.r0, n) self.mc.STR_ri(r.r1, r.r3) - self.gen_out() + self.gen_func_epilog() - def gen_out(self): + def gen_func_epilog(self): self.mc.write32(0xe50b3010) # str r3, [fp, #-16] self.mc.write32(0xe51b3010) # ldr r3, [fp, #-16] #self.mc.write32(0xe1a00003) # mov r0, r3 self.mc.write32(0xe24bd00c) # sub sp, fp, #12 ; 0xc self.mc.write32(0xe89da800) # ldm sp, {fp, sp, pc} - def gen_preamble(self): + def gen_func_prolog(self): self.mc.MOV_rr(r.ip, r.sp) - #self.mc.write32(0xe1a0c00d) # mov ip, sp - self.mc.write32(0xe92dd800) #push {fp, ip, lr, pc} + self.mc.PUSH([r.fp, r.ip, r.lr, r.pc]) self.mc.write32(0xe24cb004) # sub fp, ip, #4 ; 0x4 self.mc.write32(0xe24dd008) #sub sp, sp, #8 ; 0x8 self.mc.write32(0xe50b0014) # str r0, [fp, #-20] Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py Wed Oct 6 17:00:29 2010 @@ -70,6 +70,12 @@ | (shifttype & 0x3) << 5 | (rm & 0xFF)) + def PUSH(self, regs, cond=cond.AL): + instr = cond << 28 | 0x92D << 16 + for reg in regs: + instr |= 0x1 << reg + self.write32(instr) + def write32(self, word): self.writechar(chr(word & 0xFF)) self.writechar(chr((word >> 8) & 0xFF)) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py Wed Oct 6 17:00:29 2010 @@ -1,5 +1,6 @@ r0, r1, r2, r3, r4, r5, r6, r7, r8, r9, r10, r11, r12, r13, r14, r15 = range(16) # aliases for registers +fp = 11 ip = 12 sp = 13 lr = 14 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py Wed Oct 6 17:00:29 2010 @@ -9,28 +9,28 @@ self.a = AssemblerARM(None) def test_load_small_int_to_reg(self): - self.a.gen_preamble() + self.a.gen_func_prolog() self.a.gen_load_int(r.r0, 123) - self.a.gen_out() + self.a.gen_func_epilog() assert run_asm(self.a) == 123 def test_load_medium_int_to_reg(self): - self.a.gen_preamble() + self.a.gen_func_prolog() self.a.gen_load_int(r.r0, 0xBBD7) - self.a.gen_out() + self.a.gen_func_epilog() assert run_asm(self.a) == 48087 def test_load_int_to_reg(self): - self.a.gen_preamble() + self.a.gen_func_prolog() self.a.gen_load_int(r.r0, 0xFFFFFF85) - self.a.gen_out() + self.a.gen_func_epilog() assert run_asm(self.a) == -123 def test_or(self): - self.a.gen_preamble() + self.a.gen_func_prolog() self.a.mc.MOV_ri(r.r1, 8) self.a.mc.MOV_ri(r.r2, 8) self.a.mc.ORR_rr(r.r0, r.r1, r.r2, 4) - self.a.gen_out() + self.a.gen_func_epilog() assert run_asm(self.a) == 0x88 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Wed Oct 6 17:00:29 2010 @@ -18,32 +18,32 @@ def test_ldr(self): self.cb.LDR_ri(r.r0, r.r1) - self.assert_equal("LDR r0, [r1]") + self.assert_equal('LDR r0, [r1]') def test_add_ri(self): self.cb.ADD_ri(r.r0, r.r1, 1) - self.assert_equal("ADD r0, r1, #1") + self.assert_equal('ADD r0, r1, #1') def test_mov_rr(self): self.cb.MOV_rr(r.r7, r.r12) - self.assert_equal("MOV r7, r12") + self.assert_equal('MOV r7, r12') def test_mov_ri(self): self.cb.MOV_ri(r.r9, 123) - self.assert_equal("MOV r9, #123") + self.assert_equal('MOV r9, #123') def test_mov_ri2(self): self.cb.MOV_ri(r.r9, 255) - self.assert_equal("MOV r9, #255") + self.assert_equal('MOV r9, #255') def test_mov_ri_max(self): - py.test.skip("Check the actual largest thing") + py.test.skip('Check the actual largest thing') self.cb.MOV_ri(r.r9, 0xFFF) - self.assert_equal("MOV r9, #4095") + self.assert_equal('MOV r9, #4095') def test_str_ri(self): self.cb.STR_ri(r.r9, r.r14) - self.assert_equal("STR r9, [r14]") + self.assert_equal('STR r9, [r14]') def test_asr_ri(self): self.cb.ASR_ri(r.r7, r.r5, 24) @@ -57,6 +57,18 @@ self.cb.ORR_rr(r.r0, r.r7,r.r12, 8) self.assert_equal('ORR r0, r7, r12, lsl #8') + def test_push_one_reg(self): + self.cb.PUSH([r.r1]) + self.assert_equal('PUSH {r1}') + + def test_push_multiple(self): + self.cb.PUSH([r.r3, r.r1, r.r6, r.r8, r.sp, r.pc]) + self.assert_equal('PUSH {r3, r1, r6, r8, sp, pc}') + + def test_push_multiple2(self): + self.cb.PUSH([r.fp, r.ip, r.lr, r.pc]) + self.assert_equal('PUSH {fp, ip, lr, pc}') + def assert_equal(self, asm): assert self.cb.hexdump() == assemble(asm) From david at codespeak.net Wed Oct 6 17:36:10 2010 From: david at codespeak.net (david at codespeak.net) Date: Wed, 6 Oct 2010 17:36:10 +0200 (CEST) Subject: [pypy-svn] r77656 - in pypy/branch/arm-backend/pypy/jit/backend/arm: . test Message-ID: <20101006153610.D6628282B90@codespeak.net> Author: david Date: Wed Oct 6 17:36:09 2010 New Revision: 77656 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Log: Encoding for SUB instr Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Wed Oct 6 17:36:09 2010 @@ -34,14 +34,14 @@ self.mc.write32(0xe50b3010) # str r3, [fp, #-16] self.mc.write32(0xe51b3010) # ldr r3, [fp, #-16] #self.mc.write32(0xe1a00003) # mov r0, r3 - self.mc.write32(0xe24bd00c) # sub sp, fp, #12 ; 0xc + self.mc.SUB_ri(r.sp, r.fp, 12) self.mc.write32(0xe89da800) # ldm sp, {fp, sp, pc} def gen_func_prolog(self): self.mc.MOV_rr(r.ip, r.sp) self.mc.PUSH([r.fp, r.ip, r.lr, r.pc]) - self.mc.write32(0xe24cb004) # sub fp, ip, #4 ; 0x4 - self.mc.write32(0xe24dd008) #sub sp, sp, #8 ; 0x8 + self.mc.SUB_ri(r.fp, r.ip, 4) + self.mc.SUB_ri(r.sp, r.sp, 8) self.mc.write32(0xe50b0014) # str r0, [fp, #-20] def gen_load_int(self, reg, value): Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py Wed Oct 6 17:36:09 2010 @@ -25,6 +25,16 @@ | (rn & 0xF) << 16 | (rt & 0xF) << 12 | (imm & 0xFFF)) + + def SUB_ri(self, rd, rn, imm=0, cond=cond.AL, s=0): + self.write32(cond << 28 + | 9 << 22 + | (s & 0x1) << 20 + | (rn & 0xF) << 16 + | (rd & 0xF) << 12 + | (imm & 0xFFF)) + + def MOV_rr(self, rd, rm, cond=cond.AL, s=0): self.write32(cond << 28 | 0xD << 21 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py Wed Oct 6 17:36:09 2010 @@ -34,3 +34,10 @@ self.a.mc.ORR_rr(r.r0, r.r1, r.r2, 4) self.a.gen_func_epilog() assert run_asm(self.a) == 0x88 + + def test_sub(self): + self.a.gen_func_prolog() + self.a.gen_load_int(r.r1, 123456) + self.a.mc.SUB_ri(r.r0, r.r1, 123) + self.a.gen_func_epilog() + assert run_asm(self.a) == 123333 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Wed Oct 6 17:36:09 2010 @@ -69,6 +69,15 @@ self.cb.PUSH([r.fp, r.ip, r.lr, r.pc]) self.assert_equal('PUSH {fp, ip, lr, pc}') + def test_sub_ri(self): + self.cb.SUB_ri(r.r2, r.r4, 123) + self.assert_equal('SUB r2, r4, #123') + + def test_sub_ri2(self): + py.test.skip('XXX check the actual largest value') + self.cb.SUB_ri(r.r3, r.r7, 0xFFF) + self.assert_equal('SUB r3, r7, #4095') + def assert_equal(self, asm): assert self.cb.hexdump() == assemble(asm) From fijal at codespeak.net Wed Oct 6 18:22:49 2010 From: fijal at codespeak.net (fijal at codespeak.net) Date: Wed, 6 Oct 2010 18:22:49 +0200 (CEST) Subject: [pypy-svn] r77657 - pypy/trunk/pypy/jit/backend/x86 Message-ID: <20101006162249.1825636C22E@codespeak.net> Author: fijal Date: Wed Oct 6 18:22:47 2010 New Revision: 77657 Modified: pypy/trunk/pypy/jit/backend/x86/assembler.py Log: Don't be too smart. We get clashes if we try to index loops by code name (why not on the other hand). Instead keep numbers. Also simplify the way it's written out. Modified: pypy/trunk/pypy/jit/backend/x86/assembler.py ============================================================================== --- pypy/trunk/pypy/jit/backend/x86/assembler.py (original) +++ pypy/trunk/pypy/jit/backend/x86/assembler.py Wed Oct 6 18:22:47 2010 @@ -244,7 +244,7 @@ f = open_file_as_stream(output_log, "w") for i in range(len(self.loop_run_counters)): name, struct = self.loop_run_counters[i] - f.write(str(struct.i) + " " * (8 - len(str(struct.i))) + name + "\n") + f.write(str(name) + ":" + str(struct.i) + "\n") f.close() def _build_float_constants(self): @@ -401,7 +401,7 @@ if self._debug: struct = lltype.malloc(DEBUG_COUNTER, flavor='raw') struct.i = 0 - self.loop_run_counters.append((funcname, struct)) + self.loop_run_counters.append((len(self.loop_run_counters), struct)) return funcname def patch_jump_for_descr(self, faildescr, adr_new_target): From antocuni at codespeak.net Wed Oct 6 18:27:29 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 6 Oct 2010 18:27:29 +0200 (CEST) Subject: [pypy-svn] r77658 - pypy/branch/jitffi/pypy/jit/backend/test Message-ID: <20101006162729.854C0282B90@codespeak.net> Author: antocuni Date: Wed Oct 6 18:27:27 2010 New Revision: 77658 Modified: pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py Log: ops, I forgot to change this Modified: pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py Wed Oct 6 18:27:27 2010 @@ -520,7 +520,7 @@ from pypy.rlib.libffi import CDLL, types, ArgChain libc = CDLL('libc.so.6') c_tolower = libc.getpointer('tolower', [types.uchar], types.sint) - argchain = ArgChain().int(ord('A')) + argchain = ArgChain().arg(ord('A')) assert c_tolower.call(argchain, rffi.INT) == ord('a') func_adr = llmemory.cast_ptr_to_adr(c_tolower.funcsym) From antocuni at codespeak.net Wed Oct 6 18:28:59 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 6 Oct 2010 18:28:59 +0200 (CEST) Subject: [pypy-svn] r77659 - pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt Message-ID: <20101006162859.19E26282B90@codespeak.net> Author: antocuni Date: Wed Oct 6 18:28:57 2010 New Revision: 77659 Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py Log: disable this for now, else all translation tests fail Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py Wed Oct 6 18:28:57 2010 @@ -15,7 +15,7 @@ OptRewrite(), OptVirtualize(), OptHeap(), - OptFfiCall(), + #OptFfiCall(), ] optimizer = Optimizer(metainterp_sd, loop, optimizations, virtuals) optimizer.propagate_all_forward() From cfbolz at codespeak.net Wed Oct 6 18:47:05 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 6 Oct 2010 18:47:05 +0200 (CEST) Subject: [pypy-svn] r77660 - pypy/extradoc/talk/pepm2011 Message-ID: <20101006164705.86430282B9E@codespeak.net> Author: cfbolz Date: Wed Oct 6 18:47:04 2010 New Revision: 77660 Modified: pypy/extradoc/talk/pepm2011/math.lyx Log: shuffle stuff around Modified: pypy/extradoc/talk/pepm2011/math.lyx ============================================================================== --- pypy/extradoc/talk/pepm2011/math.lyx (original) +++ pypy/extradoc/talk/pepm2011/math.lyx Wed Oct 6 18:47:04 2010 @@ -43,15 +43,85 @@ \begin_body \begin_layout Section* +Definitions +\end_layout + +\begin_layout Standard +Object Domains: +\begin_inset Formula \[ +\begin{array}{lccc} +u,v,w & \in & V & \mathrm{are\, Variables}\\ +\\\\\\\end{array}\] + +\end_inset + + +\end_layout + +\begin_layout Standard +\begin_inset Formula \begin{eqnarray*} +u,v,w & \in & V\mathrm{\,\, are\, Variables\, in\, the\, original\, trace}\\ +u^{*},v^{*},w^{*} & \in & V^{*}\,\mathrm{\, are\, Variables\, in\, the\, optimized\, trace}\\ +T & \in & \mathfrak{T}\mathrm{\,\, are\, runtime\, types}\\ +F & \in & \left\{ L,R\right\} \,\mathrm{\, are\, fields\, of\, objects}\\ +l & \in & L\,\mathrm{\, are\, locations\, in\, the\, heap}\end{eqnarray*} + +\end_inset + + +\end_layout + +\begin_layout Standard +Available Operations: +\end_layout + +\begin_layout Standard +\begin_inset Formula \begin{eqnarray*} + & & v=\mathrm{new}(T)\,\,\mathrm{make\, a\, new\, object}\\ + & & v=\mathrm{get}(u,F)\,\,\mathrm{read\, a\, field}\\ + & & \mathrm{set}\left(v,F,u\right)\,\,\mathrm{write\, a\, field}\\ + & & \mathrm{guard}(v,T)\,\,\mathrm{check\, the\, type\, of\, an\, object}\end{eqnarray*} + +\end_inset + + +\end_layout + +\begin_layout Standard +State of the Interpreter: +\begin_inset Formula \begin{eqnarray*} +E & \in & V\rightharpoonup L\\ +H & \in & L\rightharpoonup\mathfrak{T}\times\left(L\cup\left\{ \mathrm{null}\right\} \right)\times\left(L\cup\left\{ \mathrm{null}\right\} \right)\end{eqnarray*} + +\end_inset + + +\end_layout + +\begin_layout Standard +State of the Optimizer: +\begin_inset Formula \begin{eqnarray*} +E & \in & V\rightharpoonup V^{*}\\ +H & \in & L\rightharpoonup\mathfrak{T}\times\left(V^{*}\cup\left\{ \mathrm{null}\right\} \right)\times\left(V^{*}\cup\left\{ \mathrm{null}\right\} \right)\end{eqnarray*} + +\end_inset + + +\end_layout + +\begin_layout Section* Semantics \end_layout \begin_layout Standard \begin_inset Tabular - + + + + \begin_inset Text @@ -76,15 +146,33 @@ \end_inset - - + +\begin_inset Text + +\begin_layout Plain Layout +\begin_inset space ~ +\end_inset + + +\begin_inset space ~ +\end_inset + + +\begin_inset space ~ +\end_inset + + +\end_layout + +\end_inset + \begin_inset Text \begin_layout Plain Layout \emph on -get +guard \end_layout \end_inset @@ -93,7 +181,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(u)\neq\mathrm{null}}{v=\mathrm{get}(u,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto H\left(E\left(u\right)\right)_{F}\right],H}}$ +\begin_inset Formula ${\displaystyle \frac{\mathrm{type}(H(E(v))=T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H}}$ \end_inset @@ -108,6 +196,8 @@ \begin_layout Plain Layout +\emph on +get \end_layout \end_inset @@ -116,7 +206,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle {\displaystyle \frac{E(u)=\mathrm{null}}{v=\mathrm{get}(u,F),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}}$ +\begin_inset Formula ${\displaystyle \frac{\,}{v=\mathrm{get}(u,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto H\left(E\left(u\right)\right)_{F}\right],H}}$ \end_inset @@ -124,24 +214,19 @@ \end_inset - - \begin_inset Text \begin_layout Plain Layout +\begin_inset space ~ +\end_inset -\emph on -set -\end_layout +\begin_inset space ~ \end_inset - - -\begin_inset Text -\begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(v)\neq\mathrm{null}}{\mathrm{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H\left[E\left(v\right)\mapsto\left(H\left(E\left(v\right)\right)!_{F}E(u)\right)\right]}}$ + +\begin_inset space ~ \end_inset @@ -149,8 +234,6 @@ \end_inset - - \begin_inset Text @@ -164,7 +247,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(v)=\mathrm{null}}{\mathrm{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$ +\begin_inset Formula ${\displaystyle \frac{\mathrm{type}(H(E(v))\neq T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$ \end_inset @@ -173,14 +256,14 @@ \end_inset - + \begin_inset Text \begin_layout Plain Layout \emph on -guard +set \end_layout \end_inset @@ -189,7 +272,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(v)\neq\mathrm{null},\,\mathrm{type}(H(E(v))=T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H}}$ +\begin_inset Formula ${\displaystyle \frac{\,}{\mathrm{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H\left[E\left(v\right)\mapsto\left(H\left(E\left(v\right)\right)!_{F}E(u)\right)\right]}}$ \end_inset @@ -197,12 +280,21 @@ \end_inset - - \begin_inset Text \begin_layout Plain Layout +\begin_inset space ~ +\end_inset + + +\begin_inset space ~ +\end_inset + + +\begin_inset space ~ +\end_inset + \end_layout @@ -212,9 +304,15 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(v)=\mathrm{null}\vee\mathrm{type}(H(E(v))\neq T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$ + +\end_layout + \end_inset + + +\begin_inset Text +\begin_layout Plain Layout \end_layout From cfbolz at codespeak.net Wed Oct 6 18:48:31 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 6 Oct 2010 18:48:31 +0200 (CEST) Subject: [pypy-svn] r77661 - pypy/extradoc/talk/pepm2011 Message-ID: <20101006164831.813E2282B9E@codespeak.net> Author: cfbolz Date: Wed Oct 6 18:48:30 2010 New Revision: 77661 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: first go at integrating the operational semantics. not completely happy. Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 6 18:48:30 2010 @@ -438,7 +438,7 @@ \begin{figure} \begin{center} -\includegraphics[scale=0.7]{figures/obj-lifetime.pdf} +\includegraphics[scale=0.6]{figures/obj-lifetime.pdf} \end{center} \caption{Object Lifetimes in a Trace} @@ -612,7 +612,124 @@ \section{Formal Description of the Algorithm} \label{sec:Algorithm} -XXX want some sort of pseudo-code + +\begin{figure*} +\begin{center} +\begin{tabular}{lcccc} +\emph{new} & ${\displaystyle \frac{l\,\mathrm{fresh}}{v=\mathrm{new}(T),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto l\right],H\left[l\mapsto\left(T,\mathrm{null},\mathrm{null}\right)\right]}}$ & ~~~ & +\emph{guard} & ${\displaystyle \frac{\mathrm{type}(H(E(v))=T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H}}$\tabularnewline[3em] +\emph{get} & ${\displaystyle \frac{\,}{v=\mathrm{get}(u,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto H\left(E\left(u\right)\right)_{F}\right],H}}$ & ~~~ & +& ${\displaystyle \frac{\mathrm{type}(H(E(v))\neq T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$\tabularnewline[3em] +\emph{set} & ${\displaystyle \frac{\,}{\mathrm{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H\left[E\left(v\right)\mapsto\left(H\left(E\left(v\right)\right)!_{F}E(u)\right)\right]}}$ & ~~~ & +& \tabularnewline[4em] +\end{tabular} + +\begin{minipage}[b]{7 cm} +\emph{Object Domains:} +$$\setlength\arraycolsep{0.1em} + \begin{array}{rcll} + u,v,w & \in & V & \mathrm{\ variables\ in\ trace}\\ + T & \in & \mathfrak{T} & \mathrm{\ runtime\ types}\\ + F & \in & \left\{ L,R\right\} & \mathrm{\ fields\ of\ objects}\\ + l & \in & L & \mathrm{\ locations\ on\ heap} + \end{array} +$$ +\end{minipage} +\begin{minipage}[b]{5 cm} +\emph{Semantic Values:} +$$\setlength\arraycolsep{0.1em} + \begin{array}{rcll} + E & \in & V\rightharpoonup L & \mathrm{\ Environment}\\ + H & \in & L\rightharpoonup\mathfrak{T}\times(L\cup \{ \mathrm{null} \})\times (L\cup \{ \mathrm{null}\} ) & \mathrm{\ Heap}\\ + \\ + \\ + \end{array} +$$ +\end{minipage} +\end{center} +\label{fig:semantics} +\caption{The Operational Semantics of Simplified Traces} +\end{figure*} + +In this section we want to give a formal description of the semantics of the +traces and of the optimizer and liken the optimization to partial evaluation. +We concentrate on the operations for manipulating dynamically allocated objects, +as those are the only ones that are actually optimized. Without loss of +generality we also consider only objects with two fields in this section. + +Traces are lists of operations. The operations considered here are new (to make +a new object), get (to read a field out of an object), set (to write a field +into an object) and guard (to check the type of an object). The values of all +variables are locations (i.e.~pointers). Locations are mapped to objects, which +are represented by triples of a type $T$, and two locations that represent the +fields of the object. When a new object is created, the fields are initialized +to null, but we require that they are immediately initialized to a real +location, otherwise the trace is malformed. + +We use some abbreviations when dealing with object triples. To read the type of +an object, $\mathrm{type}((T,l_1,l_2))=T$ is used. Reading a field $F$ from an +object is written $(T,l_1,l_2)_F$ which either returns $l_1$ if $F=L$ or $l_2$ +if $F=R$. To set field $F$ to a new location $l$, we use the notation +$(T,l_1,l_2)!_Fl$, which yields a new triple $(T,l,l_2)$ if $F=L$ or a new +triple $(T,l_1,l)$ if $F=R$. + +Figure~\ref{fig:semantics} shows the operational semantics for traces. The +interpreter formalized there executes one operation at a time. Its state is +represented by an environment and a heap, which are potentially changed by the +execution of an operation. The environment is a partial function from variables +to locations and the heap is a partial function from locations to objects. Note +that a variable can never be null in the environment, otherwise the trace would +be malformed. We use some syntax for changing partial functions: $E[v\mapsto l]$ +denotes the environment which is just like $E$, but maps $v$ to $l$. + +The new operation creates a new object $(T,\mathrm{null},\mathrm{null})$, on the +heap under a fresh location $l$ and adds the result variable to the environment, +mapping to the new location $l$. + +The get operation reads a field $F$ out of an object and adds the result +variable to the environment, mapping to the read location. The heap is +unchanged. + +The set operation changes field $F$ of an object stored at the location that +variable $v$ maps to. The new value of the field is the location in variable +$u$. The environment is unchanged. + +The guard operation is used to check whether the object stored at the location +that variable $v$ maps to is of type $T$. If that is the case, then execution +continues without changing heap and environment. Otherwise, execution is +stopped. + +\subsection{Optimizing Traces} +\label{sub:formalopt} + + + +% subsection Optimizing Traces (end) + + + +\begin{figure*} +\begin{center} +\fbox{ +\begin{tabular}{lc} +\emph{new} & ${\displaystyle \frac{v^{*}\,\mathrm{fresh}}{v=\mathrm{new}(T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto v^{*}\right],S\left[v^{*}\mapsto\left(T,\mathrm{null,null}\right)\right]}}$\tabularnewline[3em] +\emph{get} & ${\displaystyle \frac{E(u)\in\mathrm{dom}(S)}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto S(E(u))_{F}\right],S}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(u)\notin\mathrm{dom}(S)\, v^{*}\,\mathrm{fresh}}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle v^{*}=\mathrm{get}(E(u),F)\right\rangle ,E\left[v\mapsto v^{*}\right],S}}$\tabularnewline[3em] +\emph{set} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S\left[E\left(v\right)\mapsto\left(S(E(v))!_{F}E(u)\right)\right]}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(v)\notin\mathrm{dom}\left(S\right),\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathrm{set}\left(E(v),F,E(u)\right)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] +\emph{guard} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S}}$\tabularnewline[3em] +\emph{lifting} & ${\displaystyle \frac{v^{*}\notin\mathrm{dom}(S)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle \,\right\rangle ,S}}$\tabularnewline[3em] + & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{liftfields}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathrm{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em] + & ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathrm{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathrm{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline +\end{tabular} +} +\end{center} +\label{fig:optimization} +\caption{Optimization Rules} +\end{figure*} + % section Formal Description of the Algorithm (end) From cfbolz at codespeak.net Wed Oct 6 21:11:57 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 6 Oct 2010 21:11:57 +0200 (CEST) Subject: [pypy-svn] r77662 - pypy/extradoc/talk/pepm2011 Message-ID: <20101006191157.DD3EF282B90@codespeak.net> Author: cfbolz Date: Wed Oct 6 21:11:27 2010 New Revision: 77662 Modified: pypy/extradoc/talk/pepm2011/math.lyx Log: simplification of the guard case Modified: pypy/extradoc/talk/pepm2011/math.lyx ============================================================================== --- pypy/extradoc/talk/pepm2011/math.lyx (original) +++ pypy/extradoc/talk/pepm2011/math.lyx Wed Oct 6 21:11:27 2010 @@ -504,7 +504,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$ +\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$ \end_inset @@ -527,9 +527,6 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S}}$ -\end_inset - \end_layout From cfbolz at codespeak.net Wed Oct 6 21:13:50 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 6 Oct 2010 21:13:50 +0200 (CEST) Subject: [pypy-svn] r77663 - pypy/extradoc/talk/pepm2011 Message-ID: <20101006191350.783D8282B90@codespeak.net> Author: cfbolz Date: Wed Oct 6 21:13:29 2010 New Revision: 77663 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: some beautiful and rectangular but not very yellow text about the formal description of the optimization Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 6 21:13:29 2010 @@ -651,6 +651,9 @@ \caption{The Operational Semantics of Simplified Traces} \end{figure*} +XXX think of a way to format the operations to make them look distinct from +normal text + In this section we want to give a formal description of the semantics of the traces and of the optimizer and liken the optimization to partial evaluation. We concentrate on the operations for manipulating dynamically allocated objects, @@ -679,8 +682,13 @@ execution of an operation. The environment is a partial function from variables to locations and the heap is a partial function from locations to objects. Note that a variable can never be null in the environment, otherwise the trace would -be malformed. We use some syntax for changing partial functions: $E[v\mapsto l]$ -denotes the environment which is just like $E$, but maps $v$ to $l$. +be malformed. The environment could not directly map variables to object, +because several variables can contain a pointer to the \emph{same} object. Thus +the "indirection" is needed to express sharing. + +We use the following notation for updating partial functions: +$E[v\mapsto l]$ denotes the environment which is just like $E$, but maps $v$ to +$l$. The new operation creates a new object $(T,\mathrm{null},\mathrm{null})$, on the heap under a fresh location $l$ and adds the result variable to the environment, @@ -702,6 +710,61 @@ \subsection{Optimizing Traces} \label{sub:formalopt} +To optimize the simple traces from the last section, we use online partial +evaluation. The partial evaluator optimizes one operation of the trace at a +time. Every operation in the unoptimized trace is replaced by a list of +operations in the optimized trace. This list is empty if the operation +can be optimized away (which hopefully happens often). + +The state of the optimizer is stored in an environment $E$ and a \emph{static +heap} $S$. The environment is a partial function from variables in the +unoptimized trace to variables in the optimized trace (which are written with a +$\ ^*$ for clarity). The reason for introducing new variables in the optimized +trace is that several variables that appear in the unoptimized trace can turn +into the same variables in the optimized trace. Thus the environment of the +optimizer serves a function similar to that of the environment in the semantics. + +The static heap is a partial function from $V^*$ into the +set of static objects, which are triples of a type, and two elements of $V^*$. +An variable $v^*$ is in the domain of the static heap $S$ as long as the +optimizer can fully keep track of the object. The image of $v^*$ is what is +statically known about the object stored in it, \ie its type and its fields. The +fields of objects in the static heap are also elements of $V^*$ (or null, for +short periods of time). + +When the optimizer sees a new operation, it optimistically removes it and +assumes that the resulting object can stay static. The optimization for all +further operations is split into two cases. One case is for when the +involved variables are in the static heap, which means that the operation can be +removed. The other case is that nothing is known about the variables, which +means the operation has to be residualized. + +If the argument $u$ of a get operation is mapped to something in the static +heap, the get can be performed at optimization time. Otherwise, the get +operation needs to be emitted. + +If the first argument $v$ to a set operation is mapped to something in the +static heap, then the set can performed at optimization time and the static heap +is updated. Otherwise the set operation needs to be emitted. This needs to be +done carefully, because the new value for the field stored in the variable $u$ +could itself be static, in which case it needs to be lifted first. + +I a guard is performed on a variable that is in the static heap, the type check +can be performed at optimization time, which means the operation can be removed +if the types match. If the type check fails statically or if the object is not +in the static heap, the guard is put into the residual trace. This also needs to +lift the variable on which the guard is performed. + +Lifting takes a variable that is potentially in the static heap and makes sure +that it is turned into a dynamic variable. This means that operations are +emitted that construct an object that looks like the shape described in the +static heap, and the variable is removed from the static heap. + +Lifting a static object needs to recursively lift its fields. Some care needs to +be taken when lifting a static object, because the structures described by the +static heap can be cyclic. To make sure that the same static object is not lifted +twice, the liftfield operation removes it from the static heap \emph{before} +recursively lifting its fields. % subsection Optimizing Traces (end) @@ -710,7 +773,6 @@ \begin{figure*} \begin{center} -\fbox{ \begin{tabular}{lc} \emph{new} & ${\displaystyle \frac{v^{*}\,\mathrm{fresh}}{v=\mathrm{new}(T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto v^{*}\right],S\left[v^{*}\mapsto\left(T,\mathrm{null,null}\right)\right]}}$\tabularnewline[3em] \emph{get} & ${\displaystyle \frac{E(u)\in\mathrm{dom}(S)}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto S(E(u))_{F}\right],S}}$\tabularnewline[3em] @@ -722,9 +784,33 @@ & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S}}$\tabularnewline[3em] \emph{lifting} & ${\displaystyle \frac{v^{*}\notin\mathrm{dom}(S)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle \,\right\rangle ,S}}$\tabularnewline[3em] & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{liftfields}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathrm{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em] - & ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathrm{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathrm{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline + & ${\displaystyle + \frac{\left(S\left(v^{*}\right)_{L},S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle + \mathrm{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathrm{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline[3em] \end{tabular} -} + +\begin{minipage}[b]{7 cm} +\emph{Object Domains:} +$$\setlength\arraycolsep{0.1em} + \begin{array}{rcll} + u,v,w & \in & V & \mathrm{\ variables\ in\ trace}\\ + u^*,v^*,w^* & \in & V^* & \mathrm{\ variables\ in\ optimized\ trace}\\ + T & \in & \mathfrak{T} & \mathrm{\ runtime\ types}\\ + F & \in & \left\{ L,R\right\} & \mathrm{\ fields\ of\ objects}\\ + \end{array} +$$ +\end{minipage} +\begin{minipage}[b]{5 cm} +\emph{Semantic Values:} +$$\setlength\arraycolsep{0.1em} + \begin{array}{rcll} + E & \in & V\rightharpoonup V^* & \mathrm{\ Environment}\\ + S & \in & V^*\rightharpoonup\mathfrak{T}\times(V^*\cup \{ \mathrm{null} \})\times (V^*\cup \{ \mathrm{null}\} ) & \mathrm{\ Heap}\\ + \\ + \\ + \end{array} +$$ +\end{minipage} \end{center} \label{fig:optimization} \caption{Optimization Rules} From cfbolz at codespeak.net Wed Oct 6 21:18:36 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 6 Oct 2010 21:18:36 +0200 (CEST) Subject: [pypy-svn] r77664 - pypy/extradoc/talk/pepm2011 Message-ID: <20101006191836.4F581282B90@codespeak.net> Author: cfbolz Date: Wed Oct 6 21:18:34 2010 New Revision: 77664 Modified: pypy/extradoc/talk/pepm2011/math.lyx Log: tweaks Modified: pypy/extradoc/talk/pepm2011/math.lyx ============================================================================== --- pypy/extradoc/talk/pepm2011/math.lyx (original) +++ pypy/extradoc/talk/pepm2011/math.lyx Wed Oct 6 21:18:34 2010 @@ -340,7 +340,7 @@ \begin_layout Standard \begin_inset Tabular - + @@ -519,26 +519,6 @@ \begin_layout Plain Layout -\end_layout - -\end_inset - - -\begin_inset Text - -\begin_layout Plain Layout - -\end_layout - -\end_inset - - - - -\begin_inset Text - -\begin_layout Plain Layout - \emph on lifting \end_layout @@ -581,7 +561,7 @@ \end_inset - + \begin_inset Text From cfbolz at codespeak.net Wed Oct 6 21:19:04 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 6 Oct 2010 21:19:04 +0200 (CEST) Subject: [pypy-svn] r77665 - pypy/extradoc/talk/pepm2011 Message-ID: <20101006191904.0F6B1282B90@codespeak.net> Author: cfbolz Date: Wed Oct 6 21:19:03 2010 New Revision: 77665 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: re-copy formulas from lyx file Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 6 21:19:03 2010 @@ -780,13 +780,10 @@ \emph{set} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S\left[E\left(v\right)\mapsto\left(S(E(v))!_{F}E(u)\right)\right]}}$\tabularnewline[3em] & ${\displaystyle \frac{E(v)\notin\mathrm{dom}\left(S\right),\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathrm{set}\left(E(v),F,E(u)\right)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] \emph{guard} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$\tabularnewline[3em] - & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] - & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] \emph{lifting} & ${\displaystyle \frac{v^{*}\notin\mathrm{dom}(S)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle \,\right\rangle ,S}}$\tabularnewline[3em] - & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{liftfields}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathrm{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em] - & ${\displaystyle - \frac{\left(S\left(v^{*}\right)_{L},S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle - \mathrm{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathrm{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline[3em] + & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathrm{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em] + & ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathrm{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathrm{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline[3em] \end{tabular} \begin{minipage}[b]{7 cm} From cfbolz at codespeak.net Wed Oct 6 21:24:52 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 6 Oct 2010 21:24:52 +0200 (CEST) Subject: [pypy-svn] r77666 - pypy/extradoc/talk/pepm2011 Message-ID: <20101006192452.BA170282B90@codespeak.net> Author: cfbolz Date: Wed Oct 6 21:24:51 2010 New Revision: 77666 Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf pypy/extradoc/talk/pepm2011/paper.tex Log: shuffle stuff around Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf ============================================================================== Binary files. No diff available. Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 6 21:24:51 2010 @@ -500,7 +500,7 @@ the code is thus to analyze which objects fall in category 1 and may thus not be allocated at all. -XXX is "symbolic execution" the right word to drop? +XXX rewrite to acknowledge the existence of the formal stuff This process is called \emph{escape analysis}. The escape analysis of our tracing JIT works by using \emph{virtual objects}: The trace is walked from @@ -710,6 +710,50 @@ \subsection{Optimizing Traces} \label{sub:formalopt} + + +\begin{figure*} +\begin{center} +\begin{tabular}{lc} +\emph{new} & ${\displaystyle \frac{v^{*}\,\mathrm{fresh}}{v=\mathrm{new}(T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto v^{*}\right],S\left[v^{*}\mapsto\left(T,\mathrm{null,null}\right)\right]}}$\tabularnewline[3em] +\emph{get} & ${\displaystyle \frac{E(u)\in\mathrm{dom}(S)}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto S(E(u))_{F}\right],S}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(u)\notin\mathrm{dom}(S)\, v^{*}\,\mathrm{fresh}}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle v^{*}=\mathrm{get}(E(u),F)\right\rangle ,E\left[v\mapsto v^{*}\right],S}}$\tabularnewline[3em] +\emph{set} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S\left[E\left(v\right)\mapsto\left(S(E(v))!_{F}E(u)\right)\right]}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(v)\notin\mathrm{dom}\left(S\right),\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathrm{set}\left(E(v),F,E(u)\right)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] +\emph{guard} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] +\emph{lifting} & ${\displaystyle \frac{v^{*}\notin\mathrm{dom}(S)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle \,\right\rangle ,S}}$\tabularnewline[3em] + & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathrm{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em] + & ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathrm{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathrm{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline[3em] +\end{tabular} + +\begin{minipage}[b]{7 cm} +\emph{Object Domains:} +$$\setlength\arraycolsep{0.1em} + \begin{array}{rcll} + u,v,w & \in & V & \mathrm{\ variables\ in\ trace}\\ + u^*,v^*,w^* & \in & V^* & \mathrm{\ variables\ in\ optimized\ trace}\\ + T & \in & \mathfrak{T} & \mathrm{\ runtime\ types}\\ + F & \in & \left\{ L,R\right\} & \mathrm{\ fields\ of\ objects}\\ + \end{array} +$$ +\end{minipage} +\begin{minipage}[b]{5 cm} +\emph{Semantic Values:} +$$\setlength\arraycolsep{0.1em} + \begin{array}{rcll} + E & \in & V\rightharpoonup V^* & \mathrm{\ Environment}\\ + S & \in & V^*\rightharpoonup\mathfrak{T}\times(V^*\cup \{ \mathrm{null} \})\times (V^*\cup \{ \mathrm{null}\} ) & \mathrm{\ Heap}\\ + \\ + \\ + \end{array} +$$ +\end{minipage} +\end{center} +\label{fig:optimization} +\caption{Optimization Rules} +\end{figure*} + To optimize the simple traces from the last section, we use online partial evaluation. The partial evaluator optimizes one operation of the trace at a time. Every operation in the unoptimized trace is replaced by a list of @@ -769,49 +813,12 @@ % subsection Optimizing Traces (end) +\subsection{Analysis of the Algorithm} +\label{sub:analysis} +XXX algorithm is linear in the length of the trace -\begin{figure*} -\begin{center} -\begin{tabular}{lc} -\emph{new} & ${\displaystyle \frac{v^{*}\,\mathrm{fresh}}{v=\mathrm{new}(T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto v^{*}\right],S\left[v^{*}\mapsto\left(T,\mathrm{null,null}\right)\right]}}$\tabularnewline[3em] -\emph{get} & ${\displaystyle \frac{E(u)\in\mathrm{dom}(S)}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto S(E(u))_{F}\right],S}}$\tabularnewline[3em] - & ${\displaystyle \frac{E(u)\notin\mathrm{dom}(S)\, v^{*}\,\mathrm{fresh}}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle v^{*}=\mathrm{get}(E(u),F)\right\rangle ,E\left[v\mapsto v^{*}\right],S}}$\tabularnewline[3em] -\emph{set} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S\left[E\left(v\right)\mapsto\left(S(E(v))!_{F}E(u)\right)\right]}}$\tabularnewline[3em] - & ${\displaystyle \frac{E(v)\notin\mathrm{dom}\left(S\right),\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathrm{set}\left(E(v),F,E(u)\right)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] -\emph{guard} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$\tabularnewline[3em] - & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] -\emph{lifting} & ${\displaystyle \frac{v^{*}\notin\mathrm{dom}(S)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle \,\right\rangle ,S}}$\tabularnewline[3em] - & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathrm{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em] - & ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathrm{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathrm{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline[3em] -\end{tabular} - -\begin{minipage}[b]{7 cm} -\emph{Object Domains:} -$$\setlength\arraycolsep{0.1em} - \begin{array}{rcll} - u,v,w & \in & V & \mathrm{\ variables\ in\ trace}\\ - u^*,v^*,w^* & \in & V^* & \mathrm{\ variables\ in\ optimized\ trace}\\ - T & \in & \mathfrak{T} & \mathrm{\ runtime\ types}\\ - F & \in & \left\{ L,R\right\} & \mathrm{\ fields\ of\ objects}\\ - \end{array} -$$ -\end{minipage} -\begin{minipage}[b]{5 cm} -\emph{Semantic Values:} -$$\setlength\arraycolsep{0.1em} - \begin{array}{rcll} - E & \in & V\rightharpoonup V^* & \mathrm{\ Environment}\\ - S & \in & V^*\rightharpoonup\mathfrak{T}\times(V^*\cup \{ \mathrm{null} \})\times (V^*\cup \{ \mathrm{null}\} ) & \mathrm{\ Heap}\\ - \\ - \\ - \end{array} -$$ -\end{minipage} -\end{center} -\label{fig:optimization} -\caption{Optimization Rules} -\end{figure*} +% subsection Analysis of the Algorithm (end) % section Formal Description of the Algorithm (end) From cfbolz at codespeak.net Wed Oct 6 21:31:11 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 6 Oct 2010 21:31:11 +0200 (CEST) Subject: [pypy-svn] r77667 - pypy/extradoc/talk/pepm2011 Message-ID: <20101006193111.4707E282B90@codespeak.net> Author: cfbolz Date: Wed Oct 6 21:31:10 2010 New Revision: 77667 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: mention similarity to execution semantics Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 6 21:31:10 2010 @@ -780,8 +780,9 @@ assumes that the resulting object can stay static. The optimization for all further operations is split into two cases. One case is for when the involved variables are in the static heap, which means that the operation can be -removed. The other case is that nothing is known about the variables, which -means the operation has to be residualized. +performed at optimization time and removed from the trace. These rules mirror +the execution semantics closely. The other case is that nothing is known about +the variables, which means the operation has to be residualized. If the argument $u$ of a get operation is mapped to something in the static heap, the get can be performed at optimization time. Otherwise, the get From afa at codespeak.net Wed Oct 6 23:59:24 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Wed, 6 Oct 2010 23:59:24 +0200 (CEST) Subject: [pypy-svn] r77670 - in pypy/branch/fast-forward: lib-python pypy/module/__builtin__ pypy/module/__builtin__/test Message-ID: <20101006215924.1342736C22F@codespeak.net> Author: afa Date: Wed Oct 6 23:59:18 2010 New Revision: 77670 Modified: pypy/branch/fast-forward/lib-python/TODO pypy/branch/fast-forward/pypy/module/__builtin__/compiling.py pypy/branch/fast-forward/pypy/module/__builtin__/test/test_builtin.py Log: Fix eval('a', None, dict(a=42)) Modified: pypy/branch/fast-forward/lib-python/TODO ============================================================================== --- pypy/branch/fast-forward/lib-python/TODO (original) +++ pypy/branch/fast-forward/lib-python/TODO Wed Oct 6 23:59:18 2010 @@ -12,10 +12,6 @@ - Missing builtin: bytearray (possibly reuse module.__pypy__.bytebuffer) -- Seen in test_inspect, this has never worked in pypy:: - - assert eval('a', None, dict(a=42)) == 42 - - Missing complex.__trunc__ - Mark some tests as "implementation specific":: Modified: pypy/branch/fast-forward/pypy/module/__builtin__/compiling.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/__builtin__/compiling.py (original) +++ pypy/branch/fast-forward/pypy/module/__builtin__/compiling.py Wed Oct 6 23:59:18 2010 @@ -65,7 +65,7 @@ compile.unwrap_spec = [ObjSpace,W_Root,str,str,int,int] -def eval(space, w_code, w_globals=NoneNotWrapped, w_locals=NoneNotWrapped): +def eval(space, w_code, w_globals=None, w_locals=None): """Evaluate the source in the context of globals and locals. The source may be a string representing a Python expression or a code object as returned by compile(). The globals and locals @@ -87,13 +87,16 @@ w('eval() arg 1 must be a string or code object')) caller = space.getexecutioncontext().gettopframe_nohidden() - if w_globals is None or space.is_w(w_globals, space.w_None): + if space.is_w(w_globals, space.w_None): if caller is None: - w_globals = w_locals = space.newdict() + w_globals = space.newdict() + if space.is_w(w_locals, space.w_None): + w_locals = w_globals else: w_globals = caller.w_globals - w_locals = caller.getdictscope() - elif w_locals is None: + if space.is_w(w_locals, space.w_None): + w_locals = caller.getdictscope() + elif space.is_w(w_locals, space.w_None): w_locals = w_globals try: Modified: pypy/branch/fast-forward/pypy/module/__builtin__/test/test_builtin.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/__builtin__/test/test_builtin.py (original) +++ pypy/branch/fast-forward/pypy/module/__builtin__/test/test_builtin.py Wed Oct 6 23:59:18 2010 @@ -432,6 +432,7 @@ assert eval("3", None, None) == 3 i = 4 assert eval("i", None, None) == 4 + assert eval('a', None, dict(a=42)) == 42 def test_compile(self): co = compile('1+2', '?', 'eval') From trundle at codespeak.net Thu Oct 7 00:29:44 2010 From: trundle at codespeak.net (trundle at codespeak.net) Date: Thu, 7 Oct 2010 00:29:44 +0200 (CEST) Subject: [pypy-svn] r77671 - pypy/branch/fast-forward/lib-python Message-ID: <20101006222944.7232F282B90@codespeak.net> Author: trundle Date: Thu Oct 7 00:29:42 2010 New Revision: 77671 Modified: pypy/branch/fast-forward/lib-python/TODO Log: Remove TODO entry about bytes = str (already done in r75073). Modified: pypy/branch/fast-forward/lib-python/TODO ============================================================================== --- pypy/branch/fast-forward/lib-python/TODO (original) +++ pypy/branch/fast-forward/lib-python/TODO Thu Oct 7 00:29:42 2010 @@ -8,8 +8,6 @@ Probably easy tasks ------------------- -- Missing builtin: bytes = str - - Missing builtin: bytearray (possibly reuse module.__pypy__.bytebuffer) - Missing complex.__trunc__ From david at codespeak.net Thu Oct 7 10:42:00 2010 From: david at codespeak.net (david at codespeak.net) Date: Thu, 7 Oct 2010 10:42:00 +0200 (CEST) Subject: [pypy-svn] r77672 - in pypy/branch/arm-backend/pypy/jit/backend/arm: . test Message-ID: <20101007084200.788F1282B90@codespeak.net> Author: david Date: Thu Oct 7 10:41:57 2010 New Revision: 77672 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Log: Support negative offsets in LDR instruction Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Thu Oct 7 10:41:57 2010 @@ -32,7 +32,7 @@ def gen_func_epilog(self): self.mc.write32(0xe50b3010) # str r3, [fp, #-16] - self.mc.write32(0xe51b3010) # ldr r3, [fp, #-16] + self.mc.LDR_ri(r.r3, r.fp, -16) #self.mc.write32(0xe1a00003) # mov r0, r3 self.mc.SUB_ri(r.sp, r.fp, 12) self.mc.write32(0xe89da800) # ldm sp, {fp, sp, pc} Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py Thu Oct 7 10:41:57 2010 @@ -9,10 +9,19 @@ self._pos = 0 def LDR_ri(self, rt, rn, imm=0, cond=cond.AL): - # XXX U and P bits are not encoded yet - self.write32(cond << 28 - | 5 << 24 - | 9 << 20 + # XXX W and P bits are not encoded yet + p = 1 + u = 1 + w = 0 + if imm < 0: + u = 0 + imm = -imm + self.write32(cond << 28 + | 0x1 << 26 + | (p & 0x1) << 24 + | (u & 0x1) << 23 + | (w & 0x1) << 21 + | 0x1 << 20 | (rn & 0xF) << 16 | (rt & 0xF) << 12 | (imm & 0xFFF)) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Thu Oct 7 10:41:57 2010 @@ -20,6 +20,10 @@ self.cb.LDR_ri(r.r0, r.r1) self.assert_equal('LDR r0, [r1]') + def test_ldr_neg(self): + self.cb.LDR_ri(r.r3, r.fp, -16) + self.assert_equal('LDR r3, [fp, #-16]') + def test_add_ri(self): self.cb.ADD_ri(r.r0, r.r1, 1) self.assert_equal('ADD r0, r1, #1') From antocuni at codespeak.net Thu Oct 7 11:44:54 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 7 Oct 2010 11:44:54 +0200 (CEST) Subject: [pypy-svn] r77673 - in pypy/branch/jitffi/pypy/module: _ffi _ffi/test _rawffi Message-ID: <20101007094454.1960F282B90@codespeak.net> Author: antocuni Date: Thu Oct 7 11:44:52 2010 New Revision: 77673 Added: pypy/branch/jitffi/pypy/module/_ffi/ pypy/branch/jitffi/pypy/module/_ffi/__init__.py (contents, props changed) pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py (contents, props changed) pypy/branch/jitffi/pypy/module/_ffi/test/ pypy/branch/jitffi/pypy/module/_ffi/test/__init__.py (contents, props changed) pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py (contents, props changed) Modified: pypy/branch/jitffi/pypy/module/_rawffi/__init__.py Log: start to write an applevel module to expose rlib.libffi Added: pypy/branch/jitffi/pypy/module/_ffi/__init__.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/module/_ffi/__init__.py Thu Oct 7 11:44:52 2010 @@ -0,0 +1,10 @@ +from pypy.interpreter.mixedmodule import MixedModule + +class Module(MixedModule): + + interpleveldefs = { + 'CDLL' : 'interp_ffi.W_CDLL', +# 'FuncPtr' : 'interp_ffi.W_FuncPtr', + } + + appleveldefs = {} Added: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Thu Oct 7 11:44:52 2010 @@ -0,0 +1,29 @@ +import sys +from pypy.interpreter.baseobjspace import W_Root, ObjSpace, Wrappable, Arguments +from pypy.interpreter.error import OperationError, wrap_oserror, operationerrfmt +from pypy.interpreter.gateway import interp2app, NoneNotWrapped +from pypy.interpreter.typedef import TypeDef, GetSetProperty + +from pypy.rlib.libffi import CDLL +from pypy.rlib.rdynload import DLOpenError + +class W_CDLL(Wrappable): + def __init__(self, space, name): + try: + self.cdll = CDLL(name) + except DLOpenError, e: + raise operationerrfmt(space.w_OSError, '%s: %s', name, + e.msg or 'unspecified error') + self.name = name + self.space = space + + + +def descr_new_cdll(space, w_type, name): + return space.wrap(W_CDLL(space, name)) +descr_new_cdll.unwrap_spec = [ObjSpace, W_Root, str] + +W_CDLL.typedef = TypeDef( + 'CDLL', + __new__ = interp2app(descr_new_cdll), + ) Added: pypy/branch/jitffi/pypy/module/_ffi/test/__init__.py ============================================================================== Added: pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Thu Oct 7 11:44:52 2010 @@ -0,0 +1,49 @@ +from pypy.conftest import gettestobjspace +from pypy.translator.platform import platform +from pypy.translator.tool.cbuild import ExternalCompilationInfo +from pypy.module._rawffi.interp_rawffi import TYPEMAP +from pypy.module._rawffi.tracker import Tracker +from pypy.translator.platform import platform + +import os, sys, py + +class AppTestFfi: + + @classmethod + def prepare_c_example(cls): + from pypy.tool.udir import udir + from pypy.translator.tool.cbuild import ExternalCompilationInfo + from pypy.translator.platform import platform + + c_file = udir.ensure("test__ffi", dir=1).join("foolib.c") + # automatically collect the C source from the docstrings of the tests + snippets = [] + for name in dir(cls): + if name.startswith('test_'): + meth = getattr(cls, name) + # the heuristic to determine it it's really C code could be + # improved: so far we just check that there is a '{' :-) + if meth.__doc__ is not None and '{' in meth.__doc__: + snippets.append(meth.__doc__) + # + c_file.write(py.code.Source('\n'.join(snippets))) + eci = ExternalCompilationInfo(export_symbols=[]) + return str(platform.compile([c_file], eci, 'x', standalone=False)) + + + def setup_class(cls): + from pypy.rlib.libffi import get_libc_name + from pypy.rlib.test.test_libffi import get_libm_name + space = gettestobjspace(usemodules=('_ffi',)) + cls.space = space + cls.w_libfoo_name = space.wrap(cls.prepare_c_example()) + cls.w_libc_name = space.wrap(get_libc_name()) + cls.w_libc_name = space.wrap(get_libm_name(sys.platform)) + + def test_libload(self): + import _ffi + _ffi.CDLL(self.libc_name) + + def test_libload_fail(self): + import _ffi + raises(OSError, _ffi.CDLL, "xxxxx_this_name_does_not_exist_xxxxx") Modified: pypy/branch/jitffi/pypy/module/_rawffi/__init__.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_rawffi/__init__.py (original) +++ pypy/branch/jitffi/pypy/module/_rawffi/__init__.py Thu Oct 7 11:44:52 2010 @@ -1,5 +1,5 @@ -""" Low-level interface to libffi +""" Low-level interface to clibffi """ from pypy.interpreter.mixedmodule import MixedModule From arigo at codespeak.net Thu Oct 7 11:53:01 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 7 Oct 2010 11:53:01 +0200 (CEST) Subject: [pypy-svn] r77674 - pypy/extradoc/sprintinfo/ddorf2010 Message-ID: <20101007095301.F0F1E282B90@codespeak.net> Author: arigo Date: Thu Oct 7 11:53:00 2010 New Revision: 77674 Modified: pypy/extradoc/sprintinfo/ddorf2010/announce.txt Log: Fix typo for the future announcements. Modified: pypy/extradoc/sprintinfo/ddorf2010/announce.txt ============================================================================== --- pypy/extradoc/sprintinfo/ddorf2010/announce.txt (original) +++ pypy/extradoc/sprintinfo/ddorf2010/announce.txt Thu Oct 7 11:53:00 2010 @@ -25,7 +25,7 @@ If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. -More organisational information will be send to that list. We'll keep a +More organisational information will be sent to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). From arigo at codespeak.net Thu Oct 7 12:15:15 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 7 Oct 2010 12:15:15 +0200 (CEST) Subject: [pypy-svn] r77675 - in pypy/branch/32ptr-on-64bit/pypy: rlib rpython rpython/lltypesystem rpython/memory rpython/memory/gc rpython/memory/gc/test rpython/memory/gctransform rpython/memory/test rpython/test translator/c translator/c/src Message-ID: <20101007101515.469D2282B90@codespeak.net> Author: arigo Date: Thu Oct 7 12:15:12 2010 New Revision: 77675 Modified: pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/generation.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/inspector.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/markcompact.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/marksweep.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/semispace.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/test/test_minimark.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rdict.py pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h Log: Progress. Modified: pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py Thu Oct 7 12:15:12 2010 @@ -117,6 +117,8 @@ c_mremap, _ = external('mremap', [PTR, size_t, size_t, rffi.ULONG], PTR) + c_mmap_safe._annenforceargs_ = (None, int, int, int, int, int) + # this one is always safe _, _get_page_size = external('getpagesize', [], rffi.INT) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py Thu Oct 7 12:15:12 2010 @@ -143,6 +143,21 @@ frame.find_roots(roots) return roots + def debug_check(self): + for frame in self.frame_stack: + vars = [] + for v in frame.curr_block.inputargs: + if isinstance(v, Variable): + vars.append(v) + for op in frame.curr_block.operations[:frame.curr_operation_index]: + vars.append(op.result) + for v in vars: + TYPE = getattr(v, 'concretetype', None) + if TYPE == llmemory.HiddenGcRef32: + p = frame.getval(v) + if p: + p._obj.container._check() + def find_exception(self, exc): assert isinstance(exc, LLException) klass, inst = exc.args[0], exc.args[1] @@ -730,6 +745,7 @@ result = self.heap.malloc(obj, zero=zero, flavor='raw') self.alloca_objects.append(result) return result + self.llinterpreter.debug_check() ptr = self.heap.malloc(obj, zero=zero, flavor=flavor) if flavor == 'raw' and self.llinterpreter.malloc_check: self.llinterpreter.remember_malloc(ptr, self) @@ -882,11 +898,15 @@ raise NotImplementedError("gc_pop_alive_pyobj") def op_gc_reload_possibly_moved(self, v_newaddr, v_ptr): - assert v_newaddr.concretetype is llmemory.Address - assert isinstance(v_ptr.concretetype, lltype.Ptr) - assert v_ptr.concretetype.TO._gckind == 'gc' newaddr = self.getval(v_newaddr) - p = llmemory.cast_adr_to_ptr(newaddr, v_ptr.concretetype) + if v_newaddr.concretetype != llmemory.HiddenGcRef32: + assert v_newaddr.concretetype is llmemory.Address + assert isinstance(v_ptr.concretetype, lltype.Ptr) + assert v_ptr.concretetype.TO._gckind == 'gc' + p = llmemory.cast_adr_to_ptr(newaddr, v_ptr.concretetype) + else: + assert v_ptr.concretetype == llmemory.HiddenGcRef32 + p = newaddr if isinstance(v_ptr, Constant): assert v_ptr.value == p else: @@ -1421,6 +1441,12 @@ raise IndexError("address of local vars only support [0] indexing") p = self.frame.getval(self.v) result = llmemory.cast_ptr_to_adr(p) + if result: + from pypy.rpython.lltypesystem import llarena + try: + result = llarena.getfakearenaaddress(result) + except RuntimeError: + pass # the GC should never see instances of _gctransformed_wref result = self.unwrap_possible_weakref(result) return result Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py Thu Oct 7 12:15:12 2010 @@ -84,26 +84,44 @@ return self.ll_fasthash_function def get_ll_dummyval_obj(self, rtyper, s_value): - return DummyVal() + return DummyValueBuilder(rtyper) -class DummyVal(object): - TYPE = llmemory.HiddenGcRef32.TO - ll_dummy_value = lltype.opaqueptr(TYPE, "dummy_value", dummy_value=True) +class DummyValueBuilder(object): + TYPE = llmemory.HiddenGcRef32 + def __init__(self, rtyper): + self.rtyper = rtyper def _freeze_(self): return True + def __hash__(self): + return 42 + def __eq__(self, other): + return (isinstance(other, DummyValueBuilder) and + self.rtyper is other.rtyper) + def __ne__(self, other): + return not (self == other) + + def build_ll_dummy_value(self): + TYPE = self.TYPE + try: + return self.rtyper.cache_dummy_values[TYPE] + except KeyError: + # generate a dummy ptr to an immortal placeholder struct + p = lltype.malloc(lltype.GcStruct('dummy'), immortal=True) + p = llop.hide_into_ptr32(TYPE, p) + self.rtyper.cache_dummy_values[TYPE] = p + return p + ll_dummy_value = property(build_ll_dummy_value) class __extend__(pairtype(Repr, CompressedGcRefRepr)): def convert_from_to((r_from, r_to), v, llops): - assert r_from.lowleveltype.TO._gckind == 'gc' - assert not isinstance(r_from.lowleveltype.TO, lltype.GcOpaqueType) - return llops.genop('hide_into_ptr32', [v], + v2 = llops.convertvar(v, r_from, r_to.baserepr) + return llops.genop('hide_into_ptr32', [v2], resulttype=llmemory.HiddenGcRef32) class __extend__(pairtype(CompressedGcRefRepr, Repr)): def convert_from_to((r_from, r_to), v, llops): - assert r_to.lowleveltype.TO._gckind == 'gc' - assert not isinstance(r_to.lowleveltype.TO, lltype.GcOpaqueType) - return llops.genop('show_from_ptr32', [v], - resulttype=r_to.lowleveltype) + v2 = llops.genop('show_from_ptr32', [v], + resulttype=r_from.BASETYPE) + return llops.convertvar(v2, r_from.baserepr, r_to) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py Thu Oct 7 12:15:12 2010 @@ -3,8 +3,9 @@ from pypy.objspace.flow.model import Constant from pypy.rpython.rdict import AbstractDictRepr, AbstractDictIteratorRepr,\ rtype_newdict -from pypy.rpython.lltypesystem import lltype +from pypy.rpython.lltypesystem import lltype, llmemory from pypy.rlib.rarithmetic import r_uint, intmask +from pypy.rpython.lltypesystem.lloperation import llop from pypy.rlib.objectmodel import hlinvoke from pypy.rpython import robject from pypy.rlib import objectmodel @@ -197,7 +198,6 @@ def convert_const(self, dictobj): - from pypy.rpython.lltypesystem import llmemory # get object from bound dict methods #dictobj = getattr(dictobj, '__self__', dictobj) if dictobj is None: @@ -769,6 +769,10 @@ def recast(P, v): if isinstance(P, lltype.Ptr): + if P == llmemory.HiddenGcRef32: + if lltype.typeOf(v) == llmemory.HiddenGcRef32: + return v + return llop.hide_into_ptr32(P, v) return lltype.cast_pointer(P, v) else: return v Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/base.py Thu Oct 7 12:15:12 2010 @@ -52,7 +52,7 @@ # collection. It is automatically set to True by test_gc.py. The # checking logic is translatable, so the flag can be set to True # here before translation. - DEBUG = True + DEBUG = False def set_query_functions(self, is_varsize, has_gcptr_in_varsize, is_gcarrayofgcptr, @@ -169,7 +169,7 @@ def x_clone(self, clonedata): raise RuntimeError("no support for x_clone in the GC") - def trace(self, obj, callback, arg): + def do_trace(self, obj, callback, arg): """Enumerate the locations inside the given obj that can contain GC pointers. For each such GC pointer, callback(object, arg) is called, where 'object' is the address of the stored object. @@ -208,7 +208,7 @@ j += 1 item += itemlength length -= 1 - trace._annspecialcase_ = 'specialize:arg(2)' + do_trace._annspecialcase_ = 'specialize:arg(2)' def _trace_see(self, obj, ofs, callback, arg): if self.config.compressptr and llmemory.has_odd_value_marker(ofs): @@ -233,7 +233,7 @@ _trace_see._always_inline_ = True def trace_partial(self, obj, start, stop, callback, arg): - """Like trace(), but only walk the array part, for indices in + """Like do_trace(), but only walk the array part, for indices in range(start, stop). Must only be called if has_gcptr_in_varsize(). """ length = stop - start @@ -291,7 +291,7 @@ pending = self._debug_pending while pending.non_empty(): obj = pending.pop() - self.trace(obj, self._debug_callback2, None) + self.do_trace(obj, self._debug_callback2, None) self._debug_seen.delete() self._debug_pending.delete() @@ -306,7 +306,7 @@ ll_assert(bool(obj), "NULL address from walk_roots()") self._debug_record(obj) def _debug_callback2(self, obj, ignored): - ll_assert(bool(obj), "NULL address from self.trace()") + ll_assert(bool(obj), "NULL address from self.do_trace()") self._debug_record(obj) def debug_check_object(self, obj): @@ -397,6 +397,7 @@ else: self.id_free_list.append(id) +# ____________________________________________________________ def choose_gc_from_config(config): """Return a (GCClass, GC_PARAMS) from the given config object. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/generation.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/generation.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/generation.py Thu Oct 7 12:15:12 2010 @@ -328,14 +328,13 @@ self.header(obj).tid |= GCFLAG_NO_HEAP_PTRS # ^^^ the flag we just added will be removed immediately if # the object still contains pointers to younger objects - self.trace(obj, self._trace_external_obj, obj) + self.do_trace(obj, self._trace_external_obj, obj) stack.delete() - def _trace_external_obj(self, pointer, obj): - addr = pointer.address[0] + def _trace_external_obj(self, addr, obj): newaddr = self.copy(addr) - pointer.address[0] = newaddr self.write_into_last_generation_obj(obj, newaddr) + return newaddr # ____________________________________________________________ # Implementation of nursery-only collections @@ -432,11 +431,12 @@ """obj must not be in the nursery. This copies all the young objects it references out of the nursery. """ - self.trace(obj, self._trace_drag_out, None) + self.do_trace(obj, self._trace_drag_out, None) - def _trace_drag_out(self, pointer, ignored): - if self.is_in_nursery(pointer.address[0]): - pointer.address[0] = self.copy(pointer.address[0]) + def _trace_drag_out(self, obj, ignored): + if self.is_in_nursery(obj): + obj = self.copy(obj) + return obj # The code relies on the fact that no weakref can be an old object # weakly pointing to a young object. Indeed, weakrefs are immutable @@ -581,7 +581,7 @@ SemiSpaceGC._track_heap_root) def _track_heap_ext(self, adr, ignored): - self.trace(adr, self.track_heap_parent, adr) + self.do_trace(adr, self.track_heap_parent, adr) def debug_check_object(self, obj): """Check the invariants about 'obj' that should be true @@ -591,23 +591,22 @@ if tid & GCFLAG_NO_YOUNG_PTRS: ll_assert(not self.is_in_nursery(obj), "nursery object with GCFLAG_NO_YOUNG_PTRS") - self.trace(obj, self._debug_no_nursery_pointer, None) + self.do_trace(obj, self._debug_no_nursery_pointer, None) elif not self.is_in_nursery(obj): ll_assert(self._d_oopty.contains(obj), "missing from old_objects_pointing_to_young") if tid & GCFLAG_NO_HEAP_PTRS: ll_assert(self.is_last_generation(obj), "GCFLAG_NO_HEAP_PTRS on non-3rd-generation object") - self.trace(obj, self._debug_no_gen1or2_pointer, None) + self.do_trace(obj, self._debug_no_gen1or2_pointer, None) elif self.is_last_generation(obj): ll_assert(self._d_lgro.contains(obj), "missing from last_generation_root_objects") - def _debug_no_nursery_pointer(self, root, ignored): - ll_assert(not self.is_in_nursery(root.address[0]), + def _debug_no_nursery_pointer(self, obj, ignored): + ll_assert(not self.is_in_nursery(obj), "GCFLAG_NO_YOUNG_PTRS but found a young pointer") - def _debug_no_gen1or2_pointer(self, root, ignored): - target = root.address[0] + def _debug_no_gen1or2_pointer(self, target, ignored): ll_assert(not target or self.is_last_generation(target), "GCFLAG_NO_HEAP_PTRS but found a pointer to gen1or2") Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/inspector.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/inspector.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/inspector.py Thu Oct 7 12:15:12 2010 @@ -55,15 +55,15 @@ # ---------- implementation of pypy.rlib.rgc.get_rpy_referents() ---------- -def _count_rpy_referent(pointer, gc): +def _count_rpy_referent(address, gc): gc._count_rpy += 1 def _do_count_rpy_referents(gc, gcref): gc._count_rpy = 0 - gc.trace(llmemory.cast_ptr_to_adr(gcref), _count_rpy_referent, gc) + gc.do_trace(llmemory.cast_ptr_to_adr(gcref), _count_rpy_referent, gc) return gc._count_rpy -def _append_rpy_referent(pointer, gc): +def _append_rpy_referent(address, gc): # Can use the gc list, but should not allocate! # It is essential that the list is not resizable! lst = gc._list_rpy @@ -71,13 +71,12 @@ if index >= len(lst): raise ValueError gc._count_rpy = index + 1 - lst[index] = llmemory.cast_adr_to_ptr(pointer.address[0], - llmemory.GCREF) + lst[index] = llmemory.cast_adr_to_ptr(address, llmemory.GCREF) def _do_append_rpy_referents(gc, gcref, lst): gc._count_rpy = 0 gc._list_rpy = lst - gc.trace(llmemory.cast_ptr_to_adr(gcref), _append_rpy_referent, gc) + gc.do_trace(llmemory.cast_ptr_to_adr(gcref), _append_rpy_referent, gc) def get_rpy_referents(gc, gcref): count = _do_count_rpy_referents(gc, gcref) @@ -158,11 +157,10 @@ self.write(llmemory.cast_adr_to_int(obj)) self.write(gc.get_member_index(typeid)) self.write(gc.get_size_incl_hash(obj)) - gc.trace(obj, self._writeref, None) + gc.do_trace(obj, self._writeref, None) self.write(-1) - def _writeref(self, pointer, _): - obj = pointer.address[0] + def _writeref(self, obj, _): self.write(llmemory.cast_adr_to_int(obj)) self.add(obj) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/markcompact.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/markcompact.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/markcompact.py Thu Oct 7 12:15:12 2010 @@ -372,10 +372,10 @@ def _trace_and_mark(self): while self.to_see.non_empty(): obj = self.to_see.popleft() - self.trace(obj, self._mark_obj, None) + self.do_trace(obj, self._mark_obj, None) - def _mark_obj(self, pointer, ignored): - self.mark(pointer.address[0]) + def _mark_obj(self, obj, ignored): + self.mark(obj) def _mark_root(self, root): self.mark(root.address[0]) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/marksweep.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/marksweep.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/marksweep.py Thu Oct 7 12:15:12 2010 @@ -495,10 +495,9 @@ return hdr.typeid16 def add_reachable_to_stack(self, obj, objects): - self.trace(obj, self._add_reachable, objects) + self.do_trace(obj, self._add_reachable, objects) - def _add_reachable(pointer, objects): - obj = pointer.address[0] + def _add_reachable(obj, objects): objects.append(obj) _add_reachable = staticmethod(_add_reachable) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py Thu Oct 7 12:15:12 2010 @@ -1028,7 +1028,7 @@ """obj must not be in the nursery. This copies all the young objects it references out of the nursery. """ - self.trace(obj, self._trace_drag_out, None) + self.do_trace(obj, self._trace_drag_out, None) def trace_and_drag_out_of_nursery_partial(self, obj, start, stop): """Like trace_and_drag_out_of_nursery(), but limited to the array @@ -1322,7 +1322,7 @@ # # Trace the content of the object and put all objects it references # into the 'objects_to_trace' list. - self.trace(obj, self._collect_ref_rec, None) + self.do_trace(obj, self._collect_ref_rec, None) # ---------- @@ -1411,7 +1411,7 @@ state = self._finalization_state(y) if state == 0: self._bump_finalization_state_from_0_to_1(y) - self.trace(y, self._append_if_nonnull, pending) + self.do_trace(y, self._append_if_nonnull, pending) elif state == 2: self._recursively_bump_finalization_state_from_2_to_3(y) self._recursively_bump_finalization_state_from_1_to_2(x) @@ -1469,7 +1469,7 @@ hdr = self.header(y) if hdr.tid & GCFLAG_FINALIZATION_ORDERING: # state 2 ? hdr.tid &= ~GCFLAG_FINALIZATION_ORDERING # change to state 3 - self.trace(y, self._append_if_nonnull, pending) + self.do_trace(y, self._append_if_nonnull, pending) def _recursively_bump_finalization_state_from_1_to_2(self, obj): # recursively convert objects from state 1 to state 2. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/semispace.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/semispace.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/semispace.py Thu Oct 7 12:15:12 2010 @@ -397,10 +397,10 @@ return self._make_a_copy_with_tid(obj, objsize, tid) def trace_and_copy(self, obj): - self.trace(obj, self._trace_copy, None) + self.do_trace(obj, self._trace_copy, None) - def _trace_copy(self, pointer, ignored): - pointer.address[0] = self.copy(pointer.address[0]) + def _trace_copy(self, obj, ignored): + return self.copy(obj) def surviving(self, obj): # To use during a collection. Check if the object is currently @@ -495,7 +495,7 @@ state = self._finalization_state(y) if state == 0: self._bump_finalization_state_from_0_to_1(y) - self.trace(y, self._append_if_nonnull, pending) + self.do_trace(y, self._append_if_nonnull, pending) elif state == 2: self._recursively_bump_finalization_state_from_2_to_3(y) scan = self._recursively_bump_finalization_state_from_1_to_2( @@ -522,8 +522,8 @@ self.objects_with_finalizers = new_with_finalizer return scan - def _append_if_nonnull(pointer, stack): - stack.append(pointer.address[0]) + def _append_if_nonnull(obj, stack): + stack.append(obj) _append_if_nonnull = staticmethod(_append_if_nonnull) def _finalization_state(self, obj): @@ -559,7 +559,7 @@ hdr = self.header(y) if hdr.tid & GCFLAG_FINALIZATION_ORDERING: # state 2 ? hdr.tid &= ~GCFLAG_FINALIZATION_ORDERING # change to state 3 - self.trace(y, self._append_if_nonnull, pending) + self.do_trace(y, self._append_if_nonnull, pending) def _recursively_bump_finalization_state_from_1_to_2(self, obj, scan): # recursively convert objects from state 1 to state 2. @@ -671,8 +671,7 @@ objsize = self.get_size(obj) return self._get_object_hash(obj, objsize, hdr.tid) - def track_heap_parent(self, obj, parent): - addr = obj.address[0] + def track_heap_parent(self, addr, parent): parent_idx = llop.get_member_index(lltype.Signed, self.get_type_id(parent)) idx = llop.get_member_index(lltype.Signed, self.get_type_id(addr)) @@ -687,7 +686,7 @@ self._ll_typeid_map[idx].count += 1 totsize = self.get_size(adr) + self.size_gc_header() self._ll_typeid_map[idx].size += llmemory.raw_malloc_usage(totsize) - self.trace(adr, self.track_heap_parent, adr) + self.do_trace(adr, self.track_heap_parent, adr) def _track_heap_root(self, root): self.track_heap(root.address[0]) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/test/test_minimark.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/test/test_minimark.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/test/test_minimark.py Thu Oct 7 12:15:12 2010 @@ -6,7 +6,7 @@ def test_card_marking_words_for_length(): - gc = MiniMarkGC(None, card_page_indices=128) + gc = MiniMarkGC(None, card_page_indices=128, translated_to_c=False) assert gc.card_page_shift == 7 P = 128 * LONG_BIT assert gc.card_marking_words_for_length(1) == 1 @@ -18,7 +18,7 @@ assert gc.card_marking_words_for_length(P+P+P+P+P+P+P+P+1) == 9 def test_card_marking_bytes_for_length(): - gc = MiniMarkGC(None, card_page_indices=128) + gc = MiniMarkGC(None, card_page_indices=128, translated_to_c=False) assert gc.card_page_shift == 7 P = 128 * 8 assert gc.card_marking_bytes_for_length(1) == 1 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctransform/framework.py Thu Oct 7 12:15:12 2010 @@ -557,6 +557,13 @@ r_gcdata = self.translator.rtyper.getrepr(s_gcdata) ll_instance = rmodel.inputconst(r_gcdata, self.gcdata).value + if (self.layoutbuilder.addresses_of_static_ptrs and + self.translator.config.translation.compressptr): + raise Exception("not supported: compressptr together with a " + "non-empty list of addresses_of_static_ptrs " + "(as you would get with some GCs or with " + "DEBUG=True in rpython/memory/gc/base.py)") + addresses_of_static_ptrs = ( self.layoutbuilder.addresses_of_static_ptrs_in_nongc + self.layoutbuilder.addresses_of_static_ptrs) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gctypelayout.py Thu Oct 7 12:15:12 2010 @@ -177,7 +177,7 @@ if (isinstance(ARRAY.OF, lltype.Ptr) and ARRAY.OF.TO._gckind == 'gc'): if ARRAY.OF == llmemory.HiddenGcRef32: - pass # infobits |= T_IS_GCARRAY_OF_HIDDENGCREF32 + infobits |= T_IS_GCARRAY_OF_HIDDENGCREF32 else: infobits |= T_IS_GCARRAY_OF_GCPTR varinfo.ofstolength = llmemory.ArrayLengthOffset(ARRAY) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py Thu Oct 7 12:15:12 2010 @@ -4,7 +4,7 @@ from pypy.rpython.memory import gcwrapper from pypy.rpython.memory.test import snippet from pypy.rpython.test.test_llinterp import get_interpreter -from pypy.rpython.lltypesystem import lltype +from pypy.rpython.lltypesystem import lltype, llmemory from pypy.rpython.lltypesystem.rstr import STR from pypy.rpython.lltypesystem.lloperation import llop from pypy.rlib.objectmodel import we_are_translated @@ -146,7 +146,29 @@ res = self.interpret(concat, [100]) assert res == concat(100) #assert simulator.current_size - curr < 16000 * INT_SIZE / 4 - + + def test_double_entry_in_stack_list(self): + LOOPS = 500 + S = lltype.GcStruct("S", ('x', lltype.Signed)) + def g(s): + t = lltype.malloc(S) + s = llop.show_from_ptr32(lltype.Ptr(S), s) + t.x = s.x + 1 + return llop.hide_into_ptr32(llmemory.HiddenGcRef32, t) + def f(s): + return g(s) + def malloc_a_lot(): + s = lltype.malloc(S) + s.x = 0 + s = llop.hide_into_ptr32(llmemory.HiddenGcRef32, s) + i = 0 + while i < LOOPS: + s = f(s) + i += 1 + return llop.show_from_ptr32(lltype.Ptr(S), s).x + res = self.interpret(malloc_a_lot, []) + assert res == LOOPS + def test_finalizer(self): class B(object): pass Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py Thu Oct 7 12:15:12 2010 @@ -25,4 +25,34 @@ class TestLLtype64(MixinCompressed64, test_rclass.TestLLtype): - pass + + def test_casts_1(self): + class A: + pass + class B(A): + pass + def dummyfn(n): + if n > 5: + # this tuple is allocated as a (*, Void) tuple, and immediately + # converted into a generic (*, *) tuple. + x = (B(), None) + else: + x = (A(), A()) + return x[0] + res = self.interpret(dummyfn, [8]) + assert self.is_of_instance_type(res) + + def test_dict_recast(self): + from pypy.rlib.objectmodel import r_dict + class A(object): + pass + def myeq(n, m): + return n == m + def myhash(a): + return 42 + def fn(): + d = r_dict(myeq, myhash) + d[4] = A() + a = d.values()[0] + a.x = 5 + self.interpret(fn, []) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rdict.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rdict.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rdict.py Thu Oct 7 12:15:12 2010 @@ -578,6 +578,20 @@ res = self.interpret(fn, [3, 3]) assert res == 123 + def test_recast(self): + class A(object): + pass + def myeq(n, m): + return n == m + def myhash(a): + return 42 + def fn(): + d = r_dict(myeq, myhash) + d[4] = A() + a = d.values()[0] + a.x = 5 + self.interpret(fn, []) + class TestLLtype(BaseTestRdict, LLRtypeMixin): def test_dict_but_not_with_char_keys(self): Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/node.py Thu Oct 7 12:15:12 2010 @@ -789,8 +789,7 @@ db.late_initializations.append((access_expr, db.get(value))) expr = '0.0 /* patched later by %sinfinity */' % ( '-+'[value > 0]) - elif (TYPE == llmemory.HiddenGcRef32 - and value and not hasattr(value._obj, 'dummy_value')): + elif TYPE == llmemory.HiddenGcRef32 and value: # non-NULL HiddenGcRef32 realobj = value._obj.container realvalue = cast_opaque_ptr(Ptr(typeOf(realobj)), value) name = db.get(realvalue) Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/primitive.py Thu Oct 7 12:15:12 2010 @@ -150,15 +150,14 @@ def name_hiddengcref32(value, db): if not value: return '0' # NULL - elif hasattr(value._obj, 'dummy_value'): - return '1 /* dummy value */' - # The only prebuilt HiddenGcRef32 that should occur in a translated C - # program occur as fields or items of a GcStruct or GcArray. They are - # handled by late_initializations_hiddengcref32. + # The only prebuilt HiddenGcRef32 that occur in a translated C program + # mostly occur as fields or items of a GcStruct or GcArray. They are + # handled by late_initializations_hiddengcref32. This is for special + # cases, where it's (probably) ok to write an expression that is not + # considered as a constant by the C compiler. realobj = value._obj.container realvalue = cast_opaque_ptr(Ptr(typeOf(realobj)), value) - db.get(realvalue) # force following dependencies - return 'HIDE_INTO_PTR32(???) /* see primitive.py, name_hiddengcref32() */' + return 'HIDE_INTO_PTR32(%s)' % db.get(realvalue) def name_small_integer(value, db): """Works for integers of size at most INT or UINT.""" Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/src/hiddengcref32.h Thu Oct 7 12:15:12 2010 @@ -17,13 +17,17 @@ #endif -#define OP_SHOW_FROM_PTR32(x, r) r = (void*)(((unsigned long)(x)) << 3) +#define SHOW_FROM_PTR32(x) ((void*)(((unsigned long)(x)) << 3)) +#define HIDE_INTO_PTR32(x) ((hiddengcref32_t)(((unsigned long)(x)) >> 3)) + + +#define OP_SHOW_FROM_PTR32(x, r) r = SHOW_FROM_PTR32(x) #define OP_HIDE_INTO_PTR32_CHECK(x, r) \ - r = (hiddengcref32_t)(((unsigned long)(x)) >> 3); \ + r = HIDE_INTO_PTR32(x); \ if ((void*)(((unsigned long)(r)) << 3) != (x)) \ RPyPointerTooBig() #define OP_HIDE_INTO_PTR32(x, r) \ RPyAssert(!(((long)(x)) & ~0x7FFFFFFF8), "Pointer too big or misaligned"); \ - r = (hiddengcref32_t)(((unsigned long)(x)) >> 3) + r = HIDE_INTO_PTR32(x) From arigo at codespeak.net Thu Oct 7 12:16:04 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 7 Oct 2010 12:16:04 +0200 (CEST) Subject: [pypy-svn] r77676 - pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc Message-ID: <20101007101604.EB049282B90@codespeak.net> Author: arigo Date: Thu Oct 7 12:16:03 2010 New Revision: 77676 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py Log: Add a comment. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py Thu Oct 7 12:16:03 2010 @@ -1,3 +1,7 @@ +""" +This is a variant of minimarkpage.py, for the case of a 64-bit translation +with --compressptr. xxx it should not be a whole copy +""" from pypy.rpython.lltypesystem import lltype, llmemory, llarena, rffi from pypy.rlib.rarithmetic import LONG_BIT, r_uint from pypy.rlib.objectmodel import we_are_translated From arigo at codespeak.net Thu Oct 7 13:46:55 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Thu, 7 Oct 2010 13:46:55 +0200 (CEST) Subject: [pypy-svn] r77677 - pypy/branch/32ptr-on-64bit/pypy/rlib Message-ID: <20101007114655.699A3282B90@codespeak.net> Author: arigo Date: Thu Oct 7 13:46:52 2010 New Revision: 77677 Modified: pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py Log: Add an _annenforcearg_. Modified: pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py Thu Oct 7 13:46:52 2010 @@ -109,7 +109,7 @@ if _POSIX: has_mremap = cConfig['has_mremap'] - c_mmap, c_mmap_safe = external('mmap', [PTR, size_t, rffi.INT, rffi.INT, + c_mmap, _c_mmap_safe = external('mmap', [PTR, size_t, rffi.INT, rffi.INT, rffi.INT, off_t], PTR) c_munmap, c_munmap_safe = external('munmap', [PTR, size_t], rffi.INT) c_msync, _ = external('msync', [PTR, size_t, rffi.INT], rffi.INT) @@ -117,7 +117,9 @@ c_mremap, _ = external('mremap', [PTR, size_t, size_t, rffi.ULONG], PTR) - c_mmap_safe._annenforceargs_ = (None, int, int, int, int, int) + def c_mmap_safe(addr, length, prot, flags, fd, offset): + return _c_mmap_safe(addr, length, prot, flags, fd, offset) + c_mmap_safe._annenforceargs_ = (PTR, int, int, int, int, int) # this one is always safe _, _get_page_size = external('getpagesize', [], rffi.INT) From cfbolz at codespeak.net Thu Oct 7 14:49:49 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 14:49:49 +0200 (CEST) Subject: [pypy-svn] r77678 - pypy/extradoc/talk/pepm2011 Message-ID: <20101007124949.EC60E282B90@codespeak.net> Author: cfbolz Date: Thu Oct 7 14:49:46 2010 New Revision: 77678 Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf pypy/extradoc/talk/pepm2011/paper.tex Log: rewrite the introduction to mention partial evaluation more Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf ============================================================================== Binary files. No diff available. Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Thu Oct 7 14:49:46 2010 @@ -108,7 +108,7 @@ dispatching. Those are problems that are usually not present or at least less severe in statically typed languages. -Boxing of primitive types means that dynamic languages need to be able to handle +Boxing of primitive types is necessary because dynamic languages need to be able to handle all objects, even integers, floats, booleans etc. in the same way as user-defined instances. Thus those primitive types are usually \emph{boxed}, i.e. a small heap-structure is allocated for them, that contains the actual value. Boxing @@ -126,31 +126,34 @@ need it. A recently popular approach to implementing just-in-time compilers for dynamic -languages is that of a tracing JIT. A tracing JIT often takes the form of an -extension to an existing interpreter, which can be sped up that way. This -approach is also the one taken by the PyPy project, which is an environment for -implementing dynamic programming languages. PyPy's -approach to doing so is to straightforwardly implement an interpreter for the -to-be-implemented language, and then use powerful tools to turn the interpreter -into an efficient virtual machine (VM) that also contains a just-in-time compiler. -This compiler -is automatically generated from the interpreter using partial-evaluation-like -techniques \cite{bolz_tracing_2009}. The PyPy project and its approach to -tracing JIT compilers is described in Section~\ref{sec:Background}. - -The tracing JIT approach that the PyPy project is taking removes the overhead -of bytecode dispatch. In this paper we want to explain how the traces that are -produced by PyPy's tracing JIT can be optimized to also remove some of the +languages is that of a tracing JIT. A tracing JIT works by observing the running +program and recording linear execution traces, which are then turned into +machine code. One reason for the popularity of tracing JITs is their relative +simplicity. They can often be added to an interpreter and a lot of the +infrastructure of an interpreter can be reused. They give some important +optimizations like inlining and constant-folding for free. A tracing JIT always +produces linear pieces of code, which makes many optimizations that are usually +hard in a compiler simpler, such as register allocation. + +The usage of a tracing JIT can remove the overhead of bytecode dispatch and that +of the interpreter data structures. In this paper we want to present an approach +how an optimization can be added to a tracing JIT that also removes some of the overhead more closely associated to dynamic languages, such as boxing overhead -and type dispatching. To understand the problem more closely, we analyze the -occurring object lifetimes in Section~\ref{sec:lifetimes}. The most important -technique to achieve this is a form of escape analysis \cite{XXX} that we call -\emph{virtual objects}\footnote{The terminology comes from \cite{rigo_representation-based_2004}}, -which is described in Section~\ref{sec:virtuals}. The -goal of virtual objects is to remove allocations of temporary objects that have -a predictable lifetime and to optimize type dispatching in the process. +and type dispatching. Our experimental platform is the PyPy project, which is an +environment for implementing dynamic programming languages. PyPy and tracing +JITs are described in more detail in Section~\ref{sec:Background}. +Section~\ref{sec:lifetimes} analyzes the problem to be solved more closely. + +The most important technique we use to achieve to optimize traces is a +straightforward application of partial evaluation. The partial evaluation +performs a form of escape analysis \cite{XXX} on the traces and make some +objects that are allocated in the trace \emph{static}\footnote{These objects are +called \emph{virtual} in Psyco \cite{rigo_representation-based_2004}.} which +means that they do not occur any more in the optimized trace. This technique is +informally described in Section~\ref{sec:virtuals}, a more formal description is +given in Section~\ref{sec:formal}. -The basic approach of virtual objects can then be extended to also be used for +The basic approach of static objects can then be extended to also be used for type-specializing the traces that are produced by the tracing JIT (Section~\ref{sec:crossloop}). In Section~\ref{sec:support} we describe some supporting techniques that are not central to the approach, but are needed to @@ -161,9 +164,11 @@ \begin{enumerate} \item An efficient and effective algorithm for removing object allocations in a tracing JIT. - \item XXX + \item A characterization of this algorithm as partial evaluation. + \item A rigorous evaluation of this algorithm. \end{enumerate} + \section{Background} \label{sec:Background} @@ -610,7 +615,7 @@ and only three \texttt{guard\_class} operations, from the original seven. \section{Formal Description of the Algorithm} -\label{sec:Algorithm} +\label{sec:formal} \begin{figure*} From antocuni at codespeak.net Thu Oct 7 14:54:34 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 7 Oct 2010 14:54:34 +0200 (CEST) Subject: [pypy-svn] r77679 - in pypy/branch/jitffi/pypy: module/_ffi module/_ffi/test rlib Message-ID: <20101007125434.9877B282B90@codespeak.net> Author: antocuni Date: Thu Oct 7 14:54:32 2010 New Revision: 77679 Modified: pypy/branch/jitffi/pypy/module/_ffi/__init__.py pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: create an applevel wrapper for each ffi primitive type Modified: pypy/branch/jitffi/pypy/module/_ffi/__init__.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/__init__.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/__init__.py Thu Oct 7 14:54:32 2010 @@ -1,10 +1,12 @@ from pypy.interpreter.mixedmodule import MixedModule +from pypy.module._ffi import interp_ffi class Module(MixedModule): interpleveldefs = { 'CDLL' : 'interp_ffi.W_CDLL', # 'FuncPtr' : 'interp_ffi.W_FuncPtr', + 'types': 'interp_ffi.W_types', } appleveldefs = {} Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Thu Oct 7 14:54:32 2010 @@ -4,9 +4,42 @@ from pypy.interpreter.gateway import interp2app, NoneNotWrapped from pypy.interpreter.typedef import TypeDef, GetSetProperty -from pypy.rlib.libffi import CDLL +from pypy.rlib.libffi import CDLL, types from pypy.rlib.rdynload import DLOpenError +class W_FFIType(Wrappable): + def __init__(self, name, ffitype): + self.name = name + self.ffitype = ffitype + + def str(self, space): + return space.wrap('' % self.name) + str.unwrap_spec = ['self', ObjSpace] + + +W_FFIType.typedef = TypeDef( + 'FFIType', + __str__ = interp2app(W_FFIType.str), + ) + + +class W_types(Wrappable): + pass + +def build_ffi_types(): + tdict = {} + for key, value in types.__dict__.iteritems(): + if key.startswith('__'): + continue + tdict[key] = W_FFIType(key, value) + return tdict + +W_types.typedef = TypeDef( + 'types', + **build_ffi_types()) + +# ======================================================================== + class W_CDLL(Wrappable): def __init__(self, space, name): try: @@ -18,7 +51,6 @@ self.space = space - def descr_new_cdll(space, w_type, name): return space.wrap(W_CDLL(space, name)) descr_new_cdll.unwrap_spec = [ObjSpace, W_Root, str] @@ -27,3 +59,5 @@ 'CDLL', __new__ = interp2app(descr_new_cdll), ) + +# ======================================================================== Modified: pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Thu Oct 7 14:54:32 2010 @@ -47,3 +47,9 @@ def test_libload_fail(self): import _ffi raises(OSError, _ffi.CDLL, "xxxxx_this_name_does_not_exist_xxxxx") + + def test_simple_types(self): + from _ffi import types + assert str(types.sint) == '' + assert str(types.uint) == '' + Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Thu Oct 7 14:54:32 2010 @@ -30,6 +30,7 @@ setattr(cls, name, value) cls.slong = clibffi.cast_type_to_ffitype(rffi.LONG) cls.ulong = clibffi.cast_type_to_ffitype(rffi.ULONG) + del cls._import types._import() From david at codespeak.net Thu Oct 7 15:00:22 2010 From: david at codespeak.net (david at codespeak.net) Date: Thu, 7 Oct 2010 15:00:22 +0200 (CEST) Subject: [pypy-svn] r77680 - in pypy/branch/arm-backend/pypy/jit/backend/arm: . test Message-ID: <20101007130022.F1EFA282B90@codespeak.net> Author: david Date: Thu Oct 7 15:00:21 2010 New Revision: 77680 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Log: Support negative offsets in STR Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Thu Oct 7 15:00:21 2010 @@ -31,7 +31,7 @@ self.gen_func_epilog() def gen_func_epilog(self): - self.mc.write32(0xe50b3010) # str r3, [fp, #-16] + self.mc.STR_ri(r.r3, r.fp, -16) self.mc.LDR_ri(r.r3, r.fp, -16) #self.mc.write32(0xe1a00003) # mov r0, r3 self.mc.SUB_ri(r.sp, r.fp, 12) @@ -42,7 +42,7 @@ self.mc.PUSH([r.fp, r.ip, r.lr, r.pc]) self.mc.SUB_ri(r.fp, r.ip, 4) self.mc.SUB_ri(r.sp, r.sp, 8) - self.mc.write32(0xe50b0014) # str r0, [fp, #-20] + self.mc.STR_ri(r.r0, r.fp, -20) def gen_load_int(self, reg, value): self.mc.MOV_ri(reg, (value & 0xFF)) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py Thu Oct 7 15:00:21 2010 @@ -8,14 +8,18 @@ self._data = alloc(1024) self._pos = 0 - def LDR_ri(self, rt, rn, imm=0, cond=cond.AL): - # XXX W and P bits are not encoded yet - p = 1 + def _encode_imm(self, imm): u = 1 - w = 0 if imm < 0: u = 0 imm = -imm + return u, imm + + def LDR_ri(self, rt, rn, imm=0, cond=cond.AL): + # XXX W and P bits are not encoded yet + p = 1 + w = 0 + u, imm = self._encode_imm(imm) self.write32(cond << 28 | 0x1 << 26 | (p & 0x1) << 24 @@ -62,9 +66,15 @@ | (imm & 0xFFF)) def STR_ri(self, rt, rn, imm=0, cond=cond.AL): + # XXX W and P bits are not encoded yet + p = 1 + w = 0 + u, imm = self._encode_imm(imm) self.write32(cond << 28 - | 0x5 << 24 - | 0x8 << 20 + | 0x1 << 26 + | (p & 0x1) << 24 + | (u & 0x1) << 23 + | (w & 0x1) << 21 | (rn & 0xF) << 16 | (rt & 0xF) << 12 | (imm & 0xFFF)) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Thu Oct 7 15:00:21 2010 @@ -49,6 +49,14 @@ self.cb.STR_ri(r.r9, r.r14) self.assert_equal('STR r9, [r14]') + def test_str_ri_offset(self): + self.cb.STR_ri(r.r9, r.r14, 23) + self.assert_equal('STR r9, [r14, #23]') + + def test_str_ri_offset(self): + self.cb.STR_ri(r.r9, r.r14, -20) + self.assert_equal('STR r9, [r14, #-20]') + def test_asr_ri(self): self.cb.ASR_ri(r.r7, r.r5, 24) self.assert_equal('ASR r7, r5, #24') From cfbolz at codespeak.net Thu Oct 7 16:19:52 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 16:19:52 +0200 (CEST) Subject: [pypy-svn] r77681 - pypy/extradoc/talk/pepm2011 Message-ID: <20101007141952.B7A88282B90@codespeak.net> Author: cfbolz Date: Thu Oct 7 16:19:51 2010 New Revision: 77681 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: rewrite to speak of "static objects" everywhere Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Thu Oct 7 16:19:51 2010 @@ -150,7 +150,7 @@ objects that are allocated in the trace \emph{static}\footnote{These objects are called \emph{virtual} in Psyco \cite{rigo_representation-based_2004}.} which means that they do not occur any more in the optimized trace. This technique is -informally described in Section~\ref{sec:virtuals}, a more formal description is +informally described in Section~\ref{sec:statics}, a more formal description is given in Section~\ref{sec:formal}. The basic approach of static objects can then be extended to also be used for @@ -429,8 +429,8 @@ XXX simplify traces a bit more -In the next section, we will see how this can be improved upon, using escape -analysis. XXX +In the next section, we will see how this can be improved upon, using partial +evaluation. XXX \section{Object Lifetimes in a Tracing JIT} \label{sec:lifetimes} @@ -491,13 +491,13 @@ category 3. The creation of objects in category 1 is removed by the optimization described -in Section~\ref{sec:virtuals}. XXX +in Section~\ref{sec:statics}. XXX -\section{Escape Analysis in a Tracing JIT} % XXX imprecise title -\label{sec:virtuals} +\section{Allocation Removal in Traces} +\label{sec:statics} -\subsection{Virtual Objects} +\subsection{Static Objects} The main insight to improve the code shown in the last section is that objects in category 1 don't survive very long -- they are used only inside the loop and @@ -505,27 +505,27 @@ the code is thus to analyze which objects fall in category 1 and may thus not be allocated at all. -XXX rewrite to acknowledge the existence of the formal stuff - -This process is called \emph{escape analysis}. The escape analysis of -our tracing JIT works by using \emph{virtual objects}: The trace is walked from -beginning to end and whenever a \texttt{new} operation is seen, the operation is -removed and a shape description is constructed and associated with the variable -that would have stored the result of \texttt{new}. This result, a newly -allocated object, is called \emph{virtual}, because after the optimization is -finished, it does not need to be allocated at all. The shape description summarizes -the shape of this original object -and is used by the optimization to improve the trace. The shape describes -where the values that would be stored in the fields of the allocated objects -come from. Whenever the optimizer sees a \texttt{setfield} that writes into such -an object, that shape description is updated and the operation can be removed. -When the optimizer encounters a \texttt{getfield} from such an object, the result is read -from the shape description, and the operation is also removed. -Equivalently, a \texttt{guard\_class} on a variable that has a shape description can be removed as -well, because the shape description stores the class. +This is a process that is usually called \emph{escape analysis}. In this paper we will +perform escape analysis by using partial evaluation. The partial evalution is a +bit peculiar in that there are not actually any constant arguments to the trace, +but it is only used to optimized operations within a trace. XXX mention Prolog. + +The partial evaluation works by walking the trace from beginning to end. +Whenever a \texttt{new} operation is seen, the operation is removed and a static +object is constructed and associated with the variable that would have stored +the result of \texttt{new}. The static object describes the shape of the +original object, \eg where the values that would be stored in the fields of the +allocated object come from, as well as the type of the object. Whenever the +optimizer sees a \texttt{setfield} that writes into such an object, that shape +description is updated and the operation can be removed, which means that the +operation was done at partial evaluation time. When the optimizer encounters a +\texttt{getfield} from such an object, the result is read from the shape +description, and the operation is also removed. Equivalently, a +\texttt{guard\_class} on a variable that has a shape description can be removed +as well, because the shape description stores the type. In the example from last section, the following operations would produce two -shape descriptions, and be completely removed from the optimized trace: +static objects, and be completely removed from the optimized trace: \texttt{ \begin{tabular}{l} @@ -536,10 +536,10 @@ \end{tabular} } -The shape description associated with $p_{5}$ would know that it is an +The static object associated with $p_{5}$ would know that it is a \texttt{BoxedInteger}, and that the \texttt{intval} field contains $i_{4}$, the -associated with $p_{6}$ would know that its \texttt{intval} field contains the -constant -100. +one associated with $p_{6}$ would know that its \texttt{intval} field contains +the constant -100. The following operations, that use $p_{5}$ and $p_{6}$ could then be optimized using that knowledge: @@ -569,21 +569,23 @@ The rest of the trace is optimized similarly. -So far we have only described what happens when virtual objects are used in -operations that read and write their fields and in guards. When the virtual -object is used in -any other operation, it cannot stay virtual. For example, when a virtual object -is stored in a globally accessible place, the object needs to actually be -allocated, as it might live longer than one iteration of the loop. The actual -allocation is inserted just before the operation that requires the object to be -non-virtual. +So far we have only described what happens when static objects are used in +operations that read and write their fields and in guards. When the static +object is used in any other operation, it cannot stay static. For example, when +a static object is stored in a globally accessible place, the object needs to +actually be allocated, as it might live longer than one iteration of the loop +and because the partial evaluator looses track of it. This means that the static +objects needs to be turned into a dynamic one, \ie lifted. This makes it +necessary to put operations into the residual code that actually allocate the +static object at runtime. This is what happens at the end of the trace in Figure~\ref{fig:unopt-trace}, when the \texttt{jump} operation -is hit. The arguments of the jump are at this point virtual objects. Before the -jump is emitted, they are \emph{forced}. This means that the optimizers produces code +is hit. The arguments of the jump are at this point static objects. Before the +jump is emitted, they are \emph{lifted}. This means that the optimizers produces code that allocates a new object of the right type and sets its fields to the field -values that the virtual object has. This means that instead of the jump, the -following operations are emitted: +values that the static object has (if the static object points to other static +objects, those need to be lifted as well) This means that instead of the jump, +the following operations are emitted: \texttt{ \begin{tabular}{l} @@ -599,7 +601,7 @@ trace. It looks like for these operations we actually didn't win much, because the objects are still allocated at the end. However, the optimization was still worthwhile even in this case, because some operations that have been performed -on the forced virtual objects have been removed (some \texttt{getfield} operations +on the lifted static objects have been removed (some \texttt{getfield} operations and \texttt{guard\_class} operations). \begin{figure} @@ -840,7 +842,7 @@ optimization can be improved further. XXX Category 2 The optimization of -Section~\ref{sec:virtuals} deals with them too: the \texttt{new} that creates them and +Section~\ref{sec:statics} deals with them too: the \texttt{new} that creates them and the field accesses are deferred, until the point where the object escapes. % section Escape Analysis in a Tracing JIT (end) From david at codespeak.net Thu Oct 7 16:27:29 2010 From: david at codespeak.net (david at codespeak.net) Date: Thu, 7 Oct 2010 16:27:29 +0200 (CEST) Subject: [pypy-svn] r77682 - in pypy/branch/arm-backend/pypy/jit/backend/arm: . test Message-ID: <20101007142729.42EF836C227@codespeak.net> Author: david Date: Thu Oct 7 16:27:27 2010 New Revision: 77682 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Log: LDM instruction encoding Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Thu Oct 7 16:27:27 2010 @@ -35,7 +35,8 @@ self.mc.LDR_ri(r.r3, r.fp, -16) #self.mc.write32(0xe1a00003) # mov r0, r3 self.mc.SUB_ri(r.sp, r.fp, 12) - self.mc.write32(0xe89da800) # ldm sp, {fp, sp, pc} + self.mc.LDM(r.sp, [r.fp, r.sp, r.pc]) + #self.mc.write32(0xe89da800) # ldm sp, {fp, sp, pc} def gen_func_prolog(self): self.mc.MOV_rr(r.ip, r.sp) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py Thu Oct 7 16:27:27 2010 @@ -100,10 +100,19 @@ | (rm & 0xFF)) def PUSH(self, regs, cond=cond.AL): - instr = cond << 28 | 0x92D << 16 + instr = self._encode_reg_list(cond << 28 | 0x92D << 16, regs) + self.write32(instr) + + def LDM(self, rn, regs, cond=cond.AL): + w = 0 + instr = cond << 28 | 0x89 << 20 | w << 21 | (rn & 0xFF) << 16 + instr = self._encode_reg_list(instr, regs) + self.write32(instr) + + def _encode_reg_list(self, instr, regs): for reg in regs: instr |= 0x1 << reg - self.write32(instr) + return instr def write32(self, word): self.writechar(chr(word & 0xFF)) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Thu Oct 7 16:27:27 2010 @@ -81,6 +81,18 @@ self.cb.PUSH([r.fp, r.ip, r.lr, r.pc]) self.assert_equal('PUSH {fp, ip, lr, pc}') + def test_ldm_one_reg(self): + self.cb.LDM(r.sp, [r.fp]) + self.assert_equal('LDM sp, {fp}') + + def test_ldm_multiple_reg(self): + self.cb.LDM(r.sp, [r.fp, r.ip, r.lr]) + self.assert_equal('LDM sp, {fp, ip, lr}') + + def test_ldm_multiple_reg2(self): + self.cb.LDM(r.sp, [r.fp, r.sp, r.pc]) + self.assert_equal("LDM sp, {fp, sp, pc}") + def test_sub_ri(self): self.cb.SUB_ri(r.r2, r.r4, 123) self.assert_equal('SUB r2, r4, #123') From antocuni at codespeak.net Thu Oct 7 16:44:38 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 7 Oct 2010 16:44:38 +0200 (CEST) Subject: [pypy-svn] r77683 - in pypy/branch/jitffi/pypy: module/_ffi module/_ffi/test rlib Message-ID: <20101007144438.C3C55282B90@codespeak.net> Author: antocuni Date: Thu Oct 7 16:44:37 2010 New Revision: 77683 Modified: pypy/branch/jitffi/pypy/module/_ffi/__init__.py pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: app-level call of an ffi function works :-) (modulo tons of details left) Modified: pypy/branch/jitffi/pypy/module/_ffi/__init__.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/__init__.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/__init__.py Thu Oct 7 16:44:37 2010 @@ -5,7 +5,6 @@ interpleveldefs = { 'CDLL' : 'interp_ffi.W_CDLL', -# 'FuncPtr' : 'interp_ffi.W_FuncPtr', 'types': 'interp_ffi.W_types', } Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Thu Oct 7 16:44:37 2010 @@ -1,10 +1,12 @@ import sys from pypy.interpreter.baseobjspace import W_Root, ObjSpace, Wrappable, Arguments from pypy.interpreter.error import OperationError, wrap_oserror, operationerrfmt -from pypy.interpreter.gateway import interp2app, NoneNotWrapped +from pypy.interpreter.gateway import interp2app, NoneNotWrapped, unwrap_spec from pypy.interpreter.typedef import TypeDef, GetSetProperty - -from pypy.rlib.libffi import CDLL, types +# +from pypy.rpython.lltypesystem import lltype, rffi +# +from pypy.rlib import libffi from pypy.rlib.rdynload import DLOpenError class W_FFIType(Wrappable): @@ -12,9 +14,10 @@ self.name = name self.ffitype = ffitype + @unwrap_spec('self', ObjSpace) def str(self, space): return space.wrap('' % self.name) - str.unwrap_spec = ['self', ObjSpace] + W_FFIType.typedef = TypeDef( @@ -27,10 +30,12 @@ pass def build_ffi_types(): + from pypy.rlib.clibffi import FFI_TYPE_P tdict = {} - for key, value in types.__dict__.iteritems(): - if key.startswith('__'): + for key, value in libffi.types.__dict__.iteritems(): + if key == 'getkind' or key.startswith('__'): continue + assert lltype.typeOf(value) == FFI_TYPE_P tdict[key] = W_FFIType(key, value) return tdict @@ -40,24 +45,72 @@ # ======================================================================== +class W_FuncPtr(Wrappable): + def __init__(self, func): + self.func = func + + @unwrap_spec('self', ObjSpace, 'args_w') + def call(self, space, args_w): + assert len(args_w) == len(self.func.argtypes) # XXX: raise OperationError + argchain = libffi.ArgChain() + for i in range(len(args_w)): + argtype = self.func.argtypes[i] + w_arg = args_w[i] + kind = libffi.types.getkind(argtype) + if kind == 'i': + assert False + #argchain.arg(space.int_w(w_arg)) + elif kind == 'f': + argchain.arg(space.float_w(w_arg)) + else: + assert False # XXX + # + reskind = libffi.types.getkind(self.func.restype) + if reskind == 'f': + floatres = self.func.call(argchain, rffi.DOUBLE) + return space.wrap(floatres) + else: + assert False + +W_FuncPtr.typedef = TypeDef( + 'FuncPtr', + __call__ = interp2app(W_FuncPtr.call) + ) + + + +# ======================================================================== + class W_CDLL(Wrappable): def __init__(self, space, name): try: - self.cdll = CDLL(name) + self.cdll = libffi.CDLL(name) except DLOpenError, e: raise operationerrfmt(space.w_OSError, '%s: %s', name, e.msg or 'unspecified error') self.name = name self.space = space + def ffitype(self, w_argtype): + return self.space.interp_w(W_FFIType, w_argtype).ffitype + + @unwrap_spec('self', ObjSpace, str, W_Root, W_Root) + def getfunc(self, space, name, w_argtypes, w_restype): + argtypes = [self.ffitype(w_argtype) for w_argtype in space.listview(w_argtypes)] + restype = self.ffitype(w_restype) + func = self.cdll.getpointer(name, argtypes, restype) + return W_FuncPtr(func) + + at unwrap_spec(ObjSpace, W_Root, str) def descr_new_cdll(space, w_type, name): return space.wrap(W_CDLL(space, name)) -descr_new_cdll.unwrap_spec = [ObjSpace, W_Root, str] + W_CDLL.typedef = TypeDef( 'CDLL', __new__ = interp2app(descr_new_cdll), + getfunc = interp2app(W_CDLL.getfunc), ) # ======================================================================== Modified: pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Thu Oct 7 16:44:37 2010 @@ -38,7 +38,7 @@ cls.space = space cls.w_libfoo_name = space.wrap(cls.prepare_c_example()) cls.w_libc_name = space.wrap(get_libc_name()) - cls.w_libc_name = space.wrap(get_libm_name(sys.platform)) + cls.w_libm_name = space.wrap(get_libm_name(sys.platform)) def test_libload(self): import _ffi @@ -53,3 +53,9 @@ assert str(types.sint) == '' assert str(types.uint) == '' + def test_callfunc(self): + from _ffi import CDLL, types + libm = CDLL(self.libm_name) + pow = libm.getfunc('pow', [types.double, types.double], types.double) + assert pow(2, 3) == 8 + Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Thu Oct 7 16:44:37 2010 @@ -32,6 +32,13 @@ cls.ulong = clibffi.cast_type_to_ffitype(rffi.ULONG) del cls._import + @classmethod + def getkind(cls, ffitype): + # XXX: move this function outside the jit + from pypy.jit.backend.llsupport.ffisupport import get_ffi_type_kind + return get_ffi_type_kind(ffitype) + + types._import() @specialize.arg(0) From antocuni at codespeak.net Thu Oct 7 16:48:18 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 7 Oct 2010 16:48:18 +0200 (CEST) Subject: [pypy-svn] r77684 - in pypy/branch/jitffi/pypy/module/_ffi: . test Message-ID: <20101007144818.7CC9F36C227@codespeak.net> Author: antocuni Date: Thu Oct 7 16:48:17 2010 New Revision: 77684 Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Log: support int arguments and return value Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Thu Oct 7 16:48:17 2010 @@ -58,15 +58,17 @@ w_arg = args_w[i] kind = libffi.types.getkind(argtype) if kind == 'i': - assert False - #argchain.arg(space.int_w(w_arg)) + argchain.arg(space.int_w(w_arg)) elif kind == 'f': argchain.arg(space.float_w(w_arg)) else: assert False # XXX # reskind = libffi.types.getkind(self.func.restype) - if reskind == 'f': + if reskind == 'i': + intres = self.func.call(argchain, rffi.LONG) + return space.wrap(intres) + elif reskind == 'f': floatres = self.func.call(argchain, rffi.DOUBLE) return space.wrap(floatres) else: Modified: pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Thu Oct 7 16:48:17 2010 @@ -59,3 +59,14 @@ pow = libm.getfunc('pow', [types.double, types.double], types.double) assert pow(2, 3) == 8 + def test_int_args(self): + """ + int sum_xy(int x, int y) + { + return x+y; + } + """ + from _ffi import CDLL, types + libfoo = CDLL(self.libfoo_name) + sum_xy = libfoo.getfunc('sum_xy', [types.sint, types.sint], types.sint) + assert sum_xy(30, 12) == 42 From antocuni at codespeak.net Thu Oct 7 17:00:26 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 7 Oct 2010 17:00:26 +0200 (CEST) Subject: [pypy-svn] r77685 - in pypy/branch/jitffi/pypy: jit/backend/llgraph jit/codewriter rlib rlib/test Message-ID: <20101007150026.4B7C336C225@codespeak.net> Author: antocuni Date: Thu Oct 7 17:00:24 2010 New Revision: 77685 Modified: pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py pypy/branch/jitffi/pypy/jit/codewriter/support.py pypy/branch/jitffi/pypy/rlib/libffi.py pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Log: add support for void return value (at interp-level) Modified: pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py Thu Oct 7 17:00:24 2010 @@ -1403,7 +1403,7 @@ kind2TYPE = { 'i': lltype.Signed, 'f': lltype.Float, - 'r': rffi.VOIDP, # XXX this is probably wrong + 'v': lltype.Void, } def _do_call_common(f, args_in_order=None, calldescr=None): Modified: pypy/branch/jitffi/pypy/jit/codewriter/support.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/support.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/support.py Thu Oct 7 17:00:24 2010 @@ -241,6 +241,8 @@ def _ll_3_libffi_call_float(llfunc, funcsym, ll_args): return func(llfunc)._do_call(funcsym, ll_args, rffi.DOUBLE) +def _ll_3_libffi_call_void(llfunc, funcsym, ll_args): + return func(llfunc)._do_call(funcsym, ll_args, lltype.Void) # in the following calls to builtins, the JIT is allowed to look inside: Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Thu Oct 7 17:00:24 2010 @@ -162,6 +162,8 @@ res = self._do_call_int(self.funcsym, ll_args) elif RESULT is rffi.DOUBLE: return self._do_call_float(self.funcsym, ll_args) + elif RESULT is lltype.Void: + return self._do_call_void(self.funcsym, ll_args) else: raise TypeError, 'Unsupported result type: %s' % RESULT # @@ -201,6 +203,10 @@ def _do_call_float(self, funcsym, ll_args): return self._do_call(funcsym, ll_args, rffi.DOUBLE) + @jit.oopspec('libffi_call_void(self, funcsym, ll_args)') + def _do_call_void(self, funcsym, ll_args): + return self._do_call(funcsym, ll_args, lltype.Void) + # ------------------------------------------------------------------------ # private methods @@ -235,7 +241,8 @@ return res def _free_buffers(self, ll_result, ll_args): - lltype.free(ll_result, flavor='raw') + if ll_result: + lltype.free(ll_result, flavor='raw') for i in range(len(self.argtypes)): lltype.free(ll_args[i], flavor='raw') lltype.free(ll_args, flavor='raw') Modified: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_libffi.py Thu Oct 7 17:00:24 2010 @@ -231,3 +231,21 @@ res = self.call(func, [], LONGP, init_result=null) assert res[0] == 20 + def test_void_result(self): + """ + int dummy = 0; + void set_dummy(int val) { dummy = val; } + int get_dummy() { return dummy; } + """ + libfoo = self.get_libfoo() + set_dummy = (libfoo, 'set_dummy', [types.sint], types.void) + get_dummy = (libfoo, 'get_dummy', [], types.sint) + # + res = self.call(get_dummy, [], rffi.LONG) + assert res == 0 + # + res = self.call(set_dummy, [42], lltype.Void, init_result=None) + assert res is None + # + res = self.call(get_dummy, [], rffi.LONG) + assert res == 42 From antocuni at codespeak.net Thu Oct 7 17:02:42 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 7 Oct 2010 17:02:42 +0200 (CEST) Subject: [pypy-svn] r77686 - in pypy/branch/jitffi/pypy/module/_ffi: . test Message-ID: <20101007150242.CAC5A36C225@codespeak.net> Author: antocuni Date: Thu Oct 7 17:02:41 2010 New Revision: 77686 Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Log: add support for void return value (at app-level) Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Thu Oct 7 17:02:41 2010 @@ -72,7 +72,9 @@ floatres = self.func.call(argchain, rffi.DOUBLE) return space.wrap(floatres) else: - assert False + voidres = self.func.call(argchain, lltype.Void) + assert voidres is None + return space.w_None W_FuncPtr.typedef = TypeDef( 'FuncPtr', Modified: pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Thu Oct 7 17:02:41 2010 @@ -70,3 +70,17 @@ libfoo = CDLL(self.libfoo_name) sum_xy = libfoo.getfunc('sum_xy', [types.sint, types.sint], types.sint) assert sum_xy(30, 12) == 42 + + def test_void_result(self): + """ + int dummy = 0; + void set_dummy(int val) { dummy = val; } + int get_dummy() { return dummy; } + """ + from _ffi import CDLL, types + libfoo = CDLL(self.libfoo_name) + set_dummy = libfoo.getfunc('set_dummy', [types.sint], types.void) + get_dummy = libfoo.getfunc('get_dummy', [], types.sint) + assert get_dummy() == 0 + assert set_dummy(42) is None + assert get_dummy() == 42 From antocuni at codespeak.net Thu Oct 7 17:07:22 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 7 Oct 2010 17:07:22 +0200 (CEST) Subject: [pypy-svn] r77687 - pypy/branch/jitffi/pypy/module/_ffi Message-ID: <20101007150722.027E036C225@codespeak.net> Author: antocuni Date: Thu Oct 7 17:07:21 2010 New Revision: 77687 Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Log: try to be nicer wrt the jit: the construction of the argchain has been moved to its own function, which is marked as unroll_safe: this should be enough to make the argchain completely virtual Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Thu Oct 7 17:07:21 2010 @@ -6,6 +6,7 @@ # from pypy.rpython.lltypesystem import lltype, rffi # +from pypy.rlib import jit from pypy.rlib import libffi from pypy.rlib.rdynload import DLOpenError @@ -49,12 +50,12 @@ def __init__(self, func): self.func = func - @unwrap_spec('self', ObjSpace, 'args_w') - def call(self, space, args_w): - assert len(args_w) == len(self.func.argtypes) # XXX: raise OperationError + @jit.unroll_safe + def build_argchain(self, space, argtypes, args_w): + assert len(args_w) == len(argtypes) # XXX: raise OperationError argchain = libffi.ArgChain() for i in range(len(args_w)): - argtype = self.func.argtypes[i] + argtype = argtypes[i] w_arg = args_w[i] kind = libffi.types.getkind(argtype) if kind == 'i': @@ -63,7 +64,11 @@ argchain.arg(space.float_w(w_arg)) else: assert False # XXX - # + return argchain + + @unwrap_spec('self', ObjSpace, 'args_w') + def call(self, space, args_w): + argchain = self.build_argchain(space, self.func.argtypes, args_w) reskind = libffi.types.getkind(self.func.restype) if reskind == 'i': intres = self.func.call(argchain, rffi.LONG) From cfbolz at codespeak.net Thu Oct 7 17:07:31 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 17:07:31 +0200 (CEST) Subject: [pypy-svn] r77688 - pypy/extradoc/talk/pepm2011 Message-ID: <20101007150731.04DB2282B90@codespeak.net> Author: cfbolz Date: Thu Oct 7 17:07:30 2010 New Revision: 77688 Modified: pypy/extradoc/talk/pepm2011/paper.bib pypy/extradoc/talk/pepm2011/paper.tex Log: add which languages pypy supports Modified: pypy/extradoc/talk/pepm2011/paper.bib ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.bib (original) +++ pypy/extradoc/talk/pepm2011/paper.bib Thu Oct 7 17:07:30 2010 @@ -53,6 +53,21 @@ howpublished = {{https://developer.mozilla.org/en/SpiderMonkey/Internals/Property\_cache}} }, + at incollection{bruni_pygirl:_2009, + series = {Lecture Notes in Business Information Processing}, + title = {{PyGirl:} Generating {Whole-System} {VMs} from {High-Level} Prototypes Using {PyPy}}, + volume = {33}, + isbn = {978-3-642-02571-6}, + url = {http://dx.doi.org/10.1007/978-3-642-02571-6_19}, + booktitle = {Objects, Components, Models and Patterns}, + publisher = {Springer Berlin Heidelberg}, + author = {Camillo Bruni and Toon Verwaest}, + editor = {Will Aalst and John Mylopoulos and Norman M Sadeh and Michael J Shaw and Clemens Szyperski and Manuel Oriol and Bertrand Meyer}, + year = {2009}, + note = {10.1007/978-3-642-02571-6\_19}, + pages = {328--347} +}, + @inproceedings{davide_ancona_rpython:_2007, address = {Montreal, Quebec, Canada}, title = {{RPython:} a step towards reconciling dynamically and statically typed {OO} languages}, Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Thu Oct 7 17:07:30 2010 @@ -187,7 +187,15 @@ relatively high level, the language implementation is kept free of low-level details, such as object layout, garbage collection or memory model. Those aspects of the final VM are woven into the generated code during the translation -to C. XXX languages that are done using PyPy +to C. + +A number of languages have been implemented with PyPy. The project was started +to get a better Python implementation, which inspired the name of the project +and is still the main focus of development. In addition a number of other +languages were implemented, among them a Prolog interpreter +\cite{carl_friedrich_bolz_towards_2010}, a Smalltalk VM +\cite{carl_friedrich_bolz_back_2008} and a GameBoy emulator +\cite{bruni_pygirl:_2009}. The feature that makes PyPy more than a compiler with a runtime system is it's support for automated JIT compiler generation \cite{bolz_tracing_2009}. During @@ -196,8 +204,8 @@ automatic; it only needs to be guided by the language implementer by a small number of source-code hints. Mostly-automatically generating a JIT compiler has many advantages over writing one manually, which is an error-prone and tedious process. -By construction, the generated JIT has the same semantics as the interpreter, and -the process benefits all languages implemented as an interpreter in RPython. +By construction, the generated JIT has the same semantics as the interpreter. +Many optimizations can benefit all languages implemented as an interpreter in RPython. The JIT that is produced by PyPy's JIT generator is a \emph{tracing JIT compiler}, a concept which we now explain in more details. @@ -224,13 +232,18 @@ interpreter records all operations that it is executing while running one iteration of the hot loop. This history of executed operations of one loop is called a \emph{trace}. Because the trace corresponds to one iteration of a loop, -it always ends with a jump to its own beginning. +it always ends with a jump to its own beginning. The trace also contains all +operations that are performed in functions that were called in the loop, thus a +tracing JIT automatically performs inlining. This trace of operations is then the basis of the generated code. The trace is -optimized in some ways, and then turned into machine code. Generating machine -code is simple, because the traces are linear and the operations are very close -to machine level. The trace corresponds to one concrete execution of a loop, -therefore the code generated from it is only one possible path through the loop. +optimized in some ways, and then turned into machine code. Both optimizations +and machine code generation is simple, because the traces are linear. This +linearity makes many optimizations a lot more tractable, and the inlining that +happens gives the optimizations automatically more context to work with. + +Since the trace corresponds to one concrete execution of a loop, +the code generated from it is only one possible path through it. To make sure that the trace is maintaining the correct semantics, it contains a \emph{guard} at all places where the execution could have diverged from the path. Those guards check the assumptions under which execution can stay on the @@ -491,12 +504,12 @@ category 3. The creation of objects in category 1 is removed by the optimization described -in Section~\ref{sec:statics}. XXX +in Sections~\ref{sec:statics} and \ref{sec:formal}. We will look at objects in +category 3 in Section~\ref{sec:crossloop}. \section{Allocation Removal in Traces} \label{sec:statics} - \subsection{Static Objects} The main insight to improve the code shown in the last section is that objects From cfbolz at codespeak.net Thu Oct 7 17:08:44 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 17:08:44 +0200 (CEST) Subject: [pypy-svn] r77689 - pypy/extradoc/talk/pepm2011 Message-ID: <20101007150844.56958282B90@codespeak.net> Author: cfbolz Date: Thu Oct 7 17:08:42 2010 New Revision: 77689 Modified: pypy/extradoc/talk/pepm2011/math.lyx Log: use typewriter font for operations Modified: pypy/extradoc/talk/pepm2011/math.lyx ============================================================================== --- pypy/extradoc/talk/pepm2011/math.lyx (original) +++ pypy/extradoc/talk/pepm2011/math.lyx Thu Oct 7 17:08:42 2010 @@ -138,7 +138,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{l\,\mathrm{fresh}}{v=\mathrm{new}(T),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto l\right],H\left[l\mapsto\left(T,\mathrm{null},\mathrm{null}\right)\right]}}$ +\begin_inset Formula ${\displaystyle \frac{l\,\mathrm{fresh}}{v=\mathtt{new}(T),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto l\right],H\left[l\mapsto\left(T,\mathrm{null},\mathrm{null}\right)\right]}}$ \end_inset @@ -181,7 +181,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{\mathrm{type}(H(E(v))=T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H}}$ +\begin_inset Formula ${\displaystyle \frac{\mathrm{type}(H(E(v))=T}{\mathtt{guard\_class}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H}}$ \end_inset @@ -206,7 +206,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{\,}{v=\mathrm{get}(u,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto H\left(E\left(u\right)\right)_{F}\right],H}}$ +\begin_inset Formula ${\displaystyle \frac{\,}{v=\mathtt{get}(u,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto H\left(E\left(u\right)\right)_{F}\right],H}}$ \end_inset @@ -247,7 +247,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{\mathrm{type}(H(E(v))\neq T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$ +\begin_inset Formula ${\displaystyle \frac{\mathrm{type}(H(E(v))\neq T}{\mathtt{guard\_class}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$ \end_inset @@ -272,7 +272,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{\,}{\mathrm{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H\left[E\left(v\right)\mapsto\left(H\left(E\left(v\right)\right)!_{F}E(u)\right)\right]}}$ +\begin_inset Formula ${\displaystyle \frac{\,}{\mathtt{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H\left[E\left(v\right)\mapsto\left(H\left(E\left(v\right)\right)!_{F}E(u)\right)\right]}}$ \end_inset @@ -360,7 +360,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{v^{*}\,\mathrm{fresh}}{v=\mathrm{new}(T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto v^{*}\right],S\left[v^{*}\mapsto\left(T,\mathrm{null,null}\right)\right]}}$ +\begin_inset Formula ${\displaystyle \frac{v^{*}\,\mathrm{fresh}}{v=\mathtt{new}(T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto v^{*}\right],S\left[v^{*}\mapsto\left(T,\mathrm{null,null}\right)\right]}}$ \end_inset @@ -385,7 +385,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(u)\in\mathrm{dom}(S)}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto S(E(u))_{F}\right],S}}$ +\begin_inset Formula ${\displaystyle \frac{E(u)\in\mathrm{dom}(S)}{v=\mathtt{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto S(E(u))_{F}\right],S}}$ \end_inset @@ -408,7 +408,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(u)\notin\mathrm{dom}(S)\, v^{*}\,\mathrm{fresh}}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle v^{*}=\mathrm{get}(E(u),F)\right\rangle ,E\left[v\mapsto v^{*}\right],S}}$ +\begin_inset Formula ${\displaystyle \frac{E(u)\notin\mathrm{dom}(S)\, v^{*}\,\mathrm{fresh}}{v=\mathtt{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle v^{*}=\mathtt{get}(E(u),F)\right\rangle ,E\left[v\mapsto v^{*}\right],S}}$ \end_inset @@ -433,7 +433,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S\left[E\left(v\right)\mapsto\left(S(E(v))!_{F}E(u)\right)\right]}}$ +\begin_inset Formula ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{\mathtt{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S\left[E\left(v\right)\mapsto\left(S(E(v))!_{F}E(u)\right)\right]}}$ \end_inset @@ -456,7 +456,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}\left(S\right),\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathrm{set}\left(E(v),F,E(u)\right)\right\rangle ,E,S^{\prime}}}$ +\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}\left(S\right),\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathtt{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathtt{set}\left(E(v),F,E(u)\right)\right\rangle ,E,S^{\prime}}}$ \end_inset @@ -481,7 +481,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$ +\begin_inset Formula ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$ \end_inset @@ -504,7 +504,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$ +\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathtt{guard\_class}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$ \end_inset @@ -552,7 +552,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathrm{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$ +\begin_inset Formula ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathtt{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$ \end_inset @@ -575,7 +575,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathrm{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathrm{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$ +\begin_inset Formula ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathtt{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathtt{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$ \end_inset From david at codespeak.net Thu Oct 7 17:26:23 2010 From: david at codespeak.net (david at codespeak.net) Date: Thu, 7 Oct 2010 17:26:23 +0200 (CEST) Subject: [pypy-svn] r77691 - pypy/branch/arm-backend/pypy/jit/backend/arm Message-ID: <20101007152623.AD67D282B9E@codespeak.net> Author: david Date: Thu Oct 7 17:26:22 2010 New Revision: 77691 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py Log: updated arm calling convention code to store and restore all relevant registers on procedure entry/exit Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Thu Oct 7 17:26:22 2010 @@ -31,19 +31,10 @@ self.gen_func_epilog() def gen_func_epilog(self): - self.mc.STR_ri(r.r3, r.fp, -16) - self.mc.LDR_ri(r.r3, r.fp, -16) - #self.mc.write32(0xe1a00003) # mov r0, r3 - self.mc.SUB_ri(r.sp, r.fp, 12) - self.mc.LDM(r.sp, [r.fp, r.sp, r.pc]) - #self.mc.write32(0xe89da800) # ldm sp, {fp, sp, pc} + self.mc.LDM(r.sp, r.callee_restored_registers) def gen_func_prolog(self): - self.mc.MOV_rr(r.ip, r.sp) - self.mc.PUSH([r.fp, r.ip, r.lr, r.pc]) - self.mc.SUB_ri(r.fp, r.ip, 4) - self.mc.SUB_ri(r.sp, r.sp, 8) - self.mc.STR_ri(r.r0, r.fp, -20) + self.mc.PUSH(r.callee_saved_registers) def gen_load_int(self, reg, value): self.mc.MOV_ri(reg, (value & 0xFF)) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py Thu Oct 7 17:26:22 2010 @@ -5,3 +5,7 @@ sp = 13 lr = 14 pc = 15 + +callee_resp = [r4, r5, r6, r7, r8, r9, r10, r11] +callee_saved_registers = callee_resp+[lr] +callee_restored_registers = callee_resp+[pc] From antocuni at codespeak.net Thu Oct 7 17:27:13 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 7 Oct 2010 17:27:13 +0200 (CEST) Subject: [pypy-svn] r77692 - pypy/branch/jitffi/pypy/jit/backend/llsupport/test Message-ID: <20101007152713.679FA282BD4@codespeak.net> Author: antocuni Date: Thu Oct 7 17:27:11 2010 New Revision: 77692 Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_ffisupport.py Log: fix the test. ffi_type_pointer is really treated as an integer Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_ffisupport.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_ffisupport.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_ffisupport.py Thu Oct 7 17:27:11 2010 @@ -7,7 +7,7 @@ args = [types.sint, types.double, types.pointer] descr = get_call_descr_dynamic(args, types.void) assert isinstance(descr, VoidCallDescr) - assert descr.arg_classes == 'ifr' + assert descr.arg_classes == 'ifi' descr = get_call_descr_dynamic([], types.sint8) assert isinstance(descr, DynamicIntCallDescr) From antocuni at codespeak.net Thu Oct 7 17:32:18 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Thu, 7 Oct 2010 17:32:18 +0200 (CEST) Subject: [pypy-svn] r77693 - pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt Message-ID: <20101007153218.0EDCA36C227@codespeak.net> Author: antocuni Date: Thu Oct 7 17:32:17 2010 New Revision: 77693 Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py Log: re-enable the optimization Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py Thu Oct 7 17:32:17 2010 @@ -15,7 +15,7 @@ OptRewrite(), OptVirtualize(), OptHeap(), - #OptFfiCall(), + OptFfiCall(), ] optimizer = Optimizer(metainterp_sd, loop, optimizations, virtuals) optimizer.propagate_all_forward() Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py Thu Oct 7 17:32:17 2010 @@ -70,6 +70,10 @@ return None def optimize_CALL(self, op): + if we_are_translated(): + self.emit_operation(op) + return + # targetval = self.getvalue(op.getarg(0)) oopspec = self.get_oopspec(targetval) if oopspec == 'prepare_call': From cfbolz at codespeak.net Thu Oct 7 18:05:08 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 18:05:08 +0200 (CEST) Subject: [pypy-svn] r77694 - pypy/extradoc/talk/pepm2011 Message-ID: <20101007160508.4559B282B90@codespeak.net> Author: cfbolz Date: Thu Oct 7 18:05:06 2010 New Revision: 77694 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: rename operation to "get" and "set". Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Thu Oct 7 18:05:06 2010 @@ -364,52 +364,52 @@ \# inside f: res.add(y) \\ guard\_class($p_{1}$, BoxedInteger) \\ ~~~~\# inside BoxedInteger.add \\ -~~~~$i_{2}$ = getfield($p_{1}$, intval) \\ +~~~~$i_{2}$ = get($p_{1}$, intval) \\ ~~~~guard\_class($p_{0}$, BoxedInteger) \\ ~~~~~~~~\# inside BoxedInteger.add\_\_int \\ -~~~~~~~~$i_{3}$ = getfield($p_{0}$, intval) \\ +~~~~~~~~$i_{3}$ = get($p_{0}$, intval) \\ ~~~~~~~~$i_{4}$ = int\_add($i_{2}$, $i_{3}$) \\ ~~~~~~~~$p_{5}$ = new(BoxedInteger) \\ ~~~~~~~~~~~~\# inside BoxedInteger.\_\_init\_\_ \\ -~~~~~~~~~~~~setfield($p_{5}$, intval, $i_{4}$) \\ +~~~~~~~~~~~~set($p_{5}$, intval, $i_{4}$) \\ \# inside f: BoxedInteger(-100) \\ $p_{6}$ = new(BoxedInteger) \\ ~~~~\# inside BoxedInteger.\_\_init\_\_ \\ -~~~~setfield($p_{6}$, intval, -100) \\ +~~~~set($p_{6}$, intval, -100) \\ ~\\ \# inside f: .add(BoxedInteger(-100)) \\ guard\_class($p_{5}$, BoxedInteger) \\ ~~~~\# inside BoxedInteger.add \\ -~~~~$i_{7}$ = getfield($p_{5}$, intval) \\ +~~~~$i_{7}$ = get($p_{5}$, intval) \\ ~~~~guard\_class($p_{6}$, BoxedInteger) \\ ~~~~~~~~\# inside BoxedInteger.add\_\_int \\ -~~~~~~~~$i_{8}$ = getfield($p_{6}$, intval) \\ +~~~~~~~~$i_{8}$ = get($p_{6}$, intval) \\ ~~~~~~~~$i_{9}$ = int\_add($i_{7}$, $i_{8}$) \\ ~~~~~~~~$p_{10}$ = new(BoxedInteger) \\ ~~~~~~~~~~~~\# inside BoxedInteger.\_\_init\_\_ \\ -~~~~~~~~~~~~setfield($p_{10}$, intval, $i_{9}$) \\ +~~~~~~~~~~~~set($p_{10}$, intval, $i_{9}$) \\ ~\\ \# inside f: BoxedInteger(-1) \\ $p_{11}$ = new(BoxedInteger) \\ ~~~~\# inside BoxedInteger.\_\_init\_\_ \\ -~~~~setfield($p_{11}$, intval, -1) \\ +~~~~set($p_{11}$, intval, -1) \\ ~\\ \# inside f: y.add(BoxedInteger(-1)) \\ guard\_class($p_{0}$, BoxedInteger) \\ ~~~~\# inside BoxedInteger.add \\ -~~~~$i_{12}$ = getfield($p_{0}$, intval) \\ +~~~~$i_{12}$ = get($p_{0}$, intval) \\ ~~~~guard\_class($p_{11}$, BoxedInteger) \\ ~~~~~~~~\# inside BoxedInteger.add\_\_int \\ -~~~~~~~~$i_{13}$ = getfield($p_{11}$, intval) \\ +~~~~~~~~$i_{13}$ = get($p_{11}$, intval) \\ ~~~~~~~~$i_{14}$ = int\_add($i_{12}$, $i_{13}$) \\ ~~~~~~~~$p_{15}$ = new(BoxedInteger) \\ ~~~~~~~~~~~~\# inside BoxedInteger.\_\_init\_\_ \\ -~~~~~~~~~~~~setfield($p_{15}$, intval, $i_{14}$) \\ +~~~~~~~~~~~~set($p_{15}$, intval, $i_{14}$) \\ ~\\ \# inside f: y.is\_positive() \\ guard\_class($p_{15}$, BoxedInteger) \\ ~~~~\# inside BoxedInteger.is\_positive \\ -~~~~$i_{16}$ = getfield($p_{15}$, intval) \\ +~~~~$i_{16}$ = get($p_{15}$, intval) \\ ~~~~$i_{17}$ = int\_gt($i_{16}$, 0) \\ \# inside f \\ guard\_true($i_{17}$) \\ @@ -426,7 +426,7 @@ correspond to the stack level of the function that contains the traced operation. The trace also shows the inefficiencies of \texttt{f} clearly, if one looks at the number of \texttt{new} (corresponding to object creation), -\texttt{set/getfield} (corresponding to attribute reads/writes) and +\texttt{set/get} (corresponding to attribute reads/writes) and \texttt{guard\_class} operations (corresponding to method calls). Note how the functions that are called by \texttt{f} are automatically inlined @@ -529,10 +529,10 @@ the result of \texttt{new}. The static object describes the shape of the original object, \eg where the values that would be stored in the fields of the allocated object come from, as well as the type of the object. Whenever the -optimizer sees a \texttt{setfield} that writes into such an object, that shape +optimizer sees a \texttt{set} that writes into such an object, that shape description is updated and the operation can be removed, which means that the operation was done at partial evaluation time. When the optimizer encounters a -\texttt{getfield} from such an object, the result is read from the shape +\texttt{get} from such an object, the result is read from the shape description, and the operation is also removed. Equivalently, a \texttt{guard\_class} on a variable that has a shape description can be removed as well, because the shape description stores the type. @@ -543,9 +543,9 @@ \texttt{ \begin{tabular}{l} $p_{5}$ = new(BoxedInteger) \\ -setfield($p_{5}$, intval, $i_{4}$) \\ +set($p_{5}$, intval, $i_{4}$) \\ $p_{6}$ = new(BoxedInteger) \\ -setfield($p_{6}$, intval, -100) \\ +set($p_{6}$, intval, -100) \\ \end{tabular} } @@ -560,17 +560,17 @@ \texttt{ \begin{tabular}{l} guard\_class($p_{5}$, BoxedInteger) \\ -$i_{7}$ = getfield($p_{5}$, intval) \\ +$i_{7}$ = get($p_{5}$, intval) \\ \# inside BoxedInteger.add \\ guard\_class($p_{6}$, BoxedInteger) \\ \# inside BoxedInteger.add\_\_int \\ -$i_{8}$ = getfield($p_{6}$, intval) \\ +$i_{8}$ = get($p_{6}$, intval) \\ $i_{9}$ = int\_add($i_{7}$, $i_{8}$) \\ \end{tabular} } The \texttt{guard\_class} operations can be removed, because the classes of $p_{5}$ and -$p_{6}$ are known to be \texttt{BoxedInteger}. The \texttt{getfield} operations can be removed +$p_{6}$ are known to be \texttt{BoxedInteger}. The \texttt{get} operations can be removed and $i_{7}$ and $i_{8}$ are just replaced by $i_{4}$ and -100. Thus the only remaining operation in the optimized trace would be: @@ -603,9 +603,9 @@ \texttt{ \begin{tabular}{l} $p_{15}$ = new(BoxedInteger) \\ -setfield($p_{15}$, intval, $i_{14}$) \\ +set($p_{15}$, intval, $i_{14}$) \\ $p_{10}$ = new(BoxedInteger) \\ -setfield($p_{10}$, intval, $i_{9}$) \\ +set($p_{10}$, intval, $i_{9}$) \\ jump($p_{15}$, $p_{10}$) \\ \end{tabular} } @@ -614,7 +614,7 @@ trace. It looks like for these operations we actually didn't win much, because the objects are still allocated at the end. However, the optimization was still worthwhile even in this case, because some operations that have been performed -on the lifted static objects have been removed (some \texttt{getfield} operations +on the lifted static objects have been removed (some \texttt{get} operations and \texttt{guard\_class} operations). \begin{figure} @@ -671,18 +671,15 @@ \caption{The Operational Semantics of Simplified Traces} \end{figure*} -XXX think of a way to format the operations to make them look distinct from -normal text - In this section we want to give a formal description of the semantics of the traces and of the optimizer and liken the optimization to partial evaluation. We concentrate on the operations for manipulating dynamically allocated objects, as those are the only ones that are actually optimized. Without loss of generality we also consider only objects with two fields in this section. -Traces are lists of operations. The operations considered here are new (to make -a new object), get (to read a field out of an object), set (to write a field -into an object) and guard (to check the type of an object). The values of all +Traces are lists of operations. The operations considered here are \texttt{new} (to make +a new object), \texttt{get} (to read a field out of an object), \texttt{set} (to write a field +into an object) and \texttt{guard\_class} (to check the type of an object). The values of all variables are locations (i.e.~pointers). Locations are mapped to objects, which are represented by triples of a type $T$, and two locations that represent the fields of the object. When a new object is created, the fields are initialized @@ -714,15 +711,15 @@ heap under a fresh location $l$ and adds the result variable to the environment, mapping to the new location $l$. -The get operation reads a field $F$ out of an object and adds the result +The \texttt{get} operation reads a field $F$ out of an object and adds the result variable to the environment, mapping to the read location. The heap is unchanged. -The set operation changes field $F$ of an object stored at the location that +The \texttt{set} operation changes field $F$ of an object stored at the location that variable $v$ maps to. The new value of the field is the location in variable $u$. The environment is unchanged. -The guard operation is used to check whether the object stored at the location +The \texttt{guard\_class} operation is used to check whether the object stored at the location that variable $v$ maps to is of type $T$. If that is the case, then execution continues without changing heap and environment. Otherwise, execution is stopped. @@ -804,21 +801,21 @@ the execution semantics closely. The other case is that nothing is known about the variables, which means the operation has to be residualized. -If the argument $u$ of a get operation is mapped to something in the static -heap, the get can be performed at optimization time. Otherwise, the get +If the argument $u$ of a \texttt{get} operation is mapped to something in the static +heap, the get can be performed at optimization time. Otherwise, the \texttt{get} operation needs to be emitted. -If the first argument $v$ to a set operation is mapped to something in the -static heap, then the set can performed at optimization time and the static heap -is updated. Otherwise the set operation needs to be emitted. This needs to be +If the first argument $v$ to a \texttt{set} operation is mapped to something in the +static heap, then the \texttt{set} can performed at optimization time and the static heap +is updated. Otherwise the \texttt{set} operation needs to be emitted. This needs to be done carefully, because the new value for the field stored in the variable $u$ could itself be static, in which case it needs to be lifted first. -I a guard is performed on a variable that is in the static heap, the type check +I a \texttt{guard\_class} is performed on a variable that is in the static heap, the type check can be performed at optimization time, which means the operation can be removed if the types match. If the type check fails statically or if the object is not -in the static heap, the guard is put into the residual trace. This also needs to -lift the variable on which the guard is performed. +in the static heap, the \texttt{guard\_class} is put into the residual trace. This also needs to +lift the variable on which the \texttt{guard\_class} is performed. Lifting takes a variable that is potentially in the static heap and makes sure that it is turned into a dynamic variable. This means that operations are From cfbolz at codespeak.net Thu Oct 7 18:16:53 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 18:16:53 +0200 (CEST) Subject: [pypy-svn] r77695 - pypy/extradoc/talk/pepm2011 Message-ID: <20101007161653.EC90C36C225@codespeak.net> Author: cfbolz Date: Thu Oct 7 18:16:52 2010 New Revision: 77695 Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf pypy/extradoc/talk/pepm2011/paper.tex Log: some improvements Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf ============================================================================== Binary files. No diff available. Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Thu Oct 7 18:16:52 2010 @@ -760,7 +760,7 @@ $$\setlength\arraycolsep{0.1em} \begin{array}{rcll} E & \in & V\rightharpoonup V^* & \mathrm{\ Environment}\\ - S & \in & V^*\rightharpoonup\mathfrak{T}\times(V^*\cup \{ \mathrm{null} \})\times (V^*\cup \{ \mathrm{null}\} ) & \mathrm{\ Heap}\\ + S & \in & V^*\rightharpoonup\mathfrak{T}\times(V^*\cup \{ \mathrm{null} \})\times (V^*\cup \{ \mathrm{null}\} ) & \mathrm{\ Static\ Heap}\\ \\ \\ \end{array} @@ -836,31 +836,23 @@ XXX algorithm is linear in the length of the trace -% subsection Analysis of the Algorithm (end) +XXX Category 2 The optimization of +Section~\ref{sec:statics} deals with them too: the \texttt{new} that creates them and +the field accesses are deferred, until the point where the object escapes. +% subsection Analysis of the Algorithm (end) + % section Formal Description of the Algorithm (end) %___________________________________________________________________________ -\subsection{Summary} - -In this section we described how simple escape analysis within the scope of one -loop works. This optimizations reduces the allocation of many intermediate data -structures that become garbage quickly in an interpreter. It also removes a lot -of the type dispatching overhead. In the next section, we will explain how this -optimization can be improved further. - -XXX Category 2 The optimization of -Section~\ref{sec:statics} deals with them too: the \texttt{new} that creates them and -the field accesses are deferred, until the point where the object escapes. - % section Escape Analysis in a Tracing JIT (end) -\section{Escape Analysis Across Loop Boundaries} +\section{Allocation Removal Across Loop Boundaries} \label{sec:crossloop} -In the last section we described how escape analysis can be used to remove +In the last sections we described how partial evaluation can be used to remove many of the allocations of short-lived objects and many of the type dispatches that are present in a non-optimized trace. In this section we will improve the optimization to also handle more cases. @@ -885,9 +877,10 @@ behaviour is very similar, however, so we will use the simpler version (and we might switch to that at some point in the actual implementation).} -Let's look at the final trace obtained in the last section for the example loop. -The final trace was much better than the original one, because many allocations -were removed from it. However, it also still contained allocations: +Let's look at the final trace obtained in Section~\ref{sec:statics} for the +example loop. The final trace (Figure~\ref{fig:step1}) was much better than the +original one, because many allocations were removed from it. However, it also +still contained allocations. The two new \texttt{BoxedIntegers} stored in $p_{15}$ and $p_{10}$ are passed into the next iteration of the loop. The next iteration will check that they are @@ -919,7 +912,7 @@ Now the lifetime of the remaining allocations no longer crosses the jump, and -we can run our escape analysis a second time, to get the trace in +we can run our partial evaluation a second time, to get the trace in Figure~\ref{fig:step3}. \begin{figure} @@ -983,7 +976,7 @@ is also that the optimized loops are specialized for the types of the variables that are used inside them. -% section Escape Analysis Across Loop Boundaries (end) +% section Allocation Removal Across Loop Boundaries (end) \section{Supporting Techniques} \label{sec:support} From cfbolz at codespeak.net Thu Oct 7 18:17:18 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 18:17:18 +0200 (CEST) Subject: [pypy-svn] r77696 - pypy/extradoc/talk/pepm2011 Message-ID: <20101007161718.5343436C225@codespeak.net> Author: cfbolz Date: Thu Oct 7 18:17:16 2010 New Revision: 77696 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: this won't fit Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Thu Oct 7 18:17:16 2010 @@ -1006,9 +1006,6 @@ specifically written for one particular language. % subsection Virtualizables (end) -XXX store sinking - -do we need this? % section Supporting Techniques (end) From cfbolz at codespeak.net Thu Oct 7 18:21:11 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 18:21:11 +0200 (CEST) Subject: [pypy-svn] r77697 - pypy/extradoc/talk/pepm2011 Message-ID: <20101007162111.274D336C225@codespeak.net> Author: cfbolz Date: Thu Oct 7 18:21:09 2010 New Revision: 77697 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: these are all done Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Thu Oct 7 18:21:09 2010 @@ -90,14 +90,8 @@ \keywords{XXX}% -XXX drop the word "allocation removal" somewhere - -XXX define "escape analysis" - \section{Introduction} -XXX need to re-target introduction a bit to fit PEPMs focus - The goal of a just-in-time (JIT) compiler for a dynamic language is obviously to improve the speed of the language over an implementation of the language that uses interpretation. The first goal of a JIT is thus to remove the From cfbolz at codespeak.net Thu Oct 7 18:32:59 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 18:32:59 +0200 (CEST) Subject: [pypy-svn] r77698 - pypy/extradoc/talk/pepm2011 Message-ID: <20101007163259.40D3C36C225@codespeak.net> Author: cfbolz Date: Thu Oct 7 18:32:57 2010 New Revision: 77698 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: kill more XXXs Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Thu Oct 7 18:32:57 2010 @@ -57,7 +57,7 @@ \authorinfo{Carl Friedrich Bolz \and Antonio Cuni \and Maciej Fija?kowski \and Michael Leuschel \and Samuele Pedroni \and Armin Rigo} {Heinrich-Heine-Universit?t D?sseldorf, STUPS Group, Germany XXX} {cfbolz at gmx.de, anto.cuni at gmail.com, fijal at merlinux.eu, - samuele.pedroni at gmail.com, arigo at tunes.org} + leuschel at cs.uni-duesseldorf.de, samuele.pedroni at gmail.com, arigo at tunes.org} %\numberofauthors{3} %\author{ @@ -422,6 +422,8 @@ looks at the number of \texttt{new} (corresponding to object creation), \texttt{set/get} (corresponding to attribute reads/writes) and \texttt{guard\_class} operations (corresponding to method calls). +In the rest of the paper we will see how this trace can be optimized using +partial evaluation. Note how the functions that are called by \texttt{f} are automatically inlined into the trace. The method calls are always preceded by a \texttt{guard\_class} @@ -434,10 +436,6 @@ first \texttt{guard\_class} instruction will fail and execution will continue using the interpreter. -XXX simplify traces a bit more - -In the next section, we will see how this can be improved upon, using partial -evaluation. XXX \section{Object Lifetimes in a Tracing JIT} \label{sec:lifetimes} @@ -865,11 +863,6 @@ \subsection{Optimizing Across the Jump} -\footnote{This section is a bit -science-fictiony. The algorithm that PyPy currently uses is significantly more -complex and much harder than the one that is described here. The resulting -behaviour is very similar, however, so we will use the simpler version (and we -might switch to that at some point in the actual implementation).} Let's look at the final trace obtained in Section~\ref{sec:statics} for the example loop. The final trace (Figure~\ref{fig:step1}) was much better than the @@ -886,9 +879,14 @@ very quickly in the next iteration. In addition, the type checks at the start of the loop are superfluous, at least after the first iteration. + The reason why we cannot optimize the remaining allocations away is because their lifetime crosses the jump. To improve the situation, a little trick is -needed. The trace in Figure~\ref{fig:step1} represents a loop, i.e. the jump at the end jumps to +needed.\footnote{The algorithm that PyPy currently uses is significantly more complex +than the one that is described here. The resulting behaviour is nearly +identical, however, so we will use the simpler version (and plan to switch to +that at some point in the actual implementation).} +The trace in Figure~\ref{fig:step1} represents a loop, i.e. the jump at the end jumps to the beginning. Where in the loop the jump occurs is arbitrary, since the loop can only be left via failing guards anyway. Therefore it does not change the semantics of the loop to put the jump at another point into the trace and we @@ -896,15 +894,13 @@ appear in the current \texttt{jump}. This needs some care, because the arguments to \texttt{jump} are all currently live variables, thus they need to be adapted. -If we do that for our example trace, the trace looks like in Figure~\ref{fig:step2}. - \begin{figure} \includegraphics{figures/step2.pdf} \label{fig:step2} \caption{Shifting the Jump} \end{figure} - +If we do that for our example trace, the trace looks like in Figure~\ref{fig:step2}. Now the lifetime of the remaining allocations no longer crosses the jump, and we can run our partial evaluation a second time, to get the trace in Figure~\ref{fig:step3}. From hakanardo at codespeak.net Thu Oct 7 19:03:49 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Thu, 7 Oct 2010 19:03:49 +0200 (CEST) Subject: [pypy-svn] r77699 - in pypy/branch/jit-unroll-loops/pypy/jit/metainterp: . optimizeopt test Message-ID: <20101007170349.12BDC36C225@codespeak.net> Author: hakanardo Date: Thu Oct 7 19:03:47 2010 New Revision: 77699 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Log: only one entry bridge generated now Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py Thu Oct 7 19:03:47 2010 @@ -87,6 +87,8 @@ send_loop_to_backend(metainterp_sd, loop, "loop") send_loop_to_backend(metainterp_sd, loop.preamble, "loop") insert_loop_token(old_loop_tokens, loop.preamble.token) + jitdriver_sd.warmstate.attach_unoptimized_bridge_from_interp( + greenkey, loop.preamble.token) return loop.preamble.token else: send_loop_to_backend(metainterp_sd, loop, "loop") Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Thu Oct 7 19:03:47 2010 @@ -51,6 +51,7 @@ argmap[op.result] = newop.result descr = newop.getdescr() if isinstance(descr, ResumeGuardDescr): + op.getdescr().rd_snapshot = None #FIXME: In the right place? descr.rd_numb = None if newop.getopnum() == rop.JUMP: @@ -71,6 +72,7 @@ pass args.append(a) op.setfailargs(args) + op.getdescr().rd_snapshot = None #FIXME: In the right place? for a in op.getarglist(): if not isinstance(a, Const) and a in self.optimizer.values: Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Thu Oct 7 19:03:47 2010 @@ -348,12 +348,12 @@ myjitdriver.can_enter_jit(x=x, y=y, res=res) myjitdriver.jit_merge_point(x=x, y=y, res=res) res += x * x - if y<8: + if y<16: x += 1 y -= 1 return res - res = self.meta_interp(f, [6, 16]) - assert res == 919 + res = self.meta_interp(f, [6, 32]) + assert res == 3427 self.check_loop_count(3) def test_loop_invariant_mul_bridge_maintaining1(self): @@ -371,9 +371,9 @@ res = self.meta_interp(f, [6, 32]) assert res == 1167 self.check_loop_count(3) - self.check_loops({'int_lt': 1, 'int_gt': 1, - 'guard_false': 1, 'guard_true': 1, - 'int_sub': 2, 'int_mul': 2, 'int_add': 2, + self.check_loops({'int_lt': 2, 'int_gt': 1, + 'guard_false': 2, 'guard_true': 1, + 'int_sub': 3, 'int_mul': 1, 'int_add': 3, 'jump': 3}) def test_loop_invariant_mul_bridge_maintaining2(self): @@ -416,8 +416,8 @@ res = self.meta_interp(f, [6, 7]) assert res == 252 self.check_loop_count(2) - self.check_loops({'guard_true': 1, - 'int_add': 1, 'int_sub': 1, 'int_gt': 1, + self.check_loops({'guard_true': 2, + 'int_add': 2, 'int_sub': 2, 'int_gt': 2, 'int_mul': 1, 'getfield_gc_pure': 1, 'jump': 2}) From cfbolz at codespeak.net Thu Oct 7 19:16:36 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 19:16:36 +0200 (CEST) Subject: [pypy-svn] r77700 - pypy/extradoc/talk/pepm2011/benchmarks Message-ID: <20101007171636.F2AA4282B90@codespeak.net> Author: cfbolz Date: Thu Oct 7 19:16:35 2010 New Revision: 77700 Added: pypy/extradoc/talk/pepm2011/benchmarks/ pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json pypy/extradoc/talk/pepm2011/benchmarks/jit.json pypy/extradoc/talk/pepm2011/benchmarks/nojit.json Log: the raw benchmark data maciek gave me Added: pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json Thu Oct 7 19:16:35 2010 @@ -0,0 +1 @@ +{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [1.3408269882199999, 0.89828705787700003, 0.79922604560900001, 0.78520703315700002, 0.77960705757100002, 0.77884316444400004, 0.78138089180000003, 0.78564095497099995, 0.77933979034400003, 0.78032112121599995, 0.78032398223900001, 0.77994108200099999, 0.78709197044400003, 0.78046298027000005, 0.78100490570100001, 0.78174090385399997, 0.78066301345800004, 0.78935813903800001, 0.78030800819400004, 0.78058695793199995, 0.78065609931900004, 0.81349897384600001, 0.78035616874699998, 0.78057289123499995, 0.77895307540900005, 0.78031492233300004, 0.77951288223299997, 0.78010797500600004, 0.77941989898700004, 0.77966809272799997, 0.77928400039699997, 0.78034520149200004, 0.78004193306000003, 0.77843499183700005, 0.77900505065900005, 0.77978110313399995, 0.77990984916700001, 0.78063511848400002, 0.78277206420900003, 0.78097486496000001, 0.78050684928900005, 0.780658006668, 0.81388092041000004, 0.77942514419599995, 0.77982401847799998, 0.779586076736, 0.77912712097199999, 0.77842688560499995, 0.77977490425100005, 0.77885198593100002], "base_times": [1.96005105972, 1.9576470851900001, 1.96088910103, 1.9608309268999999, 1.9567291736600001, 1.9601850509600001, 1.9592499732999999, 1.95819783211, 1.95975589752, 1.9611279964399999, 1.9579570293399999, 1.96019601822, 1.95961117744, 1.95499491692, 1.9610810279799999, 1.95511603355, 1.9587879180900001, 1.95879602432, 1.95566296577, 1.9595639705700001, 1.95859694481, 1.9552450180100001, 1.95912909508, 1.9604980945599999, 1.95646905899, 1.9599859714500001, 1.96060585976, 1.9573318958299999, 1.95889997482, 1.9600548744199999, 1.9571371078499999, 1.96107792854, 1.9600939750699999, 1.9585239887200001, 1.9604239463799999, 1.95847797394, 1.9600219726599999, 1.9607260227200001, 1.95767593384, 1.9606490135200001, 1.9595348834999999, 1.9582331180600001, 1.95944404602, 1.96064209938, 1.9565119743299999, 1.9607801437400001, 1.9594299793200001, 1.95776391029, 1.95995807648, 1.9593291282700001]}], ["django", "RawResult", {"changed_times": [0.431762933731, 0.375938892365, 0.36341500282299999, 0.367589950562, 0.36318898200999999, 0.36300396919299999, 0.36316084861800002, 0.37451195716899999, 0.36371111869799999, 0.363828897476, 0.36306118965099998, 0.36771798133900002, 0.36362099647500001, 0.37479805946400002, 0.36376190185500001, 0.36297202110299998, 0.36336398124699998, 0.36382389068600002, 0.36317396163900001, 0.37901401519799999, 0.36382102966300001, 0.36297416687, 0.36419606208799998, 0.36393713951099999, 0.36413717269899998, 0.37551498413099998, 0.363807201385, 0.36678910255399999, 0.36428284645100001, 0.363599061966, 0.36372780799900001, 0.37592792511000001, 0.36405706405600002, 0.36310291290300001, 0.36305308341999998, 0.36352086067200001, 0.36760711669899998, 0.37471699714700002, 0.36391615867600002, 0.36355113983199999, 0.36311888694799999, 0.36337804794299999, 0.36314678192099997, 0.37514305114699997, 0.36750411987300002, 0.36400318145799998, 0.36394405364999999, 0.363122940063, 0.36291694641099997, 0.37458801269500003], "base_times": [0.62737584114099998, 0.62728309631300005, 0.627307891846, 0.62758588790900005, 0.62738704681400004, 0.62718105316200001, 0.62660717964199997, 0.62682294845599995, 0.62685489654500004, 0.626901865005, 0.62694001197799998, 0.62680888176000005, 0.62702178955099996, 0.62695503234899996, 0.626930952072, 0.62696814537000001, 0.62787008285500001, 0.62750697135900002, 0.62742996215799995, 0.62795114517200001, 0.62798190116899999, 0.62778282165499999, 0.62753701210000001, 0.62766814231900003, 0.62768888473499995, 0.62764692306500003, 0.627485990524, 0.62803101539600004, 0.62795495986899996, 0.62786889076200003, 0.62758994102499999, 0.62817692756700005, 0.62814188003500004, 0.62749600410499995, 0.62808799743699995, 0.62854099273700004, 0.62779498100300002, 0.62789297103899999, 0.62796211242699995, 0.62824296951299996, 0.62812089920000003, 0.62798500060999995, 0.62708497047400003, 0.62771701812699998, 0.62825989723200004, 0.62763118743900004, 0.62737894058200006, 0.62798690795900003, 0.62775301933299998, 0.627470970154]}], ["fannkuch", "RawResult", {"changed_times": [0.43332195282000002, 0.38041710853600003, 0.37921309471100001, 0.37792515754700001, 0.37811303138699998, 0.37825512886000001, 0.377966880798, 0.378439188004, 0.37887310981799999, 0.37781596183799998, 0.37887811660800003, 0.37811684608500001, 0.37786602973900002, 0.37938904762300002, 0.377534151077, 0.38209295272799998, 0.376986980438, 0.377496004105, 0.37841606140099998, 0.37773180007899998, 0.37749290466300001, 0.37853693962099999, 0.37957501411400002, 0.37821793556200001, 0.37789702415499998, 0.37755322456399998, 0.37845897674599999, 0.37784194946299998, 0.377678155899, 0.37787008285500001, 0.37771797180200001, 0.378336906433, 0.37751984596299998, 0.377940893173, 0.38246393203700002, 0.37781810760500001, 0.37745118141200001, 0.37853002548199999, 0.37841701507600001, 0.378228902817, 0.37974500656100002, 0.37745285034199999, 0.37854695320100001, 0.377611875534, 0.37745094299300003, 0.37820911407500002, 0.378015041351, 0.37732100486800002, 0.37784695625300002, 0.37757682800300002], "base_times": [1.3584520816800001, 1.3565759658800001, 1.3554861545600001, 1.3549780845599999, 1.3569629192399999, 1.35226488113, 1.35883998871, 1.35334992409, 1.3553960323300001, 1.35195589066, 1.3530910015099999, 1.35774278641, 1.3555450439500001, 1.35708403587, 1.3534219265, 1.3573589324999999, 1.3554661273999999, 1.35724401474, 1.3552920818300001, 1.35891509056, 1.35526394844, 1.3548588752699999, 1.3580558300000001, 1.3540120124799999, 1.3573079109199999, 1.3565089702599999, 1.3589961528800001, 1.3558809757200001, 1.35862016678, 1.3561370372799999, 1.3535299301099999, 1.35631299019, 1.35327315331, 1.3568890094799999, 1.3567941188799999, 1.35737490654, 1.35441994667, 1.35872101784, 1.35426902771, 1.3580150604200001, 1.35494399071, 1.3543949127199999, 1.3575069904299999, 1.3546030521400001, 1.3580288887, 1.3550050258599999, 1.35805010796, 1.35595703125, 1.3576319217699999, 1.3546538352999999]}], ["go", "RawResult", {"changed_times": [1.14897704124, 1.2345161438000001, 1.1498041153, 1.29407215118, 1.08174014091, 1.20440006256, 1.1048231124900001, 1.2440271377600001, 1.1091661453199999, 1.04328203201, 1.06029200554, 1.2512080669400001, 1.0826690197, 1.09385991096, 1.1346490383100001, 1.13187813759, 1.0736310482, 1.0886130332899999, 1.04829597473, 1.0568811893500001, 1.0900421142600001, 1.12933301926, 1.0717599391899999, 1.05894494057, 1.03591299057, 1.0868139267000001, 1.04581999779, 1.07085800171, 1.0427441597, 1.1460630893699999, 1.0693988800000001, 1.07336401939, 1.0726578235599999, 1.0731029510500001, 1.0713851451900001, 1.03998804092, 1.0435359477999999, 1.1242880821200001, 1.05398511887, 1.02363014221, 1.04768109322, 1.02518391609, 1.0290699005099999, 1.0308260917700001, 1.02171015739, 1.1216301918, 1.0642161369300001, 1.1219260692599999, 1.04517388344, 1.0866401195499999], "base_times": [0.59255790710400003, 0.599092006683, 0.59348702430699996, 0.59416389465300001, 0.59464716911299997, 0.59416198730500003, 0.59368491172799998, 0.59720110893199996, 0.59490489959699999, 0.59492588043200001, 0.59482812881500002, 0.59362506866499998, 0.59689593315099998, 0.59788894653299995, 0.59597611427300001, 0.59460496902500004, 0.59366416931199995, 0.59722304344199995, 0.59511184692399999, 0.59583401680000003, 0.59516787529000004, 0.59308505058299998, 0.59634900093099996, 0.59311318397500001, 0.59705591201800001, 0.595954179764, 0.59271693229699995, 0.59645009040800001, 0.59529709815999998, 0.59400391578699996, 0.59524512290999998, 0.59631705284100001, 0.59657907485999995, 0.59425806999200004, 0.59539294242899998, 0.59478211402900005, 0.59715604782099996, 0.59750199317899999, 0.596050977707, 0.59527707099899996, 0.59443402290299996, 0.59375095367399999, 0.59820485115099997, 0.59519815444900004, 0.59441995620700006, 0.59474897384600001, 0.59502887725800002, 0.59912514686600005, 0.59507703781099996, 0.59598803520200005]}], ["html5lib", "RawResult", {"changed_times": [23.0328969955, 17.374573946000002, 16.435561180099999, 15.829866170900001, 15.6528949738], "base_times": [8.6721470356000001, 8.7515590190899992, 8.73878312111, 8.7048590183299996, 8.6830830573999993]}], ["meteor-contest", "RawResult", {"changed_times": [0.35940599441499999, 0.29690194129899999, 0.29317784309400002, 0.296035051346, 0.29229211807299998, 0.29228496551499999, 0.31367707252499999, 0.291788816452, 0.29139184951800001, 0.29157185554499998, 0.29212999343899998, 0.291711091995, 0.30185389518700001, 0.29527878761300003, 0.290993928909, 0.29129290580700001, 0.29075789451599998, 0.29058814048800002, 0.29080295562699998, 0.290637969971, 0.29061007499699998, 0.29125404357899998, 0.29032993316700001, 0.29056715965300001, 0.29158401489300001, 0.29034614563, 0.29054307937599999, 0.29032897949199998, 0.29048895835900002, 0.290990114212, 0.30065107345600001, 0.29087710380600001, 0.29015898704499998, 0.29014682769799999, 0.29019498825099999, 0.29038119316099997, 0.291100978851, 0.290424108505, 0.29048800468399999, 0.29063892364499999, 0.29032087326, 0.29016399383500002, 0.29016995429999998, 0.29066991805999998, 0.29026079177899999, 0.29024004936199999, 0.29001903533899998, 0.29036879539499999, 0.29006004333500002, 0.29905819892899999], "base_times": [0.24460601806599999, 0.243553876877, 0.24350500106799999, 0.24308800697300001, 0.24368906021100001, 0.24375700950599999, 0.24353194236799999, 0.24331998825100001, 0.24436902999900001, 0.243896007538, 0.243317127228, 0.24335289001499999, 0.24417495727499999, 0.243915081024, 0.24382185936, 0.24391078949, 0.24390697479199999, 0.244327068329, 0.243896007538, 0.243628978729, 0.24348807334899999, 0.24388504028300001, 0.243416070938, 0.24379396438600001, 0.243729114532, 0.244396924973, 0.243938922882, 0.243566989899, 0.243503093719, 0.243898153305, 0.24379396438600001, 0.24365305900600001, 0.24366998672500001, 0.244452953339, 0.24378490448000001, 0.243889093399, 0.243768930435, 0.24420404434199999, 0.243615865707, 0.24389505386400001, 0.243630170822, 0.244421958923, 0.244014024734, 0.24404406547499999, 0.243921995163, 0.24391913413999999, 0.24434804916399999, 0.24426007270799999, 0.243828058243, 0.24390697479199999]}], ["nbody_modified", "RawResult", {"changed_times": [0.115092039108, 0.087569952011100002, 0.084489107132000002, 0.084752082824699995, 0.084683895111099999, 0.0844850540161, 0.084747791290299995, 0.084717988967900004, 0.0844769477844, 0.084722042083700003, 0.085180044174199998, 0.084502935409500005, 0.084743022918700006, 0.084678888320899995, 0.084486961364700006, 0.084739923477199997, 0.084715127944900007, 0.084480047225999999, 0.084748029708899997, 0.084702014923100005, 0.084467172622699996, 0.085448026657099996, 0.084717988967900004, 0.0844888687134, 0.084763050079300004, 0.084743976592999995, 0.084486961364700006, 0.0847291946411, 0.084710836410500007, 0.084522008895900003, 0.084717035293600002, 0.084691047668499997, 0.084509134292599999, 0.085023880004899993, 0.084743976592999995, 0.084515094757100007, 0.0847249031067, 0.084691047668499997, 0.084517955780000001, 0.084717988967900004, 0.084722995758099995, 0.084486007690400003, 0.084743022918700006, 0.084718227386500006, 0.084471940994299999, 0.085363149642899996, 0.084741115570100001, 0.0845909118652, 0.084722995758099995, 0.086489915847799997], "base_times": [0.40417194366499998, 0.40459799766499999, 0.40421700477599998, 0.40403985977200002, 0.40431380271900003, 0.40496802329999998, 0.40494704246500002, 0.40464997291600002, 0.40360713005100002, 0.40396285057100001, 0.403512001038, 0.403342008591, 0.403450012207, 0.40323090553300001, 0.40359306335400003, 0.40368413925199997, 0.40387392044100001, 0.403959989548, 0.40371799469000003, 0.40425086021399997, 0.40426588058500001, 0.40466904640200002, 0.40393304824800003, 0.40405297279399999, 0.40509009361300002, 0.40615010261500001, 0.40549302101099999, 0.405642986298, 0.40584611892700001, 0.40618610382100001, 0.40610885620100001, 0.40699696540800001, 0.40708398819000002, 0.406311988831, 0.40554094314599998, 0.40626502037000001, 0.40597796440099998, 0.40627288818399998, 0.40671586990399999, 0.40668511390700002, 0.40602588653600002, 0.40550208091700002, 0.40621900558500001, 0.40642905235299998, 0.40591812133799998, 0.40632581710799998, 0.40576291084299998, 0.40567016601599998, 0.40628194809000001, 0.40637803077700002]}], ["pyflate-fast", "RawResult", {"changed_times": [1.8020980358100001, 1.84628176689, 1.78963685036, 1.8260049819899999, 1.8187170028699999, 1.82588410378, 1.78930997849, 1.8257989883400001, 1.82189011574, 1.8041939735400001, 1.81843900681, 1.7986948490100001, 1.8285381794, 1.79674696922, 1.8408238887799999, 1.80955219269, 1.823802948, 1.80054306984, 1.8244709968599999, 1.82194590569, 1.8056621551500001, 1.8262178897900001, 1.7976200580599999, 1.8334770202599999, 1.81172394753, 1.8108351230599999, 1.83649802208, 1.82515788078, 1.80139780045, 1.8085131645200001, 1.81924104691, 1.83042001724, 1.8213908672300001, 1.8187279701200001, 1.81911611557, 1.8179550170900001, 1.8195929527300001, 1.8187839984900001, 1.8185229301500001, 1.85378098488, 1.79992103577, 1.8257040977500001, 1.81010389328, 1.8136050701099999, 1.83390116692, 1.8319008350399999, 1.8119609355899999, 1.81212496758, 1.8113842010500001, 1.8158957958199999], "base_times": [1.9715361595200001, 1.97649312019, 1.9744579792000001, 1.97769284248, 1.9740889072400001, 1.9755351543399999, 1.9799709320100001, 1.97855901718, 1.9788129329699999, 1.98169994354, 1.9795730114000001, 1.9760389328000001, 1.9755880832699999, 1.9830451011700001, 1.97887706757, 1.9769110679599999, 1.9802250862099999, 1.9751098156, 1.97200202942, 1.9791991710700001, 1.9742319583900001, 1.97487902641, 1.98901081085, 1.9792599678, 1.97859692574, 1.97864222527, 1.9701180458100001, 1.9805810451500001, 1.97375798225, 1.97326683998, 1.9741070270500001, 1.9746761321999999, 1.9728059768699999, 1.9711620807600001, 1.97109913826, 1.97114300728, 1.98034501076, 1.9756407737699999, 1.97721195221, 1.97855806351, 1.97502398491, 1.97638607025, 1.9746539592700001, 1.9767880439800001, 1.97603201866, 1.97480201721, 1.9750320911399999, 1.97502207756, 1.9728920459699999, 1.9767138957999999]}], ["raytrace-simple", "RawResult", {"changed_times": [1.1192121505699999, 1.12430500984, 1.14396595955, 1.1181769371000001, 1.12474393845, 1.1357970237699999, 1.1164288520800001, 1.1184740066500001, 1.1876978874199999, 1.12074708939, 1.1192080974600001, 1.11397886276, 1.11682295799, 1.1244900226600001, 1.1138560771899999, 1.1160299778, 1.1417291164400001, 1.11880779266, 1.11914396286, 1.11626505852, 1.24021792412, 1.11648702621, 1.1162600517300001, 1.11813688278, 1.1161379814100001, 1.1181910038, 1.1156840324399999, 1.1143019199399999, 1.1215739250200001, 1.23314094543, 1.1156570911399999, 1.11691904068, 1.1153750419599999, 1.1148521900199999, 1.11209821701, 1.1128268241899999, 1.11406207085, 1.1566548347500001, 1.1125919818900001, 1.1168460845899999, 1.11554002762, 1.1169860362999999, 1.11424398422, 1.11369895935, 1.12529706955, 1.16200399399, 1.11288189888, 1.1190280914299999, 1.1127638816800001, 1.1173341274299999], "base_times": [1.6308507919299999, 1.6061539650000001, 1.6082999706300001, 1.60679507256, 1.6074709892300001, 1.6028199195899999, 1.60836696625, 1.60565710068, 1.60432505608, 1.60986208916, 1.63934397697, 1.6348330974600001, 1.60514497757, 1.60835003853, 1.60388612747, 1.6093668937700001, 1.60202693939, 1.6092281341600001, 1.6030900478400001, 1.62427687645, 1.6077330112499999, 1.6068379879000001, 1.6104369163500001, 1.6036658287000001, 1.60728192329, 1.60446190834, 1.64275884628, 1.6381349563600001, 1.60937809944, 1.60457992554, 1.6079080104800001, 1.6385262012499999, 1.6040019989000001, 1.6066889762900001, 1.6061730384799999, 1.6106119155900001, 1.6066889762900001, 1.6068558692899999, 1.6067490577700001, 1.60826301575, 1.6016521453899999, 1.6092960834500001, 1.6044838428499999, 1.6094570159899999, 1.60569787025, 1.60940694809, 1.61079907417, 1.60953187943, 1.6040878295900001, 1.60577011108]}], ["richards", "RawResult", {"changed_times": [0.188634872437, 0.17832899093599999, 0.17431497573900001, 0.172646045685, 0.17323303222700001, 0.17191815376299999, 0.17156910896300001, 0.171721935272, 0.17107295989999999, 0.17158412933299999, 0.171247959137, 0.17127990722700001, 0.172159910202, 0.171751022339, 0.17034697532699999, 0.1701130867, 0.18102502822899999, 0.21005392074599999, 0.17098188400299999, 0.173319816589, 0.16994905471800001, 0.169959068298, 0.169358968735, 0.17905807495100001, 0.169506072998, 0.16935610771199999, 0.168767929077, 0.16901707649200001, 0.168606996536, 0.16865921020499999, 0.16839098930400001, 0.168842077255, 0.168585777283, 0.168391942978, 0.168664216995, 0.16862487793, 0.17172694206200001, 0.16935086250299999, 0.16864490508999999, 0.16911721229599999, 0.16891694068900001, 0.168562889099, 0.168758869171, 0.16852879524200001, 0.16824889183, 0.16894888877899999, 0.16881513595600001, 0.184386014938, 0.168937921524, 0.168543815613], "base_times": [0.21284794807400001, 0.21313500404399999, 0.21374988555900001, 0.21324205398599999, 0.21453785896300001, 0.21360492706299999, 0.21391510963400001, 0.213798999786, 0.213606119156, 0.21400713920600001, 0.21364998817399999, 0.21356511115999999, 0.214838027954, 0.21462011337299999, 0.21340608596800001, 0.212734937668, 0.21283102035500001, 0.21362400054899999, 0.214515924454, 0.21316099166899999, 0.21444606781, 0.21465396881099999, 0.21373987197899999, 0.213769197464, 0.213486909866, 0.213093996048, 0.21455001831100001, 0.21463489532499999, 0.213505029678, 0.21406984329199999, 0.215165138245, 0.215652942657, 0.21378803253199999, 0.21330499649000001, 0.21377801895099999, 0.214590072632, 0.214936971664, 0.21295380592300001, 0.212911844254, 0.213427066803, 0.21461415290800001, 0.21314406394999999, 0.213227987289, 0.21308803558299999, 0.21376299858100001, 0.21473717689499999, 0.213126897812, 0.212868928909, 0.21446895599400001, 0.21520400047300001]}], ["spambayes", "RawResult", {"changed_times": [0.35268497466999998, 0.48473596572900002, 0.48003888130200001, 0.54478287696799998, 0.58626985549900001, 0.49527096748400001, 0.45211386680600002, 0.43088984489400001, 0.30615997314499999, 0.54539084434499996, 0.397897958755, 0.34469389915499998, 0.34239602088900001, 0.322602987289, 0.68369793891899999, 0.41977381706200001, 0.42178511619600001, 0.33393692970299999, 0.383501052856, 0.358753919601, 0.30524086952200002, 0.38231086731000002, 0.31400895118700001, 0.34354496002200002, 0.30663800239599998, 0.31292319297799998, 0.38818907737699998, 0.30091691017200001, 0.39267206191999998, 0.29696297645600001, 0.372807025909, 0.456650972366, 0.29690003395100001, 0.34635186195399997, 0.29959511756899998, 0.34087109565700002, 0.37458395957899998, 0.306196928024, 0.29220890998799998, 0.35120987892200001, 0.31294512748699999, 0.29141306877099998, 0.31265306472799997, 0.34791302681000003, 0.29072403907799999, 0.28969788551300002, 0.29165005683900003, 0.34921717643700001, 0.38425016403200002, 0.28991580009500001], "base_times": [0.192424058914, 0.192607879639, 0.19253897666899999, 0.19256591796899999, 0.19250607490499999, 0.19241905212400001, 0.19255399704000001, 0.192532062531, 0.19254207611099999, 0.19256997108500001, 0.19251918792700001, 0.19549083709699999, 0.19256782531700001, 0.192401885986, 0.19272398948700001, 0.19264316558799999, 0.19307994842500001, 0.192593097687, 0.19265103340100001, 0.192397117615, 0.192836046219, 0.192632913589, 0.19270300865199999, 0.192532062531, 0.19256210327100001, 0.19262599945100001, 0.19496798515300001, 0.19257903099099999, 0.19252800941500001, 0.19245505332900001, 0.19250011444099999, 0.192535877228, 0.192492008209, 0.192594051361, 0.19248485565199999, 0.192499876022, 0.192586183548, 0.19254899025, 0.192557096481, 0.19252395629899999, 0.192557096481, 0.19257092475900001, 0.195178985596, 0.19255685806299999, 0.192538022995, 0.192622184753, 0.19253182411200001, 0.19251084327699999, 0.19250798225400001, 0.19247317314099999]}], ["spectral-norm", "RawResult", {"changed_times": [0.28835701942399999, 0.227369070053, 0.226712942123, 0.225772857666, 0.224092006683, 0.223258972168, 0.223456859589, 0.22423720359800001, 0.22428393363999999, 0.22225785255399999, 0.222506999969, 0.22108101844799999, 0.221049070358, 0.22128820419299999, 0.221066951752, 0.22229719162, 0.22106409072899999, 0.221334934235, 0.22109699249299999, 0.22112703323399999, 0.22133898734999999, 0.221033096313, 0.221338033676, 0.221124172211, 0.22107505798300001, 0.22130012512200001, 0.22103691101100001, 0.221324920654, 0.221083879471, 0.22106719017000001, 0.22131204605099999, 0.221076965332, 0.22137594223000001, 0.22109699249299999, 0.22108101844799999, 0.22130608558699999, 0.22109103202800001, 0.22142004966699999, 0.22129201889, 0.22182607650799999, 0.22131991386399999, 0.221093893051, 0.22135591506999999, 0.22108507156400001, 0.22105407714799999, 0.22134613990800001, 0.22112011909500001, 0.221364021301, 0.221080064774, 0.221827983856], "base_times": [0.38507890701300002, 0.38513684272799997, 0.38540577888499999, 0.385334968567, 0.38533091545100001, 0.38737297058100001, 0.38601493835400003, 0.38538408279399999, 0.38633489608799998, 0.38616895675700003, 0.38576793670699999, 0.38699293136599999, 0.38566589355499997, 0.38517880439800001, 0.38546013832100001, 0.385420084, 0.38523602485699998, 0.38521504402200002, 0.38548302650499999, 0.385416030884, 0.38633704185500001, 0.38570713996900002, 0.38650393486000001, 0.38614487648000001, 0.385656833649, 0.38680696487400001, 0.38568186759899997, 0.38566184043899998, 0.38578081131000003, 0.38579583167999998, 0.38546395301800002, 0.38546991348300003, 0.38550615310699998, 0.38638687133799998, 0.386216163635, 0.38625407218899999, 0.38584995269799999, 0.38524508476300001, 0.385580062866, 0.38524007797199999, 0.385420084, 0.38597679138199997, 0.38613581657399998, 0.385598182678, 0.38566184043899998, 0.38541483879100003, 0.38540506362900001, 0.38542604446399997, 0.385876178741, 0.38554000854499998]}], ["telco", "RawResult", {"changed_times": [0.69999999999999996, 0.64000000000000001, 0.64000000000000001, 0.66000000000000003, 0.63, 0.65000000000000002, 0.63, 0.62, 0.62, 0.63, 0.63, 0.62, 0.63, 0.62, 0.63, 0.63, 0.63, 0.63, 0.62, 0.62, 0.62, 0.63, 0.63, 0.64000000000000001, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62, 0.63, 0.63, 0.62, 0.62, 0.62, 0.63, 0.62, 0.62, 0.62, 0.63, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62], "base_times": [0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004]}], ["twisted_names", "RawResult", {"changed_times": [0.0089047195013357075, 0.0088105726872246704, 0.0087873462214411256, 0.0088809946714031966, 0.0088339222614840993, 0.0088417329796640146, 0.0089928057553956831, 0.009442870632672332, 0.0092336103416435829, 0.0088028169014084511, 0.0088183421516754845, 0.0090991810737033659, 0.0088495575221238937, 0.0087642418930762491, 0.0087642418930762491, 0.0096061479346781949, 0.0088809946714031966, 0.0090826521344232521, 0.0088495575221238937, 0.0087336244541484712, 0.0087642418930762491, 0.0087642418930762491, 0.008771929824561403, 0.0087412587412587419, 0.0096153846153846159, 0.0087642418930762491, 0.008771929824561403, 0.0091743119266055051, 0.0087642418930762491, 0.0090252707581227436, 0.008771929824561403, 0.0087260034904013961, 0.0087489063867016627, 0.0092506938020351526, 0.008771929824561403, 0.0087412587412587419, 0.0087565674255691769, 0.0087412587412587419, 0.0087412587412587419, 0.0087642418930762491, 0.0087412587412587419, 0.0087183958151700082, 0.0087412587412587419, 0.0087183958151700082, 0.0087412587412587419, 0.0087412587412587419, 0.0087642418930762491, 0.0087260034904013961, 0.0095693779904306216, 0.0087642418930762491], "base_times": [0.0065832784726793945, 0.0065832784726793945, 0.006587615283267457, 0.0065703022339027592, 0.0065919578114700065, 0.0065789473684210523, 0.0065789473684210523, 0.0066093853271645738, 0.0066006600660066007, 0.0066137566137566134, 0.0065832784726793945, 0.0065919578114700065, 0.0065963060686015833, 0.0065919578114700065, 0.0066006600660066007, 0.0065789473684210523, 0.0065963060686015833, 0.006587615283267457, 0.006587615283267457, 0.0065789473684210523, 0.0066006600660066007, 0.0065963060686015833, 0.0065789473684210523, 0.0066050198150594455, 0.0066093853271645738, 0.0065832784726793945, 0.0066093853271645738, 0.0065832784726793945, 0.0065746219592373442, 0.0066137566137566134, 0.0065919578114700065, 0.0065789473684210523, 0.0066006600660066007, 0.0066050198150594455, 0.0065832784726793945, 0.0066093853271645738, 0.0065963060686015833, 0.0065832784726793945, 0.0065963060686015833, 0.0065746219592373442, 0.0066050198150594455, 0.0065963060686015833, 0.0066137566137566134, 0.006587615283267457, 0.0066050198150594455, 0.0066137566137566134, 0.006587615283267457, 0.0065963060686015833, 0.006587615283267457, 0.0065746219592373442]}]], "branch": "trunk", "revision": 0} \ No newline at end of file Added: pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json Thu Oct 7 19:16:35 2010 @@ -0,0 +1 @@ +{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [0.53373098373399996, 0.23460102081299999, 0.15513491630599999, 0.15454292297399999, 0.15753602981600001, 0.16903495788600001, 0.16105413436900001, 0.15707802772500001, 0.15180110931400001, 0.152679920197, 0.15186595916699999, 0.15010499954199999, 0.15148997306799999, 0.15008211135899999, 0.15097498893700001, 0.152936935425, 0.15074682235699999, 0.148921012878, 0.15586805343599999, 0.15136885643, 0.150803089142, 0.14815711975099999, 0.144705057144, 0.145576000214, 0.162219047546, 0.145064115524, 0.14615988731400001, 0.14499902725200001, 0.14503312110899999, 0.145503044128, 0.14562892913799999, 0.15218400955200001, 0.15338587760899999, 0.153208971024, 0.15358805656399999, 0.15331101417500001, 0.14960598945600001, 0.151103019714, 0.15276718139600001, 0.15110111236599999, 0.150665998459, 0.14848804473900001, 0.151926040649, 0.15147399902299999, 0.199955940247, 0.14494299888600001, 0.14402890205400001, 0.14464497566199999, 0.14744687080400001, 0.148157835007], "base_times": [1.98487401009, 1.9759330749499999, 1.97832417488, 1.9764730930300001, 1.97814917564, 1.98089694977, 1.9775922298399999, 1.9774131774899999, 1.98496413231, 1.97159814835, 1.98496413231, 1.97353196144, 1.9848189353900001, 1.97257804871, 1.9910008907300001, 1.9682059288, 1.9917528629300001, 1.9724640846299999, 1.98371481895, 1.9713909625999999, 1.9869208335899999, 1.96859908104, 1.9883351326000001, 1.9759950637799999, 1.9792518615700001, 1.9805219173399999, 1.9756710529299999, 1.9734799861900001, 2.0159261226699998, 1.99893116951, 1.97291398048, 1.9838719368, 1.9677410125699999, 1.9859578609499999, 1.96171092987, 1.9678678512600001, 1.97432017326, 1.97341108322, 1.9648609161399999, 1.96110486984, 1.9707100391400001, 1.98760914803, 1.9710340499900001, 1.9824509620699999, 1.9752931594800001, 1.9807510375999999, 1.9799480438199999, 1.9826321601900001, 1.9798321723900001, 1.9828879833199999]}], ["django", "RawResult", {"changed_times": [0.150973081589, 0.123874902725, 0.12347698211700001, 0.12420392036400001, 0.12366008758499999, 0.123784065247, 0.13101482391399999, 0.123060941696, 0.123810052872, 0.12381196022, 0.123774051666, 0.123774051666, 0.124183893204, 0.12363195419299999, 0.130721092224, 0.121907949448, 0.12308216095000001, 0.12296700477600001, 0.123407840729, 0.123026132584, 0.12340188026399999, 0.123024940491, 0.130163908005, 0.12311792373700001, 0.123579978943, 0.123372077942, 0.123338937759, 0.123032093048, 0.12360286712599999, 0.123323917389, 0.130560159683, 0.123554944992, 0.123425006866, 0.12345290184, 0.12413501739500001, 0.12303304672199999, 0.123439073563, 0.123630046844, 0.130742788315, 0.123475074768, 0.123498916626, 0.123980045319, 0.12337899208100001, 0.123532056808, 0.12364602088899999, 0.123553037643, 0.130734920502, 0.123599052429, 0.12362909317, 0.123425960541], "base_times": [0.634876966476, 0.63119101524400001, 0.64540290832500002, 0.63067197799700003, 0.63386702537499995, 0.63004899025000005, 0.63024806976299996, 0.63103890418999997, 0.63627886772200004, 0.63133406639099998, 0.63078498840300001, 0.63178682327299995, 0.63143706321699999, 0.63687920570400003, 0.63120698928800001, 0.631051063538, 0.630378007889, 0.63146805763199998, 0.63601493835400003, 0.63147497177099998, 0.63133502006499997, 0.63147711753799995, 0.63075184822100006, 0.63706302642799995, 0.63112902641299995, 0.63136720657300005, 0.631887197495, 0.63646221160899996, 0.63175106048600005, 0.63184499740599998, 0.63149690628099997, 0.63135409355200001, 0.63489508628799995, 0.63233184814499999, 0.63151216506999996, 0.629364967346, 0.62923312187199998, 0.633197069168, 0.63016796112100004, 0.62957406043999997, 0.62918901443499997, 0.63133692741400005, 0.63574194908100001, 0.63390803337099999, 0.63227581977799996, 0.63121104240399994, 0.62836694717399999, 0.63247489929199996, 0.63313794136000001, 0.63352489471399998]}], ["fannkuch", "RawResult", {"changed_times": [0.357728004456, 0.31800484657299999, 0.30672502517700001, 0.32194614410400002, 0.31080985069299999, 0.30235886573800003, 0.30262207984900003, 0.30578708648699998, 0.31529092788700003, 0.31280684471100001, 0.31020498275800001, 0.31131386756899998, 0.31377482414199998, 0.31479120254499998, 0.31179499626200002, 0.30305600166300001, 0.30441617965700002, 0.30722284316999998, 0.301214933395, 0.30310297012300003, 0.301802873611, 0.311623096466, 0.31810212135299998, 0.31204795837400001, 0.312577962875, 0.31050682067899998, 0.31150794029200002, 0.31205916404700001, 0.30325889587400001, 0.30611991882299999, 0.30642008781399999, 0.301440000534, 0.304536104202, 0.30922794341999998, 0.313621044159, 0.31280899047900002, 0.31117606163, 0.311715126038, 0.30853891372699999, 0.31230807304399999, 0.30691790580700001, 0.30679011344899998, 0.30811786651599998, 0.301835775375, 0.30178809165999998, 0.30465817451499999, 0.31420707702599998, 0.313306093216, 0.31133508682299998, 0.31083703041100003], "base_times": [1.36353611946, 1.35942316055, 1.3685638904599999, 1.3592638969399999, 1.36338400841, 1.3641269207, 1.36004590988, 1.3620519638099999, 1.36313199997, 1.36007881165, 1.3650290966, 1.3646898269700001, 1.35792398453, 1.36864614487, 1.36348605156, 1.36114597321, 1.3651762008699999, 1.36486387253, 1.3583400249499999, 1.36733984947, 1.3646521568300001, 1.3632168769799999, 1.36906194687, 1.3584411144299999, 1.36989593506, 1.3611299991600001, 1.36594104767, 1.4054760932899999, 1.36255788803, 1.36518597603, 1.3638501167299999, 1.3670859336900001, 1.3645708560900001, 1.3667361736300001, 1.36373782158, 1.36704611778, 1.3673570156099999, 1.3680310249300001, 1.3642778396599999, 1.36619615555, 1.36799502373, 1.3654208183300001, 1.3667390346499999, 1.35933709145, 1.36695981026, 1.3632080554999999, 1.36811590195, 1.3632729053499999, 1.35716891289, 1.3655180931099999]}], ["go", "RawResult", {"changed_times": [0.205440998077, 0.237554788589, 0.178432941437, 0.22178697586099999, 0.15453481674200001, 0.184565067291, 0.181967973709, 0.20851802825900001, 0.16120600700400001, 0.12751483917199999, 0.126490831375, 0.21313405036899999, 0.13336992263799999, 0.13810706138600001, 0.17555379867599999, 0.117103099823, 0.120506048203, 0.124084234238, 0.12555694580099999, 0.13820600509600001, 0.15432786941500001, 0.12550187110899999, 0.12009692192099999, 0.126209020615, 0.120643138885, 0.12826299667400001, 0.125062942505, 0.14115309715300001, 0.112189769745, 0.12329006195099999, 0.119536876678, 0.115894079208, 0.13150215148899999, 0.120622158051, 0.133524894714, 0.10953402519200001, 0.110867977142, 0.120260953903, 0.128885030746, 0.11038398742699999, 0.113687992096, 0.105720996857, 0.110606908798, 0.106270074844, 0.12149906158400001, 0.146656036377, 0.118952989578, 0.108425855637, 0.126958847046, 0.116177082062], "base_times": [0.59508895874000001, 0.59899997711200004, 0.59387707710299997, 0.59602403640699997, 0.59553503990199996, 0.593573093414, 0.59433388710000001, 0.598137140274, 0.59456205368000004, 0.59513306617700001, 0.59438300132800004, 0.59516787529000004, 0.59493899345400003, 0.59641695022600005, 0.59443712234500001, 0.59466099739099998, 0.59525895118700001, 0.597331047058, 0.59680700302099998, 0.59574007988, 0.59577918052699996, 0.59496688842800005, 0.595326185226, 0.59544396400499999, 0.59744405746499996, 0.59654498100300002, 0.595510959625, 0.59514212608299999, 0.59548306465099998, 0.59657096862799996, 0.59839987754799995, 0.59662580490100003, 0.59663081169099996, 0.59636092186, 0.59646797180199995, 0.59581613540599998, 0.59876608848599999, 0.59532284736600005, 0.59693384170499997, 0.59681200981100002, 0.59692192077600004, 0.59601807594300005, 0.59875106811500001, 0.59596300125099999, 0.59473204612700004, 0.59547209739700002, 0.59548902511599999, 0.59676194190999998, 0.59593701362600005, 0.59788584709199999]}], ["html5lib", "RawResult", {"changed_times": [9.1950540542599999, 6.5574679374700002, 6.13121509552, 5.92117595673, 5.8948459625199998], "base_times": [8.6243629455599997, 8.6805830001800004, 8.6462841033900002, 8.6504099368999992, 8.6197869777699996]}], ["meteor-contest", "RawResult", {"changed_times": [0.32466602325400001, 0.28066992759699999, 0.27618789672900002, 0.27726387977599998, 0.27565312385599999, 0.27732300758400003, 0.283158063889, 0.27601695060699999, 0.27488493919399998, 0.27476692199699998, 0.274809122086, 0.27458906173699998, 0.28050112724300003, 0.27524900436400002, 0.27467489242600002, 0.273777008057, 0.27420091628999999, 0.27381587028499998, 0.27388405799900001, 0.27448701858500002, 0.27409100532500003, 0.27384519576999999, 0.27392005920399998, 0.27450394630399999, 0.27359294891399999, 0.27404904365499999, 0.27379584312400002, 0.27414393425, 0.27374601364099999, 0.27368783950800002, 0.279189825058, 0.27381300926199997, 0.27382278442399999, 0.27362394332899997, 0.27394199371299999, 0.27363896369899998, 0.27471089363099999, 0.27363920211800002, 0.273921966553, 0.273852109909, 0.27355313301099998, 0.27437615394600001, 0.27359104156500003, 0.27364397049, 0.27369093895000002, 0.27492690086400001, 0.27363181114200003, 0.273891925812, 0.27367687225300003, 0.27414608001700003], "base_times": [0.24080085754399999, 0.24055409431499999, 0.24092507362400001, 0.240586042404, 0.24052286148099999, 0.24070692062400001, 0.24118590354899999, 0.24084687232999999, 0.240713119507, 0.24066591262799999, 0.24061107635500001, 0.24103999137900001, 0.240550041199, 0.24071502685500001, 0.240919113159, 0.24124288559000001, 0.24059915542599999, 0.24063205719, 0.240550994873, 0.240890026093, 0.24065899848899999, 0.24082207679699999, 0.24078011512799999, 0.241106033325, 0.240442991257, 0.24064207076999999, 0.24046015739400001, 0.24054288864100001, 0.240490198135, 0.24042892456100001, 0.24063515663099999, 0.24120903015100001, 0.24073195457499999, 0.240746021271, 0.24080610275299999, 0.240445137024, 0.241003990173, 0.24048995971699999, 0.240598201752, 0.24078392982499999, 0.24111199379000001, 0.24032306671100001, 0.24051117896999999, 0.24035692215000001, 0.24025416374200001, 0.239776849747, 0.24031782150299999, 0.24069786071800001, 0.24129700660699999, 0.241048812866]}], ["nbody_modified", "RawResult", {"changed_times": [0.088541984558100001, 0.072235107421899994, 0.068952083587599994, 0.068921089172400002, 0.069000959396399997, 0.068970203399700003, 0.069295883178700002, 0.069005012512199995, 0.069174051284800003, 0.068920135497999996, 0.068984031677200006, 0.068948984146099998, 0.068947076797500007, 0.068919181823699993, 0.068933963775600002, 0.069010972976700002, 0.0692219734192, 0.068889141082800004, 0.068933963775600002, 0.068947076797500007, 0.068988084793099994, 0.069360971450800002, 0.068981885909999999, 0.069058895111099999, 0.069154977798499995, 0.069015026092500001, 0.068929910659800003, 0.068998813629200004, 0.068931102752699994, 0.068953037262, 0.068956851959199997, 0.068850040435799995, 0.069129943847700007, 0.068831920623800003, 0.068848848342900004, 0.069072008132900001, 0.068994045257600001, 0.068930149078400005, 0.068916082382199997, 0.068859100341799998, 0.069105863571199994, 0.068810939788799999, 0.068866014480599994, 0.068874120712299994, 0.0688879489899, 0.068877935409500005, 0.068838119506799994, 0.068907022476199994, 0.069243192672700002, 0.069992065429700004], "base_times": [0.430072069168, 0.43174719810500001, 0.43013787269600001, 0.42945313453700001, 0.42914891242999997, 0.43041586875900001, 0.42992877960199999, 0.43064594268799999, 0.431170940399, 0.431824922562, 0.43092703819299999, 0.43053197860699999, 0.42869901657100001, 0.43168401718100002, 0.43105721473699998, 0.43112492561299998, 0.43205499649000001, 0.43006110191300001, 0.42961001396199999, 0.428696155548, 0.429335832596, 0.42900800704999997, 0.429177999496, 0.42947387695299999, 0.42899489402800001, 0.42581796646100001, 0.42579293251, 0.426750898361, 0.42619681358299999, 0.42690300941499998, 0.426344156265, 0.425684928894, 0.42617511749300002, 0.425367116928, 0.42452383041399999, 0.42517709732100001, 0.42424011230499997, 0.42376208305399998, 0.42331886291499998, 0.42367100715599998, 0.423776865005, 0.42354416847199999, 0.42360591888400001, 0.42415189742999998, 0.423532009125, 0.42389607429499998, 0.42379999160800003, 0.42320799827599997, 0.42313408851599998, 0.42342805862400001]}], ["pyflate-fast", "RawResult", {"changed_times": [1.0659608840899999, 1.05365204811, 1.06374502182, 1.0581080913500001, 1.06380200386, 1.0373589992500001, 1.05517578125, 1.0548541545900001, 1.07799100876, 1.0740821361499999, 1.0693590640999999, 1.04684495926, 1.0749399662000001, 1.07159519196, 1.0657529830900001, 1.0503039359999999, 1.0730760097500001, 1.06034016609, 1.06997489929, 1.05047512054, 1.0729389190700001, 1.06514906883, 1.0397839546200001, 1.0728981494900001, 1.0665550231900001, 1.0433421134900001, 1.07183599472, 1.0667510032700001, 1.0408699512499999, 1.0743548870099999, 1.07203793526, 1.0681369304699999, 1.0525979995700001, 1.0704171657599999, 1.0463750362399999, 1.07289981842, 1.0518350601199999, 1.06989216805, 1.044672966, 1.0753920078300001, 1.0531618595100001, 1.0723462104799999, 1.0446758270300001, 1.07318806648, 1.0539751052899999, 1.08029985428, 1.04665017128, 1.0725378990200001, 1.05208015442, 1.0732369422900001], "base_times": [2.0108699798599998, 2.0109808445000001, 2.0061230659499998, 2.0109241008800001, 2.0078430175799999, 2.0112669467900002, 2.0139169692999999, 2.0103061199200001, 2.00638103485, 2.01138401031, 2.0138700008399999, 2.0058629512800001, 2.0086250305200002, 2.0138199329400002, 2.0063400268599998, 2.0082821846000001, 2.0112380981400002, 2.0076570510899998, 2.0080878734600001, 2.0108051300000001, 2.00607514381, 2.0063180923499999, 2.0096981525399999, 2.0101280212399999, 2.0155758857700001, 2.0102498531299999, 2.0053129196200001, 2.0141990184799998, 2.0114769935600001, 2.0126900672899999, 2.0084600448600001, 2.01082491875, 2.0114600658400001, 2.0073368549300001, 2.0094199180599999, 2.00728797913, 2.0094258785200001, 2.0149018764500002, 2.0094230175000001, 2.0109641551999999, 2.01358914375, 2.01244997978, 2.0089750289900001, 2.0090889930700002, 2.0079500675199999, 2.0080699920699998, 2.01176691055, 2.0092239379899999, 2.0114538669600002, 2.0104010105099999]}], ["raytrace-simple", "RawResult", {"changed_times": [0.104208946228, 0.106266021729, 0.098669052123999998, 0.091032028198199999, 0.091249942779500007, 0.118623018265, 0.089195013046299995, 0.088937997818000006, 0.097711086273200007, 0.097254991531400006, 0.090123891830399996, 0.089380025863599999, 0.11132407188399999, 0.088888168335000003, 0.089291095733600001, 0.088446140289299993, 0.113281011581, 0.088557004928599997, 0.088536024093599994, 0.087917804717999995, 0.114279031754, 0.096406936645499999, 0.087327957153299995, 0.087787151336699995, 0.10245609283400001, 0.086941003799399993, 0.087326049804700004, 0.087269783020000005, 0.088474035263100007, 0.116209030151, 0.087454080581699997, 0.101906776428, 0.086088895797700002, 0.086314916610699996, 0.085654973983799995, 0.086269855499299997, 0.085838079452499993, 0.086305856704700007, 0.087749958038300005, 0.14508390426600001, 0.086570978164699997, 0.085652112960799998, 0.108924865723, 0.086509943008399995, 0.086097002029400002, 0.10036110878, 0.099938154220599998, 0.085299968719500005, 0.085878849029500007, 0.0852539539337], "base_times": [1.6054999828300001, 1.6075158119199999, 1.6068139076200001, 1.6222910880999999, 1.61292099953, 1.60913705826, 1.6109960079200001, 1.62233877182, 1.6038889884900001, 1.60813307762, 1.6408219337500001, 1.60450601578, 1.6084368228899999, 1.6058139801, 1.6087529659299999, 1.6044671535499999, 1.60560894012, 1.6072280406999999, 1.6039071083100001, 1.6084260940599999, 1.6079659462, 1.6057059764899999, 1.6081471443199999, 1.60494303703, 1.60676002502, 1.60675907135, 1.6066009998299999, 1.60767817497, 1.6090710163099999, 1.6052989959699999, 1.6084320545199999, 1.61109900475, 1.6051030158999999, 1.60873317719, 1.6393661499000001, 1.6057238578799999, 1.6092879772199999, 1.6042079925499999, 1.62158799171, 1.6056730747200001, 1.60550308228, 1.60018992424, 1.62533283234, 1.60493087769, 1.61037397385, 1.6037578582800001, 1.6093821525600001, 1.60478901863, 1.63921904564, 1.60729408264]}], ["richards", "RawResult", {"changed_times": [0.0228841304779, 0.0150561332703, 0.0134420394897, 0.013730049133300001, 0.0136449337006, 0.0136260986328, 0.0137310028076, 0.0131878852844, 0.0136740207672, 0.013035058975199999, 0.0134780406952, 0.0130591392517, 0.012836933135999999, 0.0136439800262, 0.0124011039734, 0.013193130493200001, 0.0125820636749, 0.012362003326400001, 0.016746997833300001, 0.0124850273132, 0.0123841762543, 0.012007951736499999, 0.012148857116699999, 0.011607885360699999, 0.0119562149048, 0.0117020606995, 0.0120451450348, 0.0118389129639, 0.011528015136699999, 0.012020111084, 0.011619091033900001, 0.012064933776900001, 0.0116050243378, 0.0118329524994, 0.012051820755, 0.011452913284300001, 0.0119879245758, 0.0114850997925, 0.0118889808655, 0.0117588043213, 0.0115189552307, 0.0119571685791, 0.0116050243378, 0.0119731426239, 0.0115461349487, 0.0121710300446, 0.011741161346400001, 0.011747121810899999, 0.0118410587311, 0.0113570690155], "base_times": [0.21546196937600001, 0.21623301506000001, 0.21627402305599999, 0.216693162918, 0.21707105636599999, 0.21611881256099999, 0.21635699272200001, 0.21637606620800001, 0.21546506881700001, 0.21522092819200001, 0.21543502807600001, 0.21676111221300001, 0.21627402305599999, 0.21640300750700001, 0.21558499336199999, 0.21569085121199999, 0.21543717384300001, 0.21592593193099999, 0.21752214431799999, 0.21556806564299999, 0.21569395065300001, 0.21604013443, 0.21652293205299999, 0.21636700630200001, 0.215793848038, 0.215296030045, 0.215675830841, 0.21583795547500001, 0.21543502807600001, 0.216253995895, 0.21705412864699999, 0.21611094474799999, 0.216734886169, 0.21633100509600001, 0.218207120895, 0.21751618385300001, 0.21754693985000001, 0.21495008468599999, 0.21610498428300001, 0.216495037079, 0.217024087906, 0.21569108963, 0.21534991264299999, 0.215554952621, 0.21538805961599999, 0.21663308143599999, 0.215176820755, 0.21531486511199999, 0.21609115600600001, 0.216979026794]}], ["spambayes", "RawResult", {"changed_times": [0.28751707076999999, 0.34429097175599999, 0.327184915543, 0.30877208709699999, 0.30719304084799998, 0.29175710678099998, 0.30372905731200001, 0.307572126389, 0.24438500404399999, 0.303647994995, 0.273859024048, 0.278746128082, 0.23960399627699999, 0.26048088073699999, 0.33833193778999998, 0.26831603050199998, 0.28914999961900001, 0.239518880844, 0.24775385856599999, 0.24561405181900001, 0.25214099884000002, 0.25770306587199998, 0.25043201446500002, 0.22554898262, 0.24968504905700001, 0.23150801658600001, 0.25834798812900001, 0.22896385192900001, 0.24762606620800001, 0.24714899063099999, 0.26752901077300001, 0.290445804596, 0.21951293945299999, 0.241242170334, 0.22318506240800001, 0.25629496574400001, 0.22942090034500001, 0.24148082733199999, 0.21564912795999999, 0.23730015754700001, 0.21533489227300001, 0.23749184608500001, 0.22082400322000001, 0.24251198768599999, 0.214025020599, 0.235796928406, 0.227761983871, 0.23491716384899999, 0.22989392280599999, 0.23521780967700001], "base_times": [0.19353604316699999, 0.19359993934600001, 0.19369316101100001, 0.19351387023899999, 0.193557977676, 0.19352483749400001, 0.19358801841699999, 0.19363498687700001, 0.19352602958699999, 0.193531990051, 0.193488121033, 0.193665027618, 0.19356894493099999, 0.19348406791700001, 0.19356298446699999, 0.19351005554199999, 0.19349193573000001, 0.19352388381999999, 0.193593978882, 0.193593978882, 0.19361186027499999, 0.193543195724, 0.19364786148099999, 0.19361186027499999, 0.193534135818, 0.19354987144499999, 0.19374704361, 0.193845033646, 0.19361901283300001, 0.19356989860500001, 0.19359111785899999, 0.19361281394999999, 0.19363498687700001, 0.19360589981099999, 0.193613052368, 0.193671941757, 0.19370293617199999, 0.193685054779, 0.19372391700700001, 0.193596124649, 0.193512916565, 0.193508148193, 0.193747997284, 0.193646907806, 0.193574905396, 0.19359993934600001, 0.193615198135, 0.19353985786399999, 0.19352793693500001, 0.19359207153300001]}], ["spectral-norm", "RawResult", {"changed_times": [0.078598976135299994, 0.036157846450800002, 0.035543203353900002, 0.035031080245999997, 0.033834934234600003, 0.0332310199738, 0.033292055129999998, 0.0338280200958, 0.033846855163599997, 0.033032894134500002, 0.032781124114999997, 0.032047986984300003, 0.031976938247700003, 0.031943082809400003, 0.031916141510000003, 0.032538175582900002, 0.032065868377700001, 0.032016992569000001, 0.0319721698761, 0.031663894653300002, 0.031998872757000002, 0.032014846801800001, 0.031949996948199999, 0.031929016113299999, 0.031960010528600001, 0.032012939453099999, 0.031993150711100003, 0.031938076019300002, 0.031931161880499999, 0.031976938247700003, 0.032008171081500003, 0.031945943832400001, 0.031930923461899997, 0.031975030899000001, 0.032008886337300001, 0.031981945037799997, 0.031640052795399998, 0.031987905502299997, 0.031998872757000002, 0.032404184341400001, 0.031841039657599997, 0.031826019287100001, 0.0318641662598, 0.031888961792000001, 0.0318200588226, 0.031824111938500002, 0.031880140304599999, 0.031901121139499999, 0.031842947006200002, 0.032522916793799997], "base_times": [0.38017416000400001, 0.37997889518700001, 0.38001012802099998, 0.38086199760400002, 0.38064312934900002, 0.38048291206399998, 0.38021111488300002, 0.38219904899599999, 0.38127899169899998, 0.38254404067999997, 0.38078594207799998, 0.380933046341, 0.38103008270299998, 0.38071012496899997, 0.38104891777, 0.38050794601400001, 0.38025808334400002, 0.38140201568600002, 0.38133311271699999, 0.38049197197000001, 0.38048505783100001, 0.38087606430100002, 0.380648136139, 0.38163900375400001, 0.38113594055200001, 0.38124513626099998, 0.38089895248400002, 0.38078594207799998, 0.38048291206399998, 0.38178706169100002, 0.38335895538300002, 0.38262677192700001, 0.38049507141099997, 0.38140916824299997, 0.380529880524, 0.381772994995, 0.38075399398799997, 0.381628990173, 0.380851984024, 0.38191199302700002, 0.38043808937099999, 0.38140487670899997, 0.38037204742399999, 0.38166618347199999, 0.38078689575199998, 0.38159108161900002, 0.38048791885400002, 0.381910085678, 0.38064503669700001, 0.38182401657100001]}], ["telco", "RawResult", {"changed_times": [0.20000000000000001, 0.17999999999999999, 0.17000000000000001, 0.17000000000000001, 0.16, 0.17000000000000001, 0.16, 0.16, 0.16, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.16, 0.16, 0.16, 0.14999999999999999, 0.16, 0.16, 0.16, 0.14999999999999999, 0.16, 0.16, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999], "base_times": [0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.78000000000000003, 0.79000000000000004, 0.78000000000000003, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.78000000000000003, 0.79000000000000004]}], ["twisted_names", "RawResult", {"changed_times": [0.0043215211754537601, 0.0047438330170777986, 0.0043215211754537601, 0.0043421623968736434, 0.0043365134431916736, 0.0043763676148796497, 0.0042936882782310002, 0.0042789901583226361, 0.0046641791044776115, 0.0042716787697565147, 0.0043859649122807015, 0.0042735042735042739, 0.0042992261392949269, 0.0042863266180882984, 0.0042955326460481103, 0.0046860356138706651, 0.0043047783039173483, 0.004269854824935952, 0.0042936882782310002, 0.0042826552462526769, 0.0043103448275862068, 0.0042844901456726651, 0.0047169811320754715, 0.004268032437046522, 0.0042936882782310002, 0.0043029259896729772, 0.0042808219178082189, 0.0042808219178082189, 0.0042863266180882984, 0.0047103155911446069, 0.0042753313381787093, 0.0043122035360068992, 0.0042716787697565147, 0.0042973785990545769, 0.0042918454935622317, 0.0043122035360068992, 0.0046728971962616819, 0.0042863266180882984, 0.0042789901583226361, 0.0042973785990545769, 0.0043084877208099956, 0.0043047783039173483, 0.0042771599657827203, 0.0043066322136089581, 0.0047551117451260106, 0.0042918454935622317, 0.0042844901456726651, 0.0042973785990545769, 0.0042973785990545769, 0.0043066322136089581], "base_times": [0.0066225165562913907, 0.0066401062416998674, 0.0066401062416998674, 0.0066137566137566134, 0.006648936170212766, 0.0066445182724252493, 0.0066401062416998674, 0.0066533599467731202, 0.0066357000663570011, 0.006648936170212766, 0.0066401062416998674, 0.0066225165562913907, 0.0066401062416998674, 0.0066357000663570011, 0.0066312997347480109, 0.0066312997347480109, 0.0066181336863004635, 0.0066533599467731202, 0.0066181336863004635, 0.0066312997347480109, 0.0066137566137566134, 0.0066357000663570011, 0.0066225165562913907, 0.006648936170212766, 0.0066312997347480109, 0.0066577896138482022, 0.0066577896138482022, 0.0066445182724252493, 0.0066401062416998674, 0.006648936170212766, 0.0066269052352551355, 0.0066445182724252493, 0.0066533599467731202, 0.0066533599467731202, 0.0066533599467731202, 0.0066225165562913907, 0.0066401062416998674, 0.0066269052352551355, 0.0066269052352551355, 0.0066225165562913907, 0.0066445182724252493, 0.0066269052352551355, 0.0066401062416998674, 0.0066137566137566134, 0.0066445182724252493, 0.0066357000663570011, 0.0066357000663570011, 0.0066312997347480109, 0.0066357000663570011, 0.0066137566137566134]}]], "branch": "trunk", "revision": 0} \ No newline at end of file Added: pypy/extradoc/talk/pepm2011/benchmarks/jit.json ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pepm2011/benchmarks/jit.json Thu Oct 7 19:16:35 2010 @@ -0,0 +1 @@ +{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [0.48020195961000001, 0.19177389144900001, 0.133413076401, 0.13081407547000001, 0.12383103370699999, 0.123955965042, 0.12475705146799999, 0.12654089927699999, 0.123665809631, 0.1266310215, 0.12526297569299999, 0.124362945557, 0.12684011459399999, 0.12645721435500001, 0.12541103363, 0.12523484230000001, 0.124261856079, 0.124444961548, 0.12425708770799999, 0.124205112457, 0.12442111969, 0.12429785728499999, 0.12426400184600001, 0.12444806098900001, 0.124320030212, 0.127698898315, 0.124117136002, 0.12430500984200001, 0.124053001404, 0.124068021774, 0.124196052551, 0.12391710281399999, 0.12424707412700001, 0.123913049698, 0.12397909164400001, 0.124161958694, 0.12401294708299999, 0.124279022217, 0.12537884712200001, 0.124326944351, 0.17011690139800001, 0.123434066772, 0.123543977737, 0.12373208999599999, 0.123440980911, 0.123541116714, 0.123726844788, 0.12340497970600001, 0.123631000519, 0.12350797653200001], "base_times": [1.9571931362199999, 1.95759606361, 1.9574511051200001, 1.9571840763099999, 1.95754814148, 1.9592499732999999, 1.9597191810600001, 1.9576890468599999, 1.9574840068799999, 1.95859718323, 1.95902204514, 1.9580540657000001, 1.9581351280199999, 1.95984506607, 1.96052789688, 1.9583718776700001, 1.9600341320000001, 1.95734000206, 1.95849299431, 1.95821881294, 1.95806193352, 1.9597840309100001, 1.9591929912599999, 1.9574480056800001, 1.95800209045, 1.9581880569500001, 1.9591271877300001, 1.95778489113, 1.9588408470200001, 1.9598760604900001, 1.9594690799700001, 1.95824718475, 1.9581561088599999, 1.95844483376, 1.95841479301, 1.9575669765499999, 1.9578621387499999, 1.96030592918, 1.95885181427, 1.9577150344800001, 1.95695900917, 1.9573471546200001, 1.9587650299099999, 1.95756316185, 1.95798206329, 1.9601619243599999, 1.9590578079200001, 1.9581789970400001, 1.9575610160800001, 1.95715308189]}], ["django", "RawResult", {"changed_times": [0.149696826935, 0.14009809494, 0.117888212204, 0.121464014053, 0.12390708923300001, 0.116374969482, 0.116333961487, 0.11612391471899999, 0.116411924362, 0.116921901703, 0.116079092026, 0.11645913124100001, 0.12370395660400001, 0.116305828094, 0.116414070129, 0.11612296104399999, 0.11648201942399999, 0.11655402183499999, 0.11624312400800001, 0.116420030594, 0.123518943787, 0.116469860077, 0.116286993027, 0.116377830505, 0.116430997849, 0.116078853607, 0.116888046265, 0.11647796630899999, 0.12364697456400001, 0.116173028946, 0.116594076157, 0.116428852081, 0.11617898941, 0.11648607254, 0.116338014603, 0.11632394790599999, 0.123817920685, 0.116386175156, 0.11617207527200001, 0.11643409729, 0.116636991501, 0.116120100021, 0.116336107254, 0.116801977158, 0.123849868774, 0.116411924362, 0.116158008575, 0.116348981857, 0.116613864899, 0.11603999137899999], "base_times": [0.62960505485499996, 0.63032293319699995, 0.63513493537900001, 0.63052201271099995, 0.63038206100500005, 0.63098597526599998, 0.63484621048000001, 0.63156604766799995, 0.63161015510600005, 0.63133311271699999, 0.63094496727000005, 0.63588094711300003, 0.63120508194000002, 0.63055896758999996, 0.630630016327, 0.63221502304099997, 0.634573936462, 0.63075399398800003, 0.63052201271099995, 0.63049912452699997, 0.63090705871599995, 0.63108897209199999, 0.63039994239800001, 0.631347179413, 0.63096904754600003, 0.63091802597000002, 0.63096284866300001, 0.63120985031099996, 0.63101100921599995, 0.63157486915600003, 0.63075590133699999, 0.63098192215000004, 0.63011717796300004, 0.63079500198399996, 0.63095808029199996, 0.63124895095800004, 0.63111400604199996, 0.63150000572200005, 0.63068604469300005, 0.63113594055199995, 0.63090705871599995, 0.63139200210599999, 0.63083291053799995, 0.63064599037199998, 0.63013601303099998, 0.630333900452, 0.630806922913, 0.63129210472099995, 0.630800008774, 0.63049793243399999]}], ["fannkuch", "RawResult", {"changed_times": [0.34327197074900001, 0.30269885063200003, 0.29997181892399999, 0.29870891571000002, 0.299279928207, 0.30109310150099999, 0.29861617088300002, 0.299265146255, 0.300249099731, 0.30318808555600002, 0.29926800727800001, 0.298949956894, 0.29911518096900003, 0.30013704299900001, 0.29867911338800002, 0.299079179764, 0.29928493499800002, 0.29892182350199997, 0.29873299598699998, 0.30115199089099998, 0.298717975616, 0.29914808273299998, 0.29929184913599999, 0.29847502708399998, 0.29859995841999998, 0.29906201362599999, 0.29950094223000001, 0.29871892929100002, 0.298979043961, 0.30097007751499999, 0.29910802841200002, 0.29902291297900002, 0.29921507835400002, 0.29965305328399999, 0.29949402809100001, 0.29894018173199999, 0.29895210266099997, 0.29901099204999998, 0.298295974731, 0.30079817771900003, 0.29885911941499999, 0.298724889755, 0.298676013947, 0.29868197441099997, 0.29864406585699999, 0.298923969269, 0.29847002029399999, 0.298710107803, 0.29893589019799999, 0.30110883712800002], "base_times": [1.34415698051, 1.34473896027, 1.34213280678, 1.3438668251000001, 1.34420490265, 1.3415009975400001, 1.34282588959, 1.3425650596600001, 1.3436210155499999, 1.3436379432700001, 1.34295296669, 1.3431060314200001, 1.34196400642, 1.3424429893500001, 1.3429310321800001, 1.3434791564899999, 1.34407401085, 1.34443879128, 1.34361886978, 1.34351491928, 1.3432559967, 1.3444168567699999, 1.34266614914, 1.34265899658, 1.3439078331000001, 1.3449587822, 1.3445298671699999, 1.3426568508100001, 1.34537291527, 1.3459160327899999, 1.34588217735, 1.3462600708000001, 1.3434360027300001, 1.34325885773, 1.3433270454399999, 1.3445510864300001, 1.34478712082, 1.34340190887, 1.3453948497799999, 1.34363102913, 1.34446120262, 1.3437249660499999, 1.3436331748999999, 1.3450920581800001, 1.3444199562100001, 1.3433120250699999, 1.34379196167, 1.3430230617500001, 1.3436729908, 1.3451089859000001]}], ["go", "RawResult", {"changed_times": [0.224239826202, 0.21877193450900001, 0.20602703094499999, 0.252297878265, 0.15753602981600001, 0.19599390029899999, 0.17843794822699999, 0.20400905609100001, 0.15951395034800001, 0.15446615219099999, 0.14018702507, 0.21611905098, 0.140434980392, 0.14375901222199999, 0.18555498123200001, 0.12746500968899999, 0.13911294937099999, 0.12312197685200001, 0.13410091400099999, 0.136462926865, 0.15562582016000001, 0.12946891784699999, 0.124801874161, 0.13865089416500001, 0.124181985855, 0.127347230911, 0.12640810012799999, 0.14390397071800001, 0.121196985245, 0.12911605835000001, 0.131773948669, 0.12290096283, 0.12304592132599999, 0.12663388252300001, 0.13421177864100001, 0.13060092926, 0.115944862366, 0.12503790855399999, 0.115636825562, 0.12318682670599999, 0.113229990005, 0.120801925659, 0.112004995346, 0.111564874649, 0.115761041641, 0.15201306343099999, 0.12047719955400001, 0.121768951416, 0.116775035858, 0.11314201355], "base_times": [0.59340786933900003, 0.59497404098499995, 0.59325313568100002, 0.59470295906100001, 0.59381413459800003, 0.59332704544100001, 0.59409999847399997, 0.59631490707400003, 0.59585690498400001, 0.59371495246899997, 0.59403300285299998, 0.59452080726599998, 0.59437489509600006, 0.59984397888200003, 0.59600520134000001, 0.59333491325400001, 0.59451007843000003, 0.59631609916700001, 0.59571695327800001, 0.59719800949099999, 0.59574198722799998, 0.59356403350800002, 0.59414792060900001, 0.59434509277299996, 0.59581589698799997, 0.59776782989499999, 0.59241199493399999, 0.59377503395099995, 0.59570288658100001, 0.59592103958099996, 0.59745407104500003, 0.59661006927500004, 0.59465694427500004, 0.59411096572900002, 0.59395599365200002, 0.59408688545199995, 0.59843611717199996, 0.59402394294700001, 0.59620714187599999, 0.59329795837400001, 0.59584498405499997, 0.594747781754, 0.596575975418, 0.59466886520399997, 0.59498500824, 0.59656000137300003, 0.59625005722000002, 0.59529089927699996, 0.59529781341599997, 0.59693694114700002]}], ["html5lib", "RawResult", {"changed_times": [9.3101389408100008, 6.4080410003699999, 6.1054382324200001, 5.8417370319400002, 5.8308629989599998], "base_times": [8.6479589939100006, 8.7085971832299993, 8.7038509845699998, 8.6725029945400003, 8.6604359149899999]}], ["meteor-contest", "RawResult", {"changed_times": [0.32405900955200001, 0.27959895134000001, 0.275505065918, 0.27588295936599999, 0.27474308013900001, 0.27513694763199997, 0.28404498100300002, 0.27524399757399998, 0.27384400367700001, 0.273715019226, 0.274244070053, 0.27365398407000002, 0.27937602996799998, 0.27393889427200002, 0.27410411834699999, 0.27271318435699998, 0.27277207374599999, 0.27284193038900001, 0.27275490760799997, 0.27270412445100001, 0.27322816848800002, 0.27342605590800001, 0.27273511886599999, 0.27271890640300001, 0.27340292930600002, 0.27277398109399997, 0.27254891395600001, 0.27251195907600001, 0.272490024567, 0.272700786591, 0.27692794799800002, 0.27306580543499998, 0.27242612838699998, 0.27224206924400002, 0.27241110801700003, 0.27274608612099999, 0.27246308326699997, 0.27248406410199999, 0.27236604690600003, 0.27260303497299998, 0.27231597900400001, 0.27253699302700002, 0.27249503135699998, 0.27232098579399999, 0.27243399620100001, 0.27277398109399997, 0.27236318588300001, 0.272382974625, 0.27243304252599998, 0.27282595634500001], "base_times": [0.24173116684000001, 0.24125289917000001, 0.241533994675, 0.24160695075999999, 0.241540908813, 0.24194312095600001, 0.24139690399200001, 0.24161195755000001, 0.24145793914800001, 0.24141311645499999, 0.241906881332, 0.24192619323699999, 0.241802930832, 0.2417781353, 0.24143409729000001, 0.24142789840699999, 0.24163603782699999, 0.241791963577, 0.24179005622899999, 0.241909980774, 0.241487979889, 0.24162793159500001, 0.24160313606299999, 0.241613864899, 0.24137902259800001, 0.24134397506700001, 0.24174785613999999, 0.241899967194, 0.24181294441199999, 0.241572141647, 0.24126005172699999, 0.241393089294, 0.24105691909800001, 0.241027116776, 0.24095296859699999, 0.24140596389800001, 0.24158501625100001, 0.24153780937200001, 0.24153017997699999, 0.241586923599, 0.241651058197, 0.24133992195100001, 0.24141192436200001, 0.241475105286, 0.24171590805099999, 0.24158406257600001, 0.241488933563, 0.242015838623, 0.24174499511700001, 0.24191093444799999]}], ["nbody_modified", "RawResult", {"changed_times": [0.087947130203199997, 0.071905851364100007, 0.068665981292700004, 0.068542003631599999, 0.068524122238199994, 0.0686519145966, 0.068665027618400001, 0.068675041198699993, 0.068637132644700005, 0.068903923034699999, 0.068710088729900004, 0.068711996078499996, 0.068693161010699999, 0.068676948547400002, 0.068675041198699993, 0.068652868270900003, 0.068596839904800005, 0.068935155868500006, 0.068654060363800007, 0.068639039993299997, 0.068605899810799995, 0.068678855895999993, 0.068631887435899999, 0.068664073944099999, 0.068682193756100005, 0.068870067596400006, 0.068711042404200007, 0.068690061569200003, 0.068664073944099999, 0.068658113479600005, 0.068634033203100006, 0.068639993667599999, 0.068639993667599999, 0.068870067596400006, 0.068650007247900005, 0.068639039993299997, 0.068640947341900002, 0.068604946136500006, 0.0685949325562, 0.068629980087299994, 0.068645000457800004, 0.068935871124299997, 0.068669795990000004, 0.068640947341900002, 0.068603992462200003, 0.068616867065400003, 0.068652868270900003, 0.068677902221700005, 0.068669080734300003, 0.069911956787099994], "base_times": [0.39759898185699999, 0.39760899543799999, 0.39669609069799999, 0.39683794975300002, 0.39720010757399998, 0.39627695083600001, 0.39653801917999998, 0.39670896530200001, 0.39697194099400002, 0.396536827087, 0.39679622650099999, 0.396850824356, 0.397368907928, 0.39671897888199997, 0.39669322967499998, 0.39653301238999999, 0.39636301994299999, 0.39712595939599998, 0.39661812782299999, 0.396180868149, 0.396010875702, 0.39668297767600003, 0.39654588699299997, 0.39663290977499999, 0.39598798751800002, 0.39610004425000001, 0.39560699462900001, 0.39601302146900003, 0.39622092247000001, 0.39633798599199999, 0.39564895629899999, 0.39605998992899999, 0.39632201194799999, 0.39650416374199998, 0.39637398719799999, 0.39673495292700001, 0.39660906791700001, 0.39625310897799998, 0.39604401588400001, 0.395947933197, 0.39650201797500001, 0.39664816856399998, 0.39672684669500002, 0.39891719818100002, 0.39816594123799998, 0.39701795577999999, 0.39714002609299998, 0.39722919464099998, 0.39666795730600002, 0.39670610427899999]}], ["pyflate-fast", "RawResult", {"changed_times": [1.09018301964, 1.0680329799699999, 1.06872105598, 1.05886793137, 1.06418204308, 1.0598409175900001, 1.06196498871, 1.05568695068, 1.0567018985700001, 1.0555279254900001, 1.0644068717999999, 1.05560207367, 1.05402588844, 1.0528240203899999, 1.0537180900600001, 1.05323791504, 1.0666899681099999, 1.05483198166, 1.05929899216, 1.05668902397, 1.0558969974500001, 1.0555710792499999, 1.05419898033, 1.05392313004, 1.05447983742, 1.0538849830599999, 1.0565199852, 1.05395388603, 1.0541110038799999, 1.0599570274400001, 1.0534310340899999, 1.05349397659, 1.0561301708199999, 1.0588238239300001, 1.0534121990200001, 1.05571699142, 1.0535140037499999, 1.0534529686, 1.05349993706, 1.0726039409599999, 1.0234880447400001, 1.05639505386, 1.0563120842, 1.0539150237999999, 1.05793094635, 1.04236412048, 1.0372669696800001, 1.0381169319200001, 1.03758096695, 1.0407438278200001], "base_times": [1.99328494072, 1.9899220466600001, 1.9953949451399999, 1.9791898727399999, 1.9944601059, 1.9911510944399999, 1.9973587989799999, 1.9844009876299999, 1.9878590107, 1.98734712601, 1.99575614929, 1.9867370128599999, 1.99848389626, 1.9909040927899999, 1.99430203438, 1.98792386055, 1.99479794502, 1.9824438095100001, 1.9934871196699999, 1.9946010112799999, 1.98943686485, 1.98722910881, 1.9913620948799999, 1.9844691753399999, 1.9974501133, 1.98655295372, 1.99481105804, 1.9871459007300001, 1.9986989498100001, 1.99467992783, 1.99587082863, 1.97983598709, 1.9850268364000001, 1.9881370067599999, 1.9953911304500001, 1.98698186874, 1.9969110488899999, 1.98548102379, 1.9887509346000001, 1.98982906342, 1.99598193169, 1.9889919757800001, 1.9969968795799999, 1.9916138648999999, 1.99382400513, 1.98634600639, 2.0012769699100001, 1.9864809513099999, 1.9971618652300001, 1.9970350265500001]}], ["raytrace-simple", "RawResult", {"changed_times": [0.14111614227300001, 0.115634918213, 0.092776060104399996, 0.091562986373899993, 0.091269016265900005, 0.096694946289099998, 0.089217901229900004, 0.089500904083299998, 0.088824033737200006, 0.088817119598399996, 0.10380697250400001, 0.106505870819, 0.088323116302500004, 0.10829186439500001, 0.087399959564200005, 0.095005989074699995, 0.087472200393700003, 0.086698055267299998, 0.087571144104000007, 0.098669052123999998, 0.087127923965499995, 0.086586952209499995, 0.086830139160200007, 0.086702108383199999, 0.087288141250599999, 0.086775064468399998, 0.10691308975200001, 0.095185041427599995, 0.10678601265, 0.091362953186000004, 0.085355997085600002, 0.104825019836, 0.0857751369476, 0.085739135742200004, 0.085848093032799999, 0.133288145065, 0.085525035858199996, 0.085869073867800003, 0.085818052291899996, 0.085591077804599999, 0.085625886917100005, 0.115164995193, 0.088523864746099995, 0.0853309631348, 0.085108995437600005, 0.10862207412700001, 0.085693120956399998, 0.085079908371000004, 0.085324048996000004, 0.095507144927999998], "base_times": [1.5988879203799999, 1.5997819900500001, 1.59891486168, 1.6005818843799999, 1.6038961410499999, 1.6012160778, 1.61917686462, 1.60401916504, 1.59686493874, 1.5974218845399999, 1.5958590507499999, 1.5995800495100001, 1.6306281089800001, 1.6021780967699999, 1.6013729572299999, 1.6008999347699999, 1.5954790115399999, 1.59688901901, 1.5994551181800001, 1.59968185425, 1.6006410121900001, 1.5969679355599999, 1.6029269695299999, 1.5989317893999999, 1.6015889644600001, 1.5953040123, 1.5946440696699999, 1.5946199893999999, 1.60327291489, 1.5991640090899999, 1.59921884537, 1.60201787949, 1.5934991836500001, 1.59747695923, 1.5993280410799999, 1.59806799889, 1.59988999367, 1.6003260612500001, 1.5999028682700001, 1.6002368926999999, 1.5933301448799999, 1.6014490127600001, 1.5945661068000001, 1.6030778884900001, 1.59613704681, 1.60124015808, 1.6034739017499999, 1.5981848239900001, 1.59904718399, 1.5966200828599999]}], ["richards", "RawResult", {"changed_times": [0.0278170108795, 0.014379978179899999, 0.013342142105099999, 0.013777971267699999, 0.0137808322906, 0.0131430625916, 0.013561010360699999, 0.0186960697174, 0.0132620334625, 0.012887954711900001, 0.0124990940094, 0.0128400325775, 0.012853860855099999, 0.017416000366199998, 0.013220071792599999, 0.013627052307100001, 0.0134210586548, 0.0121581554413, 0.012850046157799999, 0.0116910934448, 0.011551141738899999, 0.012294054031399999, 0.011522054672200001, 0.0118951797485, 0.0120010375977, 0.0159740447998, 0.0115189552307, 0.011909008026099999, 0.011846065521199999, 0.0115439891815, 0.0119380950928, 0.0116410255432, 0.0119099617004, 0.011946916580199999, 0.0114579200745, 0.0118389129639, 0.011928081512499999, 0.011538982391400001, 0.0117189884186, 0.011837959289600001, 0.012223005294800001, 0.0117290019989, 0.0112850666046, 0.0120749473572, 0.011761903762799999, 0.0113418102264, 0.0117268562317, 0.0117020606995, 0.0116980075836, 0.011688232421900001], "base_times": [0.216315984726, 0.215498924255, 0.21641111373899999, 0.21590805053699999, 0.21580505371100001, 0.216100931168, 0.21709799766499999, 0.21659994125400001, 0.21590995788600001, 0.215686798096, 0.21582484245299999, 0.21613788604699999, 0.21666908264199999, 0.216024160385, 0.215727090836, 0.215723991394, 0.21619510650599999, 0.21609115600600001, 0.21665906906099999, 0.21585488319400001, 0.21662282943700001, 0.21619677543599999, 0.216505050659, 0.216939926147, 0.21652579307600001, 0.216482877731, 0.21655511856099999, 0.21607708931, 0.21637511253399999, 0.21669387817399999, 0.21724486351, 0.21636509895299999, 0.21614408493000001, 0.21600008010899999, 0.21747708320600001, 0.21744012832599999, 0.21655893325799999, 0.215548038483, 0.215684175491, 0.21656894683799999, 0.21586894988999999, 0.216165065765, 0.215878009796, 0.21631503105200001, 0.21671295166000001, 0.21809720993000001, 0.216701984406, 0.215636014938, 0.216233968735, 0.21654391288800001]}], ["spambayes", "RawResult", {"changed_times": [0.32702112197900002, 0.32608103752099998, 0.31628894805899999, 0.32889103889499999, 0.28262400627099998, 0.29191207885699999, 0.30786895751999999, 0.30573201179499998, 0.24465990066500001, 0.26622700691200002, 0.33094096183799998, 0.250310897827, 0.23908591270400001, 0.26244997978200002, 0.37832307815600003, 0.30173993110699998, 0.25720214843799999, 0.25671505927999999, 0.24676084518399999, 0.27264189720199999, 0.26233792304999998, 0.24248290062, 0.25211787223799997, 0.22474694251999999, 0.247483968735, 0.24680685996999999, 0.24286389350900001, 0.23023986816399999, 0.25418496131899998, 0.22227692604099999, 0.281915187836, 0.24955606460599999, 0.246785879135, 0.22405505180400001, 0.243336200714, 0.22702813148500001, 0.26623988151599998, 0.23486089706400001, 0.21349811554000001, 0.235665082932, 0.21503782272300001, 0.23597002029399999, 0.22992205619799999, 0.24228215217599999, 0.21251797676100001, 0.23505806922899999, 0.214460849762, 0.23505806922899999, 0.21343684196500001, 0.23480701446499999], "base_times": [0.19337677955599999, 0.19337487220800001, 0.19343996048000001, 0.19336819648699999, 0.193469047546, 0.19327688217200001, 0.193394184113, 0.193248033524, 0.19343686103800001, 0.19318485260000001, 0.19329404830899999, 0.19335818290699999, 0.193384885788, 0.193136930466, 0.19331407547000001, 0.193300008774, 0.19332504272500001, 0.193175077438, 0.19327211380000001, 0.193303823471, 0.19332504272500001, 0.19331717491100001, 0.19335103035000001, 0.19333100318900001, 0.19325113296499999, 0.19333791732799999, 0.19332194328300001, 0.19330096244799999, 0.19346404075599999, 0.19323682784999999, 0.19333004951499999, 0.193351984024, 0.193375825882, 0.193300008774, 0.19333696365399999, 0.19325494766199999, 0.19343209266700001, 0.19330406188999999, 0.193305969238, 0.193264007568, 0.193149089813, 0.193289995193, 0.19314718246500001, 0.19336700439499999, 0.193051099777, 0.19328284263600001, 0.193277835846, 0.19323611259500001, 0.19335293769799999, 0.19327807426499999]}], ["spectral-norm", "RawResult", {"changed_times": [0.074868917465199997, 0.033294916153000002, 0.0319759845734, 0.0320329666138, 0.0305089950562, 0.030241012573199999, 0.029862880706799998, 0.030753850936900001, 0.0307359695435, 0.029443025588999999, 0.029853820800799999, 0.0287258625031, 0.029007911682099999, 0.028622865676899999, 0.029003143310499999, 0.029177904128999999, 0.028678894042999999, 0.028953075408899999, 0.0286359786987, 0.029046058654799999, 0.028608798980700002, 0.028637886047399998, 0.029019117355299998, 0.028633117675799999, 0.029020071029700001, 0.0286231040955, 0.028687953948999999, 0.029042959213299999, 0.028628110885600001, 0.029020071029700001, 0.0286300182343, 0.028630971908599999, 0.029024124145499999, 0.0286509990692, 0.029007911682099999, 0.0286300182343, 0.028631925582900002, 0.029025077819799999, 0.0291199684143, 0.0289258956909, 0.028527975082400001, 0.0285370349884, 0.028898000717200001, 0.028551816940299998, 0.0289070606232, 0.028539896011400001, 0.028589010238599999, 0.028913974762, 0.028555154800399999, 0.029519081115700001], "base_times": [0.37576818466200002, 0.37547588348400002, 0.37559294700599999, 0.37567806243899998, 0.37582397460900002, 0.37562012672400003, 0.375561952591, 0.37594795227099997, 0.37573814392100002, 0.37600898742700001, 0.37583398819000002, 0.37612390518200001, 0.37607598304700002, 0.376045942307, 0.37554502487199998, 0.37556481361400001, 0.37557816505399999, 0.37564992904700001, 0.37574315071100001, 0.37603282928499998, 0.37587881088300001, 0.37620091438300002, 0.37593913078300001, 0.37606000900300002, 0.37613201141399999, 0.37632489204399999, 0.37581300735500001, 0.37594294548000001, 0.37562894821199999, 0.37567090988200003, 0.37568998336800002, 0.37568306922900002, 0.375794887543, 0.37602615356399999, 0.37595391273500001, 0.37603712081899998, 0.37618899345399998, 0.37632203102099998, 0.37599015235900002, 0.37571287155200001, 0.375794887543, 0.37555098533600001, 0.37568998336800002, 0.37585711479200001, 0.37600898742700001, 0.37587594986, 0.37576699256899998, 0.37570500373799998, 0.37578582763700003, 0.37625288963300002]}], ["telco", "RawResult", {"changed_times": [0.20999999999999999, 0.17000000000000001, 0.17000000000000001, 0.17999999999999999, 0.16, 0.17000000000000001, 0.16, 0.16, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999, 0.14999999999999999, 0.14999999999999999, 0.16, 0.16, 0.16, 0.14999999999999999, 0.14999999999999999, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.16, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16], "base_times": [0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.78000000000000003, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004]}], ["twisted_names", "RawResult", {"changed_times": [0.0043878894251864857, 0.0043478260869565218, 0.0046860356138706651, 0.0043421623968736434, 0.0044923629829290209, 0.0042863266180882984, 0.0042900042900042897, 0.0042918454935622317, 0.004340277777777778, 0.0046598322460391422, 0.0043159257660768235, 0.0042844901456726651, 0.0043084877208099956, 0.0042936882782310002, 0.0043103448275862068, 0.0043177892918825561, 0.0047147571900047151, 0.0043196544276457886, 0.0043159257660768235, 0.0043271311120726954, 0.0043196544276457886, 0.0043196544276457886, 0.0043215211754537601, 0.0043233895373973198, 0.0043177892918825561, 0.0043140638481449526, 0.0042955326460481103, 0.0043196544276457886, 0.0043196544276457886, 0.0043196544276457886, 0.0042955326460481103, 0.0047036688617121351, 0.0043047783039173483, 0.0043215211754537601, 0.0042992261392949269, 0.0042955326460481103, 0.0042955326460481103, 0.0043103448275862068, 0.0047192071731949033, 0.0043159257660768235, 0.0043177892918825561, 0.0043122035360068992, 0.0042955326460481103, 0.0043140638481449526, 0.0043215211754537601, 0.0047014574518100608, 0.0043271311120726954, 0.0043177892918825561, 0.004329004329004329, 0.0043103448275862068], "base_times": [0.0065919578114700065, 0.0066050198150594455, 0.0065832784726793945, 0.0066137566137566134, 0.0065832784726793945, 0.0066093853271645738, 0.006587615283267457, 0.0066225165562913907, 0.0066137566137566134, 0.0065919578114700065, 0.0066006600660066007, 0.0066181336863004635, 0.0065963060686015833, 0.0066181336863004635, 0.0065789473684210523, 0.0065832784726793945, 0.0065963060686015833, 0.0066093853271645738, 0.0065919578114700065, 0.0066050198150594455, 0.0066225165562913907, 0.006587615283267457, 0.0065919578114700065, 0.0065963060686015833, 0.0066225165562913907, 0.0066006600660066007, 0.0066137566137566134, 0.0066225165562913907, 0.0066006600660066007, 0.0066225165562913907, 0.0066137566137566134, 0.0066137566137566134, 0.0066006600660066007, 0.0066181336863004635, 0.0066050198150594455, 0.0066093853271645738, 0.0065919578114700065, 0.0066137566137566134, 0.0066225165562913907, 0.0066050198150594455, 0.0066006600660066007, 0.0066181336863004635, 0.0066093853271645738, 0.0066137566137566134, 0.0066006600660066007, 0.0065919578114700065, 0.0066093853271645738, 0.0066093853271645738, 0.0066050198150594455, 0.0066225165562913907]}]], "branch": "trunk", "revision": 0} \ No newline at end of file Added: pypy/extradoc/talk/pepm2011/benchmarks/nojit.json ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pepm2011/benchmarks/nojit.json Thu Oct 7 19:16:35 2010 @@ -0,0 +1 @@ +{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [3.24451899529, 3.2451541423800001, 3.2484319210099999, 3.24868011475, 3.2453150749200002, 3.2612857818599998, 3.24412417412, 3.2483599185899998, 3.2403240203900001, 3.24151682854, 3.2416470050799999, 3.2414860725399999, 3.2402641773199998, 3.2414939403499998, 3.2423250675199999, 3.2492020130200001, 3.24491500854, 3.24352884293, 3.2457530498499998, 3.2434449195899999, 3.2460479736300001, 3.2452580928799999, 3.2441511154199998, 3.2421209812199998, 3.2438759803799999, 3.24265193939, 3.24408507347, 3.2420589923900001, 3.2426180839500001, 3.2432379722600002, 3.24359798431, 3.2434740066500001, 3.24389314651, 3.2453119754799999, 3.24505710602, 3.24550294876, 3.2449901103999998, 3.2440829277000001, 3.2430200576799999, 3.2422559261299999, 3.2446641922000001, 3.2886731624599999, 3.24194502831, 3.2430789470699999, 3.24102115631, 3.24315214157, 3.2425389289900002, 3.2460539340999999, 3.2430410385099999, 3.24332499504], "base_times": [1.9742908477800001, 1.97254896164, 1.9728448391, 1.9736437797499999, 1.9723858833300001, 1.9721508026100001, 1.97404003143, 1.97118711472, 1.9723119735700001, 1.97428107262, 1.97056293488, 1.97442698479, 1.97402596474, 1.9740269184100001, 1.9724290370899999, 1.9739451408399999, 1.97252392769, 1.97287797928, 1.97392988205, 1.9726839065599999, 1.97153019905, 1.9746510982500001, 1.97324991226, 1.97370505333, 1.97335100174, 1.97502994537, 1.9734740257300001, 1.9735097885099999, 1.9727308750200001, 1.9733200073199999, 1.97521495819, 1.9747951030699999, 1.9745681285900001, 1.97405004501, 1.9743130207099999, 1.97423291206, 1.9736790657000001, 1.9736580848700001, 1.9753830432899999, 1.9741089344, 1.97190999985, 1.9735488891599999, 1.97480988503, 1.97467899323, 1.97327399254, 1.9745829105399999, 1.97255992889, 1.97384500504, 1.9736318588299999, 1.9745030403099999]}], ["django", "RawResult", {"changed_times": [1.01640892029, 1.0176229476900001, 1.0164048671699999, 1.0218229293800001, 1.0149660110500001, 1.01616621017, 1.0171029567700001, 1.01539587975, 1.01645708084, 1.0173921585100001, 1.01769113541, 1.0210890770000001, 1.01578497887, 1.01672196388, 1.01712584496, 1.01692390442, 1.0173480510699999, 1.0176861286200001, 1.0171391964000001, 1.0214610099799999, 1.0162498951000001, 1.01761293411, 1.01795005798, 1.0173869133, 1.0174238681800001, 1.0174930095700001, 1.0169968604999999, 1.0209729671500001, 1.0153799056999999, 1.01596999168, 1.0159599780999999, 1.01573491096, 1.01589202881, 1.0168061256400001, 1.0172579288500001, 1.0210061073300001, 1.0152461528800001, 1.0159599780999999, 1.0173289775800001, 1.01594591141, 1.0160360336300001, 1.0174479484600001, 1.0179150104500001, 1.0207760334, 1.0150790214500001, 1.0162980556500001, 1.0170168876600001, 1.0162210464500001, 1.0164031982399999, 1.01743793488], "base_times": [0.62385010719300005, 0.62486004829399999, 0.62562203407299999, 0.62786817550700003, 0.62438797950699998, 0.62284088134799998, 0.62462997436500001, 0.62638306617700001, 0.62503004074099999, 0.62424397468600001, 0.62518000602699997, 0.62529301643400004, 0.62483191490199996, 0.62550520896899997, 0.62556505203199997, 0.62616181373599999, 0.62509083747899996, 0.62512683868399999, 0.62532305717500003, 0.62479805946400002, 0.62612009048499995, 0.62651586532600001, 0.62502789497400002, 0.62546801567099997, 0.62566399574300002, 0.62599110603300001, 0.625466108322, 0.62400197982799999, 0.62511181831399998, 0.62484288215600003, 0.62472081184399997, 0.624977827072, 0.62426900863599999, 0.62370300293000003, 0.62426614761400001, 0.62449288368199996, 0.62413096427899994, 0.62434601783800003, 0.62429881095899997, 0.62487697601299996, 0.62503290176399995, 0.624408960342, 0.62468600273099995, 0.62470197677600003, 0.62522602081300005, 0.62469911575299997, 0.62464499473599999, 0.62524890899700003, 0.62433505058299998, 0.62387299537700003]}], ["fannkuch", "RawResult", {"changed_times": [1.7525610923799999, 1.75456190109, 1.7551848888399999, 1.7592298984500001, 1.7539329528800001, 1.7533161640199999, 1.7567901611300001, 1.7550489902499999, 1.7549738883999999, 1.75621986389, 1.77721691132, 1.75124502182, 1.75567388535, 1.7637329101599999, 1.75446391106, 1.7558588981600001, 1.76089882851, 1.75552797318, 1.7576568126700001, 1.75632786751, 1.7556400299099999, 1.7558760643, 1.75777602196, 1.7847819328300001, 1.7573959827400001, 1.7630050182300001, 1.75518584251, 1.7574501037600001, 1.7563450336499999, 1.7554650306699999, 1.7574830055199999, 1.76583909988, 1.7548341751100001, 1.75479102135, 1.7593448162100001, 1.75283789635, 1.7511570453600001, 1.75201010704, 1.76278305054, 1.75857520103, 1.7527630329099999, 1.75651788712, 1.7645468711900001, 1.75816082954, 1.75262498856, 1.7513329982800001, 1.7524199485800001, 1.75290083885, 1.75363206863, 1.75568318367], "base_times": [1.3310060501100001, 1.33170104027, 1.3308899402600001, 1.3305101394700001, 1.3316659927400001, 1.3297080993699999, 1.33244109154, 1.32962012291, 1.3284950256300001, 1.3292968273200001, 1.3300609588600001, 1.3295230865500001, 1.3292651176500001, 1.3311381339999999, 1.33014893532, 1.3291509151500001, 1.3284440040600001, 1.33054804802, 1.32914400101, 1.3303229808799999, 1.32920408249, 1.3317291736600001, 1.3322141170499999, 1.3311369419100001, 1.33129096031, 1.33123397827, 1.32876086235, 1.3293268680600001, 1.32887578011, 1.3325579166399999, 1.3323588371299999, 1.33169388771, 1.33165478706, 1.33138489723, 1.32928395271, 1.33160305023, 1.33113002777, 1.3318560123400001, 1.3319709301, 1.3326690197, 1.33210992813, 1.33002996445, 1.3301920890800001, 1.3290829658500001, 1.3313210010500001, 1.3313930034599999, 1.3320350647000001, 1.32845020294, 1.3319370746600001, 1.3316578865099999]}], ["go", "RawResult", {"changed_times": [1.9718310832999999, 1.9738159179699999, 1.97434592247, 1.9746530056, 1.97393703461, 1.9737930297899999, 1.9765768051099999, 1.97659111023, 1.98017787933, 1.97859406471, 1.9900450706499999, 1.9744200706499999, 1.97237300873, 1.9715647697400001, 1.9731240272499999, 1.97385907173, 1.9797570705400001, 1.9745059013399999, 1.9706268310499999, 1.98087596893, 1.97608804703, 1.9736139774300001, 1.97774386406, 1.9710638523099999, 1.97230505943, 1.97423100471, 1.98236918449, 1.9739501476300001, 1.9791419506100001, 1.97578310966, 1.9711699485800001, 1.9753818512000001, 1.9729008674599999, 1.97279191017, 1.9722950458499999, 1.97277402878, 1.97204303741, 1.97590994835, 1.97432184219, 1.97622799873, 1.9726431369799999, 1.9704701900499999, 1.9728059768699999, 1.9716351032299999, 1.97174906731, 1.97658991814, 1.97996807098, 1.97465586662, 1.97328782082, 1.98015093803], "base_times": [0.59540414810200004, 0.59541988372800003, 0.59281706810000001, 0.59443902969399998, 0.59351301193200001, 0.59394097328200002, 0.593358039856, 0.59580397605900004, 0.59437799453700002, 0.59357810020400004, 0.59425902366600003, 0.59401011466999998, 0.59433698654199996, 0.596014022827, 0.59163999557500002, 0.59366106987, 0.59340000152600003, 0.59544515609699999, 0.59593009948700004, 0.59467482566800001, 0.59426879882799999, 0.59387683868399999, 0.593913078308, 0.59401416778600002, 0.59669089317299995, 0.59634208679199996, 0.59397315978999998, 0.59430193901100004, 0.59379911422700005, 0.59470200538600004, 0.59703803062399996, 0.59560608863800002, 0.59434700012200004, 0.594178915024, 0.59457302093499997, 0.59599518775899996, 0.59800100326500005, 0.59486889839199997, 0.59505009651200003, 0.59450006484999995, 0.59540104865999999, 0.59393310546900002, 0.59670901298500001, 0.59417510032700005, 0.59537601470900003, 0.59527301788300002, 0.59467792510999995, 0.59671115875199998, 0.59430003166199996, 0.59509706497199999]}], ["html5lib", "RawResult", {"changed_times": [22.7969610691, 22.856777906400001, 22.805772066100001, 22.8616569042, 22.854540824899999], "base_times": [8.6267249584200005, 8.6838641166699997, 8.6542410850499998, 8.6537668705000002, 8.6361911296800002]}], ["meteor-contest", "RawResult", {"changed_times": [0.50923395156899998, 0.50941419601399995, 0.50952792167700001, 0.50948977470400003, 0.50936388969400004, 0.50924396514900006, 0.50919103622399997, 0.50925707817099997, 0.51003980636599999, 0.50929498672499995, 0.51118016242999997, 0.50945806503299995, 0.50993204116799995, 0.50920200347900002, 0.50964879989599998, 0.50924396514900006, 0.50979399681100002, 0.50919985771200005, 0.50959682464599998, 0.509632825851, 0.51002287864700002, 0.50988388061500001, 0.50983619689899995, 0.50931191444400004, 0.50982689857499996, 0.50927400589000005, 0.50935792923000001, 0.50939416885400002, 0.50975298881499997, 0.50955700874300003, 0.50961709022500001, 0.50914001464799996, 0.50987291336100005, 0.51021313667299995, 0.51024198532099996, 0.50933694839499999, 0.50978398323100005, 0.50937008857699995, 0.50964498519900003, 0.50951504707299999, 0.51006698608400003, 0.50996899604799995, 0.51156187057500002, 0.50929880142200001, 0.50978183746299999, 0.50929880142200001, 0.50956392288200003, 0.50975513458300004, 0.50990009307899997, 0.50932788848900001], "base_times": [0.240589857101, 0.24017000198399999, 0.24022793769799999, 0.24025583267200001, 0.240707874298, 0.240468978882, 0.24057006835899999, 0.24054598808300001, 0.24077296257, 0.240523099899, 0.24055290222199999, 0.24057507515000001, 0.240864992142, 0.24037003517200001, 0.240530967712, 0.240549087524, 0.240789890289, 0.24049592018099999, 0.24055194854699999, 0.24049782753000001, 0.24107503891000001, 0.240528821945, 0.24047994613599999, 0.24064207076999999, 0.24091005325299999, 0.240606069565, 0.24064397811900001, 0.240674972534, 0.240770816803, 0.24090790748599999, 0.240612030029, 0.24037218093900001, 0.24052786827100001, 0.240713119507, 0.240550041199, 0.24050211906399999, 0.240602970123, 0.24102997779800001, 0.240641117096, 0.240641117096, 0.24059486389199999, 0.240808010101, 0.24058890342700001, 0.240514993668, 0.240576028824, 0.240842819214, 0.240602970123, 0.24067902564999999, 0.24084615707400001, 0.24082899093599999]}], ["nbody_modified", "RawResult", {"changed_times": [0.38818812370299999, 0.38638401031500003, 0.38661384582500002, 0.38619303703300001, 0.39153003692600002, 0.38666009902999998, 0.38681101799000001, 0.38646793365499998, 0.38702797889700002, 0.38770484924300003, 0.38687992095899998, 0.38777208328200002, 0.38740205764800001, 0.386855840683, 0.38723683357200001, 0.38943696022000002, 0.38749217987099999, 0.38729500770600001, 0.38692498207100001, 0.38637304305999998, 0.38681602477999999, 0.38657712936400002, 0.38668608665499998, 0.38630890846299998, 0.38681292533900002, 0.386119127274, 0.38963198661800003, 0.38690185546900002, 0.38757920265200002, 0.38632512092600002, 0.38641405105600002, 0.38641500473000001, 0.387341022491, 0.386759996414, 0.38665986061099999, 0.38681292533900002, 0.386747837067, 0.38614678382899997, 0.38640689849900001, 0.38643479347199999, 0.38660693168600002, 0.38641810417200001, 0.38735198974599999, 0.38806200027499999, 0.38666510582000002, 0.38656806945799999, 0.38889193534900002, 0.38599085807799999, 0.39195680618299999, 0.38710403442399999], "base_times": [0.40266799926800001, 0.40421509742700001, 0.40469503402700002, 0.404855012894, 0.40383601188700002, 0.40301799774199998, 0.40401601791399999, 0.40404081344600001, 0.40401315689099998, 0.40385603904700001, 0.40401101112400001, 0.40403985977200002, 0.40431880950900001, 0.40080499649000001, 0.40367603301999999, 0.40390586852999999, 0.40407299995399998, 0.40394401550300002, 0.403142929077, 0.40314722061199998, 0.40221881866499998, 0.402665138245, 0.40284705162000001, 0.40243792533900002, 0.403349876404, 0.40271496772799997, 0.39837193489099998, 0.398493051529, 0.39839887618999997, 0.398682117462, 0.39855313301099998, 0.398659944534, 0.398689031601, 0.39893794059799997, 0.39879202842700001, 0.39923214912400001, 0.39903688430799999, 0.39881992340099998, 0.39869594574, 0.39842581749, 0.39839220047000001, 0.39844298362699998, 0.39846110343899999, 0.39831280708299999, 0.398363828659, 0.39867496490499998, 0.398545026779, 0.39788794517499998, 0.39805507659900002, 0.39805793762199998]}], ["pyflate-fast", "RawResult", {"changed_times": [3.7874670028700002, 3.7925579547899999, 3.7990520000500001, 3.7973430156700001, 3.7892198562599999, 3.8018770217900002, 3.7963681221000001, 3.7883229255700002, 3.8024299144699998, 3.8152718544000002, 3.7915089130399999, 3.8075730800600001, 3.7931261062599999, 3.78367614746, 3.7992160320299999, 3.7886638641400001, 3.7899460792499999, 3.8036901950800002, 3.79600214958, 3.79020214081, 3.80172896385, 3.7947978973400001, 3.7867920398699999, 3.8104150295300001, 3.80085086823, 3.79225111008, 3.80566692352, 3.7972359657300001, 3.7998919487, 3.8076539039599999, 3.7970049381300002, 3.7897589206700002, 3.7967648506199998, 3.7957010269200002, 3.7920849323299999, 3.8108398914300001, 3.7878198623700001, 3.7902488708500002, 3.8034110069299998, 3.7970860004400002, 3.7878341674799998, 3.8005399704, 3.7982490062699998, 3.7903809547399998, 3.80517292023, 3.7911689281499998, 3.79634594917, 3.8067698478700001, 3.78791093826, 3.79437303543], "base_times": [2.0019669532800002, 1.9968619346600001, 1.9961190223700001, 1.9985690116899999, 1.99683690071, 2.0048830509200002, 2.0027768611900001, 1.9973089694999999, 1.9973511695899999, 1.99325609207, 1.9935050010699999, 1.9935338497199999, 2.0064368248000002, 1.9978439807899999, 1.99749684334, 2.0095689296699999, 1.9977550506599999, 1.99649882317, 1.9990620613100001, 1.99493908882, 1.9962420463599999, 1.9990119934099999, 1.9993078708600001, 1.99604701996, 2.0018498897599999, 1.9966068267799999, 2.0015959739700002, 2.0015242099799999, 1.99887394905, 2.00690793991, 2.0039298534399999, 2.0020139217400001, 2.0005910396600002, 2.0046620369000001, 1.9950830936399999, 1.9967849254600001, 2.0024230480199998, 1.9941620826699999, 1.9989881515500001, 1.99789905548, 1.99698281288, 1.9983580112499999, 1.998306036, 2.0007560253099999, 2.0003230571700001, 1.9969279766100001, 1.99513196945, 1.9948849678, 1.9996719360399999, 1.9965598583199999]}], ["raytrace-simple", "RawResult", {"changed_times": [2.5259029865299998, 2.53055500984, 2.5200538635299998, 2.5136530399299999, 2.5256049633000002, 2.55257797241, 2.52219700813, 2.52172112465, 2.5238862037700001, 2.54926991463, 2.53663301468, 2.5551991462700001, 2.52897906303, 2.5599739551499998, 2.5537738800000001, 2.5579249858900002, 2.5481359958600001, 2.55539894104, 2.52555608749, 2.5228111743900001, 2.52320408821, 2.5257909297899999, 2.50518703461, 2.5582251548800001, 2.55362987518, 2.52637004852, 2.5557639598800002, 2.5231590271000002, 2.55590486526, 2.5615379810299999, 2.52569818497, 2.5533089637800002, 2.5285851955399998, 2.5255870819099999, 2.5263140201600001, 2.5171711444899998, 2.5098841190300001, 2.5515530109400002, 2.5186848640399999, 2.5520968437199998, 2.5592908859299999, 2.5509870052300001, 2.5601589679700001, 2.5387828350100001, 2.5581378936800001, 2.5617880821200001, 2.55112791061, 2.5512120723699998, 2.5230448245999999, 2.54989480972], "base_times": [1.6124339103700001, 1.60965681076, 1.6140599250800001, 1.6113469600699999, 1.6182990074200001, 1.6113588809999999, 1.61255812645, 1.6111528873400001, 1.61582493782, 1.6134300231900001, 1.60859704018, 1.6124861240399999, 1.61261200905, 1.6096360683399999, 1.61074209213, 1.6117038726799999, 1.60839509964, 1.6124470233899999, 1.6075179576900001, 1.60577487946, 1.6086390018500001, 1.6095190048200001, 1.60842394829, 1.60866212845, 1.60697889328, 1.61021304131, 1.6079950332599999, 1.6059949398, 1.6128342151599999, 1.6103730201699999, 1.616574049, 1.6216130256700001, 1.61437988281, 1.6048460006700001, 1.61348891258, 1.6078460216499999, 1.6112601757, 1.60963702202, 1.6116669178, 1.6085999012000001, 1.6102380752600001, 1.6069560051, 1.6085269451099999, 1.64675593376, 1.61165809631, 1.6143689155600001, 1.6131432056399999, 1.6108288765000001, 1.6096181869499999, 1.61089992523]}], ["richards", "RawResult", {"changed_times": [0.35733795165999999, 0.35815691947900002, 0.35844492912300002, 0.35738015174900001, 0.35734987258899997, 0.35710501670799999, 0.35719203949, 0.35757303237900001, 0.35736107826199998, 0.35771083831799999, 0.35805487632799998, 0.35751199722299998, 0.35809588432299999, 0.358512878418, 0.357872962952, 0.35806608200099999, 0.358750104904, 0.357810020447, 0.358080148697, 0.35869693756100002, 0.358117818832, 0.35854792594899998, 0.35819602012599999, 0.35885596275300002, 0.35915803909299998, 0.35868096351599998, 0.35919404029800001, 0.35971093177800001, 0.35929489135699999, 0.35918784141499999, 0.358865022659, 0.35824084281899998, 0.35879516601599998, 0.359260082245, 0.35835003852800001, 0.35905814170799999, 0.35927605628999998, 0.35836219787599999, 0.35935711860699998, 0.358397960663, 0.35834193229700001, 0.35866999626200002, 0.35795497894299999, 0.35880804061900001, 0.35883688926700003, 0.35859298706100001, 0.35823392867999998, 0.358649969101, 0.35842609405499998, 0.35812187194799999], "base_times": [0.21172690391499999, 0.21185398101799999, 0.211842060089, 0.211809158325, 0.212558031082, 0.212306976318, 0.21259713172899999, 0.21262598037700001, 0.212031126022, 0.211450099945, 0.21189093589800001, 0.21241617202800001, 0.21258687973000001, 0.212643861771, 0.21258902549700001, 0.21462988853500001, 0.213015079498, 0.21209907531700001, 0.21207809448199999, 0.21174597740199999, 0.21364402770999999, 0.21301007270799999, 0.212306976318, 0.21245503425600001, 0.21169185638400001, 0.21183514595, 0.212006092072, 0.21202707290600001, 0.21165895462000001, 0.21250700950599999, 0.2136759758, 0.212136983871, 0.21335101127600001, 0.213156938553, 0.21286082267799999, 0.21130895614600001, 0.212074995041, 0.21148419380200001, 0.211519002914, 0.21241998672500001, 0.21221494674700001, 0.212287902832, 0.212104082108, 0.212469100952, 0.21202492713900001, 0.212490797043, 0.21198487281799999, 0.21281194686900001, 0.21400499343900001, 0.21431016922000001]}], ["spambayes", "RawResult", {"changed_times": [0.50028204917899999, 0.48037195205700001, 0.48010993003800001, 0.49795198440600003, 0.48056793212900001, 0.48006296157799999, 0.49950885772699999, 0.48073792457600001, 0.480935811996, 0.497643947601, 0.47982192039499999, 0.48000788688700002, 0.50075793266299995, 0.48002386093100002, 0.48043394088699998, 0.497395992279, 0.479973077774, 0.48065590858500001, 0.49732089042700001, 0.48302197456399998, 0.48075509071400002, 0.49749898910500001, 0.47981595992999998, 0.47987699508699999, 0.49762320518499997, 0.483258008957, 0.48076701164199998, 0.49712514877300001, 0.47962999343899998, 0.48025083541899999, 0.49763703346299998, 0.48323106765700002, 0.47973179817200001, 0.49808692932100002, 0.480124950409, 0.48001098632799999, 0.497673988342, 0.48338508606000002, 0.48043513298000001, 0.49917507171600001, 0.479624986649, 0.48007702827499998, 0.498653173447, 0.483443021774, 0.480679988861, 0.49824213981600002, 0.47970604896500002, 0.479584932327, 0.498051166534, 0.48045110702499999], "base_times": [0.19211101531999999, 0.192034959793, 0.19223999977100001, 0.19230794906599999, 0.192923069, 0.19219708442700001, 0.19225502014199999, 0.19218802452100001, 0.19239997863800001, 0.192103147507, 0.192450046539, 0.19203400611900001, 0.19216108322100001, 0.192362070084, 0.19272994995100001, 0.19343900680500001, 0.19278812408400001, 0.19274902343799999, 0.19274783134500001, 0.19273900985699999, 0.19311118125900001, 0.192750930786, 0.192551136017, 0.19269800186200001, 0.19272208213799999, 0.193299055099, 0.19257998466500001, 0.19273304939300001, 0.19281101226799999, 0.192316055298, 0.192459106445, 0.192250967026, 0.19216394424399999, 0.192612171173, 0.19250798225400001, 0.192518949509, 0.19306898117099999, 0.192476034164, 0.19250583648700001, 0.19219017028800001, 0.19173789024400001, 0.192617893219, 0.192325115204, 0.19250607490499999, 0.19243192672699999, 0.19254207611099999, 0.192825078964, 0.19196105003399999, 0.19197201728800001, 0.19188404083300001]}], ["spectral-norm", "RawResult", {"changed_times": [0.77760505676299996, 0.77694511413599998, 0.78037905693099996, 0.77639794349699998, 0.77654695510899996, 0.77644205093399998, 0.77964901924100005, 0.77720713615399994, 0.77678012847900002, 0.77683305740399999, 0.78002309799199998, 0.77674102783200005, 0.77709603309599995, 0.78011703491200002, 0.77706813812300002, 0.77649903297400003, 0.77627110481299999, 0.78008413314799996, 0.77666282653800001, 0.77618098259000001, 0.77705812454199996, 0.77994203567499998, 0.77648878097499996, 0.77693796157799999, 0.77617907524100005, 0.78006911277799995, 0.77668690681499997, 0.77633309364299996, 0.77662515640300001, 0.77978491783100001, 0.77634692192099997, 0.77670788764999998, 0.77681207656899998, 0.78003311157199995, 0.77682781219499997, 0.77711296081500003, 0.77640914917000003, 0.77958297729500003, 0.77673602104200001, 0.77705597877499999, 0.77616095542899999, 0.78058195114100004, 0.77657890319800005, 0.77633595466600003, 0.780220985413, 0.77651286125200003, 0.77708983421300004, 0.77628302574200003, 0.77973103523300002, 0.77652502059899997], "base_times": [0.37738013267499998, 0.37694597244299999, 0.37648510932899998, 0.376368999481, 0.376587152481, 0.37885689735400002, 0.37677597999599999, 0.37695193290700002, 0.37683105468799999, 0.37667393684400002, 0.376702070236, 0.37671399116499998, 0.37673401832600001, 0.37877297401400001, 0.37689900398300002, 0.37723398208600001, 0.37698411941499999, 0.37705993652300002, 0.377000808716, 0.377035140991, 0.377371072769, 0.37960100174, 0.381766080856, 0.38150787353499999, 0.381573200226, 0.38162183761599999, 0.38187789916999998, 0.38156914711000001, 0.38257384300199998, 0.37775778770399998, 0.37862801551800002, 0.38168191909799998, 0.38173699378999998, 0.38146400451700002, 0.38132095336900002, 0.38183498382600001, 0.38331413268999998, 0.37714600563, 0.38064408302300001, 0.38145589828499998, 0.38117909431500002, 0.38102793693499998, 0.38102102279700001, 0.381139993668, 0.38252782821699999, 0.37691378593399999, 0.376620054245, 0.37686300277700002, 0.37692689895600001, 0.376924037933]}], ["telco", "RawResult", {"changed_times": [1.6799999999999999, 1.6799999999999999, 1.6699999999999999, 1.6799999999999999, 1.6799999999999999, 1.6699999999999999, 1.6799999999999999, 1.6799999999999999, 1.6699999999999999, 1.6699999999999999, 1.6799999999999999, 1.6699999999999999, 1.6799999999999999, 1.6699999999999999, 1.6699999999999999, 1.6799999999999999, 1.6799999999999999, 1.6699999999999999, 1.6699999999999999, 1.6799999999999999, 1.6699999999999999, 1.6699999999999999, 1.6799999999999999, 1.6699999999999999, 1.6799999999999999, 1.6699999999999999, 1.6799999999999999, 1.6699999999999999, 1.6799999999999999, 1.6699999999999999, 1.6799999999999999, 1.6799999999999999, 1.6699999999999999, 1.6799999999999999, 1.6699999999999999, 1.6699999999999999, 1.6799999999999999, 1.6699999999999999, 1.6699999999999999, 1.6799999999999999, 1.6799999999999999, 1.6699999999999999, 1.6699999999999999, 1.6699999999999999, 1.6699999999999999, 1.6699999999999999, 1.6799999999999999, 1.6699999999999999, 1.6799999999999999, 1.6799999999999999], "base_times": [0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.80000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.80000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.80000000000000004, 0.80000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.80000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.80000000000000004, 0.79000000000000004]}], ["twisted_names", "RawResult", {"changed_times": [0.014124293785310734, 0.014144271570014143, 0.014144271570014143, 0.014144271570014143, 0.014144271570014143, 0.014144271570014143, 0.014164305949008499, 0.014124293785310734, 0.015772870662460567, 0.014144271570014143, 0.014104372355430184, 0.014164305949008499, 0.014124293785310734, 0.014104372355430184, 0.014144271570014143, 0.014124293785310734, 0.014144271570014143, 0.014144271570014143, 0.014164305949008499, 0.014124293785310734, 0.014124293785310734, 0.014164305949008499, 0.014124293785310734, 0.014164305949008499, 0.014124293785310734, 0.014144271570014143, 0.014144271570014143, 0.014144271570014143, 0.014144271570014143, 0.014814814814814815, 0.014144271570014143, 0.014144271570014143, 0.014144271570014143, 0.014144271570014143, 0.014144271570014143, 0.014124293785310734, 0.014144271570014143, 0.014104372355430184, 0.014144271570014143, 0.014144271570014143, 0.014144271570014143, 0.014144271570014143, 0.014164305949008499, 0.014124293785310734, 0.014144271570014143, 0.014164305949008499, 0.014124293785310734, 0.014144271570014143, 0.014124293785310734, 0.014164305949008499], "base_times": [0.0066137566137566134, 0.0066357000663570011, 0.0066225165562913907, 0.0066711140760507001, 0.0066093853271645738, 0.0066357000663570011, 0.0066225165562913907, 0.0066577896138482022, 0.0066312997347480109, 0.0066666666666666671, 0.0066357000663570011, 0.0066225165562913907, 0.006648936170212766, 0.0066401062416998674, 0.0066181336863004635, 0.0066533599467731202, 0.0066269052352551355, 0.0066401062416998674, 0.0066269052352551355, 0.0066445182724252493, 0.0066312997347480109, 0.0066357000663570011, 0.0066445182724252493, 0.0066577896138482022, 0.0066622251832111927, 0.0066312997347480109, 0.0066312997347480109, 0.0066269052352551355, 0.006648936170212766, 0.0066181336863004635, 0.006648936170212766, 0.006648936170212766, 0.0066269052352551355, 0.0066445182724252493, 0.0066312997347480109, 0.0066312997347480109, 0.006648936170212766, 0.0066445182724252493, 0.0066269052352551355, 0.006648936170212766, 0.0066269052352551355, 0.0066533599467731202, 0.0066225165562913907, 0.0066312997347480109, 0.0066312997347480109, 0.0066533599467731202, 0.0066533599467731202, 0.0066312997347480109, 0.0066533599467731202, 0.0066445182724252493]}]], "branch": "trunk", "revision": 0} \ No newline at end of file From cfbolz at codespeak.net Thu Oct 7 19:17:10 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Thu, 7 Oct 2010 19:17:10 +0200 (CEST) Subject: [pypy-svn] r77701 - pypy/extradoc/talk/pepm2011/benchmarks Message-ID: <20101007171710.5D05A282B9E@codespeak.net> Author: cfbolz Date: Thu Oct 7 19:17:09 2010 New Revision: 77701 Added: pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric (contents, props changed) Log: the Gnumeric file I use to compute confidence intervals etc Added: pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric ============================================================================== Binary file. No diff available. From afa at codespeak.net Thu Oct 7 23:52:35 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Thu, 7 Oct 2010 23:52:35 +0200 (CEST) Subject: [pypy-svn] r77702 - in pypy/branch/fast-forward/pypy/objspace/std: . test Message-ID: <20101007215235.7799E282BD6@codespeak.net> Author: afa Date: Thu Oct 7 23:52:32 2010 New Revision: 77702 Modified: pypy/branch/fast-forward/pypy/objspace/std/complexobject.py pypy/branch/fast-forward/pypy/objspace/std/complextype.py pypy/branch/fast-forward/pypy/objspace/std/test/test_complexobject.py Log: Be more careful to preserve the sign of zero when building a complex. Also fix the repr() to display the sign of zero in both components. Modified: pypy/branch/fast-forward/pypy/objspace/std/complexobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/complexobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/complexobject.py Thu Oct 7 23:52:32 2010 @@ -3,6 +3,7 @@ from pypy.objspace.std.model import registerimplementation, W_Object from pypy.objspace.std.register_all import register_all from pypy.objspace.std.floatobject import W_FloatObject, _hash_float +from pypy.rlib.rarithmetic import formatd, isinf, isnan import math @@ -254,33 +255,37 @@ #w_imag = space.call_function(space.w_float,space.wrap(-w_self.imagval)) return space.newcomplex(w_self.realval,-w_self.imagval) -app = gateway.applevel(""" - import math - import sys - def possint(f): - ff = math.floor(f) - if f == ff and abs(f) < sys.maxint: - return int(ff) - return f - - def repr__Complex(f): - if not f.real: - return repr(possint(f.imag))+'j' - imag = f.imag - sign = ((imag >= 0) and '+') or '' - return '('+repr(possint(f.real)) + sign + repr(possint(f.imag))+'j)' - - def str__Complex(f): - if not f.real: - return str(possint(f.imag))+'j' - imag = f.imag - sign = ((imag >= 0) and '+') or '' - return '('+str(possint(f.real)) + sign + str(possint(f.imag))+'j)' - -""", filename=__file__) +def format_float(x, format): + # like float2string, except that the ".0" is not necessary + if isinf(x): + if x > 0.0: + return "inf" + else: + return "-inf" + elif isnan(x): + return "nan" + else: + return formatd(format, x) -repr__Complex = app.interphook('repr__Complex') -str__Complex = app.interphook('str__Complex') +def repr_format(x): + return format_float(x, "%.17g") +def str_format(x): + return format_float(x, "%.12g") + +def repr__Complex(space, w_complex): + print "AFA REPR", (w_complex, w_complex.realval, w_complex.imagval) + if w_complex.realval == 0 and math.copysign(1., w_complex.realval) == 1.: + return space.wrap(repr_format(w_complex.imagval) + 'j') + sign = (math.copysign(1., w_complex.imagval) == 1.) and '+' or '' + return space.wrap('(' + repr_format(w_complex.realval) + + sign + repr_format(w_complex.imagval) + 'j)') + +def str__Complex(space, w_complex): + if w_complex.realval == 0 and math.copysign(1., w_complex.realval) == 1.: + return space.wrap(str_format(w_complex.imagval) + 'j') + sign = (math.copysign(1., w_complex.imagval) == 1.) and '+' or '' + return space.wrap('(' + str_format(w_complex.realval) + + sign + str_format(w_complex.imagval) + 'j)') from pypy.objspace.std import complextype register_all(vars(), complextype) Modified: pypy/branch/fast-forward/pypy/objspace/std/complextype.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/complextype.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/complextype.py Thu Oct 7 23:52:32 2010 @@ -192,7 +192,11 @@ space.wrap("complex() second arg" " can't be a string")) else: - imagval += space.float_w(space.float(w_imag)) + a = space.float_w(space.float(w_imag)) + if imagval != 0.0: + imagval += a + else: + imagval = a # done w_obj = space.allocate_instance(W_ComplexObject, w_complextype) W_ComplexObject.__init__(w_obj, realval, imagval) Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_complexobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/test/test_complexobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/test/test_complexobject.py Thu Oct 7 23:52:32 2010 @@ -321,12 +321,16 @@ raises(ValueError, complex, unicode("1"*500)) def test_repr(self): - h = self.helper - h.assertEqual(repr(1+6j), '(1+6j)') - h.assertEqual(repr(1-6j), '(1-6j)') + assert repr(1+6j) == '(1+6j)' + assert repr(1-6j) == '(1-6j)' - h.assertNotEqual(repr(-(1+0j)), '(-1+-0j)') + assert repr(-(1+0j)) == '(-1-0j)' + assert repr(complex( 0.0, 0.0)) == '0j' + assert repr(complex( 0.0, -0.0)) == '-0j' + assert repr(complex(-0.0, 0.0)) == '(-0+0j)' + assert repr(complex(-0.0, -0.0)) == '(-0-0j)' assert repr(complex(1e45)) == "(" + repr(1e45) + "+0j)" + assert repr(complex(1e200*1e200)) == '(inf+0j)' def test_neg(self): h = self.helper From afa at codespeak.net Thu Oct 7 23:54:48 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Thu, 7 Oct 2010 23:54:48 +0200 (CEST) Subject: [pypy-svn] r77703 - pypy/branch/fast-forward/pypy/objspace/std Message-ID: <20101007215448.57F7C282BD6@codespeak.net> Author: afa Date: Thu Oct 7 23:54:46 2010 New Revision: 77703 Modified: pypy/branch/fast-forward/pypy/objspace/std/complexobject.py Log: Remove debug print. Thanks Alex! Modified: pypy/branch/fast-forward/pypy/objspace/std/complexobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/complexobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/complexobject.py Thu Oct 7 23:54:46 2010 @@ -273,7 +273,6 @@ return format_float(x, "%.12g") def repr__Complex(space, w_complex): - print "AFA REPR", (w_complex, w_complex.realval, w_complex.imagval) if w_complex.realval == 0 and math.copysign(1., w_complex.realval) == 1.: return space.wrap(repr_format(w_complex.imagval) + 'j') sign = (math.copysign(1., w_complex.imagval) == 1.) and '+' or '' From afa at codespeak.net Fri Oct 8 00:48:32 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 00:48:32 +0200 (CEST) Subject: [pypy-svn] r77704 - pypy/branch/fast-forward/pypy/objspace/std/test Message-ID: <20101007224832.B2692282BD6@codespeak.net> Author: afa Date: Fri Oct 8 00:48:31 2010 New Revision: 77704 Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_complexobject.py Log: Add one test about the __complex__ special method Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_complexobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/test/test_complexobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/test/test_complexobject.py Fri Oct 8 00:48:31 2010 @@ -362,3 +362,24 @@ def test_convert(self): raises(TypeError, int, 1+1j) raises(TypeError, float, 1+1j) + + class complex0(complex): + """Test usage of __complex__() when inheriting from 'complex'""" + def __complex__(self): + return 42j + assert complex(complex0(1j)) == 42j + + class complex1(complex): + """Test usage of __complex__() with a __new__() method""" + def __new__(self, value=0j): + return complex.__new__(self, 2*value) + def __complex__(self): + return self + assert complex(complex1(1j)) == 2j + + class complex2(complex): + """Make sure that __complex__() calls fail if anything other than a + complex is returned""" + def __complex__(self): + return None + raises(TypeError, complex, complex2(1j)) From afa at codespeak.net Fri Oct 8 00:50:15 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 00:50:15 +0200 (CEST) Subject: [pypy-svn] r77705 - pypy/branch/fast-forward/lib_pypy Message-ID: <20101007225015.173C9282BD6@codespeak.net> Author: afa Date: Fri Oct 8 00:50:14 2010 New Revision: 77705 Modified: pypy/branch/fast-forward/lib_pypy/cmath.py Log: coerce the argument of cmath function to a complex object - accept a subclass - refuse strings - but accept all other types convertible to complex Should fix most tests in test_cmath Modified: pypy/branch/fast-forward/lib_pypy/cmath.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/cmath.py (original) +++ pypy/branch/fast-forward/lib_pypy/cmath.py Fri Oct 8 00:50:14 2010 @@ -16,28 +16,36 @@ _halfi = complex(0., 0.5) -# internal function not available from Python + +# internal functions not available from Python +def _to_complex(x): + if isinstance(x, complex): + return x + if isinstance(x, (str, unicode)): + return x # and let it fail later + return complex(x) + def _prodi(x): - x = complex(x, 0) + x = _to_complex(x) real = -x.imag imag = x.real return complex(real, imag) + def phase(x): - x = complex(x) + x = _to_complex(x) return math.atan2(x.imag, x.real) def polar(x): - x = complex(x) + x = _to_complex(x) phi = math.atan2(x.imag, x.real) r = abs(x) return r, phi def rect(r, phi): - r, phi = complex(r), complex(phi) return complex(r * math.cos(phi), r * math.sin(phi)) @@ -45,7 +53,8 @@ """acos(x) Return the arc cosine of x.""" - + + x = _to_complex(x) return -(_prodi(log((x+(_i*sqrt((_one-(x*x)))))))) @@ -54,6 +63,7 @@ Return the hyperbolic arccosine of x.""" + x = _to_complex(x) z = log(_sqrt_half*(sqrt(x+_one)+sqrt(x-_one))) return z+z @@ -62,7 +72,8 @@ """asin(x) Return the arc sine of x.""" - + + x = _to_complex(x) # -i * log[(sqrt(1-x**2) + i*x] squared = x*x sqrt_1_minus_x_sq = sqrt(_one-squared) @@ -73,7 +84,8 @@ """asinh(x) Return the hyperbolic arc sine of x.""" - + + x = _to_complex(x) z = log((_sqrt_half * (sqrt(x+_i)+sqrt((x-_i))) )) return z+z @@ -82,7 +94,8 @@ """atan(x) Return the arc tangent of x.""" - + + x = _to_complex(x) return _halfi*log(((_i+x)/(_i-x))) @@ -90,7 +103,8 @@ """atanh(x) Return the hyperbolic arc tangent of x.""" - + + x = _to_complex(x) return _half*log((_one+x)/(_one-x)) @@ -99,7 +113,7 @@ Return the cosine of x.""" - x = complex(x, 0) + x = _to_complex(x) real = math.cos(x.real) * math.cosh(x.imag) imag = -math.sin(x.real) * math.sinh(x.imag) return complex(real, imag) @@ -110,7 +124,7 @@ Return the hyperbolic cosine of x.""" - x = complex(x, 0) + x = _to_complex(x) real = math.cos(x.imag) * math.cosh(x.real) imag = math.sin(x.imag) * math.sinh(x.real) return complex(real, imag) @@ -121,7 +135,7 @@ Return the exponential value e**x.""" - x = complex(x, 0) + x = _to_complex(x) l = math.exp(x.real) real = l * math.cos(x.imag) imag = l * math.sin(x.imag) @@ -135,7 +149,7 @@ if base is not None: return log(x) / log(base) - x = complex(x, 0) + x = _to_complex(x) l = math.hypot(x.real,x.imag) imag = math.atan2(x.imag, x.real) real = math.log(l) @@ -147,7 +161,7 @@ Return the base-10 logarithm of x.""" - x = complex(x, 0) + x = _to_complex(x) l = math.hypot(x.real, x.imag) imag = math.atan2(x.imag, x.real)/math.log(10.) real = math.log10(l) @@ -159,7 +173,7 @@ Return the sine of x.""" - x = complex(x, 0) + x = _to_complex(x) real = math.sin(x.real) * math.cosh(x.imag) imag = math.cos(x.real) * math.sinh(x.imag) return complex(real, imag) @@ -170,7 +184,7 @@ Return the hyperbolic sine of x.""" - x = complex(x, 0) + x = _to_complex(x) real = math.cos(x.imag) * math.sinh(x.real) imag = math.sin(x.imag) * math.cosh(x.real) return complex(real, imag) @@ -181,7 +195,7 @@ Return the square root of x.""" - x = complex(x, 0) + x = _to_complex(x) if x.real == 0. and x.imag == 0.: real, imag = 0, 0 else: @@ -206,7 +220,7 @@ Return the tangent of x.""" - x = complex(x, 0) + x = _to_complex(x) sr = math.sin(x.real) cr = math.cos(x.real) shi = math.sinh(x.imag) @@ -226,7 +240,7 @@ Return the hyperbolic tangent of x.""" - x = complex(x, 0) + x = _to_complex(x) si = math.sin(x.imag) ci = math.cos(x.imag) shr = math.sinh(x.real) @@ -239,3 +253,15 @@ real = (rs*rc + is_*ic) / d imag = (is_*rc - rs*ic) / d return complex(real, imag) + +def isnan(x): + """isnan(z) -> bool + Checks if the real or imaginary part of z not a number (NaN)""" + x = _to_complex(x) + return math.isnan(x.real) or math.isnan(x.imag) + +def isinf(x): + """isnan(z) -> bool + Checks if the real or imaginary part of z is infinite""" + x = _to_complex(x) + return math.isinf(x.real) or math.isinf(x.imag) From hakanardo at codespeak.net Fri Oct 8 07:36:04 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Fri, 8 Oct 2010 07:36:04 +0200 (CEST) Subject: [pypy-svn] r77706 - in pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test Message-ID: <20101008053604.6C024282BE9@codespeak.net> Author: hakanardo Date: Fri Oct 8 07:36:02 2010 New Revision: 77706 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Log: Fixed fail_args Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Fri Oct 8 07:36:02 2010 @@ -70,7 +70,7 @@ a = argmap[a] except KeyError: pass - args.append(a) + args.append(self.getvalue(a).force_box()) op.setfailargs(args) op.getdescr().rd_snapshot = None #FIXME: In the right place? Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Fri Oct 8 07:36:02 2010 @@ -392,9 +392,9 @@ res = self.meta_interp(f, [6, 16]) assert res == 828 self.check_loop_count(3) - self.check_loops({'int_lt': 1, 'int_gt': 1, - 'guard_false': 1, 'guard_true': 1, - 'int_sub': 2, 'int_mul': 2, 'int_add': 2, + self.check_loops({'int_lt': 2, 'int_gt': 1, + 'guard_false': 2, 'guard_true': 1, + 'int_sub': 3, 'int_mul': 1, 'int_add': 3, 'jump': 3}) def test_loop_invariant_intbox(self): From afa at codespeak.net Fri Oct 8 09:02:19 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 09:02:19 +0200 (CEST) Subject: [pypy-svn] r77707 - in pypy/branch/fast-forward/pypy/module/_io: . test Message-ID: <20101008070219.0BFC8282BE8@codespeak.net> Author: afa Date: Fri Oct 8 09:02:16 2010 New Revision: 77707 Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py pypy/branch/fast-forward/pypy/module/_io/test/test_io.py Log: Allow the creation of io.xxxBase objects. CPython can inherit tp_new from the base class, because it assumes that zeroing the memory is a sane default for most types. The closest thing we could do in PyPy is to call the __init__ function with no argument, but it's difficult since the typedef has no reference to the W_Class. Meanwhile, __new__ = GenericNew(W_Class) is not too bad. If it works, I'll move it to typedef.py. Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/interp_io.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py Fri Oct 8 09:02:16 2010 @@ -5,18 +5,19 @@ DEFAULT_BUFFER_SIZE = 8192 +def GenericNew(W_Type): + def descr_new(space, w_subtype, __args__): + self = space.allocate_instance(W_Type, w_subtype) + W_Type.__init__(self, space) + return space.wrap(self) + descr_new.unwrap_spec = [ObjSpace, W_Root, Arguments] + return interp2app(descr_new) class W_BlockingIOError(W_IOError): def __init__(self, space): W_IOError.__init__(self, space) self.written = 0 - def descr_new(space, w_subtype, __args__): - self = space.allocate_instance(W_BlockingIOError, w_subtype) - W_BlockingIOError.__init__(self, space) - return space.wrap(self) - descr_new.unwrap_spec = [ObjSpace, W_Root, Arguments] - def descr_init(self, space, w_errno, w_strerror, written=0): W_IOError.descr_init(self, space, [w_errno, w_strerror]) self.written = written @@ -26,33 +27,40 @@ 'BlockingIOError', __doc__ = ("Exception raised when I/O would block " "on a non-blocking I/O stream"), - __new__ = interp2app(W_BlockingIOError.descr_new.im_func), + __new__ = GenericNew(W_BlockingIOError), __init__ = interp2app(W_BlockingIOError.descr_init), characters_written = interp_attrproperty('written', W_BlockingIOError), ) class W_IOBase(Wrappable): - pass + def __init__(self, space): + pass + W_IOBase.typedef = TypeDef( '_IOBase', + __new__ = GenericNew(W_IOBase), ) class W_RawIOBase(W_IOBase): pass W_RawIOBase.typedef = TypeDef( '_RawIOBase', W_IOBase.typedef, + __new__ = GenericNew(W_RawIOBase), ) class W_BufferedIOBase(W_IOBase): pass + W_BufferedIOBase.typedef = TypeDef( '_BufferedIOBase', W_IOBase.typedef, + __new__ = GenericNew(W_BufferedIOBase), ) class W_TextIOBase(W_IOBase): pass W_TextIOBase.typedef = TypeDef( '_TextIOBase', W_IOBase.typedef, + __new__ = GenericNew(W_TextIOBase), ) class W_FileIO(W_RawIOBase): Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py Fri Oct 8 09:02:16 2010 @@ -6,3 +6,13 @@ def test_import(self): import io + + def test_iobase(self): + import io + io.IOBase() + + class MyFile(io.BufferedIOBase): + def __init__(self, filename): + pass + MyFile("file") + From antocuni at codespeak.net Fri Oct 8 10:58:01 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 8 Oct 2010 10:58:01 +0200 (CEST) Subject: [pypy-svn] r77709 - in pypy/branch/jitffi/pypy/jit/metainterp: optimizeopt test Message-ID: <20101008085801.0FA63282BE8@codespeak.net> Author: antocuni Date: Fri Oct 8 10:58:00 2010 New Revision: 77709 Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py Log: if the funcval is not constant, just don't do the optimization instead of crashing Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py Fri Oct 8 10:58:00 2010 @@ -52,6 +52,9 @@ return f.inst_argtypes, f.inst_restype +class NonConstantFuncVal(Exception): + pass + class OptFfiCall(Optimization): def __init__(self): @@ -76,19 +79,25 @@ # targetval = self.getvalue(op.getarg(0)) oopspec = self.get_oopspec(targetval) - if oopspec == 'prepare_call': - self.do_prepare_call(op) - return - elif oopspec == 'push_arg': - self.do_push_arg(op) + if oopspec not in ('prepare_call', 'push_arg', 'call'): + self.emit_operation(op) # normal case return - elif oopspec == 'call': - op = self.do_call(op) - self.emit_operation(op) + # + try: + if oopspec == 'prepare_call': + self.do_prepare_call(op) + elif oopspec == 'push_arg': + self.do_push_arg(op) + elif oopspec == 'call': + op = self.do_call(op) + self.emit_operation(op) + except NonConstantFuncVal: + self.emit_operation(op) def _get_funcval(self, op): funcval = self.getvalue(op.getarg(1)) - assert funcval.is_constant() # XXX: do something nice if it's not constant + if not funcval.is_constant(): + raise NonConstantFuncVal return funcval def do_prepare_call(self, op): Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py Fri Oct 8 10:58:00 2010 @@ -157,8 +157,8 @@ if arg.startswith('"') or arg.startswith("'"): # XXX ootype info = arg.strip("'\"") - return ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, - llstr(info))) + llinfo = self._cache.setdefault(info, llstr(info)) + return ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, llinfo)) if arg.startswith('ConstClass('): name = arg[len('ConstClass('):-1] return self.get_const(name, 'class') Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py Fri Oct 8 10:58:00 2010 @@ -3941,23 +3941,42 @@ namespace = namespace.__dict__ - - def test_ffi_call(self): + def test_ffi_call_opt(self): ops = """ [i0, f1] - call("_libffi_prepare_call", ConstPtr(func), descr=plaincalldescr) - call("_libffi_push_arg_Signed", ConstPtr(func), i0, descr=plaincalldescr) - call("_libffi_push_arg_Float", ConstPtr(func), f1, descr=plaincalldescr) - i3 = call("_libffi_call", ConstPtr(func), 1, descr=plaincalldescr) + call("_libffi_prepare_call", ConstPtr(func), descr=plaincalldescr) + call("_libffi_push_arg_Signed", ConstPtr(func), i0, descr=plaincalldescr) + call("_libffi_push_arg_Float", ConstPtr(func), f1, descr=plaincalldescr) + i3 = call("_libffi_call", ConstPtr(func), 12345, descr=plaincalldescr) jump(i3, f1) """ expected = """ [i0, f1] - i3 = call(1, i0, f1, descr=int_float__int) + i3 = call(12345, i0, f1, descr=int_float__int) jump(i3, f1) """ loop = self.optimize_loop(ops, 'Not, Not', expected) + def test_ffi_call_nonconst(self): + ops = """ + [i0, f1, p2] + call("_libffi_prepare_call", p2, descr=plaincalldescr) + call("_libffi_push_arg_Signed", p2, i0, descr=plaincalldescr) + call("_libffi_push_arg_Float", p2, f1, descr=plaincalldescr) + i3 = call("_libffi_call", p2, 1, descr=plaincalldescr) + jump(i3, f1, p2) + """ + expected = """ + [i0, f1, p2] + call("_libffi_prepare_call", p2, descr=plaincalldescr) + call("_libffi_push_arg_Signed", p2, i0, descr=plaincalldescr) + call("_libffi_push_arg_Float", p2, f1, descr=plaincalldescr) + i3 = call("_libffi_call", p2, 1, descr=plaincalldescr) + jump(i3, f1, p2) + + """ + loop = self.optimize_loop(ops, 'Not, Not, Not', expected) + ##class TestOOtype(OptimizeOptTest, OOtypeMixin): From antocuni at codespeak.net Fri Oct 8 11:07:23 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 8 Oct 2010 11:07:23 +0200 (CEST) Subject: [pypy-svn] r77710 - pypy/branch/jitffi/pypy/jit/tl Message-ID: <20101008090723.68A06282BE8@codespeak.net> Author: antocuni Date: Fri Oct 8 11:07:21 2010 New Revision: 77710 Modified: pypy/branch/jitffi/pypy/jit/tl/pypyjit.py pypy/branch/jitffi/pypy/jit/tl/pypyjit_child.py pypy/branch/jitffi/pypy/jit/tl/pypyjit_demo.py Log: update pypyjit* to test _ffi Modified: pypy/branch/jitffi/pypy/jit/tl/pypyjit.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/tl/pypyjit.py (original) +++ pypy/branch/jitffi/pypy/jit/tl/pypyjit.py Fri Oct 8 11:07:21 2010 @@ -40,6 +40,9 @@ config.objspace.usemodules.array = True config.objspace.usemodules._weakref = False config.objspace.usemodules._sre = False +# +config.objspace.usemodules._ffi = True +# set_pypy_opt_level(config, level='jit') config.objspace.std.withinlineddict = True Modified: pypy/branch/jitffi/pypy/jit/tl/pypyjit_child.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/tl/pypyjit_child.py (original) +++ pypy/branch/jitffi/pypy/jit/tl/pypyjit_child.py Fri Oct 8 11:07:21 2010 @@ -16,7 +16,7 @@ interp.heap.malloc_nonmovable = returns_null # XXX from pypy.jit.backend.llgraph.runner import LLtypeCPU - LLtypeCPU.supports_floats = False # for now + #LLtypeCPU.supports_floats = False # for now apply_jit(interp, graph, LLtypeCPU) Modified: pypy/branch/jitffi/pypy/jit/tl/pypyjit_demo.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/tl/pypyjit_demo.py (original) +++ pypy/branch/jitffi/pypy/jit/tl/pypyjit_demo.py Fri Oct 8 11:07:21 2010 @@ -37,26 +37,18 @@ ## t2 = time.time() ## print t2 - t1 -try: - from array import array - - def coords(w,h): - y = 0 - while y < h: - x = 0 - while x < w: - yield x,y - x += 1 - y += 1 +def main(): + from _ffi import CDLL, types + libm = CDLL('libm.so') + pow = libm.getfunc('pow', [types.double, types.double], types.double) + i = 0 + while i < 100: + res = pow(2, 3) + i += 1 + print res - def f(img): - sa=0 - for x, y in coords(4,4): - sa += x * y - return sa - - #img=array('h',(1,2,3,4)) - print f(3) +try: + main() except Exception, e: print "Exception: ", type(e) print e From david at codespeak.net Fri Oct 8 11:11:07 2010 From: david at codespeak.net (david at codespeak.net) Date: Fri, 8 Oct 2010 11:11:07 +0200 (CEST) Subject: [pypy-svn] r77711 - pypy/branch/arm-backend/pypy/jit/backend/llsupport Message-ID: <20101008091107.61095282BE8@codespeak.net> Author: david Date: Fri Oct 8 11:11:05 2010 New Revision: 77711 Modified: pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py Log: Remove self from compute_loop_consts arguments Modified: pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/llsupport/regalloc.py Fri Oct 8 11:11:05 2010 @@ -384,7 +384,7 @@ assert isinstance(arg, Box) return longevity -def compute_loop_consts(self, inputargs, jump, looptoken): +def compute_loop_consts(inputargs, jump, looptoken): if jump.getopnum() != rop.JUMP or jump.getdescr() is not looptoken: loop_consts = {} else: From antocuni at codespeak.net Fri Oct 8 11:58:18 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 8 Oct 2010 11:58:18 +0200 (CEST) Subject: [pypy-svn] r77712 - in pypy/branch/jitffi/pypy: module/_ffi module/pypyjit rlib Message-ID: <20101008095818.B3E54282BE8@codespeak.net> Author: antocuni Date: Fri Oct 8 11:58:17 2010 New Revision: 77712 Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py pypy/branch/jitffi/pypy/module/pypyjit/policy.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: be more jit-friendly Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Fri Oct 8 11:58:17 2010 @@ -47,6 +47,9 @@ # ======================================================================== class W_FuncPtr(Wrappable): + + _immutable_fields_ = ['func'] + def __init__(self, func): self.func = func @@ -68,6 +71,7 @@ @unwrap_spec('self', ObjSpace, 'args_w') def call(self, space, args_w): + self = jit.hint(self, promote=True) argchain = self.build_argchain(space, self.func.argtypes, args_w) reskind = libffi.types.getkind(self.func.restype) if reskind == 'i': Modified: pypy/branch/jitffi/pypy/module/pypyjit/policy.py ============================================================================== --- pypy/branch/jitffi/pypy/module/pypyjit/policy.py (original) +++ pypy/branch/jitffi/pypy/module/pypyjit/policy.py Fri Oct 8 11:58:17 2010 @@ -11,7 +11,7 @@ if '.' in modname: modname, _ = modname.split('.', 1) if modname in ['pypyjit', 'signal', 'micronumpy', 'math', 'exceptions', - 'imp', 'sys', 'array']: + 'imp', 'sys', 'array', '_ffi']: return True return False Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Fri Oct 8 11:58:17 2010 @@ -32,8 +32,9 @@ cls.ulong = clibffi.cast_type_to_ffitype(rffi.ULONG) del cls._import - @classmethod - def getkind(cls, ffitype): + @staticmethod + @jit.purefunction + def getkind(ffitype): # XXX: move this function outside the jit from pypy.jit.backend.llsupport.ffisupport import get_ffi_type_kind return get_ffi_type_kind(ffitype) From afa at codespeak.net Fri Oct 8 12:02:27 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 12:02:27 +0200 (CEST) Subject: [pypy-svn] r77713 - pypy/branch/fast-forward/pypy/module/_io Message-ID: <20101008100227.9F0A9282BE8@codespeak.net> Author: afa Date: Fri Oct 8 12:02:26 2010 New Revision: 77713 Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py Log: Fix translation: interp2app requires that all functions in the same module have different names Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/interp_io.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py Fri Oct 8 12:02:26 2010 @@ -1,16 +1,18 @@ from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root from pypy.interpreter.typedef import TypeDef, interp_attrproperty -from pypy.interpreter.gateway import interp2app, Arguments +from pypy.interpreter.gateway import interp2app, Arguments, unwrap_spec from pypy.module.exceptions.interp_exceptions import W_IOError +from pypy.tool.sourcetools import func_renamer DEFAULT_BUFFER_SIZE = 8192 def GenericNew(W_Type): + @unwrap_spec(ObjSpace, W_Root, Arguments) + @func_renamer('descr_new_%s' % W_Type.__name__) def descr_new(space, w_subtype, __args__): self = space.allocate_instance(W_Type, w_subtype) W_Type.__init__(self, space) return space.wrap(self) - descr_new.unwrap_spec = [ObjSpace, W_Root, Arguments] return interp2app(descr_new) class W_BlockingIOError(W_IOError): @@ -18,10 +20,10 @@ W_IOError.__init__(self, space) self.written = 0 + @unwrap_spec('self', ObjSpace, W_Root, W_Root, int) def descr_init(self, space, w_errno, w_strerror, written=0): W_IOError.descr_init(self, space, [w_errno, w_strerror]) self.written = written - descr_init.unwrap_spec = ['self', ObjSpace, W_Root, W_Root, int] W_BlockingIOError.typedef = TypeDef( 'BlockingIOError', From antocuni at codespeak.net Fri Oct 8 12:07:00 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 8 Oct 2010 12:07:00 +0200 (CEST) Subject: [pypy-svn] r77714 - pypy/branch/jitffi/pypy/rlib Message-ID: <20101008100700.D80FE282BE8@codespeak.net> Author: antocuni Date: Fri Oct 8 12:06:59 2010 New Revision: 77714 Modified: pypy/branch/jitffi/pypy/rlib/clibffi.py Log: remove JIT-related experiments from clibffi. This should fix test_newgc and test_asmgcroot Modified: pypy/branch/jitffi/pypy/rlib/clibffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/clibffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/clibffi.py Fri Oct 8 12:06:59 2010 @@ -471,8 +471,6 @@ ll_args = lltype.nullptr(rffi.VOIDPP.TO) ll_result = lltype.nullptr(rffi.VOIDP.TO) - _immutable_fields_ = ['funcsym', 'argtypes'] # XXX probably more - def __init__(self, name, argtypes, restype, funcsym, flags=FUNCFLAG_CDECL, keepalive=None): # initialize each one of pointers with null @@ -511,11 +509,7 @@ push_arg_as_ffiptr(self.argtypes[self.pushed_args], value, self.ll_args[self.pushed_args]) self.pushed_args += 1 - # XXX this is bad, fix it somehow in the future, but specialize:argtype - # doesn't work correctly with mixing non-negative and normal integers - push_arg._annenforceargs_ = [None, int] - #push_arg._annspecialcase_ = 'specialize:argtype(1)' - #push_arg.oopspec = 'libffi_push_arg(self, value)' + push_arg._annspecialcase_ = 'specialize:argtype(1)' def _check_args(self): if self.pushed_args < self.argnum: @@ -538,7 +532,6 @@ check_fficall_result(ffires, self.flags) return res call._annspecialcase_ = 'specialize:arg(1)' - #call.oopspec = 'libffi_call(self, RES_TP)' def __del__(self): if self.ll_args: From arigo at codespeak.net Fri Oct 8 13:18:08 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 8 Oct 2010 13:18:08 +0200 (CEST) Subject: [pypy-svn] r77715 - pypy/extradoc/pypy.org/source Message-ID: <20101008111808.AEC99282BE8@codespeak.net> Author: arigo Date: Fri Oct 8 13:18:07 2010 New Revision: 77715 Modified: pypy/extradoc/pypy.org/source/download.txt Log: Add a link to source dependencies. Modified: pypy/extradoc/pypy.org/source/download.txt ============================================================================== --- pypy/extradoc/pypy.org/source/download.txt (original) +++ pypy/extradoc/pypy.org/source/download.txt Fri Oct 8 13:18:07 2010 @@ -174,11 +174,15 @@ .. __: http://codespeak.net/pypy/trunk/pypy/doc/windows.html -2. Enter the ``goal`` directory:: +2. Make sure you installed the dependencies. See the list here__. + + .. __: http://codespeak.net/pypy/dist/pypy/doc/getting-started-python.html#translating-the-pypy-python-interpreter + +3. Enter the ``goal`` directory:: cd pypy-trunk/pypy/translator/goal -3. Run the ``translate.py`` script. Here are the common combinations +4. Run the ``translate.py`` script. Here are the common combinations of options:: python translate.py -Ojit # get the JIT version @@ -187,7 +191,7 @@ python translate.py --stackless # get the stackless version python translate.py -Ojit --backend=cli # only for branch/cli-jit -4. Enjoy Mandelbrot ``:-)`` It takes on the order of one hour to +5. Enjoy Mandelbrot ``:-)`` It takes on the order of one hour to finish the translation, and 1.3 GB of RAM on a 32-bit system. (Do not start a translation on a machine with 1GB or less!) From arigo at codespeak.net Fri Oct 8 13:23:41 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 8 Oct 2010 13:23:41 +0200 (CEST) Subject: [pypy-svn] r77716 - in pypy/trunk/pypy/translator/c/gcc: . test/elf64 Message-ID: <20101008112341.73B03282BE8@codespeak.net> Author: arigo Date: Fri Oct 8 13:23:39 2010 New Revision: 77716 Added: pypy/trunk/pypy/translator/c/gcc/test/elf64/track_loadconst.s pypy/trunk/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s Modified: pypy/trunk/pypy/translator/c/gcc/instruction.py pypy/trunk/pypy/translator/c/gcc/trackgcroot.py Log: Two failing tests, and corresponding fixes to trackgcroot. Modified: pypy/trunk/pypy/translator/c/gcc/instruction.py ============================================================================== --- pypy/trunk/pypy/translator/c/gcc/instruction.py (original) +++ pypy/trunk/pypy/translator/c/gcc/instruction.py Fri Oct 8 13:23:39 2010 @@ -82,6 +82,11 @@ def all_sources_of(self, localvar): return [localvar] +class InsnCondJump(Insn): # only for debugging; not used internally + _args_ = ['label'] + def __init__(self, label): + self.label = label + class Label(Insn): _args_ = ['label', 'lineno'] def __init__(self, label, lineno): @@ -170,9 +175,12 @@ self.delta = -7 # use an odd value as marker class InsnStop(Insn): - pass + _args_ = ['reason'] + def __init__(self, reason='?'): + self.reason = reason class InsnRet(InsnStop): + _args_ = [] framesize = 0 def __init__(self, registers): self.registers = registers Added: pypy/trunk/pypy/translator/c/gcc/test/elf64/track_loadconst.s ============================================================================== --- (empty file) +++ pypy/trunk/pypy/translator/c/gcc/test/elf64/track_loadconst.s Fri Oct 8 13:23:39 2010 @@ -0,0 +1,12 @@ + .type pypy_g_handler_inline_call_r_v_1, @function +pypy_g_handler_inline_call_r_v_1: + pushq %rbp + call foo_bar + ;; expected {8(%rsp) | %rbx, %r12, %r13, %r14, %r15, (%rsp) | } + movl $pypy_g_some_prebuilt_gc_object, %ebp + call bar_baz + ;; expected {8(%rsp) | %rbx, %r12, %r13, %r14, %r15, (%rsp) | %rbp} + /* GCROOT %rbp */ + popq %rbp + ret + .size pypy_g_handler_inline_call_r_v_1, .-pypy_g_handler_inline_call_r_v_1 Added: pypy/trunk/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s ============================================================================== --- (empty file) +++ pypy/trunk/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s Fri Oct 8 13:23:39 2010 @@ -0,0 +1,16 @@ + .type pypy_g_handler_inline_call_r_v_1, @function +pypy_g_handler_inline_call_r_v_1: +.L10041: + pushq %r15 + movq 8(%rsi), %r15 + call foo_bar + ;; expected {8(%rsp) | %rbx, %r12, %r13, %r14, (%rsp), %rbp | %r15} + testq %rax, %rax + jne .L10088 + movq (%ebp), %r15 + call RPyAssertFailed +.L10088: + /* GCROOT %r15 */ + popq %r15 + ret + .size pypy_g_handler_inline_call_r_v_1, .-pypy_g_handler_inline_call_r_v_1 Modified: pypy/trunk/pypy/translator/c/gcc/trackgcroot.py ============================================================================== --- pypy/trunk/pypy/translator/c/gcc/trackgcroot.py (original) +++ pypy/trunk/pypy/translator/c/gcc/trackgcroot.py Fri Oct 8 13:23:39 2010 @@ -6,7 +6,7 @@ from pypy.translator.c.gcc.instruction import InsnFunctionStart, InsnStop from pypy.translator.c.gcc.instruction import InsnSetLocal, InsnCopyLocal from pypy.translator.c.gcc.instruction import InsnPrologue, InsnEpilogue -from pypy.translator.c.gcc.instruction import InsnGCROOT +from pypy.translator.c.gcc.instruction import InsnGCROOT, InsnCondJump from pypy.translator.c.gcc.instruction import InsnStackAdjust from pypy.translator.c.gcc.instruction import InsnCannotFollowEsp from pypy.translator.c.gcc.instruction import LocalVar, somenewvalue @@ -123,18 +123,15 @@ self.labels[label] = Label(label, lineno) def find_noncollecting_calls(self): - cannot_collect = self.CANNOT_COLLECT.copy() + cannot_collect = {} for line in self.lines: match = self.r_gcnocollect_marker.search(line) if match: name = match.group(1) cannot_collect[name] = True # - if self.format in ('darwin', 'mingw32', 'msvc'): - self.cannot_collect = dict.fromkeys( - ['_' + name for name in cannot_collect]) - else: - self.cannot_collect = cannot_collect + self.cannot_collect = dict.fromkeys( + [self.function_names_prefix + name for name in cannot_collect]) def append_instruction(self, insn): # Add the instruction to the list, and link it to the previous one. @@ -410,7 +407,8 @@ return result # ____________________________________________________________ - CANNOT_COLLECT = { # some of the most used functions that cannot collect + BASE_FUNCTIONS_NOT_RETURNING = { + 'abort': None, 'pypy_debug_catch_fatal_exception': None, 'RPyAbort': None, 'RPyAssertFailed': None, @@ -644,7 +642,7 @@ if label != '0': self.register_jump_to(label) tablelin += 1 - return InsnStop() + return InsnStop("jump table") if self.r_unaryinsn_star.match(line): # that looks like an indirect tail-call. # tail-calls are equivalent to RET for us @@ -658,7 +656,7 @@ assert not target.startswith('.') # tail-calls are equivalent to RET for us return InsnRet(self.CALLEE_SAVE_REGISTERS) - return InsnStop() + return InsnStop("jump") def register_jump_to(self, label): if not isinstance(self.insns[-1], InsnStop): @@ -682,7 +680,7 @@ else: label = match.group(1) self.register_jump_to(label) - return [] + return [InsnCondJump(label)] visit_jmpl = visit_jmp visit_je = conditional_jump @@ -754,7 +752,7 @@ target, = sources if target in self.FUNCTIONS_NOT_RETURNING: - return [InsnStop(), InsnCannotFollowEsp()] + return [InsnStop(target), InsnCannotFollowEsp()] if self.format == 'mingw32' and target == '__alloca': # in functions with large stack requirements, windows # needs a call to _alloca(), to turn reserved pages @@ -885,7 +883,7 @@ # statically known pointer to a register # %eax -> %rax - new_line = re.sub(r"%e(ax|bx|cx|dx|di|si)$", r"%r\1", line) + new_line = re.sub(r"%e(ax|bx|cx|dx|di|si|bp)$", r"%r\1", line) # %r10d -> %r10 new_line = re.sub(r"%r(\d+)d$", r"%r\1", new_line) return func(self, new_line) @@ -951,6 +949,7 @@ class ElfFunctionGcRootTracker32(FunctionGcRootTracker32): format = 'elf' + function_names_prefix = '' ESP = '%esp' EBP = '%ebp' @@ -984,13 +983,14 @@ r_bottom_marker = re.compile(r"\t/[*] GC_STACK_BOTTOM [*]/") FUNCTIONS_NOT_RETURNING = { - 'abort': None, '_exit': None, '__assert_fail': None, '___assert_rtn': None, 'L___assert_rtn$stub': None, 'L___eprintf$stub': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING[_name] = None def __init__(self, lines, filetag=0): match = self.r_functionstart.match(lines[0]) @@ -1010,6 +1010,8 @@ class ElfFunctionGcRootTracker64(FunctionGcRootTracker64): format = 'elf64' + function_names_prefix = '' + ESP = '%rsp' EBP = '%rbp' EAX = '%rax' @@ -1042,13 +1044,14 @@ r_bottom_marker = re.compile(r"\t/[*] GC_STACK_BOTTOM [*]/") FUNCTIONS_NOT_RETURNING = { - 'abort': None, '_exit': None, '__assert_fail': None, '___assert_rtn': None, 'L___assert_rtn$stub': None, 'L___eprintf$stub': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING[_name] = None def __init__(self, lines, filetag=0): match = self.r_functionstart.match(lines[0]) @@ -1068,6 +1071,7 @@ class DarwinFunctionGcRootTracker(ElfFunctionGcRootTracker32): format = 'darwin' + function_names_prefix = '_' r_functionstart = re.compile(r"_(\w+):\s*$") OFFSET_LABELS = 0 @@ -1079,15 +1083,19 @@ class Mingw32FunctionGcRootTracker(DarwinFunctionGcRootTracker): format = 'mingw32' + function_names_prefix = '_' FUNCTIONS_NOT_RETURNING = { - '_abort': None, '_exit': None, '__assert': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING['_' + _name] = None class MsvcFunctionGcRootTracker(FunctionGcRootTracker32): format = 'msvc' + function_names_prefix = '_' + ESP = 'esp' EBP = 'ebp' EAX = 'eax' @@ -1127,7 +1135,6 @@ r_bottom_marker = re.compile(r"; .+\tpypy_asm_stack_bottom\(\);") FUNCTIONS_NOT_RETURNING = { - '_abort': None, '__exit': None, '__assert': None, '__wassert': None, @@ -1136,6 +1143,8 @@ 'DWORD PTR __imp__abort': None, 'DWORD PTR __imp___wassert': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING['_' + _name] = None @classmethod def init_regexp(cls): @@ -1543,9 +1552,7 @@ assert self.seen_main def _globalname(name, disp=""): - if self.format in ('darwin', 'mingw32', 'msvc'): - name = '_' + name - return name + return tracker_cls.function_names_prefix + name def _variant(**kwargs): txt = kwargs[self.format] From arigo at codespeak.net Fri Oct 8 13:24:54 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 8 Oct 2010 13:24:54 +0200 (CEST) Subject: [pypy-svn] r77717 - pypy/branch/jit-str-unicode Message-ID: <20101008112454.39188282BE8@codespeak.net> Author: arigo Date: Fri Oct 8 13:24:52 2010 New Revision: 77717 Added: pypy/branch/jit-str-unicode/ - copied from r77716, pypy/trunk/ Log: A branch where I want to duplicate for unicode the optimizations that the jit does for strings. From arigo at codespeak.net Fri Oct 8 13:25:57 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 8 Oct 2010 13:25:57 +0200 (CEST) Subject: [pypy-svn] r77718 - in pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc: . test/elf64 Message-ID: <20101008112557.0511B282BE8@codespeak.net> Author: arigo Date: Fri Oct 8 13:25:56 2010 New Revision: 77718 Added: pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf64/track_loadconst.s pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/instruction.py pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/trackgcroot.py Log: Merge r77716 from trunk. Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/instruction.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/instruction.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/instruction.py Fri Oct 8 13:25:56 2010 @@ -82,6 +82,11 @@ def all_sources_of(self, localvar): return [localvar] +class InsnCondJump(Insn): # only for debugging; not used internally + _args_ = ['label'] + def __init__(self, label): + self.label = label + class Label(Insn): _args_ = ['label', 'lineno'] def __init__(self, label, lineno): @@ -170,9 +175,12 @@ self.delta = -7 # use an odd value as marker class InsnStop(Insn): - pass + _args_ = ['reason'] + def __init__(self, reason='?'): + self.reason = reason class InsnRet(InsnStop): + _args_ = [] framesize = 0 def __init__(self, registers): self.registers = registers Added: pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf64/track_loadconst.s ============================================================================== --- (empty file) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf64/track_loadconst.s Fri Oct 8 13:25:56 2010 @@ -0,0 +1,12 @@ + .type pypy_g_handler_inline_call_r_v_1, @function +pypy_g_handler_inline_call_r_v_1: + pushq %rbp + call foo_bar + ;; expected {8(%rsp) | %rbx, %r12, %r13, %r14, %r15, (%rsp) | } + movl $pypy_g_some_prebuilt_gc_object, %ebp + call bar_baz + ;; expected {8(%rsp) | %rbx, %r12, %r13, %r14, %r15, (%rsp) | %rbp} + /* GCROOT %rbp */ + popq %rbp + ret + .size pypy_g_handler_inline_call_r_v_1, .-pypy_g_handler_inline_call_r_v_1 Added: pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s ============================================================================== --- (empty file) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s Fri Oct 8 13:25:56 2010 @@ -0,0 +1,16 @@ + .type pypy_g_handler_inline_call_r_v_1, @function +pypy_g_handler_inline_call_r_v_1: +.L10041: + pushq %r15 + movq 8(%rsi), %r15 + call foo_bar + ;; expected {8(%rsp) | %rbx, %r12, %r13, %r14, (%rsp), %rbp | %r15} + testq %rax, %rax + jne .L10088 + movq (%ebp), %r15 + call RPyAssertFailed +.L10088: + /* GCROOT %r15 */ + popq %r15 + ret + .size pypy_g_handler_inline_call_r_v_1, .-pypy_g_handler_inline_call_r_v_1 Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/trackgcroot.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/trackgcroot.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/trackgcroot.py Fri Oct 8 13:25:56 2010 @@ -6,7 +6,7 @@ from pypy.translator.c.gcc.instruction import InsnFunctionStart, InsnStop from pypy.translator.c.gcc.instruction import InsnSetLocal, InsnCopyLocal from pypy.translator.c.gcc.instruction import InsnPrologue, InsnEpilogue -from pypy.translator.c.gcc.instruction import InsnGCROOT +from pypy.translator.c.gcc.instruction import InsnGCROOT, InsnCondJump from pypy.translator.c.gcc.instruction import InsnStackAdjust from pypy.translator.c.gcc.instruction import InsnCannotFollowEsp from pypy.translator.c.gcc.instruction import LocalVar, somenewvalue @@ -123,18 +123,15 @@ self.labels[label] = Label(label, lineno) def find_noncollecting_calls(self): - cannot_collect = self.CANNOT_COLLECT.copy() + cannot_collect = {} for line in self.lines: match = self.r_gcnocollect_marker.search(line) if match: name = match.group(1) cannot_collect[name] = True # - if self.format in ('darwin', 'mingw32', 'msvc'): - self.cannot_collect = dict.fromkeys( - ['_' + name for name in cannot_collect]) - else: - self.cannot_collect = cannot_collect + self.cannot_collect = dict.fromkeys( + [self.function_names_prefix + name for name in cannot_collect]) def append_instruction(self, insn): # Add the instruction to the list, and link it to the previous one. @@ -410,7 +407,8 @@ return result # ____________________________________________________________ - CANNOT_COLLECT = { # some of the most used functions that cannot collect + BASE_FUNCTIONS_NOT_RETURNING = { + 'abort': None, 'pypy_debug_catch_fatal_exception': None, 'RPyAbort': None, 'RPyAssertFailed': None, @@ -644,7 +642,7 @@ if label != '0': self.register_jump_to(label) tablelin += 1 - return InsnStop() + return InsnStop("jump table") if self.r_unaryinsn_star.match(line): # that looks like an indirect tail-call. # tail-calls are equivalent to RET for us @@ -658,7 +656,7 @@ assert not target.startswith('.') # tail-calls are equivalent to RET for us return InsnRet(self.CALLEE_SAVE_REGISTERS) - return InsnStop() + return InsnStop("jump") def register_jump_to(self, label): if not isinstance(self.insns[-1], InsnStop): @@ -682,7 +680,7 @@ else: label = match.group(1) self.register_jump_to(label) - return [] + return [InsnCondJump(label)] visit_jmpl = visit_jmp visit_je = conditional_jump @@ -754,7 +752,7 @@ target, = sources if target in self.FUNCTIONS_NOT_RETURNING: - return [InsnStop(), InsnCannotFollowEsp()] + return [InsnStop(target), InsnCannotFollowEsp()] if self.format == 'mingw32' and target == '__alloca': # in functions with large stack requirements, windows # needs a call to _alloca(), to turn reserved pages @@ -885,7 +883,7 @@ # statically known pointer to a register # %eax -> %rax - new_line = re.sub(r"%e(ax|bx|cx|dx|di|si)$", r"%r\1", line) + new_line = re.sub(r"%e(ax|bx|cx|dx|di|si|bp)$", r"%r\1", line) # %r10d -> %r10 new_line = re.sub(r"%r(\d+)d$", r"%r\1", new_line) return func(self, new_line) @@ -951,6 +949,7 @@ class ElfFunctionGcRootTracker32(FunctionGcRootTracker32): format = 'elf' + function_names_prefix = '' ESP = '%esp' EBP = '%ebp' @@ -984,13 +983,14 @@ r_bottom_marker = re.compile(r"\t/[*] GC_STACK_BOTTOM [*]/") FUNCTIONS_NOT_RETURNING = { - 'abort': None, '_exit': None, '__assert_fail': None, '___assert_rtn': None, 'L___assert_rtn$stub': None, 'L___eprintf$stub': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING[_name] = None def __init__(self, lines, filetag=0): match = self.r_functionstart.match(lines[0]) @@ -1010,6 +1010,8 @@ class ElfFunctionGcRootTracker64(FunctionGcRootTracker64): format = 'elf64' + function_names_prefix = '' + ESP = '%rsp' EBP = '%rbp' EAX = '%rax' @@ -1042,13 +1044,14 @@ r_bottom_marker = re.compile(r"\t/[*] GC_STACK_BOTTOM [*]/") FUNCTIONS_NOT_RETURNING = { - 'abort': None, '_exit': None, '__assert_fail': None, '___assert_rtn': None, 'L___assert_rtn$stub': None, 'L___eprintf$stub': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING[_name] = None def __init__(self, lines, filetag=0): match = self.r_functionstart.match(lines[0]) @@ -1068,6 +1071,7 @@ class DarwinFunctionGcRootTracker(ElfFunctionGcRootTracker32): format = 'darwin' + function_names_prefix = '_' r_functionstart = re.compile(r"_(\w+):\s*$") OFFSET_LABELS = 0 @@ -1079,15 +1083,19 @@ class Mingw32FunctionGcRootTracker(DarwinFunctionGcRootTracker): format = 'mingw32' + function_names_prefix = '_' FUNCTIONS_NOT_RETURNING = { - '_abort': None, '_exit': None, '__assert': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING['_' + _name] = None class MsvcFunctionGcRootTracker(FunctionGcRootTracker32): format = 'msvc' + function_names_prefix = '_' + ESP = 'esp' EBP = 'ebp' EAX = 'eax' @@ -1127,7 +1135,6 @@ r_bottom_marker = re.compile(r"; .+\tpypy_asm_stack_bottom\(\);") FUNCTIONS_NOT_RETURNING = { - '_abort': None, '__exit': None, '__assert': None, '__wassert': None, @@ -1136,6 +1143,8 @@ 'DWORD PTR __imp__abort': None, 'DWORD PTR __imp___wassert': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING['_' + _name] = None @classmethod def init_regexp(cls): @@ -1543,9 +1552,7 @@ assert self.seen_main def _globalname(name, disp=""): - if self.format in ('darwin', 'mingw32', 'msvc'): - name = '_' + name - return name + return tracker_cls.function_names_prefix + name def _variant(**kwargs): txt = kwargs[self.format] From arigo at codespeak.net Fri Oct 8 13:26:17 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Fri, 8 Oct 2010 13:26:17 +0200 (CEST) Subject: [pypy-svn] r77719 - in pypy/branch/32ptr-on-64bit/pypy/rpython/memory: gc test Message-ID: <20101008112617.DD340282BE8@codespeak.net> Author: arigo Date: Fri Oct 8 13:26:16 2010 New Revision: 77719 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py Log: Tweaks. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimark.py Fri Oct 8 13:26:16 2010 @@ -190,6 +190,13 @@ else: from pypy.rpython.memory.gc import minimarkpage ArenaCollectionClass = minimarkpage.ArenaCollection + # + if self.config and self.config.compressptr: + # increase the limit to the maximum + small_request_threshold = ( + page_size - ArenaCollectionClass.PAGE_HEADER_SIZE_MAX) + self.small_request_threshold = small_request_threshold + # self.ac = ArenaCollectionClass(arena_size, page_size, small_request_threshold) # @@ -325,6 +332,15 @@ totalsize = size_gc_header + size rawtotalsize = raw_malloc_usage(totalsize) # + # It is necessary that rawtotalsize fits within + # small_request_threshold if self.config.compressptr is set. + # Note that in this case we increased small_request_threshold at + # start-up to make this more likely to always hold. The + # following check should be constant-folded. + if self.config.compressptr: + if rawtotalsize > self.small_request_threshold: + raise FixedSizeObjectTooLarge + # # If the object needs a finalizer, ask for a rawmalloc. # The following check should be constant-folded. if needs_finalizer: @@ -1534,6 +1550,7 @@ # list. class SimpleArenaCollection(object): + PAGE_HEADER_SIZE_MAX = 0 def __init__(self, arena_size, page_size, small_request_threshold): self.arena_size = arena_size # ignored Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage.py Fri Oct 8 13:26:16 2010 @@ -86,6 +86,7 @@ class ArenaCollection(object): _alloc_flavor_ = "raw" + PAGE_HEADER_SIZE_MAX = 32 def __init__(self, arena_size, page_size, small_request_threshold): # 'small_request_threshold' is the largest size that we Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/gc/minimarkpage2.py Fri Oct 8 13:26:16 2010 @@ -62,6 +62,7 @@ # '(512-2) % N' is zero or very small for various small N's, # i.e. there is not much wasted space. ) +PAGE_HEADER_SIZE_MAX = 16 # the PAGE_HEADER is at most 16 bytes PAGE_PTR.TO.become(PAGE_HEADER) PAGE_NULL = lltype.nullptr(PAGE_HEADER) @@ -73,6 +74,7 @@ class ArenaCollection2(object): _alloc_flavor_ = "raw" + PAGE_HEADER_SIZE_MAX = PAGE_HEADER_SIZE_MAX def __init__(self, arena_size, page_size, small_request_threshold): # 'small_request_threshold' is the largest size that we @@ -266,7 +268,9 @@ # and become available for reuse by any size class. Pages # not completely freed are re-chained either in # 'full_page_for_size[]' or 'page_for_size[]'. - self.mass_free_in_pages(size_class, ok_to_free_func) + if (self.full_page_for_size[size_class] != PAGE_NULL or + self.page_for_size[size_class] != PAGE_NULL): + self.mass_free_in_pages(size_class, ok_to_free_func) # size_class -= 1 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gc.py Fri Oct 8 13:26:16 2010 @@ -817,6 +817,7 @@ GC_PARAMS = {'card_page_indices': 4} class TestMiniMarkGCCompressPtr(TestMiniMarkGC): + GC_PARAMS = {'page_size': 32*WORD} def setup_class(cls): TestMiniMarkGC.setup_class.im_func(cls) # From antocuni at codespeak.net Fri Oct 8 13:40:16 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 8 Oct 2010 13:40:16 +0200 (CEST) Subject: [pypy-svn] r77720 - in pypy/branch/jitffi: . lib_pypy/_ctypes pypy pypy/annotation pypy/annotation/test pypy/config pypy/config/test pypy/doc pypy/interpreter pypy/interpreter/astcompiler/test pypy/interpreter/pyparser pypy/interpreter/test pypy/jit/backend/llgraph pypy/jit/backend/llsupport pypy/jit/backend/llsupport/test pypy/jit/backend/test pypy/jit/backend/x86 pypy/jit/backend/x86/test pypy/jit/codewriter pypy/jit/codewriter/test pypy/jit/metainterp pypy/jit/metainterp/optimizeopt pypy/jit/metainterp/test pypy/jit/tl/spli pypy/jit/tool pypy/module/array/benchmark pypy/module/array/test pypy/module/imp pypy/module/imp/test pypy/module/parser/test pypy/module/pypyjit/test pypy/module/sys pypy/module/test_lib_pypy/ctypes_tests pypy/objspace/std pypy/rlib pypy/rlib/test pypy/rpython pypy/rpython/lltypesystem pypy/rpython/lltypesystem/test pypy/rpython/memory pypy/rpython/memory/gc pypy/rpython/memory/gc/test pypy/rpython/memory/gctransform pypy/rpython/memory/test pypy/rpython/test pypy/tool pypy/translator pypy/translator/c/gcc pypy/translator/c/gcc/test/elf64 pypy/translator/c/test pypy/translator/goal pypy/translator/jvm/test pypy/translator/oosupport/test_template Message-ID: <20101008114016.A115A282BE8@codespeak.net> Author: antocuni Date: Fri Oct 8 13:40:10 2010 New Revision: 77720 Added: pypy/branch/jitffi/pypy/translator/c/gcc/test/elf64/track_loadconst.s - copied unchanged from r77719, pypy/trunk/pypy/translator/c/gcc/test/elf64/track_loadconst.s pypy/branch/jitffi/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s - copied unchanged from r77719, pypy/trunk/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s Removed: pypy/branch/jitffi/pypy/jit/backend/test/conftest.py pypy/branch/jitffi/pypy/rpython/memory/gc/inspect.py pypy/branch/jitffi/pypy/rpython/rspecialcase.py pypy/branch/jitffi/pypy/rpython/test/test_rspecialcase.py Modified: pypy/branch/jitffi/ (props changed) pypy/branch/jitffi/lib_pypy/_ctypes/function.py pypy/branch/jitffi/pypy/ (props changed) pypy/branch/jitffi/pypy/annotation/policy.py pypy/branch/jitffi/pypy/annotation/test/test_annrpython.py pypy/branch/jitffi/pypy/config/test/test_pypyoption.py pypy/branch/jitffi/pypy/config/translationoption.py pypy/branch/jitffi/pypy/doc/docindex.txt pypy/branch/jitffi/pypy/interpreter/astcompiler/test/test_compiler.py pypy/branch/jitffi/pypy/interpreter/baseobjspace.py pypy/branch/jitffi/pypy/interpreter/function.py pypy/branch/jitffi/pypy/interpreter/pycode.py pypy/branch/jitffi/pypy/interpreter/pyframe.py pypy/branch/jitffi/pypy/interpreter/pyopcode.py pypy/branch/jitffi/pypy/interpreter/pyparser/pytokenize.py pypy/branch/jitffi/pypy/interpreter/test/test_function.py pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py pypy/branch/jitffi/pypy/jit/backend/llsupport/gc.py pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_gc.py pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py pypy/branch/jitffi/pypy/jit/backend/test/test_ll_random.py pypy/branch/jitffi/pypy/jit/backend/test/test_random.py pypy/branch/jitffi/pypy/jit/backend/x86/assembler.py pypy/branch/jitffi/pypy/jit/backend/x86/codebuf.py pypy/branch/jitffi/pypy/jit/backend/x86/regalloc.py pypy/branch/jitffi/pypy/jit/backend/x86/regloc.py pypy/branch/jitffi/pypy/jit/backend/x86/runner.py pypy/branch/jitffi/pypy/jit/backend/x86/rx86.py pypy/branch/jitffi/pypy/jit/backend/x86/test/conftest.py pypy/branch/jitffi/pypy/jit/backend/x86/test/test_regloc.py pypy/branch/jitffi/pypy/jit/backend/x86/test/test_zrpy_gc.py pypy/branch/jitffi/pypy/jit/backend/x86/test/test_ztranslation.py pypy/branch/jitffi/pypy/jit/codewriter/assembler.py pypy/branch/jitffi/pypy/jit/codewriter/call.py pypy/branch/jitffi/pypy/jit/codewriter/codewriter.py pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py pypy/branch/jitffi/pypy/jit/codewriter/support.py pypy/branch/jitffi/pypy/jit/codewriter/test/test_jtransform.py pypy/branch/jitffi/pypy/jit/codewriter/test/test_list.py pypy/branch/jitffi/pypy/jit/codewriter/test/test_support.py pypy/branch/jitffi/pypy/jit/metainterp/blackhole.py pypy/branch/jitffi/pypy/jit/metainterp/executor.py pypy/branch/jitffi/pypy/jit/metainterp/gc.py pypy/branch/jitffi/pypy/jit/metainterp/history.py pypy/branch/jitffi/pypy/jit/metainterp/optimizefindnode.py pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/optimizer.py (contents, props changed) pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/rewrite.py pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/virtualize.py pypy/branch/jitffi/pypy/jit/metainterp/optimizeutil.py pypy/branch/jitffi/pypy/jit/metainterp/pyjitpl.py pypy/branch/jitffi/pypy/jit/metainterp/resoperation.py pypy/branch/jitffi/pypy/jit/metainterp/resume.py pypy/branch/jitffi/pypy/jit/metainterp/simple_optimize.py pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_oparser.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefindnode.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_resume.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_string.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_ztranslation.py pypy/branch/jitffi/pypy/jit/metainterp/warmspot.py pypy/branch/jitffi/pypy/jit/metainterp/warmstate.py pypy/branch/jitffi/pypy/jit/tl/spli/interpreter.py pypy/branch/jitffi/pypy/jit/tool/loopviewer.py pypy/branch/jitffi/pypy/module/array/benchmark/Makefile (props changed) pypy/branch/jitffi/pypy/module/array/benchmark/intimg.c (props changed) pypy/branch/jitffi/pypy/module/array/benchmark/intimgtst.c (props changed) pypy/branch/jitffi/pypy/module/array/benchmark/intimgtst.py (props changed) pypy/branch/jitffi/pypy/module/array/benchmark/loop.c (props changed) pypy/branch/jitffi/pypy/module/array/benchmark/sum.c (props changed) pypy/branch/jitffi/pypy/module/array/benchmark/sumtst.c (contents, props changed) pypy/branch/jitffi/pypy/module/array/benchmark/sumtst.py (props changed) pypy/branch/jitffi/pypy/module/array/test/test_array_old.py (props changed) pypy/branch/jitffi/pypy/module/imp/importing.py pypy/branch/jitffi/pypy/module/imp/test/test_import.py pypy/branch/jitffi/pypy/module/parser/test/test_parser.py pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py pypy/branch/jitffi/pypy/module/sys/__init__.py pypy/branch/jitffi/pypy/module/sys/state.py pypy/branch/jitffi/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py pypy/branch/jitffi/pypy/objspace/std/fake.py pypy/branch/jitffi/pypy/objspace/std/objspace.py pypy/branch/jitffi/pypy/objspace/std/stringtype.py pypy/branch/jitffi/pypy/rlib/jit.py pypy/branch/jitffi/pypy/rlib/rmmap.py pypy/branch/jitffi/pypy/rlib/rstring.py pypy/branch/jitffi/pypy/rlib/test/test_rstring.py pypy/branch/jitffi/pypy/rpython/annlowlevel.py pypy/branch/jitffi/pypy/rpython/lltypesystem/ll2ctypes.py pypy/branch/jitffi/pypy/rpython/lltypesystem/llarena.py pypy/branch/jitffi/pypy/rpython/lltypesystem/rlist.py pypy/branch/jitffi/pypy/rpython/lltypesystem/rstr.py pypy/branch/jitffi/pypy/rpython/lltypesystem/test/test_ll2ctypes.py pypy/branch/jitffi/pypy/rpython/memory/gc/base.py pypy/branch/jitffi/pypy/rpython/memory/gc/generation.py pypy/branch/jitffi/pypy/rpython/memory/gc/markcompact.py pypy/branch/jitffi/pypy/rpython/memory/gc/marksweep.py pypy/branch/jitffi/pypy/rpython/memory/gc/minimark.py pypy/branch/jitffi/pypy/rpython/memory/gc/minimarkpage.py pypy/branch/jitffi/pypy/rpython/memory/gc/semispace.py pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_direct.py pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_minimark.py pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_minimarkpage.py pypy/branch/jitffi/pypy/rpython/memory/gctransform/framework.py pypy/branch/jitffi/pypy/rpython/memory/gcwrapper.py pypy/branch/jitffi/pypy/rpython/memory/test/test_gc.py pypy/branch/jitffi/pypy/rpython/memory/test/test_transformed_gc.py pypy/branch/jitffi/pypy/rpython/rtyper.py pypy/branch/jitffi/pypy/tool/progressbar.py pypy/branch/jitffi/pypy/translator/c/gcc/instruction.py pypy/branch/jitffi/pypy/translator/c/gcc/trackgcroot.py pypy/branch/jitffi/pypy/translator/c/test/test_lltyped.py pypy/branch/jitffi/pypy/translator/driver.py pypy/branch/jitffi/pypy/translator/goal/ann_override.py pypy/branch/jitffi/pypy/translator/goal/translate.py pypy/branch/jitffi/pypy/translator/jvm/test/test_class.py pypy/branch/jitffi/pypy/translator/oosupport/test_template/class_.py Log: merge from trunk: svn merge svn+ssh://codespeak.net/svn/pypy/trunk -r77303:HEAD Modified: pypy/branch/jitffi/lib_pypy/_ctypes/function.py ============================================================================== --- pypy/branch/jitffi/lib_pypy/_ctypes/function.py (original) +++ pypy/branch/jitffi/lib_pypy/_ctypes/function.py Fri Oct 8 13:40:10 2010 @@ -171,7 +171,7 @@ return self._build_result(restype, resbuffer, argtypes, argsandobjs) def _getfuncptr(self, argtypes, restype, thisarg=None): - if self._ptr is not None: + if self._ptr is not None and argtypes is self._argtypes_: return self._ptr if restype is None or not isinstance(restype, _CDataMeta): import ctypes Modified: pypy/branch/jitffi/pypy/annotation/policy.py ============================================================================== --- pypy/branch/jitffi/pypy/annotation/policy.py (original) +++ pypy/branch/jitffi/pypy/annotation/policy.py Fri Oct 8 13:40:10 2010 @@ -1,4 +1,4 @@ -# base annotation policy for overrides and specialization +# base annotation policy for specialization from pypy.annotation.specialize import default_specialize as default from pypy.annotation.specialize import specialize_argvalue, specialize_argtype, specialize_arglistitemtype from pypy.annotation.specialize import memo @@ -41,7 +41,7 @@ if directive is None: return pol.default_specialize - # specialize|override:name[(args)] + # specialize[(args)] directive_parts = directive.split('(', 1) if len(directive_parts) == 1: [name] = directive_parts @@ -60,14 +60,6 @@ except AttributeError: raise AttributeError("%r specialize tag not defined in annotation" "policy %s" % (name, pol)) - if directive.startswith('override:'): - # different signature: override__xyz(*args_s) - if parms: - raise Exception, "override:* specialisations don't support parameters" - def specialize_override(funcdesc, args_s): - funcdesc.overridden = True - return specializer(*args_s) - return specialize_override else: if not parms: return specializer @@ -92,9 +84,5 @@ from pypy.rpython.annlowlevel import LowLevelAnnotatorPolicy return LowLevelAnnotatorPolicy.specialize__ll_and_arg(*args) - def override__ignore(pol, *args): - bk = getbookkeeper() - return bk.immutablevalue(None) - class StrictAnnotatorPolicy(AnnotatorPolicy): allow_someobjects = False Modified: pypy/branch/jitffi/pypy/annotation/test/test_annrpython.py ============================================================================== --- pypy/branch/jitffi/pypy/annotation/test/test_annrpython.py (original) +++ pypy/branch/jitffi/pypy/annotation/test/test_annrpython.py Fri Oct 8 13:40:10 2010 @@ -766,28 +766,6 @@ s = a.build_types(f, [list]) assert s.classdef is a.bookkeeper.getuniqueclassdef(IndexError) # KeyError ignored because l is a list - def test_overrides(self): - excs = [] - def record_exc(e): - """NOT_RPYTHON""" - excs.append(sys.exc_info) - record_exc._annspecialcase_ = "override:record_exc" - def g(): - pass - def f(): - try: - g() - except Exception, e: - record_exc(e) - class MyAnnotatorPolicy(policy.AnnotatorPolicy): - - def override__record_exc(pol, s_e): - return a.bookkeeper.immutablevalue(None) - - a = self.RPythonAnnotator(policy=MyAnnotatorPolicy()) - s = a.build_types(f, []) - assert s.const is None - def test_freeze_protocol(self): class Stuff: def __init__(self, flag): Modified: pypy/branch/jitffi/pypy/config/test/test_pypyoption.py ============================================================================== --- pypy/branch/jitffi/pypy/config/test/test_pypyoption.py (original) +++ pypy/branch/jitffi/pypy/config/test/test_pypyoption.py Fri Oct 8 13:40:10 2010 @@ -41,7 +41,7 @@ assert not conf.translation.backendopt.none conf = get_pypy_config() set_opt_level(conf, 'mem') - assert conf.translation.gc == 'markcompact' + assert conf.translation.gcremovetypeptr assert not conf.translation.backendopt.none def test_set_pypy_opt_level(): Modified: pypy/branch/jitffi/pypy/config/translationoption.py ============================================================================== --- pypy/branch/jitffi/pypy/config/translationoption.py (original) +++ pypy/branch/jitffi/pypy/config/translationoption.py Fri Oct 8 13:40:10 2010 @@ -11,6 +11,8 @@ DEFL_CLEVER_MALLOC_REMOVAL_INLINE_THRESHOLD = 32.4 DEFL_LOW_INLINE_THRESHOLD = DEFL_INLINE_THRESHOLD / 2.0 +DEFL_GC = "minimark" + IS_64_BITS = sys.maxint > 2147483647 PLATFORMS = [ @@ -105,7 +107,7 @@ # JIT generation: use -Ojit to enable it BoolOption("jit", "generate a JIT", default=False, - suggests=[("translation.gc", "hybrid"), + suggests=[("translation.gc", DEFL_GC), ("translation.gcrootfinder", "asmgcc"), ("translation.list_comprehension_operations", True)]), ChoiceOption("jit_backend", "choose the backend for the JIT", @@ -337,10 +339,10 @@ '0': 'boehm nobackendopt', '1': 'boehm lowinline', 'size': 'boehm lowinline remove_asserts', - 'mem': 'markcompact lowinline remove_asserts removetypeptr', - '2': 'hybrid extraopts', - '3': 'hybrid extraopts remove_asserts', - 'jit': 'hybrid extraopts jit', + 'mem': DEFL_GC + ' lowinline remove_asserts removetypeptr', + '2': DEFL_GC + ' extraopts', + '3': DEFL_GC + ' extraopts remove_asserts', + 'jit': DEFL_GC + ' extraopts jit', } def final_check_config(config): Modified: pypy/branch/jitffi/pypy/doc/docindex.txt ============================================================================== --- pypy/branch/jitffi/pypy/doc/docindex.txt (original) +++ pypy/branch/jitffi/pypy/doc/docindex.txt Fri Oct 8 13:40:10 2010 @@ -84,7 +84,7 @@ PyPy's own tests `summary`_, daily updated, run through BuildBot infrastructure. You can also find CPython's compliance tests run with compiled ``pypy-c`` -exeuctables there. +executables there. information dating from early 2007: Modified: pypy/branch/jitffi/pypy/interpreter/astcompiler/test/test_compiler.py ============================================================================== --- pypy/branch/jitffi/pypy/interpreter/astcompiler/test/test_compiler.py (original) +++ pypy/branch/jitffi/pypy/interpreter/astcompiler/test/test_compiler.py Fri Oct 8 13:40:10 2010 @@ -41,7 +41,7 @@ co_expr = compile(evalexpr, '', 'eval') space = self.space pyco_expr = PyCode._from_code(space, co_expr) - w_res = pyco_expr.exec_code(space, w_dict, w_dict) + w_res = pyco_expr.exec_host_bytecode(space, w_dict, w_dict) res = space.str_w(space.repr(w_res)) if not isinstance(expected, float): assert res == repr(expected) Modified: pypy/branch/jitffi/pypy/interpreter/baseobjspace.py ============================================================================== --- pypy/branch/jitffi/pypy/interpreter/baseobjspace.py (original) +++ pypy/branch/jitffi/pypy/interpreter/baseobjspace.py Fri Oct 8 13:40:10 2010 @@ -12,6 +12,7 @@ from pypy.rlib.timer import DummyTimer, Timer from pypy.rlib.rarithmetic import r_uint from pypy.rlib import jit +from pypy.tool.sourcetools import func_with_new_name import os, sys, py __all__ = ['ObjSpace', 'OperationError', 'Wrappable', 'W_Root'] @@ -749,12 +750,17 @@ (i, plural))) return items + unpackiterable_unroll = jit.unroll_safe(func_with_new_name(unpackiterable, + 'unpackiterable_unroll')) + def fixedview(self, w_iterable, expected_length=-1): """ A fixed list view of w_iterable. Don't modify the result """ return make_sure_not_resized(self.unpackiterable(w_iterable, expected_length)[:]) + fixedview_unroll = fixedview + def listview(self, w_iterable, expected_length=-1): """ A non-fixed view of w_iterable. Don't modify the result """ Modified: pypy/branch/jitffi/pypy/interpreter/function.py ============================================================================== --- pypy/branch/jitffi/pypy/interpreter/function.py (original) +++ pypy/branch/jitffi/pypy/interpreter/function.py Fri Oct 8 13:40:10 2010 @@ -430,8 +430,11 @@ self.w_class = w_class # possibly space.w_None def descr_method__new__(space, w_subtype, w_function, w_instance, w_class=None): - if space.is_w( w_instance, space.w_None ): + if space.is_w(w_instance, space.w_None): w_instance = None + if w_instance is None and space.is_w(w_class, space.w_None): + raise OperationError(space.w_TypeError, + space.wrap("unbound methods must have class")) method = space.allocate_instance(Method, w_subtype) Method.__init__(method, space, w_function, w_instance, w_class) return space.wrap(method) Modified: pypy/branch/jitffi/pypy/interpreter/pycode.py ============================================================================== --- pypy/branch/jitffi/pypy/interpreter/pycode.py (original) +++ pypy/branch/jitffi/pypy/interpreter/pycode.py Fri Oct 8 13:40:10 2010 @@ -253,6 +253,12 @@ tuple(self.co_freevars), tuple(self.co_cellvars) ) + def exec_host_bytecode(self, w_dict, w_globals, w_locals): + from pypy.interpreter.pyframe import CPythonFrame + frame = CPythonFrame(self.space, self, w_globals, None) + frame.setdictscope(w_locals) + return frame.run() + def dump(self): """A dis.dis() dump of the code object.""" co = self._to_code() Modified: pypy/branch/jitffi/pypy/interpreter/pyframe.py ============================================================================== --- pypy/branch/jitffi/pypy/interpreter/pyframe.py (original) +++ pypy/branch/jitffi/pypy/interpreter/pyframe.py Fri Oct 8 13:40:10 2010 @@ -13,6 +13,7 @@ from pypy.rlib.rarithmetic import intmask from pypy.rlib import jit, rstack from pypy.tool import stdlib_opcode +from pypy.tool.stdlib_opcode import host_bytecode_spec # Define some opcodes used g = globals() @@ -140,7 +141,8 @@ # the following 'assert' is an annotation hint: it hides from # the annotator all methods that are defined in PyFrame but # overridden in the {,Host}FrameClass subclasses of PyFrame. - assert isinstance(self, self.space.FrameClass) + assert (isinstance(self, self.space.FrameClass) or + not self.space.config.translating) executioncontext = self.space.getexecutioncontext() executioncontext.enter(self) try: @@ -634,6 +636,18 @@ return space.wrap(self.builtin is not space.builtin) return space.w_False +class CPythonFrame(PyFrame): + """ + Execution of host (CPython) opcodes. + """ + + bytecode_spec = host_bytecode_spec + opcode_method_names = host_bytecode_spec.method_names + opcodedesc = host_bytecode_spec.opcodedesc + opdescmap = host_bytecode_spec.opdescmap + HAVE_ARGUMENT = host_bytecode_spec.HAVE_ARGUMENT + + # ____________________________________________________________ def get_block_class(opname): Modified: pypy/branch/jitffi/pypy/interpreter/pyopcode.py ============================================================================== --- pypy/branch/jitffi/pypy/interpreter/pyopcode.py (original) +++ pypy/branch/jitffi/pypy/interpreter/pyopcode.py Fri Oct 8 13:40:10 2010 @@ -637,7 +637,7 @@ def UNPACK_SEQUENCE(self, itemcount, next_instr): w_iterable = self.popvalue() - items = self.space.fixedview(w_iterable, itemcount) + items = self.space.fixedview_unroll(w_iterable, itemcount) self.pushrevvalues(itemcount, items) def STORE_ATTR(self, nameindex, next_instr): Modified: pypy/branch/jitffi/pypy/interpreter/pyparser/pytokenize.py ============================================================================== --- pypy/branch/jitffi/pypy/interpreter/pyparser/pytokenize.py (original) +++ pypy/branch/jitffi/pypy/interpreter/pyparser/pytokenize.py Fri Oct 8 13:40:10 2010 @@ -21,40 +21,42 @@ __all__ = [ "tokenize" ] # ______________________________________________________________________ -# Automatically generated DFA's (with one or two hand tweeks): -pseudoStatesAccepts = [True, True, True, True, True, True, True, True, - True, True, False, True, True, True, False, False, - False, False, True, False, False, True, True, False, - True, False, True, False, True, False, True, False, - False, False, True, False, False, False, True] - -pseudoStates = [ - {'\t': 0, '\n': 13, '\x0c': 0, '\r': 14, ' ': 0, '!': 10, - '"': 16, '#': 18, '%': 12, '&': 12, - "'": 15, '(': 13, ')': 13, '*': 7, - '+': 12, ',': 13, '-': 12, '.': 6, - '/': 11, '0': 4, '1': 5, '2': 5, - '3': 5, '4': 5, '5': 5, '6': 5, - '7': 5, '8': 5, '9': 5, ':': 13, - ';': 13, '<': 9, '=': 12, '>': 8, - '@': 13, 'A': 1, - 'B': 1, 'C': 1, 'D': 1, 'E': 1, - 'F': 1, 'G': 1, 'H': 1, 'I': 1, - 'J': 1, 'K': 1, 'L': 1, 'M': 1, - 'N': 1, 'O': 1, 'P': 1, 'Q': 1, - 'R': 2, 'S': 1, 'T': 1, 'U': 3, - 'V': 1, 'W': 1, 'X': 1, 'Y': 1, - 'Z': 1, '[': 13, '\\': 17, ']': 13, - '^': 12, '_': 1, '`': 13, 'a': 1, - 'b': 1, 'c': 1, 'd': 1, 'e': 1, - 'f': 1, 'g': 1, 'h': 1, 'i': 1, - 'j': 1, 'k': 1, 'l': 1, 'm': 1, - 'n': 1, 'o': 1, 'p': 1, 'q': 1, - 'r': 2, 's': 1, 't': 1, 'u': 3, - 'v': 1, 'w': 1, 'x': 1, 'y': 1, - 'z': 1, '{': 13, '|': 12, '}': 13, - '~': 13}, +# Automatically generated DFA's +accepts = [False, True, True, True, True, True, True, True, + True, True, False, True, True, True, False, False, + False, False, True, True, True, False, True, + False, True, False, True, False, False, True, + False, False, False, False, True, False, False, + False, True] +states = [ + # 0 + {'\t': 0, '\n': 13, '\x0c': 0, + '\r': 14, ' ': 0, '!': 10, '"': 16, + '#': 18, '%': 12, '&': 12, "'": 15, + '(': 13, ')': 13, '*': 7, '+': 12, + ',': 13, '-': 12, '.': 6, '/': 11, + '0': 4, '1': 5, '2': 5, '3': 5, + '4': 5, '5': 5, '6': 5, '7': 5, + '8': 5, '9': 5, ':': 13, ';': 13, + '<': 9, '=': 12, '>': 8, '@': 13, + 'A': 1, 'B': 1, 'C': 1, 'D': 1, + 'E': 1, 'F': 1, 'G': 1, 'H': 1, + 'I': 1, 'J': 1, 'K': 1, 'L': 1, + 'M': 1, 'N': 1, 'O': 1, 'P': 1, + 'Q': 1, 'R': 2, 'S': 1, 'T': 1, + 'U': 3, 'V': 1, 'W': 1, 'X': 1, + 'Y': 1, 'Z': 1, '[': 13, '\\': 17, + ']': 13, '^': 12, '_': 1, '`': 13, + 'a': 1, 'b': 1, 'c': 1, 'd': 1, + 'e': 1, 'f': 1, 'g': 1, 'h': 1, + 'i': 1, 'j': 1, 'k': 1, 'l': 1, + 'm': 1, 'n': 1, 'o': 1, 'p': 1, + 'q': 1, 'r': 2, 's': 1, 't': 1, + 'u': 3, 'v': 1, 'w': 1, 'x': 1, + 'y': 1, 'z': 1, '{': 13, '|': 12, + '}': 13, '~': 13}, + # 1 {'0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, 'A': 1, 'B': 1, @@ -71,7 +73,7 @@ 'p': 1, 'q': 1, 'r': 1, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - + # 2 {'"': 16, "'": 15, '0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, @@ -89,7 +91,7 @@ 'r': 1, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - + # 3 {'"': 16, "'": 15, '0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, @@ -107,158 +109,182 @@ 'r': 2, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - - {'.': 24, '0': 22, '1': 22, '2': 22, - '3': 22, '4': 22, '5': 22, '6': 22, - '7': 22, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'L': 13, 'X': 21, 'e': 25, - 'j': 13, 'l': 13, 'x': 21}, - - {'.': 24, '0': 5, '1': 5, '2': 5, + # 4 + {'.': 22, '0': 20, '1': 20, '2': 20, + '3': 20, '4': 20, '5': 20, '6': 20, + '7': 20, '8': 21, '9': 21, 'E': 23, + 'J': 13, 'L': 13, 'X': 19, 'e': 23, + 'j': 13, 'l': 13, 'x': 19}, + # 5 + {'.': 22, '0': 5, '1': 5, '2': 5, '3': 5, '4': 5, '5': 5, '6': 5, - '7': 5, '8': 5, '9': 5, 'E': 25, - 'J': 13, 'L': 13, 'e': 25, 'j': 13, + '7': 5, '8': 5, '9': 5, 'E': 23, + 'J': 13, 'L': 13, 'e': 23, 'j': 13, 'l': 13}, - - {'0': 26, '1': 26, '2': 26, '3': 26, - '4': 26, '5': 26, '6': 26, '7': 26, - '8': 26, '9': 26}, - + # 6 + {'0': 24, '1': 24, '2': 24, '3': 24, + '4': 24, '5': 24, '6': 24, '7': 24, + '8': 24, '9': 24}, + # 7 {'*': 12, '=': 13}, - + # 8 {'=': 13, '>': 12}, - - {'=': 13, '<': 12, '>': 13}, - + # 9 + {'<': 12, '=': 13, '>': 13}, + # 10 {'=': 13}, - - {'=': 13, '/': 12}, - + # 11 + {'/': 12, '=': 13}, + # 12 {'=': 13}, - + # 13 {}, - + # 14 {'\n': 13}, - - {automata.DEFAULT: 19, '\n': 27, '\\': 29, "'": 28}, - - {automata.DEFAULT: 20, '"': 30, '\n': 27, '\\': 31}, - + # 15 + {automata.DEFAULT: 28, '\n': 25, "'": 26, '\\': 27}, + # 16 + {automata.DEFAULT: 31, '\n': 25, '"': 29, '\\': 30}, + # 17 {'\n': 13, '\r': 14}, - - {automata.DEFAULT: 18, '\n': 27, '\r': 27}, - - {automata.DEFAULT: 19, '\n': 27, '\\': 29, "'": 13}, - - {automata.DEFAULT: 20, '"': 13, '\n': 27, '\\': 31}, - - {'0': 21, '1': 21, '2': 21, '3': 21, - '4': 21, '5': 21, '6': 21, '7': 21, - '8': 21, '9': 21, 'A': 21, 'B': 21, - 'C': 21, 'D': 21, 'E': 21, 'F': 21, - 'L': 13, 'a': 21, 'b': 21, 'c': 21, - 'd': 21, 'e': 21, 'f': 21, 'l': 13}, - - {'.': 24, '0': 22, '1': 22, '2': 22, - '3': 22, '4': 22, '5': 22, '6': 22, - '7': 22, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'L': 13, 'e': 25, 'j': 13, + # 18 + {automata.DEFAULT: 18, '\n': 25, '\r': 25}, + # 19 + {'0': 19, '1': 19, '2': 19, '3': 19, + '4': 19, '5': 19, '6': 19, '7': 19, + '8': 19, '9': 19, 'A': 19, 'B': 19, + 'C': 19, 'D': 19, 'E': 19, 'F': 19, + 'L': 13, 'a': 19, 'b': 19, 'c': 19, + 'd': 19, 'e': 19, 'f': 19, 'l': 13}, + # 20 + {'.': 22, '0': 20, '1': 20, '2': 20, + '3': 20, '4': 20, '5': 20, '6': 20, + '7': 20, '8': 21, '9': 21, 'E': 23, + 'J': 13, 'L': 13, 'e': 23, 'j': 13, 'l': 13}, - - {'.': 24, '0': 23, '1': 23, '2': 23, - '3': 23, '4': 23, '5': 23, '6': 23, - '7': 23, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'e': 25, 'j': 13}, - - {'0': 24, '1': 24, '2': 24, '3': 24, - '4': 24, '5': 24, '6': 24, '7': 24, - '8': 24, '9': 24, 'E': 32, 'J': 13, + # 21 + {'.': 22, '0': 21, '1': 21, '2': 21, + '3': 21, '4': 21, '5': 21, '6': 21, + '7': 21, '8': 21, '9': 21, 'E': 23, + 'J': 13, 'e': 23, 'j': 13}, + # 22 + {'0': 22, '1': 22, '2': 22, '3': 22, + '4': 22, '5': 22, '6': 22, '7': 22, + '8': 22, '9': 22, 'E': 32, 'J': 13, 'e': 32, 'j': 13}, - + # 23 {'+': 33, '-': 33, '0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34}, - - {'0': 26, '1': 26, '2': 26, '3': 26, - '4': 26, '5': 26, '6': 26, '7': 26, - '8': 26, '9': 26, 'E': 32, 'J': 13, + # 24 + {'0': 24, '1': 24, '2': 24, '3': 24, + '4': 24, '5': 24, '6': 24, '7': 24, + '8': 24, '9': 24, 'E': 32, 'J': 13, 'e': 32, 'j': 13}, - + # 25 {}, - + # 26 {"'": 13}, - + # 27 {automata.DEFAULT: 35, '\n': 13, '\r': 14}, - + # 28 + {automata.DEFAULT: 28, '\n': 25, "'": 13, '\\': 27}, + # 29 {'"': 13}, - + # 30 {automata.DEFAULT: 36, '\n': 13, '\r': 14}, - + # 31 + {automata.DEFAULT: 31, '\n': 25, '"': 13, '\\': 30}, + # 32 {'+': 37, '-': 37, '0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38}, - - + # 33 {'0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34}, - + # 34 {'0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34, 'J': 13, 'j': 13}, - - {automata.DEFAULT: 35, '\n': 27, '\\': 29, "'": 13}, - - {automata.DEFAULT: 36, '"': 13, '\n': 27, '\\': 31}, - + # 35 + {automata.DEFAULT: 35, '\n': 25, "'": 13, '\\': 27}, + # 36 + {automata.DEFAULT: 36, '\n': 25, '"': 13, '\\': 30}, + # 37 {'0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38}, - + # 38 {'0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38, 'J': 13, 'j': 13}, ] +pseudoDFA = automata.DFA(states, accepts) -pseudoDFA = automata.DFA(pseudoStates, pseudoStatesAccepts) - -double3StatesAccepts = [False, False, False, False, False, True] -double3States = [ +accepts = [False, False, False, False, False, True] +states = [ + # 0 {automata.DEFAULT: 0, '"': 1, '\\': 2}, + # 1 {automata.DEFAULT: 4, '"': 3, '\\': 2}, + # 2 {automata.DEFAULT: 4}, + # 3 {automata.DEFAULT: 4, '"': 5, '\\': 2}, + # 4 {automata.DEFAULT: 4, '"': 1, '\\': 2}, + # 5 {automata.DEFAULT: 4, '"': 5, '\\': 2}, ] -double3DFA = automata.NonGreedyDFA(double3States, double3StatesAccepts) +double3DFA = automata.NonGreedyDFA(states, accepts) -single3StatesAccepts = [False, False, False, False, False, True] -single3States = [ - {automata.DEFAULT: 0, '\\': 2, "'": 1}, - {automata.DEFAULT: 4, '\\': 2, "'": 3}, +accepts = [False, False, False, False, False, True] +states = [ + # 0 + {automata.DEFAULT: 0, "'": 1, '\\': 2}, + # 1 + {automata.DEFAULT: 4, "'": 3, '\\': 2}, + # 2 {automata.DEFAULT: 4}, - {automata.DEFAULT: 4, '\\': 2, "'": 5}, - {automata.DEFAULT: 4, '\\': 2, "'": 1}, - {automata.DEFAULT: 4, '\\': 2, "'": 5}, + # 3 + {automata.DEFAULT: 4, "'": 5, '\\': 2}, + # 4 + {automata.DEFAULT: 4, "'": 1, '\\': 2}, + # 5 + {automata.DEFAULT: 4, "'": 5, '\\': 2}, ] -single3DFA = automata.NonGreedyDFA(single3States, single3StatesAccepts) +single3DFA = automata.NonGreedyDFA(states, accepts) -singleStatesAccepts = [False, True, False] -singleStates = [ - {automata.DEFAULT: 0, '\\': 2, "'": 1}, +accepts = [False, True, False, False] +states = [ + # 0 + {automata.DEFAULT: 0, "'": 1, '\\': 2}, + # 1 {}, - {automata.DEFAULT: 0}, + # 2 + {automata.DEFAULT: 3}, + # 3 + {automata.DEFAULT: 3, "'": 1, '\\': 2}, ] -singleDFA = automata.DFA(singleStates, singleStatesAccepts) +singleDFA = automata.DFA(states, accepts) -doubleStatesAccepts = [False, True, False] -doubleStates = [ +accepts = [False, True, False, False] +states = [ + # 0 {automata.DEFAULT: 0, '"': 1, '\\': 2}, + # 1 {}, - {automata.DEFAULT: 0}, + # 2 + {automata.DEFAULT: 3}, + # 3 + {automata.DEFAULT: 3, '"': 1, '\\': 2}, ] -doubleDFA = automata.DFA(doubleStates, doubleStatesAccepts) +doubleDFA = automata.DFA(states, accepts) + + +#_______________________________________________________________________ +# End of automatically generated DFA's endDFAs = {"'" : singleDFA, '"' : doubleDFA, Modified: pypy/branch/jitffi/pypy/interpreter/test/test_function.py ============================================================================== --- pypy/branch/jitffi/pypy/interpreter/test/test_function.py (original) +++ pypy/branch/jitffi/pypy/interpreter/test/test_function.py Fri Oct 8 13:40:10 2010 @@ -482,6 +482,11 @@ raises(TypeError, m, MyInst(None)) raises(TypeError, m, MyInst(42)) + def test_invalid_creation(self): + import new + def f(): pass + raises(TypeError, new.instancemethod, f, None) + class TestMethod: def setup_method(self, method): Modified: pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llgraph/llimpl.py Fri Oct 8 13:40:10 2010 @@ -1385,6 +1385,20 @@ uni = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), string) uni.chars[index] = unichr(newvalue) +def do_copystrcontent(src, dst, srcstart, dststart, length): + src = lltype.cast_opaque_ptr(lltype.Ptr(rstr.STR), src) + dst = lltype.cast_opaque_ptr(lltype.Ptr(rstr.STR), dst) + assert 0 <= srcstart <= srcstart + length <= len(src.chars) + assert 0 <= dststart <= dststart + length <= len(dst.chars) + rstr.copy_string_contents(src, dst, srcstart, dststart, length) + +def do_copyunicodecontent(src, dst, srcstart, dststart, length): + src = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), src) + dst = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), dst) + assert 0 <= srcstart <= srcstart + length <= len(src.chars) + assert 0 <= dststart <= dststart + length <= len(dst.chars) + rstr.copy_unicode_contents(src, dst, srcstart, dststart, length) + # ---------- call ---------- _call_args_i = [] Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/gc.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llsupport/gc.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llsupport/gc.py Fri Oct 8 13:40:10 2010 @@ -133,7 +133,7 @@ # ____________________________________________________________ -# All code below is for the hybrid GC +# All code below is for the hybrid or minimark GC class GcRefList: @@ -167,7 +167,7 @@ def alloc_gcref_list(self, n): # Important: the GRREF_LISTs allocated are *non-movable*. This - # requires support in the gc (only the hybrid GC supports it so far). + # requires support in the gc (hybrid GC or minimark GC so far). if we_are_translated(): list = rgc.malloc_nonmovable(self.GCREF_LIST, n) assert list, "malloc_nonmovable failed!" @@ -350,8 +350,9 @@ self.translator = translator self.llop1 = llop1 - # we need the hybrid GC for GcRefList.alloc_gcref_list() to work - if gcdescr.config.translation.gc != 'hybrid': + # we need the hybrid or minimark GC for GcRefList.alloc_gcref_list() + # to work + if gcdescr.config.translation.gc not in ('hybrid', 'minimark'): raise NotImplementedError("--gc=%s not implemented with the JIT" % (gcdescr.config.translation.gc,)) @@ -382,8 +383,7 @@ self.gcheaderbuilder = GCHeaderBuilder(self.HDRPTR.TO) (self.array_basesize, _, self.array_length_ofs) = \ symbolic.get_array_token(lltype.GcArray(lltype.Signed), True) - min_ns = self.GCClass.TRANSLATION_PARAMS['min_nursery_size'] - self.max_size_of_young_obj = self.GCClass.get_young_fixedsize(min_ns) + self.max_size_of_young_obj = self.GCClass.JIT_max_size_of_young_obj() # make a malloc function, with three arguments def malloc_basic(size, tid): @@ -597,6 +597,8 @@ v = op.getarg(2) if isinstance(v, BoxPtr) or (isinstance(v, ConstPtr) and bool(v.value)): # store a non-NULL + # XXX detect when we should produce a + # write_barrier_from_array self._gen_write_barrier(newops, op.getarg(0), v) op = op.copy_and_change(rop.SETARRAYITEM_RAW) # ---------- Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_gc.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_gc.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_gc.py Fri Oct 8 13:40:10 2010 @@ -149,11 +149,12 @@ class TestFramework: + gc = 'hybrid' def setup_method(self, meth): class config_: class translation: - gc = 'hybrid' + gc = self.gc gcrootfinder = 'asmgcc' gctransformer = 'framework' gcremovetypeptr = False @@ -392,3 +393,7 @@ assert operations[1].getarg(1) == v_index assert operations[1].getarg(2) == v_value assert operations[1].getdescr() == array_descr + + +class TestFrameworkMiniMark(TestFramework): + gc = 'minimark' Modified: pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py Fri Oct 8 13:40:10 2010 @@ -842,6 +842,23 @@ r = self.execute_operation(rop.STRGETITEM, [s_box, BoxInt(4)], 'int') assert r.value == 153 + def test_copystrcontent(self): + s_box = self.alloc_string("abcdef") + for s_box in [s_box, s_box.constbox()]: + for srcstart_box in [BoxInt(2), ConstInt(2)]: + for dststart_box in [BoxInt(3), ConstInt(3)]: + for length_box in [BoxInt(4), ConstInt(4)]: + for r_box_is_const in [False, True]: + r_box = self.alloc_string("!???????!") + if r_box_is_const: + r_box = r_box.constbox() + self.execute_operation(rop.COPYSTRCONTENT, + [s_box, r_box, + srcstart_box, + dststart_box, + length_box], 'void') + assert self.look_string(r_box) == "!??cdef?!" + def test_do_unicode_basic(self): u = self.cpu.bh_newunicode(5) self.cpu.bh_unicodesetitem(u, 4, 123) @@ -1225,6 +1242,10 @@ s_box = BoxPtr(lltype.cast_opaque_ptr(llmemory.GCREF, s)) return s_box + def look_string(self, string_box): + s = string_box.getref(lltype.Ptr(rstr.STR)) + return ''.join(s.chars) + def alloc_unicode(self, unicode): u = rstr.mallocunicode(len(unicode)) for i in range(len(unicode)): Modified: pypy/branch/jitffi/pypy/jit/backend/test/test_ll_random.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/test/test_ll_random.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/test/test_ll_random.py Fri Oct 8 13:40:10 2010 @@ -386,6 +386,20 @@ v_string = self.get_string(builder, r) builder.do(self.opnum, [v_string]) +class AbstractCopyContentOperation(AbstractStringOperation): + def produce_into(self, builder, r): + v_srcstring = self.get_string(builder, r) + v_dststring = self.get_string(builder, r) + if v_srcstring.value == v_dststring.value: # because it's not a + raise test_random.CannotProduceOperation # memmove(), but memcpy() + srclen = len(v_srcstring.getref(self.ptr).chars) + dstlen = len(v_dststring.getref(self.ptr).chars) + v_length = builder.get_index(min(srclen, dstlen), r) + v_srcstart = builder.get_index(srclen - v_length.value + 1, r) + v_dststart = builder.get_index(dstlen - v_length.value + 1, r) + builder.do(self.opnum, [v_srcstring, v_dststring, + v_srcstart, v_dststart, v_length]) + class StrGetItemOperation(AbstractGetItemOperation, _StrOperation): pass @@ -404,6 +418,13 @@ class UnicodeLenOperation(AbstractStringLenOperation, _UnicodeOperation): pass +class CopyStrContentOperation(AbstractCopyContentOperation, _StrOperation): + pass + +class CopyUnicodeContentOperation(AbstractCopyContentOperation, + _UnicodeOperation): + pass + # there are five options in total: # 1. non raising call and guard_no_exception @@ -577,6 +598,8 @@ OPERATIONS.append(UnicodeSetItemOperation(rop.UNICODESETITEM)) OPERATIONS.append(StrLenOperation(rop.STRLEN)) OPERATIONS.append(UnicodeLenOperation(rop.UNICODELEN)) + OPERATIONS.append(CopyStrContentOperation(rop.COPYSTRCONTENT)) + #OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT)) for i in range(2): OPERATIONS.append(GuardClassOperation(rop.GUARD_CLASS)) Modified: pypy/branch/jitffi/pypy/jit/backend/test/test_random.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/test/test_random.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/test/test_random.py Fri Oct 8 13:40:10 2010 @@ -1,7 +1,7 @@ import py, sys from pypy.rlib.rarithmetic import intmask, LONG_BIT from pypy.rpython.lltypesystem import llmemory -from pypy.jit.backend.test import conftest as demo_conftest +from pypy.jit.backend import conftest as demo_conftest from pypy.jit.metainterp.history import BasicFailDescr, TreeLoop from pypy.jit.metainterp.history import BoxInt, ConstInt, LoopToken from pypy.jit.metainterp.history import BoxPtr, ConstPtr @@ -102,7 +102,7 @@ elif isinstance(v, ConstFloat): args.append('ConstFloat(%r)' % v.value) elif isinstance(v, ConstInt): - args.append('ConstInt(%d)' % v.value) + args.append('ConstInt(%s)' % v.value) else: raise NotImplementedError(v) if op.getdescr() is None: @@ -113,7 +113,7 @@ except AttributeError: descrstr = ', descr=...' print >>s, ' ResOperation(rop.%s, [%s], %s%s),' % ( - opname[op.opnum], ', '.join(args), names[op.result], descrstr) + opname[op.getopnum()], ', '.join(args), names[op.result], descrstr) #if getattr(op, 'suboperations', None) is not None: # subops.append(op) @@ -189,7 +189,7 @@ v.value) print >>s, ' op = cpu.execute_token(looptoken)' if self.should_fail_by is None: - fail_args = self.loop.operations[-1].args + fail_args = self.loop.operations[-1].getarglist() else: fail_args = self.should_fail_by.getfailargs() for i, v in enumerate(fail_args): Modified: pypy/branch/jitffi/pypy/jit/backend/x86/assembler.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/x86/assembler.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/x86/assembler.py Fri Oct 8 13:40:10 2010 @@ -181,6 +181,7 @@ self.malloc_fixedsize_slowpath1 = 0 self.malloc_fixedsize_slowpath2 = 0 self.pending_guard_tokens = None + self.memcpy_addr = 0 self.setup_failure_recovery() self._debug = False self.debug_counter_descr = cpu.fielddescrof(DEBUG_COUNTER, 'i') @@ -212,6 +213,7 @@ ll_new_unicode = gc_ll_descr.get_funcptr_for_newunicode() self.malloc_unicode_func_addr = rffi.cast(lltype.Signed, ll_new_unicode) + self.memcpy_addr = self.cpu.cast_ptr_to_int(codebuf.memcpy_fn) self.mc = MachineCodeBlockWrapper(self, self.mc_size, self.cpu.profile_agent) self._build_failure_recovery(False) self._build_failure_recovery(True) @@ -242,7 +244,7 @@ f = open_file_as_stream(output_log, "w") for i in range(len(self.loop_run_counters)): name, struct = self.loop_run_counters[i] - f.write(str(struct.i) + " " * (8 - len(str(struct.i))) + name + "\n") + f.write(str(name) + ":" + str(struct.i) + "\n") f.close() def _build_float_constants(self): @@ -399,7 +401,7 @@ if self._debug: struct = lltype.malloc(DEBUG_COUNTER, flavor='raw') struct.i = 0 - self.loop_run_counters.append((funcname, struct)) + self.loop_run_counters.append((len(self.loop_run_counters), struct)) return funcname def patch_jump_for_descr(self, faildescr, adr_new_target): @@ -709,8 +711,8 @@ self.regalloc_perform_with_guard(None, guard_op, faillocs, arglocs, resloc, current_depths) - def load_effective_addr(self, sizereg, baseofs, scale, result): - self.mc.LEA(result, addr_add(imm(0), sizereg, baseofs, scale)) + def load_effective_addr(self, sizereg, baseofs, scale, result, frm=imm(0)): + self.mc.LEA(result, addr_add(frm, sizereg, baseofs, scale)) def _unaryop(asmop): def genop_unary(self, op, arglocs, resloc): @@ -1028,7 +1030,7 @@ if self.cpu.vtable_offset is not None: assert isinstance(loc, RegLoc) assert isinstance(loc_vtable, ImmedLoc) - self.mc.MOV_mi((loc.value, self.cpu.vtable_offset), loc_vtable.value) + self.mc.MOV(mem(loc, self.cpu.vtable_offset), loc_vtable) # XXX genop_new is abused for all varsized mallocs with Boehm, for now # (instead of genop_new_array, genop_newstr, genop_newunicode) @@ -1765,6 +1767,7 @@ jz_location = self.mc.get_relative_pos() # the following is supposed to be the slow path, so whenever possible # we choose the most compact encoding over the most efficient one. + # XXX improve a bit, particularly for IS_X86_64. for i in range(len(arglocs)-1, -1, -1): loc = arglocs[i] if isinstance(loc, RegLoc): @@ -1863,6 +1866,7 @@ offset = self.mc.get_relative_pos() - jmp_adr assert 0 < offset <= 127 self.mc.overwrite(jmp_adr-1, [chr(offset)]) + # on 64-bits, 'tid' is a value that fits in 31 bits self.mc.MOV_mi((eax.value, 0), tid) self.mc.MOV(heap(nursery_free_adr), edx) Modified: pypy/branch/jitffi/pypy/jit/backend/x86/codebuf.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/x86/codebuf.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/x86/codebuf.py Fri Oct 8 13:40:10 2010 @@ -1,6 +1,6 @@ import os, sys -from pypy.rpython.lltypesystem import lltype, rffi +from pypy.rpython.lltypesystem import lltype, llmemory, rffi from pypy.translator.tool.cbuild import ExternalCompilationInfo from pypy.jit.backend.x86.rx86 import X86_32_CodeBuilder, X86_64_CodeBuilder from pypy.jit.backend.x86.regloc import LocationCodeBuilder @@ -152,7 +152,7 @@ # Hack to make sure that mcs are not within 32-bits of one # another for testing purposes from pypy.rlib.rmmap import hint - hint.pos += 0xFFFFFFFF + hint.pos += 0x80000000 - map_size self._init(data, map_size) @@ -164,6 +164,12 @@ # ____________________________________________________________ +memcpy_fn = rffi.llexternal('memcpy', [llmemory.Address, llmemory.Address, + rffi.SIZE_T], lltype.Void, + sandboxsafe=True, _nowrapper=True) + +# ____________________________________________________________ + if sys.platform == 'win32': ensure_sse2_floats = lambda : None else: Modified: pypy/branch/jitffi/pypy/jit/backend/x86/regalloc.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/x86/regalloc.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/x86/regalloc.py Fri Oct 8 13:40:10 2010 @@ -224,7 +224,7 @@ assert tmpreg not in nonfloatlocs assert xmmtmp not in floatlocs # note: we need to make a copy of inputargs because possibly_free_vars - # is also used on op.args, which is a non-resizable list + # is also used on op args, which is a non-resizable list self.possibly_free_vars(list(inputargs)) return nonfloatlocs, floatlocs @@ -307,7 +307,7 @@ if reg not in used: self.xrm.free_regs.append(reg) # note: we need to make a copy of inputargs because possibly_free_vars - # is also used on op.args, which is a non-resizable list + # is also used on op args, which is a non-resizable list self.possibly_free_vars(list(inputargs)) self.rm._check_invariants() self.xrm._check_invariants() @@ -958,6 +958,47 @@ consider_unicodegetitem = consider_strgetitem + def consider_copystrcontent(self, op): + # compute the source address + args = op.getarglist() + base_loc = self.rm.make_sure_var_in_reg(args[0], args) + ofs_loc = self.rm.make_sure_var_in_reg(args[2], args) + assert args[0] is not args[1] # forbidden case of aliasing + self.rm.possibly_free_var(args[0]) + if args[3] is not args[2] is not args[4]: # MESS MESS MESS: don't free + self.rm.possibly_free_var(args[2]) # it if ==args[3] or args[4] + srcaddr_box = TempBox() + forbidden_vars = [args[1], args[3], args[4], srcaddr_box] + srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box, forbidden_vars) + self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc) + # compute the destination address + base_loc = self.rm.make_sure_var_in_reg(args[1], forbidden_vars) + ofs_loc = self.rm.make_sure_var_in_reg(args[3], forbidden_vars) + self.rm.possibly_free_var(args[1]) + if args[3] is not args[4]: # more of the MESS described above + self.rm.possibly_free_var(args[3]) + forbidden_vars = [args[4], srcaddr_box] + dstaddr_box = TempBox() + dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box, forbidden_vars) + self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc) + # call memcpy() + length_loc = self.loc(args[4]) + self.rm.before_call() + self.xrm.before_call() + self.assembler._emit_call(imm(self.assembler.memcpy_addr), + [dstaddr_loc, srcaddr_loc, length_loc]) + self.rm.possibly_free_var(args[4]) + self.rm.possibly_free_var(dstaddr_box) + self.rm.possibly_free_var(srcaddr_box) + + def _gen_address_inside_string(self, baseloc, ofsloc, resloc): + cpu = self.assembler.cpu + ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR, + self.translate_support_code) + assert itemsize == 1 + self.assembler.load_effective_addr(ofsloc, ofs_items, 0, + resloc, baseloc) + def consider_jump(self, op): assembler = self.assembler assert self.jump_target_descr is None Modified: pypy/branch/jitffi/pypy/jit/backend/x86/regloc.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/x86/regloc.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/x86/regloc.py Fri Oct 8 13:40:10 2010 @@ -33,6 +33,8 @@ def value_a(self): raise AssertionError("value_a undefined") def value_m(self): raise AssertionError("value_m undefined") + def find_unused_reg(self): return eax + class StackLoc(AssemblerLocation): _immutable_ = True def __init__(self, position, ebp_offset, num_words, type): @@ -88,6 +90,12 @@ def assembler(self): return '%' + repr(self) + def find_unused_reg(self): + if self.value == eax.value: + return edx + else: + return eax + class ImmedLoc(AssemblerLocation): _immutable_ = True width = WORD @@ -137,6 +145,12 @@ self._location_code = 'a' self.loc_a = (base_loc.value, scaled_loc.value, scale, static_offset) + def __repr__(self): + dict = {'j': 'value', 'a': 'loc_a', 'm': 'loc_m', 'a':'loc_a'} + attr = dict.get(self._location_code, '?') + info = getattr(self, attr, '?') + return '' % (self._location_code, info) + def location_code(self): return self._location_code @@ -146,6 +160,21 @@ def value_m(self): return self.loc_m + def find_unused_reg(self): + if self._location_code == 'm': + if self.loc_m[0] == eax.value: + return edx + elif self._location_code == 'a': + if self.loc_a[0] == eax.value: + if self.loc_a[1] == edx.value: + return ecx + return edx + if self.loc_a[1] == eax.value: + if self.loc_a[0] == edx.value: + return ecx + return edx + return eax + class ConstFloatLoc(AssemblerLocation): # XXX: We have to use this class instead of just AddressLoc because # AddressLoc is "untyped" and also we to have need some sort of unique @@ -159,6 +188,9 @@ self.value = address self.const_id = const_id + def __repr__(self): + return '' % (self.value, self.const_id) + def _getregkey(self): # XXX: 1000 is kind of magic: We just don't want to be confused # with any registers @@ -206,6 +238,32 @@ _scratch_register_value = 0 def _binaryop(name): + + def insn_with_64_bit_immediate(self, loc1, loc2): + # These are the worst cases: + val2 = loc2.value_i() + code1 = loc1.location_code() + if (code1 == 'j' + or (code1 == 'm' and not rx86.fits_in_32bits(loc1.value_m()[1])) + or (code1 == 'a' and not rx86.fits_in_32bits(loc1.value_a()[3]))): + # INSN_ji, and both operands are 64-bit; or INSN_mi or INSN_ai + # and the constant offset in the address is 64-bit. + # Hopefully this doesn't happen too often + freereg = loc1.find_unused_reg() + self.PUSH_r(freereg.value) + self.MOV_ri(freereg.value, val2) + INSN(self, loc1, freereg) + self.POP_r(freereg.value) + else: + # For this case, we should not need the scratch register more than here. + self._load_scratch(val2) + INSN(self, loc1, X86_64_SCRATCH_REG) + + def invoke(self, codes, val1, val2): + methname = name + "_" + codes + _rx86_getattr(self, methname)(val1, val2) + invoke._annspecialcase_ = 'specialize:arg(1)' + def INSN(self, loc1, loc2): code1 = loc1.location_code() code2 = loc2.location_code() @@ -218,38 +276,39 @@ if loc1 is X86_64_SCRATCH_REG and not name.startswith("MOV"): assert code2 not in ('j', 'i') - for possible_code1 in unrolling_location_codes: - if code1 == possible_code1: - for possible_code2 in unrolling_location_codes: - if code2 == possible_code2: + for possible_code2 in unrolling_location_codes: + if code2 == possible_code2: + val2 = getattr(loc2, "value_" + possible_code2)() + # + # Fake out certain operations for x86_64 + if self.WORD == 8 and possible_code2 == 'i' and not rx86.fits_in_32bits(val2): + insn_with_64_bit_immediate(self, loc1, loc2) + return + # + # Regular case + for possible_code1 in unrolling_location_codes: + if code1 == possible_code1: val1 = getattr(loc1, "value_" + possible_code1)() - val2 = getattr(loc2, "value_" + possible_code2)() - # Fake out certain operations for x86_64 - if self.WORD == 8 and possible_code2 == 'i' and not rx86.fits_in_32bits(val2): - if possible_code1 == 'j': - # This is the worst case: INSN_ji, and both operands are 64-bit - # Hopefully this doesn't happen too often - self.PUSH_r(eax.value) - self.MOV_ri(eax.value, val1) - self.MOV_ri(X86_64_SCRATCH_REG.value, val2) - methname = name + "_mr" - _rx86_getattr(self, methname)((eax.value, 0), X86_64_SCRATCH_REG.value) - self.POP_r(eax.value) - else: - self.MOV_ri(X86_64_SCRATCH_REG.value, val2) - methname = name + "_" + possible_code1 + "r" - _rx86_getattr(self, methname)(val1, X86_64_SCRATCH_REG.value) - elif self.WORD == 8 and possible_code1 == 'j': - reg_offset = self._addr_as_reg_offset(val1) - methname = name + "_" + "m" + possible_code2 - _rx86_getattr(self, methname)(reg_offset, val2) + # More faking out of certain operations for x86_64 + if self.WORD == 8 and possible_code1 == 'j': + val1 = self._addr_as_reg_offset(val1) + invoke(self, "m" + possible_code2, val1, val2) elif self.WORD == 8 and possible_code2 == 'j': - reg_offset = self._addr_as_reg_offset(val2) - methname = name + "_" + possible_code1 + "m" - _rx86_getattr(self, methname)(val1, reg_offset) + val2 = self._addr_as_reg_offset(val2) + invoke(self, possible_code1 + "m", val1, val2) + elif possible_code1 == 'm' and not rx86.fits_in_32bits(val1[1]): + val1 = self._fix_static_offset_64_m(val1) + invoke(self, "a" + possible_code2, val1, val2) + elif possible_code2 == 'm' and not rx86.fits_in_32bits(val2[1]): + val2 = self._fix_static_offset_64_m(val2) + invoke(self, possible_code1 + "a", val1, val2) else: - methname = name + "_" + possible_code1 + possible_code2 - _rx86_getattr(self, methname)(val1, val2) + if possible_code1 == 'a' and not rx86.fits_in_32bits(val1[3]): + val1 = self._fix_static_offset_64_a(val1) + if possible_code2 == 'a' and not rx86.fits_in_32bits(val2[3]): + val2 = self._fix_static_offset_64_a(val2) + invoke(self, possible_code1 + possible_code2, val1, val2) + return return func_with_new_name(INSN, "INSN_" + name) @@ -260,7 +319,7 @@ if code == possible_code: val = getattr(loc, "value_" + possible_code)() if self.WORD == 8 and possible_code == 'i' and not rx86.fits_in_32bits(val): - self.MOV_ri(X86_64_SCRATCH_REG.value, val) + self._load_scratch(val) _rx86_getattr(self, name + "_r")(X86_64_SCRATCH_REG.value) else: methname = name + "_" + possible_code @@ -280,7 +339,7 @@ _rx86_getattr(self, name + "_l")(val) else: assert self.WORD == 8 - self.MOV_ri(X86_64_SCRATCH_REG.value, val) + self._load_scratch(val) _rx86_getattr(self, name + "_r")(X86_64_SCRATCH_REG.value) else: methname = name + "_" + possible_code @@ -317,6 +376,40 @@ self.MOV_ri(X86_64_SCRATCH_REG.value, addr) return (X86_64_SCRATCH_REG.value, 0) + def _fix_static_offset_64_m(self, (basereg, static_offset)): + # For cases where an AddressLoc has the location_code 'm', but + # where the static offset does not fit in 32-bits. We have to fall + # back to the X86_64_SCRATCH_REG. Note that this returns a location + # encoded as mode 'a'. These are all possibly rare cases; don't try + # to reuse a past value of the scratch register at all. + self._scratch_register_known = False + self.MOV_ri(X86_64_SCRATCH_REG.value, static_offset) + return (basereg, X86_64_SCRATCH_REG.value, 0, 0) + + def _fix_static_offset_64_a(self, (basereg, scalereg, + scale, static_offset)): + # For cases where an AddressLoc has the location_code 'a', but + # where the static offset does not fit in 32-bits. We have to fall + # back to the X86_64_SCRATCH_REG. In one case it is even more + # annoying. These are all possibly rare cases; don't try to reuse a + # past value of the scratch register at all. + self._scratch_register_known = False + self.MOV_ri(X86_64_SCRATCH_REG.value, static_offset) + # + if basereg != rx86.NO_BASE_REGISTER: + self.LEA_ra(X86_64_SCRATCH_REG.value, + (basereg, X86_64_SCRATCH_REG.value, 0, 0)) + return (X86_64_SCRATCH_REG.value, scalereg, scale, 0) + + def _load_scratch(self, value): + if (self._scratch_register_known + and value == self._scratch_register_value): + return + if self._reuse_scratch_register: + self._scratch_register_known = True + self._scratch_register_value = value + self.MOV_ri(X86_64_SCRATCH_REG.value, value) + def begin_reuse_scratch_register(self): # Flag the beginning of a block where it is okay to reuse the value # of the scratch register. In theory we shouldn't have to do this if Modified: pypy/branch/jitffi/pypy/jit/backend/x86/runner.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/x86/runner.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/x86/runner.py Fri Oct 8 13:40:10 2010 @@ -87,7 +87,9 @@ def execute_token(self, executable_token): addr = executable_token._x86_bootstrap_code + #llop.debug_print(lltype.Void, ">>>> Entering", addr) func = rffi.cast(lltype.Ptr(self.BOOTSTRAP_TP), addr) + #llop.debug_print(lltype.Void, "<<<< Back") fail_index = self._execute_call(func) return self.get_fail_descr_from_number(fail_index) @@ -99,10 +101,7 @@ LLInterpreter.current_interpreter = self.debug_ll_interpreter res = 0 try: - #llop.debug_print(lltype.Void, ">>>> Entering", - # rffi.cast(lltype.Signed, func)) res = func() - #llop.debug_print(lltype.Void, "<<<< Back") finally: if not self.translate_support_code: LLInterpreter.current_interpreter = prev_interpreter Modified: pypy/branch/jitffi/pypy/jit/backend/x86/rx86.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/x86/rx86.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/x86/rx86.py Fri Oct 8 13:40:10 2010 @@ -506,6 +506,7 @@ LEA32_rb = insn(rex_w, '\x8D', register(1,8),stack_bp(2,force_32bits=True)) LEA_ra = insn(rex_w, '\x8D', register(1, 8), mem_reg_plus_scaled_reg_plus_const(2)) LEA_rm = insn(rex_w, '\x8D', register(1, 8), mem_reg_plus_const(2)) + LEA_rj = insn(rex_w, '\x8D', register(1, 8), '\x05', immediate(2)) CALL_l = insn('\xE8', relative(1)) CALL_r = insn(rex_nw, '\xFF', register(1), chr(0xC0 | (2<<3))) Modified: pypy/branch/jitffi/pypy/jit/backend/x86/test/conftest.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/x86/test/conftest.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/x86/test/conftest.py Fri Oct 8 13:40:10 2010 @@ -5,3 +5,6 @@ def pytest_runtest_setup(item): if cpu not in ('x86', 'x86_64'): py.test.skip("x86/x86_64 tests skipped: cpu is %r" % (cpu,)) + if cpu == 'x86_64': + from pypy.rpython.lltypesystem import ll2ctypes + ll2ctypes.do_allocation_in_far_regions() Modified: pypy/branch/jitffi/pypy/jit/backend/x86/test/test_regloc.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/x86/test/test_regloc.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/x86/test/test_regloc.py Fri Oct 8 13:40:10 2010 @@ -58,23 +58,250 @@ expected_ofs = pos_addr - (neg_addr+5) assert s.getvalue() == '\xE9' + struct.pack("= self.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE def effectinfo_from_writeanalyze(effects, cpu, - extraeffect=EffectInfo.EF_CAN_RAISE): + extraeffect=EffectInfo.EF_CAN_RAISE, + oopspecindex=EffectInfo.OS_NONE): from pypy.translator.backendopt.writeanalyze import top_set if effects is top_set: return None @@ -73,7 +94,8 @@ return EffectInfo(readonly_descrs_fields, write_descrs_fields, write_descrs_arrays, - extraeffect) + extraeffect, + oopspecindex) def consider_struct(TYPE, fieldname): if fieldType(TYPE, fieldname) is lltype.Void: @@ -104,3 +126,33 @@ def analyze_simple_operation(self, op): return op.opname in ('jit_force_virtualizable', 'jit_force_virtual') + +# ____________________________________________________________ + +_callinfo_for_oopspec = {} # {oopspecindex: (calldescr, func_as_int)} + +def callinfo_for_oopspec(oopspecindex): + """A function that returns the calldescr and the function + address (as an int) of one of the OS_XYZ functions defined above. + Don't use this if there might be several implementations of the same + OS_XYZ specialized by type, e.g. OS_ARRAYCOPY.""" + try: + return _callinfo_for_oopspec[oopspecindex] + except KeyError: + return (None, 0) + + +def _funcptr_for_oopspec_memo(oopspecindex): + from pypy.jit.codewriter import heaptracker + _, func_as_int = callinfo_for_oopspec(oopspecindex) + funcadr = heaptracker.int2adr(func_as_int) + return funcadr.ptr +_funcptr_for_oopspec_memo._annspecialcase_ = 'specialize:memo' + +def funcptr_for_oopspec(oopspecindex): + """A memo function that returns a pointer to the function described + by OS_XYZ (as a real low-level function pointer).""" + funcptr = _funcptr_for_oopspec_memo(oopspecindex) + assert funcptr + return funcptr +funcptr_for_oopspec._annspecialcase_ = 'specialize:arg(0)' Modified: pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py Fri Oct 8 13:40:10 2010 @@ -1,16 +1,18 @@ import py, sys -from pypy.rpython.lltypesystem import lltype, rstr, rclass +from pypy.rpython.lltypesystem import lltype, llmemory, rstr, rclass from pypy.rpython import rlist from pypy.jit.metainterp.history import getkind from pypy.objspace.flow.model import SpaceOperation, Variable, Constant from pypy.objspace.flow.model import Block, Link, c_last_exception from pypy.jit.codewriter.flatten import ListOfKind, IndirectCallTargets from pypy.jit.codewriter import support, heaptracker +from pypy.jit.codewriter.effectinfo import EffectInfo, _callinfo_for_oopspec from pypy.jit.codewriter.policy import log from pypy.jit.metainterp.typesystem import deref, arrayItem from pypy.rlib import objectmodel from pypy.rlib.jit import _we_are_jitted from pypy.translator.simplify import get_funcobj +from pypy.translator.unsimplify import varoftype def transform_graph(graph, cpu=None, callcontrol=None, portal_jd=None): @@ -248,11 +250,13 @@ kind = self.callcontrol.guess_call_kind(op) return getattr(self, 'handle_%s_indirect_call' % kind)(op) - def rewrite_call(self, op, namebase, initialargs): + def rewrite_call(self, op, namebase, initialargs, args=None): """Turn 'i0 = direct_call(fn, i1, i2, ref1, ref2)' into 'i0 = xxx_call_ir_i(fn, descr, [i1,i2], [ref1,ref2])'. The name is one of '{residual,direct}_call_{r,ir,irf}_{i,r,f,v}'.""" - lst_i, lst_r, lst_f = self.make_three_lists(op.args[1:]) + if args is None: + args = op.args[1:] + lst_i, lst_r, lst_f = self.make_three_lists(args) reskind = getkind(op.result.concretetype)[0] if lst_f or reskind == 'f': kinds = 'irf' elif lst_i: kinds = 'ir' @@ -310,6 +314,8 @@ # dispatch to various implementations depending on the oopspec_name if oopspec_name.startswith('list.') or oopspec_name == 'newlist': prepare = self._handle_list_call + elif oopspec_name.startswith('stroruni.'): + prepare = self._handle_stroruni_call elif oopspec_name.startswith('virtual_ref'): prepare = self._handle_virtual_ref_call else: @@ -982,10 +988,7 @@ return extraop + [op] def do_fixed_list_ll_arraycopy(self, op, args, arraydescr): - calldescr = self.callcontrol.getcalldescr(op) - return SpaceOperation('arraycopy', - [calldescr, op.args[0]] + args + [arraydescr], - op.result) + return self._handle_oopspec_call(op, args, EffectInfo.OS_ARRAYCOPY) # ---------- resizable lists ---------- @@ -1023,6 +1026,92 @@ [args[0], lengthdescr], op.result) # ---------- + # Strings and Unicodes. + + def _handle_oopspec_call(self, op, args, oopspecindex): + calldescr = self.callcontrol.getcalldescr(op, oopspecindex) + if isinstance(op.args[0].value, str): + pass # for tests only + else: + func = heaptracker.adr2int( + llmemory.cast_ptr_to_adr(op.args[0].value)) + _callinfo_for_oopspec[oopspecindex] = calldescr, func + op1 = self.rewrite_call(op, 'residual_call', + [op.args[0], calldescr], + args=args) + if self.callcontrol.calldescr_canraise(calldescr): + op1 = [op1, SpaceOperation('-live-', [], None)] + return op1 + + def _register_extra_helper(self, oopspecindex, oopspec_name, + argtypes, resulttype): + # a bit hackish + if oopspecindex in _callinfo_for_oopspec: + return + c_func, TP = support.builtin_func_for_spec(self.cpu.rtyper, + oopspec_name, argtypes, + resulttype) + op = SpaceOperation('pseudo_call', + [c_func] + [varoftype(T) for T in argtypes], + varoftype(resulttype)) + calldescr = self.callcontrol.getcalldescr(op, oopspecindex) + func = heaptracker.adr2int( + llmemory.cast_ptr_to_adr(c_func.value)) + _callinfo_for_oopspec[oopspecindex] = calldescr, func + + def _handle_stroruni_call(self, op, oopspec_name, args): + if args[0].concretetype.TO == rstr.STR: + dict = {"stroruni.concat": EffectInfo.OS_STR_CONCAT, + "stroruni.slice": EffectInfo.OS_STR_SLICE, + "stroruni.equal": EffectInfo.OS_STR_EQUAL, + } + elif args[0].concretetype.TO == rstr.UNICODE: + dict = {"stroruni.concat": EffectInfo.OS_UNI_CONCAT, + "stroruni.slice": EffectInfo.OS_UNI_SLICE, + "stroruni.equal": EffectInfo.OS_UNI_EQUAL, + } + else: + assert 0, "args[0].concretetype must be STR or UNICODE" + # + if oopspec_name == "stroruni.equal": + SoU = args[0].concretetype # Ptr(STR) or Ptr(UNICODE) + for otherindex, othername, argtypes, resulttype in [ + + (EffectInfo.OS_STREQ_SLICE_CHECKNULL, + "str.eq_slice_checknull", + [SoU, lltype.Signed, lltype.Signed, SoU], + lltype.Signed), + (EffectInfo.OS_STREQ_SLICE_NONNULL, + "str.eq_slice_nonnull", + [SoU, lltype.Signed, lltype.Signed, SoU], + lltype.Signed), + (EffectInfo.OS_STREQ_SLICE_CHAR, + "str.eq_slice_char", + [SoU, lltype.Signed, lltype.Signed, lltype.Char], + lltype.Signed), + (EffectInfo.OS_STREQ_NONNULL, + "str.eq_nonnull", + [SoU, SoU], + lltype.Signed), + (EffectInfo.OS_STREQ_NONNULL_CHAR, + "str.eq_nonnull_char", + [SoU, lltype.Char], + lltype.Signed), + (EffectInfo.OS_STREQ_CHECKNULL_CHAR, + "str.eq_checknull_char", + [SoU, lltype.Char], + lltype.Signed), + (EffectInfo.OS_STREQ_LENGTHOK, + "str.eq_lengthok", + [SoU, SoU], + lltype.Signed), + ]: + self._register_extra_helper(otherindex, othername, + argtypes, resulttype) + # + return self._handle_oopspec_call(op, args, dict[oopspec_name]) + + # ---------- # VirtualRefs. def _handle_virtual_ref_call(self, op, oopspec_name, args): Modified: pypy/branch/jitffi/pypy/jit/codewriter/support.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/support.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/support.py Fri Oct 8 13:40:10 2010 @@ -303,10 +303,86 @@ # ---------- strings and unicode ---------- - _ll_5_string_copy_contents = ll_rstr.copy_string_contents - _ll_1_str_str2unicode = ll_rstr.LLHelpers.ll_str2unicode - _ll_5_unicode_copy_contents = ll_rstr.copy_unicode_contents + + def _ll_4_str_eq_slice_checknull(s1, start, length, s2): + """str1[start : start + length] == str2.""" + if not s2: + return 0 + chars2 = s2.chars + if len(chars2) != length: + return 0 + j = 0 + chars1 = s1.chars + while j < length: + if chars1[start + j] != chars2[j]: + return 0 + j += 1 + return 1 + + def _ll_4_str_eq_slice_nonnull(s1, start, length, s2): + """str1[start : start + length] == str2, assuming str2 != NULL.""" + chars2 = s2.chars + if len(chars2) != length: + return 0 + j = 0 + chars1 = s1.chars + while j < length: + if chars1[start + j] != chars2[j]: + return 0 + j += 1 + return 1 + + def _ll_4_str_eq_slice_char(s1, start, length, c2): + """str1[start : start + length] == c2.""" + if length != 1: + return 0 + if s1.chars[start] != c2: + return 0 + return 1 + + def _ll_2_str_eq_nonnull(s1, s2): + len1 = len(s1.chars) + len2 = len(s2.chars) + if len1 != len2: + return 0 + j = 0 + chars1 = s1.chars + chars2 = s2.chars + while j < len1: + if chars1[j] != chars2[j]: + return 0 + j += 1 + return 1 + + def _ll_2_str_eq_nonnull_char(s1, c2): + chars = s1.chars + if len(chars) != 1: + return 0 + if chars[0] != c2: + return 0 + return 1 + + def _ll_2_str_eq_checknull_char(s1, c2): + if not s1: + return 0 + chars = s1.chars + if len(chars) != 1: + return 0 + if chars[0] != c2: + return 0 + return 1 + + def _ll_2_str_eq_lengthok(s1, s2): + j = 0 + chars1 = s1.chars + chars2 = s2.chars + len1 = len(chars1) + while j < len1: + if chars1[j] != chars2[j]: + return 0 + j += 1 + return 1 # ---------- malloc with del ---------- Modified: pypy/branch/jitffi/pypy/jit/codewriter/test/test_jtransform.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/test/test_jtransform.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/test/test_jtransform.py Fri Oct 8 13:40:10 2010 @@ -1,11 +1,16 @@ +import py import random from pypy.objspace.flow.model import FunctionGraph, Block, Link from pypy.objspace.flow.model import SpaceOperation, Variable, Constant from pypy.jit.codewriter.jtransform import Transformer from pypy.jit.metainterp.history import getkind -from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr +from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr, rlist from pypy.translator.unsimplify import varoftype -from pypy.jit.codewriter import heaptracker +from pypy.jit.codewriter import heaptracker, effectinfo +from pypy.jit.codewriter.flatten import ListOfKind + +def const(x): + return Constant(x, lltype.typeOf(x)) class FakeRTyper: class type_system: name = 'lltypesystem' @@ -17,6 +22,8 @@ return ('calldescr', FUNC, ARGS, RESULT) def fielddescrof(self, STRUCT, name): return ('fielddescr', STRUCT, name) + def arraydescrof(self, ARRAY): + return FakeDescr(('arraydescr', ARRAY)) def sizeof(self, STRUCT): return FakeDescr(('sizedescr', STRUCT)) @@ -67,6 +74,14 @@ def calldescr_canraise(self, calldescr): return False +class FakeBuiltinCallControl: + def guess_call_kind(self, op): + return 'builtin' + def getcalldescr(self, op, oopspecindex): + return 'calldescr-%d' % oopspecindex + def calldescr_canraise(self, calldescr): + return False + def test_optimize_goto_if_not(): v1 = Variable() @@ -107,7 +122,7 @@ assert block.operations == [] assert block.exitswitch == ('int_gt', v1, v2) assert block.exits == exits - assert exits[1].args == [Constant(True, lltype.Bool)] + assert exits[1].args == [const(True)] def test_optimize_goto_if_not__unknownop(): v3 = Variable(); v3.concretetype = lltype.Bool @@ -159,8 +174,8 @@ 'float_gt': ('float_gt', 'float_lt'), } v3 = varoftype(lltype.Signed) - for v1 in [varoftype(lltype.Signed), Constant(42, lltype.Signed)]: - for v2 in [varoftype(lltype.Signed), Constant(43, lltype.Signed)]: + for v1 in [varoftype(lltype.Signed), const(42)]: + for v2 in [varoftype(lltype.Signed), const(43)]: for name1, name2 in ops.items(): op = SpaceOperation(name1, [v1, v2], v3) op1 = Transformer(FakeCPU()).rewrite_operation(op) @@ -177,8 +192,8 @@ def test_symmetric_int_add_ovf(): v3 = varoftype(lltype.Signed) - for v1 in [varoftype(lltype.Signed), Constant(42, lltype.Signed)]: - for v2 in [varoftype(lltype.Signed), Constant(43, lltype.Signed)]: + for v1 in [varoftype(lltype.Signed), const(42)]: + for v2 in [varoftype(lltype.Signed), const(43)]: op = SpaceOperation('int_add_nonneg_ovf', [v1, v2], v3) oplist = Transformer(FakeCPU()).rewrite_operation(op) op0, op1 = oplist @@ -218,7 +233,7 @@ def get_direct_call_op(argtypes, restype): FUNC = lltype.FuncType(argtypes, restype) fnptr = lltype.functionptr(FUNC, "g") # no graph - c_fnptr = Constant(fnptr, concretetype=lltype.typeOf(fnptr)) + c_fnptr = const(fnptr) vars = [varoftype(TYPE) for TYPE in argtypes] v_result = varoftype(restype) op = SpaceOperation('direct_call', [c_fnptr] + vars, v_result) @@ -465,7 +480,7 @@ v1 = varoftype(lltype.Signed) v2 = varoftype(lltype.Signed) v3 = varoftype(lltype.Bool) - c0 = Constant(0, lltype.Signed) + c0 = const(0) # for opname, reducedname in [('int_eq', 'int_is_zero'), ('int_ne', 'int_is_true')]: @@ -488,7 +503,7 @@ v1 = varoftype(rclass.OBJECTPTR) v2 = varoftype(rclass.OBJECTPTR) v3 = varoftype(lltype.Bool) - c0 = Constant(lltype.nullptr(rclass.OBJECT), rclass.OBJECTPTR) + c0 = const(lltype.nullptr(rclass.OBJECT)) # for opname, reducedname in [('ptr_eq', 'ptr_iszero'), ('ptr_ne', 'ptr_nonzero')]: @@ -511,7 +526,7 @@ v1 = varoftype(rclass.NONGCOBJECTPTR) v2 = varoftype(rclass.NONGCOBJECTPTR) v3 = varoftype(lltype.Bool) - c0 = Constant(lltype.nullptr(rclass.NONGCOBJECT), rclass.NONGCOBJECTPTR) + c0 = const(lltype.nullptr(rclass.NONGCOBJECT)) # for opname, reducedname in [('ptr_eq', 'int_is_zero'), ('ptr_ne', 'int_is_true')]: @@ -656,3 +671,119 @@ oplist = tr.rewrite_operation(op) assert oplist[0].opname == 'inline_call_ir_i' assert oplist[0].args[0] == 'somejitcode' + +def test_str_newstr(): + c_STR = Constant(rstr.STR, lltype.Void) + c_flavor = Constant({'flavor': 'gc'}, lltype.Void) + v1 = varoftype(lltype.Signed) + v2 = varoftype(lltype.Ptr(rstr.STR)) + op = SpaceOperation('malloc_varsize', [c_STR, c_flavor, v1], v2) + op1 = Transformer().rewrite_operation(op) + assert op1.opname == 'newstr' + assert op1.args == [v1] + assert op1.result == v2 + +def test_str_concat(): + # test that the oopspec is present and correctly transformed + PSTR = lltype.Ptr(rstr.STR) + FUNC = lltype.FuncType([PSTR, PSTR], PSTR) + func = lltype.functionptr(FUNC, 'll_strconcat', + _callable=rstr.LLHelpers.ll_strconcat) + v1 = varoftype(PSTR) + v2 = varoftype(PSTR) + v3 = varoftype(PSTR) + op = SpaceOperation('direct_call', [const(func), v1, v2], v3) + tr = Transformer(FakeCPU(), FakeBuiltinCallControl()) + op1 = tr.rewrite_operation(op) + assert op1.opname == 'residual_call_r_r' + assert op1.args[0].value == func + assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_STR_CONCAT + assert op1.args[2] == ListOfKind('ref', [v1, v2]) + assert op1.result == v3 + +def test_unicode_concat(): + # test that the oopspec is present and correctly transformed + PSTR = lltype.Ptr(rstr.UNICODE) + FUNC = lltype.FuncType([PSTR, PSTR], PSTR) + func = lltype.functionptr(FUNC, 'll_strconcat', + _callable=rstr.LLHelpers.ll_strconcat) + v1 = varoftype(PSTR) + v2 = varoftype(PSTR) + v3 = varoftype(PSTR) + op = SpaceOperation('direct_call', [const(func), v1, v2], v3) + tr = Transformer(FakeCPU(), FakeBuiltinCallControl()) + op1 = tr.rewrite_operation(op) + assert op1.opname == 'residual_call_r_r' + assert op1.args[0].value == func + assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_UNI_CONCAT + assert op1.args[2] == ListOfKind('ref', [v1, v2]) + assert op1.result == v3 + # + # check the callinfo_for_oopspec + got = effectinfo.callinfo_for_oopspec(effectinfo.EffectInfo.OS_UNI_CONCAT) + assert got[0] == op1.args[1] # the calldescr + assert heaptracker.int2adr(got[1]) == llmemory.cast_ptr_to_adr(func) + +def test_str_slice(): + # test that the oopspec is present and correctly transformed + PSTR = lltype.Ptr(rstr.STR) + INT = lltype.Signed + FUNC = lltype.FuncType([PSTR, INT, INT], PSTR) + func = lltype.functionptr(FUNC, '_ll_stringslice', + _callable=rstr.LLHelpers._ll_stringslice) + v1 = varoftype(PSTR) + v2 = varoftype(INT) + v3 = varoftype(INT) + v4 = varoftype(PSTR) + op = SpaceOperation('direct_call', [const(func), v1, v2, v3], v4) + tr = Transformer(FakeCPU(), FakeBuiltinCallControl()) + op1 = tr.rewrite_operation(op) + assert op1.opname == 'residual_call_ir_r' + assert op1.args[0].value == func + assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_STR_SLICE + assert op1.args[2] == ListOfKind('int', [v2, v3]) + assert op1.args[3] == ListOfKind('ref', [v1]) + assert op1.result == v4 + +def test_unicode_slice(): + # test that the oopspec is present and correctly transformed + PUNICODE = lltype.Ptr(rstr.UNICODE) + INT = lltype.Signed + FUNC = lltype.FuncType([PUNICODE, INT, INT], PUNICODE) + func = lltype.functionptr(FUNC, '_ll_stringslice', + _callable=rstr.LLHelpers._ll_stringslice) + v1 = varoftype(PUNICODE) + v2 = varoftype(INT) + v3 = varoftype(INT) + v4 = varoftype(PUNICODE) + op = SpaceOperation('direct_call', [const(func), v1, v2, v3], v4) + tr = Transformer(FakeCPU(), FakeBuiltinCallControl()) + op1 = tr.rewrite_operation(op) + assert op1.opname == 'residual_call_ir_r' + assert op1.args[0].value == func + assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_UNI_SLICE + assert op1.args[2] == ListOfKind('int', [v2, v3]) + assert op1.args[3] == ListOfKind('ref', [v1]) + assert op1.result == v4 + +def test_list_ll_arraycopy(): + from pypy.rlib.rgc import ll_arraycopy + LIST = lltype.GcArray(lltype.Signed) + PLIST = lltype.Ptr(LIST) + INT = lltype.Signed + FUNC = lltype.FuncType([PLIST]*2+[INT]*3, lltype.Void) + func = lltype.functionptr(FUNC, 'll_arraycopy', _callable=ll_arraycopy) + v1 = varoftype(PLIST) + v2 = varoftype(PLIST) + v3 = varoftype(INT) + v4 = varoftype(INT) + v5 = varoftype(INT) + v6 = varoftype(lltype.Void) + op = SpaceOperation('direct_call', [const(func), v1, v2, v3, v4, v5], v6) + tr = Transformer(FakeCPU(), FakeBuiltinCallControl()) + op1 = tr.rewrite_operation(op) + assert op1.opname == 'residual_call_ir_v' + assert op1.args[0].value == func + assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_ARRAYCOPY + assert op1.args[2] == ListOfKind('int', [v3, v4, v5]) + assert op1.args[3] == ListOfKind('ref', [v1, v2]) Modified: pypy/branch/jitffi/pypy/jit/codewriter/test/test_list.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/test/test_list.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/test/test_list.py Fri Oct 8 13:40:10 2010 @@ -36,10 +36,16 @@ class FakeCallControl: class getcalldescr(AbstractDescr): - def __init__(self, op): + def __init__(self, op, oopspecindex=0): self.op = op + self.oopspecindex = oopspecindex def __repr__(self): - return '' + if self.oopspecindex == 0: + return '' + else: + return '' % self.oopspecindex + def calldescr_canraise(self, calldescr): + return False def builtin_test(oopspec_name, args, RESTYPE, expected): v_result = varoftype(RESTYPE) @@ -99,7 +105,7 @@ varoftype(lltype.Signed), varoftype(lltype.Signed)], lltype.Void, """ - arraycopy , $'myfunc', %r0, %r1, %i0, %i1, %i2, + residual_call_ir_v $'myfunc', , I[%i0, %i1, %i2], R[%r0, %r1] """) def test_fixed_getitem(): Modified: pypy/branch/jitffi/pypy/jit/codewriter/test/test_support.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/test/test_support.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/test/test_support.py Fri Oct 8 13:40:10 2010 @@ -1,7 +1,8 @@ import py from pypy.rpython.lltypesystem import lltype +from pypy.rpython.annlowlevel import llstr from pypy.objspace.flow.model import Variable, Constant, SpaceOperation -from pypy.jit.codewriter.support import decode_builtin_call +from pypy.jit.codewriter.support import decode_builtin_call, LLtypeHelpers def newconst(x): return Constant(x, lltype.typeOf(x)) @@ -65,3 +66,70 @@ assert opargs == [newconst(myarray), newconst(2), vc, vi] #impl = runner.get_oopspec_impl('spam.foobar', lltype.Ptr(A)) #assert impl(myarray, 2, 'A', 5) == 42 * ord('A') + +def test_streq_slice_checknull(): + p1 = llstr("hello world") + p2 = llstr("wor") + func = LLtypeHelpers._ll_4_str_eq_slice_checknull.im_func + assert func(p1, 6, 3, p2) == True + assert func(p1, 6, 2, p2) == False + assert func(p1, 5, 3, p2) == False + assert func(p1, 2, 1, llstr(None)) == False + +def test_streq_slice_nonnull(): + p1 = llstr("hello world") + p2 = llstr("wor") + func = LLtypeHelpers._ll_4_str_eq_slice_nonnull.im_func + assert func(p1, 6, 3, p2) == True + assert func(p1, 6, 2, p2) == False + assert func(p1, 5, 3, p2) == False + py.test.raises(AttributeError, func, p1, 2, 1, llstr(None)) + +def test_streq_slice_char(): + p1 = llstr("hello world") + func = LLtypeHelpers._ll_4_str_eq_slice_char.im_func + assert func(p1, 6, 3, "w") == False + assert func(p1, 6, 0, "w") == False + assert func(p1, 6, 1, "w") == True + assert func(p1, 6, 1, "x") == False + +def test_streq_nonnull(): + p1 = llstr("wor") + p2 = llstr("wor") + assert p1 != p2 + func = LLtypeHelpers._ll_2_str_eq_nonnull.im_func + assert func(p1, p1) == True + assert func(p1, p2) == True + assert func(p1, llstr("wrl")) == False + assert func(p1, llstr("world")) == False + assert func(p1, llstr("w")) == False + py.test.raises(AttributeError, func, p1, llstr(None)) + py.test.raises(AttributeError, func, llstr(None), p2) + +def test_streq_nonnull_char(): + func = LLtypeHelpers._ll_2_str_eq_nonnull_char.im_func + assert func(llstr("wor"), "x") == False + assert func(llstr("w"), "x") == False + assert func(llstr(""), "x") == False + assert func(llstr("x"), "x") == True + py.test.raises(AttributeError, func, llstr(None), "x") + +def test_streq_checknull_char(): + func = LLtypeHelpers._ll_2_str_eq_checknull_char.im_func + assert func(llstr("wor"), "x") == False + assert func(llstr("w"), "x") == False + assert func(llstr(""), "x") == False + assert func(llstr("x"), "x") == True + assert func(llstr(None), "x") == False + +def test_streq_lengthok(): + p1 = llstr("wor") + p2 = llstr("wor") + assert p1 != p2 + func = LLtypeHelpers._ll_2_str_eq_lengthok.im_func + assert func(p1, p1) == True + assert func(p1, p2) == True + assert func(p1, llstr("wrl")) == False + py.test.raises(IndexError, func, p1, llstr("w")) + py.test.raises(AttributeError, func, p1, llstr(None)) + py.test.raises(AttributeError, func, llstr(None), p2) Modified: pypy/branch/jitffi/pypy/jit/metainterp/blackhole.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/blackhole.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/blackhole.py Fri Oct 8 13:40:10 2010 @@ -1024,10 +1024,6 @@ def bhimpl_arraylen_gc(cpu, array, arraydescr): return cpu.bh_arraylen_gc(arraydescr, array) - @arguments("cpu", "d", "i", "r", "r", "i", "i", "i", "d") - def bhimpl_arraycopy(cpu, calldescr, func, x1, x2, x3, x4, x5, arraydescr): - cpu.bh_call_v(func, calldescr, [x3, x4, x5], [x1, x2], None) - @arguments("cpu", "r", "d", "d", "i", returns="i") def bhimpl_getarrayitem_vable_i(cpu, vable, fielddescr, arraydescr, index): array = cpu.bh_getfield_gc_r(vable, fielddescr) Modified: pypy/branch/jitffi/pypy/jit/metainterp/executor.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/executor.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/executor.py Fri Oct 8 13:40:10 2010 @@ -2,7 +2,7 @@ """ import py -from pypy.rpython.lltypesystem import lltype, llmemory +from pypy.rpython.lltypesystem import lltype, llmemory, rstr from pypy.rpython.ootypesystem import ootype from pypy.rpython.lltypesystem.lloperation import llop from pypy.rlib.rarithmetic import ovfcheck, r_uint, intmask @@ -168,12 +168,6 @@ def do_new_with_vtable(cpu, _, clsbox): return BoxPtr(exec_new_with_vtable(cpu, clsbox)) -def do_arraycopy(cpu, _, calldescr, funcbox, x1box, x2box, - x3box, x4box, x5box, arraydescr): - cpu.bh_call_v(funcbox.getint(), calldescr, - [x3box.getint(), x4box.getint(), x5box.getint()], - [x1box.getref_base(), x2box.getref_base()], None) - def do_int_add_ovf(cpu, metainterp, box1, box2): # the overflow operations can be called without a metainterp, if an # overflow cannot occur @@ -212,6 +206,24 @@ def do_same_as(cpu, _, box): return box.clonebox() +def do_copystrcontent(cpu, _, srcbox, dstbox, + srcstartbox, dststartbox, lengthbox): + src = srcbox.getref(lltype.Ptr(rstr.STR)) + dst = dstbox.getref(lltype.Ptr(rstr.STR)) + srcstart = srcstartbox.getint() + dststart = dststartbox.getint() + length = lengthbox.getint() + rstr.copy_string_contents(src, dst, srcstart, dststart, length) + +def do_copyunicodecontent(cpu, _, srcbox, dstbox, + srcstartbox, dststartbox, lengthbox): + src = srcbox.getref(lltype.Ptr(rstr.UNICODE)) + dst = dstbox.getref(lltype.Ptr(rstr.UNICODE)) + srcstart = srcstartbox.getint() + dststart = dststartbox.getint() + length = lengthbox.getint() + rstr.copy_unicode_contents(src, dst, srcstart, dststart, length) + # ____________________________________________________________ ##def do_force_token(cpu): @@ -419,6 +431,10 @@ if arity == 3: func = get_execute_funclist(3, False)[opnum] return func(cpu, metainterp, argboxes[0], argboxes[1], argboxes[2]) + if arity == 5: # copystrcontent, copyunicodecontent + func = get_execute_funclist(5, False)[opnum] + return func(cpu, metainterp, argboxes[0], argboxes[1], + argboxes[2], argboxes[3], argboxes[4]) raise NotImplementedError Modified: pypy/branch/jitffi/pypy/jit/metainterp/gc.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/gc.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/gc.py Fri Oct 8 13:40:10 2010 @@ -19,6 +19,9 @@ class GC_hybrid(GcDescription): malloc_zero_filled = True +class GC_minimark(GcDescription): + malloc_zero_filled = True + def get_description(config): name = config.translation.gc Modified: pypy/branch/jitffi/pypy/jit/metainterp/history.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/history.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/history.py Fri Oct 8 13:40:10 2010 @@ -685,6 +685,19 @@ return llmemory.cast_adr_to_int(adr, "emulated") return i +def get_const_ptr_for_string(s): + from pypy.rpython.annlowlevel import llstr + if not we_are_translated(): + try: + return _const_ptr_for_string[s] + except KeyError: + pass + result = ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, llstr(s))) + if not we_are_translated(): + _const_ptr_for_string[s] = result + return result +_const_ptr_for_string = {} + # ____________________________________________________________ # The TreeLoop class contains a loop or a generalized loop, i.e. a tree Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizefindnode.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizefindnode.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizefindnode.py Fri Oct 8 13:40:10 2010 @@ -174,7 +174,7 @@ find_nodes_PTR_EQ = find_nodes_no_escape find_nodes_PTR_NE = find_nodes_no_escape - find_nodes_INSTANCEOF = find_nodes_no_escape + ##find_nodes_INSTANCEOF = find_nodes_no_escape find_nodes_GUARD_NONNULL = find_nodes_no_escape find_nodes_GUARD_ISNULL = find_nodes_no_escape @@ -319,11 +319,12 @@ class PerfectSpecializationFinder(NodeFinder): node_fromstart = InstanceNode(fromstart=True) - def find_nodes_loop(self, loop): + def find_nodes_loop(self, loop, build_specnodes=True): self._loop = loop self.setup_input_nodes(loop.inputargs) self.find_nodes(loop.operations) - self.build_result_specnodes(loop) + if build_specnodes: + self.build_result_specnodes(loop) def show(self): from pypy.jit.metainterp.viewnode import viewnodes, view Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/__init__.py Fri Oct 8 13:40:10 2010 @@ -4,6 +4,7 @@ from pypy.jit.metainterp.optimizeopt.virtualize import OptVirtualize from pypy.jit.metainterp.optimizeopt.heap import OptHeap from pypy.jit.metainterp.optimizeopt.fficall import OptFfiCall +from pypy.jit.metainterp.optimizeopt.string import OptString def optimize_loop_1(metainterp_sd, loop, virtuals=True): """Optimize loop.operations to make it match the input of loop.specnodes @@ -14,6 +15,7 @@ optimizations = [OptIntBounds(), OptRewrite(), OptVirtualize(), + OptString(), OptHeap(), OptFfiCall(), ] @@ -25,4 +27,3 @@ expect 'specnodes' on the bridge. """ optimize_loop_1(metainterp_sd, bridge, False) - Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/optimizer.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/optimizer.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/optimizer.py Fri Oct 8 13:40:10 2010 @@ -12,6 +12,7 @@ from pypy.rpython.lltypesystem import lltype from pypy.jit.metainterp.history import AbstractDescr, make_hashable_int from pypy.jit.metainterp.optimizeopt.intutils import IntBound, IntUnbounded +from pypy.tool.pairtype import extendabletype LEVEL_UNKNOWN = '\x00' LEVEL_NONNULL = '\x01' @@ -23,6 +24,7 @@ MININT = -sys.maxint - 1 class OptValue(object): + __metaclass__ = extendabletype _attrs_ = ('box', 'known_class', 'last_guard_index', 'level', 'intbound') last_guard_index = -1 @@ -126,6 +128,7 @@ def setitem(self, index, value): raise NotImplementedError + class ConstantValue(OptValue): def __init__(self, box): self.make_constant(box) @@ -134,6 +137,7 @@ CONST_1 = ConstInt(1) CVAL_ZERO = ConstantValue(CONST_0) CVAL_ZERO_FLOAT = ConstantValue(ConstFloat(0.0)) +CVAL_UNINITIALIZED_ZERO = ConstantValue(CONST_0) llhelper.CVAL_NULLREF = ConstantValue(llhelper.CONST_NULL) oohelper.CVAL_NULLREF = ConstantValue(oohelper.CONST_NULL) @@ -249,6 +253,7 @@ return None def make_equal_to(self, box, value): + assert isinstance(value, OptValue) assert box not in self.values self.values[box] = value @@ -306,6 +311,9 @@ # accumulate counters self.resumedata_memo.update_counters(self.metainterp_sd.profiler) + def send_extra_operation(self, op): + self.first_optimization.propagate_forward(op) + def propagate_forward(self, op): self.producer[op.result] = op opnum = op.getopnum() Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/rewrite.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/rewrite.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/rewrite.py Fri Oct 8 13:40:10 2010 @@ -140,6 +140,7 @@ args = op.getarglist()[1:] self.emit_operation(ResOperation(rop.CALL, args, op.result, op.getdescr())) + def optimize_guard(self, op, constbox, emit_operation=True): value = self.getvalue(op.getarg(0)) if value.is_constant(): @@ -244,6 +245,9 @@ def optimize_CALL_LOOPINVARIANT(self, op): funcvalue = self.getvalue(op.getarg(0)) if not funcvalue.is_constant(): + # XXX this code path is never executed in tests nor in production. + # in fact, it can't even happen since residual_call in codewriter + # expects a compile-time constant self.emit_operation(op) return key = make_hashable_int(op.getarg(0).getint()) @@ -310,17 +314,17 @@ def optimize_PTR_EQ(self, op): self._optimize_oois_ooisnot(op, False) - def optimize_INSTANCEOF(self, op): - value = self.getvalue(op.getarg(0)) - realclassbox = value.get_constant_class(self.optimizer.cpu) - if realclassbox is not None: - checkclassbox = self.optimizer.cpu.typedescr2classbox(op.getdescr()) - result = self.optimizer.cpu.ts.subclassOf(self.optimizer.cpu, - realclassbox, - checkclassbox) - self.make_constant_int(op.result, result) - return - self.emit_operation(op) +## def optimize_INSTANCEOF(self, op): +## value = self.getvalue(op.args[0]) +## realclassbox = value.get_constant_class(self.optimizer.cpu) +## if realclassbox is not None: +## checkclassbox = self.optimizer.cpu.typedescr2classbox(op.descr) +## result = self.optimizer.cpu.ts.subclassOf(self.optimizer.cpu, +## realclassbox, +## checkclassbox) +## self.make_constant_int(op.result, result) +## return +## self.emit_operation(op) optimize_ops = _findall(OptRewrite, 'optimize_') Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/virtualize.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/virtualize.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/virtualize.py Fri Oct 8 13:40:10 2010 @@ -193,12 +193,12 @@ itemboxes.append(itemvalue.get_key_box()) modifier.register_virtual_fields(self.keybox, itemboxes) for itemvalue in self._items: - if itemvalue is not self.constvalue: - itemvalue.get_args_for_fail(modifier) + itemvalue.get_args_for_fail(modifier) def _make_virtual(self, modifier): return modifier.make_varray(self.arraydescr) + class __extend__(SpecNode): def setup_virtual_node(self, optimizer, box, newinputargs): raise NotImplementedError @@ -329,14 +329,13 @@ vrefinfo = self.optimizer.metainterp_sd.virtualref_info # op.getarg(1) should really never point to null here # - set 'forced' to point to the real object - op1 = ResOperation(rop.SETFIELD_GC, op.getarglist(), None, - descr = vrefinfo.descr_forced) - self.optimize_SETFIELD_GC(op1) + seo = self.optimizer.send_extra_operation + seo(ResOperation(rop.SETFIELD_GC, op.getarglist(), None, + descr = vrefinfo.descr_forced)) # - set 'virtual_token' to TOKEN_NONE args = [op.getarg(0), ConstInt(vrefinfo.TOKEN_NONE)] - op1 = ResOperation(rop.SETFIELD_GC, args, None, - descr = vrefinfo.descr_virtual_token) - self.optimize_SETFIELD_GC(op1) + seo(ResOperation(rop.SETFIELD_GC, args, None, + descr = vrefinfo.descr_virtual_token)) # Note that in some cases the virtual in op.getarg(1) has been forced # already. This is fine. In that case, and *if* a residual # CALL_MAY_FORCE suddenly turns out to access it, then it will @@ -363,8 +362,8 @@ def optimize_SETFIELD_GC(self, op): value = self.getvalue(op.getarg(0)) - fieldvalue = self.getvalue(op.getarg(1)) if value.is_virtual(): + fieldvalue = self.getvalue(op.getarg(1)) value.setfield(op.getdescr(), fieldvalue) else: value.ensure_nonnull() @@ -387,7 +386,7 @@ descr=op.getdescr()) self.make_varray(op.getdescr(), sizebox.getint(), op.result, op) else: - ###self.optimize_default(op) + self.getvalue(op.result).ensure_nonnull() self.emit_operation(op) def optimize_ARRAYLEN_GC(self, op): @@ -426,30 +425,6 @@ ###self.heap_op_optimizer.optimize_SETARRAYITEM_GC(op, value, fieldvalue) self.emit_operation(op) - def optimize_ARRAYCOPY(self, op): - source_value = self.getvalue(op.getarg(2)) - dest_value = self.getvalue(op.getarg(3)) - source_start_box = self.get_constant_box(op.getarg(4)) - dest_start_box = self.get_constant_box(op.getarg(5)) - length = self.get_constant_box(op.getarg(6)) - if (source_value.is_virtual() and source_start_box and dest_start_box - and length and dest_value.is_virtual()): - # XXX optimize the case where dest value is not virtual, - # but we still can avoid a mess - source_start = source_start_box.getint() - dest_start = dest_start_box.getint() - for index in range(length.getint()): - val = source_value.getitem(index + source_start) - dest_value.setitem(index + dest_start, val) - return - if length and length.getint() == 0: - return # 0-length arraycopy - descr = op.getarg(0) - assert isinstance(descr, AbstractDescr) - args = op.getarglist()[1:] - self.emit_operation(ResOperation(rop.CALL, args, op.result, - descr)) - def propagate_forward(self, op): opnum = op.getopnum() for value, func in optimize_ops: Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeutil.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeutil.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeutil.py Fri Oct 8 13:40:10 2010 @@ -14,6 +14,11 @@ def _findall(Class, name_prefix): result = [] + for name in dir(Class): + if name.startswith(name_prefix): + opname = name[len(name_prefix):] + if opname.isupper(): + assert hasattr(resoperation.rop, opname) for value, name in resoperation.opname.items(): if hasattr(Class, name_prefix + name): result.append((value, getattr(Class, name_prefix + name))) Modified: pypy/branch/jitffi/pypy/jit/metainterp/pyjitpl.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/pyjitpl.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/pyjitpl.py Fri Oct 8 13:40:10 2010 @@ -421,14 +421,6 @@ def opimpl_arraylen_gc(self, arraybox, arraydescr): return self.execute_with_descr(rop.ARRAYLEN_GC, arraydescr, arraybox) - @arguments("descr", "box", "box", "box", "box", "box", "box", "descr") - def opimpl_arraycopy(self, calldescr, fnptr, sourcebox, destbox, - source_startbox, dest_startbox, lengthbox, - arraydescr): - self.execute_with_descr(rop.ARRAYCOPY, arraydescr, calldescr, fnptr, - sourcebox, destbox, source_startbox, - dest_startbox, lengthbox) - @arguments("orgpc", "box", "descr", "box") def opimpl_check_neg_index(self, orgpc, arraybox, arraydescr, indexbox): negbox = self.metainterp.execute_and_record( Modified: pypy/branch/jitffi/pypy/jit/metainterp/resoperation.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/resoperation.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/resoperation.py Fri Oct 8 13:40:10 2010 @@ -450,7 +450,6 @@ 'SETARRAYITEM_RAW/3d', 'SETFIELD_GC/2d', 'SETFIELD_RAW/2d', - 'ARRAYCOPY/7d', # removed before it's passed to the backend 'NEWSTR/1', 'STRSETITEM/3', 'UNICODESETITEM/3', @@ -459,6 +458,8 @@ 'COND_CALL_GC_WB/2d', # [objptr, newvalue] (for the write barrier) 'DEBUG_MERGE_POINT/1', # debugging only 'VIRTUAL_REF_FINISH/2', # removed before it's passed to the backend + 'COPYSTRCONTENT/5', # src, dst, srcstart, dststart, length + 'COPYUNICODECONTENT/5', '_CANRAISE_FIRST', # ----- start of can_raise operations ----- 'CALL/*d', Modified: pypy/branch/jitffi/pypy/jit/metainterp/resume.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/resume.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/resume.py Fri Oct 8 13:40:10 2010 @@ -4,10 +4,12 @@ from pypy.jit.metainterp.history import INT, REF, FLOAT, HOLE from pypy.jit.metainterp.resoperation import rop from pypy.jit.metainterp import jitprof -from pypy.rpython.lltypesystem import lltype, llmemory, rffi +from pypy.jit.codewriter.effectinfo import EffectInfo, callinfo_for_oopspec +from pypy.jit.codewriter.effectinfo import funcptr_for_oopspec +from pypy.rpython.lltypesystem import lltype, llmemory, rffi, rstr from pypy.rlib import rarithmetic from pypy.rlib.objectmodel import we_are_translated, specialize -from pypy.rlib.debug import have_debug_prints +from pypy.rlib.debug import have_debug_prints, ll_assert from pypy.rlib.debug import debug_start, debug_stop, debug_print # Logic to encode the chain of frames and the state of the boxes at a @@ -253,6 +255,15 @@ def make_varray(self, arraydescr): return VArrayInfo(arraydescr) + def make_vstrplain(self): + return VStrPlainInfo() + + def make_vstrconcat(self): + return VStrConcatInfo() + + def make_vstrslice(self): + return VStrSliceInfo() + def register_virtual_fields(self, virtualbox, fieldboxes): tagged = self.liveboxes_from_env.get(virtualbox, UNASSIGNEDVIRTUAL) self.liveboxes[virtualbox] = tagged @@ -397,9 +408,7 @@ class AbstractVirtualInfo(object): - #def allocate(self, metainterp): - # raise NotImplementedError - #def setfields(self, decoder, struct): + #def allocate(self, decoder, index): # raise NotImplementedError def equals(self, fieldnums): return tagged_list_eq(self.fieldnums, fieldnums) @@ -419,6 +428,7 @@ for i in range(len(self.fielddescrs)): descr = self.fielddescrs[i] decoder.setfield(descr, struct, self.fieldnums[i]) + return struct def debug_prints(self): assert len(self.fielddescrs) == len(self.fieldnums) @@ -433,8 +443,10 @@ self.known_class = known_class @specialize.argtype(1) - def allocate(self, decoder): - return decoder.allocate_with_vtable(self.known_class) + def allocate(self, decoder, index): + struct = decoder.allocate_with_vtable(self.known_class) + decoder.virtuals_cache[index] = struct + return self.setfields(decoder, struct) def debug_prints(self): debug_print("\tvirtualinfo", self.known_class.repr_rpython()) @@ -446,8 +458,10 @@ self.typedescr = typedescr @specialize.argtype(1) - def allocate(self, decoder): - return decoder.allocate_struct(self.typedescr) + def allocate(self, decoder, index): + struct = decoder.allocate_struct(self.typedescr) + decoder.virtuals_cache[index] = struct + return self.setfields(decoder, struct) def debug_prints(self): debug_print("\tvstructinfo", self.typedescr.repr_rpython()) @@ -459,14 +473,11 @@ #self.fieldnums = ... @specialize.argtype(1) - def allocate(self, decoder): + def allocate(self, decoder, index): length = len(self.fieldnums) - return decoder.allocate_array(self.arraydescr, length) - - @specialize.argtype(1) - def setfields(self, decoder, array): arraydescr = self.arraydescr - length = len(self.fieldnums) + array = decoder.allocate_array(arraydescr, length) + decoder.virtuals_cache[index] = array # NB. the check for the kind of array elements is moved out of the loop if arraydescr.is_array_of_pointers(): for i in range(length): @@ -480,12 +491,65 @@ for i in range(length): decoder.setarrayitem_int(arraydescr, array, i, self.fieldnums[i]) + return array def debug_prints(self): debug_print("\tvarrayinfo", self.arraydescr) for i in self.fieldnums: debug_print("\t\t", str(untag(i))) + +class VStrPlainInfo(AbstractVirtualInfo): + """Stands for the string made out of the characters of all fieldnums.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + length = len(self.fieldnums) + string = decoder.allocate_string(length) + decoder.virtuals_cache[index] = string + for i in range(length): + decoder.string_setitem(string, i, self.fieldnums[i]) + return string + + def debug_prints(self): + debug_print("\tvstrplaininfo length", len(self.fieldnums)) + + +class VStrConcatInfo(AbstractVirtualInfo): + """Stands for the string made out of the concatenation of two + other strings.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + # xxx for blackhole resuming, this will build all intermediate + # strings and throw them away immediately, which is a bit sub- + # efficient. Not sure we care. + left, right = self.fieldnums + string = decoder.concat_strings(left, right) + decoder.virtuals_cache[index] = string + return string + + def debug_prints(self): + debug_print("\tvstrconcatinfo") + for i in self.fieldnums: + debug_print("\t\t", str(untag(i))) + + +class VStrSliceInfo(AbstractVirtualInfo): + """Stands for the string made out of slicing another string.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + largerstr, start, length = self.fieldnums + string = decoder.slice_string(largerstr, start, length) + decoder.virtuals_cache[index] = string + return string + + def debug_prints(self): + debug_print("\tvstrsliceinfo") + for i in self.fieldnums: + debug_print("\t\t", str(untag(i))) + # ____________________________________________________________ class AbstractResumeDataReader(object): @@ -496,7 +560,8 @@ blackholing and want the best performance. """ _mixin_ = True - virtuals = None + rd_virtuals = None + virtuals_cache = None virtual_default = None def _init(self, cpu, storage): @@ -508,17 +573,29 @@ self._prepare_virtuals(storage.rd_virtuals) self._prepare_pendingfields(storage.rd_pendingfields) + def getvirtual(self, index): + # Returns the index'th virtual, building it lazily if needed. + # Note that this may be called recursively; that's why the + # allocate() methods must fill in the cache as soon as they + # have the object, before they fill its fields. + v = self.virtuals_cache[index] + if not v: + v = self.rd_virtuals[index].allocate(self, index) + ll_assert(v == self.virtuals_cache[index], "resume.py: bad cache") + return v + + def force_all_virtuals(self): + rd_virtuals = self.rd_virtuals + if rd_virtuals: + for i in range(len(rd_virtuals)): + if rd_virtuals[i] is not None: + self.getvirtual(i) + return self.virtuals_cache + def _prepare_virtuals(self, virtuals): if virtuals: - self.virtuals = [self.virtual_default] * len(virtuals) - for i in range(len(virtuals)): - vinfo = virtuals[i] - if vinfo is not None: - self.virtuals[i] = vinfo.allocate(self) - for i in range(len(virtuals)): - vinfo = virtuals[i] - if vinfo is not None: - vinfo.setfields(self, self.virtuals[i]) + self.rd_virtuals = virtuals + self.virtuals_cache = [self.virtual_default] * len(virtuals) def _prepare_pendingfields(self, pendingfields): if pendingfields is not None: @@ -622,6 +699,32 @@ return self.metainterp.execute_and_record(rop.NEW_ARRAY, arraydescr, ConstInt(length)) + def allocate_string(self, length): + return self.metainterp.execute_and_record(rop.NEWSTR, + None, ConstInt(length)) + + def string_setitem(self, strbox, index, charnum): + charbox = self.decode_box(charnum, INT) + self.metainterp.execute_and_record(rop.STRSETITEM, None, + strbox, ConstInt(index), charbox) + + def concat_strings(self, str1num, str2num): + calldescr, func = callinfo_for_oopspec(EffectInfo.OS_STR_CONCAT) + str1box = self.decode_box(str1num, REF) + str2box = self.decode_box(str2num, REF) + return self.metainterp.execute_and_record_varargs( + rop.CALL, [ConstInt(func), str1box, str2box], calldescr) + + def slice_string(self, strnum, startnum, lengthnum): + calldescr, func = callinfo_for_oopspec(EffectInfo.OS_STR_SLICE) + strbox = self.decode_box(strnum, REF) + startbox = self.decode_box(startnum, INT) + lengthbox = self.decode_box(lengthnum, INT) + stopbox = self.metainterp.execute_and_record(rop.INT_ADD, None, + startbox, lengthbox) + return self.metainterp.execute_and_record_varargs( + rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr) + def setfield(self, descr, structbox, fieldnum): if descr.is_pointer_field(): kind = REF @@ -663,9 +766,7 @@ else: box = self.consts[num] elif tag == TAGVIRTUAL: - virtuals = self.virtuals - assert virtuals is not None - box = virtuals[num] + box = self.getvirtual(num) elif tag == TAGINT: box = ConstInt(num) else: @@ -750,7 +851,7 @@ resumereader.handling_async_forcing() vrefinfo = metainterp_sd.virtualref_info resumereader.consume_vref_and_vable(vrefinfo, vinfo) - return resumereader.virtuals + return resumereader.force_all_virtuals() class ResumeDataDirectReader(AbstractResumeDataReader): unique_id = lambda: None @@ -768,7 +869,9 @@ # special case for resuming after a GUARD_NOT_FORCED: we already # have the virtuals self.resume_after_guard_not_forced = 2 - self.virtuals = all_virtuals + self.virtuals_cache = all_virtuals + # self.rd_virtuals can remain None, because virtuals_cache is + # already filled def handling_async_forcing(self): self.resume_after_guard_not_forced = 1 @@ -839,6 +942,31 @@ def allocate_array(self, arraydescr, length): return self.cpu.bh_new_array(arraydescr, length) + def allocate_string(self, length): + return self.cpu.bh_newstr(length) + + def string_setitem(self, str, index, charnum): + char = self.decode_int(charnum) + self.cpu.bh_strsetitem(str, index, char) + + def concat_strings(self, str1num, str2num): + str1 = self.decode_ref(str1num) + str2 = self.decode_ref(str2num) + str1 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.STR), str1) + str2 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.STR), str2) + funcptr = funcptr_for_oopspec(EffectInfo.OS_STR_CONCAT) + result = funcptr(str1, str2) + return lltype.cast_opaque_ptr(llmemory.GCREF, result) + + def slice_string(self, strnum, startnum, lengthnum): + str = self.decode_ref(strnum) + start = self.decode_int(startnum) + length = self.decode_int(lengthnum) + str = lltype.cast_opaque_ptr(lltype.Ptr(rstr.STR), str) + funcptr = funcptr_for_oopspec(EffectInfo.OS_STR_SLICE) + result = funcptr(str, start, start + length) + return lltype.cast_opaque_ptr(llmemory.GCREF, result) + def setfield(self, descr, struct, fieldnum): if descr.is_pointer_field(): newvalue = self.decode_ref(fieldnum) @@ -881,9 +1009,7 @@ return self.cpu.ts.NULLREF return self.consts[num].getref_base() elif tag == TAGVIRTUAL: - virtuals = self.virtuals - assert virtuals is not None - return virtuals[num] + return self.getvirtual(num) else: assert tag == TAGBOX if num < 0: Modified: pypy/branch/jitffi/pypy/jit/metainterp/simple_optimize.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/simple_optimize.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/simple_optimize.py Fri Oct 8 13:40:10 2010 @@ -9,16 +9,13 @@ def transform(op): from pypy.jit.metainterp.history import AbstractDescr - # change ARRAYCOPY to call, so we don't have to pass around - # unnecessary information to the backend. Do the same with VIRTUAL_REF_*. - if op.getopnum() == rop.ARRAYCOPY: - descr = op.getarg(0) - assert isinstance(descr, AbstractDescr) - args = op.getarglist()[1:] - op = ResOperation(rop.CALL, args, op.result, descr=descr) - elif op.getopnum() == rop.CALL_PURE: - args = op.getarglist()[1:] - op = ResOperation(rop.CALL, args, op.result, op.getdescr()) + # Rename CALL_PURE and CALL_LOOPINVARIANT to CALL. + # Simplify the VIRTUAL_REF_* so that they don't show up in the backend. + if op.getopnum() == rop.CALL_PURE: + op = ResOperation(rop.CALL, op.getarglist()[1:], op.result, + op.getdescr()) + elif op.getopnum() == rop.CALL_LOOPINVARIANT: + op = op.copy_and_change(rop.CALL) elif op.getopnum() == rop.VIRTUAL_REF: op = ResOperation(rop.SAME_AS, [op.getarg(0)], op.result) elif op.getopnum() == rop.VIRTUAL_REF_FINISH: Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py Fri Oct 8 13:40:10 2010 @@ -5,13 +5,12 @@ from pypy.jit.metainterp.history import TreeLoop, BoxInt, ConstInt,\ ConstObj, ConstPtr, Box, BasicFailDescr, BoxFloat, ConstFloat,\ - LoopToken + LoopToken, get_const_ptr_for_string from pypy.jit.metainterp.resoperation import rop, ResOperation, ResOpWithDescr, N_aryOp from pypy.jit.metainterp.typesystem import llhelper from pypy.jit.codewriter.heaptracker import adr2int from pypy.rpython.lltypesystem import lltype, llmemory from pypy.rpython.ootypesystem import ootype -from pypy.rpython.annlowlevel import llstr class ParseError(Exception): pass @@ -64,7 +63,8 @@ class OpParser(object): def __init__(self, input, cpu, namespace, type_system, boxkinds, - invent_fail_descr=default_fail_descr): + invent_fail_descr=default_fail_descr, + nonstrict=False): self.input = input self.vars = {} self.cpu = cpu @@ -76,6 +76,7 @@ else: self._cache = {} self.invent_fail_descr = invent_fail_descr + self.nonstrict = nonstrict self.looptoken = LoopToken() def get_const(self, name, typ): @@ -134,11 +135,14 @@ vars = [] for elem in elements: elem = elem.strip() - box = self.box_for_var(elem) - vars.append(box) - self.vars[elem] = box + vars.append(self.newvar(elem)) return vars + def newvar(self, elem): + box = self.box_for_var(elem) + self.vars[elem] = box + return box + def is_float(self, arg): try: float(arg) @@ -157,8 +161,7 @@ if arg.startswith('"') or arg.startswith("'"): # XXX ootype info = arg.strip("'\"") - llinfo = self._cache.setdefault(info, llstr(info)) - return ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, llinfo)) + return get_const_ptr_for_string(info) if arg.startswith('ConstClass('): name = arg[len('ConstClass('):-1] return self.get_const(name, 'class') @@ -172,6 +175,8 @@ elif arg.startswith('ConstPtr('): name = arg[len('ConstPtr('):-1] return self.get_const(name, 'ptr') + if arg not in self.vars and self.nonstrict: + self.newvar(arg) return self.vars[arg] def parse_op(self, line): @@ -196,7 +201,8 @@ if opname == 'debug_merge_point': allargs = [argspec] else: - allargs = argspec.split(",") + allargs = [arg for arg in argspec.split(",") + if arg != ''] poss_descr = allargs[-1].strip() if poss_descr.startswith('descr='): @@ -211,7 +217,7 @@ if rop._GUARD_FIRST <= opnum <= rop._GUARD_LAST: i = line.find('[', endnum) + 1 j = line.find(']', i) - if i <= 0 or j <= 0: + if (i <= 0 or j <= 0) and not self.nonstrict: raise ParseError("missing fail_args for guard operation") fail_args = [] if i < j: @@ -277,11 +283,14 @@ lines = self.input.splitlines() ops = [] newlines = [] + first_comment = None for line in lines: # for simplicity comments are not allowed on # debug_merge_point lines if '#' in line and 'debug_merge_point(' not in line: if line.lstrip()[0] == '#': # comment only + if first_comment is None: + first_comment = line continue comm = line.rfind('#') rpar = line.find(')') # assume there's a op(...) @@ -290,12 +299,12 @@ if not line.strip(): continue # a comment or empty line newlines.append(line) - base_indent, inpargs = self.parse_inpargs(newlines[0]) - newlines = newlines[1:] + base_indent, inpargs, newlines = self.parse_inpargs(newlines) num, ops = self.parse_ops(base_indent, newlines, 0) if num < len(newlines): raise ParseError("unexpected dedent at line: %s" % newlines[num]) loop = ExtendedTreeLoop("loop") + loop.comment = first_comment loop.token = self.looptoken loop.operations = ops loop.inputargs = inpargs @@ -316,23 +325,27 @@ num += 1 return num, ops - def parse_inpargs(self, line): - base_indent = line.find('[') + def parse_inpargs(self, lines): + line = lines[0] + base_indent = len(line) - len(line.lstrip(' ')) line = line.strip() + if not line.startswith('[') and self.nonstrict: + return base_indent, [], lines + lines = lines[1:] if line == '[]': - return base_indent, [] - if base_indent == -1 or not line.endswith(']'): + return base_indent, [], lines + if not line.startswith('[') or not line.endswith(']'): raise ParseError("Wrong header: %s" % line) inpargs = self.parse_header_line(line[1:-1]) - return base_indent, inpargs + return base_indent, inpargs, lines def parse(input, cpu=None, namespace=None, type_system='lltype', boxkinds=None, invent_fail_descr=default_fail_descr, - no_namespace=False): + no_namespace=False, nonstrict=False): if namespace is None and not no_namespace: namespace = {} return OpParser(input, cpu, namespace, type_system, boxkinds, - invent_fail_descr).parse() + invent_fail_descr, nonstrict).parse() def pure_parse(*args, **kwds): kwds['invent_fail_descr'] = None Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_oparser.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_oparser.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_oparser.py Fri Oct 8 13:40:10 2010 @@ -174,3 +174,32 @@ def test_parse_no_namespace(): loop = parse(example_loop_log, no_namespace=True) + +def test_attach_comment_to_loop(): + loop = parse(example_loop_log, no_namespace=True) + assert loop.comment == '# bridge out of Guard12, 6 ops' + +def test_parse_new_with_comma(): + # this is generated by PYPYJITLOG, check that we can handle it + x = ''' + [] + p0 = new(, descr=) + ''' + loop = parse(x) + assert loop.operations[0].getopname() == 'new' + +def test_no_fail_args(): + x = ''' + [i0] + guard_true(i0, descr=) + ''' + loop = parse(x, nonstrict=True) + assert loop.operations[0].getfailargs() == [] + +def test_no_inputargs(): + x = ''' + i2 = int_add(i0, i1) + ''' + loop = parse(x, nonstrict=True) + assert loop.inputargs == [] + assert loop.operations[0].getopname() == 'int_add' Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefindnode.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefindnode.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefindnode.py Fri Oct 8 13:40:10 2010 @@ -1,6 +1,6 @@ import py, random -from pypy.rpython.lltypesystem import lltype, llmemory, rclass +from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr from pypy.rpython.ootypesystem import ootype from pypy.rpython.lltypesystem.rclass import OBJECT, OBJECT_VTABLE @@ -115,6 +115,36 @@ mayforcevirtdescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, EffectInfo([nextdescr], [], [], EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE)) + arraycopydescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=EffectInfo.OS_ARRAYCOPY)) + strconcatdescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_CONCAT)) + slicedescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_SLICE)) + strequaldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_EQUAL)) + streq_slice_checknull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], + oopspecindex=EffectInfo.OS_STREQ_SLICE_CHECKNULL)) + streq_slice_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], + oopspecindex=EffectInfo.OS_STREQ_SLICE_NONNULL)) + streq_slice_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], + oopspecindex=EffectInfo.OS_STREQ_SLICE_CHAR)) + streq_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], + oopspecindex=EffectInfo.OS_STREQ_NONNULL)) + streq_nonnull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], + oopspecindex=EffectInfo.OS_STREQ_NONNULL_CHAR)) + streq_checknull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], + oopspecindex=EffectInfo.OS_STREQ_CHECKNULL_CHAR)) + streq_lengthok_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], + oopspecindex=EffectInfo.OS_STREQ_LENGTHOK)) + class LoopToken(AbstractDescr): pass asmdescr = LoopToken() # it can be whatever, it's not a descr though Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py Fri Oct 8 13:40:10 2010 @@ -3086,7 +3086,7 @@ setarrayitem_gc(p1, 1, 1, descr=arraydescr) p2 = new_array(3, descr=arraydescr) setarrayitem_gc(p2, 1, 3, descr=arraydescr) - arraycopy(0, 0, p1, p2, 1, 1, 2, descr=arraydescr) + call(0, p1, p2, 1, 1, 2, descr=arraycopydescr) i2 = getarrayitem_gc(p2, 1, descr=arraydescr) jump(i2) ''' @@ -3103,7 +3103,7 @@ p2 = new_array(3, descr=arraydescr) setarrayitem_gc(p1, 0, i0, descr=arraydescr) setarrayitem_gc(p2, 0, 3, descr=arraydescr) - arraycopy(0, 0, p1, p2, 1, 1, 2, descr=arraydescr) + call(0, p1, p2, 1, 1, 2, descr=arraycopydescr) i2 = getarrayitem_gc(p2, 0, descr=arraydescr) jump(i2) ''' @@ -3120,7 +3120,7 @@ p2 = new_array(3, descr=arraydescr) setarrayitem_gc(p1, 2, 10, descr=arraydescr) setarrayitem_gc(p2, 2, 13, descr=arraydescr) - arraycopy(0, 0, p1, p2, 0, 0, 3, descr=arraydescr) + call(0, p1, p2, 0, 0, 3, descr=arraycopydescr) jump(p2) ''' expected = ''' @@ -3137,7 +3137,7 @@ ops = ''' [p1] p0 = new_array(0, descr=arraydescr) - arraycopy(0, 0, p0, p1, 0, 0, 0, descr=arraydescr) + call(0, p0, p1, 0, 0, 0, descr=arraycopydescr) jump(p1) ''' expected = ''' @@ -3897,6 +3897,607 @@ """ self.optimize_loop(ops, 'Not, Not', expected) + def test_newstr_1(self): + ops = """ + [i0] + p1 = newstr(1) + strsetitem(p1, 0, i0) + i1 = strgetitem(p1, 0) + jump(i1) + """ + expected = """ + [i0] + jump(i0) + """ + self.optimize_loop(ops, 'Not', expected) + + def test_newstr_2(self): + ops = """ + [i0, i1] + p1 = newstr(2) + strsetitem(p1, 0, i0) + strsetitem(p1, 1, i1) + i2 = strgetitem(p1, 1) + i3 = strgetitem(p1, 0) + jump(i2, i3) + """ + expected = """ + [i0, i1] + jump(i1, i0) + """ + self.optimize_loop(ops, 'Not, Not', expected) + + def test_str_concat_1(self): + ops = """ + [p1, p2] + p3 = call(0, p1, p2, descr=strconcatdescr) + jump(p2, p3) + """ + expected = """ + [p1, p2] + i1 = strlen(p1) + i2 = strlen(p2) + i3 = int_add(i1, i2) + p3 = newstr(i3) + i4 = strlen(p1) + copystrcontent(p1, p3, 0, 0, i4) + i5 = strlen(p2) + i6 = int_add(i4, i5) # will be killed by the backend + copystrcontent(p2, p3, 0, i4, i5) + jump(p2, p3) + """ + self.optimize_loop(ops, 'Not, Not', expected) + + def test_str_concat_vstr2_str(self): + ops = """ + [i0, i1, p2] + p1 = newstr(2) + strsetitem(p1, 0, i0) + strsetitem(p1, 1, i1) + p3 = call(0, p1, p2, descr=strconcatdescr) + jump(i1, i0, p3) + """ + expected = """ + [i0, i1, p2] + i2 = strlen(p2) + i3 = int_add(2, i2) + p3 = newstr(i3) + strsetitem(p3, 0, i0) + strsetitem(p3, 1, i1) + i4 = strlen(p2) + i5 = int_add(2, i4) # will be killed by the backend + copystrcontent(p2, p3, 0, 2, i4) + jump(i1, i0, p3) + """ + self.optimize_loop(ops, 'Not, Not, Not', expected) + + def test_str_concat_str_vstr2(self): + ops = """ + [i0, i1, p2] + p1 = newstr(2) + strsetitem(p1, 0, i0) + strsetitem(p1, 1, i1) + p3 = call(0, p2, p1, descr=strconcatdescr) + jump(i1, i0, p3) + """ + expected = """ + [i0, i1, p2] + i2 = strlen(p2) + i3 = int_add(i2, 2) + p3 = newstr(i3) + i4 = strlen(p2) + copystrcontent(p2, p3, 0, 0, i4) + strsetitem(p3, i4, i0) + i5 = int_add(i4, 1) + strsetitem(p3, i5, i1) + i6 = int_add(i5, 1) # will be killed by the backend + jump(i1, i0, p3) + """ + self.optimize_loop(ops, 'Not, Not, Not', expected) + + def test_str_concat_str_str_str(self): + ops = """ + [p1, p2, p3] + p4 = call(0, p1, p2, descr=strconcatdescr) + p5 = call(0, p4, p3, descr=strconcatdescr) + jump(p2, p3, p5) + """ + expected = """ + [p1, p2, p3] + i1 = strlen(p1) + i2 = strlen(p2) + i12 = int_add(i1, i2) + i3 = strlen(p3) + i123 = int_add(i12, i3) + p5 = newstr(i123) + i1b = strlen(p1) + copystrcontent(p1, p5, 0, 0, i1b) + i2b = strlen(p2) + i12b = int_add(i1b, i2b) + copystrcontent(p2, p5, 0, i1b, i2b) + i3b = strlen(p3) + i123b = int_add(i12b, i3b) # will be killed by the backend + copystrcontent(p3, p5, 0, i12b, i3b) + jump(p2, p3, p5) + """ + self.optimize_loop(ops, 'Not, Not, Not', expected) + + def test_str_concat_str_cstr1(self): + ops = """ + [p2] + p3 = call(0, p2, "x", descr=strconcatdescr) + jump(p3) + """ + expected = """ + [p2] + i2 = strlen(p2) + i3 = int_add(i2, 1) + p3 = newstr(i3) + i4 = strlen(p2) + copystrcontent(p2, p3, 0, 0, i4) + strsetitem(p3, i4, 120) # == ord('x') + i5 = int_add(i4, 1) # will be killed by the backend + jump(p3) + """ + self.optimize_loop(ops, 'Not', expected) + + def test_str_concat_consts(self): + ops = """ + [] + p1 = same_as("ab") + p2 = same_as("cde") + p3 = call(0, p1, p2, descr=strconcatdescr) + escape(p3) + jump() + """ + expected = """ + [] + escape("abcde") + jump() + """ + self.optimize_loop(ops, '', expected) + + def test_str_slice_1(self): + ops = """ + [p1, i1, i2] + p2 = call(0, p1, i1, i2, descr=slicedescr) + jump(p2, i1, i2) + """ + expected = """ + [p1, i1, i2] + i3 = int_sub(i2, i1) + p2 = newstr(i3) + copystrcontent(p1, p2, i1, 0, i3) + jump(p2, i1, i2) + """ + self.optimize_loop(ops, 'Not, Not, Not', expected) + + def test_str_slice_2(self): + ops = """ + [p1, i2] + p2 = call(0, p1, 0, i2, descr=slicedescr) + jump(p2, i2) + """ + expected = """ + [p1, i2] + p2 = newstr(i2) + copystrcontent(p1, p2, 0, 0, i2) + jump(p2, i2) + """ + self.optimize_loop(ops, 'Not, Not', expected) + + def test_str_slice_3(self): + ops = """ + [p1, i1, i2, i3, i4] + p2 = call(0, p1, i1, i2, descr=slicedescr) + p3 = call(0, p2, i3, i4, descr=slicedescr) + jump(p3, i1, i2, i3, i4) + """ + expected = """ + [p1, i1, i2, i3, i4] + i0 = int_sub(i2, i1) # killed by the backend + i5 = int_sub(i4, i3) + i6 = int_add(i1, i3) + p3 = newstr(i5) + copystrcontent(p1, p3, i6, 0, i5) + jump(p3, i1, i2, i3, i4) + """ + self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected) + + def test_str_slice_getitem1(self): + ops = """ + [p1, i1, i2, i3] + p2 = call(0, p1, i1, i2, descr=slicedescr) + i4 = strgetitem(p2, i3) + escape(i4) + jump(p1, i1, i2, i3) + """ + expected = """ + [p1, i1, i2, i3] + i6 = int_sub(i2, i1) # killed by the backend + i5 = int_add(i1, i3) + i4 = strgetitem(p1, i5) + escape(i4) + jump(p1, i1, i2, i3) + """ + self.optimize_loop(ops, 'Not, Not, Not, Not', expected) + + def test_str_slice_plain(self): + ops = """ + [i3, i4] + p1 = newstr(2) + strsetitem(p1, 0, i3) + strsetitem(p1, 1, i4) + p2 = call(0, p1, 1, 2, descr=slicedescr) + i5 = strgetitem(p2, 0) + escape(i5) + jump(i3, i4) + """ + expected = """ + [i3, i4] + escape(i4) + jump(i3, i4) + """ + self.optimize_loop(ops, 'Not, Not', expected) + + def test_str_slice_concat(self): + ops = """ + [p1, i1, i2, p2] + p3 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p3, p2, descr=strconcatdescr) + jump(p4, i1, i2, p2) + """ + expected = """ + [p1, i1, i2, p2] + i3 = int_sub(i2, i1) # length of p3 + i4 = strlen(p2) + i5 = int_add(i3, i4) + p4 = newstr(i5) + copystrcontent(p1, p4, i1, 0, i3) + i4b = strlen(p2) + i6 = int_add(i3, i4b) # killed by the backend + copystrcontent(p2, p4, 0, i3, i4b) + jump(p4, i1, i2, p2) + """ + self.optimize_loop(ops, 'Not, Not, Not, Not', expected) + + # ---------- + def optimize_loop_extradescrs(self, ops, spectext, optops): + from pypy.jit.metainterp.optimizeopt import string + def my_callinfo_for_oopspec(oopspecindex): + calldescrtype = type(LLtypeMixin.strequaldescr) + for value in LLtypeMixin.__dict__.values(): + if isinstance(value, calldescrtype): + if (value.get_extra_info() and + value.get_extra_info().oopspecindex == oopspecindex): + # returns 0 for 'func' in this test + return value, 0 + raise AssertionError("not found: oopspecindex=%d" % oopspecindex) + # + saved = string.callinfo_for_oopspec + try: + string.callinfo_for_oopspec = my_callinfo_for_oopspec + self.optimize_loop(ops, spectext, optops) + finally: + string.callinfo_for_oopspec = saved + + def test_str_equal_noop1(self): + ops = """ + [p1, p2] + i0 = call(0, p1, p2, descr=strequaldescr) + escape(i0) + jump(p1, p2) + """ + self.optimize_loop_extradescrs(ops, 'Not, Not', ops) + + def test_str_equal_noop2(self): + ops = """ + [p1, p2, p3] + p4 = call(0, p1, p2, descr=strconcatdescr) + i0 = call(0, p3, p4, descr=strequaldescr) + escape(i0) + jump(p1, p2, p3) + """ + expected = """ + [p1, p2, p3] + i1 = strlen(p1) + i2 = strlen(p2) + i3 = int_add(i1, i2) + p4 = newstr(i3) + i4 = strlen(p1) + copystrcontent(p1, p4, 0, 0, i4) + i5 = strlen(p2) + i6 = int_add(i4, i5) # will be killed by the backend + copystrcontent(p2, p4, 0, i4, i5) + i0 = call(0, p3, p4, descr=strequaldescr) + escape(i0) + jump(p1, p2, p3) + """ + self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected) + + def test_str_equal_slice1(self): + ops = """ + [p1, i1, i2, p3] + p4 = call(0, p1, i1, i2, descr=slicedescr) + i0 = call(0, p4, p3, descr=strequaldescr) + escape(i0) + jump(p1, i1, i2, p3) + """ + expected = """ + [p1, i1, i2, p3] + i3 = int_sub(i2, i1) + i0 = call(0, p1, i1, i3, p3, descr=streq_slice_checknull_descr) + escape(i0) + jump(p1, i1, i2, p3) + """ + self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + + def test_str_equal_slice2(self): + ops = """ + [p1, i1, i2, p3] + p4 = call(0, p1, i1, i2, descr=slicedescr) + i0 = call(0, p3, p4, descr=strequaldescr) + escape(i0) + jump(p1, i1, i2, p3) + """ + expected = """ + [p1, i1, i2, p3] + i4 = int_sub(i2, i1) + i0 = call(0, p1, i1, i4, p3, descr=streq_slice_checknull_descr) + escape(i0) + jump(p1, i1, i2, p3) + """ + self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + + def test_str_equal_slice3(self): + ops = """ + [p1, i1, i2, p3] + guard_nonnull(p3) [] + p4 = call(0, p1, i1, i2, descr=slicedescr) + i0 = call(0, p3, p4, descr=strequaldescr) + escape(i0) + jump(p1, i1, i2, p3) + """ + expected = """ + [p1, i1, i2, p3] + guard_nonnull(p3) [] + i4 = int_sub(i2, i1) + i0 = call(0, p1, i1, i4, p3, descr=streq_slice_nonnull_descr) + escape(i0) + jump(p1, i1, i2, p3) + """ + self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + + def test_str_equal_slice4(self): + ops = """ + [p1, i1, i2] + p3 = call(0, p1, i1, i2, descr=slicedescr) + i0 = call(0, p3, "x", descr=strequaldescr) + escape(i0) + jump(p1, i1, i2) + """ + expected = """ + [p1, i1, i2] + i3 = int_sub(i2, i1) + i0 = call(0, p1, i1, i3, 120, descr=streq_slice_char_descr) + escape(i0) + jump(p1, i1, i2) + """ + self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected) + + def test_str_equal_slice5(self): + ops = """ + [p1, i1, i2, i3] + p4 = call(0, p1, i1, i2, descr=slicedescr) + p5 = newstr(1) + strsetitem(p5, 0, i3) + i0 = call(0, p5, p4, descr=strequaldescr) + escape(i0) + jump(p1, i1, i2, i3) + """ + expected = """ + [p1, i1, i2, i3] + i4 = int_sub(i2, i1) + i0 = call(0, p1, i1, i4, i3, descr=streq_slice_char_descr) + escape(i0) + jump(p1, i1, i2, i3) + """ + self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + + def test_str_equal_none1(self): + ops = """ + [p1] + i0 = call(0, p1, NULL, descr=strequaldescr) + escape(i0) + jump(p1) + """ + expected = """ + [p1] + i0 = ptr_eq(p1, NULL) + escape(i0) + jump(p1) + """ + self.optimize_loop_extradescrs(ops, 'Not', expected) + + def test_str_equal_none2(self): + ops = """ + [p1] + i0 = call(0, NULL, p1, descr=strequaldescr) + escape(i0) + jump(p1) + """ + expected = """ + [p1] + i0 = ptr_eq(p1, NULL) + escape(i0) + jump(p1) + """ + self.optimize_loop_extradescrs(ops, 'Not', expected) + + def test_str_equal_nonnull1(self): + ops = """ + [p1] + guard_nonnull(p1) [] + i0 = call(0, p1, "hello world", descr=strequaldescr) + escape(i0) + jump(p1) + """ + expected = """ + [p1] + guard_nonnull(p1) [] + i0 = call(0, p1, "hello world", descr=streq_nonnull_descr) + escape(i0) + jump(p1) + """ + self.optimize_loop_extradescrs(ops, 'Not', expected) + + def test_str_equal_nonnull2(self): + ops = """ + [p1] + guard_nonnull(p1) [] + i0 = call(0, p1, "", descr=strequaldescr) + escape(i0) + jump(p1) + """ + expected = """ + [p1] + guard_nonnull(p1) [] + i1 = strlen(p1) + i0 = int_eq(i1, 0) + escape(i0) + jump(p1) + """ + self.optimize_loop_extradescrs(ops, 'Not', expected) + + def test_str_equal_nonnull3(self): + ops = """ + [p1] + guard_nonnull(p1) [] + i0 = call(0, p1, "x", descr=strequaldescr) + escape(i0) + jump(p1) + """ + expected = """ + [p1] + guard_nonnull(p1) [] + i0 = call(0, p1, 120, descr=streq_nonnull_char_descr) + escape(i0) + jump(p1) + """ + self.optimize_loop_extradescrs(ops, 'Not', expected) + + def test_str_equal_nonnull4(self): + ops = """ + [p1, p2] + p4 = call(0, p1, p2, descr=strconcatdescr) + i0 = call(0, "hello world", p4, descr=strequaldescr) + escape(i0) + jump(p1, p2) + """ + expected = """ + [p1, p2] + i1 = strlen(p1) + i2 = strlen(p2) + i3 = int_add(i1, i2) + p4 = newstr(i3) + i4 = strlen(p1) + copystrcontent(p1, p4, 0, 0, i4) + i5 = strlen(p2) + i6 = int_add(i4, i5) # will be killed by the backend + copystrcontent(p2, p4, 0, i4, i5) + i0 = call(0, "hello world", p4, descr=streq_nonnull_descr) + escape(i0) + jump(p1, p2) + """ + self.optimize_loop_extradescrs(ops, 'Not, Not', expected) + + def test_str_equal_chars0(self): + ops = """ + [i1] + p1 = newstr(0) + i0 = call(0, p1, "", descr=strequaldescr) + escape(i0) + jump(i1) + """ + expected = """ + [i1] + escape(1) + jump(i1) + """ + self.optimize_loop_extradescrs(ops, 'Not', expected) + + def test_str_equal_chars1(self): + ops = """ + [i1] + p1 = newstr(1) + strsetitem(p1, 0, i1) + i0 = call(0, p1, "x", descr=strequaldescr) + escape(i0) + jump(i1) + """ + expected = """ + [i1] + i0 = int_eq(i1, 120) # ord('x') + escape(i0) + jump(i1) + """ + self.optimize_loop_extradescrs(ops, 'Not', expected) + + def test_str_equal_chars2(self): + ops = """ + [i1, i2] + p1 = newstr(2) + strsetitem(p1, 0, i1) + strsetitem(p1, 1, i2) + i0 = call(0, p1, "xy", descr=strequaldescr) + escape(i0) + jump(i1, i2) + """ + expected = """ + [i1, i2] + p1 = newstr(2) + strsetitem(p1, 0, i1) + strsetitem(p1, 1, i2) + i0 = call(0, p1, "xy", descr=streq_lengthok_descr) + escape(i0) + jump(i1, i2) + """ + self.optimize_loop_extradescrs(ops, 'Not, Not', expected) + + def test_str_equal_chars3(self): + ops = """ + [p1] + i0 = call(0, "x", p1, descr=strequaldescr) + escape(i0) + jump(p1) + """ + expected = """ + [p1] + i0 = call(0, p1, 120, descr=streq_checknull_char_descr) + escape(i0) + jump(p1) + """ + self.optimize_loop_extradescrs(ops, 'Not', expected) + + def test_str_equal_lengthmismatch1(self): + ops = """ + [i1] + p1 = newstr(1) + strsetitem(p1, 0, i1) + i0 = call(0, "xy", p1, descr=strequaldescr) + escape(i0) + jump(i1) + """ + expected = """ + [i1] + escape(0) + jump(i1) + """ + self.optimize_loop_extradescrs(ops, 'Not', expected) + + # XXX unicode operations + # XXX str2unicode + # ------------------------------------------------ from pypy.rpython.lltypesystem import llmemory @@ -3928,6 +4529,7 @@ def _identityhash(self): return id(self) + class TestFfiCall(BaseTestOptimizeOpt, LLtypeMixin): class namespace: Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_resume.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_resume.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_resume.py Fri Oct 8 13:40:10 2010 @@ -199,10 +199,10 @@ def test_prepare_virtuals(): class FakeVinfo(object): - def allocate(self, decoder): - return "allocated" - def setfields(self, decoder, virtual): - assert virtual == "allocated" + def allocate(self, decoder, index): + s = "allocated" + decoder.virtuals_cache[index] = s + return s class FakeStorage(object): rd_virtuals = [FakeVinfo(), None] rd_numb = [] @@ -212,7 +212,97 @@ _already_allocated_resume_virtuals = None cpu = None reader = ResumeDataDirectReader(None, FakeStorage()) - assert reader.virtuals == ["allocated", reader.virtual_default] + assert reader.force_all_virtuals() == ["allocated", reader.virtual_default] + +# ____________________________________________________________ + +class FakeResumeDataReader(AbstractResumeDataReader): + def allocate_with_vtable(self, known_class): + return FakeBuiltObject(vtable=known_class) + def allocate_struct(self, typedescr): + return FakeBuiltObject(typedescr=typedescr) + def allocate_array(self, arraydescr, length): + return FakeBuiltObject(arraydescr=arraydescr, items=[None]*length) + def setfield(self, descr, struct, fieldnum): + setattr(struct, descr, fieldnum) + def setarrayitem_int(self, arraydescr, array, i, fieldnum): + assert 0 <= i < len(array.items) + assert arraydescr is array.arraydescr + array.items[i] = fieldnum + def allocate_string(self, length): + return FakeBuiltObject(string=[None]*length) + def string_setitem(self, string, i, fieldnum): + value, tag = untag(fieldnum) + assert tag == TAGINT + assert 0 <= i < len(string.string) + string.string[i] = value + def concat_strings(self, left, right): + return FakeBuiltObject(strconcat=[left, right]) + def slice_string(self, str, start, length): + return FakeBuiltObject(strslice=[str, start, length]) + +class FakeBuiltObject(object): + def __init__(self, **kwds): + self.__dict__ = kwds + def __eq__(self, other): + return (self.__class__ == other.__class__ and + self.__dict__ == other.__dict__) + def __repr__(self): + return 'FakeBuiltObject(%s)' % ( + ', '.join(['%s=%r' % item for item in self.__dict__.items()])) + +class FakeArrayDescr(object): + def is_array_of_pointers(self): return False + def is_array_of_floats(self): return False + +def test_virtualinfo(): + info = VirtualInfo(123, ["fielddescr1"]) + info.fieldnums = [tag(456, TAGINT)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(vtable=123, fielddescr1=tag(456, TAGINT))] + +def test_vstructinfo(): + info = VStructInfo(124, ["fielddescr1"]) + info.fieldnums = [tag(456, TAGINT)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(typedescr=124, fielddescr1=tag(456, TAGINT))] + +def test_varrayinfo(): + arraydescr = FakeArrayDescr() + info = VArrayInfo(arraydescr) + info.fieldnums = [tag(456, TAGINT)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(arraydescr=arraydescr, items=[tag(456, TAGINT)])] + +def test_vstrplaininfo(): + info = VStrPlainInfo() + info.fieldnums = [tag(60, TAGINT)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(string=[60])] + +def test_vstrconcatinfo(): + info = VStrConcatInfo() + info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(strconcat=info.fieldnums)] + +def test_vstrsliceinfo(): + info = VStrSliceInfo() + info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX), tag(30, TAGBOX)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(strslice=info.fieldnums)] # ____________________________________________________________ @@ -957,7 +1047,7 @@ metainterp = MyMetaInterp() reader = ResumeDataFakeReader(storage, newboxes, metainterp) - assert len(reader.virtuals) == 2 + assert len(reader.virtuals_cache) == 2 b2t = reader.decode_ref(modifier._gettagged(b2s)) b4t = reader.decode_ref(modifier._gettagged(b4s)) trace = metainterp.trace @@ -972,13 +1062,14 @@ b4set = [(rop.SETFIELD_GC, [b4t, b2t], None, LLtypeMixin.nextdescr), (rop.SETFIELD_GC, [b4t, b3t], None, LLtypeMixin.valuedescr), (rop.SETFIELD_GC, [b4t, b5t], None, LLtypeMixin.otherdescr)] - if untag(modifier._gettagged(b2s))[0] == -2: - expected = [b2new, b4new] + b2set + b4set - else: - expected = [b4new, b2new] + b4set + b2set - - for x, y in zip(expected, trace): - assert x == y + expected = [b2new, b4new] + b4set + b2set + + # check that we get the operations in 'expected', in a possibly different + # order. + assert len(trace) == len(expected) + for x in trace: + assert x in expected + expected.remove(x) ptr = b2t.value._obj.container._as_ptr() assert lltype.typeOf(ptr) == lltype.Ptr(LLtypeMixin.NODE) assert ptr.value == 111 @@ -1020,7 +1111,7 @@ # resume metainterp = MyMetaInterp() reader = ResumeDataFakeReader(storage, newboxes, metainterp) - assert len(reader.virtuals) == 1 + assert len(reader.virtuals_cache) == 1 b2t = reader.decode_ref(tag(0, TAGVIRTUAL)) trace = metainterp.trace expected = [ @@ -1065,7 +1156,7 @@ NULL = ConstPtr.value metainterp = MyMetaInterp() reader = ResumeDataFakeReader(storage, newboxes, metainterp) - assert len(reader.virtuals) == 1 + assert len(reader.virtuals_cache) == 1 b2t = reader.decode_ref(tag(0, TAGVIRTUAL)) trace = metainterp.trace @@ -1112,7 +1203,7 @@ metainterp = MyMetaInterp() reader = ResumeDataFakeReader(storage, newboxes, metainterp) - assert reader.virtuals is None + assert reader.virtuals_cache is None trace = metainterp.trace b2set = (rop.SETFIELD_GC, [b2t, b4t], None, LLtypeMixin.nextdescr) expected = [b2set] Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_string.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_string.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_string.py Fri Oct 8 13:40:10 2010 @@ -1,5 +1,5 @@ import py -from pypy.rlib.jit import JitDriver +from pypy.rlib.jit import JitDriver, dont_look_inside, we_are_jitted from pypy.jit.codewriter.policy import StopAtXPolicy from pypy.rpython.ootypesystem import ootype from pypy.jit.metainterp.test.test_basic import LLJitMixin, OOJitMixin @@ -72,6 +72,234 @@ res = self.meta_interp(f, [6, 10]) assert res == 6 + def test_char2string_pure(self): + for dochr in [chr, ]: #unichr]: + jitdriver = JitDriver(greens = [], reds = ['n']) + @dont_look_inside + def escape(x): + pass + def f(n): + while n > 0: + jitdriver.can_enter_jit(n=n) + jitdriver.jit_merge_point(n=n) + s = dochr(n) + if not we_are_jitted(): + s += s # forces to be a string + if n > 100: + escape(s) + n -= 1 + return 42 + self.meta_interp(f, [6]) + self.check_loops(newstr=0, strsetitem=0, strlen=0, + newunicode=0, unicodesetitem=0, unicodelen=0) + + def test_char2string_escape(self): + for dochr in [chr, ]: #unichr]: + jitdriver = JitDriver(greens = [], reds = ['n', 'total']) + @dont_look_inside + def escape(x): + return ord(x[0]) + def f(n): + total = 0 + while n > 0: + jitdriver.can_enter_jit(n=n, total=total) + jitdriver.jit_merge_point(n=n, total=total) + s = dochr(n) + if not we_are_jitted(): + s += s # forces to be a string + total += escape(s) + n -= 1 + return total + res = self.meta_interp(f, [6]) + assert res == 21 + + def test_char2string2char(self): + for dochr in [chr, ]: #unichr]: + jitdriver = JitDriver(greens = [], reds = ['m', 'total']) + def f(m): + total = 0 + while m > 0: + jitdriver.can_enter_jit(m=m, total=total) + jitdriver.jit_merge_point(m=m, total=total) + string = dochr(m) + if m > 100: + string += string # forces to be a string + # read back the character + c = string[0] + total += ord(c) + m -= 1 + return total + res = self.meta_interp(f, [6]) + assert res == 21 + self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0, + newunicode=0, unicodegetitem=0, unicodesetitem=0, + unicodelen=0) + + def test_strconcat_pure(self): + for somestr in ["abc", ]: #u"def"]: + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = [somestr+str(i) for i in range(10)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + mylist[m] + if m > 100: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=0, strsetitem=0, + newunicode=0, unicodesetitem=0, + call=0, call_pure=0) + + def test_strconcat_escape_str_str(self): + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = ["somestr"+str(i) for i in range(10)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + mylist[m] + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=1, strsetitem=0, copystrcontent=2, + call=1, call_pure=0) # escape + + def test_strconcat_escape_str_char(self): + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = ["somestr"+str(i) for i in range(10)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + chr(m) + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, + call=1, call_pure=0) # escape + + def test_strconcat_escape_char_str(self): + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = ["somestr"+str(i) for i in range(10)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = chr(n) + mylist[m] + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, + call=1, call_pure=0) # escape + + def test_strconcat_escape_char_char(self): + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = chr(n) + chr(m) + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=1, strsetitem=2, copystrcontent=0, + call=1, call_pure=0) # escape + + def test_strconcat_escape_str_char_str(self): + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = ["somestr"+str(i) for i in range(10)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + chr(n) + mylist[m] + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=1, strsetitem=1, copystrcontent=2, + call=1, call_pure=0) # escape + + def test_strconcat_guard_fail(self): + for somestr in ["abc", ]: #u"def"]: + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = [somestr+str(i) for i in range(12)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + mylist[m] + if m & 1: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 10]) + + def test_strslice(self): + for somestr in ["abc", ]: #u"def"]: + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + def f(n, m): + assert n >= 0 + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = "foobarbazetc"[m:n] + if m <= 5: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [10, 10]) + + def test_streq_char(self): + for somestr in ["?abcdefg", ]: #u"def"]: + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + def f(n, m): + assert n >= 0 + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = somestr[:m] + escape(s == "?") + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=0, newunicode=0) + + class TestOOtype(StringTests, OOJitMixin): CALL = "oosend" CALL_PURE = "oosend_pure" Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_ztranslation.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_ztranslation.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_ztranslation.py Fri Oct 8 13:40:10 2010 @@ -21,6 +21,7 @@ # - full optimizer # - jitdriver hooks # - two JITs + # - string concatenation, slicing and comparison class Frame(object): _virtualizable2_ = ['i'] @@ -60,11 +61,15 @@ frame.i -= 1 return total * 10 # - myjitdriver2 = JitDriver(greens = ['g'], reds = ['m', 'x']) + myjitdriver2 = JitDriver(greens = ['g'], reds = ['m', 'x', 's']) def f2(g, m, x): + s = "" while m > 0: - myjitdriver2.can_enter_jit(g=g, m=m, x=x) - myjitdriver2.jit_merge_point(g=g, m=m, x=x) + myjitdriver2.can_enter_jit(g=g, m=m, x=x, s=s) + myjitdriver2.jit_merge_point(g=g, m=m, x=x, s=s) + s += 'xy' + if s[:2] == 'yz': + return -666 m -= 1 x += 3 return x Modified: pypy/branch/jitffi/pypy/jit/metainterp/warmspot.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/warmspot.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/warmspot.py Fri Oct 8 13:40:10 2010 @@ -67,9 +67,16 @@ def jittify_and_run(interp, graph, args, repeat=1, backendopt=False, trace_limit=sys.maxint, debug_level=DEBUG_STEPS, inline=False, **kwds): + from pypy.config.config import ConfigError translator = interp.typer.annotator.translator - translator.config.translation.gc = "boehm" - translator.config.translation.list_comprehension_operations = True + try: + translator.config.translation.gc = "boehm" + except ConfigError: + pass + try: + translator.config.translation.list_comprehension_operations = True + except ConfigError: + pass warmrunnerdesc = WarmRunnerDesc(translator, backendopt=backendopt, **kwds) for jd in warmrunnerdesc.jitdrivers_sd: jd.warmstate.set_param_threshold(3) # for tests Modified: pypy/branch/jitffi/pypy/jit/metainterp/warmstate.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/warmstate.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/warmstate.py Fri Oct 8 13:40:10 2010 @@ -7,7 +7,8 @@ from pypy.rlib.rarithmetic import intmask from pypy.rlib.nonconst import NonConstant from pypy.rlib.unroll import unrolling_iterable -from pypy.rlib.jit import PARAMETERS, OPTIMIZER_SIMPLE, OPTIMIZER_FULL +from pypy.rlib.jit import (PARAMETERS, OPTIMIZER_SIMPLE, OPTIMIZER_FULL, + OPTIMIZER_NO_PERFECTSPEC) from pypy.rlib.jit import DEBUG_PROFILE from pypy.rlib.jit import BaseJitCell from pypy.rlib.debug import debug_start, debug_stop, debug_print @@ -87,6 +88,9 @@ return history.ConstFloat(value) else: return history.BoxFloat(value) + elif isinstance(value, str) or isinstance(value, unicode): + assert len(value) == 1 # must be a character + value = ord(value) else: value = intmask(value) if in_const_box: @@ -191,6 +195,10 @@ from pypy.jit.metainterp import simple_optimize self.optimize_loop = simple_optimize.optimize_loop self.optimize_bridge = simple_optimize.optimize_bridge + elif optimizer == OPTIMIZER_NO_PERFECTSPEC: + from pypy.jit.metainterp import optimize_nopspec + self.optimize_loop = optimize_nopspec.optimize_loop + self.optimize_bridge = optimize_nopspec.optimize_bridge elif optimizer == OPTIMIZER_FULL: from pypy.jit.metainterp import optimize self.optimize_loop = optimize.optimize_loop Modified: pypy/branch/jitffi/pypy/jit/tl/spli/interpreter.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/tl/spli/interpreter.py (original) +++ pypy/branch/jitffi/pypy/jit/tl/spli/interpreter.py Fri Oct 8 13:40:10 2010 @@ -1,12 +1,14 @@ import os -from pypy.tool import stdlib_opcode as opcode +from pypy.tool import stdlib_opcode from pypy.jit.tl.spli import objects, pycode -from pypy.tool.stdlib_opcode import unrolling_opcode_descs -from pypy.tool.stdlib_opcode import opcode_method_names from pypy.rlib.unroll import unrolling_iterable from pypy.rlib.jit import JitDriver, hint, dont_look_inside from pypy.rlib.objectmodel import we_are_translated +opcode_method_names = stdlib_opcode.host_bytecode_spec.method_names +unrolling_opcode_descs = unrolling_iterable( + stdlib_opcode.host_bytecode_spec.ordered_opdescs) +HAVE_ARGUMENT = stdlib_opcode.host_HAVE_ARGUMENT compare_ops = [ "cmp_lt", # "<" @@ -79,7 +81,7 @@ self.stack_depth = hint(self.stack_depth, promote=True) op = ord(code[instr_index]) instr_index += 1 - if op >= opcode.HAVE_ARGUMENT: + if op >= HAVE_ARGUMENT: low = ord(code[instr_index]) hi = ord(code[instr_index + 1]) oparg = (hi << 8) | low @@ -183,6 +185,12 @@ next_instr += arg return next_instr + def POP_JUMP_IF_FALSE(self, arg, next_instr, code): + w_cond = self.pop() + if not w_cond.is_true(): + next_instr = arg + return next_instr + def JUMP_FORWARD(self, arg, next_instr, code): return next_instr + arg Modified: pypy/branch/jitffi/pypy/jit/tool/loopviewer.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/tool/loopviewer.py (original) +++ pypy/branch/jitffi/pypy/jit/tool/loopviewer.py Fri Oct 8 13:40:10 2010 @@ -1,30 +1,93 @@ #!/usr/bin/env python -""" Usage: loopviewer.py [loopnum] loopfile +""" +Parse and display the traces produced by pypy-c-jit when PYPYLOG is set. """ import autopath import py import sys +import optparse +from pprint import pprint from pypy.tool import logparser from pypy.jit.metainterp.test.oparser import parse from pypy.jit.metainterp.history import ConstInt from pypy.rpython.lltypesystem import llmemory, lltype -def main(loopnum, loopfile): +def main(loopfile, options): + print 'Loading file:' log = logparser.parse_log_file(loopfile) - loops = logparser.extract_category(log, "jit-log-opt-") - inp = loops[loopnum] - loop = parse(inp, no_namespace=True) - loop.show() + loops, summary = consider_category(log, options, "jit-log-opt-") + if not options.quiet: + for loop in loops: + loop.show() + + if options.summary: + print + print 'Summary:' + print_summary(summary) -if __name__ == '__main__': - if len(sys.argv) == 2: - loopnum = -1 - loopfile = sys.argv[1] - elif len(sys.argv) == 3: - loopnum = int(sys.argv[1]) - loopfile = sys.argv[2] + if options.diff: + # non-optimized loops and summary + nloops, nsummary = consider_category(log, options, "jit-log-noopt-") + print + print 'Summary of optimized-away operations' + print + diff = {} + keys = set(summary.keys()).union(set(nsummary)) + for key in keys: + before = nsummary.get(key, 0) + after = summary.get(key, 0) + diff[key] = (before-after, before, after) + print_diff(diff) + +def consider_category(log, options, category): + loops = logparser.extract_category(log, category) + if options.loopnum is None: + input_loops = loops else: - print __doc__ - sys.exit(1) - main(loopnum, loopfile) + input_loops = [loops[options.loopnum]] + loops = [parse(inp, no_namespace=True, nonstrict=True) + for inp in input_loops] + summary = {} + for loop in loops: + summary = loop.summary(summary) + return loops, summary + + +def print_summary(summary): + ops = [(summary[key], key) for key in summary] + ops.sort(reverse=True) + for n, key in ops: + print '%5d' % n, key + +def print_diff(diff): + ops = [(d, before, after, key) for key, (d, before, after) in diff.iteritems()] + ops.sort(reverse=True) + tot_before = 0 + tot_after = 0 + for d, before, after, key in ops: + tot_before += before + tot_after += after + print '%5d - %5d = %5d ' % (before, after, d), key + print '-' * 50 + print '%5d - %5d = %5d ' % (tot_before, tot_after, tot_before-tot_after), 'TOTAL' + +if __name__ == '__main__': + parser = optparse.OptionParser(usage="%prog loopfile [options]") + parser.add_option('-n', '--loopnum', dest='loopnum', default=-1, metavar='N', type=int, + help='show the loop number N [default: last]') + parser.add_option('-a', '--all', dest='loopnum', action='store_const', const=None, + help='show all loops in the file') + parser.add_option('-s', '--summary', dest='summary', action='store_true', default=False, + help='print a summary of the operations in the loop(s)') + parser.add_option('-d', '--diff', dest='diff', action='store_true', default=False, + help='print the difference between non-optimized and optimized operations in the loop(s)') + parser.add_option('-q', '--quiet', dest='quiet', action='store_true', default=False, + help='do not show the graphical representation of the loop') + + options, args = parser.parse_args() + if len(args) != 1: + parser.print_help() + sys.exit(2) + + main(args[0], options) Modified: pypy/branch/jitffi/pypy/module/array/benchmark/sumtst.c ============================================================================== --- pypy/branch/jitffi/pypy/module/array/benchmark/sumtst.c (original) +++ pypy/branch/jitffi/pypy/module/array/benchmark/sumtst.c Fri Oct 8 13:40:10 2010 @@ -1,3 +1,4 @@ +#include double sum(double *img); @@ -5,4 +6,4 @@ double *img=malloc(640*480*4*sizeof(double)); int sa=0; for (int l=0; l<500; l++) sum(img); -} +} Modified: pypy/branch/jitffi/pypy/module/imp/importing.py ============================================================================== --- pypy/branch/jitffi/pypy/module/imp/importing.py (original) +++ pypy/branch/jitffi/pypy/module/imp/importing.py Fri Oct 8 13:40:10 2010 @@ -122,7 +122,7 @@ n = len(ctxt_name_prefix_parts)-level+1 assert n>=0 ctxt_name_prefix_parts = ctxt_name_prefix_parts[:n] - if ctxt_w_path is None: # plain module + if ctxt_name_prefix_parts and ctxt_w_path is None: # plain module ctxt_name_prefix_parts.pop() if ctxt_name_prefix_parts: rel_modulename = '.'.join(ctxt_name_prefix_parts) Modified: pypy/branch/jitffi/pypy/module/imp/test/test_import.py ============================================================================== --- pypy/branch/jitffi/pypy/module/imp/test/test_import.py (original) +++ pypy/branch/jitffi/pypy/module/imp/test/test_import.py Fri Oct 8 13:40:10 2010 @@ -360,6 +360,12 @@ """.rstrip() raises(ValueError, imp) + def test_future_relative_import_error_when_in_non_package2(self): + exec """def imp(): + from .. import inpackage + """.rstrip() + raises(ValueError, imp) + def test_relative_import_with___name__(self): import sys mydict = {'__name__': 'sys.foo'} Modified: pypy/branch/jitffi/pypy/module/parser/test/test_parser.py ============================================================================== --- pypy/branch/jitffi/pypy/module/parser/test/test_parser.py (original) +++ pypy/branch/jitffi/pypy/module/parser/test/test_parser.py Fri Oct 8 13:40:10 2010 @@ -10,6 +10,9 @@ cls.w_m = space.appexec([], """(): import parser return parser""") + cls.w_symbol = space.appexec([], """(): + import symbol + return symbol""") class AppTestParser(ParserModuleTest): @@ -36,7 +39,7 @@ seq = getattr(s, meth)() assert isinstance(seq, tp) assert len(seq) == 4 - assert seq[0] == 286 + assert seq[0] == self.symbol.file_input assert len(seq[2]) == 2 assert len(seq[3]) == 2 assert seq[2][0] == 4 Modified: pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py ============================================================================== --- pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py (original) +++ pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py Fri Oct 8 13:40:10 2010 @@ -140,7 +140,7 @@ for op in loop.operations: if op.getopname() == "debug_merge_point": sliced_loop = BytecodeTrace() - sliced_loop.bytecode = op.args[0]._get_str().rsplit(" ", 1)[1] + sliced_loop.bytecode = op.getarg(0)._get_str().rsplit(" ", 1)[1] self.sliced_loops.append(sliced_loop) else: sliced_loop.append(op) @@ -798,7 +798,6 @@ if i > 750: a = b return sa ''', 215, ([], 12481752)) - assert False def test_array_sum(self): for tc, maxops in zip('bhilBHILfd', (38,) * 6 + (40, 40, 41, 38)): @@ -869,6 +868,24 @@ return intimg[i - 1] ''', maxops, ([tc], res)) + def test_unpackiterable(self): + self.run_source(''' + from array import array + + def main(): + i = 0 + t = array('l', (1, 2)) + while i < 2000: + a, b = t + i += 1 + return 3 + + ''', 100, ([], 3)) + bytecode, = self.get_by_bytecode("UNPACK_SEQUENCE") + # we allocate virtual ref and frame, we don't want block + assert len(bytecode.get_opnames('call_may_force')) == 0 + + def test_intbound_simple(self): ops = ('<', '>', '<=', '>=', '==', '!=') nbr = (3, 7) Modified: pypy/branch/jitffi/pypy/module/sys/__init__.py ============================================================================== --- pypy/branch/jitffi/pypy/module/sys/__init__.py (original) +++ pypy/branch/jitffi/pypy/module/sys/__init__.py Fri Oct 8 13:40:10 2010 @@ -7,13 +7,15 @@ """Sys Builtin Module. """ def __init__(self, space, w_name): """NOT_RPYTHON""" # because parent __init__ isn't + if space.config.translating: + del self.__class__.interpleveldefs['pypy_getudir'] super(Module, self).__init__(space, w_name) self.checkinterval = 100 self.recursionlimit = 100 self.w_default_encoder = None self.defaultencoding = "ascii" self.filesystemencoding = None - + interpleveldefs = { '__name__' : '(space.wrap("sys"))', '__doc__' : '(space.wrap("PyPy sys module"))', @@ -37,7 +39,7 @@ 'argv' : 'state.get(space).w_argv', 'warnoptions' : 'state.get(space).w_warnoptions', 'builtin_module_names' : 'state.w_None', - 'pypy_getudir' : 'state.pypy_getudir', + 'pypy_getudir' : 'state.pypy_getudir', # not translated 'pypy_initial_path' : 'state.pypy_initial_path', '_getframe' : 'vm._getframe', Modified: pypy/branch/jitffi/pypy/module/sys/state.py ============================================================================== --- pypy/branch/jitffi/pypy/module/sys/state.py (original) +++ pypy/branch/jitffi/pypy/module/sys/state.py Fri Oct 8 13:40:10 2010 @@ -95,15 +95,8 @@ def getio(space): return space.fromcache(IOState) -def _pypy_getudir(space): - """NOT_RPYTHON""" +def pypy_getudir(space): + """NOT_RPYTHON + (should be removed from interpleveldefs before translation)""" from pypy.tool.udir import udir return space.wrap(str(udir)) -_pypy_getudir._annspecialcase_ = "override:ignore" - -# we need the indirection because this function will live in a dictionary with other -# RPYTHON functions and share call sites with them. Better it not be a special-case -# directly. -def pypy_getudir(space): - return _pypy_getudir(space) - Modified: pypy/branch/jitffi/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py ============================================================================== --- pypy/branch/jitffi/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py (original) +++ pypy/branch/jitffi/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py Fri Oct 8 13:40:10 2010 @@ -29,6 +29,7 @@ class RECT(Structure): _fields_ = [("left", c_int), ("top", c_int), ("right", c_int), ("bottom", c_int)] + class TestFunctions(BaseCTypesTestChecker): def test_mro(self): @@ -392,6 +393,18 @@ result = f("abcd", ord("b")) assert result == "bcd" + def test_caching_bug_1(self): + # the same test as test_call_some_args, with two extra lines + # in the middle that trigger caching in f._ptr, which then + # makes the last two lines fail + f = dll.my_strchr + f.argtypes = [c_char_p, c_int] + f.restype = c_char_p + result = f("abcd", ord("b")) + assert result == "bcd" + result = f("abcd", ord("b"), 42) + assert result == "bcd" + def test_sf1651235(self): py.test.skip("we are less strict in checking callback parameters") # see http://www.python.org/sf/1651235 Modified: pypy/branch/jitffi/pypy/objspace/std/fake.py ============================================================================== --- pypy/branch/jitffi/pypy/objspace/std/fake.py (original) +++ pypy/branch/jitffi/pypy/objspace/std/fake.py Fri Oct 8 13:40:10 2010 @@ -21,7 +21,6 @@ #debug_print("faking obj %s" % x) ft = fake_type(type(x)) return ft(space, x) -fake_object._annspecialcase_ = "override:fake_object" import sys @@ -47,7 +46,6 @@ w_exc = space.wrap(exc) w_value = space.wrap(value) raise OperationError, OperationError(w_exc, w_value), tb -wrap_exception._annspecialcase_ = "override:ignore" def fake_type(cpy_type): assert type(cpy_type) is type Modified: pypy/branch/jitffi/pypy/objspace/std/objspace.py ============================================================================== --- pypy/branch/jitffi/pypy/objspace/std/objspace.py (original) +++ pypy/branch/jitffi/pypy/objspace/std/objspace.py Fri Oct 8 13:40:10 2010 @@ -7,7 +7,7 @@ from pypy.objspace.std import (builtinshortcut, stdtypedef, frame, model, transparent, callmethod, proxyobject) from pypy.objspace.descroperation import DescrOperation, raiseattrerror -from pypy.rlib.objectmodel import instantiate, r_dict +from pypy.rlib.objectmodel import instantiate, r_dict, specialize from pypy.rlib.debug import make_sure_not_resized from pypy.rlib.rarithmetic import base_int from pypy.rlib.objectmodel import we_are_translated @@ -242,7 +242,6 @@ w_result = getattr(self, 'w_' + x.__name__) return w_result return None - wrap_exception_cls._annspecialcase_ = "override:wrap_exception_cls" def unwrap(self, w_obj): if isinstance(w_obj, Wrappable): @@ -350,7 +349,8 @@ raise self._wrap_expected_length(expected_length, len(t)) return t - def fixedview(self, w_obj, expected_length=-1): + @specialize.arg(3) + def fixedview(self, w_obj, expected_length=-1, unroll=False): """ Fast paths """ if isinstance(w_obj, W_TupleObject): @@ -358,18 +358,26 @@ elif isinstance(w_obj, W_ListObject): t = w_obj.wrappeditems[:] else: - return ObjSpace.fixedview(self, w_obj, expected_length) + if unroll: + return make_sure_not_resized(ObjSpace.unpackiterable_unroll( + self, w_obj, expected_length)[:]) + else: + return make_sure_not_resized(ObjSpace.unpackiterable( + self, w_obj, expected_length)[:]) if expected_length != -1 and len(t) != expected_length: raise self._wrap_expected_length(expected_length, len(t)) return t + def fixedview_unroll(self, w_obj, expected_length=-1): + return self.fixedview(w_obj, expected_length, unroll=True) + def listview(self, w_obj, expected_length=-1): if isinstance(w_obj, W_ListObject): t = w_obj.wrappeditems elif isinstance(w_obj, W_TupleObject): t = w_obj.wrappeditems[:] else: - return ObjSpace.listview(self, w_obj, expected_length) + return ObjSpace.unpackiterable(self, w_obj, expected_length) if expected_length != -1 and len(t) != expected_length: raise self._wrap_expected_length(expected_length, len(t)) return t Modified: pypy/branch/jitffi/pypy/objspace/std/stringtype.py ============================================================================== --- pypy/branch/jitffi/pypy/objspace/std/stringtype.py (original) +++ pypy/branch/jitffi/pypy/objspace/std/stringtype.py Fri Oct 8 13:40:10 2010 @@ -4,6 +4,7 @@ from sys import maxint from pypy.rlib.objectmodel import specialize +from pypy.rlib.jit import we_are_jitted def wrapstr(space, s): from pypy.objspace.std.stringobject import W_StringObject @@ -32,7 +33,7 @@ def wrapchar(space, c): from pypy.objspace.std.stringobject import W_StringObject from pypy.objspace.std.ropeobject import rope, W_RopeObject - if space.config.objspace.std.withprebuiltchar: + if space.config.objspace.std.withprebuiltchar and not we_are_jitted(): if space.config.objspace.std.withrope: return W_RopeObject.PREBUILT[ord(c)] return W_StringObject.PREBUILT[ord(c)] Modified: pypy/branch/jitffi/pypy/rlib/jit.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/jit.py (original) +++ pypy/branch/jitffi/pypy/rlib/jit.py Fri Oct 8 13:40:10 2010 @@ -230,7 +230,8 @@ """Inconsistency in the JIT hints.""" OPTIMIZER_SIMPLE = 0 -OPTIMIZER_FULL = 1 +OPTIMIZER_NO_PERFECTSPEC = 1 +OPTIMIZER_FULL = 2 DEBUG_OFF = 0 DEBUG_PROFILE = 1 Modified: pypy/branch/jitffi/pypy/rlib/rmmap.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/rmmap.py (original) +++ pypy/branch/jitffi/pypy/rlib/rmmap.py Fri Oct 8 13:40:10 2010 @@ -292,7 +292,8 @@ c_munmap(self.getptr(0), self.size) self.setdata(NODATA, 0) - __del__ = close + def __del__(self): + self.close() def unmapview(self): UnmapViewOfFile(self.getptr(0)) Modified: pypy/branch/jitffi/pypy/rlib/rstring.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/rstring.py (original) +++ pypy/branch/jitffi/pypy/rlib/rstring.py Fri Oct 8 13:40:10 2010 @@ -1,9 +1,10 @@ """ String builder interface and string functions """ -from pypy.rpython.extregistry import ExtRegistryEntry from pypy.annotation.model import SomeObject, SomeString, s_None,\ SomeChar, SomeInteger, SomeUnicodeCodePoint, SomeUnicodeString +from pypy.rlib.rarithmetic import ovfcheck +from pypy.rpython.extregistry import ExtRegistryEntry # -------------- public API for string functions ----------------------- @@ -46,9 +47,7 @@ # -------------- public API --------------------------------- -# the following number is the maximum size of an RPython unicode -# string that goes into the nursery of the minimark GC. -INIT_SIZE = 56 +INIT_SIZE = 100 # XXX tweak class AbstractStringBuilder(object): def __init__(self, init_size=INIT_SIZE): @@ -80,6 +79,11 @@ result = None factor = 1 assert mul > 0 + try: + ovfcheck(len(s) * mul) + except OverflowError: + raise MemoryError + limit = mul >> 1 while True: if mul & factor: Modified: pypy/branch/jitffi/pypy/rlib/test/test_rstring.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/test/test_rstring.py (original) +++ pypy/branch/jitffi/pypy/rlib/test/test_rstring.py Fri Oct 8 13:40:10 2010 @@ -1,5 +1,8 @@ +import sys + +from pypy.rlib.rstring import StringBuilder, UnicodeBuilder, split, rsplit, \ + string_repeat -from pypy.rlib.rstring import StringBuilder, UnicodeBuilder, split, rsplit def test_split(): assert split("", 'x') == [''] @@ -39,3 +42,6 @@ s.append_multiple_char('d', 4) assert s.build() == 'aabcbdddd' assert isinstance(s.build(), unicode) + +def test_string_repeat(): + raises(MemoryError, string_repeat, "abc", sys.maxint) Modified: pypy/branch/jitffi/pypy/rpython/annlowlevel.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/annlowlevel.py (original) +++ pypy/branch/jitffi/pypy/rpython/annlowlevel.py Fri Oct 8 13:40:10 2010 @@ -397,6 +397,8 @@ assert strtype in (str, unicode) def hlstr(ll_s): + if not ll_s: + return None if hasattr(ll_s, 'chars'): if strtype is str: return ''.join(ll_s.chars) @@ -423,9 +425,14 @@ def llstr(s): from pypy.rpython.lltypesystem.rstr import mallocstr, mallocunicode + from pypy.rpython.lltypesystem.rstr import STR, UNICODE if strtype is str: + if s is None: + return lltype.nullptr(STR) ll_s = mallocstr(len(s)) else: + if s is None: + return lltype.nullptr(UNICODE) ll_s = mallocunicode(len(s)) for i, c in enumerate(s): ll_s.chars[i] = c Modified: pypy/branch/jitffi/pypy/rpython/lltypesystem/ll2ctypes.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/lltypesystem/ll2ctypes.py (original) +++ pypy/branch/jitffi/pypy/rpython/lltypesystem/ll2ctypes.py Fri Oct 8 13:40:10 2010 @@ -26,6 +26,60 @@ from pypy.translator.platform import platform from array import array +# ____________________________________________________________ + +far_regions = None + +def allocate_ctypes(ctype): + if far_regions: + import random + pieces = far_regions._ll2ctypes_pieces + num = random.randrange(len(pieces)) + i1, stop = pieces[num] + i2 = i1 + ((ctypes.sizeof(ctype) or 1) + 7) & ~7 + if i2 > stop: + raise MemoryError("out of memory in far_regions") + pieces[num] = i2, stop + p = lltype2ctypes(far_regions.getptr(i1)) + return ctypes.cast(p, ctypes.POINTER(ctype)).contents + else: + return ctype() + +def do_allocation_in_far_regions(): + """On 32 bits: this reserves 1.25GB of address space, or 2.5GB on Linux, + which helps test this module for address values that are signed or + unsigned. + + On 64-bits: reserves 10 times 2GB of address space. This should help + to find 32-vs-64-bit issues in the JIT. It is likely that objects + are further apart than 32 bits can represent; it is also possible + to hit the corner case of being precisely e.g. 2GB - 8 bytes apart. + + Avoid this function if your OS reserves actual RAM from mmap() eagerly. + """ + global far_regions + if not far_regions: + from pypy.rlib import rmmap + if sys.maxint > 0x7FFFFFFF: + PIECESIZE = 0x80000000 + else: + if sys.platform == 'linux': + PIECESIZE = 0x10000000 + else: + PIECESIZE = 0x08000000 + PIECES = 10 + m = rmmap.mmap(-1, PIECES * PIECESIZE, + rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS, + rmmap.PROT_READ|rmmap.PROT_WRITE) + m.close = lambda : None # leak instead of giving a spurious + # error at CPython's shutdown + m._ll2ctypes_pieces = [] + for i in range(PIECES): + m._ll2ctypes_pieces.append((i * PIECESIZE, (i+1) * PIECESIZE)) + far_regions = m + +# ____________________________________________________________ + _ctypes_cache = {} _eci_cache = {} @@ -87,13 +141,13 @@ if S._arrayfld is None: if n is not None: raise TypeError("%r is not variable-sized" % (S,)) - storage = cls() + storage = allocate_ctypes(cls) return storage else: if n is None: raise TypeError("%r is variable-sized" % (S,)) biggercls = build_ctypes_struct(S, None, n) - bigstruct = biggercls() + bigstruct = allocate_ctypes(biggercls) array = getattr(bigstruct, S._arrayfld) if hasattr(array, 'length'): array.length = n @@ -135,7 +189,7 @@ if not isinstance(n, int): raise TypeError, "array length must be an int" biggercls = get_ctypes_array_of_size(A, n) - bigarray = biggercls() + bigarray = allocate_ctypes(biggercls) if hasattr(bigarray, 'length'): bigarray.length = n return bigarray @@ -375,7 +429,7 @@ "Returns the storage address as an int" if self._storage is None or self._storage is True: raise ValueError("Not a ctypes allocated structure") - return ctypes.cast(self._storage, ctypes.c_void_p).value + return intmask(ctypes.cast(self._storage, ctypes.c_void_p).value) def _free(self): self._check() # no double-frees Modified: pypy/branch/jitffi/pypy/rpython/lltypesystem/llarena.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/lltypesystem/llarena.py (original) +++ pypy/branch/jitffi/pypy/rpython/lltypesystem/llarena.py Fri Oct 8 13:40:10 2010 @@ -124,6 +124,9 @@ assert self.usagemap[i] == 'x' self.usagemap[i] = '#' + def mark_freed(self): + self.freed = True # this method is a hook for tests + class fakearenaaddress(llmemory.fakeaddress): def __init__(self, arena, offset): @@ -314,7 +317,7 @@ assert arena_addr.offset == 0 arena_addr.arena.reset(False) assert not arena_addr.arena.objectptrs - arena_addr.arena.freed = True + arena_addr.arena.mark_freed() def arena_reset(arena_addr, size, zero): """Free all objects in the arena, which can then be reused. @@ -472,22 +475,25 @@ clear_large_memory_chunk = llmemory.raw_memclear +llimpl_malloc = rffi.llexternal('malloc', [lltype.Signed], llmemory.Address, + sandboxsafe=True, _nowrapper=True) +llimpl_free = rffi.llexternal('free', [llmemory.Address], lltype.Void, + sandboxsafe=True, _nowrapper=True) + def llimpl_arena_malloc(nbytes, zero): - addr = llmemory.raw_malloc(nbytes) - if zero and bool(addr): - clear_large_memory_chunk(addr, nbytes) + addr = llimpl_malloc(nbytes) + if bool(addr): + llimpl_arena_reset(addr, nbytes, zero) return addr -register_external(arena_malloc, [int, bool], llmemory.Address, +llimpl_arena_malloc._always_inline_ = True +register_external(arena_malloc, [int, int], llmemory.Address, 'll_arena.arena_malloc', llimpl=llimpl_arena_malloc, llfakeimpl=arena_malloc, sandboxsafe=True) -def llimpl_arena_free(arena_addr): - # NB. minimark.py assumes that arena_free() is actually just a raw_free(). - llmemory.raw_free(arena_addr) register_external(arena_free, [llmemory.Address], None, 'll_arena.arena_free', - llimpl=llimpl_arena_free, + llimpl=llimpl_free, llfakeimpl=arena_free, sandboxsafe=True) @@ -497,6 +503,7 @@ clear_large_memory_chunk(arena_addr, size) else: llmemory.raw_memclear(arena_addr, size) +llimpl_arena_reset._always_inline_ = True register_external(arena_reset, [llmemory.Address, int, int], None, 'll_arena.arena_reset', llimpl=llimpl_arena_reset, Modified: pypy/branch/jitffi/pypy/rpython/lltypesystem/rlist.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/lltypesystem/rlist.py (original) +++ pypy/branch/jitffi/pypy/rpython/lltypesystem/rlist.py Fri Oct 8 13:40:10 2010 @@ -159,7 +159,6 @@ if 'item_repr' not in self.__dict__: self.external_item_repr, self.item_repr = externalvsinternal(self.rtyper, self._item_repr_computer()) if isinstance(self.LIST, GcForwardReference): - ITEM = self.item_repr.lowleveltype ITEMARRAY = self.get_itemarray_lowleveltype() self.LIST.become(ITEMARRAY) Modified: pypy/branch/jitffi/pypy/rpython/lltypesystem/rstr.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/lltypesystem/rstr.py (original) +++ pypy/branch/jitffi/pypy/rpython/lltypesystem/rstr.py Fri Oct 8 13:40:10 2010 @@ -4,7 +4,7 @@ from pypy.rlib.objectmodel import malloc_zero_filled, we_are_translated from pypy.rlib.objectmodel import _hash_string, enforceargs from pypy.rlib.debug import ll_assert -from pypy.rlib.jit import purefunction +from pypy.rlib.jit import purefunction, we_are_jitted from pypy.rpython.robject import PyObjRepr, pyobj_repr from pypy.rpython.rmodel import inputconst, IntegerRepr from pypy.rpython.rstr import AbstractStringRepr,AbstractCharRepr,\ @@ -65,8 +65,8 @@ dst = llmemory.cast_ptr_to_adr(dst) + _str_ofs(dststart) llmemory.raw_memcopy(src, dst, llmemory.sizeof(CHAR_TP) * length) copy_string_contents._always_inline_ = True - copy_string_contents.oopspec = ( - '%s.copy_contents(src, dst, srcstart, dststart, length)' % name) + #copy_string_contents.oopspec = ( + # '%s.copy_contents(src, dst, srcstart, dststart, length)' % name) return func_with_new_name(copy_string_contents, 'copy_%s_contents' % name) copy_string_contents = _new_copy_contents_fun(STR, Char, 'string') @@ -326,6 +326,7 @@ s1.copy_contents(s1, newstr, 0, 0, len1) s1.copy_contents(s2, newstr, 0, len1, len2) return newstr + ll_strconcat.oopspec = 'stroruni.concat(s1, s2)' @purefunction def ll_strip(s, ch, left, right): @@ -443,8 +444,8 @@ if chars1[j] != chars2[j]: return False j += 1 - return True + ll_streq.oopspec = 'stroruni.equal(s1, s2)' @purefunction def ll_startswith(s1, s2): @@ -694,35 +695,33 @@ return result @purefunction - def ll_stringslice_startonly(s1, start): - len1 = len(s1.chars) - newstr = s1.malloc(len1 - start) - lgt = len1 - start - assert lgt >= 0 + def _ll_stringslice(s1, start, stop): + lgt = stop - start assert start >= 0 + assert lgt >= 0 + newstr = s1.malloc(lgt) s1.copy_contents(s1, newstr, start, 0, lgt) return newstr + _ll_stringslice.oopspec = 'stroruni.slice(s1, start, stop)' + _ll_stringslice._annenforceargs_ = [None, int, int] + + def ll_stringslice_startonly(s1, start): + return LLHelpers._ll_stringslice(s1, start, len(s1.chars)) - @purefunction def ll_stringslice_startstop(s1, start, stop): - if stop >= len(s1.chars): - if start == 0: - return s1 - stop = len(s1.chars) - newstr = s1.malloc(stop - start) - assert start >= 0 - lgt = stop - start - assert lgt >= 0 - s1.copy_contents(s1, newstr, start, 0, lgt) - return newstr + if we_are_jitted(): + if stop > len(s1.chars): + stop = len(s1.chars) + else: + if stop >= len(s1.chars): + if start == 0: + return s1 + stop = len(s1.chars) + return LLHelpers._ll_stringslice(s1, start, stop) - @purefunction def ll_stringslice_minusone(s1): newlen = len(s1.chars) - 1 - newstr = s1.malloc(newlen) - assert newlen >= 0 - s1.copy_contents(s1, newstr, 0, 0, newlen) - return newstr + return LLHelpers._ll_stringslice(s1, 0, newlen) def ll_split_chr(LIST, s, c): chars = s.chars Modified: pypy/branch/jitffi/pypy/rpython/lltypesystem/test/test_ll2ctypes.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/lltypesystem/test/test_ll2ctypes.py (original) +++ pypy/branch/jitffi/pypy/rpython/lltypesystem/test/test_ll2ctypes.py Fri Oct 8 13:40:10 2010 @@ -16,6 +16,12 @@ from pypy.annotation.annrpython import RPythonAnnotator from pypy.rpython.rtyper import RPythonTyper + +if False: # for now, please keep it False by default + from pypy.rpython.lltypesystem import ll2ctypes + ll2ctypes.do_allocation_in_far_regions() + + class TestLL2Ctypes(object): def setup_method(self, meth): Modified: pypy/branch/jitffi/pypy/rpython/memory/gc/base.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gc/base.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/base.py Fri Oct 8 13:40:10 2010 @@ -20,12 +20,15 @@ prebuilt_gc_objects_are_static_roots = True object_minimal_size = 0 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE): + def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + translated_to_c=True): self.gcheaderbuilder = GCHeaderBuilder(self.HDR) self.AddressStack = get_address_stack(chunk_size) self.AddressDeque = get_address_deque(chunk_size) self.AddressDict = AddressDict self.config = config + assert isinstance(translated_to_c, bool) + self.translated_to_c = translated_to_c def setup(self): # all runtime mutable values' setup should happen here @@ -39,6 +42,9 @@ def can_malloc_nonmovable(self): return not self.moving_gc + def can_optimize_clean_setarrayitems(self): + return True # False in case of card marking + # The following flag enables costly consistency checks after each # collection. It is automatically set to True by test_gc.py. The # checking logic is translatable, so the flag can be set to True Modified: pypy/branch/jitffi/pypy/rpython/memory/gc/generation.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gc/generation.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/generation.py Fri Oct 8 13:40:10 2010 @@ -5,7 +5,6 @@ from pypy.rpython.memory.gc.base import read_from_env from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage from pypy.rpython.lltypesystem import lltype, llmemory, llarena -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rlib.objectmodel import free_non_gc_object from pypy.rlib.debug import ll_assert from pypy.rlib.debug import debug_print, debug_start, debug_stop @@ -49,15 +48,17 @@ nursery_hash_base = -1 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + def __init__(self, config, nursery_size=32*WORD, min_nursery_size=32*WORD, auto_nursery_size=False, space_size=1024*WORD, - max_space_size=sys.maxint//2+1): - SemiSpaceGC.__init__(self, config, chunk_size = chunk_size, + max_space_size=sys.maxint//2+1, + **kwds): + SemiSpaceGC.__init__(self, config, space_size = space_size, - max_space_size = max_space_size) + max_space_size = max_space_size, + **kwds) assert min_nursery_size <= nursery_size <= space_size // 2 self.initial_nursery_size = nursery_size self.auto_nursery_size = auto_nursery_size @@ -147,11 +148,24 @@ def get_young_var_basesize(nursery_size): return nursery_size // 4 - 1 + @classmethod + def JIT_max_size_of_young_obj(cls): + min_nurs_size = cls.TRANSLATION_PARAMS['min_nursery_size'] + return cls.get_young_fixedsize(min_nurs_size) + def is_in_nursery(self, addr): ll_assert(llmemory.cast_adr_to_int(addr) & 1 == 0, "odd-valued (i.e. tagged) pointer unexpected here") return self.nursery <= addr < self.nursery_top + def appears_to_be_in_nursery(self, addr): + # same as is_in_nursery(), but may return True accidentally if + # 'addr' is a tagged pointer with just the wrong value. + if not self.translated_to_c: + if not self.is_valid_gc_object(addr): + return False + return self.nursery <= addr < self.nursery_top + def malloc_fixedsize_clear(self, typeid, size, can_collect, has_finalizer=False, contains_weakptr=False): if (has_finalizer or not can_collect or @@ -453,10 +467,11 @@ JIT_WB_IF_FLAG = GCFLAG_NO_YOUNG_PTRS def write_barrier(self, newvalue, addr_struct): - if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: + if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: self.remember_young_pointer(addr_struct, newvalue) def _setup_wb(self): + DEBUG = self.DEBUG # The purpose of attaching remember_young_pointer to the instance # instead of keeping it as a regular method is to help the JIT call it. # Additionally, it makes the code in write_barrier() marginally smaller @@ -467,20 +482,19 @@ def remember_young_pointer(addr_struct, addr): #llop.debug_print(lltype.Void, "\tremember_young_pointer", # addr_struct, "<-", addr) - ll_assert(not self.is_in_nursery(addr_struct), - "nursery object with GCFLAG_NO_YOUNG_PTRS") - # if we have tagged pointers around, we first need to check whether - # we have valid pointer here, otherwise we can do it after the - # is_in_nursery check - if (self.config.taggedpointers and - not self.is_valid_gc_object(addr)): - return - if self.is_in_nursery(addr): + if DEBUG: + ll_assert(not self.is_in_nursery(addr_struct), + "nursery object with GCFLAG_NO_YOUNG_PTRS") + # + # What is important in this function is that it *must* + # clear the flag GCFLAG_NO_YOUNG_PTRS from 'addr_struct' + # if 'addr' is in the nursery. It is ok if, accidentally, + # it also clears the flag in some more rare cases, like + # 'addr' being a tagged pointer whose value happens to be + # a large integer that fools is_in_nursery(). + if self.appears_to_be_in_nursery(addr): self.old_objects_pointing_to_young.append(addr_struct) self.header(addr_struct).tid &= ~GCFLAG_NO_YOUNG_PTRS - elif (not self.config.taggedpointers and - not self.is_valid_gc_object(addr)): - return self.write_into_last_generation_obj(addr_struct, addr) remember_young_pointer._dont_inline_ = True self.remember_young_pointer = remember_young_pointer @@ -488,9 +502,11 @@ def write_into_last_generation_obj(self, addr_struct, addr): objhdr = self.header(addr_struct) if objhdr.tid & GCFLAG_NO_HEAP_PTRS: - if not self.is_last_generation(addr): + if (self.is_valid_gc_object(addr) and + not self.is_last_generation(addr)): objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS self.last_generation_root_objects.append(addr_struct) + write_into_last_generation_obj._always_inline_ = True def assume_young_pointers(self, addr_struct): objhdr = self.header(addr_struct) Modified: pypy/branch/jitffi/pypy/rpython/memory/gc/markcompact.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gc/markcompact.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/markcompact.py Fri Oct 8 13:40:10 2010 @@ -2,7 +2,6 @@ from pypy.rpython.memory.gc.base import MovingGCBase, read_from_env from pypy.rlib.debug import ll_assert, have_debug_prints from pypy.rlib.debug import debug_print, debug_start, debug_stop -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack, get_address_deque from pypy.rpython.memory.support import AddressDict from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage @@ -86,9 +85,9 @@ free = NULL next_collect_after = -1 - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, space_size=4096, - min_next_collect_after=128): - MovingGCBase.__init__(self, config, chunk_size) + def __init__(self, config, space_size=4096, + min_next_collect_after=128, **kwds): + MovingGCBase.__init__(self, config, **kwds) self.space_size = space_size self.min_next_collect_after = min_next_collect_after Modified: pypy/branch/jitffi/pypy/rpython/memory/gc/marksweep.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gc/marksweep.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/marksweep.py Fri Oct 8 13:40:10 2010 @@ -1,7 +1,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc, raw_free from pypy.rpython.lltypesystem.llmemory import raw_memcopy, raw_memclear from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack from pypy.rpython.memory.gcheader import GCHeaderBuilder from pypy.rpython.lltypesystem import lltype, llmemory, rffi, llgroup @@ -48,9 +47,9 @@ # translating to a real backend. TRANSLATION_PARAMS = {'start_heap_size': 8*1024*1024} # XXX adjust - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, start_heap_size=4096): + def __init__(self, config, start_heap_size=4096, **kwds): self.param_start_heap_size = start_heap_size - GCBase.__init__(self, config, chunk_size) + GCBase.__init__(self, config, **kwds) def setup(self): GCBase.setup(self) @@ -714,8 +713,8 @@ _alloc_flavor_ = "raw" COLLECT_EVERY = 2000 - def __init__(self, chunk_size=DEFAULT_CHUNK_SIZE, start_heap_size=4096): - MarkSweepGC.__init__(self, chunk_size, start_heap_size) + def __init__(self, config, **kwds): + MarkSweepGC.__init__(self, config, **kwds) self.count_mallocs = 0 def maybe_collect(self): Modified: pypy/branch/jitffi/pypy/rpython/memory/gc/minimark.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gc/minimark.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/minimark.py Fri Oct 8 13:40:10 2010 @@ -1,9 +1,9 @@ import sys from pypy.rpython.lltypesystem import lltype, llmemory, llarena, llgroup from pypy.rpython.lltypesystem.lloperation import llop +from pypy.rpython.lltypesystem.llmemory import raw_malloc_usage from pypy.rpython.memory.gc.base import GCBase, MovingGCBase from pypy.rpython.memory.gc import minimarkpage, base, generation -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rlib.rarithmetic import ovfcheck, LONG_BIT, intmask, r_uint from pypy.rlib.rarithmetic import LONG_BIT_SHIFT from pypy.rlib.debug import ll_assert, debug_print, debug_start, debug_stop @@ -92,26 +92,28 @@ # PYPY_GC_NURSERY and fall back to half the size of # the L2 cache. For 'major_collection_threshold' it will look # it up in the env var PYPY_GC_MAJOR_COLLECT. It also sets - # 'max_heap_size' to PYPY_GC_MAX. + # 'max_heap_size' to PYPY_GC_MAX. Finally, PYPY_GC_MIN sets + # the minimal value of 'next_major_collection_threshold'. "read_from_env": True, # The size of the nursery. Note that this is only used as a # fall-back number. "nursery_size": 896*1024, - # The system page size. Like obmalloc.c, we assume that it is 4K, - # which is OK for most systems. - "page_size": 4096, + # The system page size. Like obmalloc.c, we assume that it is 4K + # for 32-bit systems; unlike obmalloc.c, we assume that it is 8K + # for 64-bit systems, for consistent results. + "page_size": 1024*WORD, # The size of an arena. Arenas are groups of pages allocated # together. "arena_size": 65536*WORD, # The maximum size of an object allocated compactly. All objects - # that are larger are just allocated with raw_malloc(). The value - # chosen here is enough for a unicode string of length 56 (on 64-bits) - # or 60 (on 32-bits). See rlib.rstring.INIT_SIZE. - "small_request_threshold": 256-WORD, + # that are larger are just allocated with raw_malloc(). Note that + # the size limit for being first allocated in the nursery is much + # larger; see below. + "small_request_threshold": 35*WORD, # Full collection threshold: after a major collection, we record # the total size consumed; and after every minor collection, if the @@ -125,10 +127,19 @@ # in regular arrays of pointers; more in arrays whose items are # larger. A value of 0 disables card marking. "card_page_indices": 128, - "card_page_indices_min": 800, # minimum number of indices for cards + + # Objects whose total size is at least 'large_object' bytes are + # allocated out of the nursery immediately. If the object + # has GC pointers in its varsized part, we use instead the + # higher limit 'large_object_gcptrs'. The idea is that + # separately allocated objects are allocated immediately "old" + # and it's not good to have too many pointers from old to young + # objects. + "large_object": 1600*WORD, + "large_object_gcptrs": 8250*WORD, } - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, + def __init__(self, config, read_from_env=False, nursery_size=32*WORD, page_size=16*WORD, @@ -136,25 +147,34 @@ small_request_threshold=5*WORD, major_collection_threshold=2.5, card_page_indices=0, - card_page_indices_min=None, - ArenaCollectionClass=None): - MovingGCBase.__init__(self, config, chunk_size) + large_object=8*WORD, + large_object_gcptrs=10*WORD, + ArenaCollectionClass=None, + **kwds): + MovingGCBase.__init__(self, config, **kwds) assert small_request_threshold % WORD == 0 self.read_from_env = read_from_env self.nursery_size = nursery_size self.small_request_threshold = small_request_threshold self.major_collection_threshold = major_collection_threshold self.num_major_collects = 0 + self.min_heap_size = 0.0 self.max_heap_size = 0.0 self.max_heap_size_already_raised = False # self.card_page_indices = card_page_indices if self.card_page_indices > 0: - self.card_page_indices_min = card_page_indices_min self.card_page_shift = 0 while (1 << self.card_page_shift) < self.card_page_indices: self.card_page_shift += 1 # + # 'large_object' and 'large_object_gcptrs' limit how big objects + # can be in the nursery, so they give a lower bound on the allowed + # size of the nursery. + self.nonlarge_max = large_object - 1 + self.nonlarge_gcptrs_max = large_object_gcptrs - 1 + assert self.nonlarge_max <= self.nonlarge_gcptrs_max + # self.nursery = NULL self.nursery_free = NULL self.nursery_top = NULL @@ -218,7 +238,7 @@ else: # defaultsize = self.nursery_size - minsize = 18 * self.small_request_threshold + minsize = 2 * (self.nonlarge_gcptrs_max + 1) self.nursery_size = minsize self.allocate_nursery() # @@ -229,66 +249,92 @@ newsize = generation.estimate_best_nursery_size() if newsize <= 0: newsize = defaultsize + newsize = max(newsize, minsize) # major_coll = base.read_float_from_env('PYPY_GC_MAJOR_COLLECT') if major_coll >= 1.0: self.major_collection_threshold = major_coll # + min_heap_size = base.read_uint_from_env('PYPY_GC_MIN') + if min_heap_size > 0: + self.min_heap_size = float(min_heap_size) + else: + # defaults to 8 times the nursery + self.min_heap_size = newsize * 8 + # max_heap_size = base.read_uint_from_env('PYPY_GC_MAX') if max_heap_size > 0: self.max_heap_size = float(max_heap_size) # self.minor_collection() # to empty the nursery llarena.arena_free(self.nursery) - self.nursery_size = max(newsize, minsize) + self.nursery_size = newsize self.allocate_nursery() def allocate_nursery(self): debug_start("gc-set-nursery-size") debug_print("nursery size:", self.nursery_size) - # the start of the nursery: we actually allocate a tiny bit more for + # the start of the nursery: we actually allocate a bit more for # the nursery than really needed, to simplify pointer arithmetic - # in malloc_fixedsize_clear(). - extra = self.small_request_threshold - self.nursery = llarena.arena_malloc(self.nursery_size + extra, True) + # in malloc_fixedsize_clear(). The few extra pages are never used + # anyway so it doesn't even count. + extra = self.nonlarge_gcptrs_max + 1 + self.nursery = llarena.arena_malloc(self.nursery_size + extra, 2) if not self.nursery: raise MemoryError("cannot allocate nursery") # the current position in the nursery: self.nursery_free = self.nursery # the end of the nursery: self.nursery_top = self.nursery + self.nursery_size - # initialize the threshold, a bit arbitrarily - self.next_major_collection_threshold = ( - self.nursery_size * self.major_collection_threshold) + # initialize the threshold + self.min_heap_size = max(self.min_heap_size, self.nursery_size * + self.major_collection_threshold) + self.set_major_threshold_from(0.0) debug_stop("gc-set-nursery-size") + def set_major_threshold_from(self, threshold): + # Set the next_major_collection_threshold. + if threshold < self.min_heap_size: + threshold = self.min_heap_size + # + if self.max_heap_size > 0.0 and threshold > self.max_heap_size: + threshold = self.max_heap_size + bounded = True + else: + bounded = False + # + self.next_major_collection_threshold = threshold + return bounded + def malloc_fixedsize_clear(self, typeid, size, can_collect=True, needs_finalizer=False, contains_weakptr=False): ll_assert(can_collect, "!can_collect") size_gc_header = self.gcheaderbuilder.size_gc_header totalsize = size_gc_header + size - rawtotalsize = llmemory.raw_malloc_usage(totalsize) + rawtotalsize = raw_malloc_usage(totalsize) # # If the object needs a finalizer, ask for a rawmalloc. # The following check should be constant-folded. if needs_finalizer: ll_assert(not contains_weakptr, "'needs_finalizer' and 'contains_weakptr' both specified") - result = self.malloc_with_finalizer(typeid, totalsize) + obj = self.external_malloc(typeid, 0) + self.objects_with_finalizers.append(obj) # - # If totalsize is greater than small_request_threshold, ask for - # a rawmalloc. The following check should be constant-folded. - elif rawtotalsize > self.small_request_threshold: + # If totalsize is greater than nonlarge_max (which should never be + # the case in practice), ask for a rawmalloc. The following check + # should be constant-folded. + elif rawtotalsize > self.nonlarge_max: ll_assert(not contains_weakptr, "'contains_weakptr' specified for a large object") - result = self._external_malloc(typeid, totalsize) + obj = self.external_malloc(typeid, 0) # else: # If totalsize is smaller than minimal_size_in_nursery, round it # up. The following check should also be constant-folded. - min_size = llmemory.raw_malloc_usage(self.minimal_size_in_nursery) + min_size = raw_malloc_usage(self.minimal_size_in_nursery) if rawtotalsize < min_size: totalsize = rawtotalsize = min_size # @@ -306,8 +352,10 @@ # If it is a weakref, record it (check constant-folded). if contains_weakptr: self.young_objects_with_weakrefs.append(result+size_gc_header) + # + obj = result + size_gc_header # - return llmemory.cast_adr_to_ptr(result+size_gc_header, llmemory.GCREF) + return llmemory.cast_adr_to_ptr(obj, llmemory.GCREF) def malloc_varsize_clear(self, typeid, length, size, itemsize, @@ -315,32 +363,41 @@ ll_assert(can_collect, "!can_collect") size_gc_header = self.gcheaderbuilder.size_gc_header nonvarsize = size_gc_header + size - try: - varsize = ovfcheck(itemsize * length) - totalsize = ovfcheck(nonvarsize + varsize) - except OverflowError: - raise MemoryError # - # If totalsize is greater than small_request_threshold, ask for - # a rawmalloc. - if llmemory.raw_malloc_usage(totalsize) > self.small_request_threshold: - result = self._external_malloc_cardmark(typeid, totalsize, length) + # Compute the maximal length that makes the object still + # below 'nonlarge_max'. All the following logic is usually + # constant-folded because self.nonlarge_max, size and itemsize + # are all constants (the arguments are constant due to + # inlining) and self.has_gcptr_in_varsize() is constant-folded. + if self.has_gcptr_in_varsize(typeid): + nonlarge_max = self.nonlarge_gcptrs_max + else: + nonlarge_max = self.nonlarge_max + + if not raw_malloc_usage(itemsize): + too_many_items = raw_malloc_usage(nonvarsize) > nonlarge_max + else: + maxlength = nonlarge_max - raw_malloc_usage(nonvarsize) + maxlength = maxlength // raw_malloc_usage(itemsize) + too_many_items = length > maxlength + + if too_many_items: + # + # If the total size of the object would be larger than + # 'nonlarge_max', then allocate it externally. + obj = self.external_malloc(typeid, length) # else: - # Round the size up to the next multiple of WORD. Note that - # this is done only if totalsize <= self.small_request_threshold, - # i.e. it cannot overflow, and it keeps the property that - # totalsize <= self.small_request_threshold. + # With the above checks we know now that totalsize cannot be more + # than 'nonlarge_max'; in particular, the + and * cannot overflow. + totalsize = nonvarsize + itemsize * length totalsize = llarena.round_up_for_allocation(totalsize) - ll_assert(llmemory.raw_malloc_usage(totalsize) <= - self.small_request_threshold, - "round_up_for_allocation() rounded up too much?") # # 'totalsize' should contain at least the GC header and # the length word, so it should never be smaller than # 'minimal_size_in_nursery' - ll_assert(llmemory.raw_malloc_usage(totalsize) >= - llmemory.raw_malloc_usage(self.minimal_size_in_nursery), + ll_assert(raw_malloc_usage(totalsize) >= + raw_malloc_usage(self.minimal_size_in_nursery), "malloc_varsize_clear(): totalsize < minimalsize") # # Get the memory from the nursery. If there is not enough space @@ -353,10 +410,12 @@ # Build the object. llarena.arena_reserve(result, totalsize) self.init_gc_object(result, typeid, flags=0) + # + # Set the length and return the object. + obj = result + size_gc_header + (obj + offset_to_length).signed[0] = length # - # Set the length and return the object. - (result + size_gc_header + offset_to_length).signed[0] = length - return llmemory.cast_adr_to_ptr(result+size_gc_header, llmemory.GCREF) + return llmemory.cast_adr_to_ptr(obj, llmemory.GCREF) def collect(self, gen=1): @@ -389,105 +448,108 @@ collect_and_reserve._dont_inline_ = True - def _full_collect_if_needed(self, reserving_size): - reserving_size = llmemory.raw_malloc_usage(reserving_size) - if (float(self.get_total_memory_used()) + reserving_size > - self.next_major_collection_threshold): - self.minor_collection() - self.major_collection(reserving_size) - - def _external_malloc(self, typeid, totalsize): - """Allocate a large object using raw_malloc().""" - return self._external_malloc_cardmark(typeid, totalsize, 0) - - - def _external_malloc_cardmark(self, typeid, totalsize, length): - """Allocate a large object using raw_malloc(), possibly as an - object with card marking enabled, if its length is large enough. - 'length' can be specified as 0 if the object is not varsized.""" + def external_malloc(self, typeid, length): + """Allocate a large object using the ArenaCollection or + raw_malloc(), possibly as an object with card marking enabled, + if it has gc pointers in its var-sized part. 'length' should be + specified as 0 if the object is not varsized. The returned + object is fully initialized and zero-filled.""" + # + # Compute the total size, carefully checking for overflows. + size_gc_header = self.gcheaderbuilder.size_gc_header + nonvarsize = size_gc_header + self.fixed_size(typeid) + if length == 0: + # this includes the case of fixed-size objects, for which we + # should not even ask for the varsize_item_sizes(). + totalsize = nonvarsize + else: + itemsize = self.varsize_item_sizes(typeid) + try: + varsize = ovfcheck(itemsize * length) + totalsize = ovfcheck(nonvarsize + varsize) + except OverflowError: + raise MemoryError # # If somebody calls this function a lot, we must eventually # force a full collection. - self._full_collect_if_needed(totalsize) + if (float(self.get_total_memory_used()) + raw_malloc_usage(totalsize) > + self.next_major_collection_threshold): + self.minor_collection() + self.major_collection(raw_malloc_usage(totalsize)) # - # Check if we need to introduce the card marker bits area. - if (self.card_page_indices <= 0 # <- this check is constant-folded - or length < self.card_page_indices_min # <- must be large enough - or not self.has_gcptr_in_varsize(typeid)): # <- must contain ptrs + # Check if the object would fit in the ArenaCollection. + if raw_malloc_usage(totalsize) <= self.small_request_threshold: + # + # Yes. Round up 'totalsize' (it cannot overflow and it + # must remain <= self.small_request_threshold.) + totalsize = llarena.round_up_for_allocation(totalsize) + ll_assert(raw_malloc_usage(totalsize) <= + self.small_request_threshold, + "rounding up made totalsize > small_request_threshold") # - # In these cases, we don't want a card marker bits area. - cardheadersize = 0 + # Allocate from the ArenaCollection and clear the memory returned. + result = self.ac.malloc(totalsize) + llmemory.raw_memclear(result, totalsize) extra_flags = 0 # else: - # Reserve N extra words containing card bits before the object. - extra_words = self.card_marking_words_for_length(length) - cardheadersize = WORD * extra_words - extra_flags = GCFLAG_HAS_CARDS - # - allocsize = cardheadersize + llmemory.raw_malloc_usage(totalsize) - # - # Allocate the object using arena_malloc(), which we assume here - # is just the same as raw_malloc(), but allows the extra flexibility - # of saying that we have extra words in the header. - arena = llarena.arena_malloc(allocsize, False) - if not arena: - raise MemoryError("cannot allocate large object") - # - # Clear it using method 2 of llarena.arena_reset(), which is the - # same as just a raw_memclear(). - llarena.arena_reset(arena, allocsize, 2) - # - # Reserve the card mark as a list of single bytes - # (the loop is empty in C). - i = 0 - while i < cardheadersize: - llarena.arena_reserve(arena + i, llmemory.sizeof(lltype.Char)) - i += 1 - # - # Initialize the object. - result = arena + cardheadersize - llarena.arena_reserve(result, totalsize) - self.init_gc_object(result, typeid, GCFLAG_NO_YOUNG_PTRS | extra_flags) - # - # Record the newly allocated object and its size. - size_gc_header = self.gcheaderbuilder.size_gc_header - self.rawmalloced_total_size += llmemory.raw_malloc_usage(totalsize) - self.rawmalloced_objects.append(result + size_gc_header) - return result - _external_malloc_cardmark._dont_inline_ = True - - - def _malloc_nonmovable(self, typeid, totalsize): - """Allocate an object non-movable.""" - # - rawtotalsize = llmemory.raw_malloc_usage(totalsize) - if rawtotalsize > self.small_request_threshold: + # No, so proceed to allocate it externally with raw_malloc(). + # Check if we need to introduce the card marker bits area. + if (self.card_page_indices <= 0 # <- this check is constant-folded + or not self.has_gcptr_in_varsize(typeid) or + raw_malloc_usage(totalsize) <= self.nonlarge_gcptrs_max): + # + # In these cases, we don't want a card marker bits area. + # This case also includes all fixed-size objects. + cardheadersize = 0 + extra_flags = 0 + # + else: + # Reserve N extra words containing card bits before the object. + extra_words = self.card_marking_words_for_length(length) + cardheadersize = WORD * extra_words + extra_flags = GCFLAG_HAS_CARDS + # + # Detect very rare cases of overflows + if raw_malloc_usage(totalsize) > (sys.maxint - (WORD-1) + - cardheadersize): + raise MemoryError("rare case of overflow") + # + # Now we know that the following computations cannot overflow. + # Note that round_up_for_allocation() is also needed to get the + # correct number added to 'rawmalloced_total_size'. + allocsize = (cardheadersize + raw_malloc_usage( + llarena.round_up_for_allocation(totalsize))) + # + # Allocate the object using arena_malloc(), which we assume here + # is just the same as raw_malloc(), but allows the extra + # flexibility of saying that we have extra words in the header. + # The memory returned is cleared by a raw_memclear(). + arena = llarena.arena_malloc(allocsize, 2) + if not arena: + raise MemoryError("cannot allocate large object") + # + # Reserve the card mark bits as a list of single bytes + # (the loop is empty in C). + i = 0 + while i < cardheadersize: + llarena.arena_reserve(arena + i, llmemory.sizeof(lltype.Char)) + i += 1 # - # The size asked for is too large for the ArenaCollection. - return self._external_malloc(typeid, totalsize) - # - totalsize = llarena.round_up_for_allocation(totalsize) - # - # If somebody calls _malloc_nonmovable() a lot, we must eventually - # force a full collection. - self._full_collect_if_needed(totalsize) - # - # Ask the ArenaCollection to do the malloc. - result = self.ac.malloc(totalsize) - llmemory.raw_memclear(result, totalsize) - self.init_gc_object(result, typeid, GCFLAG_NO_YOUNG_PTRS) - return result - - - def malloc_with_finalizer(self, typeid, totalsize): - """Allocate an object with a finalizer.""" + # Reserve the actual object. (This is also a no-op in C). + result = arena + cardheadersize + llarena.arena_reserve(result, totalsize) + # + # Record the newly allocated object and its full malloced size. + self.rawmalloced_total_size += allocsize + self.rawmalloced_objects.append(result + size_gc_header) # - result = self._malloc_nonmovable(typeid, totalsize) - size_gc_header = self.gcheaderbuilder.size_gc_header - self.objects_with_finalizers.append(result + size_gc_header) - return result - malloc_with_finalizer._dont_inline_ = True + # Common code to fill the header and length of the object. + self.init_gc_object(result, typeid, GCFLAG_NO_YOUNG_PTRS | extra_flags) + if self.is_varsize(typeid): + offset_to_length = self.varsize_offset_to_length(typeid) + (result + size_gc_header + offset_to_length).signed[0] = length + return result + size_gc_header # ---------- @@ -502,6 +564,11 @@ def can_malloc_nonmovable(self): return True + def can_optimize_clean_setarrayitems(self): + if self.card_page_indices > 0: + return False + return MovingGCBase.can_optimize_clean_setarrayitems(self) + def can_move(self, obj): """Overrides the parent can_move().""" return self.is_in_nursery(obj) @@ -529,37 +596,16 @@ def malloc_fixedsize_nonmovable(self, typeid): - """NOT_RPYTHON: not tested translated""" - size_gc_header = self.gcheaderbuilder.size_gc_header - totalsize = size_gc_header + self.fixed_size(typeid) - # - result = self._malloc_nonmovable(typeid, totalsize) - obj = result + size_gc_header + obj = self.external_malloc(typeid, 0) return llmemory.cast_adr_to_ptr(obj, llmemory.GCREF) def malloc_varsize_nonmovable(self, typeid, length): - size_gc_header = self.gcheaderbuilder.size_gc_header - nonvarsize = size_gc_header + self.fixed_size(typeid) - itemsize = self.varsize_item_sizes(typeid) - offset_to_length = self.varsize_offset_to_length(typeid) - try: - varsize = ovfcheck(itemsize * length) - totalsize = ovfcheck(nonvarsize + varsize) - except OverflowError: - raise MemoryError - # - result = self._malloc_nonmovable(typeid, totalsize) - obj = result + size_gc_header - (obj + offset_to_length).signed[0] = length + obj = self.external_malloc(typeid, length) return llmemory.cast_adr_to_ptr(obj, llmemory.GCREF) def malloc_nonmovable(self, typeid, length, zero): # helper for testing, same as GCBase.malloc - if self.is_varsize(typeid): - gcref = self.malloc_varsize_nonmovable(typeid, length) - else: - gcref = self.malloc_fixedsize_nonmovable(typeid) - return llmemory.cast_ptr_to_adr(gcref) + return self.external_malloc(typeid, length or 0) # None -> 0 # ---------- @@ -590,6 +636,14 @@ "odd-valued (i.e. tagged) pointer unexpected here") return self.nursery <= addr < self.nursery_top + def appears_to_be_in_nursery(self, addr): + # same as is_in_nursery(), but may return True accidentally if + # 'addr' is a tagged pointer with just the wrong value. + if not self.translated_to_c: + if not self.is_valid_gc_object(addr): + return False + return self.nursery <= addr < self.nursery_top + def is_forwarded(self, obj): """Returns True if the nursery obj is marked as forwarded. Implemented a bit obscurely by checking an unrelated flag @@ -647,8 +701,9 @@ "unexpected GCFLAG_CARDS_SET") # if the GCFLAG_HAS_CARDS is set, check that all bits are zero now if self.header(obj).tid & GCFLAG_HAS_CARDS: - ll_assert(self.card_page_indices > 0, - "GCFLAG_HAS_CARDS but not using card marking") + if self.card_page_indices <= 0: + ll_assert(False, "GCFLAG_HAS_CARDS but not using card marking") + return typeid = self.get_type_id(obj) ll_assert(self.has_gcptr_in_varsize(typeid), "GCFLAG_HAS_CARDS but not has_gcptr_in_varsize") @@ -675,6 +730,10 @@ # "if addr_struct.int0 & JIT_WB_IF_FLAG: remember_young_pointer()") JIT_WB_IF_FLAG = GCFLAG_NO_YOUNG_PTRS + @classmethod + def JIT_max_size_of_young_obj(cls): + return cls.TRANSLATION_PARAMS['large_object'] + def write_barrier(self, newvalue, addr_struct): if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS: self.remember_young_pointer(addr_struct, newvalue) @@ -682,12 +741,12 @@ def write_barrier_from_array(self, newvalue, addr_array, index): if self.header(addr_array).tid & GCFLAG_NO_YOUNG_PTRS: if self.card_page_indices > 0: # <- constant-folded - self.remember_young_pointer_from_array(addr_array, index, - newvalue) + self.remember_young_pointer_from_array(addr_array, index) else: self.remember_young_pointer(addr_array, newvalue) def _init_writebarrier_logic(self): + DEBUG = self.DEBUG # The purpose of attaching remember_young_pointer to the instance # instead of keeping it as a regular method is to help the JIT call it. # Additionally, it makes the code in write_barrier() marginally smaller @@ -695,35 +754,28 @@ # For x86, there is also an extra requirement: when the JIT calls # remember_young_pointer(), it assumes that it will not touch the SSE # registers, so it does not save and restore them (that's a *hack*!). - def remember_young_pointer(addr_struct, addr): - # 'addr_struct' is the address of the object in which we write; - # 'addr' is the address that we write in 'addr_struct'. - ll_assert(not self.is_in_nursery(addr_struct), - "nursery object with GCFLAG_NO_YOUNG_PTRS") - # if we have tagged pointers around, we first need to check whether - # we have valid pointer here, otherwise we can do it after the - # is_in_nursery check - if (self.config.taggedpointers and - not self.is_valid_gc_object(addr)): - return + def remember_young_pointer(addr_struct, newvalue): + # 'addr_struct' is the address of the object in which we write. + # 'newvalue' is the address that we are going to write in there. + if DEBUG: + ll_assert(not self.is_in_nursery(addr_struct), + "nursery object with GCFLAG_NO_YOUNG_PTRS") # - # Core logic: if the 'addr' is in the nursery, then we need + # If it seems that what we are writing is a pointer to the nursery + # (as checked with appears_to_be_in_nursery()), then we need # to remove the flag GCFLAG_NO_YOUNG_PTRS and add the old object # to the list 'old_objects_pointing_to_young'. We know that # 'addr_struct' cannot be in the nursery, because nursery objects # never have the flag GCFLAG_NO_YOUNG_PTRS to start with. objhdr = self.header(addr_struct) - if self.is_in_nursery(addr): + if self.appears_to_be_in_nursery(newvalue): self.old_objects_pointing_to_young.append(addr_struct) objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS - elif (not self.config.taggedpointers and - not self.is_valid_gc_object(addr)): - return # # Second part: if 'addr_struct' is actually a prebuilt GC # object and it's the first time we see a write to it, we # add it to the list 'prebuilt_root_objects'. Note that we - # do it even in the (rare?) case of 'addr' being another + # do it even in the (rare?) case of 'addr' being NULL or another # prebuilt object, to simplify code. if objhdr.tid & GCFLAG_NO_HEAP_PTRS: objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS @@ -737,17 +789,24 @@ def _init_writebarrier_with_card_marker(self): - def remember_young_pointer_from_array(addr_array, index, addr): + DEBUG = self.DEBUG + def remember_young_pointer_from_array(addr_array, index): # 'addr_array' is the address of the object in which we write, # which must have an array part; 'index' is the index of the - # item that is (or contains) the pointer that we write; - # 'addr' is the address that we write in the array. + # item that is (or contains) the pointer that we write. + if DEBUG: + ll_assert(not self.is_in_nursery(addr_array), + "nursery array with GCFLAG_NO_YOUNG_PTRS") objhdr = self.header(addr_array) if objhdr.tid & GCFLAG_HAS_CARDS == 0: # - # no cards, use default logic. The 'nocard_logic()' is just - # 'remember_young_pointer()', but forced to be inlined here. - nocard_logic(addr_array, addr) + # no cards, use default logic. Mostly copied from above. + self.old_objects_pointing_to_young.append(addr_array) + objhdr = self.header(addr_array) + objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS + if objhdr.tid & GCFLAG_NO_HEAP_PTRS: + objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS + self.prebuilt_root_objects.append(addr_array) return # # 'addr_array' is a raw_malloc'ed array with card markers @@ -764,27 +823,16 @@ if byte & bitmask: return # - # As in remember_young_pointer, check if 'addr' is a valid - # pointer, in case it can be a tagged integer - if (self.config.taggedpointers and - not self.is_valid_gc_object(addr)): - return - # - # If the 'addr' is in the nursery, then we need to set the flag. - # Note that the following check is done after the bit check - # above, because it is expected that the "bit already set" - # situation is the most common. - if self.is_in_nursery(addr): - addr_byte.char[0] = chr(byte | bitmask) - # - if objhdr.tid & GCFLAG_CARDS_SET == 0: - self.old_objects_with_cards_set.append(addr_array) - objhdr.tid |= GCFLAG_CARDS_SET - - nocard_logic = func_with_new_name(self.remember_young_pointer, - 'remember_young_pointer_nocard') - del nocard_logic._dont_inline_ - nocard_logic._always_inline_ = True + # We set the flag (even if the newly written address does not + # actually point to the nursery, which seems to be ok -- actually + # it seems more important that remember_young_pointer_from_array() + # does not take 3 arguments). + addr_byte.char[0] = chr(byte | bitmask) + # + if objhdr.tid & GCFLAG_CARDS_SET == 0: + self.old_objects_with_cards_set.append(addr_array) + objhdr.tid |= GCFLAG_CARDS_SET + remember_young_pointer_from_array._dont_inline_ = True self.remember_young_pointer_from_array = ( remember_young_pointer_from_array) @@ -934,6 +982,8 @@ if cardbyte & 1: if interval_stop > length: interval_stop = length + ll_assert(cardbyte <= 1 and bytes == 0, + "premature end of object") self.trace_and_drag_out_of_nursery_partial( obj, interval_start, interval_stop) # @@ -997,7 +1047,7 @@ if self.header(obj).tid & GCFLAG_HAS_SHADOW == 0: # # Common case: allocate a new nonmovable location for it. - newhdr = self.ac.malloc(totalsize) + newhdr = self._malloc_out_of_nursery(totalsize) # else: # The object has already a shadow. @@ -1035,6 +1085,33 @@ self.old_objects_pointing_to_young.append(newobj) + def _malloc_out_of_nursery(self, totalsize): + """Allocate non-movable memory for an object of the given + 'totalsize' that lives so far in the nursery.""" + if raw_malloc_usage(totalsize) <= self.small_request_threshold: + # most common path + return self.ac.malloc(totalsize) + else: + # for nursery objects that are not small + return self._malloc_out_of_nursery_nonsmall(totalsize) + _malloc_out_of_nursery._always_inline_ = True + + def _malloc_out_of_nursery_nonsmall(self, totalsize): + # 'totalsize' should be aligned. + ll_assert(raw_malloc_usage(totalsize) & (WORD-1) == 0, + "misaligned totalsize in _malloc_out_of_nursery_nonsmall") + # + arena = llarena.arena_malloc(raw_malloc_usage(totalsize), False) + if not arena: + raise MemoryError("cannot allocate object") + llarena.arena_reserve(arena, totalsize) + # + size_gc_header = self.gcheaderbuilder.size_gc_header + self.rawmalloced_total_size += raw_malloc_usage(totalsize) + self.rawmalloced_objects.append(arena + size_gc_header) + return arena + + # ---------- # Full collection @@ -1104,30 +1181,26 @@ # Set the threshold for the next major collection to be when we # have allocated 'major_collection_threshold' times more than # we currently have. - self.next_major_collection_threshold = ( + bounded = self.set_major_threshold_from( (self.get_total_memory_used() * self.major_collection_threshold) + reserving_size) # # Max heap size: gives an upper bound on the threshold. If we # already have at least this much allocated, raise MemoryError. - if (self.max_heap_size > 0.0 and - self.next_major_collection_threshold > self.max_heap_size): + if bounded and (float(self.get_total_memory_used()) + reserving_size >= + self.next_major_collection_threshold): # - self.next_major_collection_threshold = self.max_heap_size - if (float(self.get_total_memory_used()) + reserving_size >= - self.next_major_collection_threshold): - # - # First raise MemoryError, giving the program a chance to - # quit cleanly. It might still allocate in the nursery, - # which might eventually be emptied, triggering another - # major collect and (possibly) reaching here again with an - # even higher memory consumption. To prevent it, if it's - # the second time we are here, then abort the program. - if self.max_heap_size_already_raised: - llop.debug_fatalerror(lltype.Void, - "Using too much memory, aborting") - self.max_heap_size_already_raised = True - raise MemoryError + # First raise MemoryError, giving the program a chance to + # quit cleanly. It might still allocate in the nursery, + # which might eventually be emptied, triggering another + # major collect and (possibly) reaching here again with an + # even higher memory consumption. To prevent it, if it's + # the second time we are here, then abort the program. + if self.max_heap_size_already_raised: + llop.debug_fatalerror(lltype.Void, + "Using too much memory, aborting") + self.max_heap_size_already_raised = True + raise MemoryError # # At the end, we can execute the finalizers of the objects # listed in 'run_finalizers'. Note that this will typically do @@ -1159,8 +1232,7 @@ self.rawmalloced_objects.append(obj) else: totalsize = size_gc_header + self.get_size(obj) - rawtotalsize = llmemory.raw_malloc_usage(totalsize) - self.rawmalloced_total_size -= rawtotalsize + allocsize = raw_malloc_usage(totalsize) arena = llarena.getfakearenaaddress(obj - size_gc_header) # # Must also include the card marker area, if any @@ -1175,8 +1247,10 @@ length = (obj + offset_to_length).signed[0] extra_words = self.card_marking_words_for_length(length) arena -= extra_words * WORD + allocsize += extra_words * WORD # llarena.arena_free(arena) + self.rawmalloced_total_size -= allocsize # list.delete() @@ -1260,7 +1334,8 @@ else: size_gc_header = self.gcheaderbuilder.size_gc_header size = self.get_size(obj) - shadowhdr = self.ac.malloc(size_gc_header + size) + shadowhdr = self._malloc_out_of_nursery(size_gc_header + + size) # initialize to an invalid tid *without* GCFLAG_VISITED, # so that if the object dies before the next minor # collection, the shadow will stay around but be collected @@ -1454,7 +1529,7 @@ self.total_memory_used = 0 def malloc(self, size): - nsize = llmemory.raw_malloc_usage(size) + nsize = raw_malloc_usage(size) ll_assert(nsize > 0, "malloc: size is null or negative") ll_assert(nsize <= self.small_request_threshold,"malloc: size too big") ll_assert((nsize & (WORD-1)) == 0, "malloc: size is not aligned") Modified: pypy/branch/jitffi/pypy/rpython/memory/gc/minimarkpage.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gc/minimarkpage.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/minimarkpage.py Fri Oct 8 13:40:10 2010 @@ -5,13 +5,44 @@ WORD = LONG_BIT // 8 NULL = llmemory.NULL +WORD_POWER_2 = {32: 2, 64: 3}[LONG_BIT] +assert 1 << WORD_POWER_2 == WORD -# Terminology: the memory is subdivided into "pages". +# Terminology: the memory is subdivided into "arenas" containing "pages". # A page contains a number of allocated objects, called "blocks". -# The actual allocation occurs in whole arenas, which are subdivided -# into pages. We don't keep track of the arenas. A page can be: +# The actual allocation occurs in whole arenas, which are then subdivided +# into pages. For each arena we allocate one of the following structures: + +ARENA_PTR = lltype.Ptr(lltype.ForwardReference()) +ARENA = lltype.Struct('ArenaReference', + # -- The address of the arena, as returned by malloc() + ('base', llmemory.Address), + # -- The number of free and the total number of pages in the arena + ('nfreepages', lltype.Signed), + ('totalpages', lltype.Signed), + # -- A chained list of free pages in the arena. Ends with NULL. + ('freepages', llmemory.Address), + # -- A linked list of arenas. See below. + ('nextarena', ARENA_PTR), + ) +ARENA_PTR.TO.become(ARENA) +ARENA_NULL = lltype.nullptr(ARENA) + +# The idea is that when we need a free page, we take it from the arena +# which currently has the *lowest* number of free pages. This allows +# arenas with a lot of free pages to eventually become entirely free, at +# which point they are returned to the OS. If an arena has a total of +# 64 pages, then we have 64 global lists, arenas_lists[0] to +# arenas_lists[63], such that arenas_lists[i] contains exactly those +# arenas that have 'nfreepages == i'. We allocate pages out of the +# arena in 'current_arena'; when it is exhausted we pick another arena +# with the smallest value for nfreepages (but > 0). + +# ____________________________________________________________ +# +# Each page in an arena can be: # # - uninitialized: never touched so far. # @@ -20,10 +51,11 @@ # room for objects of that size, unless it is completely full. # # - free: used to be partially full, and is now free again. The page is -# on the chained list of free pages. +# on the chained list of free pages 'freepages' from its arena. -# Similarily, each allocated page contains blocks of a given size, which can -# be either uninitialized, allocated or free. +# Each allocated page contains blocks of a given size, which can again be in +# one of three states: allocated, free, or uninitialized. The uninitialized +# blocks (initially all of them) are at the tail of the page. PAGE_PTR = lltype.Ptr(lltype.ForwardReference()) PAGE_HEADER = lltype.Struct('PageHeader', @@ -31,14 +63,20 @@ # pages, it is a chained list of pages having the same size class, # rooted in 'page_for_size[size_class]'. For full pages, it is a # different chained list rooted in 'full_page_for_size[size_class]'. + # For free pages, it is the list 'freepages' in the arena header. ('nextpage', PAGE_PTR), - # -- The number of free blocks, and the number of uninitialized blocks. - # The number of allocated blocks is the rest. - ('nuninitialized', lltype.Signed), + # -- The arena this page is part of. + ('arena', ARENA_PTR), + # -- The number of free blocks. The numbers of uninitialized and + # allocated blocks can be deduced from the context if needed. ('nfree', lltype.Signed), - # -- The chained list of free blocks. If there are none, points to the - # first uninitialized block. + # -- The chained list of free blocks. It ends as a pointer to the + # first uninitialized block (pointing to data that is uninitialized, + # or to the end of the page). ('freeblock', llmemory.Address), + # -- The structure above is 4 words, which is a good value: + # '(1024-4) % N' is zero or very small for various small N's, + # i.e. there is not much wasted space. ) PAGE_PTR.TO.become(PAGE_HEADER) PAGE_NULL = lltype.nullptr(PAGE_HEADER) @@ -68,13 +106,35 @@ self.nblocks_for_size = lltype.malloc(rffi.CArray(lltype.Signed), length, flavor='raw') self.hdrsize = llmemory.raw_malloc_usage(llmemory.sizeof(PAGE_HEADER)) + assert page_size > self.hdrsize self.nblocks_for_size[0] = 0 # unused for i in range(1, length): self.nblocks_for_size[i] = (page_size - self.hdrsize) // (WORD * i) # - self.uninitialized_pages = NULL + self.max_pages_per_arena = arena_size // page_size + self.arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR), + self.max_pages_per_arena, + flavor='raw', zero=True) + # this is used in mass_free() only + self.old_arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR), + self.max_pages_per_arena, + flavor='raw', zero=True) + # + # the arena currently consumed; it must have at least one page + # available, or be NULL. The arena object that we point to is + # not in any 'arenas_lists'. We will consume all its pages before + # we choose a next arena, even if there is a major collection + # in-between. + self.current_arena = ARENA_NULL + # + # guarantee that 'arenas_lists[1:min_empty_nfreepages]' are all empty + self.min_empty_nfreepages = self.max_pages_per_arena + # + # part of current_arena might still contain uninitialized pages self.num_uninitialized_pages = 0 - self.free_pages = NULL + # + # the total memory used, counting every block in use, without + # the additional bookkeeping stuff. self.total_memory_used = r_uint(0) @@ -87,7 +147,7 @@ self.total_memory_used += nsize # # Get the page to use from the size - size_class = nsize / WORD + size_class = nsize >> WORD_POWER_2 page = self.page_for_size[size_class] if page == PAGE_NULL: page = self.allocate_new_page(size_class) @@ -105,16 +165,12 @@ # else: # The 'result' is part of the uninitialized blocks. - ll_assert(page.nuninitialized > 0, - "fully allocated page found in the page_for_size list") - page.nuninitialized -= 1 - if page.nuninitialized > 0: - freeblock = result + nsize - else: - freeblock = NULL + freeblock = result + nsize # page.freeblock = freeblock - if freeblock == NULL: + # + pageaddr = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) + if freeblock - pageaddr > self.page_size - nsize: # This was the last free block, so unlink the page from the # chained list and put it in the 'full_page_for_size' list. self.page_for_size[size_class] = page.nextpage @@ -128,37 +184,88 @@ def allocate_new_page(self, size_class): """Allocate and return a new page for the given size_class.""" # - if self.free_pages != NULL: + # Allocate a new arena if needed. + if self.current_arena == ARENA_NULL: + self.allocate_new_arena() + # + # The result is simply 'current_arena.freepages'. + arena = self.current_arena + result = arena.freepages + if arena.nfreepages > 0: + # + # The 'result' was part of the chained list; read the next. + arena.nfreepages -= 1 + freepages = result.address[0] + llarena.arena_reset(result, + llmemory.sizeof(llmemory.Address), + 0) # - # Get the page from the chained list 'free_pages'. - page = self.free_pages - self.free_pages = page.address[0] - llarena.arena_reset(page, llmemory.sizeof(llmemory.Address), 0) else: - # Get the next free page from the uninitialized pages. - if self.num_uninitialized_pages == 0: - self.allocate_new_arena() # Out of memory. Get a new arena. - page = self.uninitialized_pages - self.uninitialized_pages += self.page_size + # The 'result' is part of the uninitialized pages. + ll_assert(self.num_uninitialized_pages > 0, + "fully allocated arena found in self.current_arena") self.num_uninitialized_pages -= 1 + if self.num_uninitialized_pages > 0: + freepages = result + self.page_size + else: + freepages = NULL # - # Initialize the fields of the resulting page - llarena.arena_reserve(page, llmemory.sizeof(PAGE_HEADER)) - result = llmemory.cast_adr_to_ptr(page, PAGE_PTR) + arena.freepages = freepages + if freepages == NULL: + # This was the last page, so put the arena away into + # arenas_lists[0]. + ll_assert(arena.nfreepages == 0, + "freepages == NULL but nfreepages > 0") + arena.nextarena = self.arenas_lists[0] + self.arenas_lists[0] = arena + self.current_arena = ARENA_NULL # - result.nuninitialized = self.nblocks_for_size[size_class] - result.nfree = 0 - result.freeblock = page + self.hdrsize - result.nextpage = PAGE_NULL + # Initialize the fields of the resulting page + llarena.arena_reserve(result, llmemory.sizeof(PAGE_HEADER)) + page = llmemory.cast_adr_to_ptr(result, PAGE_PTR) + page.arena = arena + page.nfree = 0 + page.freeblock = result + self.hdrsize + page.nextpage = PAGE_NULL ll_assert(self.page_for_size[size_class] == PAGE_NULL, "allocate_new_page() called but a page is already waiting") - self.page_for_size[size_class] = result - return result + self.page_for_size[size_class] = page + return page + + + def _all_arenas(self): + """For testing. Enumerates all arenas.""" + if self.current_arena: + yield self.current_arena + for arena in self.arenas_lists: + while arena: + yield arena + arena = arena.nextarena def allocate_new_arena(self): - ll_assert(self.num_uninitialized_pages == 0, - "some uninitialized pages are already waiting") + """Loads in self.current_arena the arena to allocate from next.""" + # + # Pick an arena from 'arenas_lists[i]', with i as small as possible + # but > 0. Use caching with 'min_empty_nfreepages', which guarantees + # that 'arenas_lists[1:min_empty_nfreepages]' are all empty. + i = self.min_empty_nfreepages + while i < self.max_pages_per_arena: + # + if self.arenas_lists[i] != ARENA_NULL: + # + # Found it. + self.current_arena = self.arenas_lists[i] + self.arenas_lists[i] = self.current_arena.nextarena + return + # + i += 1 + self.min_empty_nfreepages = i + # + # No more arena with any free page. We must allocate a new arena. + if not we_are_translated(): + for a in self._all_arenas(): + assert a.nfreepages == 0 # # 'arena_base' points to the start of malloced memory; it might not # be a page-aligned address @@ -173,13 +280,15 @@ # 'npages' is the number of full pages just allocated npages = (arena_end - firstpage) // self.page_size # - # add these pages to the list - self.uninitialized_pages = firstpage + # Allocate an ARENA object and initialize it + arena = lltype.malloc(ARENA, flavor='raw') + arena.base = arena_base + arena.nfreepages = 0 # they are all uninitialized pages + arena.totalpages = npages + arena.freepages = firstpage self.num_uninitialized_pages = npages + self.current_arena = arena # - # increase a bit arena_size for the next time - self.arena_size = (self.arena_size // 4 * 5) + (self.page_size - 1) - self.arena_size = (self.arena_size // self.page_size) * self.page_size allocate_new_arena._dont_inline_ = True @@ -190,21 +299,56 @@ self.total_memory_used = r_uint(0) # # For each size class: - size_class = self.small_request_threshold / WORD + size_class = self.small_request_threshold >> WORD_POWER_2 while size_class >= 1: # # Walk the pages in 'page_for_size[size_class]' and # 'full_page_for_size[size_class]' and free some objects. - # Pages completely freed are added to 'self.free_pages', and - # become available for reuse by any size class. Pages not - # completely freed are re-chained either in + # Pages completely freed are added to 'page.arena.freepages', + # and become available for reuse by any size class. Pages + # not completely freed are re-chained either in # 'full_page_for_size[]' or 'page_for_size[]'. - self.mass_free_in_page(size_class, ok_to_free_func) + self.mass_free_in_pages(size_class, ok_to_free_func) # size_class -= 1 + # + # Rehash arenas into the correct arenas_lists[i]. If + # 'self.current_arena' contains an arena too, it remains there. + (self.old_arenas_lists, self.arenas_lists) = ( + self.arenas_lists, self.old_arenas_lists) + # + i = 0 + while i < self.max_pages_per_arena: + self.arenas_lists[i] = ARENA_NULL + i += 1 + # + i = 0 + while i < self.max_pages_per_arena: + arena = self.old_arenas_lists[i] + while arena != ARENA_NULL: + nextarena = arena.nextarena + # + if arena.nfreepages == arena.totalpages: + # + # The whole arena is empty. Free it. + llarena.arena_free(arena.base) + lltype.free(arena, flavor='raw') + # + else: + # Insert 'arena' in the correct arenas_lists[n] + n = arena.nfreepages + ll_assert(n < self.max_pages_per_arena, + "totalpages != nfreepages >= max_pages_per_arena") + arena.nextarena = self.arenas_lists[n] + self.arenas_lists[n] = arena + # + arena = nextarena + i += 1 + # + self.min_empty_nfreepages = 1 - def mass_free_in_page(self, size_class, ok_to_free_func): + def mass_free_in_pages(self, size_class, ok_to_free_func): nblocks = self.nblocks_for_size[size_class] block_size = size_class * WORD remaining_partial_pages = PAGE_NULL @@ -220,8 +364,7 @@ while page != PAGE_NULL: # # Collect the page. - surviving = self.walk_page(page, block_size, - nblocks, ok_to_free_func) + surviving = self.walk_page(page, block_size, ok_to_free_func) nextpage = page.nextpage # if surviving == nblocks: @@ -255,19 +398,23 @@ def free_page(self, page): """Free a whole page.""" # - # Done by inserting it in the 'free_pages' list. + # Insert the freed page in the arena's 'freepages' list. + # If nfreepages == totalpages, then it will be freed at the + # end of mass_free(). + arena = page.arena + arena.nfreepages += 1 pageaddr = llmemory.cast_ptr_to_adr(page) pageaddr = llarena.getfakearenaaddress(pageaddr) llarena.arena_reset(pageaddr, self.page_size, 0) llarena.arena_reserve(pageaddr, llmemory.sizeof(llmemory.Address)) - pageaddr.address[0] = self.free_pages - self.free_pages = pageaddr + pageaddr.address[0] = arena.freepages + arena.freepages = pageaddr - def walk_page(self, page, block_size, nblocks, ok_to_free_func): + def walk_page(self, page, block_size, ok_to_free_func): """Walk over all objects in a page, and ask ok_to_free_func().""" # - # 'freeblock' is the next free block, or NULL if there isn't any more. + # 'freeblock' is the next free block freeblock = page.freeblock # # 'prevfreeblockat' is the address of where 'freeblock' was read from. @@ -277,22 +424,28 @@ obj = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) obj += self.hdrsize surviving = 0 # initially + skip_free_blocks = page.nfree # - nblocks -= page.nuninitialized - index = nblocks - while index > 0: + while True: # if obj == freeblock: # + if skip_free_blocks == 0: + # + # 'obj' points to the first uninitialized block, + # or to the end of the page if there are none. + break + # # 'obj' points to a free block. It means that # 'prevfreeblockat.address[0]' does not need to be updated. # Just read the next free block from 'obj.address[0]'. + skip_free_blocks -= 1 prevfreeblockat = obj freeblock = obj.address[0] # else: # 'obj' points to a valid object. - ll_assert(not freeblock or freeblock > obj, + ll_assert(freeblock > obj, "freeblocks are linked out of order") # if ok_to_free_func(obj): @@ -306,15 +459,14 @@ prevfreeblockat = obj obj.address[0] = freeblock # + # Update the number of free objects in the page. + page.nfree += 1 + # else: # The object survives. surviving += 1 # obj += block_size - index -= 1 - # - # Update the number of free objects in the page. - page.nfree = nblocks - surviving # # Update the global total size of objects. self.total_memory_used += surviving * block_size @@ -323,6 +475,20 @@ return surviving + def _nuninitialized(self, page, size_class): + # Helper for debugging: count the number of uninitialized blocks + freeblock = page.freeblock + for i in range(page.nfree): + freeblock = freeblock.address[0] + assert freeblock != NULL + pageaddr = llarena.getfakearenaaddress(llmemory.cast_ptr_to_adr(page)) + num_initialized_blocks, rem = divmod( + freeblock - pageaddr - self.hdrsize, size_class * WORD) + assert rem == 0, "page size_class misspecified?" + nblocks = self.nblocks_for_size[size_class] + return nblocks - num_initialized_blocks + + # ____________________________________________________________ # Helpers to go from a pointer to the start of its page @@ -336,7 +502,7 @@ def _start_of_page_untranslated(addr, page_size): assert isinstance(addr, llarena.fakearenaaddress) - shift = 4 # for testing, we assume that the whole arena is not + shift = WORD # for testing, we assume that the whole arena is not # on a page boundary ofs = ((addr.offset - shift) // page_size) * page_size + shift return llarena.fakearenaaddress(addr.arena, ofs) Modified: pypy/branch/jitffi/pypy/rpython/memory/gc/semispace.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gc/semispace.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/semispace.py Fri Oct 8 13:40:10 2010 @@ -1,7 +1,6 @@ from pypy.rpython.lltypesystem.llmemory import raw_malloc, raw_free from pypy.rpython.lltypesystem.llmemory import raw_memcopy, raw_memclear from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage -from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE from pypy.rpython.memory.support import get_address_stack, get_address_deque from pypy.rpython.memory.support import AddressDict from pypy.rpython.lltypesystem import lltype, llmemory, llarena, rffi, llgroup @@ -59,11 +58,11 @@ # translating to a real backend. TRANSLATION_PARAMS = {'space_size': 8*1024*1024} # XXX adjust - def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, space_size=4096, - max_space_size=sys.maxint//2+1): + def __init__(self, config, space_size=4096, max_space_size=sys.maxint//2+1, + **kwds): self.param_space_size = space_size self.param_max_space_size = max_space_size - MovingGCBase.__init__(self, config, chunk_size) + MovingGCBase.__init__(self, config, **kwds) def setup(self): #self.total_collection_time = 0.0 Modified: pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_direct.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_direct.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_direct.py Fri Oct 8 13:40:10 2010 @@ -70,6 +70,7 @@ GC_PARAMS = self.GC_PARAMS.copy() if hasattr(meth, 'GC_PARAMS'): GC_PARAMS.update(meth.GC_PARAMS) + GC_PARAMS['translated_to_c'] = False self.gc = self.GCClass(config, **GC_PARAMS) self.gc.DEBUG = True self.rootwalker = DirectRootWalker(self) @@ -507,8 +508,7 @@ for index, expected_x in nums.items(): assert a[index].x == expected_x self.stackroots.pop() - test_card_marker.GC_PARAMS = {"card_page_indices": 4, - "card_page_indices_min": 7} + test_card_marker.GC_PARAMS = {"card_page_indices": 4} class TestMiniMarkGCFull(DirectGCTest): from pypy.rpython.memory.gc.minimark import MiniMarkGC as GCClass Modified: pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_minimark.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_minimark.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_minimark.py Fri Oct 8 13:40:10 2010 @@ -5,26 +5,6 @@ # Note that most tests are in test_direct.py. -def test_stringbuilder_default_initsize_is_small(): - # Check that pypy.rlib.rstring.INIT_SIZE is short enough to let - # the allocated object be considered as a "small" object. - # Otherwise it would not be allocated in the nursery at all, - # which is kind of bad (and also prevents shrink_array() from - # being useful). - from pypy.rlib.rstring import INIT_SIZE - from pypy.rpython.lltypesystem.rstr import STR, UNICODE - # - size_gc_header = llmemory.raw_malloc_usage( - llmemory.sizeof(llmemory.Address)) - # - size1 = llmemory.raw_malloc_usage(llmemory.sizeof(STR, INIT_SIZE)) - size1 = size_gc_header + size1 - assert size1 <= MiniMarkGC.TRANSLATION_PARAMS["small_request_threshold"] - # - size2 = llmemory.raw_malloc_usage(llmemory.sizeof(UNICODE, INIT_SIZE)) - size2 = size_gc_header + size2 - assert size2 <= MiniMarkGC.TRANSLATION_PARAMS["small_request_threshold"] - def test_card_marking_words_for_length(): gc = MiniMarkGC(None, card_page_indices=128) assert gc.card_page_shift == 7 Modified: pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_minimarkpage.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_minimarkpage.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/test/test_minimarkpage.py Fri Oct 8 13:40:10 2010 @@ -7,22 +7,24 @@ from pypy.rpython.lltypesystem.llmemory import cast_ptr_to_adr NULL = llmemory.NULL -SHIFT = 4 +SHIFT = WORD hdrsize = llmemory.raw_malloc_usage(llmemory.sizeof(PAGE_HEADER)) def test_allocate_arena(): - ac = ArenaCollection(SHIFT + 8*20, 8, 1) + ac = ArenaCollection(SHIFT + 64*20, 64, 1) ac.allocate_new_arena() assert ac.num_uninitialized_pages == 20 - ac.uninitialized_pages + 8*20 # does not raise - py.test.raises(llarena.ArenaError, "ac.uninitialized_pages + 8*20 + 1") + upages = ac.current_arena.freepages + upages + 64*20 # does not raise + py.test.raises(llarena.ArenaError, "upages + 64*20 + 1") # - ac = ArenaCollection(SHIFT + 8*20 + 7, 8, 1) + ac = ArenaCollection(SHIFT + 64*20 + 7, 64, 1) ac.allocate_new_arena() assert ac.num_uninitialized_pages == 20 - ac.uninitialized_pages + 8*20 + 7 # does not raise - py.test.raises(llarena.ArenaError, "ac.uninitialized_pages + 8*20 + 8") + upages = ac.current_arena.freepages + upages + 64*20 + 7 # does not raise + py.test.raises(llarena.ArenaError, "upages + 64*20 + 64") def test_allocate_new_page(): @@ -31,7 +33,8 @@ # def checknewpage(page, size_class): size = WORD * size_class - assert page.nuninitialized == (pagesize - hdrsize) // size + assert (ac._nuninitialized(page, size_class) == + (pagesize - hdrsize) // size) assert page.nfree == 0 page1 = page.freeblock - hdrsize assert llmemory.cast_ptr_to_adr(page) == page1 @@ -44,13 +47,13 @@ page = ac.allocate_new_page(5) checknewpage(page, 5) assert ac.num_uninitialized_pages == 2 - assert ac.uninitialized_pages - pagesize == cast_ptr_to_adr(page) + assert ac.current_arena.freepages - pagesize == cast_ptr_to_adr(page) assert ac.page_for_size[5] == page # page = ac.allocate_new_page(3) checknewpage(page, 3) assert ac.num_uninitialized_pages == 1 - assert ac.uninitialized_pages - pagesize == cast_ptr_to_adr(page) + assert ac.current_arena.freepages - pagesize == cast_ptr_to_adr(page) assert ac.page_for_size[3] == page # page = ac.allocate_new_page(4) @@ -71,17 +74,17 @@ page = llmemory.cast_adr_to_ptr(pageaddr, PAGE_PTR) if step == 1: page.nfree = 0 - page.nuninitialized = nblocks - nusedblocks + nuninitialized = nblocks - nusedblocks else: page.nfree = nusedblocks - page.nuninitialized = nblocks - 2*nusedblocks + nuninitialized = nblocks - 2*nusedblocks + page.freeblock = pageaddr + hdrsize + nusedblocks * size_block if nusedblocks < nblocks: - page.freeblock = pageaddr + hdrsize + nusedblocks * size_block chainedlists = ac.page_for_size else: - page.freeblock = NULL chainedlists = ac.full_page_for_size page.nextpage = chainedlists[size_class] + page.arena = ac.current_arena chainedlists[size_class] = page if fill_with_objects: for i in range(0, nusedblocks*step, step): @@ -98,11 +101,15 @@ prev = 'prevhole.address[0]' endaddr = pageaddr + hdrsize + 2*nusedblocks * size_block exec '%s = endaddr' % prev in globals(), locals() + assert ac._nuninitialized(page, size_class) == nuninitialized # ac.allocate_new_arena() num_initialized_pages = len(pagelayout.rstrip(" ")) - ac._startpageaddr = ac.uninitialized_pages - ac.uninitialized_pages += pagesize * num_initialized_pages + ac._startpageaddr = ac.current_arena.freepages + if pagelayout.endswith(" "): + ac.current_arena.freepages += pagesize * num_initialized_pages + else: + ac.current_arena.freepages = NULL ac.num_uninitialized_pages -= num_initialized_pages # for i in reversed(range(num_initialized_pages)): @@ -115,8 +122,9 @@ link(pageaddr, size_class, size_block, nblocks, nblocks-1) elif c == '.': # a free, but initialized, page llarena.arena_reserve(pageaddr, llmemory.sizeof(llmemory.Address)) - pageaddr.address[0] = ac.free_pages - ac.free_pages = pageaddr + pageaddr.address[0] = ac.current_arena.freepages + ac.current_arena.freepages = pageaddr + ac.current_arena.nfreepages += 1 elif c == '#': # a random full page, in the list 'full_pages' size_class = fill_with_objects or 1 size_block = WORD * size_class @@ -142,26 +150,29 @@ def checkpage(ac, page, expected_position): assert llmemory.cast_ptr_to_adr(page) == pagenum(ac, expected_position) +def freepages(ac): + return ac.current_arena.freepages + def test_simple_arena_collection(): pagesize = hdrsize + 16 ac = arena_collection_for_test(pagesize, "##....# ") # - assert ac.free_pages == pagenum(ac, 2) + assert freepages(ac) == pagenum(ac, 2) page = ac.allocate_new_page(1); checkpage(ac, page, 2) - assert ac.free_pages == pagenum(ac, 3) + assert freepages(ac) == pagenum(ac, 3) page = ac.allocate_new_page(2); checkpage(ac, page, 3) - assert ac.free_pages == pagenum(ac, 4) + assert freepages(ac) == pagenum(ac, 4) page = ac.allocate_new_page(3); checkpage(ac, page, 4) - assert ac.free_pages == pagenum(ac, 5) + assert freepages(ac) == pagenum(ac, 5) page = ac.allocate_new_page(4); checkpage(ac, page, 5) - assert ac.free_pages == NULL and ac.num_uninitialized_pages == 3 + assert freepages(ac) == pagenum(ac, 7) and ac.num_uninitialized_pages == 3 page = ac.allocate_new_page(5); checkpage(ac, page, 7) - assert ac.free_pages == NULL and ac.num_uninitialized_pages == 2 + assert freepages(ac) == pagenum(ac, 8) and ac.num_uninitialized_pages == 2 page = ac.allocate_new_page(6); checkpage(ac, page, 8) - assert ac.free_pages == NULL and ac.num_uninitialized_pages == 1 + assert freepages(ac) == pagenum(ac, 9) and ac.num_uninitialized_pages == 1 page = ac.allocate_new_page(7); checkpage(ac, page, 9) - assert ac.free_pages == NULL and ac.num_uninitialized_pages == 0 + assert not ac.current_arena and ac.num_uninitialized_pages == 0 def chkob(ac, num_page, pos_obj, obj): @@ -205,47 +216,47 @@ ac = arena_collection_for_test(pagesize, "/.", fill_with_objects=2) page = getpage(ac, 0) assert page.nfree == 3 - assert page.nuninitialized == 3 + assert ac._nuninitialized(page, 2) == 3 chkob(ac, 0, 2*WORD, page.freeblock) # obj = ac.malloc(2*WORD); chkob(ac, 0, 2*WORD, obj) obj = ac.malloc(2*WORD); chkob(ac, 0, 6*WORD, obj) assert page.nfree == 1 - assert page.nuninitialized == 3 + assert ac._nuninitialized(page, 2) == 3 chkob(ac, 0, 10*WORD, page.freeblock) # obj = ac.malloc(2*WORD); chkob(ac, 0, 10*WORD, obj) assert page.nfree == 0 - assert page.nuninitialized == 3 + assert ac._nuninitialized(page, 2) == 3 chkob(ac, 0, 12*WORD, page.freeblock) # obj = ac.malloc(2*WORD); chkob(ac, 0, 12*WORD, obj) - assert page.nuninitialized == 2 + assert ac._nuninitialized(page, 2) == 2 obj = ac.malloc(2*WORD); chkob(ac, 0, 14*WORD, obj) obj = ac.malloc(2*WORD); chkob(ac, 0, 16*WORD, obj) assert page.nfree == 0 - assert page.nuninitialized == 0 + assert ac._nuninitialized(page, 2) == 0 obj = ac.malloc(2*WORD); chkob(ac, 1, 0*WORD, obj) def test_malloc_new_arena(): pagesize = hdrsize + 7*WORD ac = arena_collection_for_test(pagesize, "### ") + arena_size = ac.arena_size obj = ac.malloc(2*WORD); chkob(ac, 3, 0*WORD, obj) # 3rd page -> size 2 # del ac.allocate_new_arena # restore the one from the class - arena_size = ac.arena_size obj = ac.malloc(3*WORD) # need a new arena assert ac.num_uninitialized_pages == (arena_size // ac.page_size - - 1 # for start_of_page() - 1 # the just-allocated page ) class OkToFree(object): - def __init__(self, ac, answer): + def __init__(self, ac, answer, multiarenas=False): assert callable(answer) or 0.0 <= answer <= 1.0 self.ac = ac self.answer = answer + self.multiarenas = multiarenas self.lastnum = 0.0 self.seen = {} @@ -257,7 +268,10 @@ ok_to_free = self.lastnum >= 1.0 if ok_to_free: self.lastnum -= 1.0 - key = addr - self.ac._startpageaddr + if self.multiarenas: + key = (addr.arena, addr.offset) + else: + key = addr - self.ac._startpageaddr assert key not in self.seen self.seen[key] = ok_to_free return ok_to_free @@ -272,10 +286,10 @@ page = getpage(ac, 0) assert page == ac.page_for_size[2] assert page.nextpage == PAGE_NULL - assert page.nuninitialized == 1 + assert ac._nuninitialized(page, 2) == 1 assert page.nfree == 0 chkob(ac, 0, 4*WORD, page.freeblock) - assert ac.free_pages == NULL + assert freepages(ac) == NULL def test_mass_free_emptied_page(): pagesize = hdrsize + 7*WORD @@ -285,7 +299,7 @@ assert ok_to_free.seen == {hdrsize + 0*WORD: True, hdrsize + 2*WORD: True} pageaddr = pagenum(ac, 0) - assert pageaddr == ac.free_pages + assert pageaddr == freepages(ac) assert pageaddr.address[0] == NULL assert ac.page_for_size[2] == PAGE_NULL @@ -300,10 +314,9 @@ page = getpage(ac, 0) assert page == ac.full_page_for_size[2] assert page.nextpage == PAGE_NULL - assert page.nuninitialized == 0 + assert ac._nuninitialized(page, 2) == 0 assert page.nfree == 0 - assert page.freeblock == NULL - assert ac.free_pages == NULL + assert freepages(ac) == NULL assert ac.page_for_size[2] == PAGE_NULL def test_mass_free_full_is_partially_emptied(): @@ -319,19 +332,19 @@ pageaddr = pagenum(ac, 0) assert page == ac.page_for_size[2] assert page.nextpage == PAGE_NULL - assert page.nuninitialized == 0 + assert ac._nuninitialized(page, 2) == 0 assert page.nfree == 2 assert page.freeblock == pageaddr + hdrsize + 2*WORD assert page.freeblock.address[0] == pageaddr + hdrsize + 6*WORD - assert page.freeblock.address[0].address[0] == NULL - assert ac.free_pages == NULL + assert page.freeblock.address[0].address[0] == pageaddr + hdrsize + 8*WORD + assert freepages(ac) == NULL assert ac.full_page_for_size[2] == PAGE_NULL def test_mass_free_half_page_remains(): pagesize = hdrsize + 24*WORD ac = arena_collection_for_test(pagesize, "/", fill_with_objects=2) page = getpage(ac, 0) - assert page.nuninitialized == 4 + assert ac._nuninitialized(page, 2) == 4 assert page.nfree == 4 # ok_to_free = OkToFree(ac, False) @@ -344,7 +357,7 @@ pageaddr = pagenum(ac, 0) assert page == ac.page_for_size[2] assert page.nextpage == PAGE_NULL - assert page.nuninitialized == 4 + assert ac._nuninitialized(page, 2) == 4 assert page.nfree == 4 assert page.freeblock == pageaddr + hdrsize + 2*WORD assert page.freeblock.address[0] == pageaddr + hdrsize + 6*WORD @@ -352,14 +365,14 @@ pageaddr + hdrsize + 10*WORD assert page.freeblock.address[0].address[0].address[0] == \ pageaddr + hdrsize + 14*WORD - assert ac.free_pages == NULL + assert freepages(ac) == NULL assert ac.full_page_for_size[2] == PAGE_NULL def test_mass_free_half_page_becomes_more_free(): pagesize = hdrsize + 24*WORD ac = arena_collection_for_test(pagesize, "/", fill_with_objects=2) page = getpage(ac, 0) - assert page.nuninitialized == 4 + assert ac._nuninitialized(page, 2) == 4 assert page.nfree == 4 # ok_to_free = OkToFree(ac, 0.5) @@ -372,7 +385,7 @@ pageaddr = pagenum(ac, 0) assert page == ac.page_for_size[2] assert page.nextpage == PAGE_NULL - assert page.nuninitialized == 4 + assert ac._nuninitialized(page, 2) == 4 assert page.nfree == 6 fb = page.freeblock assert fb == pageaddr + hdrsize + 2*WORD @@ -384,7 +397,7 @@ pageaddr + hdrsize + 12*WORD assert fb.address[0].address[0].address[0].address[0].address[0] == \ pageaddr + hdrsize + 14*WORD - assert ac.free_pages == NULL + assert freepages(ac) == NULL assert ac.full_page_for_size[2] == PAGE_NULL # ____________________________________________________________ @@ -392,17 +405,29 @@ def test_random(): import random pagesize = hdrsize + 24*WORD - num_pages = 28 + num_pages = 3 ac = arena_collection_for_test(pagesize, " " * num_pages) live_objects = {} # - # Run the test until ac.allocate_new_arena() is called. + # Run the test until three arenas are freed. This is a quick test + # that the arenas are really freed by the logic. class DoneTesting(Exception): - pass - def done_testing(): - raise DoneTesting - ac.allocate_new_arena = done_testing - # + counter = 0 + def my_allocate_new_arena(): + # the following output looks cool on a 112-character-wide terminal. + lst = sorted(ac._all_arenas(), key=lambda a: a.base.arena._arena_index) + for a in lst: + print a.base.arena, a.base.arena.usagemap + print '-' * 80 + ac.__class__.allocate_new_arena(ac) + a = ac.current_arena.base.arena + def my_mark_freed(): + a.freed = True + DoneTesting.counter += 1 + if DoneTesting.counter > 3: + raise DoneTesting + a.mark_freed = my_mark_freed + ac.allocate_new_arena = my_allocate_new_arena try: while True: # @@ -410,12 +435,13 @@ for i in range(random.randrange(50, 100)): size_class = random.randrange(1, 7) obj = ac.malloc(size_class * WORD) - at = obj - ac._startpageaddr + at = (obj.arena, obj.offset) assert at not in live_objects live_objects[at] = size_class * WORD # # Free half the objects, randomly - ok_to_free = OkToFree(ac, lambda obj: random.random() < 0.5) + ok_to_free = OkToFree(ac, lambda obj: random.random() < 0.5, + multiarenas=True) ac.mass_free(ok_to_free) # # Check that we have seen all objects @@ -428,5 +454,4 @@ surviving_total_size += live_objects[at] assert ac.total_memory_used == surviving_total_size except DoneTesting: - # the following output looks cool on a 112-character-wide terminal. - print ac._startpageaddr.arena.usagemap + pass Modified: pypy/branch/jitffi/pypy/rpython/memory/gctransform/framework.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gctransform/framework.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gctransform/framework.py Fri Oct 8 13:40:10 2010 @@ -139,7 +139,7 @@ def __init__(self, translator): from pypy.rpython.memory.gc.base import choose_gc_from_config from pypy.rpython.memory.gc.base import ARRAY_TYPEID_MAP - from pypy.rpython.memory.gc import inspect + from pypy.rpython.memory.gc import inspector super(FrameworkGCTransformer, self).__init__(translator, inline=True) if hasattr(self, 'GC_PARAMS'): @@ -391,27 +391,27 @@ else: self.id_ptr = None - self.get_rpy_roots_ptr = getfn(inspect.get_rpy_roots, + self.get_rpy_roots_ptr = getfn(inspector.get_rpy_roots, [s_gc], rgc.s_list_of_gcrefs(), minimal_transform=False) - self.get_rpy_referents_ptr = getfn(inspect.get_rpy_referents, + self.get_rpy_referents_ptr = getfn(inspector.get_rpy_referents, [s_gc, s_gcref], rgc.s_list_of_gcrefs(), minimal_transform=False) - self.get_rpy_memory_usage_ptr = getfn(inspect.get_rpy_memory_usage, + self.get_rpy_memory_usage_ptr = getfn(inspector.get_rpy_memory_usage, [s_gc, s_gcref], annmodel.SomeInteger(), minimal_transform=False) - self.get_rpy_type_index_ptr = getfn(inspect.get_rpy_type_index, + self.get_rpy_type_index_ptr = getfn(inspector.get_rpy_type_index, [s_gc, s_gcref], annmodel.SomeInteger(), minimal_transform=False) - self.is_rpy_instance_ptr = getfn(inspect.is_rpy_instance, + self.is_rpy_instance_ptr = getfn(inspector.is_rpy_instance, [s_gc, s_gcref], annmodel.SomeBool(), minimal_transform=False) - self.dump_rpy_heap_ptr = getfn(inspect.dump_rpy_heap, + self.dump_rpy_heap_ptr = getfn(inspector.dump_rpy_heap, [s_gc, annmodel.SomeInteger()], annmodel.s_Bool, minimal_transform=False) @@ -455,8 +455,7 @@ self.write_barrier_from_array_failing_case_ptr = \ getfn(func, [annmodel.SomeAddress(), - annmodel.SomeInteger(), - annmodel.SomeAddress()], + annmodel.SomeInteger()], annmodel.s_None) self.statistics_ptr = getfn(GCClass.statistics.im_func, [s_gc, annmodel.SomeInteger()], @@ -610,8 +609,10 @@ if self.write_barrier_ptr: self.clean_sets = ( - find_clean_setarrayitems(self.collect_analyzer, graph).union( - find_initializing_stores(self.collect_analyzer, graph))) + find_initializing_stores(self.collect_analyzer, graph)) + if self.gcdata.gc.can_optimize_clean_setarrayitems(): + self.clean_sets = self.clean_sets.union( + find_clean_setarrayitems(self.collect_analyzer, graph)) super(FrameworkGCTransformer, self).transform_graph(graph) if self.write_barrier_ptr: self.clean_sets = None Modified: pypy/branch/jitffi/pypy/rpython/memory/gcwrapper.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/gcwrapper.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/gcwrapper.py Fri Oct 8 13:40:10 2010 @@ -9,7 +9,10 @@ def __init__(self, llinterp, flowgraphs, gc_class, GC_PARAMS={}): translator = llinterp.typer.annotator.translator config = translator.config.translation - self.gc = gc_class(config, chunk_size = 10, **GC_PARAMS) + self.gc = gc_class(config, + chunk_size = 10, + translated_to_c = False, + **GC_PARAMS) self.gc.set_root_walker(LLInterpRootWalker(self)) self.gc.DEBUG = True self.llinterp = llinterp Modified: pypy/branch/jitffi/pypy/rpython/memory/test/test_gc.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/test/test_gc.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/test/test_gc.py Fri Oct 8 13:40:10 2010 @@ -29,6 +29,7 @@ GC_CAN_MALLOC_NONMOVABLE = True GC_CAN_SHRINK_ARRAY = False GC_CAN_SHRINK_BIG_ARRAY = False + BUT_HOW_BIG_IS_A_BIG_STRING = 3*WORD def setup_class(cls): cls._saved_logstate = py.log._getstate() @@ -495,7 +496,8 @@ # with larger numbers, it gets allocated outside the semispace # with some GCs. flag = self.GC_CAN_SHRINK_BIG_ARRAY - assert self.interpret(f, [12, 0, flag]) == 0x62024241 + bigsize = self.BUT_HOW_BIG_IS_A_BIG_STRING + assert self.interpret(f, [bigsize, 0, flag]) == 0x62024241 def test_tagged_simple(self): from pypy.rlib.objectmodel import UnboxedValue @@ -770,7 +772,7 @@ from pypy.rpython.memory.gc.minimark import MiniMarkGC as GCClass GC_CAN_SHRINK_BIG_ARRAY = False GC_CAN_MALLOC_NONMOVABLE = True + BUT_HOW_BIG_IS_A_BIG_STRING = 11*WORD class TestMiniMarkGCCardMarking(TestMiniMarkGC): - GC_PARAMS = {'card_page_indices': 4, - 'card_page_indices_min': 10} + GC_PARAMS = {'card_page_indices': 4} Modified: pypy/branch/jitffi/pypy/rpython/memory/test/test_transformed_gc.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/memory/test/test_transformed_gc.py (original) +++ pypy/branch/jitffi/pypy/rpython/memory/test/test_transformed_gc.py Fri Oct 8 13:40:10 2010 @@ -906,7 +906,8 @@ gcname = "marksweep" class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 @@ -1144,7 +1145,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.marksweep import PrintingMarkSweepGC as GCClass - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestSemiSpaceGC(GenericMovingGCTests): @@ -1154,7 +1156,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.semispace import SemiSpaceGC as GCClass - GC_PARAMS = {'space_size': 512*WORD} + GC_PARAMS = {'space_size': 512*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestMarkCompactGC(GenericMovingGCTests): @@ -1163,7 +1166,8 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.markcompact import MarkCompactGC as GCClass - GC_PARAMS = {'space_size': 4096*WORD} + GC_PARAMS = {'space_size': 4096*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestGenerationGC(GenericMovingGCTests): @@ -1175,7 +1179,8 @@ from pypy.rpython.memory.gc.generation import GenerationGC as \ GCClass GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 32*WORD} + 'nursery_size': 32*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_weakref_across_minor_collection(cls): @@ -1372,7 +1377,8 @@ GenerationGC._teardown(self) GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 128*WORD} + 'nursery_size': 128*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_working_nursery(cls): @@ -1404,7 +1410,8 @@ from pypy.rpython.memory.gc.hybrid import HybridGC as GCClass GC_PARAMS = {'space_size': 512*WORD, 'nursery_size': 32*WORD, - 'large_object': 8*WORD} + 'large_object': 8*WORD, + 'translated_to_c': False} root_stack_depth = 200 def define_ref_from_rawmalloced_to_regular(cls): @@ -1474,11 +1481,38 @@ 'page_size': 16*WORD, 'arena_size': 64*WORD, 'small_request_threshold': 5*WORD, + 'large_object': 8*WORD, + 'large_object_gcptrs': 10*WORD, 'card_page_indices': 4, - 'card_page_indices_min': 10, + 'translated_to_c': False, } root_stack_depth = 200 + def define_no_clean_setarrayitems(cls): + # The optimization find_clean_setarrayitems() in + # gctransformer/framework.py does not work with card marking. + # Check that it is turned off. + S = lltype.GcStruct('S', ('x', lltype.Signed)) + A = lltype.GcArray(lltype.Ptr(S)) + def sub(lst): + lst[15] = lltype.malloc(S) # 'lst' is set the single mark "12-15" + lst[15].x = 123 + lst[0] = lst[15] # that would be a "clean_setarrayitem" + def f(): + lst = lltype.malloc(A, 16) # 16 > 10 + rgc.collect() + sub(lst) + null = lltype.nullptr(S) + lst[15] = null # clear, so that A() is only visible via lst[0] + rgc.collect() # -> crash + return lst[0].x + return f + + def test_no_clean_setarrayitems(self): + run = self.runner("no_clean_setarrayitems") + res = run([]) + assert res == 123 + # ________________________________________________________________ # tagged pointers @@ -1559,7 +1593,8 @@ gcname = "marksweep" class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): - GC_PARAMS = {'start_heap_size': 1024*WORD } + GC_PARAMS = {'start_heap_size': 1024*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestHybridTaggedPointerGC(TaggedPointerGCTests): @@ -1570,7 +1605,8 @@ from pypy.rpython.memory.gc.generation import GenerationGC as \ GCClass GC_PARAMS = {'space_size': 512*WORD, - 'nursery_size': 32*WORD} + 'nursery_size': 32*WORD, + 'translated_to_c': False} root_stack_depth = 200 class TestMarkCompactTaggedpointerGC(TaggedPointerGCTests): @@ -1579,5 +1615,6 @@ class gcpolicy(gc.FrameworkGcPolicy): class transformerclass(framework.FrameworkGCTransformer): from pypy.rpython.memory.gc.markcompact import MarkCompactGC as GCClass - GC_PARAMS = {'space_size': 4096*WORD} + GC_PARAMS = {'space_size': 4096*WORD, + 'translated_to_c': False} root_stack_depth = 200 Modified: pypy/branch/jitffi/pypy/rpython/rtyper.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/rtyper.py (original) +++ pypy/branch/jitffi/pypy/rpython/rtyper.py Fri Oct 8 13:40:10 2010 @@ -421,7 +421,7 @@ assert noexclink.exitcase is None if pos == "removed": # the exception cannot actually occur at all. - # See for example rspecialcase.rtype_call_specialcase(). + # This is set by calling exception_cannot_occur(). # We just remove all exception links. block.exitswitch = None block.exits = block.exits[:1] @@ -1019,7 +1019,7 @@ from pypy.rpython import rint, rbool, rfloat from pypy.rpython import rrange from pypy.rpython import rstr, rdict, rlist -from pypy.rpython import rclass, rbuiltin, rpbc, rspecialcase +from pypy.rpython import rclass, rbuiltin, rpbc from pypy.rpython import rexternalobj from pypy.rpython import rptr from pypy.rpython import rgeneric Modified: pypy/branch/jitffi/pypy/tool/progressbar.py ============================================================================== --- pypy/branch/jitffi/pypy/tool/progressbar.py (original) +++ pypy/branch/jitffi/pypy/tool/progressbar.py Fri Oct 8 13:40:10 2010 @@ -17,11 +17,11 @@ ) PADDING = 7 - def __init__(self, color=None, width=None, block='?', empty=' '): + def __init__(self, color=None, width=None, block='.', empty=' '): """ color -- color name (BLUE GREEN CYAN RED MAGENTA YELLOW WHITE BLACK) width -- bar width (optinal) - block -- progress display character (default '?') + block -- progress display character (default '.') empty -- bar display character (default ' ') """ if color: Modified: pypy/branch/jitffi/pypy/translator/c/gcc/instruction.py ============================================================================== --- pypy/branch/jitffi/pypy/translator/c/gcc/instruction.py (original) +++ pypy/branch/jitffi/pypy/translator/c/gcc/instruction.py Fri Oct 8 13:40:10 2010 @@ -82,6 +82,11 @@ def all_sources_of(self, localvar): return [localvar] +class InsnCondJump(Insn): # only for debugging; not used internally + _args_ = ['label'] + def __init__(self, label): + self.label = label + class Label(Insn): _args_ = ['label', 'lineno'] def __init__(self, label, lineno): @@ -170,9 +175,12 @@ self.delta = -7 # use an odd value as marker class InsnStop(Insn): - pass + _args_ = ['reason'] + def __init__(self, reason='?'): + self.reason = reason class InsnRet(InsnStop): + _args_ = [] framesize = 0 def __init__(self, registers): self.registers = registers Modified: pypy/branch/jitffi/pypy/translator/c/gcc/trackgcroot.py ============================================================================== --- pypy/branch/jitffi/pypy/translator/c/gcc/trackgcroot.py (original) +++ pypy/branch/jitffi/pypy/translator/c/gcc/trackgcroot.py Fri Oct 8 13:40:10 2010 @@ -6,7 +6,7 @@ from pypy.translator.c.gcc.instruction import InsnFunctionStart, InsnStop from pypy.translator.c.gcc.instruction import InsnSetLocal, InsnCopyLocal from pypy.translator.c.gcc.instruction import InsnPrologue, InsnEpilogue -from pypy.translator.c.gcc.instruction import InsnGCROOT +from pypy.translator.c.gcc.instruction import InsnGCROOT, InsnCondJump from pypy.translator.c.gcc.instruction import InsnStackAdjust from pypy.translator.c.gcc.instruction import InsnCannotFollowEsp from pypy.translator.c.gcc.instruction import LocalVar, somenewvalue @@ -123,18 +123,15 @@ self.labels[label] = Label(label, lineno) def find_noncollecting_calls(self): - cannot_collect = self.CANNOT_COLLECT.copy() + cannot_collect = {} for line in self.lines: match = self.r_gcnocollect_marker.search(line) if match: name = match.group(1) cannot_collect[name] = True # - if self.format in ('darwin', 'mingw32', 'msvc'): - self.cannot_collect = dict.fromkeys( - ['_' + name for name in cannot_collect]) - else: - self.cannot_collect = cannot_collect + self.cannot_collect = dict.fromkeys( + [self.function_names_prefix + name for name in cannot_collect]) def append_instruction(self, insn): # Add the instruction to the list, and link it to the previous one. @@ -410,7 +407,8 @@ return result # ____________________________________________________________ - CANNOT_COLLECT = { # some of the most used functions that cannot collect + BASE_FUNCTIONS_NOT_RETURNING = { + 'abort': None, 'pypy_debug_catch_fatal_exception': None, 'RPyAbort': None, 'RPyAssertFailed': None, @@ -644,7 +642,7 @@ if label != '0': self.register_jump_to(label) tablelin += 1 - return InsnStop() + return InsnStop("jump table") if self.r_unaryinsn_star.match(line): # that looks like an indirect tail-call. # tail-calls are equivalent to RET for us @@ -658,7 +656,7 @@ assert not target.startswith('.') # tail-calls are equivalent to RET for us return InsnRet(self.CALLEE_SAVE_REGISTERS) - return InsnStop() + return InsnStop("jump") def register_jump_to(self, label): if not isinstance(self.insns[-1], InsnStop): @@ -682,7 +680,7 @@ else: label = match.group(1) self.register_jump_to(label) - return [] + return [InsnCondJump(label)] visit_jmpl = visit_jmp visit_je = conditional_jump @@ -754,7 +752,7 @@ target, = sources if target in self.FUNCTIONS_NOT_RETURNING: - return [InsnStop(), InsnCannotFollowEsp()] + return [InsnStop(target), InsnCannotFollowEsp()] if self.format == 'mingw32' and target == '__alloca': # in functions with large stack requirements, windows # needs a call to _alloca(), to turn reserved pages @@ -856,7 +854,6 @@ visit_and = FunctionGcRootTracker._visit_and visit_xchgl = FunctionGcRootTracker._visit_xchg - visit_xchgq = FunctionGcRootTracker._visit_xchg # used in "xor reg, reg" to create a NULL GC ptr visit_xorl = FunctionGcRootTracker.binary_insn @@ -886,7 +883,7 @@ # statically known pointer to a register # %eax -> %rax - new_line = re.sub(r"%e(ax|bx|cx|dx|di|si)$", r"%r\1", line) + new_line = re.sub(r"%e(ax|bx|cx|dx|di|si|bp)$", r"%r\1", line) # %r10d -> %r10 new_line = re.sub(r"%r(\d+)d$", r"%r\1", new_line) return func(self, new_line) @@ -914,6 +911,7 @@ visit_leaq = FunctionGcRootTracker._visit_lea visit_xorq = FunctionGcRootTracker.binary_insn + visit_xchgq = FunctionGcRootTracker._visit_xchg # FIXME: similar to visit_popl for 32-bit def visit_popq(self, line): @@ -951,6 +949,7 @@ class ElfFunctionGcRootTracker32(FunctionGcRootTracker32): format = 'elf' + function_names_prefix = '' ESP = '%esp' EBP = '%ebp' @@ -984,13 +983,14 @@ r_bottom_marker = re.compile(r"\t/[*] GC_STACK_BOTTOM [*]/") FUNCTIONS_NOT_RETURNING = { - 'abort': None, '_exit': None, '__assert_fail': None, '___assert_rtn': None, 'L___assert_rtn$stub': None, 'L___eprintf$stub': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING[_name] = None def __init__(self, lines, filetag=0): match = self.r_functionstart.match(lines[0]) @@ -1010,6 +1010,8 @@ class ElfFunctionGcRootTracker64(FunctionGcRootTracker64): format = 'elf64' + function_names_prefix = '' + ESP = '%rsp' EBP = '%rbp' EAX = '%rax' @@ -1042,13 +1044,14 @@ r_bottom_marker = re.compile(r"\t/[*] GC_STACK_BOTTOM [*]/") FUNCTIONS_NOT_RETURNING = { - 'abort': None, '_exit': None, '__assert_fail': None, '___assert_rtn': None, 'L___assert_rtn$stub': None, 'L___eprintf$stub': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING[_name] = None def __init__(self, lines, filetag=0): match = self.r_functionstart.match(lines[0]) @@ -1068,6 +1071,7 @@ class DarwinFunctionGcRootTracker(ElfFunctionGcRootTracker32): format = 'darwin' + function_names_prefix = '_' r_functionstart = re.compile(r"_(\w+):\s*$") OFFSET_LABELS = 0 @@ -1079,15 +1083,19 @@ class Mingw32FunctionGcRootTracker(DarwinFunctionGcRootTracker): format = 'mingw32' + function_names_prefix = '_' FUNCTIONS_NOT_RETURNING = { - '_abort': None, '_exit': None, '__assert': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING['_' + _name] = None class MsvcFunctionGcRootTracker(FunctionGcRootTracker32): format = 'msvc' + function_names_prefix = '_' + ESP = 'esp' EBP = 'ebp' EAX = 'eax' @@ -1127,7 +1135,6 @@ r_bottom_marker = re.compile(r"; .+\tpypy_asm_stack_bottom\(\);") FUNCTIONS_NOT_RETURNING = { - '_abort': None, '__exit': None, '__assert': None, '__wassert': None, @@ -1136,6 +1143,8 @@ 'DWORD PTR __imp__abort': None, 'DWORD PTR __imp___wassert': None, } + for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING: + FUNCTIONS_NOT_RETURNING['_' + _name] = None @classmethod def init_regexp(cls): @@ -1543,9 +1552,7 @@ assert self.seen_main def _globalname(name, disp=""): - if self.format in ('darwin', 'mingw32', 'msvc'): - name = '_' + name - return name + return tracker_cls.function_names_prefix + name def _variant(**kwargs): txt = kwargs[self.format] Modified: pypy/branch/jitffi/pypy/translator/c/test/test_lltyped.py ============================================================================== --- pypy/branch/jitffi/pypy/translator/c/test/test_lltyped.py (original) +++ pypy/branch/jitffi/pypy/translator/c/test/test_lltyped.py Fri Oct 8 13:40:10 2010 @@ -783,6 +783,17 @@ res = fn() assert res == 42 + def test_llarena(self): + from pypy.rpython.lltypesystem import llmemory, llarena + # + def f(): + a = llarena.arena_malloc(800, False) + llarena.arena_reset(a, 800, 2) + llarena.arena_free(a) + # + fn = self.getcompiled(f, []) + fn() + def test_padding_in_prebuilt_struct(self): from pypy.rpython.lltypesystem import rffi from pypy.rpython.tool import rffi_platform Modified: pypy/branch/jitffi/pypy/translator/driver.py ============================================================================== --- pypy/branch/jitffi/pypy/translator/driver.py (original) +++ pypy/branch/jitffi/pypy/translator/driver.py Fri Oct 8 13:40:10 2010 @@ -426,6 +426,22 @@ [OOTYPE], "JIT compiler generation") + def task_jittest_lltype(self): + """ Run with the JIT on top of the llgraph backend + """ + # parent process loop: spawn a child, wait for the child to finish, + # print a message, and restart + from pypy.translator.goal import unixcheckpoint + unixcheckpoint.restartable_point(auto='run') + # load the module pypy/jit/tl/jittest.py, which you can hack at + # and restart without needing to restart the whole translation process + from pypy.jit.tl import jittest + jittest.jittest(self) + # + task_jittest_lltype = taskdef(task_jittest_lltype, + [RTYPE], + "test of the JIT on the llgraph backend") + def task_backendopt_lltype(self): """ Run all backend optimizations - lltype version """ @@ -433,7 +449,8 @@ backend_optimizations(self.translator) # task_backendopt_lltype = taskdef(task_backendopt_lltype, - [RTYPE, '??pyjitpl_lltype'], + [RTYPE, '??pyjitpl_lltype', + '??jittest_lltype'], "lltype back-end optimisations") BACKENDOPT = 'backendopt_lltype' Modified: pypy/branch/jitffi/pypy/translator/goal/ann_override.py ============================================================================== --- pypy/branch/jitffi/pypy/translator/goal/ann_override.py (original) +++ pypy/branch/jitffi/pypy/translator/goal/ann_override.py Fri Oct 8 13:40:10 2010 @@ -27,21 +27,6 @@ pol.pypytypes = {} pol.single_space = single_space - #def override__wrap_exception_cls(pol, space, x): - # import pypy.objspace.std.typeobject as typeobject - # clsdef = getbookkeeper().getuniqueclassdef(typeobject.W_TypeObject) - # return annmodel.SomeInstance(clsdef, can_be_None=True) - # - #def override__fake_object(pol, space, x): - # from pypy.interpreter import typedef - # clsdef = getbookkeeper().getuniqueclassdef(typedef.W_Root) - # return annmodel.SomeInstance(clsdef) - # - #def override__cpy_compile(pol, self, source, filename, mode, flags): - # from pypy.interpreter import pycode - # clsdef = getbookkeeper().getuniqueclassdef(pycode.PyCode) - # return annmodel.SomeInstance(clsdef) - def specialize__wrap(pol, funcdesc, args_s): from pypy.interpreter.baseobjspace import Wrappable from pypy.annotation.classdef import ClassDef Modified: pypy/branch/jitffi/pypy/translator/goal/translate.py ============================================================================== --- pypy/branch/jitffi/pypy/translator/goal/translate.py (original) +++ pypy/branch/jitffi/pypy/translator/goal/translate.py Fri Oct 8 13:40:10 2010 @@ -27,6 +27,7 @@ ("annotate", "do type inference", "-a --annotate", ""), ("rtype", "do rtyping", "-t --rtype", ""), ("pyjitpl", "JIT generation step", "--pyjitpl", ""), + ("jittest", "JIT test with llgraph backend", "--jittest", ""), ("backendopt", "do backend optimizations", "--backendopt", ""), ("source", "create source", "-s --source", ""), ("compile", "compile", "-c --compile", " (default goal)"), Modified: pypy/branch/jitffi/pypy/translator/jvm/test/test_class.py ============================================================================== --- pypy/branch/jitffi/pypy/translator/jvm/test/test_class.py (original) +++ pypy/branch/jitffi/pypy/translator/jvm/test/test_class.py Fri Oct 8 13:40:10 2010 @@ -1,6 +1,6 @@ import py from pypy.translator.jvm.test.runtest import JvmTest -from pypy.translator.oosupport.test_template.class_ import BaseTestClass, BaseTestSpecialcase +from pypy.translator.oosupport.test_template.class_ import BaseTestClass class TestJvmClass(JvmTest, BaseTestClass): def test_overridden_classattr_as_defaults(self): @@ -26,6 +26,3 @@ def test_specialize_methods(self): py.test.skip('ABSTRACT METHOD FIX: RE-TEST AFTER MERGE') - -class TestJvmSpecialCase(JvmTest, BaseTestSpecialcase): - pass Modified: pypy/branch/jitffi/pypy/translator/oosupport/test_template/class_.py ============================================================================== --- pypy/branch/jitffi/pypy/translator/oosupport/test_template/class_.py (original) +++ pypy/branch/jitffi/pypy/translator/oosupport/test_template/class_.py Fri Oct 8 13:40:10 2010 @@ -1,6 +1,5 @@ import py from pypy.rpython.test import test_rclass -from pypy.rpython.test.test_rspecialcase import BaseTestRspecialcase class BaseTestClass(test_rclass.TestOOtype): def test_abstract_method(self): @@ -66,6 +65,3 @@ def test_cast_object_mix_null(self): py.test.skip('cannot return ootype.NULL from translated functions') - -class BaseTestSpecialcase(BaseTestRspecialcase): - pass From antocuni at codespeak.net Fri Oct 8 15:26:15 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 8 Oct 2010 15:26:15 +0200 (CEST) Subject: [pypy-svn] r77722 - pypy/extradoc/talk/pepm2011/benchmarks Message-ID: <20101008132615.399BC282BE8@codespeak.net> Author: antocuni Date: Fri Oct 8 15:26:13 2010 New Revision: 77722 Removed: pypy/extradoc/talk/pepm2011/benchmarks/crypto_pyaes.trace pypy/extradoc/talk/pepm2011/benchmarks/fannkuch.trace Log: bah, these traces just contain jit-log-opt :-( From antocuni at codespeak.net Fri Oct 8 15:39:04 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 8 Oct 2010 15:39:04 +0200 (CEST) Subject: [pypy-svn] r77724 - pypy/extradoc/talk/pepm2011/benchmarks Message-ID: <20101008133904.EA9CD282BE8@codespeak.net> Author: antocuni Date: Fri Oct 8 15:38:06 2010 New Revision: 77724 Removed: pypy/extradoc/talk/pepm2011/benchmarks/crypto_pyaes.py.trace pypy/extradoc/talk/pepm2011/benchmarks/fannkuch.py.trace pypy/extradoc/talk/pepm2011/benchmarks/go.py.trace Log: these are too big. Try another approach to reduce the size From cfbolz at codespeak.net Fri Oct 8 16:00:21 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Fri, 8 Oct 2010 16:00:21 +0200 (CEST) Subject: [pypy-svn] r77725 - pypy/extradoc/talk/pepm2011 Message-ID: <20101008140021.D11AD282BE8@codespeak.net> Author: cfbolz Date: Fri Oct 8 15:53:19 2010 New Revision: 77725 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: preliminary numbers, psyco and pypy-nojit are missing Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Fri Oct 8 15:53:19 2010 @@ -1021,6 +1021,28 @@ \textbf{twisted\_names}: A DNS server benchmark using the Twisted networking framework\footnote{\texttt{http://twistedmatrix.com/trac/}}. +\begin{figure*} +\begin{tabular}{lrrrr} + &CPython &JIT no optimizations &JIT allocation removal &JIT full\\ +crypto\_pyaes &1958.40 $\pm$ 0.32 &782.14 $\pm$ 3.08 &150.86 $\pm$ 3.62 &125.68 $\pm$ 3.02\\ +django &630.91 $\pm$ 0.13 &365.90 $\pm$ 1.57 &124.41 $\pm$ 0.88 &117.36 $\pm$ 0.91\\ +fannkuch &1344.15 $\pm$ 0.36 &378.18 $\pm$ 0.36 &309.04 $\pm$ 1.56 &299.12 $\pm$ 0.25\\ +go &595.31 $\pm$ 0.51 &1067.26 $\pm$ 12.20 &121.76 $\pm$ 4.15 &125.57 $\pm$ 3.90\\ +html5lib &8678.67 $\pm$ 23.37 &17665.16 $\pm$ 2695.45 &6739.95 $\pm$ 1225.25 &6699.24 $\pm$ 1295.92\\ +meteor-contest &241.53 $\pm$ 0.09 &291.11 $\pm$ 0.86 &274.09 $\pm$ 0.37 &272.77 $\pm$ 0.30\\ +nbody\_modified &396.55 $\pm$ 0.24 &84.77 $\pm$ 0.14 &69.01 $\pm$ 0.08 &68.71 $\pm$ 0.09\\ +pyflate-fast &1991.33 $\pm$ 1.90 &1818.70 $\pm$ 4.30 &1062.03 $\pm$ 4.56 &1052.02 $\pm$ 3.24\\ +raytrace-simple &1598.84 $\pm$ 1.07 &1126.93 $\pm$ 11.42 &93.45 $\pm$ 4.75 &92.17 $\pm$ 4.15\\ +richards &216.47 $\pm$ 0.20 &169.84 $\pm$ 1.21 &11.81 $\pm$ 0.09 &11.90 $\pm$ 0.29\\ +spambayes &193.30 $\pm$ 0.03 &331.37 $\pm$ 14.93 &238.83 $\pm$ 6.10 &237.23 $\pm$ 5.97\\ +spectral-norm &375.91 $\pm$ 0.08 &221.24 $\pm$ 0.07 &31.96 $\pm$ 0.06 &28.79 $\pm$ 0.09\\ +telco &789.67 $\pm$ 1.14 &622.67 $\pm$ 1.86 &155.33 $\pm$ 1.82 &154.67 $\pm$ 1.82\\ +twisted\_names &6.61 $\pm$ 0.00 &8.85 $\pm$ 0.09 &4.35 $\pm$ 0.05 &4.35 $\pm$ 0.04\\ +\end{tabular} +\caption{Benchmark Times} +\label{fig:times} +\end{figure*} + \section{Related Work} From antocuni at codespeak.net Fri Oct 8 16:02:40 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 8 Oct 2010 16:02:40 +0200 (CEST) Subject: [pypy-svn] r77726 - pypy/extradoc/talk/pepm2011/benchmarks Message-ID: <20101008140240.1CA1C282BE8@codespeak.net> Author: antocuni Date: Fri Oct 8 16:00:11 2010 New Revision: 77726 Added: pypy/extradoc/talk/pepm2011/benchmarks/crypto_pyaes.py.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/fannkuch.py.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/get_own_bench_trace (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/go.py.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/meteor-contest.py.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/nbody_modified.py.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/pyflate-fast.py.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/raytrace-simple.py.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/spectral-norm.py.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/telco.py.trace.bz2 (contents, props changed) Log: script and traces for own benchmarks Added: pypy/extradoc/talk/pepm2011/benchmarks/crypto_pyaes.py.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/fannkuch.py.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/get_own_bench_trace ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pepm2011/benchmarks/get_own_bench_trace Fri Oct 8 16:00:11 2010 @@ -0,0 +1,9 @@ +#!/bin/bash + +for i in crypto_pyaes.py fannkuch.py go.py meteor-contest.py nbody_modified.py pyflate-fast.py raytrace-simple.py spectral-norm.py telco.py +do + echo $i + PYPYLOG=jit-log:$i.trace /tmp/pypy-c-jit-77705-linux/bin/pypy --jit optimizer=1 $i -n 20 +done + + Added: pypy/extradoc/talk/pepm2011/benchmarks/go.py.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/meteor-contest.py.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/nbody_modified.py.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/pyflate-fast.py.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/raytrace-simple.py.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/spectral-norm.py.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/telco.py.trace.bz2 ============================================================================== Binary file. No diff available. From afa at codespeak.net Fri Oct 8 16:16:17 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 16:16:17 +0200 (CEST) Subject: [pypy-svn] r77727 - in pypy/branch/fast-forward/pypy/module/_io: . test Message-ID: <20101008141617.0FD93282BE8@codespeak.net> Author: afa Date: Fri Oct 8 16:13:42 2010 New Revision: 77727 Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py pypy/branch/fast-forward/pypy/module/_io/test/test_io.py Log: io.BaseIO basic operations: close(), flush(), context manager. Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/interp_io.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py Fri Oct 8 16:13:42 2010 @@ -1,6 +1,8 @@ from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root -from pypy.interpreter.typedef import TypeDef, interp_attrproperty +from pypy.interpreter.typedef import ( + TypeDef, interp_attrproperty, GetSetProperty) from pypy.interpreter.gateway import interp2app, Arguments, unwrap_spec +from pypy.interpreter.error import OperationError from pypy.module.exceptions.interp_exceptions import W_IOError from pypy.tool.sourcetools import func_renamer @@ -36,11 +38,67 @@ class W_IOBase(Wrappable): def __init__(self, space): - pass + # XXX: IOBase thinks it has to maintain its own internal state in + # `__IOBase_closed` and call flush() by itself, but it is redundant + # with whatever behaviour a non-trivial derived class will implement. + self.__IOBase_closed = False + + def _closed(self, space): + # This gets the derived attribute, which is *not* __IOBase_closed + # in most cases! + w_closed = space.findattr(self, space.wrap('closed')) + if w_closed is not None and space.is_true(w_closed): + return True + return False + + def _CLOSED(self): + # Use this macro whenever you want to check the internal `closed` + # status of the IOBase object rather than the virtual `closed` + # attribute as returned by whatever subclass. + return self.__IOBase_closed + + def _check_closed(self, space): + if self._closed(space): + raise OperationError( + space.w_ValueError, + space.wrap("I/O operation on closed file")) + + def closed_get_w(space, self): + return space.newbool(self.__IOBase_closed) + + @unwrap_spec('self', ObjSpace) + def close_w(self, space): + if self._CLOSED(): + return + try: + space.call_method(self, "flush") + finally: + self.__IOBase_closed = True + + @unwrap_spec('self', ObjSpace) + def flush_w(self, space): + if self._CLOSED(): + raise OperationError( + space.w_ValueError, + space.wrap("I/O operation on closed file")) + + @unwrap_spec('self', ObjSpace) + def enter_w(self, space): + self._check_closed(space) + return space.wrap(self) + + @unwrap_spec('self', ObjSpace, Arguments) + def exit_w(self, space, __args__): + space.call_method(self, "close") W_IOBase.typedef = TypeDef( '_IOBase', __new__ = GenericNew(W_IOBase), + __enter__ = interp2app(W_IOBase.enter_w), + __exit__ = interp2app(W_IOBase.exit_w), + close = interp2app(W_IOBase.close_w), + flush = interp2app(W_IOBase.flush_w), + closed = GetSetProperty(W_IOBase.closed_get_w), ) class W_RawIOBase(W_IOBase): Modified: pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py Fri Oct 8 16:13:42 2010 @@ -9,7 +9,7 @@ self.buf = [] self.pos = 0 - def _check_closed(self): + def _check_closed(self, space): pass def _check_initialized(self): pass @@ -46,7 +46,7 @@ raise operationerrfmt(space.w_TypeError, "string argument expected, got '%s'", space.type(self).getname(space, '?')) - self._check_closed() + self._check_closed(space) string = space.unicode_w(w_obj) size = len(string) if size: @@ -56,7 +56,7 @@ @unwrap_spec('self', ObjSpace) def getvalue_w(self, space): self._check_initialized() - self._check_closed() + self._check_closed(space) return space.wrap(u''.join(self.buf)) W_StringIO.typedef = TypeDef( Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py Fri Oct 8 16:13:42 2010 @@ -16,3 +16,9 @@ pass MyFile("file") + def test_openclose(self): + import io + with io.BufferedIOBase() as f: + assert not f.closed + assert f.closed + From antocuni at codespeak.net Fri Oct 8 16:53:46 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 8 Oct 2010 16:53:46 +0200 (CEST) Subject: [pypy-svn] r77728 - pypy/extradoc/talk/pepm2011/benchmarks Message-ID: <20101008145346.90162282BE8@codespeak.net> Author: antocuni Date: Fri Oct 8 16:47:45 2010 New Revision: 77728 Added: pypy/extradoc/talk/pepm2011/benchmarks/django.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/html5lib.trace.bz2 (contents, props changed) Log: two more traces Added: pypy/extradoc/talk/pepm2011/benchmarks/django.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/html5lib.trace.bz2 ============================================================================== Binary file. No diff available. From antocuni at codespeak.net Fri Oct 8 16:59:54 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Fri, 8 Oct 2010 16:59:54 +0200 (CEST) Subject: [pypy-svn] r77729 - pypy/extradoc/talk/pepm2011/benchmarks Message-ID: <20101008145954.30A49282BE8@codespeak.net> Author: antocuni Date: Fri Oct 8 16:58:35 2010 New Revision: 77729 Added: pypy/extradoc/talk/pepm2011/benchmarks/richards.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/spambayes.trace.bz2 (contents, props changed) pypy/extradoc/talk/pepm2011/benchmarks/twisted-names.trace.bz2 (contents, props changed) Log: more traces Added: pypy/extradoc/talk/pepm2011/benchmarks/richards.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/spambayes.trace.bz2 ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pepm2011/benchmarks/twisted-names.trace.bz2 ============================================================================== Binary file. No diff available. From afa at codespeak.net Fri Oct 8 18:50:55 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 18:50:55 +0200 (CEST) Subject: [pypy-svn] r77730 - in pypy/branch/fast-forward/pypy/module/__builtin__: . test Message-ID: <20101008165055.2F707282BDD@codespeak.net> Author: afa Date: Fri Oct 8 18:50:53 2010 New Revision: 77730 Modified: pypy/branch/fast-forward/pypy/module/__builtin__/__init__.py pypy/branch/fast-forward/pypy/module/__builtin__/operation.py pypy/branch/fast-forward/pypy/module/__builtin__/test/test_buffer.py Log: Very very basic start of a memoryview I can't get my mind around the PyPy's buffer interface. Modified: pypy/branch/fast-forward/pypy/module/__builtin__/__init__.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/__builtin__/__init__.py (original) +++ pypy/branch/fast-forward/pypy/module/__builtin__/__init__.py Fri Oct 8 18:50:53 2010 @@ -52,7 +52,8 @@ 'object' : '(space.w_object)', 'bytes' : '(space.w_str)', 'unicode' : '(space.w_unicode)', - 'buffer' : 'operation.Buffer', + 'buffer' : 'operation.W_Buffer', + 'memoryview' : 'operation.W_Memoryview', 'file' : 'state.get(space).w_file', 'open' : 'state.get(space).w_file', Modified: pypy/branch/fast-forward/pypy/module/__builtin__/operation.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/__builtin__/operation.py (original) +++ pypy/branch/fast-forward/pypy/module/__builtin__/operation.py Fri Oct 8 18:50:53 2010 @@ -3,13 +3,32 @@ """ from pypy.interpreter import gateway, buffer -from pypy.interpreter.baseobjspace import ObjSpace +from pypy.interpreter.baseobjspace import ObjSpace, W_Root from pypy.interpreter.error import OperationError +from pypy.interpreter.gateway import interp2app, unwrap_spec +from pypy.interpreter.typedef import TypeDef from pypy.rlib.runicode import UNICHR import __builtin__ NoneNotWrapped = gateway.NoneNotWrapped -Buffer = buffer.Buffer +W_Buffer = buffer.Buffer + +class W_Memoryview(buffer.StringLikeBuffer): + @unwrap_spec(ObjSpace, W_Root, W_Root) + def descr_new(space, w_subtype, w_object): + self = space.allocate_instance(W_Memoryview, w_subtype) + buffer.StringLikeBuffer.__init__(self, space, w_object) + return space.wrap(self) + + @unwrap_spec('self', ObjSpace) + def to_bytes_w(self, space): + return space.wrap(self.as_str()) + +W_Memoryview.typedef = TypeDef( + "memoryview", + __new__=interp2app(W_Memoryview.descr_new.im_func), + to_bytes=interp2app(W_Memoryview.to_bytes_w), + ) def abs(space, w_val): "abs(number) -> number\n\nReturn the absolute value of the argument." Modified: pypy/branch/fast-forward/pypy/module/__builtin__/test/test_buffer.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/__builtin__/test/test_buffer.py (original) +++ pypy/branch/fast-forward/pypy/module/__builtin__/test/test_buffer.py Fri Oct 8 18:50:53 2010 @@ -159,3 +159,8 @@ raises(ValueError, buffer, a, -1) raises(ValueError, buffer, a, 0, -2) + +class AppTestMemoryview: + def test_basic(self): + v = memoryview("abc") + assert v.to_bytes() == "abc" From afa at codespeak.net Fri Oct 8 18:51:29 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 18:51:29 +0200 (CEST) Subject: [pypy-svn] r77731 - in pypy/branch/fast-forward/pypy: interpreter module/_io Message-ID: <20101008165129.9B261282BDD@codespeak.net> Author: afa Date: Fri Oct 8 18:51:28 2010 New Revision: 77731 Modified: pypy/branch/fast-forward/pypy/interpreter/typedef.py pypy/branch/fast-forward/pypy/module/_io/interp_io.py Log: Move generic_new_descr to typedef.py Modified: pypy/branch/fast-forward/pypy/interpreter/typedef.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/typedef.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/typedef.py Fri Oct 8 18:51:28 2010 @@ -552,6 +552,17 @@ Member.typedef.acceptable_as_base_class = False # ____________________________________________________________ + +def generic_new_descr(W_Type): + @unwrap_spec(ObjSpace, W_Root, Arguments) + @func_renamer('descr_new_%s' % W_Type.__name__) + def descr_new(space, w_subtype, __args__): + self = space.allocate_instance(W_Type, w_subtype) + W_Type.__init__(self, space) + return space.wrap(self) + return interp2app(descr_new) + +# ____________________________________________________________ # # Definition of the type's descriptors for all the internal types Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/interp_io.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py Fri Oct 8 18:51:28 2010 @@ -1,6 +1,6 @@ from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root from pypy.interpreter.typedef import ( - TypeDef, interp_attrproperty, GetSetProperty) + TypeDef, interp_attrproperty, GetSetProperty, generic_new_descr) from pypy.interpreter.gateway import interp2app, Arguments, unwrap_spec from pypy.interpreter.error import OperationError from pypy.module.exceptions.interp_exceptions import W_IOError @@ -8,15 +8,6 @@ DEFAULT_BUFFER_SIZE = 8192 -def GenericNew(W_Type): - @unwrap_spec(ObjSpace, W_Root, Arguments) - @func_renamer('descr_new_%s' % W_Type.__name__) - def descr_new(space, w_subtype, __args__): - self = space.allocate_instance(W_Type, w_subtype) - W_Type.__init__(self, space) - return space.wrap(self) - return interp2app(descr_new) - class W_BlockingIOError(W_IOError): def __init__(self, space): W_IOError.__init__(self, space) @@ -31,7 +22,7 @@ 'BlockingIOError', __doc__ = ("Exception raised when I/O would block " "on a non-blocking I/O stream"), - __new__ = GenericNew(W_BlockingIOError), + __new__ = generic_new_descr(W_BlockingIOError), __init__ = interp2app(W_BlockingIOError.descr_init), characters_written = interp_attrproperty('written', W_BlockingIOError), ) @@ -93,7 +84,7 @@ W_IOBase.typedef = TypeDef( '_IOBase', - __new__ = GenericNew(W_IOBase), + __new__ = generic_new_descr(W_IOBase), __enter__ = interp2app(W_IOBase.enter_w), __exit__ = interp2app(W_IOBase.exit_w), close = interp2app(W_IOBase.close_w), @@ -105,7 +96,7 @@ pass W_RawIOBase.typedef = TypeDef( '_RawIOBase', W_IOBase.typedef, - __new__ = GenericNew(W_RawIOBase), + __new__ = generic_new_descr(W_RawIOBase), ) class W_BufferedIOBase(W_IOBase): @@ -113,14 +104,14 @@ W_BufferedIOBase.typedef = TypeDef( '_BufferedIOBase', W_IOBase.typedef, - __new__ = GenericNew(W_BufferedIOBase), + __new__ = generic_new_descr(W_BufferedIOBase), ) class W_TextIOBase(W_IOBase): pass W_TextIOBase.typedef = TypeDef( '_TextIOBase', W_IOBase.typedef, - __new__ = GenericNew(W_TextIOBase), + __new__ = generic_new_descr(W_TextIOBase), ) class W_FileIO(W_RawIOBase): From afa at codespeak.net Fri Oct 8 19:58:45 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 19:58:45 +0200 (CEST) Subject: [pypy-svn] r77732 - pypy/branch/fast-forward/pypy/interpreter Message-ID: <20101008175845.45E95282BDD@codespeak.net> Author: afa Date: Fri Oct 8 19:58:43 2010 New Revision: 77732 Modified: pypy/branch/fast-forward/pypy/interpreter/typedef.py Log: Bah. Should have run tests before checkin Modified: pypy/branch/fast-forward/pypy/interpreter/typedef.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/typedef.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/typedef.py Fri Oct 8 19:58:43 2010 @@ -554,12 +554,12 @@ # ____________________________________________________________ def generic_new_descr(W_Type): - @unwrap_spec(ObjSpace, W_Root, Arguments) - @func_renamer('descr_new_%s' % W_Type.__name__) def descr_new(space, w_subtype, __args__): self = space.allocate_instance(W_Type, w_subtype) W_Type.__init__(self, space) return space.wrap(self) + descr_new.unwrap_spec = [ObjSpace, W_Root, Arguments] + descr_new = func_with_new_name(descr_new, 'descr_new_%s' % W_Type.__name__) return interp2app(descr_new) # ____________________________________________________________ From afa at codespeak.net Fri Oct 8 22:01:04 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 22:01:04 +0200 (CEST) Subject: [pypy-svn] r77733 - pypy/branch/fast-forward/lib-python Message-ID: <20101008200104.56681282BDD@codespeak.net> Author: afa Date: Fri Oct 8 22:01:02 2010 New Revision: 77733 Modified: pypy/branch/fast-forward/lib-python/TODO Log: update TODO Modified: pypy/branch/fast-forward/lib-python/TODO ============================================================================== --- pypy/branch/fast-forward/lib-python/TODO (original) +++ pypy/branch/fast-forward/lib-python/TODO Fri Oct 8 22:01:02 2010 @@ -12,6 +12,12 @@ - Missing complex.__trunc__ +- New complex syntax:: + + assert complex("(1+2j)") == (1+2j) + assert complex("(1-2j)") == (1-2j) + assert complex("1e500") == complex(INF, 0.0) + - Mark some tests as "implementation specific":: @test_support.cpython_only From afa at codespeak.net Fri Oct 8 22:41:31 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 22:41:31 +0200 (CEST) Subject: [pypy-svn] r77734 - pypy/branch/fast-forward/pypy/interpreter/pyparser Message-ID: <20101008204131.9994B282BDD@codespeak.net> Author: afa Date: Fri Oct 8 22:41:29 2010 New Revision: 77734 Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py Log: Allow 'whitespace' to form a valid token, when nothing more can match. This gives a better error position: when given "if ?:", the parser will return whitespace as a token, and stop on the ?. Previously, the error would be at the space character, and one test in interpreter/test/test_syntax.py failed. This effectively reverts the only effective change in r77650, which I did not understand correctly at the time. Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py Fri Oct 8 22:41:29 2010 @@ -182,6 +182,7 @@ pseudoToken = chain(states, makeWhitespace(), group(states, + newArcPair(states, EMPTY), pseudoExtras, number, funny, contStr, name)) dfaStates, dfaAccepts = nfaToDfa(states, *pseudoToken) return DFA(dfaStates, dfaAccepts) Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py Fri Oct 8 22:41:29 2010 @@ -23,7 +23,7 @@ # ______________________________________________________________________ # Automatically generated DFA's -accepts = [False, True, True, True, True, True, True, True, +accepts = [True, True, True, True, True, True, True, True, True, True, False, True, True, True, False, False, False, False, True, True, True, True, True, False, True, False, True, False, True, False, False, From afa at codespeak.net Fri Oct 8 22:44:02 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 22:44:02 +0200 (CEST) Subject: [pypy-svn] r77735 - pypy/trunk/pypy/interpreter/pyparser Message-ID: <20101008204402.34017282BDD@codespeak.net> Author: afa Date: Fri Oct 8 22:44:00 2010 New Revision: 77735 Modified: pypy/trunk/pypy/interpreter/pyparser/genpytokenize.py pypy/trunk/pypy/interpreter/pyparser/pytokenize.py Log: Merge r77734 from the fast-forward branch. Should fix the failure in test_syntax.py Modified: pypy/trunk/pypy/interpreter/pyparser/genpytokenize.py ============================================================================== --- pypy/trunk/pypy/interpreter/pyparser/genpytokenize.py (original) +++ pypy/trunk/pypy/interpreter/pyparser/genpytokenize.py Fri Oct 8 22:44:00 2010 @@ -176,6 +176,7 @@ pseudoToken = chain(states, makeWhitespace(), group(states, + newArcPair(states, EMPTY), pseudoExtras, number, funny, contStr, name)) dfaStates, dfaAccepts = nfaToDfa(states, *pseudoToken) return DFA(dfaStates, dfaAccepts) Modified: pypy/trunk/pypy/interpreter/pyparser/pytokenize.py ============================================================================== --- pypy/trunk/pypy/interpreter/pyparser/pytokenize.py (original) +++ pypy/trunk/pypy/interpreter/pyparser/pytokenize.py Fri Oct 8 22:44:00 2010 @@ -23,7 +23,7 @@ # ______________________________________________________________________ # Automatically generated DFA's -accepts = [False, True, True, True, True, True, True, True, +accepts = [True, True, True, True, True, True, True, True, True, True, False, True, True, True, False, False, False, False, True, True, True, False, True, False, True, False, True, False, False, True, From afa at codespeak.net Fri Oct 8 22:53:45 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Fri, 8 Oct 2010 22:53:45 +0200 (CEST) Subject: [pypy-svn] r77736 - in pypy/branch/fast-forward/pypy/module/__builtin__: . test Message-ID: <20101008205345.AF626282BDD@codespeak.net> Author: afa Date: Fri Oct 8 22:53:43 2010 New Revision: 77736 Modified: pypy/branch/fast-forward/pypy/module/__builtin__/operation.py pypy/branch/fast-forward/pypy/module/__builtin__/test/test_buffer.py Log: Let's make memoryview inherit from buffer and all its operations. I wonder how far this can go. Modified: pypy/branch/fast-forward/pypy/module/__builtin__/operation.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/__builtin__/operation.py (original) +++ pypy/branch/fast-forward/pypy/module/__builtin__/operation.py Fri Oct 8 22:53:43 2010 @@ -25,7 +25,7 @@ return space.wrap(self.as_str()) W_Memoryview.typedef = TypeDef( - "memoryview", + "memoryview", W_Buffer.typedef, __new__=interp2app(W_Memoryview.descr_new.im_func), to_bytes=interp2app(W_Memoryview.to_bytes_w), ) Modified: pypy/branch/fast-forward/pypy/module/__builtin__/test/test_buffer.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/__builtin__/test/test_buffer.py (original) +++ pypy/branch/fast-forward/pypy/module/__builtin__/test/test_buffer.py Fri Oct 8 22:53:43 2010 @@ -164,3 +164,5 @@ def test_basic(self): v = memoryview("abc") assert v.to_bytes() == "abc" + assert len(v) == 3 + assert list(v) == ['a', 'b', 'c'] From afa at codespeak.net Sat Oct 9 00:00:04 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sat, 9 Oct 2010 00:00:04 +0200 (CEST) Subject: [pypy-svn] r77737 - in pypy/branch/fast-forward/pypy/module/__builtin__: . test Message-ID: <20101008220004.BEC16282BDD@codespeak.net> Author: afa Date: Sat Oct 9 00:00:03 2010 New Revision: 77737 Modified: pypy/branch/fast-forward/pypy/module/__builtin__/functional.py pypy/branch/fast-forward/pypy/module/__builtin__/test/test_functional.py Log: xrange objects are now pickable Modified: pypy/branch/fast-forward/pypy/module/__builtin__/functional.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/__builtin__/functional.py (original) +++ pypy/branch/fast-forward/pypy/module/__builtin__/functional.py Sat Oct 9 00:00:03 2010 @@ -631,6 +631,15 @@ return self.space.wrap(W_XRangeIterator(self.space, lastitem, self.len, -self.step)) + def descr_reduce(self): + space = self.space + return space.newtuple( + [space.type(self), + space.newtuple([space.wrap(self.start), + space.wrap(self.start + self.len * self.step), + space.wrap(self.step)]) + ]) + def _toint(space, w_obj): # this also supports float arguments. CPython still does, too. # needs a bit more thinking in general... @@ -644,6 +653,7 @@ __iter__ = interp2app(W_XRange.descr_iter), __len__ = interp2app(W_XRange.descr_len), __reversed__ = interp2app(W_XRange.descr_reversed), + __reduce__ = interp2app(W_XRange.descr_reduce), ) class W_XRangeIterator(Wrappable): Modified: pypy/branch/fast-forward/pypy/module/__builtin__/test/test_functional.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/__builtin__/test/test_functional.py (original) +++ pypy/branch/fast-forward/pypy/module/__builtin__/test/test_functional.py Sat Oct 9 00:00:03 2010 @@ -158,6 +158,12 @@ raises(OverflowError, xrange, 0, a) raises(OverflowError, xrange, 0, 1, a) + def test_xrange_reduce(self): + x = xrange(2, 9, 3) + callable, args = x.__reduce__() + y = callable(*args) + assert list(y) == list(x) + class AppTestReversed: def test_reversed(self): r = reversed("hello") From afa at codespeak.net Sat Oct 9 00:05:39 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sat, 9 Oct 2010 00:05:39 +0200 (CEST) Subject: [pypy-svn] r77738 - in pypy/branch/fast-forward/pypy/interpreter: . test Message-ID: <20101008220539.BADC9282BDD@codespeak.net> Author: afa Date: Sat Oct 9 00:05:38 2010 New Revision: 77738 Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_function.py pypy/branch/fast-forward/pypy/interpreter/typedef.py Log: add function.__code__ and function.__defaults__ (py3k renamed the attributes) Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_function.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/test/test_function.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/test/test_function.py Sat Oct 9 00:05:38 2010 @@ -32,6 +32,8 @@ assert f.__name__ == f.func_name assert f.__doc__ == f.func_doc assert f.__dict__ is f.func_dict + assert f.__code__ is f.func_code + assert f.__defaults__ is f.func_defaults assert hasattr(f, '__class__') def test_write_doc(self): Modified: pypy/branch/fast-forward/pypy/interpreter/typedef.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/typedef.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/typedef.py Sat Oct 9 00:05:38 2010 @@ -770,9 +770,11 @@ func_defaults = getset_func_defaults, func_globals = interp_attrproperty_w('w_func_globals', cls=Function), func_closure = GetSetProperty( Function.fget_func_closure ), + __code__ = getset_func_code, __doc__ = getset_func_doc, __name__ = getset_func_name, __dict__ = getset_func_dict, + __defaults__ = getset_func_defaults, __module__ = getset___module__, __weakref__ = make_weakref_descr(Function), ) From afa at codespeak.net Sat Oct 9 00:45:35 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sat, 9 Oct 2010 00:45:35 +0200 (CEST) Subject: [pypy-svn] r77739 - in pypy/branch/fast-forward/pypy/module/_io: . test Message-ID: <20101008224535.8F00B282BDD@codespeak.net> Author: afa Date: Sat Oct 9 00:45:33 2010 New Revision: 77739 Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py pypy/branch/fast-forward/pypy/module/_io/test/test_io.py Log: io.IOBase is iterable, and calls readline(). Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/interp_io.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py Sat Oct 9 00:45:33 2010 @@ -82,11 +82,25 @@ def exit_w(self, space, __args__): space.call_method(self, "close") + @unwrap_spec('self', ObjSpace) + def iter_w(self, space): + self._check_closed(space) + return space.wrap(self) + + @unwrap_spec('self', ObjSpace) + def next_w(self, space): + w_line = space.call_method(self, "readline") + if space.int_w(space.len(w_line)) == 0: + raise OperationError(space.w_StopIteration, space.w_None) + return w_line + W_IOBase.typedef = TypeDef( '_IOBase', __new__ = generic_new_descr(W_IOBase), __enter__ = interp2app(W_IOBase.enter_w), __exit__ = interp2app(W_IOBase.exit_w), + __iter__ = interp2app(W_IOBase.iter_w), + next = interp2app(W_IOBase.next_w), close = interp2app(W_IOBase.close_w), flush = interp2app(W_IOBase.flush_w), closed = GetSetProperty(W_IOBase.closed_get_w), Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py (original) +++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py Sat Oct 9 00:45:33 2010 @@ -22,3 +22,17 @@ assert not f.closed assert f.closed + def test_iter(self): + import io + class MyFile(io.IOBase): + def __init__(self): + self.lineno = 0 + def readline(self): + self.lineno += 1 + if self.lineno == 1: + return "line1" + elif self.lineno == 2: + return "line2" + return "" + + assert list(MyFile()) == ["line1", "line2"] From afa at codespeak.net Sat Oct 9 09:35:13 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sat, 9 Oct 2010 09:35:13 +0200 (CEST) Subject: [pypy-svn] r77740 - in pypy/branch/fast-forward/lib_pypy: . pypy_test Message-ID: <20101009073513.58670282BDD@codespeak.net> Author: afa Date: Sat Oct 9 09:35:10 2010 New Revision: 77740 Modified: pypy/branch/fast-forward/lib_pypy/_collections.py pypy/branch/fast-forward/lib_pypy/pypy_test/test_defaultdict.py Log: defaultdict accepts None for the __missing__ function Modified: pypy/branch/fast-forward/lib_pypy/_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/_collections.py Sat Oct 9 09:35:10 2010 @@ -317,7 +317,7 @@ self.default_factory = None if 'default_factory' in kwds: self.default_factory = kwds.pop('default_factory') - elif len(args) > 0 and callable(args[0]): + elif len(args) > 0 and (callable(args[0]) or args[0] is None): self.default_factory = args[0] args = args[1:] super(defaultdict, self).__init__(*args, **kwds) Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_defaultdict.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/pypy_test/test_defaultdict.py (original) +++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_defaultdict.py Sat Oct 9 09:35:10 2010 @@ -48,6 +48,10 @@ py.test.raises(KeyError, d2.__getitem__, 15) py.test.raises(TypeError, defaultdict, 1) + def test_constructor(self): + assert defaultdict(None) == {} + assert defaultdict(None, {1: 2}) == {1: 2} + def test_missing(self): d1 = defaultdict() py.test.raises(KeyError, d1.__missing__, 42) From hakanardo at codespeak.net Sat Oct 9 10:08:47 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Sat, 9 Oct 2010 10:08:47 +0200 (CEST) Subject: [pypy-svn] r77741 - in pypy/branch/jit-unroll-loops/pypy/jit/metainterp: . test Message-ID: <20101009080847.61283282BDD@codespeak.net> Author: hakanardo Date: Sat Oct 9 10:08:45 2010 New Revision: 77741 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Log: mark preamble as entry bridge and dont count it Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py Sat Oct 9 10:08:45 2010 @@ -85,7 +85,7 @@ loop.token.specnodes = [prebuiltNotSpecNode] * len(loop.inputargs) # FIXME loop.preamble.token.specnodes = [prebuiltNotSpecNode] * len(loop.preamble.inputargs) # FIXME send_loop_to_backend(metainterp_sd, loop, "loop") - send_loop_to_backend(metainterp_sd, loop.preamble, "loop") + send_loop_to_backend(metainterp_sd, loop.preamble, "entry bridge") insert_loop_token(old_loop_tokens, loop.preamble.token) jitdriver_sd.warmstate.attach_unoptimized_bridge_from_interp( greenkey, loop.preamble.token) Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Sat Oct 9 10:08:45 2010 @@ -315,11 +315,10 @@ return res res = self.meta_interp(f, [6, 7]) assert res == 252 - self.check_loop_count(2) - self.check_loops({'guard_true': 2, - 'int_add': 2, 'int_sub': 2, 'int_gt': 2, - 'int_mul': 1, - 'jump': 2}) + self.check_loop_count(1) + self.check_loops({'guard_true': 1, + 'int_add': 1, 'int_sub': 1, 'int_gt': 1, + 'jump': 1}) def test_loop_invariant_mul_ovf(self): myjitdriver = JitDriver(greens = [], reds = ['y', 'res', 'x']) @@ -334,11 +333,11 @@ return res res = self.meta_interp(f, [6, 7]) assert res == 308 - self.check_loop_count(2) - self.check_loops({'guard_true': 2, 'guard_no_overflow': 1, - 'int_add': 4, 'int_sub': 2, 'int_gt': 2, - 'int_mul': 2, 'int_mul_ovf': 1, - 'jump': 2}) + self.check_loop_count(1) + self.check_loops({'guard_true': 1, + 'int_add': 2, 'int_sub': 1, 'int_gt': 1, + 'int_mul': 1, + 'jump': 1}) def test_loop_invariant_mul_bridge1(self): myjitdriver = JitDriver(greens = [], reds = ['y', 'res', 'x']) @@ -354,7 +353,7 @@ return res res = self.meta_interp(f, [6, 32]) assert res == 3427 - self.check_loop_count(3) + self.check_loop_count(2) def test_loop_invariant_mul_bridge_maintaining1(self): myjitdriver = JitDriver(greens = [], reds = ['y', 'res', 'x']) @@ -370,11 +369,11 @@ return res res = self.meta_interp(f, [6, 32]) assert res == 1167 - self.check_loop_count(3) - self.check_loops({'int_lt': 2, 'int_gt': 1, - 'guard_false': 2, 'guard_true': 1, - 'int_sub': 3, 'int_mul': 1, 'int_add': 3, - 'jump': 3}) + self.check_loop_count(2) + self.check_loops({'int_add': 1, 'int_lt': 1, + 'int_sub': 1, 'guard_false': 1, + 'jump': 1}) + def test_loop_invariant_mul_bridge_maintaining2(self): myjitdriver = JitDriver(greens = [], reds = ['y', 'res', 'x']) @@ -391,11 +390,12 @@ return res res = self.meta_interp(f, [6, 16]) assert res == 828 - self.check_loop_count(3) - self.check_loops({'int_lt': 2, 'int_gt': 1, - 'guard_false': 2, 'guard_true': 1, - 'int_sub': 3, 'int_mul': 1, 'int_add': 3, - 'jump': 3}) + self.check_loop_count(2) + self.check_loops({'int_add': 1, 'int_lt': 1, + 'int_sub': 1, 'guard_false': 1, + 'jump': 1}) + + def test_loop_invariant_intbox(self): myjitdriver = JitDriver(greens = [], reds = ['y', 'res', 'x']) @@ -415,11 +415,10 @@ return res res = self.meta_interp(f, [6, 7]) assert res == 252 - self.check_loop_count(2) - self.check_loops({'guard_true': 2, - 'int_add': 2, 'int_sub': 2, 'int_gt': 2, - 'int_mul': 1, 'getfield_gc_pure': 1, - 'jump': 2}) + self.check_loop_count(1) + self.check_loops({'guard_true': 1, + 'int_add': 1, 'int_sub': 1, 'int_gt': 1, + 'jump': 1}) def test_loops_are_transient(self): import gc, weakref From afa at codespeak.net Sat Oct 9 10:42:18 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sat, 9 Oct 2010 10:42:18 +0200 (CEST) Subject: [pypy-svn] r77742 - pypy/branch/fast-forward/lib_pypy/xml Message-ID: <20101009084218.B8A5C282BDD@codespeak.net> Author: afa Date: Sat Oct 9 10:42:17 2010 New Revision: 77742 Removed: pypy/branch/fast-forward/lib_pypy/xml/ Log: Remove our copy of the xml package. This fixes test_minidom From hakanardo at codespeak.net Sat Oct 9 15:33:51 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Sat, 9 Oct 2010 15:33:51 +0200 (CEST) Subject: [pypy-svn] r77744 - in pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test Message-ID: <20101009133351.A9383282BDD@codespeak.net> Author: hakanardo Date: Sat Oct 9 15:33:49 2010 New Revision: 77744 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py Log: support some virtuals Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py Sat Oct 9 15:33:49 2010 @@ -47,6 +47,10 @@ def get_key_box(self): return self.box + # FIXME: Reise get_args_for_fail instead? + def get_forced_boxes(self): + return [self.force_box()] + def get_args_for_fail(self, modifier): pass Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Sat Oct 9 15:33:49 2010 @@ -14,13 +14,13 @@ if not self.enabled: self.emit_operation(op) return - + if op.getopnum() == rop.JUMP: loop = self.optimizer.loop loop.preamble.operations = self.optimizer.newoperations self.optimizer.newoperations = [] jump_args = op.getarglist() - inputargs = self.inline(loop.preamble.operations + [op], + inputargs = self.inline(loop.operations, loop.inputargs, jump_args) loop.inputargs = inputargs jmp = ResOperation(rop.JUMP, loop.inputargs[:], None) @@ -38,7 +38,10 @@ for v in self.optimizer.values.values(): v.fromstart = True - inputargs = jump_args[:] + inputargs = [] + for arg in jump_args: + inputargs.extend(self.getvalue(arg).get_forced_boxes()) + for op in loop_operations: newop = op.clone() for i in range(op.numargs()): @@ -55,9 +58,10 @@ descr.rd_numb = None if newop.getopnum() == rop.JUMP: - args = newop.getarglist() + args = [] + for arg in newop.getarglist(): + args.extend(self.getvalue(arg).get_forced_boxes()) newop.initarglist(args + inputargs[len(args):]) - # FIXME: Assumes no virtuals current = len(self.optimizer.newoperations) self.emit_operation(newop) Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py Sat Oct 9 15:33:49 2010 @@ -121,6 +121,13 @@ fieldvalue = self._fields[ofs] fieldvalue.get_args_for_fail(modifier) + def get_forced_boxes(self): + lst = self._get_field_descr_list() + fieldboxes = [] + for ofs in lst: + fieldboxes.extend(self._fields[ofs].get_forced_boxes()) + return fieldboxes + class VirtualValue(AbstractVirtualStructValue): level = LEVEL_KNOWNCLASS Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py Sat Oct 9 15:33:49 2010 @@ -12,7 +12,7 @@ def _freeze_(self): return True - def test_virtualized(self): + def test_virtualized1(self): myjitdriver = JitDriver(greens = [], reds = ['n', 'node']) def f(n): node = self._new() From david at codespeak.net Sat Oct 9 16:45:20 2010 From: david at codespeak.net (david at codespeak.net) Date: Sat, 9 Oct 2010 16:45:20 +0200 (CEST) Subject: [pypy-svn] r77745 - pypy/branch/arm-backend/pypy/jit/backend/arm/test Message-ID: <20101009144520.DD292282BAD@codespeak.net> Author: david Date: Sat Oct 9 16:45:19 2010 New Revision: 77745 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Log: refactor test support code, add a skipping macro for as version Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py Sat Oct 9 16:45:19 2010 @@ -1,11 +1,8 @@ import os import tempfile +from pypy.jit.backend.arm.test.support import AS class ASMInstruction(object): - if os.uname()[0] == 'Darwin': - asm = '~/Code/arm-jit/android/android-ndk-r4b//build/prebuilt/darwin-x86/arm-eabi-4.4.0/arm-eabi/bin/as' - else: - asm = 'as' asm_opts = '-mcpu=cortex-a8 -march=armv7' body = """.section .text .arm @@ -41,7 +38,7 @@ res = self.body % (self.instr) self.file.write(res) self.file.flush() - os.system("%s %s %s -o %s/a.out" % (self.asm, self.asm_opts, self.name, self.tmpdir)) + os.system("%s %s %s -o %s/a.out" % (AS, self.asm_opts, self.name, self.tmpdir)) def __del__(self): self.file.close() Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py Sat Oct 9 16:45:19 2010 @@ -1,21 +1,33 @@ +import os + from pypy.rpython.lltypesystem import lltype, rffi +if os.uname()[1] == 'llaima.local': + AS = '~/Code/arm-jit/android/android-ndk-r4b//build/prebuilt/darwin-x86/arm-eabi-4.4.0/arm-eabi/bin/as' +else: + AS = 'as' + def run_asm(asm): BOOTSTRAP_TP = lltype.FuncType([], lltype.Signed) - addr = asm.mc.baseaddr()#executable_token._arm_bootstrap_code + addr = asm.mc.baseaddr() assert addr % 8 == 0 func = rffi.cast(lltype.Ptr(BOOTSTRAP_TP), addr) return func() def skip_unless_arm(): - import os import py + check_skip(os.uname()[4]) +def requires_arm_as(): + import commands + i = commands.getoutput("%s -version &1" % AS) + check_skip(i) + +def check_skip(inp, search='arm', msg='only for arm'): skip = True try: - if os.uname()[4].index('arm') >= 0: + if inp.index(search) >= 0: skip = False finally: if skip: - py.test.skip('only for arm') - + py.test.skip(msg) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Sat Oct 9 16:45:19 2010 @@ -1,7 +1,11 @@ from pypy.jit.backend.arm import registers as r from pypy.jit.backend.arm import codebuilder +from pypy.jit.backend.arm.test.support import requires_arm_as from gen import assemble import py + +requires_arm_as() + class CodeBuilder(codebuilder.ARMv7Builder): def __init__(self): self.buffer = [] From hakanardo at codespeak.net Sat Oct 9 17:12:41 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Sat, 9 Oct 2010 17:12:41 +0200 (CEST) Subject: [pypy-svn] r77746 - in pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test Message-ID: <20101009151241.6D622282BAD@codespeak.net> Author: hakanardo Date: Sat Oct 9 17:12:39 2010 New Revision: 77746 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Log: arguments of jump fixed Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Sat Oct 9 17:12:39 2010 @@ -20,7 +20,8 @@ loop.preamble.operations = self.optimizer.newoperations self.optimizer.newoperations = [] jump_args = op.getarglist() - inputargs = self.inline(loop.operations, + op.initarglist([]) + inputargs = self.inline(loop.preamble.operations + [op], loop.inputargs, jump_args) loop.inputargs = inputargs jmp = ResOperation(rop.JUMP, loop.inputargs[:], None) @@ -59,7 +60,8 @@ if newop.getopnum() == rop.JUMP: args = [] - for arg in newop.getarglist(): + #for arg in newop.getarglist(): + for arg in [argmap[a] for a in inputargs]: args.extend(self.getvalue(arg).get_forced_boxes()) newop.initarglist(args + inputargs[len(args):]) Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Sat Oct 9 17:12:39 2010 @@ -595,7 +595,7 @@ res = self.meta_interp(f, [21, 5]) assert res == -1 # the CALL_PURE is constant-folded away by optimizeopt.py - self.check_loops(int_sub=1, call=0, call_pure=0, getfield_gc=1) + self.check_loops(int_sub=1, call=0, call_pure=0, getfield_gc=0) def test_constant_across_mp(self): myjitdriver = JitDriver(greens = [], reds = ['n']) @@ -968,10 +968,9 @@ self.meta_interp(f, [20], repeat=7) self.check_tree_loop_count(2) # the loop and the entry path # we get: - # ENTER - compile the new loop - # ENTER - compile the entry bridge + # ENTER - compile the new loop and the entry bridge # ENTER - compile the leaving path - self.check_enter_count(3) + self.check_enter_count(2) def test_bridge_from_interpreter_2(self): # one case for backend - computing of framesize on guard failure From hakanardo at codespeak.net Sat Oct 9 17:39:21 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Sat, 9 Oct 2010 17:39:21 +0200 (CEST) Subject: [pypy-svn] r77747 - in pypy/branch/jit-unroll-loops/pypy/jit/metainterp: . optimizeopt Message-ID: <20101009153921.7C1E3282BAD@codespeak.net> Author: hakanardo Date: Sat Oct 9 17:39:19 2010 New Revision: 77747 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resoperation.py Log: bugfix Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Sat Oct 9 17:39:19 2010 @@ -41,7 +41,9 @@ inputargs = [] for arg in jump_args: - inputargs.extend(self.getvalue(arg).get_forced_boxes()) + for a in self.getvalue(arg).get_forced_boxes(): + if not isinstance(a, Const): + inputargs.append(a) for op in loop_operations: newop = op.clone() Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resoperation.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resoperation.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resoperation.py Sat Oct 9 17:39:19 2010 @@ -84,7 +84,7 @@ descr = self.getdescr() if descr is not None: descr = descr.clone_if_mutable() - op = ResOperation(self.getopnum(), args, self.result, descr) + op = ResOperation(self.getopnum(), args[:], self.result, descr) if not we_are_translated(): op.name = self.name op.pc = self.pc From afa at codespeak.net Sat Oct 9 22:09:51 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sat, 9 Oct 2010 22:09:51 +0200 (CEST) Subject: [pypy-svn] r77748 - pypy/branch/fast-forward/lib-python/modified-2.7.0/test Message-ID: <20101009200951.AA1EC282BAD@codespeak.net> Author: afa Date: Sat Oct 9 22:09:49 2010 New Revision: 77748 Added: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_code.py - copied, changed from r77733, pypy/branch/fast-forward/lib-python/2.7.0/test/test_code.py Log: Skip a CPython specific test Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_code.py (from r77733, pypy/branch/fast-forward/lib-python/2.7.0/test/test_code.py) ============================================================================== --- pypy/branch/fast-forward/lib-python/2.7.0/test/test_code.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_code.py Sat Oct 9 22:09:49 2010 @@ -82,7 +82,7 @@ import unittest import weakref -import _testcapi +from test import test_support def consts(t): @@ -104,7 +104,9 @@ class CodeTest(unittest.TestCase): + @test_support.impl_detail("test for PyCode_NewEmpty") def test_newempty(self): + import _testcapi co = _testcapi.code_newempty("filename", "funcname", 15) self.assertEquals(co.co_filename, "filename") self.assertEquals(co.co_name, "funcname") From afa at codespeak.net Sun Oct 10 00:27:55 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 00:27:55 +0200 (CEST) Subject: [pypy-svn] r77749 - in pypy/branch/fast-forward/pypy/module/_random: . test Message-ID: <20101009222755.A1E18282BAD@codespeak.net> Author: afa Date: Sun Oct 10 00:27:53 2010 New Revision: 77749 Modified: pypy/branch/fast-forward/pypy/module/_random/interp_random.py pypy/branch/fast-forward/pypy/module/_random/test/test_random.py Log: random.jumpahead() now accepts long integers Modified: pypy/branch/fast-forward/pypy/module/_random/interp_random.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_random/interp_random.py (original) +++ pypy/branch/fast-forward/pypy/module/_random/interp_random.py Sun Oct 10 00:27:53 2010 @@ -81,9 +81,14 @@ self._rnd.index = space.int_w(w_item) setstate.unwrap_spec = ['self', ObjSpace, W_Root] - def jumpahead(self, n): + def jumpahead(self, space, w_n): + if space.is_true(space.isinstance(w_n, space.w_long)): + num = space.bigint_w(w_n) + n = intmask(num.uintmask()) + else: + n = space.int_w(w_n) self._rnd.jumpahead(n) - jumpahead.unwrap_spec = ['self', int] + jumpahead.unwrap_spec = ['self', ObjSpace, W_Root] def getrandbits(self, space, k): if k <= 0: Modified: pypy/branch/fast-forward/pypy/module/_random/test/test_random.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_random/test/test_random.py (original) +++ pypy/branch/fast-forward/pypy/module/_random/test/test_random.py Sun Oct 10 00:27:53 2010 @@ -82,8 +82,14 @@ state2 = rnd.getstate() # seed() to improve the resolution) assert state1 != state2 + def test_jumpahead(self): + import sys + import _random + rnd = _random.Random() + rnd.jumpahead(100) + rnd.jumpahead(sys.maxint + 2) + def test_randbits(self): - import math import _random rnd = _random.Random() for n in range(1, 10) + range(10, 1000, 15): From afa at codespeak.net Sun Oct 10 00:55:51 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 00:55:51 +0200 (CEST) Subject: [pypy-svn] r77750 - pypy/branch/fast-forward/lib-python/modified-2.7.0 Message-ID: <20101009225551.CE1EC282BAD@codespeak.net> Author: afa Date: Sun Oct 10 00:55:49 2010 New Revision: 77750 Added: pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py - copied, changed from r77736, pypy/branch/fast-forward/lib-python/2.7.0/random.py Log: Use a more pythonic (and less cpython specific) way to test whether a method has been overriden Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py (from r77736, pypy/branch/fast-forward/lib-python/2.7.0/random.py) ============================================================================== --- pypy/branch/fast-forward/lib-python/2.7.0/random.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py Sun Oct 10 00:55:49 2010 @@ -41,7 +41,6 @@ from __future__ import division from warnings import warn as _warn -from types import MethodType as _MethodType, BuiltinMethodType as _BuiltinMethodType from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil from math import sqrt as _sqrt, acos as _acos, cos as _cos, sin as _sin from os import urandom as _urandom @@ -240,8 +239,7 @@ return self.randrange(a, b+1) - def _randbelow(self, n, _log=_log, int=int, _maxwidth=1L< n-1 > 2**(k-2) r = getrandbits(k) while r >= n: From afa at codespeak.net Sun Oct 10 01:56:18 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 01:56:18 +0200 (CEST) Subject: [pypy-svn] r77751 - in pypy/branch/fast-forward/pypy/rlib/rsre: . test Message-ID: <20101009235618.244B5282BAD@codespeak.net> Author: afa Date: Sun Oct 10 01:56:16 2010 New Revision: 77751 Modified: pypy/branch/fast-forward/pypy/rlib/rsre/rsre_char.py pypy/branch/fast-forward/pypy/rlib/rsre/test/test_char.py Log: CPython issue6561: fix characters matched by regexp r'\d' Modified: pypy/branch/fast-forward/pypy/rlib/rsre/rsre_char.py ============================================================================== --- pypy/branch/fast-forward/pypy/rlib/rsre/rsre_char.py (original) +++ pypy/branch/fast-forward/pypy/rlib/rsre/rsre_char.py Sun Oct 10 01:56:16 2010 @@ -80,7 +80,7 @@ def is_uni_digit(code): assert unicodedb is not None - return unicodedb.isdigit(code) + return unicodedb.isdecimal(code) def is_space(code): return code < 128 and (ascii_char_info[code] & 2 != 0) Modified: pypy/branch/fast-forward/pypy/rlib/rsre/test/test_char.py ============================================================================== --- pypy/branch/fast-forward/pypy/rlib/rsre/test/test_char.py (original) +++ pypy/branch/fast-forward/pypy/rlib/rsre/test/test_char.py Sun Oct 10 01:56:16 2010 @@ -8,10 +8,13 @@ UPPER_PI = 0x3a0 LOWER_PI = 0x3c0 INDIAN_DIGIT = 0x966 +ROMAN_NUMERAL = 0x2165 +FULLWIDTH_DIGIT = 0xff10 +CIRCLED_NUMBER = 0x32b4 +DINGBAT_CIRCLED = 0x2781 EM_SPACE = 0x2001 LINE_SEP = 0x2028 - # XXX very incomplete test def test_getlower(): @@ -105,3 +108,15 @@ assert cat(CHCODES["category_uni_not_linebreak"], ord(' ')) assert cat(CHCODES["category_uni_not_linebreak"], ord('s')) assert not cat(CHCODES["category_uni_not_linebreak"], LINE_SEP) + # + assert cat(CHCODES["category_uni_digit"], INDIAN_DIGIT) + assert cat(CHCODES["category_uni_digit"], FULLWIDTH_DIGIT) + assert not cat(CHCODES["category_uni_digit"], ROMAN_NUMERAL) + assert not cat(CHCODES["category_uni_digit"], CIRCLED_NUMBER) + assert not cat(CHCODES["category_uni_digit"], DINGBAT_CIRCLED) + # + assert not cat(CHCODES["category_uni_not_digit"], INDIAN_DIGIT) + assert not cat(CHCODES["category_uni_not_digit"], FULLWIDTH_DIGIT) + assert cat(CHCODES["category_uni_not_digit"], ROMAN_NUMERAL) + assert cat(CHCODES["category_uni_not_digit"], CIRCLED_NUMBER) + assert cat(CHCODES["category_uni_not_digit"], DINGBAT_CIRCLED) From afa at codespeak.net Sun Oct 10 10:46:22 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 10:46:22 +0200 (CEST) Subject: [pypy-svn] r77752 - pypy/branch/fast-forward/lib-python/modified-2.7.0 Message-ID: <20101010084622.1D5F736C22F@codespeak.net> Author: afa Date: Sun Oct 10 10:46:20 2010 New Revision: 77752 Added: pypy/branch/fast-forward/lib-python/modified-2.7.0/pprint.py - copied, changed from r77751, pypy/branch/fast-forward/lib-python/2.7.0/pprint.py Log: unlike CPython, PyPy has real "unbound builtin methods" objects and dict.__repr__ yields a different object each time. Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/pprint.py (from r77751, pypy/branch/fast-forward/lib-python/2.7.0/pprint.py) ============================================================================== --- pypy/branch/fast-forward/lib-python/2.7.0/pprint.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/pprint.py Sun Oct 10 10:46:20 2010 @@ -144,7 +144,7 @@ return r = getattr(typ, "__repr__", None) - if issubclass(typ, dict) and r is dict.__repr__: + if issubclass(typ, dict) and r == dict.__repr__: write('{') if self._indent_per_level > 1: write((self._indent_per_level - 1) * ' ') @@ -173,10 +173,10 @@ write('}') return - if ((issubclass(typ, list) and r is list.__repr__) or - (issubclass(typ, tuple) and r is tuple.__repr__) or - (issubclass(typ, set) and r is set.__repr__) or - (issubclass(typ, frozenset) and r is frozenset.__repr__) + if ((issubclass(typ, list) and r == list.__repr__) or + (issubclass(typ, tuple) and r == tuple.__repr__) or + (issubclass(typ, set) and r == set.__repr__) or + (issubclass(typ, frozenset) and r == frozenset.__repr__) ): length = _len(object) if issubclass(typ, list): @@ -266,7 +266,7 @@ return ("%s%s%s" % (closure, sio.getvalue(), closure)), True, False r = getattr(typ, "__repr__", None) - if issubclass(typ, dict) and r is dict.__repr__: + if issubclass(typ, dict) and r == dict.__repr__: if not object: return "{}", True, False objid = _id(object) @@ -291,8 +291,8 @@ del context[objid] return "{%s}" % _commajoin(components), readable, recursive - if (issubclass(typ, list) and r is list.__repr__) or \ - (issubclass(typ, tuple) and r is tuple.__repr__): + if (issubclass(typ, list) and r == list.__repr__) or \ + (issubclass(typ, tuple) and r == tuple.__repr__): if issubclass(typ, list): if not object: return "[]", True, False From hakanardo at codespeak.net Sun Oct 10 12:26:35 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Sun, 10 Oct 2010 12:26:35 +0200 (CEST) Subject: [pypy-svn] r77753 - in pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test Message-ID: <20101010102635.7FF85282BAD@codespeak.net> Author: hakanardo Date: Sun Oct 10 12:26:33 2010 New Revision: 77753 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Log: saner fail_args handling Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Sun Oct 10 12:26:33 2010 @@ -1,6 +1,7 @@ from pypy.jit.metainterp.optimizeopt.optimizer import * from pypy.jit.metainterp.resoperation import rop, ResOperation from pypy.jit.metainterp.compile import ResumeGuardDescr +from pypy.jit.metainterp.resume import Snapshot class OptUnroll(Optimization): """Unroll the loop into two itterations. The first one will @@ -14,14 +15,15 @@ if not self.enabled: self.emit_operation(op) return - + if op.getopnum() == rop.JUMP: loop = self.optimizer.loop loop.preamble.operations = self.optimizer.newoperations self.optimizer.newoperations = [] jump_args = op.getarglist() op.initarglist([]) - inputargs = self.inline(loop.preamble.operations + [op], + inputargs = self.inline(loop.operations, + #loop.preamble.operations + [op], loop.inputargs, jump_args) loop.inputargs = inputargs jmp = ResOperation(rop.JUMP, loop.inputargs[:], None) @@ -31,7 +33,7 @@ self.emit_operation(op) def inline(self, loop_operations, loop_args, jump_args): - argmap = {} + self.argmap = argmap = {} assert len(loop_args) == len(jump_args) for i in range(len(loop_args)): argmap[loop_args[i]] = jump_args[i] @@ -39,19 +41,17 @@ for v in self.optimizer.values.values(): v.fromstart = True + self.snapshot_map ={None: None} + inputargs = [] for arg in jump_args: for a in self.getvalue(arg).get_forced_boxes(): if not isinstance(a, Const): - inputargs.append(a) + inputargs.append(a) for op in loop_operations: newop = op.clone() - for i in range(op.numargs()): - a = op.getarg(i) - if not isinstance(a, Const): - newa = argmap[a] - newop.setarg(i, newa) + newop.initarglist([self.inline_arg(a) for a in newop.getarglist()]) if op.result: newop.result = op.result.clonebox() argmap[op.result] = newop.result @@ -59,7 +59,8 @@ if isinstance(descr, ResumeGuardDescr): op.getdescr().rd_snapshot = None #FIXME: In the right place? descr.rd_numb = None - + descr.rd_snapshot = self.inline_snapshot(descr.rd_snapshot) + if newop.getopnum() == rop.JUMP: args = [] #for arg in newop.getarglist(): @@ -67,19 +68,12 @@ args.extend(self.getvalue(arg).get_forced_boxes()) newop.initarglist(args + inputargs[len(args):]) + print "P: ", newop current = len(self.optimizer.newoperations) self.emit_operation(newop) for op in self.optimizer.newoperations[current:]: if op.is_guard(): - args = [] - for a in op._fail_args: - try: - a = argmap[a] - except KeyError: - pass - args.append(self.getvalue(a).force_box()) - op.setfailargs(args) op.getdescr().rd_snapshot = None #FIXME: In the right place? for a in op.getarglist(): @@ -90,7 +84,22 @@ return inputargs - + def inline_arg(self, arg): + if isinstance(arg, Const): + return arg + return self.argmap[arg] + + def inline_snapshot(self, snapshot): + if snapshot in self.snapshot_map: + return self.snapshot_map[snapshot] + boxes = [] + for a in snapshot.boxes: + if isinstance(a, Const): + boxes.append(a) + else: + boxes.append(self.inline_arg(a)) + return Snapshot(self.inline_snapshot(snapshot.prev), boxes) + Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py Sun Oct 10 12:26:33 2010 @@ -1348,7 +1348,7 @@ res = self.meta_interp(f, [10, 3]) assert res == 9 + 8 + 7 + 6 + 5 + 4 + 3 + 2 + 1 + 0 - self.check_tree_loop_count(1) + self.check_tree_loop_count(2) res = self.meta_interp(f, [10, 13]) assert res == 9 + 8 + 7 + 6 + 5 + 4 + 3 + 2 + 1 + 0 From david at codespeak.net Sun Oct 10 12:47:52 2010 From: david at codespeak.net (david at codespeak.net) Date: Sun, 10 Oct 2010 12:47:52 +0200 (CEST) Subject: [pypy-svn] r77754 - in pypy/branch/arm-backend/pypy/jit/backend/arm: . test Message-ID: <20101010104752.0F52C282BAD@codespeak.net> Author: david Date: Sun Oct 10 12:47:51 2010 New Revision: 77754 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Log: Implemtent INT_LE and GUARD_TRUE operations and make test_compile_loop pass Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Sun Oct 10 12:47:51 2010 @@ -1,5 +1,6 @@ from pypy.jit.backend.arm.codebuilder import ARMv7Builder from pypy.jit.backend.arm import registers as r +from pypy.jit.backend.arm import conditions as c #from pypy.jit.backend.arm.regalloc import RegAlloc, ARMRegisterManager from pypy.jit.metainterp.resoperation import rop from pypy.rpython.lltypesystem import lltype @@ -20,25 +21,43 @@ self.gen_func_prolog() addr = self.fail_boxes_int.get_addr_for_num(0) self.gen_load_int(r.r3, addr) - self.mc.LDR_ri(r.r2, r.r3) + self.mc.LDR_ri(r.r1, r.r3) + loop_head=self.mc.curraddr() + fcond=c.AL for op in operations: - if op.getopnum() == rop.INT_ADD: - self.mc.ADD_ri(r.r1, r.r2, op.getarg(1).getint()) - elif op.getopnum() == rop.FINISH: + opnum = op.getopnum() + if opnum == rop.INT_ADD: + self.mc.ADD_ri(r.r1, r.r1, op.getarg(1).getint()) + elif opnum == rop.INT_LE: + self.mc.CMP(r.r1, op.getarg(1).getint()) + fcond = c.GT + elif opnum == rop.GUARD_TRUE: + n = self.cpu.get_fail_descr_number(op.getdescr()) + self.mc.MOV_ri(r.r0, n, cond=fcond) + self.mc.STR_ri(r.r1, r.r3, cond=fcond) + self.gen_func_epilog(cond=fcond) + fcond = c.AL + elif opnum == rop.JUMP: + self.gen_load_int(r.r7, loop_head) + self.mc.MOV_rr(r.pc, r.r7) + elif opnum == rop.FINISH: n = self.cpu.get_fail_descr_number(op.getdescr()) self.mc.MOV_ri(r.r0, n) self.mc.STR_ri(r.r1, r.r3) - self.gen_func_epilog() + else: + raise ValueError("Unknown op %r" % op) + self.gen_func_epilog() - def gen_func_epilog(self): - self.mc.LDM(r.sp, r.callee_restored_registers) + def gen_func_epilog(self,cond=c.AL): + self.mc.LDM(r.sp, r.callee_restored_registers, cond=cond) def gen_func_prolog(self): self.mc.PUSH(r.callee_saved_registers) - def gen_load_int(self, reg, value): - self.mc.MOV_ri(reg, (value & 0xFF)) + def gen_load_int(self, reg, value, cond=c.AL): + assert reg != r.ip, 'ip is used to load int' + self.mc.MOV_ri(reg, (value & 0xFF), cond=cond) for offset in range(8, 25, 8): - self.mc.MOV_ri(r.ip, (value >> offset) & 0xFF) - self.mc.ORR_rr(reg, reg, r.ip, offset) + self.mc.MOV_ri(r.ip, (value >> offset) & 0xFF, cond=cond) + self.mc.ORR_rr(reg, reg, r.ip, offset, cond=cond) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py Sun Oct 10 12:47:51 2010 @@ -8,13 +8,6 @@ self._data = alloc(1024) self._pos = 0 - def _encode_imm(self, imm): - u = 1 - if imm < 0: - u = 0 - imm = -imm - return u, imm - def LDR_ri(self, rt, rn, imm=0, cond=cond.AL): # XXX W and P bits are not encoded yet p = 1 @@ -109,11 +102,27 @@ instr = self._encode_reg_list(instr, regs) self.write32(instr) + def CMP(self, rn, imm, cond=cond.AL): + if 0 <= imm <= 255: + self.write32(cond << 28 + | 0x35 << 20 + | (rn & 0xFF) << 16 + | (imm & 0xFFF)) + else: + raise NotImplentedError + def _encode_reg_list(self, instr, regs): for reg in regs: instr |= 0x1 << reg return instr + def _encode_imm(self, imm): + u = 1 + if imm < 0: + u = 0 + imm = -imm + return u, imm + def write32(self, word): self.writechar(chr(word & 0xFF)) self.writechar(chr((word >> 8) & 0xFF)) @@ -127,4 +136,7 @@ def baseaddr(self): return rffi.cast(lltype.Signed, self._data) + def curraddr(self): + return self.baseaddr() + self._pos + Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py Sun Oct 10 12:47:51 2010 @@ -1,4 +1,5 @@ import os +import py from pypy.rpython.lltypesystem import lltype, rffi @@ -15,7 +16,6 @@ return func() def skip_unless_arm(): - import py check_skip(os.uname()[4]) def requires_arm_as(): Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py Sun Oct 10 12:47:51 2010 @@ -1,4 +1,5 @@ from pypy.jit.backend.arm import registers as r +from pypy.jit.backend.arm import conditions as c from pypy.jit.backend.arm.assembler import AssemblerARM from pypy.jit.backend.arm.test.support import skip_unless_arm, run_asm @@ -41,3 +42,47 @@ self.a.mc.SUB_ri(r.r0, r.r1, 123) self.a.gen_func_epilog() assert run_asm(self.a) == 123333 + + def test_int_le(self): + self.a.gen_func_prolog() + self.a.gen_load_int(r.r1, 22) + self.a.mc.CMP(r.r1, 123) + self.a.mc.MOV_ri(r.r0, 1, c.LE) + self.a.mc.MOV_ri(r.r0, 0, c.GT) + self.a.gen_func_epilog() + assert run_asm(self.a) == 1 + + def test_int_le_false(self): + self.a.gen_func_prolog() + self.a.gen_load_int(r.r1, 2222) + self.a.mc.CMP(r.r1, 123) + self.a.mc.MOV_ri(r.r0, 1, c.LE) + self.a.mc.MOV_ri(r.r0, 0, c.GT) + self.a.gen_func_epilog() + assert run_asm(self.a) == 0 + + def test_simple_jump(self): + self.a.gen_func_prolog() + self.a.mc.MOV_ri(r.r1, 1) + loop_head = self.a.mc.curraddr() + self.a.mc.CMP(r.r1, 0) # z=0, z=1 + self.a.mc.MOV_ri(r.r1, 0, cond=c.NE) + self.a.mc.MOV_ri(r.r1, 7, cond=c.EQ) + self.a.gen_load_int(r.r4, loop_head, cond=c.NE) + self.a.mc.MOV_rr(r.pc, r.r4, cond=c.NE) + self.a.mc.MOV_rr(r.r0, r.r1) + self.a.gen_func_epilog() + assert run_asm(self.a) == 7 + + def test_jump(self): + self.a.gen_func_prolog() + self.a.mc.MOV_ri(r.r1, 1) + loop_head = self.a.mc.curraddr() + self.a.mc.ADD_ri(r.r1, r.r1, 1) + self.a.mc.CMP(r.r1, 9) + self.a.gen_load_int(r.r4, loop_head, cond=c.NE) + self.a.mc.MOV_rr(r.pc, r.r4, cond=c.NE) + self.a.mc.MOV_rr(r.r0, r.r1) + self.a.gen_func_epilog() + assert run_asm(self.a) == 9 + Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py Sun Oct 10 12:47:51 2010 @@ -106,6 +106,10 @@ self.cb.SUB_ri(r.r3, r.r7, 0xFFF) self.assert_equal('SUB r3, r7, #4095') + def test_cmp_ri(self): + self.cb.CMP(r.r3, 123) + self.assert_equal('CMP r3, #123') + def assert_equal(self, asm): assert self.cb.hexdump() == assemble(asm) From david at codespeak.net Sun Oct 10 12:55:14 2010 From: david at codespeak.net (david at codespeak.net) Date: Sun, 10 Oct 2010 12:55:14 +0200 (CEST) Subject: [pypy-svn] r77755 - pypy/branch/arm-backend/pypy/jit/backend/arm Message-ID: <20101010105514.4C676282BAD@codespeak.net> Author: david Date: Sun Oct 10 12:55:12 2010 New Revision: 77755 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py Log: Store address of machine code in looptoken and read it from there Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Sun Oct 10 12:55:12 2010 @@ -47,6 +47,8 @@ else: raise ValueError("Unknown op %r" % op) self.gen_func_epilog() + looptoken._arm_bootstrap_code = self.mc.baseaddr() + looptoken._arm_loop_code = loop_head def gen_func_epilog(self,cond=c.AL): self.mc.LDM(r.sp, r.callee_restored_registers, cond=cond) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py Sun Oct 10 12:55:12 2010 @@ -25,7 +25,7 @@ return self.assembler.fail_boxes_int.getitem(index) def execute_token(self, executable_token): - addr = self.assembler.mc.baseaddr()#executable_token._arm_bootstrap_code + addr = executable_token._arm_bootstrap_code assert addr % 8 == 0 func = rffi.cast(lltype.Ptr(self.BOOTSTRAP_TP), addr) fail_index = self._execute_call(func) From afa at codespeak.net Sun Oct 10 13:32:25 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 13:32:25 +0200 (CEST) Subject: [pypy-svn] r77756 - in pypy/branch/fast-forward/lib_pypy: . pypy_test Message-ID: <20101010113225.0BDF0282BAD@codespeak.net> Author: afa Date: Sun Oct 10 13:32:23 2010 New Revision: 77756 Modified: pypy/branch/fast-forward/lib_pypy/_collections.py pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Log: Add deque.maxlen Modified: pypy/branch/fast-forward/lib_pypy/_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/_collections.py Sun Oct 10 13:32:23 2010 @@ -18,6 +18,12 @@ RGTLNK = n+1 BLOCKSIZ = n+2 +# The deque's size limit is d.maxlen. The limit can be zero or positive, or +# None. After an item is added to a deque, we check to see if the size has +# grown past the limit. If it has, we get the size back down to the limit by +# popping an item off of the opposite end. The methods that can trigger this +# are append(), appendleft(), extend(), and extendleft(). + class deque(object): def __new__(cls, iterable=(), *args, **kw): @@ -25,7 +31,11 @@ self.clear() return self - def __init__(self, iterable=()): + def __init__(self, iterable=(), maxlen=None): + if maxlen is not None: + if maxlen < 0: + raise ValueError("maxlen must be non-negative") + self.maxlen = maxlen add = self.append for elem in iterable: add(elem) @@ -48,6 +58,8 @@ self.rightndx = 0 self.length += 1 self.right[self.rightndx] = x + if self.maxlen is not None and self.length > self.maxlen: + self.popleft() def appendleft(self, x): self.state += 1 @@ -60,6 +72,8 @@ self.leftndx = n-1 self.length += 1 self.left[self.leftndx] = x + if self.maxlen is not None and self.length > self.maxlen: + self.pop() def extend(self, iterable): for elem in iterable: @@ -144,7 +158,10 @@ else: self.__dict__[threadlocalattr] = True try: - return 'deque(%r)' % (list(self),) + if self.maxlen is not None: + return 'deque(%r, maxlen=%s)' % (list(self), self.maxlen) + else: + return 'deque(%r)' % (list(self),) finally: del self.__dict__[threadlocalattr] @@ -249,13 +266,13 @@ self.rotate(-index) def __reduce_ex__(self, proto): - return type(self), (), self.__dict__, iter(self), None + return type(self), (list(self), self.maxlen) def __hash__(self): raise TypeError, "deque objects are unhashable" def __copy__(self): - return self.__class__(self) + return self.__class__(self, self.maxlen) # XXX make comparison more efficient def __eq__(self, other): Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Sun Oct 10 13:32:23 2010 @@ -14,7 +14,21 @@ d = collections.deque([MutatingCmp()]) py.test.raises(IndexError, d.remove, 1) - + +def test_deque_maxlen(): + d = collections.deque([], 3) + d.append(1); d.append(2); d.append(3); d.append(4) + assert list(d) == [2, 3, 4] + assert repr(d) == "deque([2, 3, 4], maxlen=3)" + + import pickle + d2 = pickle.loads(pickle.dumps(d)) + assert repr(d2) == "deque([2, 3, 4], maxlen=3)" + + import copy + d3 = copy.copy(d) + assert repr(d3) == "deque([2, 3, 4], maxlen=3)" + class SubclassWithKwargs(collections.deque): def __init__(self, newarg=1): collections.deque.__init__(self) From afa at codespeak.net Sun Oct 10 13:36:09 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 13:36:09 +0200 (CEST) Subject: [pypy-svn] r77757 - in pypy/branch/fast-forward/lib_pypy: . pypy_test Message-ID: <20101010113609.B456736C230@codespeak.net> Author: afa Date: Sun Oct 10 13:36:08 2010 New Revision: 77757 Modified: pypy/branch/fast-forward/lib_pypy/_collections.py pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Log: Add deque.count() Modified: pypy/branch/fast-forward/lib_pypy/_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/_collections.py Sun Oct 10 13:36:08 2010 @@ -125,6 +125,13 @@ self.leftndx = 0 return x + def count(self, value): + c = 0 + for i in range(len(self)): + if self[i] == value: + c += 1 + return c + def remove(self, value): # Need to be defensive for mutating comparisons for i in range(len(self)): Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Sun Oct 10 13:36:08 2010 @@ -29,6 +29,11 @@ d3 = copy.copy(d) assert repr(d3) == "deque([2, 3, 4], maxlen=3)" +def test_deque_count(): + d = collections.deque([1, 2, 2, 3, 2]) + assert d.count(2) == 3 + assert d.count(4) == 0 + class SubclassWithKwargs(collections.deque): def __init__(self, newarg=1): collections.deque.__init__(self) From afa at codespeak.net Sun Oct 10 13:38:01 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 13:38:01 +0200 (CEST) Subject: [pypy-svn] r77758 - pypy/branch/fast-forward/lib_pypy/pypy_test Message-ID: <20101010113801.8DC1936C230@codespeak.net> Author: afa Date: Sun Oct 10 13:37:59 2010 New Revision: 77758 Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Log: Move all tests into a class Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Sun Oct 10 13:37:59 2010 @@ -2,42 +2,43 @@ from .. import _collections as collections import py -def test_deque_remove_empty(): - d = collections.deque([]) - py.test.raises(ValueError, d.remove, 1) - -def test_deque_remove_mutating(): - class MutatingCmp(object): - def __eq__(self, other): - d.clear() - return True - - d = collections.deque([MutatingCmp()]) - py.test.raises(IndexError, d.remove, 1) - -def test_deque_maxlen(): - d = collections.deque([], 3) - d.append(1); d.append(2); d.append(3); d.append(4) - assert list(d) == [2, 3, 4] - assert repr(d) == "deque([2, 3, 4], maxlen=3)" - - import pickle - d2 = pickle.loads(pickle.dumps(d)) - assert repr(d2) == "deque([2, 3, 4], maxlen=3)" - - import copy - d3 = copy.copy(d) - assert repr(d3) == "deque([2, 3, 4], maxlen=3)" - -def test_deque_count(): - d = collections.deque([1, 2, 2, 3, 2]) - assert d.count(2) == 3 - assert d.count(4) == 0 - -class SubclassWithKwargs(collections.deque): - def __init__(self, newarg=1): - collections.deque.__init__(self) - -def test_subclass_with_kwargs(): - # SF bug #1486663 -- this used to erroneously raise a TypeError - SubclassWithKwargs(newarg=1) +class TestDeque: + def test_remove_empty(self): + d = collections.deque([]) + py.test.raises(ValueError, d.remove, 1) + + def test_remove_mutating(self): + class MutatingCmp(object): + def __eq__(self, other): + d.clear() + return True + + d = collections.deque([MutatingCmp()]) + py.test.raises(IndexError, d.remove, 1) + + def test_maxlen(self): + d = collections.deque([], 3) + d.append(1); d.append(2); d.append(3); d.append(4) + assert list(d) == [2, 3, 4] + assert repr(d) == "deque([2, 3, 4], maxlen=3)" + + import pickle + d2 = pickle.loads(pickle.dumps(d)) + assert repr(d2) == "deque([2, 3, 4], maxlen=3)" + + import copy + d3 = copy.copy(d) + assert repr(d3) == "deque([2, 3, 4], maxlen=3)" + + def test_count(self): + d = collections.deque([1, 2, 2, 3, 2]) + assert d.count(2) == 3 + assert d.count(4) == 0 + + def test_subclass_with_kwargs(self): + class SubclassWithKwargs(collections.deque): + def __init__(self, newarg=1): + collections.deque.__init__(self) + + # SF bug #1486663 -- this used to erroneously raise a TypeError + SubclassWithKwargs(newarg=1) From afa at codespeak.net Sun Oct 10 13:51:34 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 13:51:34 +0200 (CEST) Subject: [pypy-svn] r77759 - in pypy/branch/fast-forward/lib_pypy: . pypy_test Message-ID: <20101010115134.240DF282BAD@codespeak.net> Author: afa Date: Sun Oct 10 13:51:31 2010 New Revision: 77759 Modified: pypy/branch/fast-forward/lib_pypy/_collections.py pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Log: implement deque.reverse() Modified: pypy/branch/fast-forward/lib_pypy/_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/_collections.py Sun Oct 10 13:51:31 2010 @@ -158,6 +158,34 @@ self.append(self.popleft()) n += 1 + def reverse(self): + "reverse *IN PLACE*" + leftblock = self.left + rightblock = self.right + leftindex = self.leftndx + rightindex = self.rightndx + for i in range(self.length // 2): + # Validate that pointers haven't met in the middle + assert leftblock != rightblock or leftindex < rightindex + + # Swap + (self.right[rightindex], self.left[leftindex]) = ( + self.left[leftindex], self.right[rightindex]) + + # Advance left block/index pair + leftindex += 1 + if leftindex == n: + leftblock = leftblock[RGTLNK] + assert leftblock is not None + leftindex = 0 + + # Step backwards with the right block/index pair + rightindex -= 1 + if rightindex == -1: + rightblock = leftblock[LFTLNK] + assert rightblock is not None + leftindex = n - 1 + def __repr__(self): threadlocalattr = '__repr' + str(_thread_ident()) if threadlocalattr in self.__dict__: Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Sun Oct 10 13:51:31 2010 @@ -35,6 +35,11 @@ assert d.count(2) == 3 assert d.count(4) == 0 + def test_reverse(self): + d = collections.deque([1, 2, 2, 3, 2]) + d.reverse() + assert list(d) == [2, 3, 2, 2, 1] + def test_subclass_with_kwargs(self): class SubclassWithKwargs(collections.deque): def __init__(self, newarg=1): From afa at codespeak.net Sun Oct 10 14:14:03 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 14:14:03 +0200 (CEST) Subject: [pypy-svn] r77760 - in pypy/branch/fast-forward/lib_pypy: . pypy_test Message-ID: <20101010121403.C0291282BDD@codespeak.net> Author: afa Date: Sun Oct 10 14:14:02 2010 New Revision: 77760 Modified: pypy/branch/fast-forward/lib_pypy/_collections.py pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Log: deque.reverse() was completely wrong for larger sizes Modified: pypy/branch/fast-forward/lib_pypy/_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/_collections.py Sun Oct 10 14:14:02 2010 @@ -169,8 +169,8 @@ assert leftblock != rightblock or leftindex < rightindex # Swap - (self.right[rightindex], self.left[leftindex]) = ( - self.left[leftindex], self.right[rightindex]) + (rightblock[rightindex], leftblock[leftindex]) = ( + leftblock[leftindex], rightblock[rightindex]) # Advance left block/index pair leftindex += 1 @@ -182,9 +182,9 @@ # Step backwards with the right block/index pair rightindex -= 1 if rightindex == -1: - rightblock = leftblock[LFTLNK] + rightblock = rightblock[LFTLNK] assert rightblock is not None - leftindex = n - 1 + rightindex = n - 1 def __repr__(self): threadlocalattr = '__repr' + str(_thread_ident()) Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_collections.py Sun Oct 10 14:14:02 2010 @@ -40,6 +40,10 @@ d.reverse() assert list(d) == [2, 3, 2, 2, 1] + d = collections.deque(range(100)) + d.reverse() + assert list(d) == range(99, -1, -1) + def test_subclass_with_kwargs(self): class SubclassWithKwargs(collections.deque): def __init__(self, newarg=1): From afa at codespeak.net Sun Oct 10 18:39:28 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 18:39:28 +0200 (CEST) Subject: [pypy-svn] r77761 - in pypy/branch/fast-forward/pypy/objspace/std: . test Message-ID: <20101010163928.DC21A282BDD@codespeak.net> Author: afa Date: Sun Oct 10 18:39:24 2010 New Revision: 77761 Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py pypy/branch/fast-forward/pypy/objspace/std/typeobject.py Log: Declare an __eq__ slot for type objects. Later, __lt__ will have to raise a py3k warning. Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py Sun Oct 10 18:39:24 2010 @@ -856,6 +856,22 @@ assert Abc.__name__ == 'Def' raises(TypeError, "Abc.__name__ = 42") + def test_compare(self): + class A(object): + pass + class B(A): + pass + A.__eq__ + A.__ne__ + assert A.__eq__(A) + assert not A.__eq__(B) + assert A.__ne__(B) + assert not A.__ne__(A) + assert A == A + assert A != B + assert not A == B + assert not A != A + def test_class_variations(self): class A(object): pass Modified: pypy/branch/fast-forward/pypy/objspace/std/typeobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/typeobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/typeobject.py Sun Oct 10 18:39:24 2010 @@ -790,6 +790,9 @@ w_type.mutated() w_type.dict_w[name] = w_value +def eq__Type_Type(space, w_self, w_other): + return space.is_(w_self, w_other) + def delattr__Type_ANY(space, w_type, w_name): if w_type.lazyloaders: w_type._freeze_() # force un-lazification From hakanardo at codespeak.net Sun Oct 10 19:06:17 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Sun, 10 Oct 2010 19:06:17 +0200 (CEST) Subject: [pypy-svn] r77762 - in pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test Message-ID: <20101010170617.6AD31282BDD@codespeak.net> Author: hakanardo Date: Sun Oct 10 19:06:15 2010 New Revision: 77762 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py Log: test_basic.py and test_virtual.py probably ok Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py Sun Oct 10 19:06:15 2010 @@ -209,6 +209,7 @@ # the following assert should always be true for now, # because invalid loops that would fail it are detected # earlier, in optimizefindnode.py. + raise InvalidLoop assert realclassbox.same_constant(expectedclassbox) return emit_operation = True Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Sun Oct 10 19:06:15 2010 @@ -19,6 +19,7 @@ if op.getopnum() == rop.JUMP: loop = self.optimizer.loop loop.preamble.operations = self.optimizer.newoperations + print '\n'.join([str(o) for o in loop.preamble.operations]) self.optimizer.newoperations = [] jump_args = op.getarglist() op.initarglist([]) @@ -39,7 +40,8 @@ argmap[loop_args[i]] = jump_args[i] for v in self.optimizer.values.values(): - v.fromstart = True + if not v.is_constant() and v.box: + v.fromstart = True self.snapshot_map ={None: None} @@ -47,9 +49,11 @@ for arg in jump_args: for a in self.getvalue(arg).get_forced_boxes(): if not isinstance(a, Const): - inputargs.append(a) - + inputargs.append(a) + print "Inputargs: ", inputargs + for op in loop_operations: + #import pdb; pdb.set_trace() newop = op.clone() newop.initarglist([self.inline_arg(a) for a in newop.getarglist()]) if op.result: @@ -73,14 +77,20 @@ self.emit_operation(newop) for op in self.optimizer.newoperations[current:]: + print "E: ", op if op.is_guard(): op.getdescr().rd_snapshot = None #FIXME: In the right place? - - for a in op.getarglist(): + args = op.getarglist() + if op.is_guard(): + args = args + op.getfailargs() + for a in args: if not isinstance(a, Const) and a in self.optimizer.values: v = self.getvalue(a) if v.fromstart and a not in inputargs: + print "Arg: ", a inputargs.append(a) + if op.getopnum() == rop.JUMP: + op.initarglist(op.getarglist() + [argmap[a]]) return inputargs Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py Sun Oct 10 19:06:15 2010 @@ -122,11 +122,14 @@ fieldvalue.get_args_for_fail(modifier) def get_forced_boxes(self): - lst = self._get_field_descr_list() - fieldboxes = [] - for ofs in lst: - fieldboxes.extend(self._fields[ofs].get_forced_boxes()) - return fieldboxes + if self.box is None: + lst = self._get_field_descr_list() + fieldboxes = [] + for ofs in lst: + fieldboxes.extend(self._fields[ofs].get_forced_boxes()) + return fieldboxes + else: + return [self.box] class VirtualValue(AbstractVirtualStructValue): Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py Sun Oct 10 19:06:15 2010 @@ -49,6 +49,24 @@ res = self.meta_interp(f, [10]) assert res == f(10) self.check_loop_count(1) + self.check_loops(new=0, float_add=0) + + def test_virtualized_float2(self): + myjitdriver = JitDriver(greens = [], reds = ['n', 'node']) + def f(n): + node = self._new() + node.floatval = 0.0 + while n > 0: + myjitdriver.can_enter_jit(n=n, node=node) + myjitdriver.jit_merge_point(n=n, node=node) + next = self._new() + next.floatval = node.floatval + .5 + node = next + n -= 1 + return node.floatval + res = self.meta_interp(f, [10]) + assert res == f(10) + self.check_loop_count(1) self.check_loops(new=0, float_add=1) def test_virtualized_2(self): @@ -143,10 +161,10 @@ node = next n -= 1 return node.value - res = self.meta_interp(f, [11], policy=StopAtXPolicy(externfn)) - assert res == f(11) + res = self.meta_interp(f, [15], policy=StopAtXPolicy(externfn)) + assert res == f(15) self.check_loop_count(2) - self.check_loops(**{self._new_op: 2}) # XXX was 1 + self.check_loops(**{self._new_op: 1}) self.check_loops(int_mul=0, call=1) def test_two_virtuals(self): From afa at codespeak.net Sun Oct 10 20:16:31 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 20:16:31 +0200 (CEST) Subject: [pypy-svn] r77764 - in pypy/branch/fast-forward/pypy/objspace/std: . test Message-ID: <20101010181631.443D9282BDD@codespeak.net> Author: afa Date: Sun Oct 10 20:16:29 2010 New Revision: 77764 Modified: pypy/branch/fast-forward/pypy/objspace/std/slicetype.py pypy/branch/fast-forward/pypy/objspace/std/test/test_sliceobject.py Log: Make slices picklable Modified: pypy/branch/fast-forward/pypy/objspace/std/slicetype.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/slicetype.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/slicetype.py Sun Oct 10 20:16:29 2010 @@ -72,6 +72,15 @@ descr__new__.unwrap_spec = [baseobjspace.ObjSpace, baseobjspace.W_Root, 'args_w'] +def descr__reduce__(space, w_self): + return space.newtuple([ + space.type(w_self), + space.newtuple([w_self.w_start, + w_self.w_stop, + w_self.w_step]), + ]) +descr__reduce__.unwrap_spec = [baseobjspace.ObjSpace, baseobjspace.W_Root] + # ____________________________________________________________ def slicewprop(name): @@ -90,6 +99,7 @@ Create a slice object. This is used for extended slicing (e.g. a[0:10:2]).''', __new__ = gateway.interp2app(descr__new__), __hash__ = None, + __reduce__ = gateway.interp2app(descr__reduce__), start = slicewprop('w_start'), stop = slicewprop('w_stop'), step = slicewprop('w_step'), Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_sliceobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/test/test_sliceobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/test/test_sliceobject.py Sun Oct 10 20:16:29 2010 @@ -93,3 +93,6 @@ assert stop == 1000 assert step >= 1000 raises(OverflowError, "slice(0, 1000, 1).indices(2 ** 100)") + + def test_reduce(self): + assert slice(1, 2, 3).__reduce__() == (slice, (1, 2, 3)) From afa at codespeak.net Sun Oct 10 20:23:58 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 20:23:58 +0200 (CEST) Subject: [pypy-svn] r77765 - pypy/branch/fast-forward/lib_pypy Message-ID: <20101010182358.8F76E282BDD@codespeak.net> Author: afa Date: Sun Oct 10 20:23:56 2010 New Revision: 77765 Modified: pypy/branch/fast-forward/lib_pypy/_collections.py Log: Avoid quadratic behavior in deque.count(). Thanks Trundle! Modified: pypy/branch/fast-forward/lib_pypy/_collections.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_collections.py (original) +++ pypy/branch/fast-forward/lib_pypy/_collections.py Sun Oct 10 20:23:56 2010 @@ -127,8 +127,8 @@ def count(self, value): c = 0 - for i in range(len(self)): - if self[i] == value: + for item in self: + if item == value: c += 1 return c From afa at codespeak.net Sun Oct 10 21:16:47 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 21:16:47 +0200 (CEST) Subject: [pypy-svn] r77766 - in pypy/branch/fast-forward/pypy/objspace/std: . test Message-ID: <20101010191647.C0221282BDD@codespeak.net> Author: afa Date: Sun Oct 10 21:16:09 2010 New Revision: 77766 Modified: pypy/branch/fast-forward/pypy/objspace/std/sliceobject.py pypy/branch/fast-forward/pypy/objspace/std/test/test_sliceobject.py Log: cpython issue3004 slightly changed the behavior of slice.indices() Modified: pypy/branch/fast-forward/pypy/objspace/std/sliceobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/sliceobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/sliceobject.py Sun Oct 10 21:16:09 2010 @@ -59,9 +59,15 @@ if stop < 0: stop += length if stop < 0: - stop =-1 - elif stop > length: - stop = length + if step < 0: + stop = -1 + else: + stop = 0 + elif stop >= length: + if step < 0: + stop = length - 1 + else: + stop = length return start, stop, step def indices4(w_slice, space, length): Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_sliceobject.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/test/test_sliceobject.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/test/test_sliceobject.py Sun Oct 10 21:16:09 2010 @@ -60,7 +60,11 @@ assert slice(4,11,2).indices(2) == (2, 2, 2) assert slice(11,4,-2).indices(28) == (11, 4, -2) assert slice(11,4,-2).indices(8) == (7, 4, -2) - assert slice(11,4,-2).indices(2) == (1, 2, -2) + assert slice(11,4,-2).indices(2) == (1, 1, -2) + assert slice(None, -9).indices(10) == (0, 1, 1) + assert slice(None, -10, -1).indices(10) == (9, 0, -1) + assert slice(None, 10, -1).indices(10) == (9, 9, -1) + def test_repr(self): assert repr(slice(1, 2, 3)) == 'slice(1, 2, 3)' @@ -85,7 +89,7 @@ def test_long_indices(self): assert slice(-2 ** 100, 10, 1).indices(1000) == (0, 10, 1) - assert slice(-2 ** 200, -2 ** 100, 1).indices(1000) == (0, -1, 1) + assert slice(-2 ** 200, -2 ** 100, 1).indices(1000) == (0, 0, 1) assert slice(2 ** 100, 0, -1).indices(1000) == (999, 0, -1) assert slice(2 ** 100, -2 ** 100, -1).indices(1000) == (999, -1, -1) start, stop, step = slice(0, 1000, 2 ** 200).indices(1000) From afa at codespeak.net Sun Oct 10 21:56:03 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 21:56:03 +0200 (CEST) Subject: [pypy-svn] r77767 - pypy/branch/fast-forward/lib-python/modified-2.7.0/test Message-ID: <20101010195603.5A59A282BDD@codespeak.net> Author: afa Date: Sun Oct 10 21:56:01 2010 New Revision: 77767 Added: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_scope.py - copied, changed from r77766, pypy/branch/fast-forward/lib-python/2.7.0/test/test_scope.py Log: gc.collect() before checking for reference leaks Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_scope.py (from r77766, pypy/branch/fast-forward/lib-python/2.7.0/test/test_scope.py) ============================================================================== --- pypy/branch/fast-forward/lib-python/2.7.0/test/test_scope.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_scope.py Sun Oct 10 21:56:01 2010 @@ -1,6 +1,6 @@ import unittest from test.test_support import check_syntax_error, check_py3k_warnings, \ - check_warnings, run_unittest + check_warnings, run_unittest, gc_collect class ScopeTests(unittest.TestCase): @@ -432,6 +432,7 @@ for i in range(100): f1() + gc_collect() self.assertEqual(Foo.count, 0) From afa at codespeak.net Sun Oct 10 22:32:58 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 22:32:58 +0200 (CEST) Subject: [pypy-svn] r77768 - in pypy/branch/fast-forward/pypy/interpreter/astcompiler: . test Message-ID: <20101010203258.23E5C282BDD@codespeak.net> Author: afa Date: Sun Oct 10 22:32:56 2010 New Revision: 77768 Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/symtable.py pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_symtable.py Log: Add syntax warning: "import * only allowed at module level" Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/symtable.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/symtable.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/symtable.py Sun Oct 10 22:32:56 2010 @@ -237,6 +237,7 @@ def note_import_star(self, imp): self.optimized = False self.import_star = imp + return True def note_variable_arg(self, vararg): self.has_variable_arg = True @@ -382,7 +383,11 @@ def visit_ImportFrom(self, imp): for alias in imp.names: if self._visit_alias(alias): - self.scope.note_import_star(imp) + if self.scope.note_import_star(imp): + msg = "import * only allowed at module level" + misc.syntax_warning( + self.space, msg, self.compile_info.filename, + imp.lineno, imp.col_offset) def _visit_alias(self, alias): assert isinstance(alias, ast.alias) Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_symtable.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_symtable.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_symtable.py Sun Oct 10 22:32:56 2010 @@ -309,6 +309,17 @@ exc = py.test.raises(SyntaxError, self.mod_scope, input).value assert exc.msg == error + " contains a nested function with free variables" + def test_importstar_warning(self, capfd): + self.mod_scope("def f():\n from re import *") + _, err1 = capfd.readouterr() + + self.mod_scope("if 1:\n from re import *") + _, err2 = capfd.readouterr() + + capfd.close() + assert "import * only allowed at module level" in err1 + assert not "import * only allowed at module level" in err2 + def test_exec(self): self.mod_scope("exec 'hi'") scp = self.func_scope("def f(): exec 'hi'") From afa at codespeak.net Sun Oct 10 23:11:36 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Sun, 10 Oct 2010 23:11:36 +0200 (CEST) Subject: [pypy-svn] r77769 - pypy/branch/fast-forward/pypy/objspace/std Message-ID: <20101010211136.5A79E282BDD@codespeak.net> Author: afa Date: Sun Oct 10 23:11:31 2010 New Revision: 77769 Modified: pypy/branch/fast-forward/pypy/objspace/std/slicetype.py Log: Fix translation Modified: pypy/branch/fast-forward/pypy/objspace/std/slicetype.py ============================================================================== --- pypy/branch/fast-forward/pypy/objspace/std/slicetype.py (original) +++ pypy/branch/fast-forward/pypy/objspace/std/slicetype.py Sun Oct 10 23:11:31 2010 @@ -73,6 +73,8 @@ 'args_w'] def descr__reduce__(space, w_self): + from pypy.objspace.std.sliceobject import W_SliceObject + assert isinstance(w_self, W_SliceObject) return space.newtuple([ space.type(w_self), space.newtuple([w_self.w_start, From benjamin at codespeak.net Mon Oct 11 03:33:07 2010 From: benjamin at codespeak.net (benjamin at codespeak.net) Date: Mon, 11 Oct 2010 03:33:07 +0200 (CEST) Subject: [pypy-svn] r77770 - pypy/branch/fast-forward/lib-python/modified-2.7.0 Message-ID: <20101011013307.C179D282BDD@codespeak.net> Author: benjamin Date: Mon Oct 11 03:33:05 2010 New Revision: 77770 Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py Log: compare methods with 'is' Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py ============================================================================== --- pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py Mon Oct 11 03:33:05 2010 @@ -254,8 +254,8 @@ # Only call self.getrandbits if the original random() builtin method # has not been overridden or if a new getrandbits() was supplied. # This assures that the two methods correspond. - if (self.random == super(Random, self).random or - getrandbits != super(Random, self).getrandbits): + if (self.random is super(Random, self).random or + getrandbits is not super(Random, self).getrandbits): k = int(1.00001 + _log(n-1, 2.0)) # 2**k > n-1 > 2**(k-2) r = getrandbits(k) while r >= n: From afa at codespeak.net Mon Oct 11 08:26:13 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 08:26:13 +0200 (CEST) Subject: [pypy-svn] r77771 - pypy/branch/fast-forward/pypy/interpreter/astcompiler Message-ID: <20101011062613.25796282BF3@codespeak.net> Author: afa Date: Mon Oct 11 08:26:11 2010 New Revision: 77771 Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/symtable.py Log: Another translation fix Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/symtable.py ============================================================================== --- pypy/branch/fast-forward/pypy/interpreter/astcompiler/symtable.py (original) +++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/symtable.py Mon Oct 11 08:26:11 2010 @@ -90,7 +90,7 @@ def note_import_star(self, imp): """Called when a star import is found.""" - pass + return False def mangle(self, name): if self.parent: From afa at codespeak.net Mon Oct 11 08:30:54 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 08:30:54 +0200 (CEST) Subject: [pypy-svn] r77772 - pypy/branch/fast-forward/lib-python/modified-2.7.0 Message-ID: <20101011063054.B3FFF282BF3@codespeak.net> Author: afa Date: Mon Oct 11 08:30:53 2010 New Revision: 77772 Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py Log: Revert r77770. Bound methods cannot be compared by identity, "self.random" yields a different object every time. Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py ============================================================================== --- pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/random.py Mon Oct 11 08:30:53 2010 @@ -254,8 +254,8 @@ # Only call self.getrandbits if the original random() builtin method # has not been overridden or if a new getrandbits() was supplied. # This assures that the two methods correspond. - if (self.random is super(Random, self).random or - getrandbits is not super(Random, self).getrandbits): + if (self.random == super(Random, self).random or + getrandbits != super(Random, self).getrandbits): k = int(1.00001 + _log(n-1, 2.0)) # 2**k > n-1 > 2**(k-2) r = getrandbits(k) while r >= n: From antocuni at codespeak.net Mon Oct 11 10:58:29 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 11 Oct 2010 10:58:29 +0200 (CEST) Subject: [pypy-svn] r77773 - in pypy/branch/jitffi/pypy/module/_ffi: . test Message-ID: <20101011085829.15B9E282BF3@codespeak.net> Author: antocuni Date: Mon Oct 11 10:58:22 2010 New Revision: 77773 Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Log: raise TypeError if the number of actual arguments is not the expected one Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Mon Oct 11 10:58:22 2010 @@ -55,7 +55,14 @@ @jit.unroll_safe def build_argchain(self, space, argtypes, args_w): - assert len(args_w) == len(argtypes) # XXX: raise OperationError + if len(args_w) != len(argtypes): + arg = 'arguments' + if len(argtypes) == 1: + arg = 'argument' + raise operationerrfmt(space.w_TypeError, + '%s() takes exactly %d %s (%d given)', + self.func.name, len(argtypes), arg, len(args_w)) + # argchain = libffi.ArgChain() for i in range(len(args_w)): argtype = argtypes[i] Modified: pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Mon Oct 11 10:58:22 2010 @@ -84,3 +84,10 @@ assert get_dummy() == 0 assert set_dummy(42) is None assert get_dummy() == 42 + + def test_TypeError_numargs(self): + from _ffi import CDLL, types + libfoo = CDLL(self.libfoo_name) + sum_xy = libfoo.getfunc('sum_xy', [types.sint, types.sint], types.sint) + raises(TypeError, "sum_xy(1, 2, 3)") + raises(TypeError, "sum_xy(1)") From antocuni at codespeak.net Mon Oct 11 11:26:55 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 11 Oct 2010 11:26:55 +0200 (CEST) Subject: [pypy-svn] r77774 - in pypy/branch/jitffi/pypy/module/_ffi: . test Message-ID: <20101011092655.2C794282BF3@codespeak.net> Author: antocuni Date: Mon Oct 11 11:26:53 2010 New Revision: 77774 Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Log: raise TypeError if we try to use void as an argument type Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Mon Oct 11 11:26:53 2010 @@ -73,7 +73,7 @@ elif kind == 'f': argchain.arg(space.float_w(w_arg)) else: - assert False # XXX + assert False, "Argument kind '%s' not supported" % kind return argchain @unwrap_spec('self', ObjSpace, 'args_w') @@ -111,13 +111,19 @@ self.name = name self.space = space - def ffitype(self, w_argtype): - return self.space.interp_w(W_FFIType, w_argtype).ffitype + def ffitype(self, w_argtype, allow_void=False): + res = self.space.interp_w(W_FFIType, w_argtype).ffitype + if res is libffi.types.void and not allow_void: + space = self.space + msg = 'void is not a valid argument type' + raise OperationError(space.w_TypeError, space.wrap(msg)) + return res @unwrap_spec('self', ObjSpace, str, W_Root, W_Root) def getfunc(self, space, name, w_argtypes, w_restype): - argtypes = [self.ffitype(w_argtype) for w_argtype in space.listview(w_argtypes)] - restype = self.ffitype(w_restype) + argtypes = [self.ffitype(w_argtype) for w_argtype in + space.listview(w_argtypes)] + restype = self.ffitype(w_restype, allow_void=True) func = self.cdll.getpointer(name, argtypes, restype) return W_FuncPtr(func) Modified: pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Mon Oct 11 11:26:53 2010 @@ -91,3 +91,9 @@ sum_xy = libfoo.getfunc('sum_xy', [types.sint, types.sint], types.sint) raises(TypeError, "sum_xy(1, 2, 3)") raises(TypeError, "sum_xy(1)") + + def test_TypeError_voidarg(self): + from _ffi import CDLL, types + libfoo = CDLL(self.libfoo_name) + raises(TypeError, "libfoo.getfunc('sum_xy', [types.void], types.sint)") + From antocuni at codespeak.net Mon Oct 11 11:30:21 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 11 Oct 2010 11:30:21 +0200 (CEST) Subject: [pypy-svn] r77775 - pypy/branch/jitffi/pypy/module/_ffi Message-ID: <20101011093021.DDBA7282BF3@codespeak.net> Author: antocuni Date: Mon Oct 11 11:30:20 2010 New Revision: 77775 Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Log: reformat to stay in 80 columns Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py Mon Oct 11 11:30:20 2010 @@ -1,6 +1,8 @@ import sys -from pypy.interpreter.baseobjspace import W_Root, ObjSpace, Wrappable, Arguments -from pypy.interpreter.error import OperationError, wrap_oserror, operationerrfmt +from pypy.interpreter.baseobjspace import W_Root, ObjSpace, Wrappable, \ + Arguments +from pypy.interpreter.error import OperationError, wrap_oserror, \ + operationerrfmt from pypy.interpreter.gateway import interp2app, NoneNotWrapped, unwrap_spec from pypy.interpreter.typedef import TypeDef, GetSetProperty # @@ -55,16 +57,18 @@ @jit.unroll_safe def build_argchain(self, space, argtypes, args_w): - if len(args_w) != len(argtypes): + expected = len(argtypes) + given = len(args_w) + if given != expected: arg = 'arguments' if len(argtypes) == 1: arg = 'argument' raise operationerrfmt(space.w_TypeError, '%s() takes exactly %d %s (%d given)', - self.func.name, len(argtypes), arg, len(args_w)) + self.func.name, expected, arg, given) # argchain = libffi.ArgChain() - for i in range(len(args_w)): + for i in range(expected): argtype = argtypes[i] w_arg = args_w[i] kind = libffi.types.getkind(argtype) From antocuni at codespeak.net Mon Oct 11 11:38:19 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 11 Oct 2010 11:38:19 +0200 (CEST) Subject: [pypy-svn] r77776 - in pypy/branch/jitffi/pypy: jit/backend/llsupport rlib Message-ID: <20101011093819.DB42F282BF3@codespeak.net> Author: antocuni Date: Mon Oct 11 11:38:17 2010 New Revision: 77776 Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: move the logic from the jit to libffi, which is a more proper place Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py Mon Oct 11 11:38:17 2010 @@ -21,43 +21,13 @@ return VoidCallDescr(arg_classes, extrainfo) assert False - -# XXX: maybe we can turn this into a dictionary, but we need to do it at -# runtime as libffi.types.* pointers def get_ffi_type_kind(ffi_type): from pypy.rlib.libffi import types - if ffi_type is types.void: - return history.VOID - elif ffi_type is types.pointer: + kind = types.getkind(ffi_type) + if kind == 'i': return history.INT - elif ffi_type is types.double: + elif kind == 'f': return history.FLOAT - elif ffi_type is types.uchar: - return history.INT - elif ffi_type is types.uint8: - return history.INT - elif ffi_type is types.schar: - return history.INT - elif ffi_type is types.sint8: - return history.INT - elif ffi_type is types.uint16: - return history.INT - elif ffi_type is types.ushort: - return history.INT - elif ffi_type is types.sint16: - return history.INT - elif ffi_type is types.sshort: - return history.INT - elif ffi_type is types.uint: - return history.INT - elif ffi_type is types.uint32: - return history.INT - elif ffi_type is types.sint: - return history.INT - elif ffi_type is types.sint32: - return history.INT - ## elif ffi_type is types.uint64: - ## return history.INT - ## elif ffi_type is types.sint64: - ## return history.INT - raise KeyError + elif kind == 'v': + return history.VOID + assert False, "Unsuported kind '%s'" % kind Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Mon Oct 11 11:38:17 2010 @@ -34,10 +34,42 @@ @staticmethod @jit.purefunction - def getkind(ffitype): - # XXX: move this function outside the jit - from pypy.jit.backend.llsupport.ffisupport import get_ffi_type_kind - return get_ffi_type_kind(ffitype) + def getkind(ffi_type): + if ffi_type is types.void: + return 'v' + elif ffi_type is types.double: + return 'f' + elif ffi_type is types.pointer: + return 'i' + elif ffi_type is types.uchar: + return 'i' + elif ffi_type is types.uint8: + return 'i' + elif ffi_type is types.schar: + return 'i' + elif ffi_type is types.sint8: + return 'i' + elif ffi_type is types.uint16: + return 'i' + elif ffi_type is types.ushort: + return 'i' + elif ffi_type is types.sint16: + return 'i' + elif ffi_type is types.sshort: + return 'i' + elif ffi_type is types.uint: + return 'i' + elif ffi_type is types.uint32: + return 'i' + elif ffi_type is types.sint: + return 'i' + elif ffi_type is types.sint32: + return 'i' + ## elif ffi_type is types.uint64: + ## return 'i' + ## elif ffi_type is types.sint64: + ## return 'i' + raise KeyError types._import() From antocuni at codespeak.net Mon Oct 11 11:44:29 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Mon, 11 Oct 2010 11:44:29 +0200 (CEST) Subject: [pypy-svn] r77777 - pypy/branch/jitffi/pypy/module/_ffi/test Message-ID: <20101011094429.36C39282BF3@codespeak.net> Author: antocuni Date: Mon Oct 11 11:44:27 2010 New Revision: 77777 Modified: pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Log: write a passing test Modified: pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py ============================================================================== --- pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py (original) +++ pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py Mon Oct 11 11:44:27 2010 @@ -97,3 +97,6 @@ libfoo = CDLL(self.libfoo_name) raises(TypeError, "libfoo.getfunc('sum_xy', [types.void], types.sint)") + def test_OSError_loading(self): + from _ffi import CDLL, types + raises(OSError, "CDLL('I do not exist')") From fijal at codespeak.net Mon Oct 11 12:18:05 2010 From: fijal at codespeak.net (fijal at codespeak.net) Date: Mon, 11 Oct 2010 12:18:05 +0200 (CEST) Subject: [pypy-svn] r77778 - in pypy/extradoc/talk/pyconpl-2010: . demos Message-ID: <20101011101805.DB301282BF4@codespeak.net> Author: fijal Date: Mon Oct 11 12:18:03 2010 New Revision: 77778 Added: pypy/extradoc/talk/pyconpl-2010/ pypy/extradoc/talk/pyconpl-2010/Makefile pypy/extradoc/talk/pyconpl-2010/author.latex pypy/extradoc/talk/pyconpl-2010/beamerouterthememy.sty pypy/extradoc/talk/pyconpl-2010/beamerthemeWarsaw.sty pypy/extradoc/talk/pyconpl-2010/demos/ pypy/extradoc/talk/pyconpl-2010/demos/attrsfast.py pypy/extradoc/talk/pyconpl-2010/demos/attrsslow.py pypy/extradoc/talk/pyconpl-2010/demos/bytecode.py pypy/extradoc/talk/pyconpl-2010/demos/complex.py pypy/extradoc/talk/pyconpl-2010/demos/jit.py pypy/extradoc/talk/pyconpl-2010/demos/jit2.py pypy/extradoc/talk/pyconpl-2010/talk.pdf (contents, props changed) pypy/extradoc/talk/pyconpl-2010/telescope.jpg (contents, props changed) pypy/extradoc/talk/pyconpl-2010/title.latex Log: presentation as it went Added: pypy/extradoc/talk/pyconpl-2010/Makefile ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/Makefile Mon Oct 11 12:18:03 2010 @@ -0,0 +1,18 @@ +# you can find rst2beamer.py here: +# http://codespeak.net/svn/user/antocuni/bin/rst2beamer.py + +# WARNING: to work, it needs this patch for docutils +# https://sourceforge.net/tracker/?func=detail&atid=422032&aid=1459707&group_id=38414 + +talk.pdf: talk.rst author.latex title.latex stylesheet.latex + ~/src/user/antocuni/bin/rst2beamer.py --stylesheet=stylesheet.latex --documentoptions=14pt talk.rst talk.latex --output-encoding=utf8|| exit + sed 's/\\date{}/\\input{author.latex}/' -i talk.latex || exit + sed 's/\\maketitle/\\input{title.latex}/' -i talk.latex || exit + pdflatex talk.latex || exit + pdflatex talk.latex || exit + +view: talk.pdf + evince talk.pdf & + +xpdf: talk.pdf + xpdf talk.pdf & Added: pypy/extradoc/talk/pyconpl-2010/author.latex ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/author.latex Mon Oct 11 12:18:03 2010 @@ -0,0 +1,5 @@ +\definecolor{rrblitbackground}{rgb}{0.0, 0.0, 0.0} +\title{PyPy: Jak uczyni? pythona szybszym} +\author[fijal]{Maciej Fija?kowski} +\institute{SKA South Africa} +\date{PyCon PL, 8.10.2010} Added: pypy/extradoc/talk/pyconpl-2010/beamerouterthememy.sty ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/beamerouterthememy.sty Mon Oct 11 12:18:03 2010 @@ -0,0 +1,39 @@ +\ProvidesPackageRCS $Header: /cvsroot/latex-beamer/latex-beamer/themes/outer/beamerouterthemesplit.sty,v 1.4 2004/10/07 22:21:16 tantau Exp $ + +% Copyright 2003 by Till Tantau +% +% This program can be redistributed and/or modified under the terms +% of the GNU Public License, version 2. + +\mode + +\setbeamercolor{section in head/foot}{parent=palette quaternary} +\setbeamercolor{subsection in head/foot}{parent=palette primary} + +\setbeamercolor{author in head/foot}{parent=section in head/foot} +\setbeamercolor{title in head/foot}{parent=subsection in head/foot} + + + +\usesectionheadtemplate + {\hfill\insertsectionhead} + {\hfill\color{fg!50!bg}\insertsectionhead} + + + + +\defbeamertemplate*{footline}{split theme} +{% + \leavevmode% + \hbox{\begin{beamercolorbox}[wd=.6\paperwidth,ht=2.5ex,dp=1.125ex,leftskip=.3cm plus1fill,rightskip=.3cm]{author in head/foot}% + \usebeamerfont{author in head/foot}\insertshortauthor + \end{beamercolorbox}% + \begin{beamercolorbox}[wd=.4\paperwidth,ht=2.5ex,dp=1.125ex,leftskip=.3cm,rightskip=.3cm plus1fil]{title in head/foot}% + \usebeamerfont{title in head/foot}\insertshorttitle + \end{beamercolorbox}}% + \vskip0pt% +} + + +\mode + Added: pypy/extradoc/talk/pyconpl-2010/beamerthemeWarsaw.sty ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/beamerthemeWarsaw.sty Mon Oct 11 12:18:03 2010 @@ -0,0 +1,18 @@ +\ProvidesPackageRCS $Header: /cvsroot/latex-beamer/latex-beamer/themes/theme/beamerthemeWarsaw.sty,v 1.8 2004/10/07 20:53:10 tantau Exp $ + +% Copyright 2003 by Till Tantau +% +% This program can be redistributed and/or modified under the terms +% of the GNU Public License, version 2. + +\mode + +\useinnertheme[shadow=true]{rounded} +\useoutertheme{my} +\usecolortheme{orchid} +\usecolortheme{whale} + +\setbeamerfont{block title}{size={}} + +\mode + Added: pypy/extradoc/talk/pyconpl-2010/demos/attrsfast.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/demos/attrsfast.py Mon Oct 11 12:18:03 2010 @@ -0,0 +1,15 @@ + +class A(object): + def __init__(self): + self.a = 3 + self.b = 4 + +def f(): + a = A() + i = 0 + while i < 1000000: + a.a + i += 1 + +f() + Added: pypy/extradoc/talk/pyconpl-2010/demos/attrsslow.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/demos/attrsslow.py Mon Oct 11 12:18:03 2010 @@ -0,0 +1,16 @@ + +class A(object): + def __init__(self): + self.d = {'a': 3, 'b': 4} + + def __getattr__(self, attr): + return self.d[attr] + +def f(): + a = A() + i = 0 + while i < 1000000: + a.a + i += 1 + +f() Added: pypy/extradoc/talk/pyconpl-2010/demos/bytecode.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/demos/bytecode.py Mon Oct 11 12:18:03 2010 @@ -0,0 +1,8 @@ + +def f(a, b): + while a < b: + a += 1 + return b + +import dis +dis.dis(f) Added: pypy/extradoc/talk/pyconpl-2010/demos/complex.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/demos/complex.py Mon Oct 11 12:18:03 2010 @@ -0,0 +1,16 @@ + +class Number(object): + def __init__(self, no): + self.no = no + + def __add__(self, other): + return Number(self.no + other.no) + +def f(): + i = 0 + sum = Number(0) + while i < 10000: + sum = Number(1) + sum + i += 1 + +f() Added: pypy/extradoc/talk/pyconpl-2010/demos/jit.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/demos/jit.py Mon Oct 11 12:18:03 2010 @@ -0,0 +1,7 @@ + +def f(): + i = 0 + while i < 20000000: + i += 1 + +f() Added: pypy/extradoc/talk/pyconpl-2010/demos/jit2.py ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/demos/jit2.py Mon Oct 11 12:18:03 2010 @@ -0,0 +1,6 @@ +def f(): + i = 0 + while i < 1001: + i += 1 + +f() Added: pypy/extradoc/talk/pyconpl-2010/talk.pdf ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pyconpl-2010/telescope.jpg ============================================================================== Binary file. No diff available. Added: pypy/extradoc/talk/pyconpl-2010/title.latex ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pyconpl-2010/title.latex Mon Oct 11 12:18:03 2010 @@ -0,0 +1,5 @@ +\begin{titlepage} +\begin{figure}[h] +\includegraphics[width=80px]{../img/py-web.png} +\end{figure} +\end{titlepage} \ No newline at end of file From arigo at codespeak.net Mon Oct 11 14:26:23 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 14:26:23 +0200 (CEST) Subject: [pypy-svn] r77780 - in pypy/branch/jit-str-unicode/pypy/jit: backend/test backend/x86 backend/x86/test codewriter codewriter/test metainterp metainterp/optimizeopt metainterp/test Message-ID: <20101011122623.54BAD282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 14:26:19 2010 New Revision: 77780 Modified: pypy/branch/jit-str-unicode/pypy/jit/backend/test/runner_test.py pypy/branch/jit-str-unicode/pypy/jit/backend/test/test_ll_random.py pypy/branch/jit-str-unicode/pypy/jit/backend/x86/regalloc.py pypy/branch/jit-str-unicode/pypy/jit/backend/x86/test/test_string.py pypy/branch/jit-str-unicode/pypy/jit/codewriter/effectinfo.py pypy/branch/jit-str-unicode/pypy/jit/codewriter/jtransform.py pypy/branch/jit-str-unicode/pypy/jit/codewriter/test/test_jtransform.py pypy/branch/jit-str-unicode/pypy/jit/metainterp/history.py pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/intbounds.py pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/rewrite.py pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/string.py pypy/branch/jit-str-unicode/pypy/jit/metainterp/resume.py pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/oparser.py pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_optimizefindnode.py pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_optimizeopt.py pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_resume.py pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_string.py Log: Check-in. Should be done. Modified: pypy/branch/jit-str-unicode/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/backend/test/runner_test.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/backend/test/runner_test.py Mon Oct 11 14:26:19 2010 @@ -833,6 +833,23 @@ length_box], 'void') assert self.look_string(r_box) == "!??cdef?!" + def test_copyunicodecontent(self): + s_box = self.alloc_unicode(u"abcdef") + for s_box in [s_box, s_box.constbox()]: + for srcstart_box in [BoxInt(2), ConstInt(2)]: + for dststart_box in [BoxInt(3), ConstInt(3)]: + for length_box in [BoxInt(4), ConstInt(4)]: + for r_box_is_const in [False, True]: + r_box = self.alloc_unicode(u"!???????!") + if r_box_is_const: + r_box = r_box.constbox() + self.execute_operation(rop.COPYUNICODECONTENT, + [s_box, r_box, + srcstart_box, + dststart_box, + length_box], 'void') + assert self.look_unicode(r_box) == u"!??cdef?!" + def test_do_unicode_basic(self): u = self.cpu.bh_newunicode(5) self.cpu.bh_unicodesetitem(u, 4, 123) @@ -1227,6 +1244,10 @@ u_box = BoxPtr(lltype.cast_opaque_ptr(llmemory.GCREF, u)) return u_box + def look_unicode(self, unicode_box): + u = unicode_box.getref(lltype.Ptr(rstr.UNICODE)) + return u''.join(u.chars) + def test_casts(self): py.test.skip("xxx fix or kill") Modified: pypy/branch/jit-str-unicode/pypy/jit/backend/test/test_ll_random.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/backend/test/test_ll_random.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/backend/test/test_ll_random.py Mon Oct 11 14:26:19 2010 @@ -599,7 +599,7 @@ OPERATIONS.append(StrLenOperation(rop.STRLEN)) OPERATIONS.append(UnicodeLenOperation(rop.UNICODELEN)) OPERATIONS.append(CopyStrContentOperation(rop.COPYSTRCONTENT)) - #OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT)) + OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT)) for i in range(2): OPERATIONS.append(GuardClassOperation(rop.GUARD_CLASS)) Modified: pypy/branch/jit-str-unicode/pypy/jit/backend/x86/regalloc.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/backend/x86/regalloc.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/backend/x86/regalloc.py Mon Oct 11 14:26:19 2010 @@ -778,15 +778,11 @@ loc = self.loc(op.getarg(0)) return self._call(op, [loc]) # boehm GC (XXX kill the following code at some point) - ofs_items, itemsize, ofs = symbolic.get_array_token(rstr.UNICODE, self.translate_support_code) - if itemsize == 4: - return self._malloc_varsize(ofs_items, ofs, 2, op.getarg(0), - op.result) - elif itemsize == 2: - return self._malloc_varsize(ofs_items, ofs, 1, op.getarg(0), - op.result) - else: - assert False, itemsize + ofs_items, _, ofs = symbolic.get_array_token(rstr.UNICODE, + self.translate_support_code) + scale = self._get_unicode_item_scale() + return self._malloc_varsize(ofs_items, ofs, scale, op.getarg(0), + op.result) def _malloc_varsize(self, ofs_items, ofs_length, scale, v, res_v): # XXX kill this function at some point @@ -959,6 +955,12 @@ consider_unicodegetitem = consider_strgetitem def consider_copystrcontent(self, op): + self._consider_copystrcontent(op, is_unicode=False) + + def consider_copyunicodecontent(self, op): + self._consider_copystrcontent(op, is_unicode=True) + + def _consider_copystrcontent(self, op, is_unicode): # compute the source address args = op.getarglist() base_loc = self.rm.make_sure_var_in_reg(args[0], args) @@ -970,7 +972,8 @@ srcaddr_box = TempBox() forbidden_vars = [args[1], args[3], args[4], srcaddr_box] srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box, forbidden_vars) - self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc) + self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc, + is_unicode=is_unicode) # compute the destination address base_loc = self.rm.make_sure_var_in_reg(args[1], forbidden_vars) ofs_loc = self.rm.make_sure_var_in_reg(args[3], forbidden_vars) @@ -980,25 +983,57 @@ forbidden_vars = [args[4], srcaddr_box] dstaddr_box = TempBox() dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box, forbidden_vars) - self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc) + self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc, + is_unicode=is_unicode) + # compute the length in bytes + length_box = args[4] + length_loc = self.loc(length_box) + if is_unicode: + self.rm.possibly_free_var(length_box) + forbidden_vars = [srcaddr_box, dstaddr_box] + bytes_box = TempBox() + bytes_loc = self.rm.force_allocate_reg(bytes_box, forbidden_vars) + scale = self._get_unicode_item_scale() + if not (isinstance(length_loc, ImmedLoc) or + isinstance(length_loc, RegLoc)): + self.assembler.mov(length_loc, bytes_loc) + length_loc = bytes_loc + self.assembler.load_effective_addr(length_loc, 0, scale, bytes_loc) + length_box = bytes_box + length_loc = bytes_loc # call memcpy() - length_loc = self.loc(args[4]) self.rm.before_call() self.xrm.before_call() self.assembler._emit_call(imm(self.assembler.memcpy_addr), [dstaddr_loc, srcaddr_loc, length_loc]) - self.rm.possibly_free_var(args[4]) + self.rm.possibly_free_var(length_box) self.rm.possibly_free_var(dstaddr_box) self.rm.possibly_free_var(srcaddr_box) - def _gen_address_inside_string(self, baseloc, ofsloc, resloc): + def _gen_address_inside_string(self, baseloc, ofsloc, resloc, is_unicode): cpu = self.assembler.cpu - ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR, + if is_unicode: + ofs_items, _, _ = symbolic.get_array_token(rstr.UNICODE, + self.translate_support_code) + scale = self._get_unicode_item_scale() + else: + ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR, self.translate_support_code) - assert itemsize == 1 - self.assembler.load_effective_addr(ofsloc, ofs_items, 0, + assert itemsize == 1 + scale = 0 + self.assembler.load_effective_addr(ofsloc, ofs_items, scale, resloc, baseloc) + def _get_unicode_item_scale(self): + _, itemsize, _ = symbolic.get_array_token(rstr.UNICODE, + self.translate_support_code) + if itemsize == 4: + return 2 + elif itemsize == 2: + return 1 + else: + raise AssertionError("bad unicode item size") + def consider_jump(self, op): assembler = self.assembler assert self.jump_target_descr is None Modified: pypy/branch/jit-str-unicode/pypy/jit/backend/x86/test/test_string.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/backend/x86/test/test_string.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/backend/x86/test/test_string.py Mon Oct 11 14:26:19 2010 @@ -2,8 +2,12 @@ from pypy.jit.metainterp.test import test_string from pypy.jit.backend.x86.test.test_basic import Jit386Mixin -class TestString(Jit386Mixin, test_string.StringTests): +class TestString(Jit386Mixin, test_string.TestLLtype): # for the individual tests see # ====> ../../../metainterp/test/test_string.py - CALL = 'call' - CALL_PURE = 'call_pure' + pass + +class TestUnicode(Jit386Mixin, test_string.TestLLtypeUnicode): + # for the individual tests see + # ====> ../../../metainterp/test/test_string.py + pass Modified: pypy/branch/jit-str-unicode/pypy/jit/codewriter/effectinfo.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/codewriter/effectinfo.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/codewriter/effectinfo.py Mon Oct 11 14:26:19 2010 @@ -18,19 +18,30 @@ # the 'oopspecindex' field is one of the following values: OS_NONE = 0 # normal case, no oopspec OS_ARRAYCOPY = 1 # "list.ll_arraycopy" - OS_STR_CONCAT = 2 # "stroruni.concat" - OS_UNI_CONCAT = 3 # "stroruni.concat" - OS_STR_SLICE = 4 # "stroruni.slice" - OS_UNI_SLICE = 5 # "stroruni.slice" - OS_STR_EQUAL = 6 # "stroruni.equal" - OS_UNI_EQUAL = 7 # "stroruni.equal" - OS_STREQ_SLICE_CHECKNULL = 8 # s2!=NULL and s1[x:x+length]==s2 - OS_STREQ_SLICE_NONNULL = 9 # s1[x:x+length]==s2 (assert s2!=NULL) - OS_STREQ_SLICE_CHAR = 10 # s1[x:x+length]==char - OS_STREQ_NONNULL = 11 # s1 == s2 (assert s1!=NULL,s2!=NULL) - OS_STREQ_NONNULL_CHAR = 12 # s1 == char (assert s1!=NULL) - OS_STREQ_CHECKNULL_CHAR = 13 # s1!=NULL and s1==char - OS_STREQ_LENGTHOK = 14 # s1 == s2 (assert len(s1)==len(s2)) + OS_STR2UNICODE = 2 # "str.str2unicode" + + OS_STR_CONCAT = 22 # "stroruni.concat" + OS_STR_SLICE = 23 # "stroruni.slice" + OS_STR_EQUAL = 24 # "stroruni.equal" + OS_STREQ_SLICE_CHECKNULL = 25 # s2!=NULL and s1[x:x+length]==s2 + OS_STREQ_SLICE_NONNULL = 26 # s1[x:x+length]==s2 (assert s2!=NULL) + OS_STREQ_SLICE_CHAR = 27 # s1[x:x+length]==char + OS_STREQ_NONNULL = 28 # s1 == s2 (assert s1!=NULL,s2!=NULL) + OS_STREQ_NONNULL_CHAR = 29 # s1 == char (assert s1!=NULL) + OS_STREQ_CHECKNULL_CHAR = 30 # s1!=NULL and s1==char + OS_STREQ_LENGTHOK = 31 # s1 == s2 (assert len(s1)==len(s2)) + + OS_UNI_CONCAT = 42 # + OS_UNI_SLICE = 43 # + OS_UNI_EQUAL = 44 # + OS_UNIEQ_SLICE_CHECKNULL = 45 # + OS_UNIEQ_SLICE_NONNULL = 46 # + OS_UNIEQ_SLICE_CHAR = 47 # + OS_UNIEQ_NONNULL = 48 # the same for unicode + OS_UNIEQ_NONNULL_CHAR = 49 # (must be the same amount as for + OS_UNIEQ_CHECKNULL_CHAR = 50 # STR, in the same order) + OS_UNIEQ_LENGTHOK = 51 # + _OS_offset_uni = OS_UNI_CONCAT - OS_STR_CONCAT def __new__(cls, readonly_descrs_fields, write_descrs_fields, write_descrs_arrays, Modified: pypy/branch/jit-str-unicode/pypy/jit/codewriter/jtransform.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/codewriter/jtransform.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/codewriter/jtransform.py Mon Oct 11 14:26:19 2010 @@ -316,6 +316,8 @@ prepare = self._handle_list_call elif oopspec_name.startswith('stroruni.'): prepare = self._handle_stroruni_call + elif oopspec_name == 'str.str2unicode': + prepare = self._handle_str2unicode_call elif oopspec_name.startswith('virtual_ref'): prepare = self._handle_virtual_ref_call else: @@ -1106,11 +1108,16 @@ [SoU, SoU], lltype.Signed), ]: + if args[0].concretetype.TO == rstr.UNICODE: + otherindex += EffectInfo._OS_offset_uni self._register_extra_helper(otherindex, othername, argtypes, resulttype) # return self._handle_oopspec_call(op, args, dict[oopspec_name]) + def _handle_str2unicode_call(self, op, oopspec_name, args): + return self._handle_oopspec_call(op, args, EffectInfo.OS_STR2UNICODE) + # ---------- # VirtualRefs. Modified: pypy/branch/jit-str-unicode/pypy/jit/codewriter/test/test_jtransform.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/codewriter/test/test_jtransform.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/codewriter/test/test_jtransform.py Mon Oct 11 14:26:19 2010 @@ -77,7 +77,8 @@ class FakeBuiltinCallControl: def guess_call_kind(self, op): return 'builtin' - def getcalldescr(self, op, oopspecindex): + def getcalldescr(self, op, oopspecindex=None): + assert oopspecindex is not None # in this test return 'calldescr-%d' % oopspecindex def calldescr_canraise(self, calldescr): return False @@ -766,6 +767,24 @@ assert op1.args[3] == ListOfKind('ref', [v1]) assert op1.result == v4 +def test_str2unicode(): + # test that the oopspec is present and correctly transformed + PSTR = lltype.Ptr(rstr.STR) + PUNICODE = lltype.Ptr(rstr.UNICODE) + FUNC = lltype.FuncType([PSTR], PUNICODE) + func = lltype.functionptr(FUNC, 'll_str2unicode', + _callable=rstr.LLHelpers.ll_str2unicode) + v1 = varoftype(PSTR) + v2 = varoftype(PUNICODE) + op = SpaceOperation('direct_call', [const(func), v1], v2) + tr = Transformer(FakeCPU(), FakeBuiltinCallControl()) + op1 = tr.rewrite_operation(op) + assert op1.opname == 'residual_call_r_r' + assert op1.args[0].value == func + assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_STR2UNICODE + assert op1.args[2] == ListOfKind('ref', [v1]) + assert op1.result == v2 + def test_list_ll_arraycopy(): from pypy.rlib.rgc import ll_arraycopy LIST = lltype.GcArray(lltype.Signed) Modified: pypy/branch/jit-str-unicode/pypy/jit/metainterp/history.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/metainterp/history.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/metainterp/history.py Mon Oct 11 14:26:19 2010 @@ -698,6 +698,21 @@ return result _const_ptr_for_string = {} +def get_const_ptr_for_unicode(s): + from pypy.rpython.annlowlevel import llunicode + if not we_are_translated(): + try: + return _const_ptr_for_unicode[s] + except KeyError: + pass + if isinstance(s, str): + s = unicode(s) + result = ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, llunicode(s))) + if not we_are_translated(): + _const_ptr_for_unicode[s] = result + return result +_const_ptr_for_unicode = {} + # ____________________________________________________________ # The TreeLoop class contains a loop or a generalized loop, i.e. a tree Modified: pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/intbounds.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/intbounds.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/intbounds.py Mon Oct 11 14:26:19 2010 @@ -191,6 +191,7 @@ v1.intbound.make_ge(IntLowerBound(0)) optimize_STRLEN = optimize_ARRAYLEN_GC + optimize_UNICODELEN = optimize_ARRAYLEN_GC def make_int_lt(self, box1, box2): v1 = self.getvalue(box1) Modified: pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/rewrite.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/rewrite.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/rewrite.py Mon Oct 11 14:26:19 2010 @@ -3,6 +3,7 @@ from pypy.jit.metainterp.history import ConstInt from pypy.jit.metainterp.optimizeutil import _findall from pypy.jit.metainterp.resoperation import rop, ResOperation +from pypy.jit.codewriter.effectinfo import EffectInfo class OptRewrite(Optimization): """Rewrite operations into equivalent, cheaper operations. @@ -326,8 +327,37 @@ ## return ## self.emit_operation(op) -optimize_ops = _findall(OptRewrite, 'optimize_') - + def optimize_CALL(self, op): + # dispatch based on 'oopspecindex' to a method that handles + # specifically the given oopspec call. For non-oopspec calls, + # oopspecindex is just zero. + effectinfo = op.getdescr().get_extra_info() + if effectinfo is not None: + oopspecindex = effectinfo.oopspecindex + if oopspecindex == EffectInfo.OS_ARRAYCOPY: + if self._optimize_CALL_ARRAYCOPY(op): + return + self.emit_operation(op) + def _optimize_CALL_ARRAYCOPY(self, op): + source_value = self.getvalue(op.getarg(1)) + dest_value = self.getvalue(op.getarg(2)) + source_start_box = self.get_constant_box(op.getarg(3)) + dest_start_box = self.get_constant_box(op.getarg(4)) + length = self.get_constant_box(op.getarg(5)) + if (source_value.is_virtual() and source_start_box and dest_start_box + and length and dest_value.is_virtual()): + # XXX optimize the case where dest value is not virtual, + # but we still can avoid a mess + source_start = source_start_box.getint() + dest_start = dest_start_box.getint() + for index in range(length.getint()): + val = source_value.getitem(index + source_start) + dest_value.setitem(index + dest_start, val) + return True + if length and length.getint() == 0: + return True # 0-length arraycopy + return False + +optimize_ops = _findall(OptRewrite, 'optimize_') - Modified: pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/string.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/string.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/metainterp/optimizeopt/string.py Mon Oct 11 14:26:19 2010 @@ -3,6 +3,7 @@ from pypy.jit.metainterp.history import Box, BoxInt, BoxPtr from pypy.jit.metainterp.history import Const, ConstInt, ConstPtr from pypy.jit.metainterp.history import get_const_ptr_for_string +from pypy.jit.metainterp.history import get_const_ptr_for_unicode from pypy.jit.metainterp.resoperation import rop, ResOperation from pypy.jit.metainterp.optimizeopt import optimizer, virtualize from pypy.jit.metainterp.optimizeopt.optimizer import CONST_0, CONST_1 @@ -11,56 +12,103 @@ from pypy.jit.codewriter.effectinfo import EffectInfo, callinfo_for_oopspec from pypy.jit.codewriter import heaptracker from pypy.rlib.unroll import unrolling_iterable +from pypy.rlib.objectmodel import specialize + + +class StrOrUnicode(object): + def __init__(self, LLTYPE, hlstr, emptystr, chr, + NEWSTR, STRLEN, STRGETITEM, STRSETITEM, COPYSTRCONTENT, + OS_offset): + self.LLTYPE = LLTYPE + self.hlstr = hlstr + self.emptystr = emptystr + self.chr = chr + self.NEWSTR = NEWSTR + self.STRLEN = STRLEN + self.STRGETITEM = STRGETITEM + self.STRSETITEM = STRSETITEM + self.COPYSTRCONTENT = COPYSTRCONTENT + self.OS_offset = OS_offset + + def _freeze_(self): + return True + +mode_string = StrOrUnicode(rstr.STR, annlowlevel.hlstr, '', chr, + rop.NEWSTR, rop.STRLEN, rop.STRGETITEM, + rop.STRSETITEM, rop.COPYSTRCONTENT, 0) +mode_unicode = StrOrUnicode(rstr.UNICODE, annlowlevel.hlunicode, u'', unichr, + rop.NEWUNICODE, rop.UNICODELEN, rop.UNICODEGETITEM, + rop.UNICODESETITEM, rop.COPYUNICODECONTENT, + EffectInfo._OS_offset_uni) + +# ____________________________________________________________ class __extend__(optimizer.OptValue): """New methods added to the base class OptValue for this file.""" - def getstrlen(self, newoperations): - s = self.get_constant_string() - if s is not None: - return ConstInt(len(s)) + def getstrlen(self, newoperations, mode): + if mode is mode_string: + s = self.get_constant_string_spec(mode_string) + if s is not None: + return ConstInt(len(s)) else: - if newoperations is None: - return None - self.ensure_nonnull() - box = self.force_box() - lengthbox = BoxInt() - newoperations.append(ResOperation(rop.STRLEN, [box], lengthbox)) - return lengthbox + s = self.get_constant_string_spec(mode_unicode) + if s is not None: + return ConstInt(len(s)) + if newoperations is None: + return None + self.ensure_nonnull() + box = self.force_box() + lengthbox = BoxInt() + newoperations.append(ResOperation(mode.STRLEN, [box], lengthbox)) + return lengthbox - def get_constant_string(self): + @specialize.arg(1) + def get_constant_string_spec(self, mode): if self.is_constant(): - s = self.box.getref(lltype.Ptr(rstr.STR)) - return annlowlevel.hlstr(s) + s = self.box.getref(lltype.Ptr(mode.LLTYPE)) + return mode.hlstr(s) else: return None - def string_copy_parts(self, newoperations, targetbox, offsetbox): + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): # Copies the pointer-to-string 'self' into the target string # given by 'targetbox', at the specified offset. Returns the offset # at the end of the copy. - lengthbox = self.getstrlen(newoperations) + lengthbox = self.getstrlen(newoperations, mode) srcbox = self.force_box() return copy_str_content(newoperations, srcbox, targetbox, - CONST_0, offsetbox, lengthbox) + CONST_0, offsetbox, lengthbox, mode) class VAbstractStringValue(virtualize.AbstractVirtualValue): - _attrs_ = () + _attrs_ = ('mode',) + + def __init__(self, optimizer, keybox, source_op, mode): + virtualize.AbstractVirtualValue.__init__(self, optimizer, keybox, + source_op) + self.mode = mode def _really_force(self): - s = self.get_constant_string() - if s is not None: - c_s = get_const_ptr_for_string(s) - self.make_constant(c_s) - return + if self.mode is mode_string: + s = self.get_constant_string_spec(mode_string) + if s is not None: + c_s = get_const_ptr_for_string(s) + self.make_constant(c_s) + return + else: + s = self.get_constant_string_spec(mode_unicode) + if s is not None: + c_s = get_const_ptr_for_unicode(s) + self.make_constant(c_s) + return assert self.source_op is not None self.box = box = self.source_op.result newoperations = self.optimizer.newoperations - lengthbox = self.getstrlen(newoperations) - newoperations.append(ResOperation(rop.NEWSTR, [lengthbox], box)) - self.string_copy_parts(newoperations, box, CONST_0) + lengthbox = self.getstrlen(newoperations, self.mode) + newoperations.append(ResOperation(self.mode.NEWSTR, [lengthbox], box)) + self.string_copy_parts(newoperations, box, CONST_0, self.mode) class VStringPlainValue(VAbstractStringValue): @@ -74,7 +122,7 @@ assert 0 <= start <= stop <= len(longerlist) self._chars = longerlist[start:stop] - def getstrlen(self, _): + def getstrlen(self, _, mode): if self._lengthbox is None: self._lengthbox = ConstInt(len(self._chars)) return self._lengthbox @@ -86,18 +134,21 @@ assert isinstance(charvalue, optimizer.OptValue) self._chars[index] = charvalue - def get_constant_string(self): + @specialize.arg(1) + def get_constant_string_spec(self, mode): for c in self._chars: if c is optimizer.CVAL_UNINITIALIZED_ZERO or not c.is_constant(): return None - return ''.join([chr(c.box.getint()) for c in self._chars]) + return mode.emptystr.join([mode.chr(c.box.getint()) + for c in self._chars]) - def string_copy_parts(self, newoperations, targetbox, offsetbox): + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): for i in range(len(self._chars)): charbox = self._chars[i].force_box() - newoperations.append(ResOperation(rop.STRSETITEM, [targetbox, - offsetbox, - charbox], None)) + newoperations.append(ResOperation(mode.STRSETITEM, [targetbox, + offsetbox, + charbox], + None)) offsetbox = _int_add(newoperations, offsetbox, CONST_1) return offsetbox @@ -109,7 +160,7 @@ value.get_args_for_fail(modifier) def _make_virtual(self, modifier): - return modifier.make_vstrplain() + return modifier.make_vstrplain(self.mode is mode_unicode) class VStringConcatValue(VAbstractStringValue): @@ -120,23 +171,24 @@ self.right = right self.lengthbox = lengthbox - def getstrlen(self, _): + def getstrlen(self, _, mode): return self.lengthbox - def get_constant_string(self): - s1 = self.left.get_constant_string() + @specialize.arg(1) + def get_constant_string_spec(self, mode): + s1 = self.left.get_constant_string_spec(mode) if s1 is None: return None - s2 = self.right.get_constant_string() + s2 = self.right.get_constant_string_spec(mode) if s2 is None: return None return s1 + s2 - def string_copy_parts(self, newoperations, targetbox, offsetbox): + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): offsetbox = self.left.string_copy_parts(newoperations, targetbox, - offsetbox) + offsetbox, mode) offsetbox = self.right.string_copy_parts(newoperations, targetbox, - offsetbox) + offsetbox, mode) return offsetbox def get_args_for_fail(self, modifier): @@ -150,7 +202,7 @@ self.right.get_args_for_fail(modifier) def _make_virtual(self, modifier): - return modifier.make_vstrconcat() + return modifier.make_vstrconcat(self.mode is mode_unicode) class VStringSliceValue(VAbstractStringValue): @@ -162,12 +214,13 @@ self.vstart = vstart self.vlength = vlength - def getstrlen(self, _): + def getstrlen(self, _, mode): return self.vlength.force_box() - def get_constant_string(self): + @specialize.arg(1) + def get_constant_string_spec(self, mode): if self.vstart.is_constant() and self.vlength.is_constant(): - s1 = self.vstr.get_constant_string() + s1 = self.vstr.get_constant_string_spec(mode) if s1 is None: return None start = self.vstart.box.getint() @@ -177,12 +230,12 @@ return s1[start : start + length] return None - def string_copy_parts(self, newoperations, targetbox, offsetbox): - lengthbox = self.getstrlen(newoperations) + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): + lengthbox = self.getstrlen(newoperations, mode) return copy_str_content(newoperations, self.vstr.force_box(), targetbox, self.vstart.force_box(), offsetbox, - lengthbox) + lengthbox, mode) def get_args_for_fail(self, modifier): if self.box is None and not modifier.already_seen_virtual(self.keybox): @@ -195,11 +248,11 @@ self.vlength.get_args_for_fail(modifier) def _make_virtual(self, modifier): - return modifier.make_vstrslice() + return modifier.make_vstrslice(self.mode is mode_unicode) def copy_str_content(newoperations, srcbox, targetbox, - srcoffsetbox, offsetbox, lengthbox): + srcoffsetbox, offsetbox, lengthbox, mode): if isinstance(srcbox, ConstPtr) and isinstance(srcoffsetbox, Const): M = 5 else: @@ -208,17 +261,18 @@ # up to M characters are done "inline", i.e. with STRGETITEM/STRSETITEM # instead of just a COPYSTRCONTENT. for i in range(lengthbox.value): - charbox = _strgetitem(newoperations, srcbox, srcoffsetbox) + charbox = _strgetitem(newoperations, srcbox, srcoffsetbox, mode) srcoffsetbox = _int_add(newoperations, srcoffsetbox, CONST_1) - newoperations.append(ResOperation(rop.STRSETITEM, [targetbox, - offsetbox, - charbox], None)) + newoperations.append(ResOperation(mode.STRSETITEM, [targetbox, + offsetbox, + charbox], + None)) offsetbox = _int_add(newoperations, offsetbox, CONST_1) else: nextoffsetbox = _int_add(newoperations, offsetbox, lengthbox) - op = ResOperation(rop.COPYSTRCONTENT, [srcbox, targetbox, - srcoffsetbox, offsetbox, - lengthbox], None) + op = ResOperation(mode.COPYSTRCONTENT, [srcbox, targetbox, + srcoffsetbox, offsetbox, + lengthbox], None) newoperations.append(op) offsetbox = nextoffsetbox return offsetbox @@ -245,12 +299,16 @@ newoperations.append(ResOperation(rop.INT_SUB, [box1, box2], resbox)) return resbox -def _strgetitem(newoperations, strbox, indexbox): +def _strgetitem(newoperations, strbox, indexbox, mode): if isinstance(strbox, ConstPtr) and isinstance(indexbox, ConstInt): - s = strbox.getref(lltype.Ptr(rstr.STR)) - return ConstInt(ord(s.chars[indexbox.getint()])) + if mode is mode_string: + s = strbox.getref(lltype.Ptr(rstr.STR)) + return ConstInt(ord(s.chars[indexbox.getint()])) + else: + s = strbox.getref(lltype.Ptr(rstr.UNICODE)) + return ConstInt(ord(s.chars[indexbox.getint()])) resbox = BoxInt() - newoperations.append(ResOperation(rop.STRGETITEM, [strbox, indexbox], + newoperations.append(ResOperation(mode.STRGETITEM, [strbox, indexbox], resbox)) return resbox @@ -258,62 +316,34 @@ class OptString(optimizer.Optimization): "Handling of strings and unicodes." - def make_vstring_plain(self, box, source_op=None): - vvalue = VStringPlainValue(self.optimizer, box, source_op) + def make_vstring_plain(self, box, source_op, mode): + vvalue = VStringPlainValue(self.optimizer, box, source_op, mode) self.make_equal_to(box, vvalue) return vvalue - def make_vstring_concat(self, box, source_op=None): - vvalue = VStringConcatValue(self.optimizer, box, source_op) + def make_vstring_concat(self, box, source_op, mode): + vvalue = VStringConcatValue(self.optimizer, box, source_op, mode) self.make_equal_to(box, vvalue) return vvalue - def make_vstring_slice(self, box, source_op=None): - vvalue = VStringSliceValue(self.optimizer, box, source_op) + def make_vstring_slice(self, box, source_op, mode): + vvalue = VStringSliceValue(self.optimizer, box, source_op, mode) self.make_equal_to(box, vvalue) return vvalue - def optimize_CALL(self, op): - # dispatch based on 'oopspecindex' to a method that handles - # specifically the given oopspec call. For non-oopspec calls, - # oopspecindex is just zero. - effectinfo = op.getdescr().get_extra_info() - if effectinfo is not None: - oopspecindex = effectinfo.oopspecindex - for value, meth in opt_call_oopspec_ops: - if oopspecindex == value: - if meth(self, op): - return - self.emit_operation(op) - - def opt_call_oopspec_ARRAYCOPY(self, op): - source_value = self.getvalue(op.getarg(1)) - dest_value = self.getvalue(op.getarg(2)) - source_start_box = self.get_constant_box(op.getarg(3)) - dest_start_box = self.get_constant_box(op.getarg(4)) - length = self.get_constant_box(op.getarg(5)) - if (source_value.is_virtual() and source_start_box and dest_start_box - and length and dest_value.is_virtual()): - # XXX optimize the case where dest value is not virtual, - # but we still can avoid a mess - source_start = source_start_box.getint() - dest_start = dest_start_box.getint() - for index in range(length.getint()): - val = source_value.getitem(index + source_start) - dest_value.setitem(index + dest_start, val) - return True - if length and length.getint() == 0: - return True # 0-length arraycopy - return False - def optimize_NEWSTR(self, op): + self._optimize_NEWSTR(op, mode_string) + def optimize_NEWUNICODE(self, op): + self._optimize_NEWSTR(op, mode_unicode) + + def _optimize_NEWSTR(self, op, mode): length_box = self.get_constant_box(op.getarg(0)) if length_box: # if the original 'op' did not have a ConstInt as argument, # build a new one with the ConstInt argument if not isinstance(op.getarg(0), ConstInt): - op = ResOperation(rop.NEWSTR, [length_box], op.result) - vvalue = self.make_vstring_plain(op.result, op) + op = ResOperation(mode.NEWSTR, [length_box], op.result) + vvalue = self.make_vstring_plain(op.result, op, mode) vvalue.setup(length_box.getint()) else: self.getvalue(op.result).ensure_nonnull() @@ -329,13 +359,20 @@ value.ensure_nonnull() self.emit_operation(op) + optimize_UNICODESETITEM = optimize_STRSETITEM + def optimize_STRGETITEM(self, op): + self._optimize_STRGETITEM(op, mode_string) + def optimize_UNICODEGETITEM(self, op): + self._optimize_STRGETITEM(op, mode_unicode) + + def _optimize_STRGETITEM(self, op, mode): value = self.getvalue(op.getarg(0)) vindex = self.getvalue(op.getarg(1)) - vresult = self.strgetitem(value, vindex) + vresult = self.strgetitem(value, vindex, mode) self.make_equal_to(op.result, vresult) - def strgetitem(self, value, vindex): + def strgetitem(self, value, vindex, mode): value.ensure_nonnull() # if value.is_virtual() and isinstance(value, VStringSliceValue): @@ -350,28 +387,71 @@ return value.getitem(vindex.box.getint()) # resbox = _strgetitem(self.optimizer.newoperations, - value.force_box(),vindex.force_box()) + value.force_box(),vindex.force_box(), mode) return self.getvalue(resbox) def optimize_STRLEN(self, op): + self._optimize_STRLEN(op, mode_string) + def optimize_UNICODELEN(self, op): + self._optimize_STRLEN(op, mode_unicode) + + def _optimize_STRLEN(self, op, mode): value = self.getvalue(op.getarg(0)) - lengthbox = value.getstrlen(self.optimizer.newoperations) + lengthbox = value.getstrlen(self.optimizer.newoperations, mode) self.make_equal_to(op.result, self.getvalue(lengthbox)) - def opt_call_oopspec_STR_CONCAT(self, op): + def optimize_CALL(self, op): + # dispatch based on 'oopspecindex' to a method that handles + # specifically the given oopspec call. For non-oopspec calls, + # oopspecindex is just zero. + effectinfo = op.getdescr().get_extra_info() + if effectinfo is not None: + oopspecindex = effectinfo.oopspecindex + for value, meth in opt_call_oopspec_ops: + if oopspecindex == value: # a match with the OS_STR_xxx + if meth(self, op, mode_string): + return + break + if oopspecindex == value + EffectInfo._OS_offset_uni: + # a match with the OS_UNI_xxx + if meth(self, op, mode_unicode): + return + break + if oopspecindex == EffectInfo.OS_STR2UNICODE: + if self.opt_call_str_STR2UNICODE(op): + return + self.emit_operation(op) + + def opt_call_str_STR2UNICODE(self, op): + # Constant-fold unicode("constant string"). + # More generally, supporting non-constant but virtual cases is + # not obvious, because of the exception UnicodeDecodeError that + # can be raised by ll_str2unicode() + varg = self.getvalue(op.getarg(1)) + s = varg.get_constant_string_spec(mode_string) + if s is None: + return False + try: + u = unicode(s) + except UnicodeDecodeError: + return False + self.make_constant(op.result, get_const_ptr_for_unicode(u)) + return True + + def opt_call_stroruni_STR_CONCAT(self, op, mode): vleft = self.getvalue(op.getarg(1)) vright = self.getvalue(op.getarg(2)) vleft.ensure_nonnull() vright.ensure_nonnull() newoperations = self.optimizer.newoperations - len1box = vleft.getstrlen(newoperations) - len2box = vright.getstrlen(newoperations) + len1box = vleft.getstrlen(newoperations, mode) + len2box = vright.getstrlen(newoperations, mode) lengthbox = _int_add(newoperations, len1box, len2box) - value = self.make_vstring_concat(op.result, op) + value = self.make_vstring_concat(op.result, op, mode) value.setup(vleft, vright, lengthbox) return True - def opt_call_oopspec_STR_SLICE(self, op): + def opt_call_stroruni_STR_SLICE(self, op, mode): newoperations = self.optimizer.newoperations vstr = self.getvalue(op.getarg(1)) vstart = self.getvalue(op.getarg(2)) @@ -380,7 +460,7 @@ if (isinstance(vstr, VStringPlainValue) and vstart.is_constant() and vstop.is_constant()): # slicing with constant bounds of a VStringPlainValue - value = self.make_vstring_plain(op.result, op) + value = self.make_vstring_plain(op.result, op, mode) value.setup_slice(vstr._chars, vstart.box.getint(), vstop.box.getint()) return True @@ -398,16 +478,16 @@ vstart.force_box()) vstart = self.getvalue(startbox) # - value = self.make_vstring_slice(op.result, op) + value = self.make_vstring_slice(op.result, op, mode) value.setup(vstr, vstart, self.getvalue(lengthbox)) return True - def opt_call_oopspec_STR_EQUAL(self, op): + def opt_call_stroruni_STR_EQUAL(self, op, mode): v1 = self.getvalue(op.getarg(1)) v2 = self.getvalue(op.getarg(2)) # - l1box = v1.getstrlen(None) - l2box = v2.getstrlen(None) + l1box = v1.getstrlen(None, mode) + l2box = v2.getstrlen(None, mode) if (l1box is not None and l2box is not None and isinstance(l1box, ConstInt) and isinstance(l2box, ConstInt) and @@ -416,13 +496,13 @@ self.make_constant(op.result, CONST_0) return True # - if self.handle_str_equal_level1(v1, v2, op.result): + if self.handle_str_equal_level1(v1, v2, op.result, mode): return True - if self.handle_str_equal_level1(v2, v1, op.result): + if self.handle_str_equal_level1(v2, v1, op.result, mode): return True - if self.handle_str_equal_level2(v1, v2, op.result): + if self.handle_str_equal_level2(v1, v2, op.result, mode): return True - if self.handle_str_equal_level2(v2, v1, op.result): + if self.handle_str_equal_level2(v2, v1, op.result, mode): return True # if v1.is_nonnull() and v2.is_nonnull(): @@ -434,37 +514,37 @@ else: do = EffectInfo.OS_STREQ_NONNULL self.generate_modified_call(do, [v1.force_box(), - v2.force_box()], op.result) + v2.force_box()], op.result, mode) return True return False - def handle_str_equal_level1(self, v1, v2, resultbox): - l2box = v2.getstrlen(None) + def handle_str_equal_level1(self, v1, v2, resultbox, mode): + l2box = v2.getstrlen(None, mode) if isinstance(l2box, ConstInt): if l2box.value == 0: - lengthbox = v1.getstrlen(self.optimizer.newoperations) + lengthbox = v1.getstrlen(self.optimizer.newoperations, mode) seo = self.optimizer.send_extra_operation seo(ResOperation(rop.INT_EQ, [lengthbox, CONST_0], resultbox)) return True if l2box.value == 1: - l1box = v1.getstrlen(None) + l1box = v1.getstrlen(None, mode) if isinstance(l1box, ConstInt) and l1box.value == 1: # comparing two single chars - vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO) - vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO) + vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO, mode) + vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO, mode) seo = self.optimizer.send_extra_operation seo(ResOperation(rop.INT_EQ, [vchar1.force_box(), vchar2.force_box()], resultbox)) return True if isinstance(v1, VStringSliceValue): - vchar = self.strgetitem(v2, optimizer.CVAL_ZERO) + vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode) do = EffectInfo.OS_STREQ_SLICE_CHAR self.generate_modified_call(do, [v1.vstr.force_box(), v1.vstart.force_box(), v1.vlength.force_box(), vchar.force_box()], - resultbox) + resultbox, mode) return True # if v2.is_null(): @@ -482,17 +562,18 @@ # return False - def handle_str_equal_level2(self, v1, v2, resultbox): - l2box = v2.getstrlen(None) + def handle_str_equal_level2(self, v1, v2, resultbox, mode): + l2box = v2.getstrlen(None, mode) if isinstance(l2box, ConstInt): if l2box.value == 1: - vchar = self.strgetitem(v2, optimizer.CVAL_ZERO) + vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode) if v1.is_nonnull(): do = EffectInfo.OS_STREQ_NONNULL_CHAR else: do = EffectInfo.OS_STREQ_CHECKNULL_CHAR self.generate_modified_call(do, [v1.force_box(), - vchar.force_box()], resultbox) + vchar.force_box()], resultbox, + mode) return True # if v1.is_virtual() and isinstance(v1, VStringSliceValue): @@ -503,11 +584,12 @@ self.generate_modified_call(do, [v1.vstr.force_box(), v1.vstart.force_box(), v1.vlength.force_box(), - v2.force_box()], resultbox) + v2.force_box()], resultbox, mode) return True return False - def generate_modified_call(self, oopspecindex, args, result): + def generate_modified_call(self, oopspecindex, args, result, mode): + oopspecindex += mode.OS_offset calldescr, func = callinfo_for_oopspec(oopspecindex) op = ResOperation(rop.CALL, [ConstInt(func)] + args, result, descr=calldescr) @@ -525,7 +607,7 @@ optimize_ops = _findall(OptString, 'optimize_') def _findall_call_oopspec(): - prefix = 'opt_call_oopspec_' + prefix = 'opt_call_stroruni_' result = [] for name in dir(OptString): if name.startswith(prefix): Modified: pypy/branch/jit-str-unicode/pypy/jit/metainterp/resume.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/metainterp/resume.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/metainterp/resume.py Mon Oct 11 14:26:19 2010 @@ -255,13 +255,19 @@ def make_varray(self, arraydescr): return VArrayInfo(arraydescr) - def make_vstrplain(self): + def make_vstrplain(self, is_unicode=False): + if is_unicode: + return VUniPlainInfo() return VStrPlainInfo() - def make_vstrconcat(self): + def make_vstrconcat(self, is_unicode=False): + if is_unicode: + return VUniConcatInfo() return VStrConcatInfo() - def make_vstrslice(self): + def make_vstrslice(self, is_unicode=False): + if is_unicode: + return VUniSliceInfo() return VStrSliceInfo() def register_virtual_fields(self, virtualbox, fieldboxes): @@ -550,6 +556,60 @@ for i in self.fieldnums: debug_print("\t\t", str(untag(i))) + +class VUniPlainInfo(AbstractVirtualInfo): + """Stands for the unicode string made out of the characters of all + fieldnums.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + length = len(self.fieldnums) + string = decoder.allocate_unicode(length) + decoder.virtuals_cache[index] = string + for i in range(length): + decoder.unicode_setitem(string, i, self.fieldnums[i]) + return string + + def debug_prints(self): + debug_print("\tvuniplaininfo length", len(self.fieldnums)) + + +class VUniConcatInfo(AbstractVirtualInfo): + """Stands for the unicode string made out of the concatenation of two + other unicode strings.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + # xxx for blackhole resuming, this will build all intermediate + # strings and throw them away immediately, which is a bit sub- + # efficient. Not sure we care. + left, right = self.fieldnums + string = decoder.concat_unicodes(left, right) + decoder.virtuals_cache[index] = string + return string + + def debug_prints(self): + debug_print("\tvuniconcatinfo") + for i in self.fieldnums: + debug_print("\t\t", str(untag(i))) + + +class VUniSliceInfo(AbstractVirtualInfo): + """Stands for the unicode string made out of slicing another + unicode string.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + largerstr, start, length = self.fieldnums + string = decoder.slice_unicode(largerstr, start, length) + decoder.virtuals_cache[index] = string + return string + + def debug_prints(self): + debug_print("\tvunisliceinfo") + for i in self.fieldnums: + debug_print("\t\t", str(untag(i))) + # ____________________________________________________________ class AbstractResumeDataReader(object): @@ -725,6 +785,32 @@ return self.metainterp.execute_and_record_varargs( rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr) + def allocate_unicode(self, length): + return self.metainterp.execute_and_record(rop.NEWUNICODE, + None, ConstInt(length)) + + def unicode_setitem(self, strbox, index, charnum): + charbox = self.decode_box(charnum, INT) + self.metainterp.execute_and_record(rop.UNICODESETITEM, None, + strbox, ConstInt(index), charbox) + + def concat_unicodes(self, str1num, str2num): + calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_CONCAT) + str1box = self.decode_box(str1num, REF) + str2box = self.decode_box(str2num, REF) + return self.metainterp.execute_and_record_varargs( + rop.CALL, [ConstInt(func), str1box, str2box], calldescr) + + def slice_unicode(self, strnum, startnum, lengthnum): + calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_SLICE) + strbox = self.decode_box(strnum, REF) + startbox = self.decode_box(startnum, INT) + lengthbox = self.decode_box(lengthnum, INT) + stopbox = self.metainterp.execute_and_record(rop.INT_ADD, None, + startbox, lengthbox) + return self.metainterp.execute_and_record_varargs( + rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr) + def setfield(self, descr, structbox, fieldnum): if descr.is_pointer_field(): kind = REF @@ -967,6 +1053,31 @@ result = funcptr(str, start, start + length) return lltype.cast_opaque_ptr(llmemory.GCREF, result) + def allocate_unicode(self, length): + return self.cpu.bh_newunicode(length) + + def unicode_setitem(self, str, index, charnum): + char = self.decode_int(charnum) + self.cpu.bh_unicodesetitem(str, index, char) + + def concat_unicodes(self, str1num, str2num): + str1 = self.decode_ref(str1num) + str2 = self.decode_ref(str2num) + str1 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str1) + str2 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str2) + funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_CONCAT) + result = funcptr(str1, str2) + return lltype.cast_opaque_ptr(llmemory.GCREF, result) + + def slice_unicode(self, strnum, startnum, lengthnum): + str = self.decode_ref(strnum) + start = self.decode_int(startnum) + length = self.decode_int(lengthnum) + str = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str) + funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_SLICE) + result = funcptr(str, start, start + length) + return lltype.cast_opaque_ptr(llmemory.GCREF, result) + def setfield(self, descr, struct, fieldnum): if descr.is_pointer_field(): newvalue = self.decode_ref(fieldnum) Modified: pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/oparser.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/oparser.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/oparser.py Mon Oct 11 14:26:19 2010 @@ -5,7 +5,7 @@ from pypy.jit.metainterp.history import TreeLoop, BoxInt, ConstInt,\ ConstObj, ConstPtr, Box, BasicFailDescr, BoxFloat, ConstFloat,\ - LoopToken, get_const_ptr_for_string + LoopToken, get_const_ptr_for_string, get_const_ptr_for_unicode from pypy.jit.metainterp.resoperation import rop, ResOperation, ResOpWithDescr, N_aryOp from pypy.jit.metainterp.typesystem import llhelper from pypy.jit.codewriter.heaptracker import adr2int @@ -158,10 +158,15 @@ except ValueError: if self.is_float(arg): return ConstFloat(float(arg)) - if arg.startswith('"') or arg.startswith("'"): + if (arg.startswith('"') or arg.startswith("'") or + arg.startswith('s"')): # XXX ootype - info = arg.strip("'\"") + info = arg[1:].strip("'\"") return get_const_ptr_for_string(info) + if arg.startswith('u"'): + # XXX ootype + info = arg[1:].strip("'\"") + return get_const_ptr_for_unicode(info) if arg.startswith('ConstClass('): name = arg[len('ConstClass('):-1] return self.get_const(name, 'class') Modified: pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_optimizefindnode.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_optimizefindnode.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_optimizefindnode.py Mon Oct 11 14:26:19 2010 @@ -117,33 +117,32 @@ EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE)) arraycopydescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, EffectInfo([], [], [], oopspecindex=EffectInfo.OS_ARRAYCOPY)) - strconcatdescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_CONCAT)) - slicedescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_SLICE)) - strequaldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_EQUAL)) - streq_slice_checknull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_SLICE_CHECKNULL)) - streq_slice_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_SLICE_NONNULL)) - streq_slice_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_SLICE_CHAR)) - streq_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_NONNULL)) - streq_nonnull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_NONNULL_CHAR)) - streq_checknull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_CHECKNULL_CHAR)) - streq_lengthok_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_LENGTHOK)) + + for _name, _os in [ + ('strconcatdescr', 'OS_STR_CONCAT'), + ('strslicedescr', 'OS_STR_SLICE'), + ('strequaldescr', 'OS_STR_EQUAL'), + ('streq_slice_checknull_descr', 'OS_STREQ_SLICE_CHECKNULL'), + ('streq_slice_nonnull_descr', 'OS_STREQ_SLICE_NONNULL'), + ('streq_slice_char_descr', 'OS_STREQ_SLICE_CHAR'), + ('streq_nonnull_descr', 'OS_STREQ_NONNULL'), + ('streq_nonnull_char_descr', 'OS_STREQ_NONNULL_CHAR'), + ('streq_checknull_char_descr', 'OS_STREQ_CHECKNULL_CHAR'), + ('streq_lengthok_descr', 'OS_STREQ_LENGTHOK'), + ]: + _oopspecindex = getattr(EffectInfo, _os) + locals()[_name] = \ + cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=_oopspecindex)) + # + _oopspecindex = getattr(EffectInfo, _os.replace('STR', 'UNI')) + locals()[_name.replace('str', 'unicode')] = \ + cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=_oopspecindex)) + + s2u_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR2UNICODE)) + # class LoopToken(AbstractDescr): pass Modified: pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_optimizeopt.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_optimizeopt.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_optimizeopt.py Mon Oct 11 14:26:19 2010 @@ -3487,7 +3487,7 @@ i0 = strlen(p0) jump(p0) """ - self.optimize_loop(ops, 'Not', expected) + self.optimize_strunicode_loop(ops, 'Not', expected) def test_addsub_const(self): ops = """ @@ -3893,6 +3893,15 @@ """ self.optimize_loop(ops, 'Not, Not', expected) + # ---------- + def optimize_strunicode_loop(self, ops, spectext, optops): + # check with the arguments passed in + self.optimize_loop(ops, spectext, optops) + # check with replacing 'str' with 'unicode' everywhere + self.optimize_loop(ops.replace('str','unicode').replace('s"', 'u"'), + spectext, + optops.replace('str','unicode').replace('s"', 'u"')) + def test_newstr_1(self): ops = """ [i0] @@ -3905,7 +3914,7 @@ [i0] jump(i0) """ - self.optimize_loop(ops, 'Not', expected) + self.optimize_strunicode_loop(ops, 'Not', expected) def test_newstr_2(self): ops = """ @@ -3921,7 +3930,7 @@ [i0, i1] jump(i1, i0) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_concat_1(self): ops = """ @@ -3942,7 +3951,7 @@ copystrcontent(p2, p3, 0, i4, i5) jump(p2, p3) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_concat_vstr2_str(self): ops = """ @@ -3965,7 +3974,7 @@ copystrcontent(p2, p3, 0, 2, i4) jump(i1, i0, p3) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_concat_str_vstr2(self): ops = """ @@ -3989,7 +3998,7 @@ i6 = int_add(i5, 1) # will be killed by the backend jump(i1, i0, p3) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_concat_str_str_str(self): ops = """ @@ -4016,12 +4025,12 @@ copystrcontent(p3, p5, 0, i12b, i3b) jump(p2, p3, p5) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_concat_str_cstr1(self): ops = """ [p2] - p3 = call(0, p2, "x", descr=strconcatdescr) + p3 = call(0, p2, s"x", descr=strconcatdescr) jump(p3) """ expected = """ @@ -4035,28 +4044,28 @@ i5 = int_add(i4, 1) # will be killed by the backend jump(p3) """ - self.optimize_loop(ops, 'Not', expected) + self.optimize_strunicode_loop(ops, 'Not', expected) def test_str_concat_consts(self): ops = """ [] - p1 = same_as("ab") - p2 = same_as("cde") + p1 = same_as(s"ab") + p2 = same_as(s"cde") p3 = call(0, p1, p2, descr=strconcatdescr) escape(p3) jump() """ expected = """ [] - escape("abcde") + escape(s"abcde") jump() """ - self.optimize_loop(ops, '', expected) + self.optimize_strunicode_loop(ops, '', expected) def test_str_slice_1(self): ops = """ [p1, i1, i2] - p2 = call(0, p1, i1, i2, descr=slicedescr) + p2 = call(0, p1, i1, i2, descr=strslicedescr) jump(p2, i1, i2) """ expected = """ @@ -4066,12 +4075,12 @@ copystrcontent(p1, p2, i1, 0, i3) jump(p2, i1, i2) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_slice_2(self): ops = """ [p1, i2] - p2 = call(0, p1, 0, i2, descr=slicedescr) + p2 = call(0, p1, 0, i2, descr=strslicedescr) jump(p2, i2) """ expected = """ @@ -4080,13 +4089,13 @@ copystrcontent(p1, p2, 0, 0, i2) jump(p2, i2) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_slice_3(self): ops = """ [p1, i1, i2, i3, i4] - p2 = call(0, p1, i1, i2, descr=slicedescr) - p3 = call(0, p2, i3, i4, descr=slicedescr) + p2 = call(0, p1, i1, i2, descr=strslicedescr) + p3 = call(0, p2, i3, i4, descr=strslicedescr) jump(p3, i1, i2, i3, i4) """ expected = """ @@ -4098,12 +4107,12 @@ copystrcontent(p1, p3, i6, 0, i5) jump(p3, i1, i2, i3, i4) """ - self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not, Not', expected) def test_str_slice_getitem1(self): ops = """ [p1, i1, i2, i3] - p2 = call(0, p1, i1, i2, descr=slicedescr) + p2 = call(0, p1, i1, i2, descr=strslicedescr) i4 = strgetitem(p2, i3) escape(i4) jump(p1, i1, i2, i3) @@ -4116,7 +4125,7 @@ escape(i4) jump(p1, i1, i2, i3) """ - self.optimize_loop(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected) def test_str_slice_plain(self): ops = """ @@ -4124,7 +4133,7 @@ p1 = newstr(2) strsetitem(p1, 0, i3) strsetitem(p1, 1, i4) - p2 = call(0, p1, 1, 2, descr=slicedescr) + p2 = call(0, p1, 1, 2, descr=strslicedescr) i5 = strgetitem(p2, 0) escape(i5) jump(i3, i4) @@ -4134,12 +4143,12 @@ escape(i4) jump(i3, i4) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_slice_concat(self): ops = """ [p1, i1, i2, p2] - p3 = call(0, p1, i1, i2, descr=slicedescr) + p3 = call(0, p1, i1, i2, descr=strslicedescr) p4 = call(0, p3, p2, descr=strconcatdescr) jump(p4, i1, i2, p2) """ @@ -4155,10 +4164,10 @@ copystrcontent(p2, p4, 0, i3, i4b) jump(p4, i1, i2, p2) """ - self.optimize_loop(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected) # ---------- - def optimize_loop_extradescrs(self, ops, spectext, optops): + def optimize_strunicode_loop_extradescrs(self, ops, spectext, optops): from pypy.jit.metainterp.optimizeopt import string def my_callinfo_for_oopspec(oopspecindex): calldescrtype = type(LLtypeMixin.strequaldescr) @@ -4173,7 +4182,7 @@ saved = string.callinfo_for_oopspec try: string.callinfo_for_oopspec = my_callinfo_for_oopspec - self.optimize_loop(ops, spectext, optops) + self.optimize_strunicode_loop(ops, spectext, optops) finally: string.callinfo_for_oopspec = saved @@ -4184,7 +4193,7 @@ escape(i0) jump(p1, p2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not', ops) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', ops) def test_str_equal_noop2(self): ops = """ @@ -4209,12 +4218,13 @@ escape(i0) jump(p1, p2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not', + expected) def test_str_equal_slice1(self): ops = """ [p1, i1, i2, p3] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) i0 = call(0, p4, p3, descr=strequaldescr) escape(i0) jump(p1, i1, i2, p3) @@ -4226,12 +4236,13 @@ escape(i0) jump(p1, i1, i2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_slice2(self): ops = """ [p1, i1, i2, p3] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) i0 = call(0, p3, p4, descr=strequaldescr) escape(i0) jump(p1, i1, i2, p3) @@ -4243,13 +4254,14 @@ escape(i0) jump(p1, i1, i2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_slice3(self): ops = """ [p1, i1, i2, p3] guard_nonnull(p3) [] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) i0 = call(0, p3, p4, descr=strequaldescr) escape(i0) jump(p1, i1, i2, p3) @@ -4262,13 +4274,14 @@ escape(i0) jump(p1, i1, i2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_slice4(self): ops = """ [p1, i1, i2] - p3 = call(0, p1, i1, i2, descr=slicedescr) - i0 = call(0, p3, "x", descr=strequaldescr) + p3 = call(0, p1, i1, i2, descr=strslicedescr) + i0 = call(0, p3, s"x", descr=strequaldescr) escape(i0) jump(p1, i1, i2) """ @@ -4279,12 +4292,13 @@ escape(i0) jump(p1, i1, i2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not', + expected) def test_str_equal_slice5(self): ops = """ [p1, i1, i2, i3] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) p5 = newstr(1) strsetitem(p5, 0, i3) i0 = call(0, p5, p4, descr=strequaldescr) @@ -4298,7 +4312,8 @@ escape(i0) jump(p1, i1, i2, i3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_none1(self): ops = """ @@ -4313,7 +4328,7 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_none2(self): ops = """ @@ -4328,30 +4343,30 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull1(self): ops = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "hello world", descr=strequaldescr) + i0 = call(0, p1, s"hello world", descr=strequaldescr) escape(i0) jump(p1) """ expected = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "hello world", descr=streq_nonnull_descr) + i0 = call(0, p1, s"hello world", descr=streq_nonnull_descr) escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull2(self): ops = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "", descr=strequaldescr) + i0 = call(0, p1, s"", descr=strequaldescr) escape(i0) jump(p1) """ @@ -4363,13 +4378,13 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull3(self): ops = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "x", descr=strequaldescr) + i0 = call(0, p1, s"x", descr=strequaldescr) escape(i0) jump(p1) """ @@ -4380,13 +4395,13 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull4(self): ops = """ [p1, p2] p4 = call(0, p1, p2, descr=strconcatdescr) - i0 = call(0, "hello world", p4, descr=strequaldescr) + i0 = call(0, s"hello world", p4, descr=strequaldescr) escape(i0) jump(p1, p2) """ @@ -4401,17 +4416,17 @@ i5 = strlen(p2) i6 = int_add(i4, i5) # will be killed by the backend copystrcontent(p2, p4, 0, i4, i5) - i0 = call(0, "hello world", p4, descr=streq_nonnull_descr) + i0 = call(0, s"hello world", p4, descr=streq_nonnull_descr) escape(i0) jump(p1, p2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected) def test_str_equal_chars0(self): ops = """ [i1] p1 = newstr(0) - i0 = call(0, p1, "", descr=strequaldescr) + i0 = call(0, p1, s"", descr=strequaldescr) escape(i0) jump(i1) """ @@ -4420,14 +4435,14 @@ escape(1) jump(i1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_chars1(self): ops = """ [i1] p1 = newstr(1) strsetitem(p1, 0, i1) - i0 = call(0, p1, "x", descr=strequaldescr) + i0 = call(0, p1, s"x", descr=strequaldescr) escape(i0) jump(i1) """ @@ -4437,7 +4452,7 @@ escape(i0) jump(i1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_chars2(self): ops = """ @@ -4445,7 +4460,7 @@ p1 = newstr(2) strsetitem(p1, 0, i1) strsetitem(p1, 1, i2) - i0 = call(0, p1, "xy", descr=strequaldescr) + i0 = call(0, p1, s"xy", descr=strequaldescr) escape(i0) jump(i1, i2) """ @@ -4454,16 +4469,16 @@ p1 = newstr(2) strsetitem(p1, 0, i1) strsetitem(p1, 1, i2) - i0 = call(0, p1, "xy", descr=streq_lengthok_descr) + i0 = call(0, p1, s"xy", descr=streq_lengthok_descr) escape(i0) jump(i1, i2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected) def test_str_equal_chars3(self): ops = """ [p1] - i0 = call(0, "x", p1, descr=strequaldescr) + i0 = call(0, s"x", p1, descr=strequaldescr) escape(i0) jump(p1) """ @@ -4473,14 +4488,14 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_lengthmismatch1(self): ops = """ [i1] p1 = newstr(1) strsetitem(p1, 0, i1) - i0 = call(0, "xy", p1, descr=strequaldescr) + i0 = call(0, s"xy", p1, descr=strequaldescr) escape(i0) jump(i1) """ @@ -4489,10 +4504,33 @@ escape(0) jump(i1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) - # XXX unicode operations - # XXX str2unicode + def test_str2unicode_constant(self): + ops = """ + [] + p0 = call(0, "xy", descr=s2u_descr) # string -> unicode + escape(p0) + jump() + """ + expected = """ + [] + escape(u"xy") + jump() + """ + self.optimize_strunicode_loop_extradescrs(ops, '', expected) + + def test_str2unicode_nonconstant(self): + ops = """ + [p0] + p1 = call(0, p0, descr=s2u_descr) # string -> unicode + escape(p1) + jump(p1) + """ + self.optimize_strunicode_loop_extradescrs(ops, 'Not', ops) + # more generally, supporting non-constant but virtual cases is + # not obvious, because of the exception UnicodeDecodeError that + # can be raised by ll_str2unicode() ##class TestOOtype(BaseTestOptimizeOpt, OOtypeMixin): Modified: pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_resume.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_resume.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_resume.py Mon Oct 11 14:26:19 2010 @@ -240,6 +240,17 @@ return FakeBuiltObject(strconcat=[left, right]) def slice_string(self, str, start, length): return FakeBuiltObject(strslice=[str, start, length]) + def allocate_unicode(self, length): + return FakeBuiltObject(unistring=[None]*length) + def unicode_setitem(self, unistring, i, fieldnum): + value, tag = untag(fieldnum) + assert tag == TAGINT + assert 0 <= i < len(unistring.unistring) + unistring.unistring[i] = value + def concat_unicodes(self, left, right): + return FakeBuiltObject(uniconcat=[left, right]) + def slice_unicode(self, str, start, length): + return FakeBuiltObject(unislice=[str, start, length]) class FakeBuiltObject(object): def __init__(self, **kwds): @@ -304,6 +315,30 @@ assert reader.force_all_virtuals() == [ FakeBuiltObject(strslice=info.fieldnums)] +def test_vuniplaininfo(): + info = VUniPlainInfo() + info.fieldnums = [tag(60, TAGINT)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(unistring=[60])] + +def test_vuniconcatinfo(): + info = VUniConcatInfo() + info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(uniconcat=info.fieldnums)] + +def test_vunisliceinfo(): + info = VUniSliceInfo() + info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX), tag(30, TAGBOX)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(unislice=info.fieldnums)] + # ____________________________________________________________ Modified: pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_string.py ============================================================================== --- pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_string.py (original) +++ pypy/branch/jit-str-unicode/pypy/jit/metainterp/test/test_string.py Mon Oct 11 14:26:19 2010 @@ -6,14 +6,17 @@ class StringTests: + _str, _chr = str, chr + def test_eq_residual(self): + _str = self._str jitdriver = JitDriver(greens = [], reds = ['n', 'i', 's']) - global_s = "hello" + global_s = _str("hello") def f(n, b, s): if b: - s += "ello" + s += _str("ello") else: - s += "allo" + s += _str("allo") i = 0 while n > 0: jitdriver.can_enter_jit(s=s, n=n, i=i) @@ -21,18 +24,19 @@ n -= 1 + (s == global_s) i += 1 return i - res = self.meta_interp(f, [10, True, 'h'], listops=True) + res = self.meta_interp(f, [10, True, _str('h')], listops=True) assert res == 5 self.check_loops(**{self.CALL: 1, self.CALL_PURE: 0}) def test_eq_folded(self): + _str = self._str jitdriver = JitDriver(greens = ['s'], reds = ['n', 'i']) - global_s = "hello" + global_s = _str("hello") def f(n, b, s): if b: - s += "ello" + s += _str("ello") else: - s += "allo" + s += _str("allo") i = 0 while n > 0: jitdriver.can_enter_jit(s=s, n=n, i=i) @@ -40,31 +44,18 @@ n -= 1 + (s == global_s) i += 1 return i - res = self.meta_interp(f, [10, True, 'h'], listops=True) + res = self.meta_interp(f, [10, True, _str('h')], listops=True) assert res == 5 self.check_loops(**{self.CALL: 0, self.CALL_PURE: 0}) def test_newstr(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['n', 'm']) def f(n, m): while True: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - bytecode = 'adlfkj' + chr(n) - res = bytecode[n] - m -= 1 - if m < 0: - return ord(res) - res = self.meta_interp(f, [6, 10]) - assert res == 6 - - def test_newunicode(self): - jitdriver = JitDriver(greens = [], reds = ['n', 'm']) - def f(n, m): - while True: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - bytecode = u'adlfkj' + unichr(n) + bytecode = _str('adlfkj') + _chr(n) res = bytecode[n] m -= 1 if m < 0: @@ -73,95 +64,96 @@ assert res == 6 def test_char2string_pure(self): - for dochr in [chr, ]: #unichr]: - jitdriver = JitDriver(greens = [], reds = ['n']) - @dont_look_inside - def escape(x): - pass - def f(n): - while n > 0: - jitdriver.can_enter_jit(n=n) - jitdriver.jit_merge_point(n=n) - s = dochr(n) - if not we_are_jitted(): - s += s # forces to be a string - if n > 100: - escape(s) - n -= 1 - return 42 - self.meta_interp(f, [6]) - self.check_loops(newstr=0, strsetitem=0, strlen=0, - newunicode=0, unicodesetitem=0, unicodelen=0) + _str, _chr = self._str, self._chr + jitdriver = JitDriver(greens = [], reds = ['n']) + @dont_look_inside + def escape(x): + pass + def f(n): + while n > 0: + jitdriver.can_enter_jit(n=n) + jitdriver.jit_merge_point(n=n) + s = _chr(n) + if not we_are_jitted(): + s += s # forces to be a string + if n > 100: + escape(s) + n -= 1 + return 42 + self.meta_interp(f, [6]) + self.check_loops(newstr=0, strsetitem=0, strlen=0, + newunicode=0, unicodesetitem=0, unicodelen=0) def test_char2string_escape(self): - for dochr in [chr, ]: #unichr]: - jitdriver = JitDriver(greens = [], reds = ['n', 'total']) - @dont_look_inside - def escape(x): - return ord(x[0]) - def f(n): - total = 0 - while n > 0: - jitdriver.can_enter_jit(n=n, total=total) - jitdriver.jit_merge_point(n=n, total=total) - s = dochr(n) - if not we_are_jitted(): - s += s # forces to be a string - total += escape(s) - n -= 1 - return total - res = self.meta_interp(f, [6]) - assert res == 21 + _str, _chr = self._str, self._chr + jitdriver = JitDriver(greens = [], reds = ['n', 'total']) + @dont_look_inside + def escape(x): + return ord(x[0]) + def f(n): + total = 0 + while n > 0: + jitdriver.can_enter_jit(n=n, total=total) + jitdriver.jit_merge_point(n=n, total=total) + s = _chr(n) + if not we_are_jitted(): + s += s # forces to be a string + total += escape(s) + n -= 1 + return total + res = self.meta_interp(f, [6]) + assert res == 21 def test_char2string2char(self): - for dochr in [chr, ]: #unichr]: - jitdriver = JitDriver(greens = [], reds = ['m', 'total']) - def f(m): - total = 0 - while m > 0: - jitdriver.can_enter_jit(m=m, total=total) - jitdriver.jit_merge_point(m=m, total=total) - string = dochr(m) - if m > 100: - string += string # forces to be a string - # read back the character - c = string[0] - total += ord(c) - m -= 1 - return total - res = self.meta_interp(f, [6]) - assert res == 21 - self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0, - newunicode=0, unicodegetitem=0, unicodesetitem=0, - unicodelen=0) + _str, _chr = self._str, self._chr + jitdriver = JitDriver(greens = [], reds = ['m', 'total']) + def f(m): + total = 0 + while m > 0: + jitdriver.can_enter_jit(m=m, total=total) + jitdriver.jit_merge_point(m=m, total=total) + string = _chr(m) + if m > 100: + string += string # forces to be a string + # read back the character + c = string[0] + total += ord(c) + m -= 1 + return total + res = self.meta_interp(f, [6]) + assert res == 21 + self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0, + newunicode=0, unicodegetitem=0, unicodesetitem=0, + unicodelen=0) def test_strconcat_pure(self): - for somestr in ["abc", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - mylist = [somestr+str(i) for i in range(10)] - def f(n, m): - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + mylist[m] - if m > 100: - escape(s) - m -= 1 - return 42 - self.meta_interp(f, [6, 7]) - self.check_loops(newstr=0, strsetitem=0, - newunicode=0, unicodesetitem=0, - call=0, call_pure=0) + _str = self._str + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = [_str("abc") + _str(i) for i in range(10)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + mylist[m] + if m > 100: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=0, strsetitem=0, + newunicode=0, unicodesetitem=0, + call=0, call_pure=0) def test_strconcat_escape_str_str(self): + _str = self._str jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) @@ -171,46 +163,64 @@ m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=0, copystrcontent=2, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=0, copystrcontent=2, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=0, + copyunicodecontent=2, + call=1, call_pure=0) # escape def test_strconcat_escape_str_char(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + chr(m) + s = mylist[n] + _chr(m) escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=1, + copyunicodecontent=1, + call=1, call_pure=0) # escape def test_strconcat_escape_char_str(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = chr(n) + mylist[m] + s = _chr(n) + mylist[m] escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=1, + copyunicodecontent=1, + call=1, call_pure=0) # escape def test_strconcat_escape_char_char(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): @@ -219,91 +229,132 @@ while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = chr(n) + chr(m) + s = _chr(n) + _chr(m) escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=2, copystrcontent=0, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=2, copystrcontent=0, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=2, + copyunicodecontent=0, + call=1, call_pure=0) # escape def test_strconcat_escape_str_char_str(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + chr(n) + mylist[m] + s = mylist[n] + _chr(n) + mylist[m] escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=1, copystrcontent=2, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=1, copystrcontent=2, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=1, + copyunicodecontent=2, + call=1, call_pure=0) # escape def test_strconcat_guard_fail(self): - for somestr in ["abc", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - mylist = [somestr+str(i) for i in range(12)] - def f(n, m): - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + mylist[m] - if m & 1: - escape(s) - m -= 1 - return 42 - self.meta_interp(f, [6, 10]) + _str = self._str + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = [_str("abc") + _str(i) for i in range(12)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + mylist[m] + if m & 1: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 10]) def test_strslice(self): - for somestr in ["abc", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - def f(n, m): - assert n >= 0 - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = "foobarbazetc"[m:n] - if m <= 5: - escape(s) - m -= 1 - return 42 - self.meta_interp(f, [10, 10]) + _str = self._str + longstring = _str("foobarbazetc") + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + def f(n, m): + assert n >= 0 + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = longstring[m:n] + if m <= 5: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [10, 10]) def test_streq_char(self): - for somestr in ["?abcdefg", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - def f(n, m): - assert n >= 0 - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = somestr[:m] - escape(s == "?") - m -= 1 - return 42 - self.meta_interp(f, [6, 7]) - self.check_loops(newstr=0, newunicode=0) - - -class TestOOtype(StringTests, OOJitMixin): - CALL = "oosend" - CALL_PURE = "oosend_pure" + _str = self._str + longstring = _str("?abcdefg") + somechar = _str("?") + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + def f(n, m): + assert n >= 0 + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = longstring[:m] + escape(s == somechar) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=0, newunicode=0) + + +#class TestOOtype(StringTests, OOJitMixin): +# CALL = "oosend" +# CALL_PURE = "oosend_pure" class TestLLtype(StringTests, LLJitMixin): CALL = "call" CALL_PURE = "call_pure" + +class TestLLtypeUnicode(TestLLtype): + _str, _chr = unicode, unichr + + def test_str2unicode(self): + _str = self._str + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + class Foo: + pass + @dont_look_inside + def escape(x): + assert x == _str("6y") + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + foo = Foo() + foo.y = chr(m) + foo.y = "y" + s = _str(str(n)) + _str(foo.y) + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(call=3, # str(), _str(), escape() + newunicode=1, unicodegetitem=0, + unicodesetitem=1, copyunicodecontent=1) From arigo at codespeak.net Mon Oct 11 14:30:01 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 14:30:01 +0200 (CEST) Subject: [pypy-svn] r77781 - pypy/trunk/pypy/jit/backend/x86/test Message-ID: <20101011123001.99D8A282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 14:29:59 2010 New Revision: 77781 Modified: pypy/trunk/pypy/jit/backend/x86/test/test_runner.py Log: Unsure, but it seems that the debugging info changed. Fixed the test. Modified: pypy/trunk/pypy/jit/backend/x86/test/test_runner.py ============================================================================== --- pypy/trunk/pypy/jit/backend/x86/test/test_runner.py (original) +++ pypy/trunk/pypy/jit/backend/x86/test/test_runner.py Mon Oct 11 14:29:59 2010 @@ -506,8 +506,8 @@ self.cpu.execute_token(ops.token) # check debugging info name, struct = self.cpu.assembler.loop_run_counters[0] - assert name == 'xyz' + assert name == 0 # 'xyz' assert struct.i == 10 self.cpu.finish_once() lines = py.path.local(self.logfile + ".count").readlines() - assert lines[0] == '10 xyz\n' + assert lines[0] == '0:10\n' # '10 xyz\n' From arigo at codespeak.net Mon Oct 11 14:31:43 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 14:31:43 +0200 (CEST) Subject: [pypy-svn] r77782 - in pypy/trunk/pypy: module/signal translator/c/src Message-ID: <20101011123143.4CADE282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 14:31:40 2010 New Revision: 77782 Modified: pypy/trunk/pypy/module/signal/interp_signal.py pypy/trunk/pypy/translator/c/src/signals.h Log: In signals.h, set the bit 30 instead of the highest bit. Helps on 64-bit platforms because the constant is within 32 bits. Modified: pypy/trunk/pypy/module/signal/interp_signal.py ============================================================================== --- pypy/trunk/pypy/module/signal/interp_signal.py (original) +++ pypy/trunk/pypy/module/signal/interp_signal.py Mon Oct 11 14:31:40 2010 @@ -1,7 +1,6 @@ from pypy.interpreter.error import OperationError from pypy.interpreter.baseobjspace import W_Root, ObjSpace from pypy.interpreter.executioncontext import AsyncAction, AbstractActionFlag -from pypy.rlib.rarithmetic import LONG_BIT, intmask import signal as cpy_signal from pypy.rpython.lltypesystem import lltype, rffi from pypy.translator.tool.cbuild import ExternalCompilationInfo @@ -64,8 +63,8 @@ class CheckSignalAction(AsyncAction): """An action that is automatically invoked when a signal is received.""" - # The C-level signal handler sets the highest bit of pypysig_occurred: - bitmask = intmask(1 << (LONG_BIT-1)) + # The C-level signal handler sets the bit 30 of pypysig_occurred: + bitmask = 1 << 30 def __init__(self, space): AsyncAction.__init__(self, space) Modified: pypy/trunk/pypy/translator/c/src/signals.h ============================================================================== --- pypy/trunk/pypy/translator/c/src/signals.h (original) +++ pypy/trunk/pypy/translator/c/src/signals.h Mon Oct 11 14:31:40 2010 @@ -6,20 +6,6 @@ #include -#ifndef LONG_MAX -#if SIZEOF_LONG == 4 -#define LONG_MAX 0X7FFFFFFFL -#elif SIZEOF_LONG == 8 -#define LONG_MAX 0X7FFFFFFFFFFFFFFFL -#else -#error "could not set LONG_MAX in pyport.h" -#endif -#endif - -#ifndef LONG_MIN -#define LONG_MIN (-LONG_MAX-1) -#endif - #include #ifdef MS_WINDOWS @@ -28,10 +14,6 @@ #include -#ifndef SIG_ERR -#define SIG_ERR ((PyOS_sighandler_t)(-1)) -#endif - #if defined(PYOS_OS2) && !defined(PYCC_GCC) #define NSIG 12 #include @@ -65,11 +47,11 @@ /* utility to poll for signals that arrived */ int pypysig_poll(void); /* => signum or -1 */ -/* When a signal is received, the high bit of pypysig_occurred is set. - After all signals are processed by pypysig_poll(), the high bit is +/* When a signal is received, the bit 30 of pypysig_occurred is set. + After all signals are processed by pypysig_poll(), the bit 30 is cleared again. The variable is exposed and RPython code is free to use the other bits in any way. */ -#define PENDING_SIGNAL_BIT (LONG_MIN) /* high bit */ +#define PENDING_SIGNAL_BIT (1 << 30) /* This is a struct for the JIT. See interp_signal.py. */ struct pypysig_long_struct { long value; From arigo at codespeak.net Mon Oct 11 14:31:57 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 14:31:57 +0200 (CEST) Subject: [pypy-svn] r77783 - pypy/extradoc/planning Message-ID: <20101011123157.C840E282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 14:31:55 2010 New Revision: 77783 Modified: pypy/extradoc/planning/jit.txt Log: Fixed by r77782. Modified: pypy/extradoc/planning/jit.txt ============================================================================== --- pypy/extradoc/planning/jit.txt (original) +++ pypy/extradoc/planning/jit.txt Mon Oct 11 14:31:55 2010 @@ -81,9 +81,8 @@ - guard_true(frame.is_being_profiled) all over the place -- at the end of every loop there is on x86-64: - int_and(i49, Const(0x8000000000f00000)) - which is not a constant that fits in 32-bits. Optimize that. +- 64-bit issues: in test_pypy_c, test_array_intimg, the STORE_SUBSCR turns + into strange code (truncate the int to 32 bits). - xxx (find more examples :-) From arigo at codespeak.net Mon Oct 11 14:34:40 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 14:34:40 +0200 (CEST) Subject: [pypy-svn] r77784 - in pypy/trunk/pypy/rpython: lltypesystem test Message-ID: <20101011123440.D219C282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 14:34:35 2010 New Revision: 77784 Modified: pypy/trunk/pypy/rpython/lltypesystem/rclass.py pypy/trunk/pypy/rpython/test/test_rclass.py Log: Change the order of the instance attributes: order them by decreasing size. It might help make some instances more compact by removing the padding that the C compiler puts. Modified: pypy/trunk/pypy/rpython/lltypesystem/rclass.py ============================================================================== --- pypy/trunk/pypy/rpython/lltypesystem/rclass.py (original) +++ pypy/trunk/pypy/rpython/lltypesystem/rclass.py Mon Oct 11 14:34:35 2010 @@ -329,16 +329,33 @@ fields['__class__'] = 'typeptr', get_type_repr(self.rtyper) else: # instance attributes - if llfields is None: - llfields = [] attrs = self.classdef.attrs.items() attrs.sort() + myllfields = [] for name, attrdef in attrs: if not attrdef.readonly: r = self.rtyper.getrepr(attrdef.s_value) mangled_name = 'inst_' + name fields[name] = mangled_name, r - llfields.append((mangled_name, r.lowleveltype)) + myllfields.append((mangled_name, r.lowleveltype)) + + # Sort the instance attributes by decreasing "likely size", + # as reported by rffi.sizeof(), to minimize padding holes in C. + # Fields of the same size are sorted by name (by attrs.sort() + # above) just to minimize randomness. + def keysize((_, T)): + if T is lltype.Void: + return None + from pypy.rpython.lltypesystem.rffi import sizeof + try: + return -sizeof(T) + except StandardError: + return None + myllfields.sort(key = keysize) + if llfields is None: + llfields = myllfields + else: + llfields = llfields + myllfields self.rbase = getinstancerepr(self.rtyper, self.classdef.basedef, self.gcflavor) Modified: pypy/trunk/pypy/rpython/test/test_rclass.py ============================================================================== --- pypy/trunk/pypy/rpython/test/test_rclass.py (original) +++ pypy/trunk/pypy/rpython/test/test_rclass.py Mon Oct 11 14:34:35 2010 @@ -3,7 +3,7 @@ from pypy.translator.translator import TranslationContext, graphof from pypy.rpython.lltypesystem.lltype import * from pypy.rpython.ootypesystem import ootype -from pypy.rlib.rarithmetic import intmask +from pypy.rlib.rarithmetic import intmask, r_longlong from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin from pypy.objspace.flow.model import summary @@ -1001,6 +1001,40 @@ res = self.interpret(f, [5]) assert res == 0 + def test_order_of_fields(self): + class A(object): + pass + def f(n): + a = A() + a.as_int = n + a.as_char = chr(n) + a.as_unichar = unichr(n) + a.as_double = n + 0.5 + a.as_bool = bool(n) + a.as_void = None + a.as_longlong = r_longlong(n) + a.as_reference = A() + return a + + res = self.interpret(f, [5]) + names = list(typeOf(res).TO._names) + i = names.index('inst_as_int') + c = names.index('inst_as_char') + u = names.index('inst_as_unichar') + d = names.index('inst_as_double') + b = names.index('inst_as_bool') + v = names.index('inst_as_void') + l = names.index('inst_as_longlong') + r = names.index('inst_as_reference') + assert v == 1 # void fields are first + assert sorted([c, b]) == [7, 8] + if sys.maxint == 2147483647: + assert sorted([u, i, r]) == [4, 5, 6] # 32-bit types + assert sorted([d, l]) == [2, 3] # 64-bit types + else: + assert sorted([u]) == [6] # 32-bit types + assert sorted([i, r, d, l]) == [2, 3, 4, 5] # 64-bit types + class TestOOtype(BaseTestRclass, OORtypeMixin): From arigo at codespeak.net Mon Oct 11 14:35:17 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 14:35:17 +0200 (CEST) Subject: [pypy-svn] r77785 - in pypy/trunk/pypy: config objspace/std objspace/std/test rlib rlib/test rpython rpython/lltypesystem rpython/test Message-ID: <20101011123517.B338C282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 14:35:15 2010 New Revision: 77785 Added: pypy/trunk/pypy/objspace/std/strbufobject.py pypy/trunk/pypy/objspace/std/test/test_strbufobject.py Modified: pypy/trunk/pypy/config/pypyoption.py pypy/trunk/pypy/objspace/std/model.py pypy/trunk/pypy/objspace/std/stringobject.py pypy/trunk/pypy/objspace/std/stringtype.py pypy/trunk/pypy/rlib/rstring.py pypy/trunk/pypy/rlib/test/test_rstring.py pypy/trunk/pypy/rpython/lltypesystem/rbuilder.py pypy/trunk/pypy/rpython/rbuilder.py pypy/trunk/pypy/rpython/test/test_rbuilder.py Log: Experimental (disabled by default): add a W_StringBufferObject, similar to W_StringJoinObject but based on the StringBuilder class. Modified: pypy/trunk/pypy/config/pypyoption.py ============================================================================== --- pypy/trunk/pypy/config/pypyoption.py (original) +++ pypy/trunk/pypy/config/pypyoption.py Mon Oct 11 14:35:15 2010 @@ -198,6 +198,9 @@ BoolOption("withstrslice", "use strings optimized for slicing", default=False), + BoolOption("withstrbuf", "use strings optimized for addition (ver 2)", + default=False), + BoolOption("withprebuiltchar", "use prebuilt single-character string objects", default=False), @@ -210,7 +213,8 @@ BoolOption("withrope", "use ropes as the string implementation", default=False, requires=[("objspace.std.withstrslice", False), - ("objspace.std.withstrjoin", False)], + ("objspace.std.withstrjoin", False), + ("objspace.std.withstrbuf", False)], suggests=[("objspace.std.withprebuiltchar", True), ("objspace.std.sharesmallstr", True)]), Modified: pypy/trunk/pypy/objspace/std/model.py ============================================================================== --- pypy/trunk/pypy/objspace/std/model.py (original) +++ pypy/trunk/pypy/objspace/std/model.py Mon Oct 11 14:35:15 2010 @@ -18,6 +18,7 @@ "withsmallint" : ["smallintobject.W_SmallIntObject"], "withstrslice" : ["strsliceobject.W_StringSliceObject"], "withstrjoin" : ["strjoinobject.W_StringJoinObject"], + "withstrbuf" : ["strbufobject.W_StringBufferObject"], "withrope" : ["ropeobject.W_RopeObject", "ropeobject.W_RopeIterObject"], "withropeunicode": ["ropeunicodeobject.W_RopeUnicodeObject", @@ -75,6 +76,7 @@ from pypy.objspace.std import ropeunicodeobject from pypy.objspace.std import strsliceobject from pypy.objspace.std import strjoinobject + from pypy.objspace.std import strbufobject from pypy.objspace.std import typeobject from pypy.objspace.std import sliceobject from pypy.objspace.std import longobject @@ -222,6 +224,13 @@ (unicodeobject.W_UnicodeObject, strjoinobject.delegate_join2unicode) ] + elif config.objspace.std.withstrbuf: + self.typeorder[strbufobject.W_StringBufferObject] += [ + (stringobject.W_StringObject, + strbufobject.delegate_buf2str), + (unicodeobject.W_UnicodeObject, + strbufobject.delegate_buf2unicode) + ] if config.objspace.std.withrangelist: self.typeorder[rangeobject.W_RangeListObject] += [ (listobject.W_ListObject, Added: pypy/trunk/pypy/objspace/std/strbufobject.py ============================================================================== --- (empty file) +++ pypy/trunk/pypy/objspace/std/strbufobject.py Mon Oct 11 14:35:15 2010 @@ -0,0 +1,76 @@ +from pypy.objspace.std.model import registerimplementation, W_Object +from pypy.objspace.std.register_all import register_all +from pypy.objspace.std.stringobject import W_StringObject +from pypy.objspace.std.unicodeobject import delegate_String2Unicode +from pypy.rlib.rstring import StringBuilder +from pypy.interpreter.buffer import Buffer + +class W_StringBufferObject(W_Object): + from pypy.objspace.std.stringtype import str_typedef as typedef + + w_str = None + + def __init__(self, builder): + self.builder = builder # StringBuilder + self.length = builder.getlength() + + def force(self): + if self.w_str is None: + s = self.builder.build() + if self.length < len(s): + s = s[:self.length] + self.w_str = W_StringObject(s) + return s + else: + return self.w_str._value + + def __repr__(w_self): + """ representation for debugging purposes """ + return "%s(%r[:%d])" % ( + w_self.__class__.__name__, w_self.builder, w_self.length) + + def unwrap(self, space): + return self.force() + +registerimplementation(W_StringBufferObject) + +# ____________________________________________________________ + +def joined2(str1, str2): + builder = StringBuilder() + builder.append(str1) + builder.append(str2) + return W_StringBufferObject(builder) + +# ____________________________________________________________ + +def delegate_buf2str(space, w_strbuf): + w_strbuf.force() + return w_strbuf.w_str + +def delegate_buf2unicode(space, w_strbuf): + w_strbuf.force() + return delegate_String2Unicode(space, w_strbuf.w_str) + +def len__StringBuffer(space, w_self): + return space.wrap(w_self.length) + +def str_w__StringBuffer(space, w_strbuf): + return w_strbuf.force() + +def add__StringBuffer_String(space, w_self, w_other): + if w_self.builder.getlength() != w_self.length: + builder = StringBuilder() + builder.append(w_self.force()) + else: + builder = w_self.builder + builder.append(w_other._value) + return W_StringBufferObject(builder) + +def str__StringBuffer(space, w_self): + # you cannot get subclasses of W_StringBufferObject here + assert type(w_self) is W_StringBufferObject + return w_self + +from pypy.objspace.std import stringtype +register_all(vars(), stringtype) Modified: pypy/trunk/pypy/objspace/std/stringobject.py ============================================================================== --- pypy/trunk/pypy/objspace/std/stringobject.py (original) +++ pypy/trunk/pypy/objspace/std/stringobject.py Mon Oct 11 14:35:15 2010 @@ -14,7 +14,7 @@ from pypy.rlib.rstring import StringBuilder, string_repeat from pypy.interpreter.buffer import StringBuffer -from pypy.objspace.std.stringtype import sliced, joined, wrapstr, wrapchar, \ +from pypy.objspace.std.stringtype import sliced, wrapstr, wrapchar, \ stringendswith, stringstartswith, joined2 from pypy.objspace.std.formatting import mod_format Modified: pypy/trunk/pypy/objspace/std/stringtype.py ============================================================================== --- pypy/trunk/pypy/objspace/std/stringtype.py (original) +++ pypy/trunk/pypy/objspace/std/stringtype.py Mon Oct 11 14:35:15 2010 @@ -55,19 +55,14 @@ return W_StringSliceObject(s, start, stop) return wrapstr(space, s[start:stop]) -def joined(space, strlist): - assert not space.config.objspace.std.withrope - if space.config.objspace.std.withstrjoin: - from pypy.objspace.std.strjoinobject import W_StringJoinObject - return W_StringJoinObject(strlist) - else: - return wrapstr(space, "".join(strlist)) - def joined2(space, str1, str2): assert not space.config.objspace.std.withrope if space.config.objspace.std.withstrjoin: from pypy.objspace.std.strjoinobject import W_StringJoinObject return W_StringJoinObject([str1, str2]) + elif space.config.objspace.std.withstrbuf: + from pypy.objspace.std.strbufobject import joined2 + return joined2(str1, str2) else: return wrapstr(space, str1 + str2) Added: pypy/trunk/pypy/objspace/std/test/test_strbufobject.py ============================================================================== --- (empty file) +++ pypy/trunk/pypy/objspace/std/test/test_strbufobject.py Mon Oct 11 14:35:15 2010 @@ -0,0 +1,78 @@ +import py + +from pypy.objspace.std.test import test_stringobject +from pypy.conftest import gettestobjspace + +class AppTestStringObject(test_stringobject.AppTestStringObject): + + def setup_class(cls): + cls.space = gettestobjspace(**{"objspace.std.withstrbuf": True}) + + def test_basic(self): + import __pypy__ + # cannot do "Hello, " + "World!" because cpy2.5 optimises this + # away on AST level + s = "Hello, ".__add__("World!") + assert type(s) is str + assert 'W_StringBufferObject' in __pypy__.internal_repr(s) + + def test_add_twice(self): + x = "a".__add__("b") + y = x + "c" + c = x + "d" + assert y == "abc" + assert c == "abd" + + def test_add(self): + import __pypy__ + all = "" + for i in range(20): + all += str(i) + assert 'W_StringBufferObject' in __pypy__.internal_repr(all) + assert all == "012345678910111213141516171819" + + def test_hash(self): + import __pypy__ + def join(s): return s[:len(s) // 2] + s[len(s) // 2:] + t = 'a' * 101 + s = join(t) + assert 'W_StringBufferObject' in __pypy__.internal_repr(s) + assert hash(s) == hash(t) + + def test_len(self): + s = "a".__add__("b") + r = "c".__add__("d") + t = s + r + assert len(s) == 2 + assert len(r) == 2 + assert len(t) == 4 + + def test_add_strbuf(self): + # make three strbuf objects + s = 'a'.__add__('b') + t = 'x'.__add__('c') + u = 'y'.__add__('d') + + # add two different strbufs to the same string + v = s + t + w = s + u + + # check that insanity hasn't resulted. + assert v == "abxc" + assert w == "abyd" + + def test_more_adding_fun(self): + s = 'a'.__add__('b') # s is a strbuf now + t = s + 'c' + u = s + 'd' + v = s + 'e' + assert v == 'abe' + assert u == 'abd' + assert t == 'abc' + + def test_buh_even_more(self): + a = 'a'.__add__('b') + b = a + 'c' + c = '0'.__add__('1') + x = c + a + assert x == '01ab' Modified: pypy/trunk/pypy/rlib/rstring.py ============================================================================== --- pypy/trunk/pypy/rlib/rstring.py (original) +++ pypy/trunk/pypy/rlib/rstring.py Mon Oct 11 14:35:15 2010 @@ -54,6 +54,7 @@ self.l = [] def append(self, s): + assert isinstance(s, self._type) self.l.append(s) def append_slice(self, s, start, end): @@ -63,11 +64,16 @@ def append_multiple_char(self, c, times): self.l.append(c * times) + def getlength(self): + return len(self.build()) + class StringBuilder(AbstractStringBuilder): + _type = str def build(self): return "".join(self.l) class UnicodeBuilder(AbstractStringBuilder): + _type = unicode def build(self): return u''.join(self.l) @@ -121,9 +127,12 @@ assert s_times.nonneg return s_None + def method_getlength(self): + return SomeInteger(nonneg=True) + def method_build(self): return SomeString() - + def rtyper_makerepr(self, rtyper): return rtyper.type_system.rbuilder.stringbuilder_repr @@ -146,6 +155,9 @@ assert s_times.nonneg return s_None + def method_getlength(self): + return SomeInteger(nonneg=True) + def method_build(self): return SomeUnicodeString() Modified: pypy/trunk/pypy/rlib/test/test_rstring.py ============================================================================== --- pypy/trunk/pypy/rlib/test/test_rstring.py (original) +++ pypy/trunk/pypy/rlib/test/test_rstring.py Mon Oct 11 14:35:15 2010 @@ -29,6 +29,7 @@ s = StringBuilder() s.append("a") s.append("abc") + assert s.getlength() == len('aabc') s.append("a") s.append_slice("abc", 1, 2) s.append_multiple_char('d', 4) @@ -39,6 +40,7 @@ s.append(u'a') s.append(u'abc') s.append_slice(u'abcdef', 1, 2) + assert s.getlength() == len('aabcb') s.append_multiple_char('d', 4) assert s.build() == 'aabcbdddd' assert isinstance(s.build(), unicode) Modified: pypy/trunk/pypy/rpython/lltypesystem/rbuilder.py ============================================================================== --- pypy/trunk/pypy/rpython/lltypesystem/rbuilder.py (original) +++ pypy/trunk/pypy/rpython/lltypesystem/rbuilder.py Mon Oct 11 14:35:15 2010 @@ -100,6 +100,10 @@ ll_builder.used = used @staticmethod + def ll_getlength(ll_builder): + return ll_builder.used + + @staticmethod def ll_build(ll_builder): final_size = ll_builder.used assert final_size >= 0 Modified: pypy/trunk/pypy/rpython/rbuilder.py ============================================================================== --- pypy/trunk/pypy/rpython/rbuilder.py (original) +++ pypy/trunk/pypy/rpython/rbuilder.py Mon Oct 11 14:35:15 2010 @@ -36,8 +36,12 @@ hop.exception_cannot_occur() return hop.gendirectcall(self.ll_append_multiple_char, *vlist) + def rtype_method_getlength(self, hop): + vlist = hop.inputargs(self) + hop.exception_cannot_occur() + return hop.gendirectcall(self.ll_getlength, *vlist) + def rtype_method_build(self, hop): vlist = hop.inputargs(self) hop.exception_cannot_occur() return hop.gendirectcall(self.ll_build, *vlist) - Modified: pypy/trunk/pypy/rpython/test/test_rbuilder.py ============================================================================== --- pypy/trunk/pypy/rpython/test/test_rbuilder.py (original) +++ pypy/trunk/pypy/rpython/test/test_rbuilder.py Mon Oct 11 14:35:15 2010 @@ -1,4 +1,4 @@ - +import py from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin from pypy.rpython.lltypesystem.rbuilder import * from pypy.rpython.annlowlevel import llstr, hlstr @@ -55,8 +55,29 @@ assert res == 'aabcabcdefbuuuu' assert isinstance(res, unicode) + def test_string_getlength(self): + def func(): + s = StringBuilder() + s.append("a") + s.append("abc") + return s.getlength() + res = self.interpret(func, []) + assert res == 4 + + def test_unicode_getlength(self): + def func(): + s = UnicodeBuilder() + s.append(u"a") + s.append(u"abc") + return s.getlength() + res = self.interpret(func, []) + assert res == 4 + class TestLLtype(BaseTestStringBuilder, LLRtypeMixin): pass class TestOOtype(BaseTestStringBuilder, OORtypeMixin): - pass + def test_string_getlength(self): + py.test.skip("getlength(): not implemented on ootype") + def test_unicode_getlength(self): + py.test.skip("getlength(): not implemented on ootype") From arigo at codespeak.net Mon Oct 11 14:36:10 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 14:36:10 +0200 (CEST) Subject: [pypy-svn] r77786 - in pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem: . test Message-ID: <20101011123610.7BE25282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 14:36:08 2010 New Revision: 77786 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rffi.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_rffi.py Log: Test and fix. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rffi.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rffi.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rffi.py Mon Oct 11 14:36:08 2010 @@ -796,6 +796,8 @@ size = llmemory.sizeof(tp) # a symbolic result in this case return size if isinstance(tp, lltype.Ptr): + if tp == llmemory.HiddenGcRef32: + return 4 tp = ULONG # XXX! if tp is lltype.Char or tp is lltype.Bool: return 1 @@ -805,8 +807,6 @@ return 8 if tp is lltype.SingleFloat: return 4 - if tp is llmemory.HiddenGcRef32: - return 4 assert isinstance(tp, lltype.Number) if tp is lltype.Signed: return ULONG._type.BITS/8 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_rffi.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_rffi.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/test/test_rffi.py Mon Oct 11 14:36:08 2010 @@ -750,6 +750,9 @@ assert offsetof(S, "c_b") == struct.calcsize("hi") - struct.calcsize("i") assert offsetof(S, "c_c") == struct.calcsize("hii") - struct.calcsize("i") + def test_rffi_sizeof_hiddengcref(self): + assert sizeof(llmemory.HiddenGcRef32) == 4 # even on 64-bit platforms + ARRAY_OF_CHAR = lltype.Array(CHAR, hints={'nolength': True}) def test_ptradd(): From arigo at codespeak.net Mon Oct 11 14:37:19 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 14:37:19 +0200 (CEST) Subject: [pypy-svn] r77787 - in pypy/branch/32ptr-on-64bit/pypy/rpython: . lltypesystem test Message-ID: <20101011123719.D6122282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 14:37:17 2010 New Revision: 77787 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lltype.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rlist.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rstr.py pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py pypy/branch/32ptr-on-64bit/pypy/rpython/rtuple.py pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rdict.py Log: Tests and fixes. Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/llinterp.py Mon Oct 11 14:37:17 2010 @@ -1445,7 +1445,7 @@ from pypy.rpython.lltypesystem import llarena try: result = llarena.getfakearenaaddress(result) - except RuntimeError: + except (RuntimeError, AttributeError): # xxx bad style pass # the GC should never see instances of _gctransformed_wref result = self.unwrap_possible_weakref(result) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lltype.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lltype.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/lltype.py Mon Oct 11 14:37:17 2010 @@ -542,6 +542,9 @@ class ForwardReference(ContainerType): _gckind = 'raw' + def __init__(self, will_be_varsize=False): + self._will_be_varsize = will_be_varsize + def become(self, realcontainertype): if not isinstance(realcontainertype, ContainerType): raise TypeError("ForwardReference can only be to a container, " @@ -549,12 +552,22 @@ if realcontainertype._gckind != self._gckind: raise TypeError("become() gives conflicting gckind, use the " "correct XxForwardReference") + if realcontainertype._is_varsize() != self._will_be_varsize: + raise TypeError("become() between non-varsize and varsize " + "containers -- initialize the ForwardReference " + "correctly") self.__class__ = realcontainertype.__class__ self.__dict__ = realcontainertype.__dict__ def __hash__(self): raise TypeError("%r object is not hashable" % self.__class__.__name__) + def _inline_is_varsize(self, last): + raise TypeError("%s._inline_is_varsize()" % self.__class__.__name__) + + def _is_varsize(self): + return self._will_be_varsize + class GcForwardReference(ForwardReference): _gckind = 'gc' Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py Mon Oct 11 14:37:17 2010 @@ -38,6 +38,7 @@ class CompressedGcRefRepr(Repr): lowleveltype = llmemory.HiddenGcRef32 + ll_eq_function = None ll_hash_function = None ll_fasthash_function = None @@ -53,9 +54,19 @@ return llop.hide_into_ptr32(self.lowleveltype, ptr) def get_ll_eq_function(self): - if self.baserepr.get_ll_eq_function() is not None: - raise TyperError("%r has an eq function" % (self.baserepr,)) - return None + if self.ll_eq_function is None: + basefunc = self.baserepr.get_ll_eq_function() + if basefunc is None: + return None + BASETYPE = self.BASETYPE + # + def ll_eq_function(x, y): + x = llop.show_from_ptr32(BASETYPE, x) + y = llop.show_from_ptr32(BASETYPE, y) + return basefunc(x, y) + # + self.ll_eq_function = ll_eq_function + return self.ll_eq_function def get_ll_hash_function(self): if self.ll_hash_function is None: Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py Mon Oct 11 14:37:17 2010 @@ -275,10 +275,14 @@ def _rtype_method_kvi(self, hop, ll_func): v_dic, = hop.inputargs(self) + s_list = hop.s_result + s_value = s_list.listdef.listitem.s_value + r_value = hop.rtyper.getrepr(s_value) + cLISTITEM = hop.inputconst(lltype.Void, r_value.lowleveltype) r_list = hop.r_result cLIST = hop.inputconst(lltype.Void, r_list.lowleveltype.TO) hop.exception_cannot_occur() - return hop.gendirectcall(ll_func, cLIST, v_dic) + return hop.gendirectcall(ll_func, cLIST, cLISTITEM, v_dic) def rtype_method_keys(self, hop): return self._rtype_method_kvi(hop, ll_dict_keys) @@ -778,7 +782,7 @@ return v def _make_ll_keys_values_items(kind): - def ll_kvi(LIST, dic): + def ll_kvi(LIST, LISTITEM, dic): res = LIST.ll_newlist(dic.num_items) entries = dic.entries dlen = len(entries) @@ -787,18 +791,18 @@ p = 0 while i < dlen: if entries.valid(i): - ELEM = lltype.typeOf(items).TO.OF - if ELEM is not lltype.Void: + if LISTITEM is not lltype.Void: entry = entries[i] if kind == 'items': - r = lltype.malloc(ELEM.TO) - r.item0 = recast(ELEM.TO.item0, entry.key) - r.item1 = recast(ELEM.TO.item1, entry.value) - items[p] = r + r = lltype.malloc(LISTITEM.TO) + r.item0 = recast(LISTITEM.TO.item0, entry.key) + r.item1 = recast(LISTITEM.TO.item1, entry.value) elif kind == 'keys': - items[p] = recast(ELEM, entry.key) + r = entry.key elif kind == 'values': - items[p] = recast(ELEM, entry.value) + r = entry.value + ELEM = lltype.typeOf(items).TO.OF + items[p] = recast(ELEM, r) p += 1 i += 1 assert p == res.ll_length() Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rlist.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rlist.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rlist.py Mon Oct 11 14:37:17 2010 @@ -44,7 +44,8 @@ # known_maxlength is ignored by lltype but used by ootype def __init__(self, rtyper, item_repr, listitem=None, known_maxlength=False): self.rtyper = rtyper - self.LIST = GcForwardReference() + will_be_varsize = isinstance(self, FixedSizeListRepr) + self.LIST = GcForwardReference(will_be_varsize=will_be_varsize) self.lowleveltype = Ptr(self.LIST) if not isinstance(item_repr, Repr): # not computed yet, done by setup() assert callable(item_repr) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rstr.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rstr.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rstr.py Mon Oct 11 14:37:17 2010 @@ -28,8 +28,8 @@ # chars: array of Char # } -STR = GcForwardReference() -UNICODE = GcForwardReference() +STR = GcForwardReference(will_be_varsize=True) +UNICODE = GcForwardReference(will_be_varsize=True) def new_malloc(TP, name): def mallocstr(length): Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/rmodel.py Mon Oct 11 14:37:17 2010 @@ -435,14 +435,16 @@ return alloc_flavor def externalvsinternal(rtyper, item_repr): # -> external_item_repr, (internal_)item_repr + if (rtyper is not None and rtyper.annotator is not None and # in tests + rtyper.annotator.translator.config.translation.compressptr): + item_repr, internal_item_repr = externalvsinternalfield(rtyper, + item_repr) + if internal_item_repr is not item_repr: + return item_repr, internal_item_repr + # from pypy.rpython import rclass if (isinstance(item_repr, rclass.AbstractInstanceRepr) and getattr(item_repr, 'gcflavor', 'gc') == 'gc'): - if rtyper.annotator.translator.config.translation.compressptr: - item_repr, internal_item_repr = externalvsinternalfield(rtyper, - item_repr) - if internal_item_repr is not item_repr: - return item_repr, internal_item_repr return item_repr, rclass.getinstancerepr(rtyper, None) return item_repr, item_repr Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/rtuple.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/rtuple.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/rtuple.py Mon Oct 11 14:37:17 2010 @@ -46,7 +46,7 @@ _gen_eq_function_cache[key] = ll_eq return ll_eq -import os + def gen_cmp_function(items_r, op_funcs, eq_funcs, strict): """generates <= and >= comparison ll_op for tuples cmp_funcs is a tuple of (strict_comp, equality) functions Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py Mon Oct 11 14:37:17 2010 @@ -1,6 +1,8 @@ import py from pypy.config.translationoption import IS_64_BITS from pypy.rpython.test import test_rclass +from pypy.rpython import rmodel, rint, rclass +from pypy.rpython.lltypesystem import llmemory def setup_module(mod): @@ -24,6 +26,148 @@ return super(MixinCompressed64, self).interpret_raises(*args, **kwds) +class TestExternalVsInternal(MixinCompressed64): + def setup_method(self, _): + class FakeRTyper(object): + class annotator: + class translator: + config = self._get_config() + class type_system: + from pypy.rpython.lltypesystem import rclass + def add_pendingsetup(self, r): + pass + self.rtyper = FakeRTyper() + self.rtyper.instance_reprs = {} + + def get_r_A(self): + class FakeClassDesc: + pyobj = None + def read_attribute(self, name, default): + return default + class FakeClassDef(object): + classdesc = FakeClassDesc() + def getallsubdefs(self): + return [self] + classdef = FakeClassDef() + return rclass.getinstancerepr(self.rtyper, classdef) + + def get_r_L(self, itemrepr, fixed=True): + from pypy.rpython.lltypesystem import rlist + class FakeListItem(object): + pass + if fixed: + cls = rlist.FixedSizeListRepr + else: + cls = rlist.ListRepr + return cls(self.rtyper, itemrepr, FakeListItem()) + + def test_simple(self): + er, ir = rmodel.externalvsinternal(self.rtyper, rint.signed_repr) + assert er is ir is rint.signed_repr + er, ir = rmodel.externalvsinternalfield(self.rtyper, rint.signed_repr) + assert er is ir is rint.signed_repr + + def test_instance(self): + r_A = self.get_r_A() + er, ir = rmodel.externalvsinternal(self.rtyper, r_A) + assert er is r_A + assert ir.lowleveltype == llmemory.HiddenGcRef32 + er, ir = rmodel.externalvsinternalfield(self.rtyper, r_A) + assert er is r_A + assert ir.lowleveltype == llmemory.HiddenGcRef32 + + def test_fixedlist_of_ints(self): + r_L = self.get_r_L(rint.signed_repr) + assert r_L.lowleveltype.TO._is_varsize() + assert r_L.external_item_repr is rint.signed_repr + assert r_L.item_repr is rint.signed_repr + er, ir = rmodel.externalvsinternal(self.rtyper, r_L) + assert er is ir is r_L + er, ir = rmodel.externalvsinternalfield(self.rtyper, r_L) + assert er is ir is r_L + + def test_varlist_of_ints(self): + r_L = self.get_r_L(rint.signed_repr, fixed=False) + assert r_L.external_item_repr is rint.signed_repr + assert r_L.item_repr is rint.signed_repr + er, ir = rmodel.externalvsinternal(self.rtyper, r_L) + assert er is r_L + assert ir.lowleveltype == llmemory.HiddenGcRef32 + er, ir = rmodel.externalvsinternalfield(self.rtyper, r_L) + assert er is r_L + assert ir.lowleveltype == llmemory.HiddenGcRef32 + + def test_fixedlist_of_instance(self): + r_A = self.get_r_A() + r_L = self.get_r_L(r_A) + assert r_L.external_item_repr is r_A + assert r_L.item_repr.lowleveltype == llmemory.HiddenGcRef32 + er, ir = rmodel.externalvsinternal(self.rtyper, r_L) + assert er is ir is r_L + er, ir = rmodel.externalvsinternalfield(self.rtyper, r_L) + assert er is ir is r_L + + def test_varlist_of_instance(self): + r_A = self.get_r_A() + r_L = self.get_r_L(r_A, fixed=False) + assert r_L.external_item_repr is r_A + assert r_L.item_repr.lowleveltype == llmemory.HiddenGcRef32 + er, ir = rmodel.externalvsinternal(self.rtyper, r_L) + assert er is r_L + assert ir.lowleveltype == llmemory.HiddenGcRef32 + er, ir = rmodel.externalvsinternalfield(self.rtyper, r_L) + assert er is r_L + assert ir.lowleveltype == llmemory.HiddenGcRef32 + + def test_fixedlist_of_fixedlist_of_instance(self): + r_A = self.get_r_A() + r_L1 = self.get_r_L(r_A) + r_L = self.get_r_L(r_L1) + assert r_L.external_item_repr is r_L1 + assert r_L.item_repr is r_L1 + er, ir = rmodel.externalvsinternal(self.rtyper, r_L) + assert er is ir is r_L + er, ir = rmodel.externalvsinternalfield(self.rtyper, r_L) + assert er is ir is r_L + + def test_fixedlist_of_varlist_of_instance(self): + r_A = self.get_r_A() + r_L1 = self.get_r_L(r_A, fixed=False) + r_L = self.get_r_L(r_L1) + assert r_L.external_item_repr is r_L1 + assert r_L.item_repr.lowleveltype == llmemory.HiddenGcRef32 + er, ir = rmodel.externalvsinternal(self.rtyper, r_L) + assert er is ir is r_L + er, ir = rmodel.externalvsinternalfield(self.rtyper, r_L) + assert er is ir is r_L + + def test_varlist_of_fixedlist_of_instance(self): + r_A = self.get_r_A() + r_L1 = self.get_r_L(r_A) + r_L = self.get_r_L(r_L1, fixed=False) + assert r_L.external_item_repr is r_L1 + assert r_L.item_repr is r_L1 + er, ir = rmodel.externalvsinternal(self.rtyper, r_L) + assert er is r_L + assert ir.lowleveltype == llmemory.HiddenGcRef32 + er, ir = rmodel.externalvsinternalfield(self.rtyper, r_L) + assert er is r_L + assert ir.lowleveltype == llmemory.HiddenGcRef32 + + def test_varlist_of_varlist_of_instance(self): + r_A = self.get_r_A() + r_L1 = self.get_r_L(r_A, fixed=False) + r_L = self.get_r_L(r_L1, fixed=False) + assert r_L.external_item_repr is r_L1 + assert r_L.item_repr.lowleveltype == llmemory.HiddenGcRef32 + er, ir = rmodel.externalvsinternal(self.rtyper, r_L) + assert er is r_L + assert ir.lowleveltype == llmemory.HiddenGcRef32 + er, ir = rmodel.externalvsinternalfield(self.rtyper, r_L) + assert er is r_L + assert ir.lowleveltype == llmemory.HiddenGcRef32 + + class TestLLtype64(MixinCompressed64, test_rclass.TestLLtype): def test_casts_1(self): @@ -56,3 +200,20 @@ a = d.values()[0] a.x = 5 self.interpret(fn, []) + + def test_dict_recast_2(self): + from pypy.rlib.objectmodel import r_dict + def fn(): + d = {4: 5} + return d.items()[0][1] + res = self.interpret(fn, []) + assert res == 5 + + def test_tuple_eq(self): + base = (5, (6, (7,))) + def fn(i, j, k): + return (i, (j, (k,))) == base + res = self.interpret(fn, [5, 6, 7]) + assert res == True + res = self.interpret(fn, [5, 6, 8]) + assert res == False Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rdict.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rdict.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rdict.py Mon Oct 11 14:37:17 2010 @@ -936,6 +936,7 @@ class PseudoRTyper: cache_dummy_values = {} + annotator = None dictrepr = rdict.DictRepr(PseudoRTyper(), string_repr, string_repr, DictKey(None, annmodel.SomeString(key_can_be_none)), DictValue(None, annmodel.SomeString(value_can_be_none))) From afa at codespeak.net Mon Oct 11 15:07:20 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 15:07:20 +0200 (CEST) Subject: [pypy-svn] r77788 - pypy/branch/fast-forward/lib_pypy Message-ID: <20101011130720.B7834282BF3@codespeak.net> Author: afa Date: Mon Oct 11 15:07:17 2010 New Revision: 77788 Modified: pypy/branch/fast-forward/lib_pypy/_subprocess.py Log: the multiprocessing module calls CreateProcess with startup_info=None. Modified: pypy/branch/fast-forward/lib_pypy/_subprocess.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_subprocess.py (original) +++ pypy/branch/fast-forward/lib_pypy/_subprocess.py Mon Oct 11 15:07:17 2010 @@ -133,14 +133,15 @@ def CreateProcess(name, command_line, process_attr, thread_attr, inherit, flags, env, start_dir, startup_info): si = _STARTUPINFO() - si.dwFlags = startup_info.dwFlags - si.wShowWindow = getattr(startup_info, 'wShowWindow', 0) - if startup_info.hStdInput: - si.hStdInput = startup_info.hStdInput.handle - if startup_info.hStdOutput: - si.hStdOutput = startup_info.hStdOutput.handle - if startup_info.hStdError: - si.hStdError = startup_info.hStdError.handle + if startup_info is not None: + si.dwFlags = startup_info.dwFlags + si.wShowWindow = startup_info.wShowWindow + if startup_info.hStdInput: + si.hStdInput = startup_info.hStdInput.handle + if startup_info.hStdOutput: + si.hStdOutput = startup_info.hStdOutput.handle + if startup_info.hStdError: + si.hStdError = startup_info.hStdError.handle pi = _PROCESS_INFORMATION() From afa at codespeak.net Mon Oct 11 15:12:31 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 15:12:31 +0200 (CEST) Subject: [pypy-svn] r77789 - in pypy/branch/fast-forward/pypy: module/_multiprocessing module/_multiprocessing/test rlib Message-ID: <20101011131231.F00E9282BF3@codespeak.net> Author: afa Date: Mon Oct 11 15:12:30 2010 New Revision: 77789 Added: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py (contents, props changed) pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py (contents, props changed) Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/__init__.py pypy/branch/fast-forward/pypy/rlib/rwin32.py Log: Partially implement _multiprocessing.SemLock. Won't work on Unix for now. Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/__init__.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/__init__.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/__init__.py Mon Oct 11 15:12:30 2010 @@ -6,6 +6,7 @@ interpleveldefs = { 'Connection' : 'interp_connection.W_FileConnection', 'PipeConnection' : 'interp_connection.W_PipeConnection', + 'SemLock' : 'interp_semaphore.W_SemLock', } appleveldefs = { @@ -13,3 +14,7 @@ if sys.platform == 'win32': interpleveldefs['win32'] = 'interp_win32.win32_namespace(space)' + + def startup(self, space): + from pypy.module._multiprocessing.interp_semaphore import CounterState + space.fromcache(CounterState).startup(space) Added: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Mon Oct 11 15:12:30 2010 @@ -0,0 +1,357 @@ +from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root +from pypy.interpreter.typedef import TypeDef, GetSetProperty +from pypy.interpreter.gateway import interp2app, unwrap_spec +from pypy.interpreter.error import ( + wrap_windowserror, wrap_oserror, OperationError) +from pypy.rpython.lltypesystem import rffi, lltype +from pypy.rlib.rarithmetic import r_uint +from pypy.translator.tool.cbuild import ExternalCompilationInfo +from pypy.module.thread import ll_thread +import sys, os, time + +RECURSIVE_MUTEX, SEMAPHORE = range(2) + +if sys.platform == 'win32': + from pypy.rlib import rwin32 + + _CreateSemaphore = rwin32.winexternal( + 'CreateSemaphoreA', [rffi.VOIDP, rffi.LONG, rffi.LONG, rwin32.LPCSTR], + rwin32.HANDLE) + _ReleaseSemaphore = rwin32.winexternal( + 'ReleaseSemaphore', [rwin32.HANDLE, rffi.LONG, rffi.LONGP], + rwin32.BOOL) + _GetTickCount = rwin32.winexternal( + 'GetTickCount', [], rwin32.DWORD) + + CtrlHandler_type = lltype.Ptr(lltype.FuncType([], rwin32.BOOL)) + _CreateEvent = rwin32.winexternal( + 'CreateEventA', [rffi.VOIDP, rwin32.BOOL, rwin32.BOOL, rwin32.LPCSTR], + rwin32.HANDLE) + _SetEvent = rwin32.winexternal( + 'SetEvent', [rwin32.HANDLE], rwin32.BOOL) + _ResetEvent = rwin32.winexternal( + 'ResetEvent', [rwin32.HANDLE], rwin32.BOOL) + + # This is needed because the handler function must have the "WINAPI" + # callinf convention, which is not supported by lltype.Ptr. + eci = ExternalCompilationInfo( + separate_module_sources=[''' + #include + + static BOOL (*CtrlHandlerRoutine)( + DWORD dwCtrlType); + + static BOOL WINAPI winapi_CtrlHandlerRoutine( + DWORD dwCtrlType) + { + return CtrlHandlerRoutine(dwCtrlType); + } + + BOOL pypy_multiprocessing_setCtrlHandlerRoutine(BOOL (*f)(DWORD)) + { + CtrlHandlerRoutine = f; + SetConsoleCtrlHandler(winapi_CtrlHandlerRoutine, TRUE); + } + + '''], + export_symbols=['pypy_multiprocessing_setCtrlHandlerRoutine'], + ) + _setCtrlHandlerRoutine = rffi.llexternal( + 'pypy_multiprocessing_setCtrlHandlerRoutine', + [CtrlHandler_type], rwin32.BOOL, + compilation_info=eci) + + def ProcessingCtrlHandler(): + _SetEvent(globalState.sigint_event) + return False + + class GlobalState: + def __init__(self): + self.init() + + def init(self): + self.sigint_event = rwin32.NULL_HANDLE + + def startup(self, space): + # Initialize the event handle used to signal Ctrl-C + globalState.sigint_event = _CreateEvent( + rffi.NULL, True, False, rffi.NULL) + if globalState.sigint_event == rwin32.NULL_HANDLE: + raise wrap_windowserror( + space, rwin32.lastWindowsError("CreateEvent")) + if not _setCtrlHandlerRoutine(ProcessingCtrlHandler): + raise wrap_windowserror( + space, rwin32.lastWindowsError("SetConsoleCtrlHandler")) + + +else: + class GlobalState: + def init(self): + pass + + def startup(self, space): + pass + +globalState = GlobalState() + +class CounterState: + def __init__(self, space): + self.counter = 0 + + def _freeze_(self): + self.counter = 0 + globalState.init() + + def startup(self, space): + globalState.startup(space) + + def getCount(self): + value = self.counter + self.counter += 1 + return value + +if sys.platform == 'win32': + SEM_VALUE_MAX = sys.maxint + from pypy.module._multiprocessing.interp_win32 import w_handle + + def create_semaphore(space, name, val, max): + rwin32.SetLastError(0) + handle = _CreateSemaphore(rffi.NULL, val, max, rffi.NULL) + # On Windows we should fail on ERROR_ALREADY_EXISTS + err = rwin32.GetLastError() + if err != 0: + raise wrap_windowserror( + space, WindowsError(err, "CreateSemaphore")) + return handle + + def semlock_acquire(self, space, block, w_timeout): + if not block: + full_msecs = 0 + elif space.is_w(w_timeout, space.w_None): + full_msecs = rwin32.INFINITE + else: + timeout = space.float_w(w_timeout) + timeout *= 1000.0 + if timeout < 0.0: + timeout = 0.0 + elif timeout >= 0.5 * rwin32.INFINITE: # 25 days + raise OperationError(space.w_OverflowError, + space.wrap("timeout is too large")) + full_msecs = int(timeout + 0.5) + + # check whether we can acquire without blocking + try: + res = rwin32.WaitForSingleObject(self.handle, 0) + except WindowsError, e: + raise wrap_windowserror(space, e) + + if res != rwin32.WAIT_TIMEOUT: + self.last_tid = ll_thread.get_ident() + self.count += 1 + return True + + msecs = r_uint(full_msecs) + start = _GetTickCount() + + while True: + handles = [self.handle, globalState.sigint_event] + + # do the wait + _ResetEvent(globalState.sigint_event) + try: + res = rwin32.WaitForMultipleObjects(handles, timeout=msecs) + except WindowsError, e: + raise wrap_windowserror(space, e) + + if res != rwin32.WAIT_OBJECT_0 + 1: + break + + # got SIGINT so give signal handler a chance to run + time.sleep(0.001) + + # if this is main thread let KeyboardInterrupt be raised + # XXX PyErr_CheckSignals() + + # recalculate timeout + if msecs != rwin32.INFINITE: + ticks = _GetTickCount() + if r_uint(ticks - start) >= full_msecs: + return False + msecs = r_uint(full_msecs - (ticks - start)) + + # handle result + if res != rwin32.WAIT_TIMEOUT: + self.last_tid = ll_thread.get_ident() + self.count += 1 + return True + return False + + def semlock_release(self, space): + if not _ReleaseSemaphore(self.handle, 1, + lltype.nullptr(rffi.LONGP.TO)): + err = rwin32.GetLastError() + if err == 0x0000012a: # ERROR_TOO_MANY_POSTS + raise OperationError( + space.w_ValueError, + space.wrap("semaphore or lock released too many times")) + else: + raise wrap_windowserror( + space, WindowsError(err, "ReleaseSemaphore")) + +else: + HAVE_BROKEN_SEM_GETVALUE = False + + def create_semaphore(space, name, val, max): + sem_open(name, os.O_CREAT | os.O_EXCL, 0600, val) + sem_unlink(name) + + def semlock_acquire(self, space, block, w_timeout): + if not block: + deadline = lltype.nullptr(TIMESPEC.TO) + elif space.is_w(w_timeout, space.w_None): + deadline = lltype.nullptr(TIMESPEC.TO) + else: + timeout = space.float_w(w_timeout) + sec = int(timeout) + nsec = int(1e9 * (timeout - sec) + 0.5) + + deadline = lltype.malloc(TIMESPEC.TO, 1, flavor='raw') + deadline.c_tv_sec = now.c_tv_sec + sec + deadline.c_tv_nsec = now.c_tv_usec * 1000 + nsec + deadline.c_tv_sec += (deadline.c_tv_nsec / 1000000000) + deadline.c_tv_nsec %= 1000000000 + try: + while True: + if not block: + res = sem_trywait(self.handle) + elif not deadline: + res = sem_wait(self.handle) + else: + res = sem_timedwait(self.handle, deadline) + if res >= 0: + break + elif errno != EINTR: + break + # elif PyErr_CheckSignals(): + # raise... + finally: + if deadline: + lltype.free(deadline, flavor='raw') + + if res < 0: + if errno == EAGAIN or errno == ETIMEDOUT: + return False + raise wrap_oserror(space, errno) + return True + + def semlock_release(self, space): + if self.kind == RECURSIVE_MUTEX: + return + if HAVE_BROKEN_SEM_GETVALUE: + # We will only check properly the maxvalue == 1 case + if self.maxvalue == 1: + # make sure that already locked + if sem_trywait(self.handle) < 0: + if errno != EAGAIN: + raise + # it is already locked as expected + else: + # it was not locked so undo wait and raise + if sem_post(self.handle) < 0: + raise + raise OperationError( + space.w_ValueError, space.wrap( + "semaphore or lock released too many times")) + else: + # This check is not an absolute guarantee that the semaphore does + # not rise above maxvalue. + if sem_getvalue(self.handle, sval_ptr) < 0: + raise + if sval_ptr[0] >= self.maxvalue: + raise OperationError( + space.w_ValueError, space.wrap( + "semaphore or lock released too many times")) + + if sem_post(self.handle) < 0: + raise + + +class W_SemLock(Wrappable): + def __init__(self, handle, kind, maxvalue): + self.handle = handle + self.kind = kind + self.count = 0 + self.maxvalue = maxvalue + + def kind_get(space, self): + return space.newint(self.kind) + def maxvalue_get(space, self): + return space.newint(self.maxvalue) + def handle_get(space, self): + return w_handle(space, self.handle) + + @unwrap_spec('self', ObjSpace) + def get_count(self, space): + return space.wrap(self.count) + + def _ismine(self): + return self.count > 0 and ll_thread.get_ident() == self.last_tid + + @unwrap_spec('self', ObjSpace) + def is_mine(self, space): + return space.wrap(self._ismine()) + + @unwrap_spec('self', ObjSpace, bool, W_Root) + def acquire(self, space, block=True, w_timeout=None): + # check whether we already own the lock + if self.kind == RECURSIVE_MUTEX and self._ismine(): + self.count += 1 + return space.w_True + + res = semlock_acquire(self, space, block, w_timeout) + return space.wrap(res) + + @unwrap_spec('self', ObjSpace) + def release(self, space): + if self.kind == RECURSIVE_MUTEX: + if not self._ismine(): + raise OperationError( + space.w_AssertionError, + space.wrap("attempt to release recursive lock" + " not owned by thread")) + if self.count > 1: + self.count -= 1 + return + + semlock_release(self, space) + + self.count -= 1 + + at unwrap_spec(ObjSpace, W_Root, int, int, int) +def descr_new(space, w_subtype, kind, value, maxvalue): + if kind != RECURSIVE_MUTEX and kind != SEMAPHORE: + raise OperationError(space.w_ValueError, + space.wrap("unrecognized kind")) + + counter = space.fromcache(CounterState).getCount() + name = "/mp%d-%d" % (os.getpid(), counter) + + handle = create_semaphore(space, name, value, maxvalue) + + self = space.allocate_instance(W_SemLock, w_subtype) + self.__init__(handle, kind, maxvalue) + + return space.wrap(self) + +W_SemLock.typedef = TypeDef( + "SemLock", + __new__ = interp2app(descr_new), + kind = GetSetProperty(W_SemLock.kind_get), + maxvalue = GetSetProperty(W_SemLock.maxvalue_get), + handle = GetSetProperty(W_SemLock.handle_get), + _count = interp2app(W_SemLock.get_count), + _is_mine = interp2app(W_SemLock.is_mine), + acquire = interp2app(W_SemLock.acquire), + release = interp2app(W_SemLock.release), + SEM_VALUE_MAX=SEM_VALUE_MAX, + ) Added: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py Mon Oct 11 15:12:30 2010 @@ -0,0 +1,38 @@ +from pypy.conftest import gettestobjspace +from pypy.module._multiprocessing.interp_semaphore import ( + RECURSIVE_MUTEX, SEMAPHORE) + +class AppTestSemaphore: + def setup_class(cls): + space = gettestobjspace(usemodules=('_multiprocessing', 'thread')) + cls.space = space + cls.w_SEMAPHORE = space.wrap(SEMAPHORE) + + def test_semaphore(self): + from _multiprocessing import SemLock + assert SemLock.SEM_VALUE_MAX > 10 + + kind = self.SEMAPHORE + value = 1 + maxvalue = 1 + sem = SemLock(kind, value, maxvalue) + assert sem.kind == kind + assert sem.maxvalue == maxvalue + assert isinstance(sem.handle, int) + + assert sem._count() == 0 + sem.acquire() + assert sem._is_mine() + assert sem._count() == 1 + sem.release() + assert sem._count() == 0 + + def test_semaphore_wait(self): + from _multiprocessing import SemLock + kind = self.SEMAPHORE + value = 1 + maxvalue = 1 + sem = SemLock(kind, value, maxvalue) + + assert sem.acquire() + assert not sem.acquire(timeout=0.1) Modified: pypy/branch/fast-forward/pypy/rlib/rwin32.py ============================================================================== --- pypy/branch/fast-forward/pypy/rlib/rwin32.py (original) +++ pypy/branch/fast-forward/pypy/rlib/rwin32.py Mon Oct 11 15:12:30 2010 @@ -71,6 +71,7 @@ for name in """FORMAT_MESSAGE_ALLOCATE_BUFFER FORMAT_MESSAGE_FROM_SYSTEM MAX_PATH + WAIT_OBJECT_0 WAIT_TIMEOUT INFINITE """.split(): locals()[name] = rffi_platform.ConstantInteger(name) @@ -241,3 +242,36 @@ info.c_szCSDVersion))) finally: lltype.free(info, flavor='raw') + + _WaitForSingleObject = winexternal( + 'WaitForSingleObject', [HANDLE, DWORD], DWORD) + + def WaitForSingleObject(handle, timeout): + """Return values: + - WAIT_OBJECT_0 when the object is signaled + - WAIT_TIMEOUT when the timeout elapsed""" + res = _WaitForSingleObject(handle, timeout) + if res == rffi.cast(DWORD, -1): + raise lastWindowsError("WaitForSingleObject") + return res + + _WaitForMultipleObjects = winexternal( + 'WaitForMultipleObjects', [ + DWORD, rffi.CArrayPtr(HANDLE), BOOL, DWORD], DWORD) + + def WaitForMultipleObjects(handles, waitall=False, timeout=INFINITE): + """Return values: + - WAIT_OBJECT_0 + index when an object is signaled + - WAIT_TIMEOUT when the timeout elapsed""" + nb = len(handles) + handle_array = lltype.malloc(rffi.CArrayPtr(HANDLE).TO, nb, + flavor='raw') + try: + for i in range(nb): + handle_array[i] = handles[i] + res = _WaitForMultipleObjects(nb, handle_array, waitall, timeout) + if res == rffi.cast(DWORD, -1): + raise lastWindowsError("WaitForMultipleObjects") + return res + finally: + lltype.free(handle_array, flavor='raw') From arigo at codespeak.net Mon Oct 11 15:19:43 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 15:19:43 +0200 (CEST) Subject: [pypy-svn] r77790 - in pypy/trunk/pypy/jit: backend/test backend/x86 backend/x86/test codewriter codewriter/test metainterp metainterp/optimizeopt metainterp/test Message-ID: <20101011131943.70DB5282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 15:19:40 2010 New Revision: 77790 Modified: pypy/trunk/pypy/jit/backend/test/runner_test.py pypy/trunk/pypy/jit/backend/test/test_ll_random.py pypy/trunk/pypy/jit/backend/x86/regalloc.py pypy/trunk/pypy/jit/backend/x86/test/test_string.py pypy/trunk/pypy/jit/codewriter/effectinfo.py pypy/trunk/pypy/jit/codewriter/jtransform.py pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py pypy/trunk/pypy/jit/metainterp/history.py pypy/trunk/pypy/jit/metainterp/optimizeopt/intbounds.py pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py pypy/trunk/pypy/jit/metainterp/optimizeopt/string.py pypy/trunk/pypy/jit/metainterp/resume.py pypy/trunk/pypy/jit/metainterp/test/oparser.py pypy/trunk/pypy/jit/metainterp/test/test_optimizefindnode.py pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py pypy/trunk/pypy/jit/metainterp/test/test_resume.py pypy/trunk/pypy/jit/metainterp/test/test_string.py Log: Merge branch/jit-str-unicode, adding support for optimizing unicode strings too in the jit. Modified: pypy/trunk/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/trunk/pypy/jit/backend/test/runner_test.py (original) +++ pypy/trunk/pypy/jit/backend/test/runner_test.py Mon Oct 11 15:19:40 2010 @@ -833,6 +833,23 @@ length_box], 'void') assert self.look_string(r_box) == "!??cdef?!" + def test_copyunicodecontent(self): + s_box = self.alloc_unicode(u"abcdef") + for s_box in [s_box, s_box.constbox()]: + for srcstart_box in [BoxInt(2), ConstInt(2)]: + for dststart_box in [BoxInt(3), ConstInt(3)]: + for length_box in [BoxInt(4), ConstInt(4)]: + for r_box_is_const in [False, True]: + r_box = self.alloc_unicode(u"!???????!") + if r_box_is_const: + r_box = r_box.constbox() + self.execute_operation(rop.COPYUNICODECONTENT, + [s_box, r_box, + srcstart_box, + dststart_box, + length_box], 'void') + assert self.look_unicode(r_box) == u"!??cdef?!" + def test_do_unicode_basic(self): u = self.cpu.bh_newunicode(5) self.cpu.bh_unicodesetitem(u, 4, 123) @@ -1227,6 +1244,10 @@ u_box = BoxPtr(lltype.cast_opaque_ptr(llmemory.GCREF, u)) return u_box + def look_unicode(self, unicode_box): + u = unicode_box.getref(lltype.Ptr(rstr.UNICODE)) + return u''.join(u.chars) + def test_casts(self): py.test.skip("xxx fix or kill") Modified: pypy/trunk/pypy/jit/backend/test/test_ll_random.py ============================================================================== --- pypy/trunk/pypy/jit/backend/test/test_ll_random.py (original) +++ pypy/trunk/pypy/jit/backend/test/test_ll_random.py Mon Oct 11 15:19:40 2010 @@ -599,7 +599,7 @@ OPERATIONS.append(StrLenOperation(rop.STRLEN)) OPERATIONS.append(UnicodeLenOperation(rop.UNICODELEN)) OPERATIONS.append(CopyStrContentOperation(rop.COPYSTRCONTENT)) - #OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT)) + OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT)) for i in range(2): OPERATIONS.append(GuardClassOperation(rop.GUARD_CLASS)) Modified: pypy/trunk/pypy/jit/backend/x86/regalloc.py ============================================================================== --- pypy/trunk/pypy/jit/backend/x86/regalloc.py (original) +++ pypy/trunk/pypy/jit/backend/x86/regalloc.py Mon Oct 11 15:19:40 2010 @@ -778,15 +778,11 @@ loc = self.loc(op.getarg(0)) return self._call(op, [loc]) # boehm GC (XXX kill the following code at some point) - ofs_items, itemsize, ofs = symbolic.get_array_token(rstr.UNICODE, self.translate_support_code) - if itemsize == 4: - return self._malloc_varsize(ofs_items, ofs, 2, op.getarg(0), - op.result) - elif itemsize == 2: - return self._malloc_varsize(ofs_items, ofs, 1, op.getarg(0), - op.result) - else: - assert False, itemsize + ofs_items, _, ofs = symbolic.get_array_token(rstr.UNICODE, + self.translate_support_code) + scale = self._get_unicode_item_scale() + return self._malloc_varsize(ofs_items, ofs, scale, op.getarg(0), + op.result) def _malloc_varsize(self, ofs_items, ofs_length, scale, v, res_v): # XXX kill this function at some point @@ -959,6 +955,12 @@ consider_unicodegetitem = consider_strgetitem def consider_copystrcontent(self, op): + self._consider_copystrcontent(op, is_unicode=False) + + def consider_copyunicodecontent(self, op): + self._consider_copystrcontent(op, is_unicode=True) + + def _consider_copystrcontent(self, op, is_unicode): # compute the source address args = op.getarglist() base_loc = self.rm.make_sure_var_in_reg(args[0], args) @@ -970,7 +972,8 @@ srcaddr_box = TempBox() forbidden_vars = [args[1], args[3], args[4], srcaddr_box] srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box, forbidden_vars) - self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc) + self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc, + is_unicode=is_unicode) # compute the destination address base_loc = self.rm.make_sure_var_in_reg(args[1], forbidden_vars) ofs_loc = self.rm.make_sure_var_in_reg(args[3], forbidden_vars) @@ -980,25 +983,57 @@ forbidden_vars = [args[4], srcaddr_box] dstaddr_box = TempBox() dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box, forbidden_vars) - self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc) + self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc, + is_unicode=is_unicode) + # compute the length in bytes + length_box = args[4] + length_loc = self.loc(length_box) + if is_unicode: + self.rm.possibly_free_var(length_box) + forbidden_vars = [srcaddr_box, dstaddr_box] + bytes_box = TempBox() + bytes_loc = self.rm.force_allocate_reg(bytes_box, forbidden_vars) + scale = self._get_unicode_item_scale() + if not (isinstance(length_loc, ImmedLoc) or + isinstance(length_loc, RegLoc)): + self.assembler.mov(length_loc, bytes_loc) + length_loc = bytes_loc + self.assembler.load_effective_addr(length_loc, 0, scale, bytes_loc) + length_box = bytes_box + length_loc = bytes_loc # call memcpy() - length_loc = self.loc(args[4]) self.rm.before_call() self.xrm.before_call() self.assembler._emit_call(imm(self.assembler.memcpy_addr), [dstaddr_loc, srcaddr_loc, length_loc]) - self.rm.possibly_free_var(args[4]) + self.rm.possibly_free_var(length_box) self.rm.possibly_free_var(dstaddr_box) self.rm.possibly_free_var(srcaddr_box) - def _gen_address_inside_string(self, baseloc, ofsloc, resloc): + def _gen_address_inside_string(self, baseloc, ofsloc, resloc, is_unicode): cpu = self.assembler.cpu - ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR, + if is_unicode: + ofs_items, _, _ = symbolic.get_array_token(rstr.UNICODE, + self.translate_support_code) + scale = self._get_unicode_item_scale() + else: + ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR, self.translate_support_code) - assert itemsize == 1 - self.assembler.load_effective_addr(ofsloc, ofs_items, 0, + assert itemsize == 1 + scale = 0 + self.assembler.load_effective_addr(ofsloc, ofs_items, scale, resloc, baseloc) + def _get_unicode_item_scale(self): + _, itemsize, _ = symbolic.get_array_token(rstr.UNICODE, + self.translate_support_code) + if itemsize == 4: + return 2 + elif itemsize == 2: + return 1 + else: + raise AssertionError("bad unicode item size") + def consider_jump(self, op): assembler = self.assembler assert self.jump_target_descr is None Modified: pypy/trunk/pypy/jit/backend/x86/test/test_string.py ============================================================================== --- pypy/trunk/pypy/jit/backend/x86/test/test_string.py (original) +++ pypy/trunk/pypy/jit/backend/x86/test/test_string.py Mon Oct 11 15:19:40 2010 @@ -2,8 +2,12 @@ from pypy.jit.metainterp.test import test_string from pypy.jit.backend.x86.test.test_basic import Jit386Mixin -class TestString(Jit386Mixin, test_string.StringTests): +class TestString(Jit386Mixin, test_string.TestLLtype): # for the individual tests see # ====> ../../../metainterp/test/test_string.py - CALL = 'call' - CALL_PURE = 'call_pure' + pass + +class TestUnicode(Jit386Mixin, test_string.TestLLtypeUnicode): + # for the individual tests see + # ====> ../../../metainterp/test/test_string.py + pass Modified: pypy/trunk/pypy/jit/codewriter/effectinfo.py ============================================================================== --- pypy/trunk/pypy/jit/codewriter/effectinfo.py (original) +++ pypy/trunk/pypy/jit/codewriter/effectinfo.py Mon Oct 11 15:19:40 2010 @@ -18,19 +18,30 @@ # the 'oopspecindex' field is one of the following values: OS_NONE = 0 # normal case, no oopspec OS_ARRAYCOPY = 1 # "list.ll_arraycopy" - OS_STR_CONCAT = 2 # "stroruni.concat" - OS_UNI_CONCAT = 3 # "stroruni.concat" - OS_STR_SLICE = 4 # "stroruni.slice" - OS_UNI_SLICE = 5 # "stroruni.slice" - OS_STR_EQUAL = 6 # "stroruni.equal" - OS_UNI_EQUAL = 7 # "stroruni.equal" - OS_STREQ_SLICE_CHECKNULL = 8 # s2!=NULL and s1[x:x+length]==s2 - OS_STREQ_SLICE_NONNULL = 9 # s1[x:x+length]==s2 (assert s2!=NULL) - OS_STREQ_SLICE_CHAR = 10 # s1[x:x+length]==char - OS_STREQ_NONNULL = 11 # s1 == s2 (assert s1!=NULL,s2!=NULL) - OS_STREQ_NONNULL_CHAR = 12 # s1 == char (assert s1!=NULL) - OS_STREQ_CHECKNULL_CHAR = 13 # s1!=NULL and s1==char - OS_STREQ_LENGTHOK = 14 # s1 == s2 (assert len(s1)==len(s2)) + OS_STR2UNICODE = 2 # "str.str2unicode" + + OS_STR_CONCAT = 22 # "stroruni.concat" + OS_STR_SLICE = 23 # "stroruni.slice" + OS_STR_EQUAL = 24 # "stroruni.equal" + OS_STREQ_SLICE_CHECKNULL = 25 # s2!=NULL and s1[x:x+length]==s2 + OS_STREQ_SLICE_NONNULL = 26 # s1[x:x+length]==s2 (assert s2!=NULL) + OS_STREQ_SLICE_CHAR = 27 # s1[x:x+length]==char + OS_STREQ_NONNULL = 28 # s1 == s2 (assert s1!=NULL,s2!=NULL) + OS_STREQ_NONNULL_CHAR = 29 # s1 == char (assert s1!=NULL) + OS_STREQ_CHECKNULL_CHAR = 30 # s1!=NULL and s1==char + OS_STREQ_LENGTHOK = 31 # s1 == s2 (assert len(s1)==len(s2)) + + OS_UNI_CONCAT = 42 # + OS_UNI_SLICE = 43 # + OS_UNI_EQUAL = 44 # + OS_UNIEQ_SLICE_CHECKNULL = 45 # + OS_UNIEQ_SLICE_NONNULL = 46 # + OS_UNIEQ_SLICE_CHAR = 47 # + OS_UNIEQ_NONNULL = 48 # the same for unicode + OS_UNIEQ_NONNULL_CHAR = 49 # (must be the same amount as for + OS_UNIEQ_CHECKNULL_CHAR = 50 # STR, in the same order) + OS_UNIEQ_LENGTHOK = 51 # + _OS_offset_uni = OS_UNI_CONCAT - OS_STR_CONCAT def __new__(cls, readonly_descrs_fields, write_descrs_fields, write_descrs_arrays, Modified: pypy/trunk/pypy/jit/codewriter/jtransform.py ============================================================================== --- pypy/trunk/pypy/jit/codewriter/jtransform.py (original) +++ pypy/trunk/pypy/jit/codewriter/jtransform.py Mon Oct 11 15:19:40 2010 @@ -316,6 +316,8 @@ prepare = self._handle_list_call elif oopspec_name.startswith('stroruni.'): prepare = self._handle_stroruni_call + elif oopspec_name == 'str.str2unicode': + prepare = self._handle_str2unicode_call elif oopspec_name.startswith('virtual_ref'): prepare = self._handle_virtual_ref_call else: @@ -1106,11 +1108,16 @@ [SoU, SoU], lltype.Signed), ]: + if args[0].concretetype.TO == rstr.UNICODE: + otherindex += EffectInfo._OS_offset_uni self._register_extra_helper(otherindex, othername, argtypes, resulttype) # return self._handle_oopspec_call(op, args, dict[oopspec_name]) + def _handle_str2unicode_call(self, op, oopspec_name, args): + return self._handle_oopspec_call(op, args, EffectInfo.OS_STR2UNICODE) + # ---------- # VirtualRefs. Modified: pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py ============================================================================== --- pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py (original) +++ pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py Mon Oct 11 15:19:40 2010 @@ -77,7 +77,8 @@ class FakeBuiltinCallControl: def guess_call_kind(self, op): return 'builtin' - def getcalldescr(self, op, oopspecindex): + def getcalldescr(self, op, oopspecindex=None): + assert oopspecindex is not None # in this test return 'calldescr-%d' % oopspecindex def calldescr_canraise(self, calldescr): return False @@ -766,6 +767,24 @@ assert op1.args[3] == ListOfKind('ref', [v1]) assert op1.result == v4 +def test_str2unicode(): + # test that the oopspec is present and correctly transformed + PSTR = lltype.Ptr(rstr.STR) + PUNICODE = lltype.Ptr(rstr.UNICODE) + FUNC = lltype.FuncType([PSTR], PUNICODE) + func = lltype.functionptr(FUNC, 'll_str2unicode', + _callable=rstr.LLHelpers.ll_str2unicode) + v1 = varoftype(PSTR) + v2 = varoftype(PUNICODE) + op = SpaceOperation('direct_call', [const(func), v1], v2) + tr = Transformer(FakeCPU(), FakeBuiltinCallControl()) + op1 = tr.rewrite_operation(op) + assert op1.opname == 'residual_call_r_r' + assert op1.args[0].value == func + assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_STR2UNICODE + assert op1.args[2] == ListOfKind('ref', [v1]) + assert op1.result == v2 + def test_list_ll_arraycopy(): from pypy.rlib.rgc import ll_arraycopy LIST = lltype.GcArray(lltype.Signed) Modified: pypy/trunk/pypy/jit/metainterp/history.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/history.py (original) +++ pypy/trunk/pypy/jit/metainterp/history.py Mon Oct 11 15:19:40 2010 @@ -698,6 +698,21 @@ return result _const_ptr_for_string = {} +def get_const_ptr_for_unicode(s): + from pypy.rpython.annlowlevel import llunicode + if not we_are_translated(): + try: + return _const_ptr_for_unicode[s] + except KeyError: + pass + if isinstance(s, str): + s = unicode(s) + result = ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, llunicode(s))) + if not we_are_translated(): + _const_ptr_for_unicode[s] = result + return result +_const_ptr_for_unicode = {} + # ____________________________________________________________ # The TreeLoop class contains a loop or a generalized loop, i.e. a tree Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/intbounds.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/optimizeopt/intbounds.py (original) +++ pypy/trunk/pypy/jit/metainterp/optimizeopt/intbounds.py Mon Oct 11 15:19:40 2010 @@ -191,6 +191,7 @@ v1.intbound.make_ge(IntLowerBound(0)) optimize_STRLEN = optimize_ARRAYLEN_GC + optimize_UNICODELEN = optimize_ARRAYLEN_GC def make_int_lt(self, box1, box2): v1 = self.getvalue(box1) Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py (original) +++ pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py Mon Oct 11 15:19:40 2010 @@ -3,6 +3,7 @@ from pypy.jit.metainterp.history import ConstInt from pypy.jit.metainterp.optimizeutil import _findall from pypy.jit.metainterp.resoperation import rop, ResOperation +from pypy.jit.codewriter.effectinfo import EffectInfo class OptRewrite(Optimization): """Rewrite operations into equivalent, cheaper operations. @@ -326,8 +327,37 @@ ## return ## self.emit_operation(op) -optimize_ops = _findall(OptRewrite, 'optimize_') - + def optimize_CALL(self, op): + # dispatch based on 'oopspecindex' to a method that handles + # specifically the given oopspec call. For non-oopspec calls, + # oopspecindex is just zero. + effectinfo = op.getdescr().get_extra_info() + if effectinfo is not None: + oopspecindex = effectinfo.oopspecindex + if oopspecindex == EffectInfo.OS_ARRAYCOPY: + if self._optimize_CALL_ARRAYCOPY(op): + return + self.emit_operation(op) + def _optimize_CALL_ARRAYCOPY(self, op): + source_value = self.getvalue(op.getarg(1)) + dest_value = self.getvalue(op.getarg(2)) + source_start_box = self.get_constant_box(op.getarg(3)) + dest_start_box = self.get_constant_box(op.getarg(4)) + length = self.get_constant_box(op.getarg(5)) + if (source_value.is_virtual() and source_start_box and dest_start_box + and length and dest_value.is_virtual()): + # XXX optimize the case where dest value is not virtual, + # but we still can avoid a mess + source_start = source_start_box.getint() + dest_start = dest_start_box.getint() + for index in range(length.getint()): + val = source_value.getitem(index + source_start) + dest_value.setitem(index + dest_start, val) + return True + if length and length.getint() == 0: + return True # 0-length arraycopy + return False + +optimize_ops = _findall(OptRewrite, 'optimize_') - Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/string.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/optimizeopt/string.py (original) +++ pypy/trunk/pypy/jit/metainterp/optimizeopt/string.py Mon Oct 11 15:19:40 2010 @@ -3,6 +3,7 @@ from pypy.jit.metainterp.history import Box, BoxInt, BoxPtr from pypy.jit.metainterp.history import Const, ConstInt, ConstPtr from pypy.jit.metainterp.history import get_const_ptr_for_string +from pypy.jit.metainterp.history import get_const_ptr_for_unicode from pypy.jit.metainterp.resoperation import rop, ResOperation from pypy.jit.metainterp.optimizeopt import optimizer, virtualize from pypy.jit.metainterp.optimizeopt.optimizer import CONST_0, CONST_1 @@ -11,56 +12,103 @@ from pypy.jit.codewriter.effectinfo import EffectInfo, callinfo_for_oopspec from pypy.jit.codewriter import heaptracker from pypy.rlib.unroll import unrolling_iterable +from pypy.rlib.objectmodel import specialize + + +class StrOrUnicode(object): + def __init__(self, LLTYPE, hlstr, emptystr, chr, + NEWSTR, STRLEN, STRGETITEM, STRSETITEM, COPYSTRCONTENT, + OS_offset): + self.LLTYPE = LLTYPE + self.hlstr = hlstr + self.emptystr = emptystr + self.chr = chr + self.NEWSTR = NEWSTR + self.STRLEN = STRLEN + self.STRGETITEM = STRGETITEM + self.STRSETITEM = STRSETITEM + self.COPYSTRCONTENT = COPYSTRCONTENT + self.OS_offset = OS_offset + + def _freeze_(self): + return True + +mode_string = StrOrUnicode(rstr.STR, annlowlevel.hlstr, '', chr, + rop.NEWSTR, rop.STRLEN, rop.STRGETITEM, + rop.STRSETITEM, rop.COPYSTRCONTENT, 0) +mode_unicode = StrOrUnicode(rstr.UNICODE, annlowlevel.hlunicode, u'', unichr, + rop.NEWUNICODE, rop.UNICODELEN, rop.UNICODEGETITEM, + rop.UNICODESETITEM, rop.COPYUNICODECONTENT, + EffectInfo._OS_offset_uni) + +# ____________________________________________________________ class __extend__(optimizer.OptValue): """New methods added to the base class OptValue for this file.""" - def getstrlen(self, newoperations): - s = self.get_constant_string() - if s is not None: - return ConstInt(len(s)) + def getstrlen(self, newoperations, mode): + if mode is mode_string: + s = self.get_constant_string_spec(mode_string) + if s is not None: + return ConstInt(len(s)) else: - if newoperations is None: - return None - self.ensure_nonnull() - box = self.force_box() - lengthbox = BoxInt() - newoperations.append(ResOperation(rop.STRLEN, [box], lengthbox)) - return lengthbox + s = self.get_constant_string_spec(mode_unicode) + if s is not None: + return ConstInt(len(s)) + if newoperations is None: + return None + self.ensure_nonnull() + box = self.force_box() + lengthbox = BoxInt() + newoperations.append(ResOperation(mode.STRLEN, [box], lengthbox)) + return lengthbox - def get_constant_string(self): + @specialize.arg(1) + def get_constant_string_spec(self, mode): if self.is_constant(): - s = self.box.getref(lltype.Ptr(rstr.STR)) - return annlowlevel.hlstr(s) + s = self.box.getref(lltype.Ptr(mode.LLTYPE)) + return mode.hlstr(s) else: return None - def string_copy_parts(self, newoperations, targetbox, offsetbox): + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): # Copies the pointer-to-string 'self' into the target string # given by 'targetbox', at the specified offset. Returns the offset # at the end of the copy. - lengthbox = self.getstrlen(newoperations) + lengthbox = self.getstrlen(newoperations, mode) srcbox = self.force_box() return copy_str_content(newoperations, srcbox, targetbox, - CONST_0, offsetbox, lengthbox) + CONST_0, offsetbox, lengthbox, mode) class VAbstractStringValue(virtualize.AbstractVirtualValue): - _attrs_ = () + _attrs_ = ('mode',) + + def __init__(self, optimizer, keybox, source_op, mode): + virtualize.AbstractVirtualValue.__init__(self, optimizer, keybox, + source_op) + self.mode = mode def _really_force(self): - s = self.get_constant_string() - if s is not None: - c_s = get_const_ptr_for_string(s) - self.make_constant(c_s) - return + if self.mode is mode_string: + s = self.get_constant_string_spec(mode_string) + if s is not None: + c_s = get_const_ptr_for_string(s) + self.make_constant(c_s) + return + else: + s = self.get_constant_string_spec(mode_unicode) + if s is not None: + c_s = get_const_ptr_for_unicode(s) + self.make_constant(c_s) + return assert self.source_op is not None self.box = box = self.source_op.result newoperations = self.optimizer.newoperations - lengthbox = self.getstrlen(newoperations) - newoperations.append(ResOperation(rop.NEWSTR, [lengthbox], box)) - self.string_copy_parts(newoperations, box, CONST_0) + lengthbox = self.getstrlen(newoperations, self.mode) + newoperations.append(ResOperation(self.mode.NEWSTR, [lengthbox], box)) + self.string_copy_parts(newoperations, box, CONST_0, self.mode) class VStringPlainValue(VAbstractStringValue): @@ -74,7 +122,7 @@ assert 0 <= start <= stop <= len(longerlist) self._chars = longerlist[start:stop] - def getstrlen(self, _): + def getstrlen(self, _, mode): if self._lengthbox is None: self._lengthbox = ConstInt(len(self._chars)) return self._lengthbox @@ -86,18 +134,21 @@ assert isinstance(charvalue, optimizer.OptValue) self._chars[index] = charvalue - def get_constant_string(self): + @specialize.arg(1) + def get_constant_string_spec(self, mode): for c in self._chars: if c is optimizer.CVAL_UNINITIALIZED_ZERO or not c.is_constant(): return None - return ''.join([chr(c.box.getint()) for c in self._chars]) + return mode.emptystr.join([mode.chr(c.box.getint()) + for c in self._chars]) - def string_copy_parts(self, newoperations, targetbox, offsetbox): + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): for i in range(len(self._chars)): charbox = self._chars[i].force_box() - newoperations.append(ResOperation(rop.STRSETITEM, [targetbox, - offsetbox, - charbox], None)) + newoperations.append(ResOperation(mode.STRSETITEM, [targetbox, + offsetbox, + charbox], + None)) offsetbox = _int_add(newoperations, offsetbox, CONST_1) return offsetbox @@ -109,7 +160,7 @@ value.get_args_for_fail(modifier) def _make_virtual(self, modifier): - return modifier.make_vstrplain() + return modifier.make_vstrplain(self.mode is mode_unicode) class VStringConcatValue(VAbstractStringValue): @@ -120,23 +171,24 @@ self.right = right self.lengthbox = lengthbox - def getstrlen(self, _): + def getstrlen(self, _, mode): return self.lengthbox - def get_constant_string(self): - s1 = self.left.get_constant_string() + @specialize.arg(1) + def get_constant_string_spec(self, mode): + s1 = self.left.get_constant_string_spec(mode) if s1 is None: return None - s2 = self.right.get_constant_string() + s2 = self.right.get_constant_string_spec(mode) if s2 is None: return None return s1 + s2 - def string_copy_parts(self, newoperations, targetbox, offsetbox): + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): offsetbox = self.left.string_copy_parts(newoperations, targetbox, - offsetbox) + offsetbox, mode) offsetbox = self.right.string_copy_parts(newoperations, targetbox, - offsetbox) + offsetbox, mode) return offsetbox def get_args_for_fail(self, modifier): @@ -150,7 +202,7 @@ self.right.get_args_for_fail(modifier) def _make_virtual(self, modifier): - return modifier.make_vstrconcat() + return modifier.make_vstrconcat(self.mode is mode_unicode) class VStringSliceValue(VAbstractStringValue): @@ -162,12 +214,13 @@ self.vstart = vstart self.vlength = vlength - def getstrlen(self, _): + def getstrlen(self, _, mode): return self.vlength.force_box() - def get_constant_string(self): + @specialize.arg(1) + def get_constant_string_spec(self, mode): if self.vstart.is_constant() and self.vlength.is_constant(): - s1 = self.vstr.get_constant_string() + s1 = self.vstr.get_constant_string_spec(mode) if s1 is None: return None start = self.vstart.box.getint() @@ -177,12 +230,12 @@ return s1[start : start + length] return None - def string_copy_parts(self, newoperations, targetbox, offsetbox): - lengthbox = self.getstrlen(newoperations) + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): + lengthbox = self.getstrlen(newoperations, mode) return copy_str_content(newoperations, self.vstr.force_box(), targetbox, self.vstart.force_box(), offsetbox, - lengthbox) + lengthbox, mode) def get_args_for_fail(self, modifier): if self.box is None and not modifier.already_seen_virtual(self.keybox): @@ -195,11 +248,11 @@ self.vlength.get_args_for_fail(modifier) def _make_virtual(self, modifier): - return modifier.make_vstrslice() + return modifier.make_vstrslice(self.mode is mode_unicode) def copy_str_content(newoperations, srcbox, targetbox, - srcoffsetbox, offsetbox, lengthbox): + srcoffsetbox, offsetbox, lengthbox, mode): if isinstance(srcbox, ConstPtr) and isinstance(srcoffsetbox, Const): M = 5 else: @@ -208,17 +261,18 @@ # up to M characters are done "inline", i.e. with STRGETITEM/STRSETITEM # instead of just a COPYSTRCONTENT. for i in range(lengthbox.value): - charbox = _strgetitem(newoperations, srcbox, srcoffsetbox) + charbox = _strgetitem(newoperations, srcbox, srcoffsetbox, mode) srcoffsetbox = _int_add(newoperations, srcoffsetbox, CONST_1) - newoperations.append(ResOperation(rop.STRSETITEM, [targetbox, - offsetbox, - charbox], None)) + newoperations.append(ResOperation(mode.STRSETITEM, [targetbox, + offsetbox, + charbox], + None)) offsetbox = _int_add(newoperations, offsetbox, CONST_1) else: nextoffsetbox = _int_add(newoperations, offsetbox, lengthbox) - op = ResOperation(rop.COPYSTRCONTENT, [srcbox, targetbox, - srcoffsetbox, offsetbox, - lengthbox], None) + op = ResOperation(mode.COPYSTRCONTENT, [srcbox, targetbox, + srcoffsetbox, offsetbox, + lengthbox], None) newoperations.append(op) offsetbox = nextoffsetbox return offsetbox @@ -245,12 +299,16 @@ newoperations.append(ResOperation(rop.INT_SUB, [box1, box2], resbox)) return resbox -def _strgetitem(newoperations, strbox, indexbox): +def _strgetitem(newoperations, strbox, indexbox, mode): if isinstance(strbox, ConstPtr) and isinstance(indexbox, ConstInt): - s = strbox.getref(lltype.Ptr(rstr.STR)) - return ConstInt(ord(s.chars[indexbox.getint()])) + if mode is mode_string: + s = strbox.getref(lltype.Ptr(rstr.STR)) + return ConstInt(ord(s.chars[indexbox.getint()])) + else: + s = strbox.getref(lltype.Ptr(rstr.UNICODE)) + return ConstInt(ord(s.chars[indexbox.getint()])) resbox = BoxInt() - newoperations.append(ResOperation(rop.STRGETITEM, [strbox, indexbox], + newoperations.append(ResOperation(mode.STRGETITEM, [strbox, indexbox], resbox)) return resbox @@ -258,62 +316,34 @@ class OptString(optimizer.Optimization): "Handling of strings and unicodes." - def make_vstring_plain(self, box, source_op=None): - vvalue = VStringPlainValue(self.optimizer, box, source_op) + def make_vstring_plain(self, box, source_op, mode): + vvalue = VStringPlainValue(self.optimizer, box, source_op, mode) self.make_equal_to(box, vvalue) return vvalue - def make_vstring_concat(self, box, source_op=None): - vvalue = VStringConcatValue(self.optimizer, box, source_op) + def make_vstring_concat(self, box, source_op, mode): + vvalue = VStringConcatValue(self.optimizer, box, source_op, mode) self.make_equal_to(box, vvalue) return vvalue - def make_vstring_slice(self, box, source_op=None): - vvalue = VStringSliceValue(self.optimizer, box, source_op) + def make_vstring_slice(self, box, source_op, mode): + vvalue = VStringSliceValue(self.optimizer, box, source_op, mode) self.make_equal_to(box, vvalue) return vvalue - def optimize_CALL(self, op): - # dispatch based on 'oopspecindex' to a method that handles - # specifically the given oopspec call. For non-oopspec calls, - # oopspecindex is just zero. - effectinfo = op.getdescr().get_extra_info() - if effectinfo is not None: - oopspecindex = effectinfo.oopspecindex - for value, meth in opt_call_oopspec_ops: - if oopspecindex == value: - if meth(self, op): - return - self.emit_operation(op) - - def opt_call_oopspec_ARRAYCOPY(self, op): - source_value = self.getvalue(op.getarg(1)) - dest_value = self.getvalue(op.getarg(2)) - source_start_box = self.get_constant_box(op.getarg(3)) - dest_start_box = self.get_constant_box(op.getarg(4)) - length = self.get_constant_box(op.getarg(5)) - if (source_value.is_virtual() and source_start_box and dest_start_box - and length and dest_value.is_virtual()): - # XXX optimize the case where dest value is not virtual, - # but we still can avoid a mess - source_start = source_start_box.getint() - dest_start = dest_start_box.getint() - for index in range(length.getint()): - val = source_value.getitem(index + source_start) - dest_value.setitem(index + dest_start, val) - return True - if length and length.getint() == 0: - return True # 0-length arraycopy - return False - def optimize_NEWSTR(self, op): + self._optimize_NEWSTR(op, mode_string) + def optimize_NEWUNICODE(self, op): + self._optimize_NEWSTR(op, mode_unicode) + + def _optimize_NEWSTR(self, op, mode): length_box = self.get_constant_box(op.getarg(0)) if length_box: # if the original 'op' did not have a ConstInt as argument, # build a new one with the ConstInt argument if not isinstance(op.getarg(0), ConstInt): - op = ResOperation(rop.NEWSTR, [length_box], op.result) - vvalue = self.make_vstring_plain(op.result, op) + op = ResOperation(mode.NEWSTR, [length_box], op.result) + vvalue = self.make_vstring_plain(op.result, op, mode) vvalue.setup(length_box.getint()) else: self.getvalue(op.result).ensure_nonnull() @@ -329,13 +359,20 @@ value.ensure_nonnull() self.emit_operation(op) + optimize_UNICODESETITEM = optimize_STRSETITEM + def optimize_STRGETITEM(self, op): + self._optimize_STRGETITEM(op, mode_string) + def optimize_UNICODEGETITEM(self, op): + self._optimize_STRGETITEM(op, mode_unicode) + + def _optimize_STRGETITEM(self, op, mode): value = self.getvalue(op.getarg(0)) vindex = self.getvalue(op.getarg(1)) - vresult = self.strgetitem(value, vindex) + vresult = self.strgetitem(value, vindex, mode) self.make_equal_to(op.result, vresult) - def strgetitem(self, value, vindex): + def strgetitem(self, value, vindex, mode): value.ensure_nonnull() # if value.is_virtual() and isinstance(value, VStringSliceValue): @@ -350,28 +387,71 @@ return value.getitem(vindex.box.getint()) # resbox = _strgetitem(self.optimizer.newoperations, - value.force_box(),vindex.force_box()) + value.force_box(),vindex.force_box(), mode) return self.getvalue(resbox) def optimize_STRLEN(self, op): + self._optimize_STRLEN(op, mode_string) + def optimize_UNICODELEN(self, op): + self._optimize_STRLEN(op, mode_unicode) + + def _optimize_STRLEN(self, op, mode): value = self.getvalue(op.getarg(0)) - lengthbox = value.getstrlen(self.optimizer.newoperations) + lengthbox = value.getstrlen(self.optimizer.newoperations, mode) self.make_equal_to(op.result, self.getvalue(lengthbox)) - def opt_call_oopspec_STR_CONCAT(self, op): + def optimize_CALL(self, op): + # dispatch based on 'oopspecindex' to a method that handles + # specifically the given oopspec call. For non-oopspec calls, + # oopspecindex is just zero. + effectinfo = op.getdescr().get_extra_info() + if effectinfo is not None: + oopspecindex = effectinfo.oopspecindex + for value, meth in opt_call_oopspec_ops: + if oopspecindex == value: # a match with the OS_STR_xxx + if meth(self, op, mode_string): + return + break + if oopspecindex == value + EffectInfo._OS_offset_uni: + # a match with the OS_UNI_xxx + if meth(self, op, mode_unicode): + return + break + if oopspecindex == EffectInfo.OS_STR2UNICODE: + if self.opt_call_str_STR2UNICODE(op): + return + self.emit_operation(op) + + def opt_call_str_STR2UNICODE(self, op): + # Constant-fold unicode("constant string"). + # More generally, supporting non-constant but virtual cases is + # not obvious, because of the exception UnicodeDecodeError that + # can be raised by ll_str2unicode() + varg = self.getvalue(op.getarg(1)) + s = varg.get_constant_string_spec(mode_string) + if s is None: + return False + try: + u = unicode(s) + except UnicodeDecodeError: + return False + self.make_constant(op.result, get_const_ptr_for_unicode(u)) + return True + + def opt_call_stroruni_STR_CONCAT(self, op, mode): vleft = self.getvalue(op.getarg(1)) vright = self.getvalue(op.getarg(2)) vleft.ensure_nonnull() vright.ensure_nonnull() newoperations = self.optimizer.newoperations - len1box = vleft.getstrlen(newoperations) - len2box = vright.getstrlen(newoperations) + len1box = vleft.getstrlen(newoperations, mode) + len2box = vright.getstrlen(newoperations, mode) lengthbox = _int_add(newoperations, len1box, len2box) - value = self.make_vstring_concat(op.result, op) + value = self.make_vstring_concat(op.result, op, mode) value.setup(vleft, vright, lengthbox) return True - def opt_call_oopspec_STR_SLICE(self, op): + def opt_call_stroruni_STR_SLICE(self, op, mode): newoperations = self.optimizer.newoperations vstr = self.getvalue(op.getarg(1)) vstart = self.getvalue(op.getarg(2)) @@ -380,7 +460,7 @@ if (isinstance(vstr, VStringPlainValue) and vstart.is_constant() and vstop.is_constant()): # slicing with constant bounds of a VStringPlainValue - value = self.make_vstring_plain(op.result, op) + value = self.make_vstring_plain(op.result, op, mode) value.setup_slice(vstr._chars, vstart.box.getint(), vstop.box.getint()) return True @@ -398,16 +478,16 @@ vstart.force_box()) vstart = self.getvalue(startbox) # - value = self.make_vstring_slice(op.result, op) + value = self.make_vstring_slice(op.result, op, mode) value.setup(vstr, vstart, self.getvalue(lengthbox)) return True - def opt_call_oopspec_STR_EQUAL(self, op): + def opt_call_stroruni_STR_EQUAL(self, op, mode): v1 = self.getvalue(op.getarg(1)) v2 = self.getvalue(op.getarg(2)) # - l1box = v1.getstrlen(None) - l2box = v2.getstrlen(None) + l1box = v1.getstrlen(None, mode) + l2box = v2.getstrlen(None, mode) if (l1box is not None and l2box is not None and isinstance(l1box, ConstInt) and isinstance(l2box, ConstInt) and @@ -416,13 +496,13 @@ self.make_constant(op.result, CONST_0) return True # - if self.handle_str_equal_level1(v1, v2, op.result): + if self.handle_str_equal_level1(v1, v2, op.result, mode): return True - if self.handle_str_equal_level1(v2, v1, op.result): + if self.handle_str_equal_level1(v2, v1, op.result, mode): return True - if self.handle_str_equal_level2(v1, v2, op.result): + if self.handle_str_equal_level2(v1, v2, op.result, mode): return True - if self.handle_str_equal_level2(v2, v1, op.result): + if self.handle_str_equal_level2(v2, v1, op.result, mode): return True # if v1.is_nonnull() and v2.is_nonnull(): @@ -434,37 +514,37 @@ else: do = EffectInfo.OS_STREQ_NONNULL self.generate_modified_call(do, [v1.force_box(), - v2.force_box()], op.result) + v2.force_box()], op.result, mode) return True return False - def handle_str_equal_level1(self, v1, v2, resultbox): - l2box = v2.getstrlen(None) + def handle_str_equal_level1(self, v1, v2, resultbox, mode): + l2box = v2.getstrlen(None, mode) if isinstance(l2box, ConstInt): if l2box.value == 0: - lengthbox = v1.getstrlen(self.optimizer.newoperations) + lengthbox = v1.getstrlen(self.optimizer.newoperations, mode) seo = self.optimizer.send_extra_operation seo(ResOperation(rop.INT_EQ, [lengthbox, CONST_0], resultbox)) return True if l2box.value == 1: - l1box = v1.getstrlen(None) + l1box = v1.getstrlen(None, mode) if isinstance(l1box, ConstInt) and l1box.value == 1: # comparing two single chars - vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO) - vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO) + vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO, mode) + vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO, mode) seo = self.optimizer.send_extra_operation seo(ResOperation(rop.INT_EQ, [vchar1.force_box(), vchar2.force_box()], resultbox)) return True if isinstance(v1, VStringSliceValue): - vchar = self.strgetitem(v2, optimizer.CVAL_ZERO) + vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode) do = EffectInfo.OS_STREQ_SLICE_CHAR self.generate_modified_call(do, [v1.vstr.force_box(), v1.vstart.force_box(), v1.vlength.force_box(), vchar.force_box()], - resultbox) + resultbox, mode) return True # if v2.is_null(): @@ -482,17 +562,18 @@ # return False - def handle_str_equal_level2(self, v1, v2, resultbox): - l2box = v2.getstrlen(None) + def handle_str_equal_level2(self, v1, v2, resultbox, mode): + l2box = v2.getstrlen(None, mode) if isinstance(l2box, ConstInt): if l2box.value == 1: - vchar = self.strgetitem(v2, optimizer.CVAL_ZERO) + vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode) if v1.is_nonnull(): do = EffectInfo.OS_STREQ_NONNULL_CHAR else: do = EffectInfo.OS_STREQ_CHECKNULL_CHAR self.generate_modified_call(do, [v1.force_box(), - vchar.force_box()], resultbox) + vchar.force_box()], resultbox, + mode) return True # if v1.is_virtual() and isinstance(v1, VStringSliceValue): @@ -503,11 +584,12 @@ self.generate_modified_call(do, [v1.vstr.force_box(), v1.vstart.force_box(), v1.vlength.force_box(), - v2.force_box()], resultbox) + v2.force_box()], resultbox, mode) return True return False - def generate_modified_call(self, oopspecindex, args, result): + def generate_modified_call(self, oopspecindex, args, result, mode): + oopspecindex += mode.OS_offset calldescr, func = callinfo_for_oopspec(oopspecindex) op = ResOperation(rop.CALL, [ConstInt(func)] + args, result, descr=calldescr) @@ -525,7 +607,7 @@ optimize_ops = _findall(OptString, 'optimize_') def _findall_call_oopspec(): - prefix = 'opt_call_oopspec_' + prefix = 'opt_call_stroruni_' result = [] for name in dir(OptString): if name.startswith(prefix): Modified: pypy/trunk/pypy/jit/metainterp/resume.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/resume.py (original) +++ pypy/trunk/pypy/jit/metainterp/resume.py Mon Oct 11 15:19:40 2010 @@ -255,13 +255,19 @@ def make_varray(self, arraydescr): return VArrayInfo(arraydescr) - def make_vstrplain(self): + def make_vstrplain(self, is_unicode=False): + if is_unicode: + return VUniPlainInfo() return VStrPlainInfo() - def make_vstrconcat(self): + def make_vstrconcat(self, is_unicode=False): + if is_unicode: + return VUniConcatInfo() return VStrConcatInfo() - def make_vstrslice(self): + def make_vstrslice(self, is_unicode=False): + if is_unicode: + return VUniSliceInfo() return VStrSliceInfo() def register_virtual_fields(self, virtualbox, fieldboxes): @@ -550,6 +556,60 @@ for i in self.fieldnums: debug_print("\t\t", str(untag(i))) + +class VUniPlainInfo(AbstractVirtualInfo): + """Stands for the unicode string made out of the characters of all + fieldnums.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + length = len(self.fieldnums) + string = decoder.allocate_unicode(length) + decoder.virtuals_cache[index] = string + for i in range(length): + decoder.unicode_setitem(string, i, self.fieldnums[i]) + return string + + def debug_prints(self): + debug_print("\tvuniplaininfo length", len(self.fieldnums)) + + +class VUniConcatInfo(AbstractVirtualInfo): + """Stands for the unicode string made out of the concatenation of two + other unicode strings.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + # xxx for blackhole resuming, this will build all intermediate + # strings and throw them away immediately, which is a bit sub- + # efficient. Not sure we care. + left, right = self.fieldnums + string = decoder.concat_unicodes(left, right) + decoder.virtuals_cache[index] = string + return string + + def debug_prints(self): + debug_print("\tvuniconcatinfo") + for i in self.fieldnums: + debug_print("\t\t", str(untag(i))) + + +class VUniSliceInfo(AbstractVirtualInfo): + """Stands for the unicode string made out of slicing another + unicode string.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + largerstr, start, length = self.fieldnums + string = decoder.slice_unicode(largerstr, start, length) + decoder.virtuals_cache[index] = string + return string + + def debug_prints(self): + debug_print("\tvunisliceinfo") + for i in self.fieldnums: + debug_print("\t\t", str(untag(i))) + # ____________________________________________________________ class AbstractResumeDataReader(object): @@ -725,6 +785,32 @@ return self.metainterp.execute_and_record_varargs( rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr) + def allocate_unicode(self, length): + return self.metainterp.execute_and_record(rop.NEWUNICODE, + None, ConstInt(length)) + + def unicode_setitem(self, strbox, index, charnum): + charbox = self.decode_box(charnum, INT) + self.metainterp.execute_and_record(rop.UNICODESETITEM, None, + strbox, ConstInt(index), charbox) + + def concat_unicodes(self, str1num, str2num): + calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_CONCAT) + str1box = self.decode_box(str1num, REF) + str2box = self.decode_box(str2num, REF) + return self.metainterp.execute_and_record_varargs( + rop.CALL, [ConstInt(func), str1box, str2box], calldescr) + + def slice_unicode(self, strnum, startnum, lengthnum): + calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_SLICE) + strbox = self.decode_box(strnum, REF) + startbox = self.decode_box(startnum, INT) + lengthbox = self.decode_box(lengthnum, INT) + stopbox = self.metainterp.execute_and_record(rop.INT_ADD, None, + startbox, lengthbox) + return self.metainterp.execute_and_record_varargs( + rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr) + def setfield(self, descr, structbox, fieldnum): if descr.is_pointer_field(): kind = REF @@ -967,6 +1053,31 @@ result = funcptr(str, start, start + length) return lltype.cast_opaque_ptr(llmemory.GCREF, result) + def allocate_unicode(self, length): + return self.cpu.bh_newunicode(length) + + def unicode_setitem(self, str, index, charnum): + char = self.decode_int(charnum) + self.cpu.bh_unicodesetitem(str, index, char) + + def concat_unicodes(self, str1num, str2num): + str1 = self.decode_ref(str1num) + str2 = self.decode_ref(str2num) + str1 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str1) + str2 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str2) + funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_CONCAT) + result = funcptr(str1, str2) + return lltype.cast_opaque_ptr(llmemory.GCREF, result) + + def slice_unicode(self, strnum, startnum, lengthnum): + str = self.decode_ref(strnum) + start = self.decode_int(startnum) + length = self.decode_int(lengthnum) + str = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str) + funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_SLICE) + result = funcptr(str, start, start + length) + return lltype.cast_opaque_ptr(llmemory.GCREF, result) + def setfield(self, descr, struct, fieldnum): if descr.is_pointer_field(): newvalue = self.decode_ref(fieldnum) Modified: pypy/trunk/pypy/jit/metainterp/test/oparser.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/test/oparser.py (original) +++ pypy/trunk/pypy/jit/metainterp/test/oparser.py Mon Oct 11 15:19:40 2010 @@ -5,7 +5,7 @@ from pypy.jit.metainterp.history import TreeLoop, BoxInt, ConstInt,\ ConstObj, ConstPtr, Box, BasicFailDescr, BoxFloat, ConstFloat,\ - LoopToken, get_const_ptr_for_string + LoopToken, get_const_ptr_for_string, get_const_ptr_for_unicode from pypy.jit.metainterp.resoperation import rop, ResOperation, ResOpWithDescr, N_aryOp from pypy.jit.metainterp.typesystem import llhelper from pypy.jit.codewriter.heaptracker import adr2int @@ -158,10 +158,15 @@ except ValueError: if self.is_float(arg): return ConstFloat(float(arg)) - if arg.startswith('"') or arg.startswith("'"): + if (arg.startswith('"') or arg.startswith("'") or + arg.startswith('s"')): # XXX ootype - info = arg.strip("'\"") + info = arg[1:].strip("'\"") return get_const_ptr_for_string(info) + if arg.startswith('u"'): + # XXX ootype + info = arg[1:].strip("'\"") + return get_const_ptr_for_unicode(info) if arg.startswith('ConstClass('): name = arg[len('ConstClass('):-1] return self.get_const(name, 'class') Modified: pypy/trunk/pypy/jit/metainterp/test/test_optimizefindnode.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/test/test_optimizefindnode.py (original) +++ pypy/trunk/pypy/jit/metainterp/test/test_optimizefindnode.py Mon Oct 11 15:19:40 2010 @@ -117,33 +117,32 @@ EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE)) arraycopydescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, EffectInfo([], [], [], oopspecindex=EffectInfo.OS_ARRAYCOPY)) - strconcatdescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_CONCAT)) - slicedescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_SLICE)) - strequaldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_EQUAL)) - streq_slice_checknull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_SLICE_CHECKNULL)) - streq_slice_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_SLICE_NONNULL)) - streq_slice_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_SLICE_CHAR)) - streq_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_NONNULL)) - streq_nonnull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_NONNULL_CHAR)) - streq_checknull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_CHECKNULL_CHAR)) - streq_lengthok_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_LENGTHOK)) + + for _name, _os in [ + ('strconcatdescr', 'OS_STR_CONCAT'), + ('strslicedescr', 'OS_STR_SLICE'), + ('strequaldescr', 'OS_STR_EQUAL'), + ('streq_slice_checknull_descr', 'OS_STREQ_SLICE_CHECKNULL'), + ('streq_slice_nonnull_descr', 'OS_STREQ_SLICE_NONNULL'), + ('streq_slice_char_descr', 'OS_STREQ_SLICE_CHAR'), + ('streq_nonnull_descr', 'OS_STREQ_NONNULL'), + ('streq_nonnull_char_descr', 'OS_STREQ_NONNULL_CHAR'), + ('streq_checknull_char_descr', 'OS_STREQ_CHECKNULL_CHAR'), + ('streq_lengthok_descr', 'OS_STREQ_LENGTHOK'), + ]: + _oopspecindex = getattr(EffectInfo, _os) + locals()[_name] = \ + cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=_oopspecindex)) + # + _oopspecindex = getattr(EffectInfo, _os.replace('STR', 'UNI')) + locals()[_name.replace('str', 'unicode')] = \ + cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=_oopspecindex)) + + s2u_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR2UNICODE)) + # class LoopToken(AbstractDescr): pass Modified: pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py (original) +++ pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py Mon Oct 11 15:19:40 2010 @@ -3487,7 +3487,7 @@ i0 = strlen(p0) jump(p0) """ - self.optimize_loop(ops, 'Not', expected) + self.optimize_strunicode_loop(ops, 'Not', expected) def test_addsub_const(self): ops = """ @@ -3893,6 +3893,15 @@ """ self.optimize_loop(ops, 'Not, Not', expected) + # ---------- + def optimize_strunicode_loop(self, ops, spectext, optops): + # check with the arguments passed in + self.optimize_loop(ops, spectext, optops) + # check with replacing 'str' with 'unicode' everywhere + self.optimize_loop(ops.replace('str','unicode').replace('s"', 'u"'), + spectext, + optops.replace('str','unicode').replace('s"', 'u"')) + def test_newstr_1(self): ops = """ [i0] @@ -3905,7 +3914,7 @@ [i0] jump(i0) """ - self.optimize_loop(ops, 'Not', expected) + self.optimize_strunicode_loop(ops, 'Not', expected) def test_newstr_2(self): ops = """ @@ -3921,7 +3930,7 @@ [i0, i1] jump(i1, i0) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_concat_1(self): ops = """ @@ -3942,7 +3951,7 @@ copystrcontent(p2, p3, 0, i4, i5) jump(p2, p3) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_concat_vstr2_str(self): ops = """ @@ -3965,7 +3974,7 @@ copystrcontent(p2, p3, 0, 2, i4) jump(i1, i0, p3) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_concat_str_vstr2(self): ops = """ @@ -3989,7 +3998,7 @@ i6 = int_add(i5, 1) # will be killed by the backend jump(i1, i0, p3) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_concat_str_str_str(self): ops = """ @@ -4016,12 +4025,12 @@ copystrcontent(p3, p5, 0, i12b, i3b) jump(p2, p3, p5) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_concat_str_cstr1(self): ops = """ [p2] - p3 = call(0, p2, "x", descr=strconcatdescr) + p3 = call(0, p2, s"x", descr=strconcatdescr) jump(p3) """ expected = """ @@ -4035,28 +4044,28 @@ i5 = int_add(i4, 1) # will be killed by the backend jump(p3) """ - self.optimize_loop(ops, 'Not', expected) + self.optimize_strunicode_loop(ops, 'Not', expected) def test_str_concat_consts(self): ops = """ [] - p1 = same_as("ab") - p2 = same_as("cde") + p1 = same_as(s"ab") + p2 = same_as(s"cde") p3 = call(0, p1, p2, descr=strconcatdescr) escape(p3) jump() """ expected = """ [] - escape("abcde") + escape(s"abcde") jump() """ - self.optimize_loop(ops, '', expected) + self.optimize_strunicode_loop(ops, '', expected) def test_str_slice_1(self): ops = """ [p1, i1, i2] - p2 = call(0, p1, i1, i2, descr=slicedescr) + p2 = call(0, p1, i1, i2, descr=strslicedescr) jump(p2, i1, i2) """ expected = """ @@ -4066,12 +4075,12 @@ copystrcontent(p1, p2, i1, 0, i3) jump(p2, i1, i2) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_slice_2(self): ops = """ [p1, i2] - p2 = call(0, p1, 0, i2, descr=slicedescr) + p2 = call(0, p1, 0, i2, descr=strslicedescr) jump(p2, i2) """ expected = """ @@ -4080,13 +4089,13 @@ copystrcontent(p1, p2, 0, 0, i2) jump(p2, i2) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_slice_3(self): ops = """ [p1, i1, i2, i3, i4] - p2 = call(0, p1, i1, i2, descr=slicedescr) - p3 = call(0, p2, i3, i4, descr=slicedescr) + p2 = call(0, p1, i1, i2, descr=strslicedescr) + p3 = call(0, p2, i3, i4, descr=strslicedescr) jump(p3, i1, i2, i3, i4) """ expected = """ @@ -4098,12 +4107,12 @@ copystrcontent(p1, p3, i6, 0, i5) jump(p3, i1, i2, i3, i4) """ - self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not, Not', expected) def test_str_slice_getitem1(self): ops = """ [p1, i1, i2, i3] - p2 = call(0, p1, i1, i2, descr=slicedescr) + p2 = call(0, p1, i1, i2, descr=strslicedescr) i4 = strgetitem(p2, i3) escape(i4) jump(p1, i1, i2, i3) @@ -4116,7 +4125,7 @@ escape(i4) jump(p1, i1, i2, i3) """ - self.optimize_loop(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected) def test_str_slice_plain(self): ops = """ @@ -4124,7 +4133,7 @@ p1 = newstr(2) strsetitem(p1, 0, i3) strsetitem(p1, 1, i4) - p2 = call(0, p1, 1, 2, descr=slicedescr) + p2 = call(0, p1, 1, 2, descr=strslicedescr) i5 = strgetitem(p2, 0) escape(i5) jump(i3, i4) @@ -4134,12 +4143,12 @@ escape(i4) jump(i3, i4) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_slice_concat(self): ops = """ [p1, i1, i2, p2] - p3 = call(0, p1, i1, i2, descr=slicedescr) + p3 = call(0, p1, i1, i2, descr=strslicedescr) p4 = call(0, p3, p2, descr=strconcatdescr) jump(p4, i1, i2, p2) """ @@ -4155,10 +4164,10 @@ copystrcontent(p2, p4, 0, i3, i4b) jump(p4, i1, i2, p2) """ - self.optimize_loop(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected) # ---------- - def optimize_loop_extradescrs(self, ops, spectext, optops): + def optimize_strunicode_loop_extradescrs(self, ops, spectext, optops): from pypy.jit.metainterp.optimizeopt import string def my_callinfo_for_oopspec(oopspecindex): calldescrtype = type(LLtypeMixin.strequaldescr) @@ -4173,7 +4182,7 @@ saved = string.callinfo_for_oopspec try: string.callinfo_for_oopspec = my_callinfo_for_oopspec - self.optimize_loop(ops, spectext, optops) + self.optimize_strunicode_loop(ops, spectext, optops) finally: string.callinfo_for_oopspec = saved @@ -4184,7 +4193,7 @@ escape(i0) jump(p1, p2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not', ops) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', ops) def test_str_equal_noop2(self): ops = """ @@ -4209,12 +4218,13 @@ escape(i0) jump(p1, p2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not', + expected) def test_str_equal_slice1(self): ops = """ [p1, i1, i2, p3] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) i0 = call(0, p4, p3, descr=strequaldescr) escape(i0) jump(p1, i1, i2, p3) @@ -4226,12 +4236,13 @@ escape(i0) jump(p1, i1, i2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_slice2(self): ops = """ [p1, i1, i2, p3] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) i0 = call(0, p3, p4, descr=strequaldescr) escape(i0) jump(p1, i1, i2, p3) @@ -4243,13 +4254,14 @@ escape(i0) jump(p1, i1, i2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_slice3(self): ops = """ [p1, i1, i2, p3] guard_nonnull(p3) [] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) i0 = call(0, p3, p4, descr=strequaldescr) escape(i0) jump(p1, i1, i2, p3) @@ -4262,13 +4274,14 @@ escape(i0) jump(p1, i1, i2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_slice4(self): ops = """ [p1, i1, i2] - p3 = call(0, p1, i1, i2, descr=slicedescr) - i0 = call(0, p3, "x", descr=strequaldescr) + p3 = call(0, p1, i1, i2, descr=strslicedescr) + i0 = call(0, p3, s"x", descr=strequaldescr) escape(i0) jump(p1, i1, i2) """ @@ -4279,12 +4292,13 @@ escape(i0) jump(p1, i1, i2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not', + expected) def test_str_equal_slice5(self): ops = """ [p1, i1, i2, i3] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) p5 = newstr(1) strsetitem(p5, 0, i3) i0 = call(0, p5, p4, descr=strequaldescr) @@ -4298,7 +4312,8 @@ escape(i0) jump(p1, i1, i2, i3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_none1(self): ops = """ @@ -4313,7 +4328,7 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_none2(self): ops = """ @@ -4328,30 +4343,30 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull1(self): ops = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "hello world", descr=strequaldescr) + i0 = call(0, p1, s"hello world", descr=strequaldescr) escape(i0) jump(p1) """ expected = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "hello world", descr=streq_nonnull_descr) + i0 = call(0, p1, s"hello world", descr=streq_nonnull_descr) escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull2(self): ops = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "", descr=strequaldescr) + i0 = call(0, p1, s"", descr=strequaldescr) escape(i0) jump(p1) """ @@ -4363,13 +4378,13 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull3(self): ops = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "x", descr=strequaldescr) + i0 = call(0, p1, s"x", descr=strequaldescr) escape(i0) jump(p1) """ @@ -4380,13 +4395,13 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull4(self): ops = """ [p1, p2] p4 = call(0, p1, p2, descr=strconcatdescr) - i0 = call(0, "hello world", p4, descr=strequaldescr) + i0 = call(0, s"hello world", p4, descr=strequaldescr) escape(i0) jump(p1, p2) """ @@ -4401,17 +4416,17 @@ i5 = strlen(p2) i6 = int_add(i4, i5) # will be killed by the backend copystrcontent(p2, p4, 0, i4, i5) - i0 = call(0, "hello world", p4, descr=streq_nonnull_descr) + i0 = call(0, s"hello world", p4, descr=streq_nonnull_descr) escape(i0) jump(p1, p2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected) def test_str_equal_chars0(self): ops = """ [i1] p1 = newstr(0) - i0 = call(0, p1, "", descr=strequaldescr) + i0 = call(0, p1, s"", descr=strequaldescr) escape(i0) jump(i1) """ @@ -4420,14 +4435,14 @@ escape(1) jump(i1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_chars1(self): ops = """ [i1] p1 = newstr(1) strsetitem(p1, 0, i1) - i0 = call(0, p1, "x", descr=strequaldescr) + i0 = call(0, p1, s"x", descr=strequaldescr) escape(i0) jump(i1) """ @@ -4437,7 +4452,7 @@ escape(i0) jump(i1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_chars2(self): ops = """ @@ -4445,7 +4460,7 @@ p1 = newstr(2) strsetitem(p1, 0, i1) strsetitem(p1, 1, i2) - i0 = call(0, p1, "xy", descr=strequaldescr) + i0 = call(0, p1, s"xy", descr=strequaldescr) escape(i0) jump(i1, i2) """ @@ -4454,16 +4469,16 @@ p1 = newstr(2) strsetitem(p1, 0, i1) strsetitem(p1, 1, i2) - i0 = call(0, p1, "xy", descr=streq_lengthok_descr) + i0 = call(0, p1, s"xy", descr=streq_lengthok_descr) escape(i0) jump(i1, i2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected) def test_str_equal_chars3(self): ops = """ [p1] - i0 = call(0, "x", p1, descr=strequaldescr) + i0 = call(0, s"x", p1, descr=strequaldescr) escape(i0) jump(p1) """ @@ -4473,14 +4488,14 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_lengthmismatch1(self): ops = """ [i1] p1 = newstr(1) strsetitem(p1, 0, i1) - i0 = call(0, "xy", p1, descr=strequaldescr) + i0 = call(0, s"xy", p1, descr=strequaldescr) escape(i0) jump(i1) """ @@ -4489,10 +4504,33 @@ escape(0) jump(i1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) - # XXX unicode operations - # XXX str2unicode + def test_str2unicode_constant(self): + ops = """ + [] + p0 = call(0, "xy", descr=s2u_descr) # string -> unicode + escape(p0) + jump() + """ + expected = """ + [] + escape(u"xy") + jump() + """ + self.optimize_strunicode_loop_extradescrs(ops, '', expected) + + def test_str2unicode_nonconstant(self): + ops = """ + [p0] + p1 = call(0, p0, descr=s2u_descr) # string -> unicode + escape(p1) + jump(p1) + """ + self.optimize_strunicode_loop_extradescrs(ops, 'Not', ops) + # more generally, supporting non-constant but virtual cases is + # not obvious, because of the exception UnicodeDecodeError that + # can be raised by ll_str2unicode() ##class TestOOtype(BaseTestOptimizeOpt, OOtypeMixin): Modified: pypy/trunk/pypy/jit/metainterp/test/test_resume.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/test/test_resume.py (original) +++ pypy/trunk/pypy/jit/metainterp/test/test_resume.py Mon Oct 11 15:19:40 2010 @@ -240,6 +240,17 @@ return FakeBuiltObject(strconcat=[left, right]) def slice_string(self, str, start, length): return FakeBuiltObject(strslice=[str, start, length]) + def allocate_unicode(self, length): + return FakeBuiltObject(unistring=[None]*length) + def unicode_setitem(self, unistring, i, fieldnum): + value, tag = untag(fieldnum) + assert tag == TAGINT + assert 0 <= i < len(unistring.unistring) + unistring.unistring[i] = value + def concat_unicodes(self, left, right): + return FakeBuiltObject(uniconcat=[left, right]) + def slice_unicode(self, str, start, length): + return FakeBuiltObject(unislice=[str, start, length]) class FakeBuiltObject(object): def __init__(self, **kwds): @@ -304,6 +315,30 @@ assert reader.force_all_virtuals() == [ FakeBuiltObject(strslice=info.fieldnums)] +def test_vuniplaininfo(): + info = VUniPlainInfo() + info.fieldnums = [tag(60, TAGINT)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(unistring=[60])] + +def test_vuniconcatinfo(): + info = VUniConcatInfo() + info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(uniconcat=info.fieldnums)] + +def test_vunisliceinfo(): + info = VUniSliceInfo() + info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX), tag(30, TAGBOX)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(unislice=info.fieldnums)] + # ____________________________________________________________ Modified: pypy/trunk/pypy/jit/metainterp/test/test_string.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/test/test_string.py (original) +++ pypy/trunk/pypy/jit/metainterp/test/test_string.py Mon Oct 11 15:19:40 2010 @@ -6,14 +6,17 @@ class StringTests: + _str, _chr = str, chr + def test_eq_residual(self): + _str = self._str jitdriver = JitDriver(greens = [], reds = ['n', 'i', 's']) - global_s = "hello" + global_s = _str("hello") def f(n, b, s): if b: - s += "ello" + s += _str("ello") else: - s += "allo" + s += _str("allo") i = 0 while n > 0: jitdriver.can_enter_jit(s=s, n=n, i=i) @@ -21,18 +24,19 @@ n -= 1 + (s == global_s) i += 1 return i - res = self.meta_interp(f, [10, True, 'h'], listops=True) + res = self.meta_interp(f, [10, True, _str('h')], listops=True) assert res == 5 self.check_loops(**{self.CALL: 1, self.CALL_PURE: 0}) def test_eq_folded(self): + _str = self._str jitdriver = JitDriver(greens = ['s'], reds = ['n', 'i']) - global_s = "hello" + global_s = _str("hello") def f(n, b, s): if b: - s += "ello" + s += _str("ello") else: - s += "allo" + s += _str("allo") i = 0 while n > 0: jitdriver.can_enter_jit(s=s, n=n, i=i) @@ -40,31 +44,18 @@ n -= 1 + (s == global_s) i += 1 return i - res = self.meta_interp(f, [10, True, 'h'], listops=True) + res = self.meta_interp(f, [10, True, _str('h')], listops=True) assert res == 5 self.check_loops(**{self.CALL: 0, self.CALL_PURE: 0}) def test_newstr(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['n', 'm']) def f(n, m): while True: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - bytecode = 'adlfkj' + chr(n) - res = bytecode[n] - m -= 1 - if m < 0: - return ord(res) - res = self.meta_interp(f, [6, 10]) - assert res == 6 - - def test_newunicode(self): - jitdriver = JitDriver(greens = [], reds = ['n', 'm']) - def f(n, m): - while True: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - bytecode = u'adlfkj' + unichr(n) + bytecode = _str('adlfkj') + _chr(n) res = bytecode[n] m -= 1 if m < 0: @@ -73,95 +64,96 @@ assert res == 6 def test_char2string_pure(self): - for dochr in [chr, ]: #unichr]: - jitdriver = JitDriver(greens = [], reds = ['n']) - @dont_look_inside - def escape(x): - pass - def f(n): - while n > 0: - jitdriver.can_enter_jit(n=n) - jitdriver.jit_merge_point(n=n) - s = dochr(n) - if not we_are_jitted(): - s += s # forces to be a string - if n > 100: - escape(s) - n -= 1 - return 42 - self.meta_interp(f, [6]) - self.check_loops(newstr=0, strsetitem=0, strlen=0, - newunicode=0, unicodesetitem=0, unicodelen=0) + _str, _chr = self._str, self._chr + jitdriver = JitDriver(greens = [], reds = ['n']) + @dont_look_inside + def escape(x): + pass + def f(n): + while n > 0: + jitdriver.can_enter_jit(n=n) + jitdriver.jit_merge_point(n=n) + s = _chr(n) + if not we_are_jitted(): + s += s # forces to be a string + if n > 100: + escape(s) + n -= 1 + return 42 + self.meta_interp(f, [6]) + self.check_loops(newstr=0, strsetitem=0, strlen=0, + newunicode=0, unicodesetitem=0, unicodelen=0) def test_char2string_escape(self): - for dochr in [chr, ]: #unichr]: - jitdriver = JitDriver(greens = [], reds = ['n', 'total']) - @dont_look_inside - def escape(x): - return ord(x[0]) - def f(n): - total = 0 - while n > 0: - jitdriver.can_enter_jit(n=n, total=total) - jitdriver.jit_merge_point(n=n, total=total) - s = dochr(n) - if not we_are_jitted(): - s += s # forces to be a string - total += escape(s) - n -= 1 - return total - res = self.meta_interp(f, [6]) - assert res == 21 + _str, _chr = self._str, self._chr + jitdriver = JitDriver(greens = [], reds = ['n', 'total']) + @dont_look_inside + def escape(x): + return ord(x[0]) + def f(n): + total = 0 + while n > 0: + jitdriver.can_enter_jit(n=n, total=total) + jitdriver.jit_merge_point(n=n, total=total) + s = _chr(n) + if not we_are_jitted(): + s += s # forces to be a string + total += escape(s) + n -= 1 + return total + res = self.meta_interp(f, [6]) + assert res == 21 def test_char2string2char(self): - for dochr in [chr, ]: #unichr]: - jitdriver = JitDriver(greens = [], reds = ['m', 'total']) - def f(m): - total = 0 - while m > 0: - jitdriver.can_enter_jit(m=m, total=total) - jitdriver.jit_merge_point(m=m, total=total) - string = dochr(m) - if m > 100: - string += string # forces to be a string - # read back the character - c = string[0] - total += ord(c) - m -= 1 - return total - res = self.meta_interp(f, [6]) - assert res == 21 - self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0, - newunicode=0, unicodegetitem=0, unicodesetitem=0, - unicodelen=0) + _str, _chr = self._str, self._chr + jitdriver = JitDriver(greens = [], reds = ['m', 'total']) + def f(m): + total = 0 + while m > 0: + jitdriver.can_enter_jit(m=m, total=total) + jitdriver.jit_merge_point(m=m, total=total) + string = _chr(m) + if m > 100: + string += string # forces to be a string + # read back the character + c = string[0] + total += ord(c) + m -= 1 + return total + res = self.meta_interp(f, [6]) + assert res == 21 + self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0, + newunicode=0, unicodegetitem=0, unicodesetitem=0, + unicodelen=0) def test_strconcat_pure(self): - for somestr in ["abc", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - mylist = [somestr+str(i) for i in range(10)] - def f(n, m): - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + mylist[m] - if m > 100: - escape(s) - m -= 1 - return 42 - self.meta_interp(f, [6, 7]) - self.check_loops(newstr=0, strsetitem=0, - newunicode=0, unicodesetitem=0, - call=0, call_pure=0) + _str = self._str + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = [_str("abc") + _str(i) for i in range(10)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + mylist[m] + if m > 100: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=0, strsetitem=0, + newunicode=0, unicodesetitem=0, + call=0, call_pure=0) def test_strconcat_escape_str_str(self): + _str = self._str jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) @@ -171,46 +163,64 @@ m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=0, copystrcontent=2, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=0, copystrcontent=2, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=0, + copyunicodecontent=2, + call=1, call_pure=0) # escape def test_strconcat_escape_str_char(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + chr(m) + s = mylist[n] + _chr(m) escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=1, + copyunicodecontent=1, + call=1, call_pure=0) # escape def test_strconcat_escape_char_str(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = chr(n) + mylist[m] + s = _chr(n) + mylist[m] escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=1, + copyunicodecontent=1, + call=1, call_pure=0) # escape def test_strconcat_escape_char_char(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): @@ -219,91 +229,132 @@ while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = chr(n) + chr(m) + s = _chr(n) + _chr(m) escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=2, copystrcontent=0, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=2, copystrcontent=0, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=2, + copyunicodecontent=0, + call=1, call_pure=0) # escape def test_strconcat_escape_str_char_str(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + chr(n) + mylist[m] + s = mylist[n] + _chr(n) + mylist[m] escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=1, copystrcontent=2, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=1, copystrcontent=2, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=1, + copyunicodecontent=2, + call=1, call_pure=0) # escape def test_strconcat_guard_fail(self): - for somestr in ["abc", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - mylist = [somestr+str(i) for i in range(12)] - def f(n, m): - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + mylist[m] - if m & 1: - escape(s) - m -= 1 - return 42 - self.meta_interp(f, [6, 10]) + _str = self._str + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = [_str("abc") + _str(i) for i in range(12)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + mylist[m] + if m & 1: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 10]) def test_strslice(self): - for somestr in ["abc", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - def f(n, m): - assert n >= 0 - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = "foobarbazetc"[m:n] - if m <= 5: - escape(s) - m -= 1 - return 42 - self.meta_interp(f, [10, 10]) + _str = self._str + longstring = _str("foobarbazetc") + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + def f(n, m): + assert n >= 0 + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = longstring[m:n] + if m <= 5: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [10, 10]) def test_streq_char(self): - for somestr in ["?abcdefg", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - def f(n, m): - assert n >= 0 - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = somestr[:m] - escape(s == "?") - m -= 1 - return 42 - self.meta_interp(f, [6, 7]) - self.check_loops(newstr=0, newunicode=0) - - -class TestOOtype(StringTests, OOJitMixin): - CALL = "oosend" - CALL_PURE = "oosend_pure" + _str = self._str + longstring = _str("?abcdefg") + somechar = _str("?") + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + def f(n, m): + assert n >= 0 + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = longstring[:m] + escape(s == somechar) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=0, newunicode=0) + + +#class TestOOtype(StringTests, OOJitMixin): +# CALL = "oosend" +# CALL_PURE = "oosend_pure" class TestLLtype(StringTests, LLJitMixin): CALL = "call" CALL_PURE = "call_pure" + +class TestLLtypeUnicode(TestLLtype): + _str, _chr = unicode, unichr + + def test_str2unicode(self): + _str = self._str + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + class Foo: + pass + @dont_look_inside + def escape(x): + assert x == _str("6y") + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + foo = Foo() + foo.y = chr(m) + foo.y = "y" + s = _str(str(n)) + _str(foo.y) + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(call=3, # str(), _str(), escape() + newunicode=1, unicodegetitem=0, + unicodesetitem=1, copyunicodecontent=1) From arigo at codespeak.net Mon Oct 11 15:19:53 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 15:19:53 +0200 (CEST) Subject: [pypy-svn] r77791 - pypy/branch/jit-str-unicode Message-ID: <20101011131953.D42C1282BF7@codespeak.net> Author: arigo Date: Mon Oct 11 15:19:51 2010 New Revision: 77791 Removed: pypy/branch/jit-str-unicode/ Log: Branch merged. From afa at codespeak.net Mon Oct 11 15:55:46 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 15:55:46 +0200 (CEST) Subject: [pypy-svn] r77792 - pypy/trunk/pypy/rpython/module Message-ID: <20101011135546.208D9282BF3@codespeak.net> Author: afa Date: Mon Oct 11 15:55:43 2010 New Revision: 77792 Modified: pypy/trunk/pypy/rpython/module/ll_time.py Log: Fix a potential error on some platforms. Modified: pypy/trunk/pypy/rpython/module/ll_time.py ============================================================================== --- pypy/trunk/pypy/rpython/module/ll_time.py (original) +++ pypy/trunk/pypy/rpython/module/ll_time.py Mon Oct 11 15:55:43 2010 @@ -108,7 +108,7 @@ errcode = -1 if self.GETTIMEOFDAY_NO_TZ: - errcode = g_gettimeofday(t) + errcode = c_gettimeofday(t) else: errcode = c_gettimeofday(t, void) From afa at codespeak.net Mon Oct 11 16:45:32 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 16:45:32 +0200 (CEST) Subject: [pypy-svn] r77793 - in pypy/branch/fast-forward/pypy/module: _multiprocessing _multiprocessing/test sys Message-ID: <20101011144532.D422A282BF3@codespeak.net> Author: afa Date: Mon Oct 11 16:45:30 2010 New Revision: 77793 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py pypy/branch/fast-forward/pypy/module/sys/__init__.py Log: Tests pass on Linux Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Mon Oct 11 16:45:30 2010 @@ -1,18 +1,19 @@ from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root from pypy.interpreter.typedef import TypeDef, GetSetProperty from pypy.interpreter.gateway import interp2app, unwrap_spec -from pypy.interpreter.error import ( - wrap_windowserror, wrap_oserror, OperationError) +from pypy.interpreter.error import wrap_oserror, OperationError from pypy.rpython.lltypesystem import rffi, lltype from pypy.rlib.rarithmetic import r_uint from pypy.translator.tool.cbuild import ExternalCompilationInfo +from pypy.rpython.tool import rffi_platform as platform from pypy.module.thread import ll_thread -import sys, os, time +import sys, os, time, errno RECURSIVE_MUTEX, SEMAPHORE = range(2) if sys.platform == 'win32': from pypy.rlib import rwin32 + from pypy.interpreter_error import wrap_windowserror _CreateSemaphore = rwin32.winexternal( 'CreateSemaphoreA', [rffi.VOIDP, rffi.LONG, rffi.LONG, rwin32.LPCSTR], @@ -85,6 +86,100 @@ else: + from pypy.rlib import rposix + + eci = ExternalCompilationInfo( + includes = ['sys/time.h', + 'semaphore.h'], + libraries = ['rt'], + ) + + class CConfig: + _compilation_info_ = eci + TIMEVAL = platform.Struct('struct timeval', [('tv_sec', rffi.LONG), + ('tv_usec', rffi.LONG)]) + TIMESPEC = platform.Struct('struct timespec', [('tv_sec', rffi.TIME_T), + ('tv_nsec', rffi.LONG)]) + SEM_FAILED = platform.ConstantInteger('SEM_FAILED') + SEM_VALUE_MAX = platform.ConstantInteger('SEM_VALUE_MAX') + + config = platform.configure(CConfig) + TIMEVAL = config['TIMEVAL'] + TIMESPEC = config['TIMESPEC'] + TIMEVALP = rffi.CArrayPtr(TIMEVAL) + TIMESPECP = rffi.CArrayPtr(TIMESPEC) + SEM_T = rffi.COpaquePtr('sem_t', compilation_info=eci) + SEM_FAILED = rffi.cast(SEM_T, config['SEM_FAILED']) + SEM_VALUE_MAX = config['SEM_VALUE_MAX'] + HAVE_BROKEN_SEM_GETVALUE = False + + def external(name, args, result): + return rffi.llexternal(name, args, result, + compilation_info=eci) + + _sem_open = external('sem_open', + [rffi.CCHARP, rffi.INT, rffi.INT, rffi.UINT], + SEM_T) + _sem_unlink = external('sem_unlink', [rffi.CCHARP], rffi.INT) + _sem_wait = external('sem_wait', [SEM_T], rffi.INT) + _sem_trywait = external('sem_trywait', [SEM_T], rffi.INT) + _sem_timedwait = external('sem_timedwait', [SEM_T, TIMESPECP], rffi.INT) + _sem_post = external('sem_post', [SEM_T], rffi.INT) + _sem_getvalue = external('sem_getvalue', [SEM_T, rffi.INTP], rffi.INT) + + _gettimeofday = external('gettimeofday', [TIMEVALP, rffi.VOIDP], rffi.INT) + + def sem_open(name, oflag, mode, value): + res = _sem_open(name, oflag, mode, value) + if res == SEM_FAILED: + raise OSError(rposix.get_errno(), "sem_open failed") + return res + + def sem_unlink(name): + res = _sem_unlink(name) + if res < 0: + raise OSError(rposix.get_errno(), "sem_unlink failed") + + def sem_wait(sem): + res = _sem_wait(sem) + if res < 0: + raise OSError(rposix.get_errno(), "sem_wait failed") + + def sem_trywait(sem): + res = _sem_trywait(sem) + if res < 0: + raise OSError(rposix.get_errno(), "sem_trywait failed") + + def sem_timedwait(sem, deadline): + res = _sem_timedwait(sem, deadline) + if res < 0: + raise OSError(rposix.get_errno(), "sem_timedwait failed") + + def sem_post(sem): + res = _sem_post(sem) + if res < 0: + raise OSError(rposix.get_errno(), "sem_post failed") + + def sem_getvalue(sem): + sval_ptr = lltype.malloc(rffi.INTP.TO, 1, flavor='raw') + try: + res = _sem_getvalue(sem, sval_ptr) + if res < 0: + raise OSError(rposix.get_errno(), "sem_getvalue failed") + return sval_ptr[0] + finally: + lltype.free(sval_ptr, flavor='raw') + + def gettimeofday(): + now = lltype.malloc(TIMEVALP.TO, 1, flavor='raw') + try: + res = _gettimeofday(now, None) + if res < 0: + raise OSError(rposix.get_errno(), "sem_getvalue failed") + return now[0].c_tv_sec, now[0].c_tv_usec + finally: + lltype.free(now, flavor='raw') + class GlobalState: def init(self): pass @@ -146,8 +241,6 @@ raise wrap_windowserror(space, e) if res != rwin32.WAIT_TIMEOUT: - self.last_tid = ll_thread.get_ident() - self.count += 1 return True msecs = r_uint(full_msecs) @@ -181,8 +274,6 @@ # handle result if res != rwin32.WAIT_TIMEOUT: - self.last_tid = ll_thread.get_ident() - self.count += 1 return True return False @@ -199,50 +290,57 @@ space, WindowsError(err, "ReleaseSemaphore")) else: - HAVE_BROKEN_SEM_GETVALUE = False + def w_handle(space, value): + return space.newint(value) def create_semaphore(space, name, val, max): - sem_open(name, os.O_CREAT | os.O_EXCL, 0600, val) - sem_unlink(name) + res = sem_open(name, os.O_CREAT | os.O_EXCL, 0600, val) + try: + sem_unlink(name) + except OSError: + pass + return res def semlock_acquire(self, space, block, w_timeout): if not block: - deadline = lltype.nullptr(TIMESPEC.TO) + deadline = lltype.nullptr(TIMESPECP.TO) elif space.is_w(w_timeout, space.w_None): - deadline = lltype.nullptr(TIMESPEC.TO) + deadline = lltype.nullptr(TIMESPECP.TO) else: timeout = space.float_w(w_timeout) sec = int(timeout) nsec = int(1e9 * (timeout - sec) + 0.5) - deadline = lltype.malloc(TIMESPEC.TO, 1, flavor='raw') - deadline.c_tv_sec = now.c_tv_sec + sec - deadline.c_tv_nsec = now.c_tv_usec * 1000 + nsec - deadline.c_tv_sec += (deadline.c_tv_nsec / 1000000000) - deadline.c_tv_nsec %= 1000000000 + now_sec, now_usec = gettimeofday() + + deadline = lltype.malloc(TIMESPECP.TO, 1, flavor='raw') + deadline[0].c_tv_sec = now_sec + sec + deadline[0].c_tv_nsec = now_usec * 1000 + nsec + deadline[0].c_tv_sec += (deadline[0].c_tv_nsec / 1000000000) + deadline[0].c_tv_nsec %= 1000000000 try: while True: - if not block: - res = sem_trywait(self.handle) - elif not deadline: - res = sem_wait(self.handle) - else: - res = sem_timedwait(self.handle, deadline) - if res >= 0: - break - elif errno != EINTR: - break - # elif PyErr_CheckSignals(): - # raise... + try: + if not block: + sem_trywait(self.handle) + elif not deadline: + sem_wait(self.handle) + else: + sem_timedwait(self.handle, deadline) + except OSError, e: + if e.errno == errno.EINTR: + # again + continue + elif e.errno in (errno.EAGAIN, errno.ETIMEDOUT): + return False + raise wrap_oserror(space, e) + # XXX PyErr_CheckSignals() + + return True finally: if deadline: lltype.free(deadline, flavor='raw') - if res < 0: - if errno == EAGAIN or errno == ETIMEDOUT: - return False - raise wrap_oserror(space, errno) - return True def semlock_release(self, space): if self.kind == RECURSIVE_MUTEX: @@ -251,29 +349,27 @@ # We will only check properly the maxvalue == 1 case if self.maxvalue == 1: # make sure that already locked - if sem_trywait(self.handle) < 0: - if errno != EAGAIN: + try: + sem_trywait(self.handle) + except OSError, e: + if e.errno != errno.EAGAIN: raise # it is already locked as expected else: # it was not locked so undo wait and raise - if sem_post(self.handle) < 0: - raise + sem_post(self.handle) raise OperationError( space.w_ValueError, space.wrap( "semaphore or lock released too many times")) else: # This check is not an absolute guarantee that the semaphore does # not rise above maxvalue. - if sem_getvalue(self.handle, sval_ptr) < 0: - raise - if sval_ptr[0] >= self.maxvalue: - raise OperationError( - space.w_ValueError, space.wrap( - "semaphore or lock released too many times")) + if sem_getvalue(self.handle) >= self.maxvalue: + raise OperationError( + space.w_ValueError, space.wrap( + "semaphore or lock released too many times")) - if sem_post(self.handle) < 0: - raise + sem_post(self.handle) class W_SemLock(Wrappable): @@ -308,8 +404,12 @@ self.count += 1 return space.w_True - res = semlock_acquire(self, space, block, w_timeout) - return space.wrap(res) + if semlock_acquire(self, space, block, w_timeout): + self.last_tid = ll_thread.get_ident() + self.count += 1 + return space.w_True + else: + return space.w_False @unwrap_spec('self', ObjSpace) def release(self, space): Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py Mon Oct 11 16:45:30 2010 @@ -2,9 +2,10 @@ import sys from pypy.conftest import gettestobjspace -class TestConnection: +class TestImport: def test_simple(self): from pypy.module._multiprocessing import interp_connection + from pypy.module._multiprocessing import interp_semaphore class AppTestConnection: def setup_class(cls): Modified: pypy/branch/fast-forward/pypy/module/sys/__init__.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/sys/__init__.py (original) +++ pypy/branch/fast-forward/pypy/module/sys/__init__.py Mon Oct 11 16:45:30 2010 @@ -20,7 +20,7 @@ 'platform' : 'space.wrap(sys.platform)', 'maxint' : 'space.wrap(sys.maxint)', - 'maxsize' : 'space.wrap(sys.maxsize)', + 'maxsize' : 'space.wrap(sys.maxint)', 'byteorder' : 'space.wrap(sys.byteorder)', 'maxunicode' : 'space.wrap(vm.MAXUNICODE)', 'maxint' : 'space.wrap(sys.maxint)', From afa at codespeak.net Mon Oct 11 16:46:47 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 16:46:47 +0200 (CEST) Subject: [pypy-svn] r77794 - pypy/branch/fast-forward/pypy/module/_multiprocessing Message-ID: <20101011144647.EF200282BF3@codespeak.net> Author: afa Date: Mon Oct 11 16:46:45 2010 New Revision: 77794 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Log: Typo Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Mon Oct 11 16:46:45 2010 @@ -13,7 +13,7 @@ if sys.platform == 'win32': from pypy.rlib import rwin32 - from pypy.interpreter_error import wrap_windowserror + from pypy.interpreter.error import wrap_windowserror _CreateSemaphore = rwin32.winexternal( 'CreateSemaphoreA', [rffi.VOIDP, rffi.LONG, rffi.LONG, rwin32.LPCSTR], From arigo at codespeak.net Mon Oct 11 17:13:24 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 17:13:24 +0200 (CEST) Subject: [pypy-svn] r77795 - in pypy/trunk/pypy/translator/c/gcc: . test/elf Message-ID: <20101011151324.AD47D282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 17:13:22 2010 New Revision: 77795 Added: pypy/trunk/pypy/translator/c/gcc/test/elf/track10.s pypy/trunk/pypy/translator/c/gcc/test/elf/track11.s Modified: pypy/trunk/pypy/translator/c/gcc/test/elf/track5.s pypy/trunk/pypy/translator/c/gcc/trackgcroot.py Log: Fix TestAsmGCRootWithHybridTagged, which crashes because the hybrid gc's set_max_heap_size() method is somehow inlined, but always raises NotImplementedError -- and you get a large function that starts with "call RPyAssertFailed" and then more stuff that is completely unreachable. Fixed (after a few unsuccessful tries) by trimming the list of instructions to only keep the reachable ones. Added: pypy/trunk/pypy/translator/c/gcc/test/elf/track10.s ============================================================================== --- (empty file) +++ pypy/trunk/pypy/translator/c/gcc/test/elf/track10.s Mon Oct 11 17:13:22 2010 @@ -0,0 +1,14 @@ + .type main, @function +main: + pushl %ebx + call pypy_f + ;; expected {4(%esp) | (%esp), %esi, %edi, %ebp | %ebx} + je .L1 + call RPyAssertFailed + ;; the following call is not reachable: it should be ignored + call pypy_malloc_something +.L1: + /* GCROOT %ebx */ + popl %ebx + ret + .size main, .-main Added: pypy/trunk/pypy/translator/c/gcc/test/elf/track11.s ============================================================================== --- (empty file) +++ pypy/trunk/pypy/translator/c/gcc/test/elf/track11.s Mon Oct 11 17:13:22 2010 @@ -0,0 +1,451 @@ + .type pypy_g_f_gc_set_max_heap_size, @function +pypy_g_f_gc_set_max_heap_size: + ;; This really occurred in real-life (see around r77794). + ;; This function is large, but actually in all paths it + ;; will soon hit a RPyAssertFailed and abort. +.L22962: +.L22963: +.L22964: + pushl %ebp + movl $pypy_g_exceptions_NotImplementedError_vtable, %eax + pushl %edi + pushl %esi + pushl %ebx + subl $28, %esp + cmpl $pypy_g_py__code_assertion_AssertionError_vtable, %eax + je .L23136 +.L22965: +.L22967: +.L22969: +.L22971: +.L22972: +.L22974: + movl $.LC1, (%esp) + movl $.LC2, %ebp + movl $__FUNCTION__.1761, %esi + movl %ebp, 12(%esp) + movl $726, %edi + movl $loc.982, %ebp + movl %esi, 8(%esp) + xorl %esi, %esi + movl $pypy_g_exceptions_NotImplementedError_vtable, %ebx + movl %edi, 4(%esp) + call RPyAssertFailed + movl %ebx, pypy_g_ExcData + movl pypydtcount, %edi + xorl %edx, %edx + movl $400000, (%esp) + movl $pypy_g_exceptions_NotImplementedError_vtable, %eax + movl $pypy_g_exceptions_NotImplementedError, %ecx + movl %ecx, pypy_g_ExcData+4 + movl %edx, pypy_debug_tracebacks(,%edi,8) + movl %eax, pypy_debug_tracebacks+4(,%edi,8) + incl %edi + andl $127, %edi + movl %esi, pypy_debug_tracebacks+4(,%edi,8) + movl %ebp, pypy_debug_tracebacks(,%edi,8) + incl %edi + andl $127, %edi + movl %edi, pypydtcount + call pypy_g_mallocstr__Signed + movl pypy_g_ExcData, %ebx + movl %eax, %esi + testl %ebx, %ebx + jne .L22976 +.L22977: + xorl %ebx, %ebx + testl %eax, %eax + je .L23117 +.L23127: +.L22981: + movb $120, 12(%esi,%ebx) + incl %ebx + cmpl $399999, %ebx + jle .L23127 + .p2align 4,,15 +.L23112: + movl pypy_g_ExcData, %ebx + movl %esi, %edi + testl %ebx, %ebx + jne .L22990 +.L22991: +.L22992: +.L22993: +.L23137: +.L22995: + movl $4000000, (%esp) + call pypy_g_mallocstr__Signed + movl pypy_g_ExcData, %ebx + movl %eax, %esi + testl %ebx, %ebx + jne .L22997 +.L22998: + xorl %ebx, %ebx + testl %eax, %eax + je .L23120 +.L23129: +.L23002: + movb $120, 12(%esi,%ebx) + incl %ebx + cmpl $3999999, %ebx + jle .L23129 +.L23114: + movl pypy_g_ExcData, %ebx + movl %esi, %ebp + jmp .L23004 +.L22980: + .p2align 4,,7 +.L22976: +.L22983: +.L22987: + movl pypydtcount, %eax + xorl %edx, %edx + movl $loc.586, %ecx + xorl %edi, %edi + movl %ecx, pypy_debug_tracebacks(,%eax,8) + movl %edx, pypy_debug_tracebacks+4(,%eax,8) + incl %eax + andl $127, %eax + movl %eax, pypydtcount + testl %ebx, %ebx + je .L23137 +.L22990: + movl pypydtcount, %ecx + movl $loc.1444, %edx + movl pypy_g_ExcData+4, %edi + movl %edx, pypy_debug_tracebacks(,%ecx,8) + movl %ebx, pypy_debug_tracebacks+4(,%ecx,8) + incl %ecx + andl $127, %ecx + movl %ecx, pypydtcount + cmpl $pypy_g_py__code_assertion_AssertionError_vtable, %ebx + je .L23075 + cmpl $pypy_g_exceptions_NotImplementedError_vtable, %ebx + je .L23075 +.L23074: +.L23076: + xorl %eax, %eax + xorl %ebp, %ebp + movl %eax, pypy_g_ExcData+4 + movl %ebp, pypy_g_ExcData +#APP + /* keepalive %edi */ +#NO_APP + testl %ebx, %ebx + movl pypy_g_exceptions_MemoryError_vtable, %esi + je .L23138 +.L23080: +.L23083: + movl (%ebx), %ebp + movl pypy_g_exceptions_MemoryError_vtable+4, %edx + subl %esi, %ebp + subl %esi, %edx + cmpl %edx, %ebp + setb %cl +#APP + /* GC_NOCOLLECT pypy_g_ll_issubclass */ +#NO_APP + testb %cl, %cl + je .L23086 + xorl %edx, %edx + xorl %ecx, %ecx + xorl %eax, %eax +.L23036: +.L23038: +.L23040: + testl %eax, %eax + movzbl %dl, %ebp + setne %bl + movzbl %bl, %esi + xorl %eax, %eax + addl %esi, %ebp + testl %ecx, %ecx + setne %al + leal (%ebp,%eax), %eax +.L23041: + addl $28, %esp + popl %ebx + popl %esi + popl %edi + popl %ebp + ret + .p2align 4,,7 +.L23117: + call RPyAbort + movb $120, 12(%esi,%ebx) + incl %ebx + cmpl $399999, %ebx + jg .L23112 + call RPyAbort + movb $120, 12(%esi,%ebx) + incl %ebx + cmpl $399999, %ebx + jle .L23117 + jmp .L23112 +.L23075: + call pypy_debug_catch_fatal_exception + jmp .L23074 +.L23138: + call RPyAbort + jmp .L23080 +.L23136: + movl $.LC1, (%esp) + movl $.LC0, %ebx + movl $__FUNCTION__.1761, %ecx + movl %ebx, 12(%esp) + movl $724, %edx + movl %ecx, 8(%esp) + movl %edx, 4(%esp) + call RPyAssertFailed + jmp .L22965 +.L23001: +.L22997: + movl pypydtcount, %ebp + movl $loc.586, %eax + xorl %esi, %esi + movl %eax, pypy_debug_tracebacks(,%ebp,8) + movl %esi, pypy_debug_tracebacks+4(,%ebp,8) + incl %ebp + andl $127, %ebp + movl %ebp, pypydtcount + xorl %ebp, %ebp +.L23004: +.L23008: + movl %edi, %esi +#APP + /* GCROOT %esi */ +#NO_APP + movl %esi, 24(%esp) + testl %ebx, %ebx + je .L23139 +.L23011: + movl pypydtcount, %ecx + movl $loc.1443, %edx + movl pypy_g_ExcData+4, %edi + movl %edx, pypy_debug_tracebacks(,%ecx,8) + movl %ebx, pypy_debug_tracebacks+4(,%ecx,8) + incl %ecx + andl $127, %ecx + movl %ecx, pypydtcount + cmpl $pypy_g_py__code_assertion_AssertionError_vtable, %ebx + je .L23059 + cmpl $pypy_g_exceptions_NotImplementedError_vtable, %ebx + je .L23059 +.L23058: +.L23060: + xorl %ebp, %ebp + xorl %eax, %eax + movl %ebp, pypy_g_ExcData+4 + movl %eax, pypy_g_ExcData +#APP + /* keepalive %edi */ + /* keepalive %esi */ +#NO_APP + testl %ebx, %ebx + movl pypy_g_exceptions_MemoryError_vtable, %esi + je .L23140 +.L23064: +.L23067: + movl (%ebx), %eax + xorl %ecx, %ecx + movl pypy_g_exceptions_MemoryError_vtable+4, %ebp + subl %esi, %eax + subl %esi, %ebp + xorl %esi, %esi + cmpl %ebp, %eax + setb %dl +#APP + /* GC_NOCOLLECT pypy_g_ll_issubclass */ +#NO_APP + testb %dl, %dl + je .L23141 +.L23033: +.L23034: + movl 24(%esp), %ebx + movb $1, %dl + testl %ebx, %ebx + je .L23142 + movl %esi, %eax + jmp .L23036 +.L23086: +.L23087: +.L23088: + movl %edi, pypy_g_ExcData+4 + movl pypydtcount, %esi + movl $-1, %edi + movl %ebx, pypy_g_ExcData + movl %edi, pypy_debug_tracebacks(,%esi,8) + movl %ebx, pypy_debug_tracebacks+4(,%esi,8) + incl %esi + andl $127, %esi + movl %esi, pypydtcount +#APP + /* GC_NOCOLLECT pypy_g_RPyReRaiseException */ +#NO_APP + movl $-1, %eax + jmp .L23041 + .p2align 4,,7 +.L23120: + call RPyAbort + movb $120, 12(%esi,%ebx) + incl %ebx + cmpl $3999999, %ebx + jg .L23114 + call RPyAbort + movb $120, 12(%esi,%ebx) + incl %ebx + cmpl $3999999, %ebx + jle .L23120 + jmp .L23114 +.L23142: + xorl %edx, %edx + movl %esi, %eax + jmp .L23036 +.L23059: + call pypy_debug_catch_fatal_exception + jmp .L23058 +.L23140: + call RPyAbort + jmp .L23064 +.L23012: +.L23013: +.L23014: +.L23139: +.L23016: + movl $40000000, (%esp) + call pypy_g_mallocstr__Signed + movl pypy_g_ExcData, %ebx + movl %eax, %edi + testl %ebx, %ebx + jne .L23018 +.L23019: + xorl %ebx, %ebx + testl %eax, %eax + je .L23123 +.L23131: +.L23023: + movb $120, 12(%edi,%ebx) + incl %ebx + cmpl $39999999, %ebx + jle .L23131 +.L23116: + movl pypy_g_ExcData, %ebx + movl %edi, %eax +.L23025: +.L23029: +#APP + /* GCROOT %esi */ +#NO_APP + movl %esi, 24(%esp) + movl %ebp, %edi +#APP + /* GCROOT %edi */ +#NO_APP + testl %ebx, %ebx + jne .L23032 + movl %eax, %ecx + movl %edi, %esi + jmp .L23033 +.L23123: + call RPyAbort + movb $120, 12(%edi,%ebx) + incl %ebx + cmpl $39999999, %ebx + jle .L23123 + jmp .L23116 +.L23022: +.L23018: + movl pypydtcount, %ecx + movl $loc.586, %edx + xorl %edi, %edi + xorl %eax, %eax + movl %edx, pypy_debug_tracebacks(,%ecx,8) + movl %edi, pypy_debug_tracebacks+4(,%ecx,8) + incl %ecx + andl $127, %ecx + movl %ecx, pypydtcount + jmp .L23025 +.L23141: +.L23070: +.L23071: +.L23072: + movl %edi, pypy_g_ExcData+4 + movl pypydtcount, %esi + movl $-1, %edi + movl %ebx, pypy_g_ExcData + movl %edi, pypy_debug_tracebacks(,%esi,8) + movl %ebx, pypy_debug_tracebacks+4(,%esi,8) + incl %esi + andl $127, %esi + movl %esi, pypydtcount +#APP + /* GC_NOCOLLECT pypy_g_RPyReRaiseException */ +#NO_APP + movl $-1, %eax + jmp .L23041 +.L23032: + movl pypydtcount, %ecx + movl $loc.1442, %edx + movl pypy_g_ExcData+4, %ebp + movl %edx, pypy_debug_tracebacks(,%ecx,8) + movl %ebx, pypy_debug_tracebacks+4(,%ecx,8) + incl %ecx + andl $127, %ecx + movl %ecx, pypydtcount + cmpl $pypy_g_py__code_assertion_AssertionError_vtable, %ebx + je .L23043 + cmpl $pypy_g_exceptions_NotImplementedError_vtable, %ebx + je .L23043 +.L23042: +.L23044: + xorl %ecx, %ecx + xorl %eax, %eax + movl %ecx, pypy_g_ExcData+4 + movl %eax, pypy_g_ExcData +#APP + /* keepalive %ebp */ + /* keepalive %esi */ + /* keepalive %edi */ +#NO_APP + testl %ebx, %ebx + movl pypy_g_exceptions_MemoryError_vtable, %esi + je .L23143 +.L23048: +.L23051: + movl (%ebx), %eax + xorl %ecx, %ecx + movl pypy_g_exceptions_MemoryError_vtable+4, %edx + subl %esi, %eax + subl %esi, %edx + cmpl %edx, %eax + movl %edi, %esi + setb %dl +#APP + /* GC_NOCOLLECT pypy_g_ll_issubclass */ +#NO_APP + testb %dl, %dl + jne .L23033 +.L23054: +.L23055: +.L23056: + movl %ebp, pypy_g_ExcData+4 + movl pypydtcount, %edi + movl $-1, %ebp + movl %ebx, pypy_g_ExcData + movl %ebp, pypy_debug_tracebacks(,%edi,8) + movl %ebx, pypy_debug_tracebacks+4(,%edi,8) + incl %edi + andl $127, %edi + movl %edi, pypydtcount +#APP + /* GC_NOCOLLECT pypy_g_RPyReRaiseException */ +#NO_APP + movl $-1, %eax + jmp .L23041 +.L23043: + call pypy_debug_catch_fatal_exception + jmp .L23042 +.L23143: + call RPyAbort + jmp .L23048 + .size pypy_g_f_gc_set_max_heap_size, .-pypy_g_f_gc_set_max_heap_size Modified: pypy/trunk/pypy/translator/c/gcc/test/elf/track5.s ============================================================================== --- pypy/trunk/pypy/translator/c/gcc/test/elf/track5.s (original) +++ pypy/trunk/pypy/translator/c/gcc/test/elf/track5.s Mon Oct 11 17:13:22 2010 @@ -44,7 +44,7 @@ addl %eax, %ebx jmp .L1221 .L1227: - call RPyAbort + ;;call RPyAbort cmpl 12(%esi), %ebx jb .L1229 addl $20, %esp Modified: pypy/trunk/pypy/translator/c/gcc/trackgcroot.py ============================================================================== --- pypy/trunk/pypy/translator/c/gcc/trackgcroot.py (original) +++ pypy/trunk/pypy/translator/c/gcc/trackgcroot.py Mon Oct 11 17:13:22 2010 @@ -46,6 +46,7 @@ self.findlabels() self.parse_instructions() try: + self.trim_unreachable_instructions() self.find_noncollecting_calls() if not self.list_collecting_call_insns(): return [] @@ -122,6 +123,26 @@ assert label not in self.labels, "duplicate label: %s" % label self.labels[label] = Label(label, lineno) + def trim_unreachable_instructions(self): + reached = set([self.insns[0]]) + prevlen = 0 + while len(reached) > prevlen: + prevlen = len(reached) + for insn in self.insns: + if insn not in reached: + for previnsn in insn.previous_insns: + if previnsn in reached: + # this instruction is reachable too + reached.add(insn) + break + # now kill all unreachable instructions + i = 0 + while i < len(self.insns): + if self.insns[i] in reached: + i += 1 + else: + del self.insns[i] + def find_noncollecting_calls(self): cannot_collect = {} for line in self.lines: @@ -752,7 +773,7 @@ target, = sources if target in self.FUNCTIONS_NOT_RETURNING: - return [InsnStop(target), InsnCannotFollowEsp()] + return [InsnStop(target)] if self.format == 'mingw32' and target == '__alloca': # in functions with large stack requirements, windows # needs a call to _alloca(), to turn reserved pages From david at codespeak.net Mon Oct 11 17:40:36 2010 From: david at codespeak.net (david at codespeak.net) Date: Mon, 11 Oct 2010 17:40:36 +0200 (CEST) Subject: [pypy-svn] r77796 - in pypy/branch/arm-backend/pypy/jit/backend/arm: . test Message-ID: <20101011154036.28BE0282BF3@codespeak.net> Author: david Date: Mon Oct 11 17:40:34 2010 New Revision: 77796 Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py Log: (antocuni, david) some refactoring of the resoperation encoding, start using the register allocator, Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py Mon Oct 11 17:40:34 2010 @@ -1,8 +1,10 @@ from pypy.jit.backend.arm.codebuilder import ARMv7Builder from pypy.jit.backend.arm import registers as r from pypy.jit.backend.arm import conditions as c -#from pypy.jit.backend.arm.regalloc import RegAlloc, ARMRegisterManager +from pypy.jit.backend.llsupport.regalloc import compute_vars_longevity +from pypy.jit.backend.arm.regalloc import ARMRegisterManager from pypy.jit.metainterp.resoperation import rop +from pypy.jit.metainterp.history import ConstInt, Box from pypy.rpython.lltypesystem import lltype # XXX Move to llsupport from pypy.jit.backend.x86.support import values_array @@ -13,42 +15,81 @@ def __init__(self, cpu, failargs_limit=1000): self.mc = ARMv7Builder() self.cpu = cpu + self.input_arg_boxes_int = values_array(lltype.Signed, failargs_limit) # merge with fail_boxes_int later self.fail_boxes_int = values_array(lltype.Signed, failargs_limit) def assemble_loop(self, inputargs, operations, looptoken): assert len(inputargs) == 1 - reg = 0 + longevity = compute_vars_longevity(inputargs, operations) + regalloc = ARMRegisterManager(longevity, assembler=self.mc) self.gen_func_prolog() - addr = self.fail_boxes_int.get_addr_for_num(0) - self.gen_load_int(r.r3, addr) - self.mc.LDR_ri(r.r1, r.r3) + self.gen_bootstrap_code(inputargs, regalloc) loop_head=self.mc.curraddr() + looptoken._arm_bootstrap_code = self.mc.baseaddr() + looptoken._arm_loop_code = loop_head + looptoken._temp_inputargs = inputargs#XXX remove fcond=c.AL for op in operations: opnum = op.getopnum() - if opnum == rop.INT_ADD: - self.mc.ADD_ri(r.r1, r.r1, op.getarg(1).getint()) - elif opnum == rop.INT_LE: - self.mc.CMP(r.r1, op.getarg(1).getint()) - fcond = c.GT - elif opnum == rop.GUARD_TRUE: - n = self.cpu.get_fail_descr_number(op.getdescr()) - self.mc.MOV_ri(r.r0, n, cond=fcond) - self.mc.STR_ri(r.r1, r.r3, cond=fcond) - self.gen_func_epilog(cond=fcond) - fcond = c.AL - elif opnum == rop.JUMP: - self.gen_load_int(r.r7, loop_head) - self.mc.MOV_rr(r.pc, r.r7) - elif opnum == rop.FINISH: - n = self.cpu.get_fail_descr_number(op.getdescr()) - self.mc.MOV_ri(r.r0, n) - self.mc.STR_ri(r.r1, r.r3) - else: - raise ValueError("Unknown op %r" % op) + fcond = self.operations[opnum](self, op, regalloc, fcond) self.gen_func_epilog() - looptoken._arm_bootstrap_code = self.mc.baseaddr() - looptoken._arm_loop_code = loop_head + f = open('loop.asm', 'wb') + for i in range(self.mc._pos): + f.write(self.mc._data[i]) + f.close() + print 'Done assembling' + + def emit_op_jump(self, op, regalloc, fcond): + tmp = Box() + tmpreg = regalloc.try_allocate_reg(tmp) + inputargs = op.getdescr()._temp_inputargs + for i in range(op.numargs()): + reg = regalloc.try_allocate_reg(op.getarg(i)) + inpreg = regalloc.try_allocate_reg(inputargs[i]) + # XXX only if every value is in a register + self.mc.MOV_rr(inpreg, reg) + loop_code = op.getdescr()._arm_loop_code + self.gen_load_int(tmpreg, loop_code) + self.mc.MOV_rr(r.pc, tmpreg) + regalloc.possibly_free_var(tmpreg) + return fcond + + def emit_op_finish(self, op, regalloc, fcond): + self.gen_write_back(op, op.getarglist(), regalloc, fcond) + return fcond + + def emit_op_int_le(self, op, regalloc, fcond): + reg = regalloc.try_allocate_reg(op.getarg(0)) + assert isinstance(op.getarg(1), ConstInt) + self.mc.CMP(reg, op.getarg(1).getint()) + return c.GT + + def emit_op_int_add(self, op, regalloc, fcond): + reg = regalloc.try_allocate_reg(op.getarg(0)) + res = regalloc.try_allocate_reg(op.result) + assert isinstance(op.getarg(1), ConstInt) + self.mc.ADD_ri(res, reg, op.getarg(1).getint()) + regalloc.possibly_free_vars_for_op(op) + return fcond + + def emit_op_guard_true(self, op, regalloc, fcond): + assert fcond == c.GT + self.gen_write_back(op, op.getfailargs(), regalloc, fcond) + self.gen_func_epilog(cond=fcond) + return c.AL + + def gen_write_back(self, op, args, regalloc, fcond): + temp = Box() + temp_reg = regalloc.try_allocate_reg(temp) + for i in range(len(args)): + reg = regalloc.try_allocate_reg(args[i]) + addr = self.fail_boxes_int.get_addr_for_num(i) + self.gen_load_int(temp_reg, addr, cond=fcond) + self.mc.STR_ri(reg, temp_reg, cond=fcond) + + regalloc.possibly_free_var(temp_reg) + n = self.cpu.get_fail_descr_number(op.getdescr()) + self.mc.MOV_ri(r.r0, n, cond=fcond) def gen_func_epilog(self,cond=c.AL): self.mc.LDM(r.sp, r.callee_restored_registers, cond=cond) @@ -56,6 +97,13 @@ def gen_func_prolog(self): self.mc.PUSH(r.callee_saved_registers) + def gen_bootstrap_code(self, inputargs, regalloc): + for i in range(len(inputargs)): + reg = regalloc.try_allocate_reg(inputargs[i]) + addr = self.input_arg_boxes_int.get_addr_for_num(i) + self.gen_load_int(reg, addr) + self.mc.LDR_ri(reg, reg) + def gen_load_int(self, reg, value, cond=c.AL): assert reg != r.ip, 'ip is used to load int' self.mc.MOV_ri(reg, (value & 0xFF), cond=cond) @@ -63,3 +111,24 @@ for offset in range(8, 25, 8): self.mc.MOV_ri(r.ip, (value >> offset) & 0xFF, cond=cond) self.mc.ORR_rr(reg, reg, r.ip, offset, cond=cond) + + + +def make_operation_list(): + def notimplemented(self, op, regalloc, fcond): + raise NotImplementedError + + operations = [None] * (rop._LAST+1) + for key, value in rop.__dict__.items(): + key = key.lower() + if key.startswith('_'): + continue + methname = 'emit_op_%s' % key + if hasattr(AssemblerARM, methname): + func = getattr(AssemblerARM, methname).im_func + else: + func = notimplemented + operations[value] = func + return operations + +AssemblerARM.operations = make_operation_list() Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py Mon Oct 11 17:40:34 2010 @@ -6,6 +6,7 @@ lr = 14 pc = 15 +all_regs = range(12) callee_resp = [r4, r5, r6, r7, r8, r9, r10, r11] callee_saved_registers = callee_resp+[lr] callee_restored_registers = callee_resp+[pc] Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py Mon Oct 11 17:40:34 2010 @@ -19,7 +19,7 @@ self.assembler.assemble_loop(inputargs, operations, looptoken) def set_future_value_int(self, index, intvalue): - self.assembler.fail_boxes_int.setitem(index, intvalue) + self.assembler.input_arg_boxes_int.setitem(index, intvalue) def get_latest_value_int(self, index): return self.assembler.fail_boxes_int.getitem(index) Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py ============================================================================== --- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py (original) +++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py Mon Oct 11 17:40:34 2010 @@ -2,6 +2,7 @@ from pypy.jit.backend.arm import conditions as c from pypy.jit.backend.arm.assembler import AssemblerARM from pypy.jit.backend.arm.test.support import skip_unless_arm, run_asm +from pypy.jit.metainterp.resoperation import rop skip_unless_arm() @@ -9,6 +10,10 @@ def setup_method(self, method): self.a = AssemblerARM(None) + def test_make_operation_list(self): + i = rop.INT_ADD + assert self.a.operations[i] is AssemblerARM.emit_op_int_add.im_func + def test_load_small_int_to_reg(self): self.a.gen_func_prolog() self.a.gen_load_int(r.r0, 123) @@ -43,7 +48,7 @@ self.a.gen_func_epilog() assert run_asm(self.a) == 123333 - def test_int_le(self): + def test_cmp(self): self.a.gen_func_prolog() self.a.gen_load_int(r.r1, 22) self.a.mc.CMP(r.r1, 123) @@ -86,3 +91,4 @@ self.a.gen_func_epilog() assert run_asm(self.a) == 9 + From arigo at codespeak.net Mon Oct 11 17:40:43 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 17:40:43 +0200 (CEST) Subject: [pypy-svn] r77797 - in pypy/trunk/pypy: rlib rpython/lltypesystem Message-ID: <20101011154043.CFA18282BF7@codespeak.net> Author: arigo Date: Mon Oct 11 17:40:41 2010 New Revision: 77797 Modified: pypy/trunk/pypy/rlib/rmmap.py pypy/trunk/pypy/rpython/lltypesystem/ll2ctypes.py Log: Need to specify MAP_NORESERVE here, otherwise the mmap will fail if the system doesn't have enough RAM+SWAP (about 20GB). Modified: pypy/trunk/pypy/rlib/rmmap.py ============================================================================== --- pypy/trunk/pypy/rlib/rmmap.py (original) +++ pypy/trunk/pypy/rlib/rmmap.py Mon Oct 11 17:40:41 2010 @@ -50,7 +50,7 @@ constant_names = ['MAP_SHARED', 'MAP_PRIVATE', 'PROT_READ', 'PROT_WRITE', 'MS_SYNC'] - opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS', + opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS', 'MAP_NORESERVE', 'PROT_EXEC', 'MAP_DENYWRITE', 'MAP_EXECUTABLE'] for name in constant_names: Modified: pypy/trunk/pypy/rpython/lltypesystem/ll2ctypes.py ============================================================================== --- pypy/trunk/pypy/rpython/lltypesystem/ll2ctypes.py (original) +++ pypy/trunk/pypy/rpython/lltypesystem/ll2ctypes.py Mon Oct 11 17:40:41 2010 @@ -69,7 +69,7 @@ PIECESIZE = 0x08000000 PIECES = 10 m = rmmap.mmap(-1, PIECES * PIECESIZE, - rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS, + rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS|rmmap.MAP_NORESERVE, rmmap.PROT_READ|rmmap.PROT_WRITE) m.close = lambda : None # leak instead of giving a spurious # error at CPython's shutdown From afa at codespeak.net Mon Oct 11 17:52:45 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 17:52:45 +0200 (CEST) Subject: [pypy-svn] r77798 - pypy/branch/fast-forward/pypy/module/_multiprocessing Message-ID: <20101011155245.D2EE9282BF3@codespeak.net> Author: afa Date: Mon Oct 11 17:52:43 2010 New Revision: 77798 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py Log: multiprocessing Connection objects need to be picklable Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py Mon Oct 11 17:52:43 2010 @@ -290,6 +290,7 @@ W_FileConnection.typedef = TypeDef( 'Connection', W_BaseConnection.typedef, __new__ = interp2app(W_FileConnection.descr_new_file.im_func), + __module__ = '_multiprocessing', fileno = interp2app(W_FileConnection.fileno), ) @@ -407,5 +408,6 @@ W_PipeConnection.typedef = TypeDef( 'PipeConnection', W_BaseConnection.typedef, __new__ = interp2app(W_PipeConnection.descr_new_pipe.im_func), + __module__ = '_multiprocessing', fileno = interp2app(W_PipeConnection.fileno), ) From afa at codespeak.net Mon Oct 11 17:55:30 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 17:55:30 +0200 (CEST) Subject: [pypy-svn] r77799 - pypy/branch/fast-forward/pypy/module/_multiprocessing Message-ID: <20101011155530.6E00C282BF3@codespeak.net> Author: afa Date: Mon Oct 11 17:55:28 2010 New Revision: 77799 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py Log: Various translation fixes Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py Mon Oct 11 17:55:28 2010 @@ -25,6 +25,9 @@ return OperationError(space.w_ValueError, space.wrap("BUFFERTOOSHORT")) +def w_handle(space, handle): + return space.wrap(rffi.cast(rffi.INTPTR_T, handle)) + class W_BaseConnection(Wrappable): BUFFER_SIZE = 1024 @@ -323,7 +326,6 @@ @unwrap_spec('self', ObjSpace) def fileno(self, space): - from pypy.module._multiprocessing.interp_win32 import w_handle return w_handle(space, self.handle) def do_close(self): Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Mon Oct 11 17:55:28 2010 @@ -7,6 +7,7 @@ from pypy.translator.tool.cbuild import ExternalCompilationInfo from pypy.rpython.tool import rffi_platform as platform from pypy.module.thread import ll_thread +from pypy.module._multiprocessing.interp_connection import w_handle import sys, os, time, errno RECURSIVE_MUTEX, SEMAPHORE = range(2) @@ -15,6 +16,8 @@ from pypy.rlib import rwin32 from pypy.interpreter.error import wrap_windowserror + SEM_VALUE_MAX = sys.maxint + _CreateSemaphore = rwin32.winexternal( 'CreateSemaphoreA', [rffi.VOIDP, rffi.LONG, rffi.LONG, rwin32.LPCSTR], rwin32.HANDLE) @@ -34,7 +37,7 @@ 'ResetEvent', [rwin32.HANDLE], rwin32.BOOL) # This is needed because the handler function must have the "WINAPI" - # callinf convention, which is not supported by lltype.Ptr. + # calling convention, which is not supported by lltype.Ptr. eci = ExternalCompilationInfo( separate_module_sources=[''' #include @@ -166,7 +169,7 @@ res = _sem_getvalue(sem, sval_ptr) if res < 0: raise OSError(rposix.get_errno(), "sem_getvalue failed") - return sval_ptr[0] + return rffi.cast(lltype.Signed, sval_ptr[0]) finally: lltype.free(sval_ptr, flavor='raw') @@ -175,7 +178,7 @@ try: res = _gettimeofday(now, None) if res < 0: - raise OSError(rposix.get_errno(), "sem_getvalue failed") + raise OSError(rposix.get_errno(), "gettimeofday failed") return now[0].c_tv_sec, now[0].c_tv_usec finally: lltype.free(now, flavor='raw') @@ -206,9 +209,6 @@ return value if sys.platform == 'win32': - SEM_VALUE_MAX = sys.maxint - from pypy.module._multiprocessing.interp_win32 import w_handle - def create_semaphore(space, name, val, max): rwin32.SetLastError(0) handle = _CreateSemaphore(rffi.NULL, val, max, rffi.NULL) @@ -290,16 +290,13 @@ space, WindowsError(err, "ReleaseSemaphore")) else: - def w_handle(space, value): - return space.newint(value) - def create_semaphore(space, name, val, max): - res = sem_open(name, os.O_CREAT | os.O_EXCL, 0600, val) + sem = sem_open(name, os.O_CREAT | os.O_EXCL, 0600, val) try: sem_unlink(name) except OSError: pass - return res + return sem def semlock_acquire(self, space, block, w_timeout): if not block: Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py Mon Oct 11 17:55:28 2010 @@ -6,6 +6,7 @@ from pypy.rpython.lltypesystem import rffi, lltype from pypy.translator.tool.cbuild import ExternalCompilationInfo from pypy.rpython.tool import rffi_platform +from pypy.module._multiprocessing import w_handle CONSTANTS = """ PIPE_ACCESS_INBOUND PIPE_ACCESS_DUPLEX @@ -31,8 +32,6 @@ def handle_w(space, w_handle): return rffi.cast(rwin32.HANDLE, space.int_w(w_handle)) -def w_handle(space, handle): - return space.wrap(rffi.cast(rffi.INTPTR_T, handle)) _CreateNamedPipe = rwin32.winexternal( 'CreateNamedPipeA', [ From arigo at codespeak.net Mon Oct 11 17:58:25 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 17:58:25 +0200 (CEST) Subject: [pypy-svn] r77800 - pypy/trunk/pypy/rlib/test Message-ID: <20101011155825.9070A282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 17:58:23 2010 New Revision: 77800 Modified: pypy/trunk/pypy/rlib/test/test_rsocket.py Log: Add a (failing) test that _rsocket_rffi.py is really importable on platforms where AF_NETLINK or AF_PACKET are not defined. Modified: pypy/trunk/pypy/rlib/test/test_rsocket.py ============================================================================== --- pypy/trunk/pypy/rlib/test/test_rsocket.py (original) +++ pypy/trunk/pypy/rlib/test/test_rsocket.py Mon Oct 11 17:58:23 2010 @@ -437,3 +437,27 @@ foo = self.serv.accept() py.test.raises(SocketError, raise_error) +def test_cond_includes(): + import re + # Test that _rsocket_rffi is importable even on platforms where + # AF_PACKET or AF_NETLINK is not defined. + for cond in ['AF_PACKET', 'AF_NETLINK']: + print cond + from pypy.rlib import _rsocket_rffi + srcfile = _rsocket_rffi.__file__ + if srcfile.lower().endswith('c') or srcfile.lower().endswith('o'): + srcfile = srcfile[:-1] # .pyc => .py + assert srcfile.lower().endswith('.py') + sourcelines = open(srcfile, 'rb').read().splitlines() + found = False + for i, line in enumerate(sourcelines): + line2 = re.sub(r"(\s*COND_HEADER\s*=)", + r"\1'#undef %s\\n'+" % cond, + line) + if line2 != line: + found = True + sourcelines[i] = line2 + assert found + d = {} + sourcelines.append('') + exec '\n'.join(sourcelines) in d From arigo at codespeak.net Mon Oct 11 17:59:56 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 17:59:56 +0200 (CEST) Subject: [pypy-svn] r77801 - pypy/trunk/pypy/rlib/test Message-ID: <20101011155956.BDB78282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 17:59:54 2010 New Revision: 77801 Modified: pypy/trunk/pypy/rlib/test/test_rsocket.py Log: Rewrite the test. Modified: pypy/trunk/pypy/rlib/test/test_rsocket.py ============================================================================== --- pypy/trunk/pypy/rlib/test/test_rsocket.py (original) +++ pypy/trunk/pypy/rlib/test/test_rsocket.py Mon Oct 11 17:59:54 2010 @@ -437,27 +437,31 @@ foo = self.serv.accept() py.test.raises(SocketError, raise_error) -def test_cond_includes(): - import re +def _test_cond_include(cond): # Test that _rsocket_rffi is importable even on platforms where # AF_PACKET or AF_NETLINK is not defined. - for cond in ['AF_PACKET', 'AF_NETLINK']: - print cond - from pypy.rlib import _rsocket_rffi - srcfile = _rsocket_rffi.__file__ - if srcfile.lower().endswith('c') or srcfile.lower().endswith('o'): - srcfile = srcfile[:-1] # .pyc => .py - assert srcfile.lower().endswith('.py') - sourcelines = open(srcfile, 'rb').read().splitlines() - found = False - for i, line in enumerate(sourcelines): - line2 = re.sub(r"(\s*COND_HEADER\s*=)", - r"\1'#undef %s\\n'+" % cond, - line) - if line2 != line: - found = True - sourcelines[i] = line2 - assert found - d = {} - sourcelines.append('') - exec '\n'.join(sourcelines) in d + import re + from pypy.rlib import _rsocket_rffi + srcfile = _rsocket_rffi.__file__ + if srcfile.lower().endswith('c') or srcfile.lower().endswith('o'): + srcfile = srcfile[:-1] # .pyc => .py + assert srcfile.lower().endswith('.py') + sourcelines = open(srcfile, 'rb').read().splitlines() + found = False + for i, line in enumerate(sourcelines): + line2 = re.sub(r"(\s*COND_HEADER\s*=)", + r"\1'#undef %s\\n'+" % cond, + line) + if line2 != line: + found = True + sourcelines[i] = line2 + assert found + d = {} + sourcelines.append('') + exec '\n'.join(sourcelines) in d + +def test_no_AF_PACKET(): + _test_cond_include('AF_PACKET') + +def test_no_AF_NETLINK(): + _test_cond_include('AF_NETLINK') From arigo at codespeak.net Mon Oct 11 18:04:23 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 18:04:23 +0200 (CEST) Subject: [pypy-svn] r77802 - pypy/trunk/pypy/rlib Message-ID: <20101011160423.B1309282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 18:04:21 2010 New Revision: 77802 Modified: pypy/trunk/pypy/rlib/_rsocket_rffi.py Log: Fix. Modified: pypy/trunk/pypy/rlib/_rsocket_rffi.py ============================================================================== --- pypy/trunk/pypy/rlib/_rsocket_rffi.py (original) +++ pypy/trunk/pypy/rlib/_rsocket_rffi.py Mon Oct 11 18:04:21 2010 @@ -324,10 +324,11 @@ ('sll_hatype', rffi.INT), ('sll_addr', rffi.CFixedArray(rffi.CHAR, 8)), ('sll_halen', rffi.INT)], - ) + ifdef='AF_PACKET') CConfig.ifreq = platform.Struct('struct ifreq', [('ifr_ifindex', rffi.INT), - ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))]) + ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))], + ifdef='AF_PACKET') if _WIN32: CConfig.WSAEVENT = platform.SimpleType('WSAEVENT', rffi.VOIDP) @@ -532,8 +533,9 @@ socketpair_t = rffi.CArray(socketfd_type) socketpair = external('socketpair', [rffi.INT, rffi.INT, rffi.INT, lltype.Ptr(socketpair_t)], rffi.INT) - ioctl = external('ioctl', [socketfd_type, rffi.INT, lltype.Ptr(ifreq)], - rffi.INT) + if ifreq is not None: + ioctl = external('ioctl', [socketfd_type, rffi.INT, lltype.Ptr(ifreq)], + rffi.INT) if _WIN32: ioctlsocket = external('ioctlsocket', From arigo at codespeak.net Mon Oct 11 18:07:38 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 18:07:38 +0200 (CEST) Subject: [pypy-svn] r77803 - pypy/build/bot2/pypybuildbot Message-ID: <20101011160738.AC0FD282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 18:07:36 2010 New Revision: 77803 Modified: pypy/build/bot2/pypybuildbot/master.py Log: Make a separate category {jit64}, as there are already {own64} and {applevel64}. Modified: pypy/build/bot2/pypybuildbot/master.py ============================================================================== --- pypy/build/bot2/pypybuildbot/master.py (original) +++ pypy/build/bot2/pypybuildbot/master.py Mon Oct 11 18:07:36 2010 @@ -286,7 +286,7 @@ 'slavenames': ['tannit64'], 'builddir': JITLINUX64, 'factory': pypyJITTranslatedTestFactory64, - 'category': 'jit', + 'category': 'jit64', }, {"name" : JITMACOSX32, "slavenames": ["minime"], From afa at codespeak.net Mon Oct 11 18:42:03 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 18:42:03 +0200 (CEST) Subject: [pypy-svn] r77804 - in pypy/branch/fast-forward/pypy/module/_multiprocessing: . test Message-ID: <20101011164203.268A1282BF3@codespeak.net> Author: afa Date: Mon Oct 11 18:42:01 2010 New Revision: 77804 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py Log: implement SemLock._rebuild() Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Mon Oct 11 18:42:01 2010 @@ -15,6 +15,7 @@ if sys.platform == 'win32': from pypy.rlib import rwin32 from pypy.interpreter.error import wrap_windowserror + from pypy.module._multiprocessing.interp_win32 import handle_w SEM_VALUE_MAX = sys.maxint @@ -183,6 +184,9 @@ finally: lltype.free(now, flavor='raw') + def handle_w(space, w_handle): + return rffi.cast(SEM_T, space.uint_w(w_handle)) + class GlobalState: def init(self): pass @@ -424,6 +428,12 @@ self.count -= 1 + @unwrap_spec('self', ObjSpace, W_Root, int, int) + def rebuild(self, space, w_handle, kind, maxvalue): + self.handle = handle_w(space, w_handle) + self.kind = kind + self.maxvalue = maxvalue + @unwrap_spec(ObjSpace, W_Root, int, int, int) def descr_new(space, w_subtype, kind, value, maxvalue): if kind != RECURSIVE_MUTEX and kind != SEMAPHORE: @@ -450,5 +460,6 @@ _is_mine = interp2app(W_SemLock.is_mine), acquire = interp2app(W_SemLock.acquire), release = interp2app(W_SemLock.release), + _rebuild = interp2app(W_SemLock.rebuild), SEM_VALUE_MAX=SEM_VALUE_MAX, ) Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py Mon Oct 11 18:42:01 2010 @@ -6,7 +6,7 @@ from pypy.rpython.lltypesystem import rffi, lltype from pypy.translator.tool.cbuild import ExternalCompilationInfo from pypy.rpython.tool import rffi_platform -from pypy.module._multiprocessing import w_handle +from pypy.module._multiprocessing.interp_connection import w_handle CONSTANTS = """ PIPE_ACCESS_INBOUND PIPE_ACCESS_DUPLEX Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py Mon Oct 11 18:42:01 2010 @@ -36,3 +36,12 @@ assert sem.acquire() assert not sem.acquire(timeout=0.1) + + def test_semaphore_rebuild(self): + from _multiprocessing import SemLock + kind = self.SEMAPHORE + value = 1 + maxvalue = 1 + sem = SemLock(kind, value, maxvalue) + + sem._rebuild(sem.handle, kind, value) From arigo at codespeak.net Mon Oct 11 18:45:26 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Mon, 11 Oct 2010 18:45:26 +0200 (CEST) Subject: [pypy-svn] r77805 - pypy/trunk/pypy/module/gc Message-ID: <20101011164526.DE68C282BF3@codespeak.net> Author: arigo Date: Mon Oct 11 18:45:24 2010 New Revision: 77805 Modified: pypy/trunk/pypy/module/gc/referents.py Log: Ignore instances of exactly the class W_Root. Seems that we get one on 64-bit translations, for some reason. See crashes of module/gc/test e.g. on {applevel64}. Modified: pypy/trunk/pypy/module/gc/referents.py ============================================================================== --- pypy/trunk/pypy/module/gc/referents.py (original) +++ pypy/trunk/pypy/module/gc/referents.py Mon Oct 11 18:45:24 2010 @@ -15,6 +15,10 @@ def try_cast_gcref_to_w_root(gcref): w_obj = rgc.try_cast_gcref_to_instance(W_Root, gcref) + # Ignore instances of exactly the class W_Root. Seems that + # we get one on 64-bit translations, for some reason. + if w_obj is not None and type(w_obj) is W_Root: + w_obj = None if not we_are_translated() and not hasattr(w_obj, 'typedef'): w_obj = None return w_obj From afa at codespeak.net Mon Oct 11 19:16:20 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 19:16:20 +0200 (CEST) Subject: [pypy-svn] r77806 - in pypy/branch/fast-forward/pypy/module/_multiprocessing: . test Message-ID: <20101011171620.ED86B282B9D@codespeak.net> Author: afa Date: Mon Oct 11 19:16:18 2010 New Revision: 77806 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py Log: Arg, SemLock._rebuild is a classmethod. Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Mon Oct 11 19:16:18 2010 @@ -428,11 +428,11 @@ self.count -= 1 - @unwrap_spec('self', ObjSpace, W_Root, int, int) - def rebuild(self, space, w_handle, kind, maxvalue): - self.handle = handle_w(space, w_handle) - self.kind = kind - self.maxvalue = maxvalue + @unwrap_spec(ObjSpace, W_Root, W_Root, int, int) + def rebuild(space, w_cls, w_handle, kind, maxvalue): + self = space.allocate_instance(W_SemLock, w_cls) + self.__init__(handle_w(space, w_handle), kind, maxvalue) + return space.wrap(self) @unwrap_spec(ObjSpace, W_Root, int, int, int) def descr_new(space, w_subtype, kind, value, maxvalue): @@ -460,6 +460,6 @@ _is_mine = interp2app(W_SemLock.is_mine), acquire = interp2app(W_SemLock.acquire), release = interp2app(W_SemLock.release), - _rebuild = interp2app(W_SemLock.rebuild), + _rebuild = interp2app(W_SemLock.rebuild.im_func, as_classmethod=True), SEM_VALUE_MAX=SEM_VALUE_MAX, ) Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py Mon Oct 11 19:16:18 2010 @@ -44,4 +44,5 @@ maxvalue = 1 sem = SemLock(kind, value, maxvalue) - sem._rebuild(sem.handle, kind, value) + sem2 = SemLock._rebuild(sem.handle, kind, value) + assert sem.handle == sem2.handle From afa at codespeak.net Mon Oct 11 19:46:25 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 19:46:25 +0200 (CEST) Subject: [pypy-svn] r77807 - pypy/branch/fast-forward/lib_pypy Message-ID: <20101011174625.6BB75282B9D@codespeak.net> Author: afa Date: Mon Oct 11 19:46:23 2010 New Revision: 77807 Modified: pypy/branch/fast-forward/lib_pypy/_subprocess.py Log: Fix _subprocess, for multiprocessing which directly calls this internal package :-( Modified: pypy/branch/fast-forward/lib_pypy/_subprocess.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_subprocess.py (original) +++ pypy/branch/fast-forward/lib_pypy/_subprocess.py Mon Oct 11 19:46:23 2010 @@ -137,11 +137,11 @@ si.dwFlags = startup_info.dwFlags si.wShowWindow = startup_info.wShowWindow if startup_info.hStdInput: - si.hStdInput = startup_info.hStdInput.handle + si.hStdInput = int(startup_info.hStdInput) if startup_info.hStdOutput: - si.hStdOutput = startup_info.hStdOutput.handle + si.hStdOutput = int(startup_info.hStdOutput) if startup_info.hStdError: - si.hStdError = startup_info.hStdError.handle + si.hStdError = int(startup_info.hStdError) pi = _PROCESS_INFORMATION() @@ -167,7 +167,7 @@ CREATE_NEW_PROCESS_GROUP = 0x200 def WaitForSingleObject(handle, milliseconds): - res = _WaitForSingleObject(handle.handle, milliseconds) + res = _WaitForSingleObject(int(handle), milliseconds) if res < 0: raise WindowsError("Error") @@ -179,7 +179,7 @@ def GetExitCodeProcess(handle): code = _c_int() - res = _GetExitCodeProcess(handle.handle, _byref(code)) + res = _GetExitCodeProcess(int(handle), _byref(code)) if not res: raise WindowsError("Error") From afa at codespeak.net Mon Oct 11 23:39:32 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 23:39:32 +0200 (CEST) Subject: [pypy-svn] r77809 - pypy/branch/fast-forward/pypy/module/_multiprocessing/test Message-ID: <20101011213932.9F6E9282BF3@codespeak.net> Author: afa Date: Mon Oct 11 23:39:30 2010 New Revision: 77809 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py Log: Refactor: one test for two implementations. Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py Mon Oct 11 23:39:30 2010 @@ -7,43 +7,48 @@ from pypy.module._multiprocessing import interp_connection from pypy.module._multiprocessing import interp_semaphore -class AppTestConnection: - def setup_class(cls): - space = gettestobjspace(usemodules=('_multiprocessing', 'thread')) - cls.space = space - if sys.platform == "win32": - # stubs for some modules, - # just for multiprocessing to import correctly. - w_modules = space.sys.get('modules') - space.setitem(w_modules, space.wrap('msvcrt'), space.sys) - space.setitem(w_modules, space.wrap('_subprocess'), space.sys) - - # import multiprocessing once - space.appexec([], """(): import multiprocessing""") - - def test_winpipe_connection(self): - import sys - if sys.platform != "win32": - skip("win32 only") - - import multiprocessing - rhandle, whandle = multiprocessing.Pipe() +class BaseConnectionTest(object): + def test_connection(self): + rhandle, whandle = self.make_pair() obj = [1, 2.0, "hello"] whandle.send(obj) obj2 = rhandle.recv() assert obj == obj2 - def test_ospipe_connection(self): - import _multiprocessing - import os - fd1, fd2 = os.pipe() - rhandle = _multiprocessing.Connection(fd1, writable=False) - whandle = _multiprocessing.Connection(fd2, readable=False) +class AppTestWinpipeConnection(BaseConnectionTest): + def setup_class(cls): + if sys.platform != "win32": + py.test.skip("win32 only") - obj = [1, 2.0, "hello"] - whandle.send(obj) - obj2 = rhandle.recv() - assert obj == obj2 + space = gettestobjspace(usemodules=('_multiprocessing', 'thread')) + cls.space = space + # stubs for some modules, + # just for multiprocessing to import correctly on Windows + w_modules = space.sys.get('modules') + space.setitem(w_modules, space.wrap('msvcrt'), space.sys) + space.setitem(w_modules, space.wrap('_subprocess'), space.sys) + + cls.w_make_pair = space.appexec([], """(): + import multiprocessing + def make_pair(): + rhandle, whandle = multiprocessing.Pipe() + return rhandle, whandle + return make_pair + """) +class AppTestSocketConnection(BaseConnectionTest): + def setup_class(cls): + space = gettestobjspace(usemodules=('_multiprocessing', 'thread')) + cls.space = space + cls.w_make_pair = space.appexec([], """(): + import _multiprocessing + import os + def make_pair(): + fd1, fd2 = os.pipe() + rhandle = _multiprocessing.Connection(fd1, writable=False) + whandle = _multiprocessing.Connection(fd2, readable=False) + return rhandle, whandle + return make_pair + """) From afa at codespeak.net Mon Oct 11 23:53:45 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Mon, 11 Oct 2010 23:53:45 +0200 (CEST) Subject: [pypy-svn] r77810 - pypy/branch/fast-forward/pypy/module/_multiprocessing Message-ID: <20101011215345.18421282BF3@codespeak.net> Author: afa Date: Mon Oct 11 23:53:44 2010 New Revision: 77810 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Log: Fix compilation on tannit Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Mon Oct 11 23:53:44 2010 @@ -94,6 +94,7 @@ eci = ExternalCompilationInfo( includes = ['sys/time.h', + 'limits.h', 'semaphore.h'], libraries = ['rt'], ) From afa at codespeak.net Tue Oct 12 01:40:28 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 12 Oct 2010 01:40:28 +0200 (CEST) Subject: [pypy-svn] r77811 - in pypy/branch/fast-forward/pypy/module/imp: . test Message-ID: <20101011234028.3832C282BF3@codespeak.net> Author: afa Date: Tue Oct 12 01:40:25 2010 New Revision: 77811 Modified: pypy/branch/fast-forward/pypy/module/imp/importing.py pypy/branch/fast-forward/pypy/module/imp/test/test_import.py Log: __import__ accepts anything for its "globals" and "locals" :-( Seen in lib2to3:: pkg = __import__(fixer_pkg, [], [], ["*"]) Modified: pypy/branch/fast-forward/pypy/module/imp/importing.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/imp/importing.py (original) +++ pypy/branch/fast-forward/pypy/module/imp/importing.py Tue Oct 12 01:40:25 2010 @@ -107,7 +107,7 @@ rel_modulename = None if (level != 0 and w_globals is not None and - not space.is_w(w_globals, space.w_None)): + space.isinstance_w(w_globals, space.w_dict)): ctxt_w_name = space.finditem(w_globals, w('__name__')) ctxt_w_path = space.finditem(w_globals, w('__path__')) if ctxt_w_name is not None: Modified: pypy/branch/fast-forward/pypy/module/imp/test/test_import.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/imp/test/test_import.py (original) +++ pypy/branch/fast-forward/pypy/module/imp/test/test_import.py Tue Oct 12 01:40:25 2010 @@ -285,6 +285,8 @@ assert sys == m n = __import__('sys', None, None, ['']) assert sys == n + o = __import__('sys', [], [], ['']) # CPython accepts this + assert sys == o def test_import_relative_back_to_absolute2(self): from pkg import abs_x_y From afa at codespeak.net Tue Oct 12 01:41:39 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 12 Oct 2010 01:41:39 +0200 (CEST) Subject: [pypy-svn] r77812 - in pypy/branch/fast-forward/pypy/module/_multiprocessing: . test Message-ID: <20101011234139.4B6B9282BF3@codespeak.net> Author: afa Date: Tue Oct 12 01:41:37 2010 New Revision: 77812 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py Log: multiprocessing: implement Connection.poll() Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py Tue Oct 12 01:41:37 2010 @@ -187,6 +187,16 @@ return w_unpickled + @unwrap_spec('self', ObjSpace, W_Root) + def poll(self, space, w_timeout=0.0): + if space.is_w(w_timeout, space.w_None): + timeout = -1.0 # block forever + else: + timeout = space.float_w(w_timeout) + if timeout < 0.0: + timeout = 0.0 + return space.newbool(self.do_poll(space, timeout)) + W_BaseConnection.typedef = TypeDef( 'BaseConnection', closed = GetSetProperty(W_BaseConnection.closed_get), @@ -198,7 +208,7 @@ recv_bytes_into = interp2app(W_BaseConnection.recv_bytes_into), send = interp2app(W_BaseConnection.send), recv = interp2app(W_BaseConnection.recv), - ## poll = interp2app(W_BaseConnection.poll), + poll = interp2app(W_BaseConnection.poll), close = interp2app(W_BaseConnection.close), ) @@ -290,6 +300,21 @@ remaining -= count buffer = rffi.ptradd(buffer, count) + def do_poll(self, space): + # XXX Won't work on Windows + + from pypy.rlib import rpoll + # just verify the fd + if self.fd < 0 or self.fd > FD_SETSIZE: + raise OperationError(space.w_IOError, space.wrap( + "handle out of range in select()")) + + r, w, e = rpoll.select([self.fd], [], [], timeout) + if r: + return True + else: + return False + W_FileConnection.typedef = TypeDef( 'Connection', W_BaseConnection.typedef, __new__ = interp2app(W_FileConnection.descr_new_file.im_func), @@ -406,6 +431,63 @@ finally: lltype.free(read_ptr, flavor='raw') + def do_poll(self, space, timeout): + from pypy.module._multiprocessing.interp_win32 import ( + _PeekNamedPipe) + from pypy.rlib import rwin32 + bytes_ptr = lltype.malloc(rffi.CArrayPtr(rwin32.DWORD).TO, 1, + flavor='raw') + try: + if not _PeekNamedPipe(self.handle, rffi.NULL, 0, + lltype.nullptr(rwin32.LPDWORD.TO), + bytes_ptr, + lltype.nullptr(rwin32.LPDWORD.TO)): + raise wrap_windowserror(space, rwin32.lastWindowsError()) + bytes = bytes_ptr[0] + finally: + lltype.free(bytes_ptr, flavor='raw') + + if timeout == 0.0: + return bytes > 0 + + block = timeout < 0 + if not block: + # XXX does not check for overflow + deadline = _GetTickCount() + int(1000 * timeout + 0.5) + + _Sleep(0) + + delay = 1 + while True: + try: + if not _PeekNamedPipe(self.handle, rffi.NULL, 0, + lltype.nullptr(rwin32.LPDWORD.TO), + bytes_ptr, + lltype.nullptr(rwin32.LPDWORD.TO)): + raise wrap_windowserror(space, rwin32.lastWindowsError()) + bytes = bytes_ptr[0] + finally: + lltype.free(bytes_ptr, flavor='raw') + + if bytes > 0: + return True + + if not block: + diff = deadline - _GetTickCount() + if diff < 0: + return False + if delay > difference: + delay = difference + else: + delay += 1 + + if delay >= 20: + delay = 20 + Sleep(delay) + + # check for signals + # PyErr_CheckSignals() + if sys.platform == 'win32': W_PipeConnection.typedef = TypeDef( 'PipeConnection', W_BaseConnection.typedef, Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py Tue Oct 12 01:41:37 2010 @@ -16,6 +16,15 @@ obj2 = rhandle.recv() assert obj == obj2 + def test_poll(self): + rhandle, whandle = self.make_pair() + + assert rhandle.poll(0) == False + whandle.send(1) + assert rhandle.poll(0) == True + assert rhandle.recv() == 1 + assert rhandle.poll(0) == False + class AppTestWinpipeConnection(BaseConnectionTest): def setup_class(cls): if sys.platform != "win32": @@ -52,3 +61,7 @@ return rhandle, whandle return make_pair """) + + if sys.platform == "win32": + def test_poll(self): + skip("poll does not accept file handles on Windows") From afa at codespeak.net Tue Oct 12 09:35:22 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 12 Oct 2010 09:35:22 +0200 (CEST) Subject: [pypy-svn] r77813 - pypy/branch/fast-forward/pypy/module/_multiprocessing Message-ID: <20101012073522.232FE282BF5@codespeak.net> Author: afa Date: Tue Oct 12 09:35:20 2010 New Revision: 77813 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py Log: Fix translation error Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py Tue Oct 12 09:35:20 2010 @@ -435,6 +435,7 @@ from pypy.module._multiprocessing.interp_win32 import ( _PeekNamedPipe) from pypy.rlib import rwin32 + from pypy.interpreter.error import wrap_windowserror bytes_ptr = lltype.malloc(rffi.CArrayPtr(rwin32.DWORD).TO, 1, flavor='raw') try: From antocuni at codespeak.net Tue Oct 12 11:25:24 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 12 Oct 2010 11:25:24 +0200 (CEST) Subject: [pypy-svn] r77815 - in pypy/branch/jitffi/pypy/jit: codewriter metainterp/optimizeopt metainterp/test Message-ID: <20101012092524.42988282BF7@codespeak.net> Author: antocuni Date: Tue Oct 12 11:25:21 2010 New Revision: 77815 Modified: pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py Log: use the new "official" way to detect oopspec, i.e. by using Effectinfo.OS_*. This should be make the code translatable Modified: pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py Tue Oct 12 11:25:21 2010 @@ -31,6 +31,9 @@ OS_STREQ_NONNULL_CHAR = 12 # s1 == char (assert s1!=NULL) OS_STREQ_CHECKNULL_CHAR = 13 # s1!=NULL and s1==char OS_STREQ_LENGTHOK = 14 # s1 == s2 (assert len(s1)==len(s2)) + OS_LIBFFI_PREPARE = 15 + OS_LIBFFI_PUSH_ARG = 16 + OS_LIBFFI_CALL = 17 def __new__(cls, readonly_descrs_fields, write_descrs_fields, write_descrs_arrays, Modified: pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py (original) +++ pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py Tue Oct 12 11:25:21 2010 @@ -318,6 +318,8 @@ prepare = self._handle_stroruni_call elif oopspec_name.startswith('virtual_ref'): prepare = self._handle_virtual_ref_call + elif oopspec_name.startswith('libffi_'): + prepare = self._handle_libffi_call else: prepare = self.prepare_builtin_call try: @@ -1121,6 +1123,20 @@ vrefinfo.JIT_VIRTUAL_REF) return SpaceOperation(oopspec_name, list(args), op.result) + # ----------- + # rlib.libffi + + def _handle_libffi_call(self, op, oopspec_name, args): + if oopspec_name == 'libffi_prepare_call': + oopspecindex = EffectInfo.OS_LIBFFI_PREPARE + elif oopspec_name.startswith('libffi_push_'): + oopspecindex = EffectInfo.OS_LIBFFI_PUSH_ARG + elif oopspec_name.startswith('libffi_call_'): + oopspecindex = EffectInfo.OS_LIBFFI_CALL + else: + assert False, 'unsupported oopspec: %s' % oopspec_name + return self._handle_oopspec_call(op, args, oopspecindex) + def rewrite_op_jit_force_virtual(self, op): return self._do_builtin_call(op) Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py Tue Oct 12 11:25:21 2010 @@ -1,6 +1,7 @@ from pypy.rpython.annlowlevel import cast_base_ptr_to_instance from pypy.rlib.objectmodel import we_are_translated from pypy.rlib.libffi import Func +from pypy.jit.codewriter.effectinfo import EffectInfo from pypy.jit.metainterp.resoperation import rop, ResOperation from pypy.jit.metainterp.optimizeutil import _findall from pypy.jit.metainterp.optimizeopt.optimizer import Optimization @@ -60,35 +61,25 @@ def __init__(self): self.func_infos = {} - def get_oopspec(self, funcval): - # XXX: not RPython at all, just a hack while waiting to have an - # "official" way to know if and which oopspec we are calling - funcname = str(funcval.box) - if '_libffi_prepare_call' in funcname: - return 'prepare_call' - elif '_libffi_push_' in funcname: - return 'push_arg' - elif '_libffi_call' in funcname: - return 'call' - return None - def optimize_CALL(self, op): if we_are_translated(): self.emit_operation(op) return # - targetval = self.getvalue(op.getarg(0)) - oopspec = self.get_oopspec(targetval) - if oopspec not in ('prepare_call', 'push_arg', 'call'): + effectinfo = op.getdescr().get_extra_info() + oopspec = effectinfo.oopspecindex + if oopspec not in (EffectInfo.OS_LIBFFI_PREPARE, + EffectInfo.OS_LIBFFI_PUSH_ARG, + EffectInfo.OS_LIBFFI_CALL): self.emit_operation(op) # normal case return # try: - if oopspec == 'prepare_call': + if oopspec == EffectInfo.OS_LIBFFI_PREPARE: self.do_prepare_call(op) - elif oopspec == 'push_arg': + elif oopspec == EffectInfo.OS_LIBFFI_PUSH_ARG: self.do_push_arg(op) - elif oopspec == 'call': + elif oopspec == EffectInfo.OS_LIBFFI_CALL: op = self.do_call(op) self.emit_operation(op) except NonConstantFuncVal: Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py Tue Oct 12 11:25:21 2010 @@ -4503,6 +4503,7 @@ from pypy.rpython.lltypesystem import llmemory from pypy.rlib.libffi import Func, types from pypy.jit.metainterp.history import AbstractDescr +from pypy.jit.codewriter.effectinfo import EffectInfo class MyCallDescr(AbstractDescr): """ @@ -4534,22 +4535,30 @@ class namespace: cpu = LLtypeMixin.cpu - plaincalldescr = LLtypeMixin.plaincalldescr + FUNC = LLtypeMixin.FUNC int_float__int = MyCallDescr('if', 'i') funcptr = FakeLLObject() func = FakeLLObject(_fake_class=Func, argtypes=[types.sint, types.double], restype=types.sint) - + # + def calldescr(cpu, FUNC, oopspecindex): + einfo = EffectInfo([], [], [], oopspecindex=oopspecindex) + return cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, einfo) + # + libffi_prepare = calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_PREPARE) + libffi_push_arg = calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_PUSH_ARG) + libffi_call = calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_CALL) + namespace = namespace.__dict__ def test_ffi_call_opt(self): ops = """ [i0, f1] - call("_libffi_prepare_call", ConstPtr(func), descr=plaincalldescr) - call("_libffi_push_arg_Signed", ConstPtr(func), i0, descr=plaincalldescr) - call("_libffi_push_arg_Float", ConstPtr(func), f1, descr=plaincalldescr) - i3 = call("_libffi_call", ConstPtr(func), 12345, descr=plaincalldescr) + call(0, ConstPtr(func), descr=libffi_prepare) + call(0, ConstPtr(func), i0, descr=libffi_push_arg) + call(0, ConstPtr(func), f1, descr=libffi_push_arg) + i3 = call(0, ConstPtr(func), 12345, descr=libffi_call) jump(i3, f1) """ expected = """ @@ -4562,21 +4571,13 @@ def test_ffi_call_nonconst(self): ops = """ [i0, f1, p2] - call("_libffi_prepare_call", p2, descr=plaincalldescr) - call("_libffi_push_arg_Signed", p2, i0, descr=plaincalldescr) - call("_libffi_push_arg_Float", p2, f1, descr=plaincalldescr) - i3 = call("_libffi_call", p2, 1, descr=plaincalldescr) - jump(i3, f1, p2) - """ - expected = """ - [i0, f1, p2] - call("_libffi_prepare_call", p2, descr=plaincalldescr) - call("_libffi_push_arg_Signed", p2, i0, descr=plaincalldescr) - call("_libffi_push_arg_Float", p2, f1, descr=plaincalldescr) - i3 = call("_libffi_call", p2, 1, descr=plaincalldescr) + call(0, p2, descr=libffi_prepare) + call(0, p2, i0, descr=libffi_push_arg) + call(0, p2, f1, descr=libffi_push_arg) + i3 = call(0, p2, 12345, descr=libffi_call) jump(i3, f1, p2) - """ + expected = ops loop = self.optimize_loop(ops, 'Not, Not, Not', expected) From antocuni at codespeak.net Tue Oct 12 11:27:46 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 12 Oct 2010 11:27:46 +0200 (CEST) Subject: [pypy-svn] r77816 - pypy/branch/jitffi/pypy/config Message-ID: <20101012092746.CCFE0282BF7@codespeak.net> Author: antocuni Date: Tue Oct 12 11:27:43 2010 New Revision: 77816 Modified: pypy/branch/jitffi/pypy/config/pypyoption.py Log: enable _ffi by default on this branch Modified: pypy/branch/jitffi/pypy/config/pypyoption.py ============================================================================== --- pypy/branch/jitffi/pypy/config/pypyoption.py (original) +++ pypy/branch/jitffi/pypy/config/pypyoption.py Tue Oct 12 11:27:43 2010 @@ -30,7 +30,7 @@ "rctime" , "select", "zipimport", "_lsprof", "crypt", "signal", "_rawffi", "termios", "zlib", "struct", "md5", "sha", "bz2", "_minimal_curses", "cStringIO", - "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array"] + "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array", "_ffi"] )) working_oo_modules = default_modules.copy() From afa at codespeak.net Tue Oct 12 11:27:47 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 12 Oct 2010 11:27:47 +0200 (CEST) Subject: [pypy-svn] r77817 - in pypy/branch/fast-forward/pypy: module/_multiprocessing module/_multiprocessing/test rlib Message-ID: <20101012092747.32746282BF8@codespeak.net> Author: afa Date: Tue Oct 12 11:27:43 2010 New Revision: 77817 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py pypy/branch/fast-forward/pypy/rlib/rpoll.py Log: Improve tests, and fix. Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_connection.py Tue Oct 12 11:27:43 2010 @@ -300,12 +300,12 @@ remaining -= count buffer = rffi.ptradd(buffer, count) - def do_poll(self, space): + def do_poll(self, space, timeout): # XXX Won't work on Windows from pypy.rlib import rpoll # just verify the fd - if self.fd < 0 or self.fd > FD_SETSIZE: + if self.fd < 0 or self.fd > rpoll.FD_SETSIZE: raise OperationError(space.w_IOError, space.wrap( "handle out of range in select()")) @@ -433,7 +433,7 @@ def do_poll(self, space, timeout): from pypy.module._multiprocessing.interp_win32 import ( - _PeekNamedPipe) + _PeekNamedPipe, _GetTickCount, _Sleep) from pypy.rlib import rwin32 from pypy.interpreter.error import wrap_windowserror bytes_ptr = lltype.malloc(rffi.CArrayPtr(rwin32.DWORD).TO, 1, @@ -455,11 +455,15 @@ if not block: # XXX does not check for overflow deadline = _GetTickCount() + int(1000 * timeout + 0.5) + else: + deadline = 0 _Sleep(0) delay = 1 while True: + bytes_ptr = lltype.malloc(rffi.CArrayPtr(rwin32.DWORD).TO, 1, + flavor='raw') try: if not _PeekNamedPipe(self.handle, rffi.NULL, 0, lltype.nullptr(rwin32.LPDWORD.TO), @@ -474,17 +478,18 @@ return True if not block: - diff = deadline - _GetTickCount() - if diff < 0: + now = _GetTickCount() + if now > deadline: return False - if delay > difference: - delay = difference + diff = deadline - now + if delay > diff: + delay = diff else: delay += 1 if delay >= 20: delay = 20 - Sleep(delay) + _Sleep(delay) # check for signals # PyErr_CheckSignals() Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Tue Oct 12 11:27:43 2010 @@ -15,7 +15,8 @@ if sys.platform == 'win32': from pypy.rlib import rwin32 from pypy.interpreter.error import wrap_windowserror - from pypy.module._multiprocessing.interp_win32 import handle_w + from pypy.module._multiprocessing.interp_win32 import ( + handle_w, _GetTickCount) SEM_VALUE_MAX = sys.maxint @@ -25,8 +26,6 @@ _ReleaseSemaphore = rwin32.winexternal( 'ReleaseSemaphore', [rwin32.HANDLE, rffi.LONG, rffi.LONGP], rwin32.BOOL) - _GetTickCount = rwin32.winexternal( - 'GetTickCount', [], rwin32.DWORD) CtrlHandler_type = lltype.Ptr(lltype.FuncType([], rwin32.BOOL)) _CreateEvent = rwin32.winexternal( @@ -68,7 +67,7 @@ def ProcessingCtrlHandler(): _SetEvent(globalState.sigint_event) - return False + return 0 class GlobalState: def __init__(self): Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py Tue Oct 12 11:27:43 2010 @@ -82,6 +82,12 @@ _ExitProcess = rwin32.winexternal( 'ExitProcess', [rffi.UINT], lltype.Void) +_GetTickCount = rwin32.winexternal( + 'GetTickCount', [], rwin32.DWORD) + +_Sleep = rwin32.winexternal( + 'Sleep', [rwin32.DWORD], lltype.Void) + def CloseHandle(space, w_handle): handle = handle_w(space, w_handle) if not rwin32.CloseHandle(handle): @@ -90,6 +96,9 @@ def GetLastError(space): return space.wrap(rwin32.GetLastError()) +# __________________________________________________________ +# functions for the "win32" namespace + @unwrap_spec(ObjSpace, str, r_uint, r_uint, r_uint, r_uint, r_uint, r_uint, W_Root) def CreateNamedPipe(space, name, openmode, pipemode, maxinstances, outputsize, inputsize, timeout, w_security): Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_connection.py Tue Oct 12 11:27:43 2010 @@ -19,11 +19,13 @@ def test_poll(self): rhandle, whandle = self.make_pair() - assert rhandle.poll(0) == False + assert rhandle.poll() == False + assert rhandle.poll(1) == False whandle.send(1) - assert rhandle.poll(0) == True + assert rhandle.poll() == True + assert rhandle.poll(None) == True assert rhandle.recv() == 1 - assert rhandle.poll(0) == False + assert rhandle.poll() == False class AppTestWinpipeConnection(BaseConnectionTest): def setup_class(cls): @@ -64,4 +66,4 @@ if sys.platform == "win32": def test_poll(self): - skip("poll does not accept file handles on Windows") + skip("poll() does not accept file handles on Windows") Modified: pypy/branch/fast-forward/pypy/rlib/rpoll.py ============================================================================== --- pypy/branch/fast-forward/pypy/rlib/rpoll.py (original) +++ pypy/branch/fast-forward/pypy/rlib/rpoll.py Tue Oct 12 11:27:43 2010 @@ -15,7 +15,8 @@ # events # eventnames = '''POLLIN POLLPRI POLLOUT POLLERR POLLHUP POLLNVAL - POLLRDNORM POLLRDBAND POLLWRNORM POLLWEBAND POLLMSG'''.split() + POLLRDNORM POLLRDBAND POLLWRNORM POLLWEBAND POLLMSG + FD_SETSIZE'''.split() eventnames = [name for name in eventnames if _c.constants.get(name) is not None] From fijall at gmail.com Tue Oct 12 12:07:50 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 12 Oct 2010 12:07:50 +0200 Subject: [pypy-svn] r77781 - pypy/trunk/pypy/jit/backend/x86/test In-Reply-To: <20101011123001.99D8A282BF3@codespeak.net> References: <20101011123001.99D8A282BF3@codespeak.net> Message-ID: Sorry about that, my fault. The reason was that code strings are not unique (obviously) On Mon, Oct 11, 2010 at 2:30 PM, wrote: > Author: arigo > Date: Mon Oct 11 14:29:59 2010 > New Revision: 77781 > > Modified: > ? pypy/trunk/pypy/jit/backend/x86/test/test_runner.py > Log: > Unsure, but it seems that the debugging info changed. ?Fixed the test. > > > Modified: pypy/trunk/pypy/jit/backend/x86/test/test_runner.py > ============================================================================== > --- pypy/trunk/pypy/jit/backend/x86/test/test_runner.py (original) > +++ pypy/trunk/pypy/jit/backend/x86/test/test_runner.py Mon Oct 11 14:29:59 2010 > @@ -506,8 +506,8 @@ > ? ? ? ? self.cpu.execute_token(ops.token) > ? ? ? ? # check debugging info > ? ? ? ? name, struct = self.cpu.assembler.loop_run_counters[0] > - ? ? ? ?assert name == 'xyz' > + ? ? ? ?assert name == 0 ? ? ? # 'xyz' > ? ? ? ? assert struct.i == 10 > ? ? ? ? self.cpu.finish_once() > ? ? ? ? lines = py.path.local(self.logfile + ".count").readlines() > - ? ? ? ?assert lines[0] == '10 ? ? ?xyz\n' > + ? ? ? ?assert lines[0] == '0:10\n' ?# '10 ? ? ?xyz\n' > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From antocuni at codespeak.net Tue Oct 12 12:45:05 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 12 Oct 2010 12:45:05 +0200 (CEST) Subject: [pypy-svn] r77818 - pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt Message-ID: <20101012104505.08B9A282BF6@codespeak.net> Author: antocuni Date: Tue Oct 12 12:45:03 2010 New Revision: 77818 Added: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/string.py (contents, props changed) Log: I don't know why, but it seems that this file was not checked in during the merge that happened in r77720 Added: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/string.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/string.py Tue Oct 12 12:45:03 2010 @@ -0,0 +1,536 @@ +from pypy.rpython.lltypesystem import lltype, rstr, llmemory +from pypy.rpython import annlowlevel +from pypy.jit.metainterp.history import Box, BoxInt, BoxPtr +from pypy.jit.metainterp.history import Const, ConstInt, ConstPtr +from pypy.jit.metainterp.history import get_const_ptr_for_string +from pypy.jit.metainterp.resoperation import rop, ResOperation +from pypy.jit.metainterp.optimizeopt import optimizer, virtualize +from pypy.jit.metainterp.optimizeopt.optimizer import CONST_0, CONST_1 +from pypy.jit.metainterp.optimizeopt.optimizer import llhelper +from pypy.jit.metainterp.optimizeutil import _findall +from pypy.jit.codewriter.effectinfo import EffectInfo, callinfo_for_oopspec +from pypy.jit.codewriter import heaptracker +from pypy.rlib.unroll import unrolling_iterable + + +class __extend__(optimizer.OptValue): + """New methods added to the base class OptValue for this file.""" + + def getstrlen(self, newoperations): + s = self.get_constant_string() + if s is not None: + return ConstInt(len(s)) + else: + if newoperations is None: + return None + self.ensure_nonnull() + box = self.force_box() + lengthbox = BoxInt() + newoperations.append(ResOperation(rop.STRLEN, [box], lengthbox)) + return lengthbox + + def get_constant_string(self): + if self.is_constant(): + s = self.box.getref(lltype.Ptr(rstr.STR)) + return annlowlevel.hlstr(s) + else: + return None + + def string_copy_parts(self, newoperations, targetbox, offsetbox): + # Copies the pointer-to-string 'self' into the target string + # given by 'targetbox', at the specified offset. Returns the offset + # at the end of the copy. + lengthbox = self.getstrlen(newoperations) + srcbox = self.force_box() + return copy_str_content(newoperations, srcbox, targetbox, + CONST_0, offsetbox, lengthbox) + + +class VAbstractStringValue(virtualize.AbstractVirtualValue): + _attrs_ = () + + def _really_force(self): + s = self.get_constant_string() + if s is not None: + c_s = get_const_ptr_for_string(s) + self.make_constant(c_s) + return + assert self.source_op is not None + self.box = box = self.source_op.result + newoperations = self.optimizer.newoperations + lengthbox = self.getstrlen(newoperations) + newoperations.append(ResOperation(rop.NEWSTR, [lengthbox], box)) + self.string_copy_parts(newoperations, box, CONST_0) + + +class VStringPlainValue(VAbstractStringValue): + """A string built with newstr(const).""" + _lengthbox = None # cache only + + def setup(self, size): + self._chars = [optimizer.CVAL_UNINITIALIZED_ZERO] * size + + def setup_slice(self, longerlist, start, stop): + assert 0 <= start <= stop <= len(longerlist) + self._chars = longerlist[start:stop] + + def getstrlen(self, _): + if self._lengthbox is None: + self._lengthbox = ConstInt(len(self._chars)) + return self._lengthbox + + def getitem(self, index): + return self._chars[index] + + def setitem(self, index, charvalue): + assert isinstance(charvalue, optimizer.OptValue) + self._chars[index] = charvalue + + def get_constant_string(self): + for c in self._chars: + if c is optimizer.CVAL_UNINITIALIZED_ZERO or not c.is_constant(): + return None + return ''.join([chr(c.box.getint()) for c in self._chars]) + + def string_copy_parts(self, newoperations, targetbox, offsetbox): + for i in range(len(self._chars)): + charbox = self._chars[i].force_box() + newoperations.append(ResOperation(rop.STRSETITEM, [targetbox, + offsetbox, + charbox], None)) + offsetbox = _int_add(newoperations, offsetbox, CONST_1) + return offsetbox + + def get_args_for_fail(self, modifier): + if self.box is None and not modifier.already_seen_virtual(self.keybox): + charboxes = [value.get_key_box() for value in self._chars] + modifier.register_virtual_fields(self.keybox, charboxes) + for value in self._chars: + value.get_args_for_fail(modifier) + + def _make_virtual(self, modifier): + return modifier.make_vstrplain() + + +class VStringConcatValue(VAbstractStringValue): + """The concatenation of two other strings.""" + + def setup(self, left, right, lengthbox): + self.left = left + self.right = right + self.lengthbox = lengthbox + + def getstrlen(self, _): + return self.lengthbox + + def get_constant_string(self): + s1 = self.left.get_constant_string() + if s1 is None: + return None + s2 = self.right.get_constant_string() + if s2 is None: + return None + return s1 + s2 + + def string_copy_parts(self, newoperations, targetbox, offsetbox): + offsetbox = self.left.string_copy_parts(newoperations, targetbox, + offsetbox) + offsetbox = self.right.string_copy_parts(newoperations, targetbox, + offsetbox) + return offsetbox + + def get_args_for_fail(self, modifier): + if self.box is None and not modifier.already_seen_virtual(self.keybox): + # we don't store the lengthvalue in guards, because the + # guard-failed code starts with a regular STR_CONCAT again + leftbox = self.left.get_key_box() + rightbox = self.right.get_key_box() + modifier.register_virtual_fields(self.keybox, [leftbox, rightbox]) + self.left.get_args_for_fail(modifier) + self.right.get_args_for_fail(modifier) + + def _make_virtual(self, modifier): + return modifier.make_vstrconcat() + + +class VStringSliceValue(VAbstractStringValue): + """A slice.""" + _attrs_ = ('vstr', 'vstart', 'vlength') + + def setup(self, vstr, vstart, vlength): + self.vstr = vstr + self.vstart = vstart + self.vlength = vlength + + def getstrlen(self, _): + return self.vlength.force_box() + + def get_constant_string(self): + if self.vstart.is_constant() and self.vlength.is_constant(): + s1 = self.vstr.get_constant_string() + if s1 is None: + return None + start = self.vstart.box.getint() + length = self.vlength.box.getint() + assert start >= 0 + assert length >= 0 + return s1[start : start + length] + return None + + def string_copy_parts(self, newoperations, targetbox, offsetbox): + lengthbox = self.getstrlen(newoperations) + return copy_str_content(newoperations, + self.vstr.force_box(), targetbox, + self.vstart.force_box(), offsetbox, + lengthbox) + + def get_args_for_fail(self, modifier): + if self.box is None and not modifier.already_seen_virtual(self.keybox): + boxes = [self.vstr.get_key_box(), + self.vstart.get_key_box(), + self.vlength.get_key_box()] + modifier.register_virtual_fields(self.keybox, boxes) + self.vstr.get_args_for_fail(modifier) + self.vstart.get_args_for_fail(modifier) + self.vlength.get_args_for_fail(modifier) + + def _make_virtual(self, modifier): + return modifier.make_vstrslice() + + +def copy_str_content(newoperations, srcbox, targetbox, + srcoffsetbox, offsetbox, lengthbox): + if isinstance(srcbox, ConstPtr) and isinstance(srcoffsetbox, Const): + M = 5 + else: + M = 2 + if isinstance(lengthbox, ConstInt) and lengthbox.value <= M: + # up to M characters are done "inline", i.e. with STRGETITEM/STRSETITEM + # instead of just a COPYSTRCONTENT. + for i in range(lengthbox.value): + charbox = _strgetitem(newoperations, srcbox, srcoffsetbox) + srcoffsetbox = _int_add(newoperations, srcoffsetbox, CONST_1) + newoperations.append(ResOperation(rop.STRSETITEM, [targetbox, + offsetbox, + charbox], None)) + offsetbox = _int_add(newoperations, offsetbox, CONST_1) + else: + nextoffsetbox = _int_add(newoperations, offsetbox, lengthbox) + op = ResOperation(rop.COPYSTRCONTENT, [srcbox, targetbox, + srcoffsetbox, offsetbox, + lengthbox], None) + newoperations.append(op) + offsetbox = nextoffsetbox + return offsetbox + +def _int_add(newoperations, box1, box2): + if isinstance(box1, ConstInt): + if box1.value == 0: + return box2 + if isinstance(box2, ConstInt): + return ConstInt(box1.value + box2.value) + elif isinstance(box2, ConstInt) and box2.value == 0: + return box1 + resbox = BoxInt() + newoperations.append(ResOperation(rop.INT_ADD, [box1, box2], resbox)) + return resbox + +def _int_sub(newoperations, box1, box2): + if isinstance(box2, ConstInt): + if box2.value == 0: + return box1 + if isinstance(box1, ConstInt): + return ConstInt(box1.value - box2.value) + resbox = BoxInt() + newoperations.append(ResOperation(rop.INT_SUB, [box1, box2], resbox)) + return resbox + +def _strgetitem(newoperations, strbox, indexbox): + if isinstance(strbox, ConstPtr) and isinstance(indexbox, ConstInt): + s = strbox.getref(lltype.Ptr(rstr.STR)) + return ConstInt(ord(s.chars[indexbox.getint()])) + resbox = BoxInt() + newoperations.append(ResOperation(rop.STRGETITEM, [strbox, indexbox], + resbox)) + return resbox + + +class OptString(optimizer.Optimization): + "Handling of strings and unicodes." + + def make_vstring_plain(self, box, source_op=None): + vvalue = VStringPlainValue(self.optimizer, box, source_op) + self.make_equal_to(box, vvalue) + return vvalue + + def make_vstring_concat(self, box, source_op=None): + vvalue = VStringConcatValue(self.optimizer, box, source_op) + self.make_equal_to(box, vvalue) + return vvalue + + def make_vstring_slice(self, box, source_op=None): + vvalue = VStringSliceValue(self.optimizer, box, source_op) + self.make_equal_to(box, vvalue) + return vvalue + + def optimize_CALL(self, op): + # dispatch based on 'oopspecindex' to a method that handles + # specifically the given oopspec call. For non-oopspec calls, + # oopspecindex is just zero. + effectinfo = op.getdescr().get_extra_info() + if effectinfo is not None: + oopspecindex = effectinfo.oopspecindex + for value, meth in opt_call_oopspec_ops: + if oopspecindex == value: + if meth(self, op): + return + self.emit_operation(op) + + def opt_call_oopspec_ARRAYCOPY(self, op): + source_value = self.getvalue(op.getarg(1)) + dest_value = self.getvalue(op.getarg(2)) + source_start_box = self.get_constant_box(op.getarg(3)) + dest_start_box = self.get_constant_box(op.getarg(4)) + length = self.get_constant_box(op.getarg(5)) + if (source_value.is_virtual() and source_start_box and dest_start_box + and length and dest_value.is_virtual()): + # XXX optimize the case where dest value is not virtual, + # but we still can avoid a mess + source_start = source_start_box.getint() + dest_start = dest_start_box.getint() + for index in range(length.getint()): + val = source_value.getitem(index + source_start) + dest_value.setitem(index + dest_start, val) + return True + if length and length.getint() == 0: + return True # 0-length arraycopy + return False + + def optimize_NEWSTR(self, op): + length_box = self.get_constant_box(op.getarg(0)) + if length_box: + # if the original 'op' did not have a ConstInt as argument, + # build a new one with the ConstInt argument + if not isinstance(op.getarg(0), ConstInt): + op = ResOperation(rop.NEWSTR, [length_box], op.result) + vvalue = self.make_vstring_plain(op.result, op) + vvalue.setup(length_box.getint()) + else: + self.getvalue(op.result).ensure_nonnull() + self.emit_operation(op) + + def optimize_STRSETITEM(self, op): + value = self.getvalue(op.getarg(0)) + if value.is_virtual() and isinstance(value, VStringPlainValue): + indexbox = self.get_constant_box(op.getarg(1)) + if indexbox is not None: + value.setitem(indexbox.getint(), self.getvalue(op.getarg(2))) + return + value.ensure_nonnull() + self.emit_operation(op) + + def optimize_STRGETITEM(self, op): + value = self.getvalue(op.getarg(0)) + vindex = self.getvalue(op.getarg(1)) + vresult = self.strgetitem(value, vindex) + self.make_equal_to(op.result, vresult) + + def strgetitem(self, value, vindex): + value.ensure_nonnull() + # + if value.is_virtual() and isinstance(value, VStringSliceValue): + fullindexbox = _int_add(self.optimizer.newoperations, + value.vstart.force_box(), + vindex.force_box()) + value = value.vstr + vindex = self.getvalue(fullindexbox) + # + if isinstance(value, VStringPlainValue): # even if no longer virtual + if vindex.is_constant(): + return value.getitem(vindex.box.getint()) + # + resbox = _strgetitem(self.optimizer.newoperations, + value.force_box(),vindex.force_box()) + return self.getvalue(resbox) + + def optimize_STRLEN(self, op): + value = self.getvalue(op.getarg(0)) + lengthbox = value.getstrlen(self.optimizer.newoperations) + self.make_equal_to(op.result, self.getvalue(lengthbox)) + + def opt_call_oopspec_STR_CONCAT(self, op): + vleft = self.getvalue(op.getarg(1)) + vright = self.getvalue(op.getarg(2)) + vleft.ensure_nonnull() + vright.ensure_nonnull() + newoperations = self.optimizer.newoperations + len1box = vleft.getstrlen(newoperations) + len2box = vright.getstrlen(newoperations) + lengthbox = _int_add(newoperations, len1box, len2box) + value = self.make_vstring_concat(op.result, op) + value.setup(vleft, vright, lengthbox) + return True + + def opt_call_oopspec_STR_SLICE(self, op): + newoperations = self.optimizer.newoperations + vstr = self.getvalue(op.getarg(1)) + vstart = self.getvalue(op.getarg(2)) + vstop = self.getvalue(op.getarg(3)) + # + if (isinstance(vstr, VStringPlainValue) and vstart.is_constant() + and vstop.is_constant()): + # slicing with constant bounds of a VStringPlainValue + value = self.make_vstring_plain(op.result, op) + value.setup_slice(vstr._chars, vstart.box.getint(), + vstop.box.getint()) + return True + # + vstr.ensure_nonnull() + lengthbox = _int_sub(newoperations, vstop.force_box(), + vstart.force_box()) + # + if isinstance(vstr, VStringSliceValue): + # double slicing s[i:j][k:l] + vintermediate = vstr + vstr = vintermediate.vstr + startbox = _int_add(newoperations, + vintermediate.vstart.force_box(), + vstart.force_box()) + vstart = self.getvalue(startbox) + # + value = self.make_vstring_slice(op.result, op) + value.setup(vstr, vstart, self.getvalue(lengthbox)) + return True + + def opt_call_oopspec_STR_EQUAL(self, op): + v1 = self.getvalue(op.getarg(1)) + v2 = self.getvalue(op.getarg(2)) + # + l1box = v1.getstrlen(None) + l2box = v2.getstrlen(None) + if (l1box is not None and l2box is not None and + isinstance(l1box, ConstInt) and + isinstance(l2box, ConstInt) and + l1box.value != l2box.value): + # statically known to have a different length + self.make_constant(op.result, CONST_0) + return True + # + if self.handle_str_equal_level1(v1, v2, op.result): + return True + if self.handle_str_equal_level1(v2, v1, op.result): + return True + if self.handle_str_equal_level2(v1, v2, op.result): + return True + if self.handle_str_equal_level2(v2, v1, op.result): + return True + # + if v1.is_nonnull() and v2.is_nonnull(): + if l1box is not None and l2box is not None and ( + l1box == l2box or (isinstance(l1box, ConstInt) and + isinstance(l2box, ConstInt) and + l1box.value == l2box.value)): + do = EffectInfo.OS_STREQ_LENGTHOK + else: + do = EffectInfo.OS_STREQ_NONNULL + self.generate_modified_call(do, [v1.force_box(), + v2.force_box()], op.result) + return True + return False + + def handle_str_equal_level1(self, v1, v2, resultbox): + l2box = v2.getstrlen(None) + if isinstance(l2box, ConstInt): + if l2box.value == 0: + lengthbox = v1.getstrlen(self.optimizer.newoperations) + seo = self.optimizer.send_extra_operation + seo(ResOperation(rop.INT_EQ, [lengthbox, CONST_0], resultbox)) + return True + if l2box.value == 1: + l1box = v1.getstrlen(None) + if isinstance(l1box, ConstInt) and l1box.value == 1: + # comparing two single chars + vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO) + vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO) + seo = self.optimizer.send_extra_operation + seo(ResOperation(rop.INT_EQ, [vchar1.force_box(), + vchar2.force_box()], + resultbox)) + return True + if isinstance(v1, VStringSliceValue): + vchar = self.strgetitem(v2, optimizer.CVAL_ZERO) + do = EffectInfo.OS_STREQ_SLICE_CHAR + self.generate_modified_call(do, [v1.vstr.force_box(), + v1.vstart.force_box(), + v1.vlength.force_box(), + vchar.force_box()], + resultbox) + return True + # + if v2.is_null(): + if v1.is_nonnull(): + self.make_constant(resultbox, CONST_0) + return True + if v1.is_null(): + self.make_constant(resultbox, CONST_1) + return True + op = ResOperation(rop.PTR_EQ, [v1.force_box(), + llhelper.CONST_NULL], + resultbox) + self.optimizer.newoperations.append(op) + return True + # + return False + + def handle_str_equal_level2(self, v1, v2, resultbox): + l2box = v2.getstrlen(None) + if isinstance(l2box, ConstInt): + if l2box.value == 1: + vchar = self.strgetitem(v2, optimizer.CVAL_ZERO) + if v1.is_nonnull(): + do = EffectInfo.OS_STREQ_NONNULL_CHAR + else: + do = EffectInfo.OS_STREQ_CHECKNULL_CHAR + self.generate_modified_call(do, [v1.force_box(), + vchar.force_box()], resultbox) + return True + # + if v1.is_virtual() and isinstance(v1, VStringSliceValue): + if v2.is_nonnull(): + do = EffectInfo.OS_STREQ_SLICE_NONNULL + else: + do = EffectInfo.OS_STREQ_SLICE_CHECKNULL + self.generate_modified_call(do, [v1.vstr.force_box(), + v1.vstart.force_box(), + v1.vlength.force_box(), + v2.force_box()], resultbox) + return True + return False + + def generate_modified_call(self, oopspecindex, args, result): + calldescr, func = callinfo_for_oopspec(oopspecindex) + op = ResOperation(rop.CALL, [ConstInt(func)] + args, result, + descr=calldescr) + self.optimizer.newoperations.append(op) + + def propagate_forward(self, op): + opnum = op.getopnum() + for value, func in optimize_ops: + if opnum == value: + func(self, op) + break + else: + self.emit_operation(op) + +optimize_ops = _findall(OptString, 'optimize_') + +def _findall_call_oopspec(): + prefix = 'opt_call_oopspec_' + result = [] + for name in dir(OptString): + if name.startswith(prefix): + value = getattr(EffectInfo, 'OS_' + name[len(prefix):]) + assert isinstance(value, int) and value != 0 + result.append((value, getattr(OptString, name))) + return unrolling_iterable(result) +opt_call_oopspec_ops = _findall_call_oopspec() From cfbolz at codespeak.net Tue Oct 12 12:47:31 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 12:47:31 +0200 (CEST) Subject: [pypy-svn] r77819 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012104731.361C3282BF6@codespeak.net> Author: cfbolz Date: Tue Oct 12 12:47:28 2010 New Revision: 77819 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: flesh out the benchmarking and related work sections a bit more. add some acknowlegdements Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 12:47:28 2010 @@ -88,7 +88,7 @@ \terms Languages, Performance, Experimentation -\keywords{XXX}% +\keywords{Tracing JIT, Partial Evaluation, Optimization} \section{Introduction} @@ -832,6 +832,7 @@ Section~\ref{sec:statics} deals with them too: the \texttt{new} that creates them and the field accesses are deferred, until the point where the object escapes. +XXX optimization particularly effective for chains of operations % subsection Analysis of the Algorithm (end) @@ -954,7 +955,6 @@ \caption{A Way to Enter the Loop From the Interpreter} \end{figure} -XXX optimization particularly effective for chains of operations %___________________________________________________________________________ @@ -1002,24 +1002,88 @@ \section{Evaluation} \label{sec:Evaluation} -Benchmarks from the Computer Language Benchmark Game are: fannkuch, nbody, -meteor-contest, spectral-norm. +To evaluate the effectiveness of our allocation removal algorithm, we look at +the effectiveness when used in the tracing JIT of PyPy's Python interpreter. The +benchmarks we used are small-to-medium Python programs, some synthetic +benchmarks, some real applications. + +Some of them are from the Computer Language Benchmark +Game\footnote{\texttt{http://shootout.alioth.debian.org/}}: \textbf{fannkuch}, +\textbf{nbody}, \textbf{meteor-contest}, \textbf{spectral-norm}. + +Furthermore there are the following benchmarks: +\begin{itemize} + \item \textbf{crypto\_pyaes}: AES implementation. + \item \textbf{django}: The templating engine of the Django web + framework\footnote{\texttt{http://www.djangoproject.com/}}. + \item \textbf{go}: A Monte-Carlo Go + AI\footnote{\texttt{http://shed-skin.blogspot.com/2009/07/ disco-elegant-python-go-player.html}}. + \item \textbf{html5lib}: HTML5 parser + \item \textbf{pyflate-fast}: BZ2 decoder + \item \textbf{raytrace-simple}: ray tracer + \item \textbf{richards}: The Richards benchmark \cite{XXX} + \item \textbf{spambayes}: A Bayesian spam filter\footnote{\texttt{http://spambayes.sourceforge.net/}}. + \item \textbf{telco}: A Python version of the Telco decimal + benchmark\footnote{\texttt{http://speleotrove.com/decimal/telco.html}}, + using a pure Python decimal floating point implementation. + \item \textbf{twisted\_names}: A DNS server benchmark using the Twisted networking + framework\footnote{\texttt{http://twistedmatrix.com/}}. +\end{itemize} + +We evaluate the allocation removal algorithm along two lines: First we want to +know how many allocations could be optimized away. On the other hand, we want +to know how much the run times of the benchmarks is improved. + +For the former we counted the occurring operations in all generated traces +before and after the optimization phase for all benchmarks. The results can be +seen in Figure~\ref{fig:numops}. The optimization removes as many as XXX and as +little as XXX percent of allocation operations in the benchmarks. All benchmarks +taken together, the optimization removes XXX percent of allocation operations. + +\begin{figure*} +\begin{tabular}{lrrrrrrrrrrrrrrrrrrrrrrr} + &\#loops &new & & &get & & &set & & &guard & & &rest & & & & & &total & &\\ +crypto\_pyaes &75 &1663 &446 &27\% &18842 &2127 &11\% &14552 &2020 &14\% &8952 &235 &3\% &53339 &7794 &15\% &44009 &4828 &11\% &97348 &12622 &13\%\\ +django &53 &453 &155 &34\% &11970 &1457 &12\% &8738 &991 &11\% &4093 &275 &7\% &33988 &4076 &12\% &25254 &2878 &11\% &59242 &6954 &12\%\\ +fannkuch &40 &161 &76 &47\% &574 &306 &53\% &278 &139 &50\% &1121 &202 &18\% &2581 &1685 &65\% &2134 &723 &34\% &4715 &2408 &51\%\\ +go &510 &5604 &1651 &29\% &112403 &12612 &11\% &88439 &11915 &13\% &51621 &3194 &6\% &300831 &30834 &10\% &258067 &29372 &11\% &558898 &60206 &11\%\\ +html5lib &416 &5939 &1793 &30\% &241402 &15863 &7\% &178723 &13960 &8\% &58486 &3037 &5\% &682654 &40341 &6\% &484550 &34653 &7\% &1167204 &74994 &6\%\\ +meteor-contest &56 &221 &141 &64\% &2326 &606 &26\% &2047 &645 &32\% &1040 &169 &16\% &7018 &2250 &32\% &5634 &1561 &28\% &12652 &3811 &30\%\\ +nbody &10 &84 &48 &57\% &287 &159 &55\% &121 &74 &61\% &411 &90 &22\% &986 &737 &75\% &903 &371 &41\% &1889 &1108 &59\%\\ +pyflate-fast &161 &1878 &723 &38\% &22815 &2592 &11\% &18415 &3690 &20\% &8291 &628 &8\% &64649 &11327 &18\% &51399 &7633 &15\% &116048 &18960 &16\%\\ +raytrace-simple &117 &1906 &359 &19\% &52995 &2202 &4\% &39518 &1823 &5\% &13451 &447 &3\% &140723 &6533 &5\% &107870 &4831 &4\% &248593 &11364 &5\%\\ +richards &79 &375 &191 &51\% &27507 &2095 &8\% &21217 &2004 &9\% &3972 &332 &8\% &76751 &6171 &8\% &53071 &4622 &9\% &129822 &10793 &8\%\\ +spambayes &267 &2413 &505 &21\% &53327 &3033 &6\% &40938 &2428 &6\% &19343 &718 &4\% &143575 &10322 &7\% &116021 &6684 &6\% &259596 &17006 &7\%\\ +spectral-norm &35 &266 &88 &33\% &3004 &398 &13\% &2544 &414 &16\% &1084 &78 &7\% &4803 &1184 &25\% &6898 &978 &14\% &11701 &2162 &18\%\\ +telco &42 &660 &70 &11\% &20716 &424 &2\% &14444 &608 &4\% &6574 &56 &1\% &53674 &1224 &2\% &42394 &1158 &3\% &96068 &2382 &2\%\\ +twisted-names &201 &2706 &317 &12\% &64438 &2795 &4\% &47960 &1968 &4\% &24091 &555 &2\% &171582 &6828 &4\% &139195 &5635 &4\% &310777 &12463 &4\%\\ +total &2062 &24329 &6563 &27\% &632606 &46669 &7\% &477934 &42679 &9\% &202530 &10016 &5\% &1737154 &131306 &8\% &1337399 &105927 &8\% &3074553 &237233 &8\%\\ +\end{tabular} +\caption{Number of Operations Before and After Optimization} +\label{fig:numops} +\end{figure*} + +In addition to the count of operations we also performed time measurements. The +machine the benchmarks were run on is XXX. We compared the performance of +various Python implementations on the benchmarks. As a baseline, we used the +standard Python implementation in C, called +CPython\footnote{\texttt{http://python.org}}, which uses a bytecode-based +interpreter. Furthermore we compared against Psyco \cite{XXX}, an extension to +CPython which is a just-in-time compiler that produces machine code at run-time. +It is not based on traces. Of PyPy's Python interpreter we used three versions, +one without a JIT, one including the JIT but not using the allocation removal +optimization, and one using the allocation removal optimizations. + +All benchmarks were run 50 times in the same process, to give the JIT time to produce machine +code. The arithmetic mean of the times of the last 30 runs were used as the +result. The errors were computed using a confidence interval with a 95\% +confidence level \cite{XXX}. The results are reported in Figure~\ref{fig:times}. +With the optimization turned on, PyPy's Python interpreter outperforms CPython +in all benchmarks except spambayes (which heavily relies on regular expression +performance). All benchmarks are improved by the allocation removal +optimization, some as much as XXX. XXX Psyco -\textbf{crypto\_pyaes}: AES implementation. -\textbf{django}: The templating engine of the Django web -framework\footnote{\texttt{http://www.djangoproject.com/}}. -\textbf{go}: A Monte-Carlo Go -AI\footnote{\texttt{http://shed-skin.blogspot.com/2009/07/disco-elegant-python-go-player.html}}. -\textbf{html5lib}: HTML5 parser -\textbf{pyflate-fast}: BZ2 decoder -\textbf{raytrace-simple}: ray tracer -\textbf{richards}: The Richards benchmark \cite{XXX} -\textbf{spambayes}: A Bayesian spam filter\footnote{\texttt{http://spambayes.sourceforge.net/}}. -\textbf{telco}: A Python version of the Telco decimal -benchmark\footnote{\texttt{http://speleotrove.com/decimal/telco.html}}, using a pure -Python decimal floating point implementation. -\textbf{twisted\_names}: A DNS server benchmark using the Twisted networking -framework\footnote{\texttt{http://twistedmatrix.com/trac/}}. +XXX runtimes of the algorithm somehow? \begin{figure*} \begin{tabular}{lrrrr} @@ -1030,7 +1094,7 @@ go &595.31 $\pm$ 0.51 &1067.26 $\pm$ 12.20 &121.76 $\pm$ 4.15 &125.57 $\pm$ 3.90\\ html5lib &8678.67 $\pm$ 23.37 &17665.16 $\pm$ 2695.45 &6739.95 $\pm$ 1225.25 &6699.24 $\pm$ 1295.92\\ meteor-contest &241.53 $\pm$ 0.09 &291.11 $\pm$ 0.86 &274.09 $\pm$ 0.37 &272.77 $\pm$ 0.30\\ -nbody\_modified &396.55 $\pm$ 0.24 &84.77 $\pm$ 0.14 &69.01 $\pm$ 0.08 &68.71 $\pm$ 0.09\\ +nbody &396.55 $\pm$ 0.24 &84.77 $\pm$ 0.14 &69.01 $\pm$ 0.08 &68.71 $\pm$ 0.09\\ pyflate-fast &1991.33 $\pm$ 1.90 &1818.70 $\pm$ 4.30 &1062.03 $\pm$ 4.56 &1052.02 $\pm$ 3.24\\ raytrace-simple &1598.84 $\pm$ 1.07 &1126.93 $\pm$ 11.42 &93.45 $\pm$ 4.75 &92.17 $\pm$ 4.15\\ richards &216.47 $\pm$ 0.20 &169.84 $\pm$ 1.21 &11.81 $\pm$ 0.09 &11.90 $\pm$ 0.29\\ @@ -1043,14 +1107,32 @@ \label{fig:times} \end{figure*} - - \section{Related Work} \label{sec:related} +Escape analysis: XXX + +partial evaluation: + +Prolog + +partially static data structures: kenichi asai's thesis? + +hint in first js paper by michael franz \cite{mason_chang_efficient_2007} + +SPUR, a tracing JIT for C# seems to be able to remove allocations in a similar +way to the approach described here, as hinted at in the technical report +\cite{XXX}. However, no details for the approach and its implementation are +given. + \section{Conclusions} \label{sec:conclusions} +\section*{Acknowledgements} + +The authors would like to thank Stefan Hallerstede and Thomas Stiehl for +fruitful discussions during the writing of the paper. + \bibliographystyle{abbrv} \bibliography{paper} From afa at codespeak.net Tue Oct 12 12:56:03 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 12 Oct 2010 12:56:03 +0200 (CEST) Subject: [pypy-svn] r77820 - pypy/branch/fast-forward/lib-python/modified-2.7.0/multiprocessing Message-ID: <20101012105603.820AC282BF6@codespeak.net> Author: afa Date: Tue Oct 12 12:56:01 2010 New Revision: 77820 Added: pypy/branch/fast-forward/lib-python/modified-2.7.0/multiprocessing/ - copied from r77812, pypy/branch/fast-forward/lib-python/2.7.0/multiprocessing/ Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/multiprocessing/forking.py Log: PyPy does not have a separate type for builtin methods: list.append is a regular unbound method, with im_func & co. (this fix means that complex objects are being transferred between processes. Progress!) Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/multiprocessing/forking.py ============================================================================== --- pypy/branch/fast-forward/lib-python/2.7.0/multiprocessing/forking.py (original) +++ pypy/branch/fast-forward/lib-python/modified-2.7.0/multiprocessing/forking.py Tue Oct 12 12:56:01 2010 @@ -47,15 +47,12 @@ return getattr, (m.im_self, m.im_func.func_name) ForkingPickler.register(type(ForkingPickler.save), _reduce_method) -def _reduce_method_descriptor(m): - return getattr, (m.__objclass__, m.__name__) -ForkingPickler.register(type(list.append), _reduce_method_descriptor) -ForkingPickler.register(type(int.__add__), _reduce_method_descriptor) - -#def _reduce_builtin_function_or_method(m): -# return getattr, (m.__self__, m.__name__) -#ForkingPickler.register(type(list().append), _reduce_builtin_function_or_method) -#ForkingPickler.register(type(int().__add__), _reduce_builtin_function_or_method) +if type(list.append) is not type(ForkingPickler.save): + # Some python implementations have unbound methods even for builtin types + def _reduce_method_descriptor(m): + return getattr, (m.__objclass__, m.__name__) + ForkingPickler.register(type(list.append), _reduce_method_descriptor) + ForkingPickler.register(type(int.__add__), _reduce_method_descriptor) try: from functools import partial From afa at codespeak.net Tue Oct 12 13:05:31 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 12 Oct 2010 13:05:31 +0200 (CEST) Subject: [pypy-svn] r77821 - pypy/branch/fast-forward/lib_pypy Message-ID: <20101012110531.6E83C282BF6@codespeak.net> Author: afa Date: Tue Oct 12 13:05:29 2010 New Revision: 77821 Modified: pypy/branch/fast-forward/lib_pypy/_subprocess.py Log: Add _subprocess.TerminateProcess(), called directly by the multiprocessing module Modified: pypy/branch/fast-forward/lib_pypy/_subprocess.py ============================================================================== --- pypy/branch/fast-forward/lib_pypy/_subprocess.py (original) +++ pypy/branch/fast-forward/lib_pypy/_subprocess.py Tue Oct 12 13:05:29 2010 @@ -42,6 +42,10 @@ _GetExitCodeProcess.argtypes = [ctypes.c_int, ctypes.POINTER(ctypes.c_int)] _GetExitCodeProcess.restype = ctypes.c_int +_TerminateProcess = _kernel32.TerminateProcess +_TerminateProcess.argtypes = [ctypes.c_int, ctypes.c_int] +_TerminateProcess.restype = ctypes.c_int + _GetStdHandle = _kernel32.GetStdHandle _GetStdHandle.argtypes = [ctypes.c_int] _GetStdHandle.restype = ctypes.c_int @@ -186,6 +190,12 @@ return code.value +def TerminateProcess(handle, exitcode): + res = _TerminateProcess(int(handle), exitcode) + + if not res: + raise WindowsError("Error") + def GetStdHandle(stdhandle): res = _GetStdHandle(stdhandle) From afa at codespeak.net Tue Oct 12 13:16:10 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 12 Oct 2010 13:16:10 +0200 (CEST) Subject: [pypy-svn] r77822 - in pypy/branch/fast-forward/pypy/module/_multiprocessing: . test Message-ID: <20101012111610.4DA5D282BF6@codespeak.net> Author: afa Date: Tue Oct 12 13:16:08 2010 New Revision: 77822 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_win32.py Log: Add multiprocessing.win32.WaitNamedPipe() Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py Tue Oct 12 13:16:08 2010 @@ -50,6 +50,10 @@ rwin32.LPDWORD, rwin32.LPDWORD, rwin32.LPDWORD], rwin32.BOOL) +_WaitNamedPipe = rwin32.winexternal( + 'WaitNamedPipeA', [rwin32.LPCSTR, rwin32.DWORD], + rwin32.BOOL) + _PeekNamedPipe = rwin32.winexternal( 'PeekNamedPipe', [ rwin32.HANDLE, @@ -145,6 +149,12 @@ lltype.free(state, flavor='raw') lltype.free(statep, flavor='raw') + at unwrap_spec(ObjSpace, str, r_uint) +def WaitNamedPipe(space, name, timeout): + # Careful: zero means "default value specified by CreateNamedPipe()" + if not _WaitNamedPipe(name, timeout): + raise wrap_windowserror(space, rwin32.lastWindowsError()) + @unwrap_spec(ObjSpace, str, r_uint, r_uint, W_Root, r_uint, r_uint, W_Root) def CreateFile(space, filename, access, share, w_security, disposition, flags, w_templatefile): @@ -184,7 +194,7 @@ # functions for name in ['CloseHandle', 'GetLastError', 'CreateFile', 'CreateNamedPipe', 'ConnectNamedPipe', - 'SetNamedPipeHandleState', + 'SetNamedPipeHandleState', 'WaitNamedPipe', 'ExitProcess', ]: function = globals()[name] Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_win32.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_win32.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_win32.py Tue Oct 12 13:16:08 2010 @@ -43,5 +43,8 @@ if e.args[0] != win32.ERROR_PIPE_CONNECTED: raise + exc = raises(WindowsError, win32.WaitNamedPipe, address, timeout=100) + assert exc.value.winerror == 121 # ERROR_SEM_TIMEOUT + win32.CloseHandle(readhandle) win32.CloseHandle(writehandle) From antocuni at codespeak.net Tue Oct 12 13:35:23 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 12 Oct 2010 13:35:23 +0200 (CEST) Subject: [pypy-svn] r77825 - pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt Message-ID: <20101012113523.427E6282BF6@codespeak.net> Author: antocuni Date: Tue Oct 12 13:35:21 2010 New Revision: 77825 Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py Log: simplify the logic, fix it when effectinfo is None, enable the optimization when translated Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py Tue Oct 12 13:35:21 2010 @@ -61,19 +61,14 @@ def __init__(self): self.func_infos = {} - def optimize_CALL(self, op): - if we_are_translated(): - self.emit_operation(op) - return - # + def _get_oopspec(self, op): effectinfo = op.getdescr().get_extra_info() - oopspec = effectinfo.oopspecindex - if oopspec not in (EffectInfo.OS_LIBFFI_PREPARE, - EffectInfo.OS_LIBFFI_PUSH_ARG, - EffectInfo.OS_LIBFFI_CALL): - self.emit_operation(op) # normal case - return - # + if effectinfo is not None: + return effectinfo.oopspecindex + return EffectInfo.OS_NONE + + def optimize_CALL(self, op): + oopspec = self._get_oopspec(op) try: if oopspec == EffectInfo.OS_LIBFFI_PREPARE: self.do_prepare_call(op) @@ -82,7 +77,10 @@ elif oopspec == EffectInfo.OS_LIBFFI_CALL: op = self.do_call(op) self.emit_operation(op) + else: + raise NonConstantFuncVal # it's not a libffi call except NonConstantFuncVal: + # normal case self.emit_operation(op) def _get_funcval(self, op): From antocuni at codespeak.net Tue Oct 12 13:41:53 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 12 Oct 2010 13:41:53 +0200 (CEST) Subject: [pypy-svn] r77826 - in pypy/branch/jitffi/pypy: interpreter/pyparser jit/backend jit/backend/x86/test jit/metainterp jit/metainterp/test jit/tl rpython/memory/gc Message-ID: <20101012114153.54D58282BF6@codespeak.net> Author: antocuni Date: Tue Oct 12 13:41:51 2010 New Revision: 77826 Added: pypy/branch/jitffi/pypy/interpreter/pyparser/autopath.py (contents, props changed) pypy/branch/jitffi/pypy/interpreter/pyparser/genpytokenize.py (contents, props changed) pypy/branch/jitffi/pypy/interpreter/pyparser/pylexer.py (contents, props changed) pypy/branch/jitffi/pypy/jit/backend/conftest.py (contents, props changed) pypy/branch/jitffi/pypy/jit/backend/x86/test/test_string.py (contents, props changed) pypy/branch/jitffi/pypy/jit/metainterp/optimize_nopspec.py (contents, props changed) pypy/branch/jitffi/pypy/jit/metainterp/test/test_loop_nopspec.py (contents, props changed) pypy/branch/jitffi/pypy/jit/tl/jittest.py (contents, props changed) pypy/branch/jitffi/pypy/rpython/memory/gc/inspector.py (contents, props changed) Log: add more files that were lost during the merge :-( Added: pypy/branch/jitffi/pypy/interpreter/pyparser/autopath.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/interpreter/pyparser/autopath.py Tue Oct 12 13:41:51 2010 @@ -0,0 +1,134 @@ +""" +self cloning, automatic path configuration + +copy this into any subdirectory of pypy from which scripts need +to be run, typically all of the test subdirs. +The idea is that any such script simply issues + + import autopath + +and this will make sure that the parent directory containing "pypy" +is in sys.path. + +If you modify the master "autopath.py" version (in pypy/tool/autopath.py) +you can directly run it which will copy itself on all autopath.py files +it finds under the pypy root directory. + +This module always provides these attributes: + + pypydir pypy root directory path + this_dir directory where this autopath.py resides + +""" + +def __dirinfo(part): + """ return (partdir, this_dir) and insert parent of partdir + into sys.path. If the parent directories don't have the part + an EnvironmentError is raised.""" + + import sys, os + try: + head = this_dir = os.path.realpath(os.path.dirname(__file__)) + except NameError: + head = this_dir = os.path.realpath(os.path.dirname(sys.argv[0])) + + error = None + while head: + partdir = head + head, tail = os.path.split(head) + if tail == part: + checkfile = os.path.join(partdir, os.pardir, 'pypy', '__init__.py') + if not os.path.exists(checkfile): + error = "Cannot find %r" % (os.path.normpath(checkfile),) + break + else: + error = "Cannot find the parent directory %r of the path %r" % ( + partdir, this_dir) + if not error: + # check for bogus end-of-line style (e.g. files checked out on + # Windows and moved to Unix) + f = open(__file__.replace('.pyc', '.py'), 'r') + data = f.read() + f.close() + if data.endswith('\r\n') or data.endswith('\r'): + error = ("Bad end-of-line style in the .py files. Typically " + "caused by a zip file or a checkout done on Windows and " + "moved to Unix or vice-versa.") + if error: + raise EnvironmentError("Invalid source tree - bogus checkout! " + + error) + + pypy_root = os.path.join(head, '') + try: + sys.path.remove(head) + except ValueError: + pass + sys.path.insert(0, head) + + munged = {} + for name, mod in sys.modules.items(): + if '.' in name: + continue + fn = getattr(mod, '__file__', None) + if not isinstance(fn, str): + continue + newname = os.path.splitext(os.path.basename(fn))[0] + if not newname.startswith(part + '.'): + continue + path = os.path.join(os.path.dirname(os.path.realpath(fn)), '') + if path.startswith(pypy_root) and newname != part: + modpaths = os.path.normpath(path[len(pypy_root):]).split(os.sep) + if newname != '__init__': + modpaths.append(newname) + modpath = '.'.join(modpaths) + if modpath not in sys.modules: + munged[modpath] = mod + + for name, mod in munged.iteritems(): + if name not in sys.modules: + sys.modules[name] = mod + if '.' in name: + prename = name[:name.rfind('.')] + postname = name[len(prename)+1:] + if prename not in sys.modules: + __import__(prename) + if not hasattr(sys.modules[prename], postname): + setattr(sys.modules[prename], postname, mod) + + return partdir, this_dir + +def __clone(): + """ clone master version of autopath.py into all subdirs """ + from os.path import join, walk + if not this_dir.endswith(join('pypy','tool')): + raise EnvironmentError("can only clone master version " + "'%s'" % join(pypydir, 'tool',_myname)) + + + def sync_walker(arg, dirname, fnames): + if _myname in fnames: + fn = join(dirname, _myname) + f = open(fn, 'rwb+') + try: + if f.read() == arg: + print "checkok", fn + else: + print "syncing", fn + f = open(fn, 'w') + f.write(arg) + finally: + f.close() + s = open(join(pypydir, 'tool', _myname), 'rb').read() + walk(pypydir, sync_walker, s) + +_myname = 'autopath.py' + +# set guaranteed attributes + +pypydir, this_dir = __dirinfo('pypy') +import py # note: py is imported only AFTER the path has been set +libpythondir = str(py.path.local(pypydir).dirpath().join('lib-python', '2.5.2')) +libpythonmodifieddir = str(py.path.local(libpythondir).dirpath().join('modified-2.5.2')) + +if __name__ == '__main__': + __clone() Added: pypy/branch/jitffi/pypy/interpreter/pyparser/genpytokenize.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/interpreter/pyparser/genpytokenize.py Tue Oct 12 13:41:51 2010 @@ -0,0 +1,306 @@ +#! /usr/bin/env python +"""Module genPytokenize + +Generates finite state automata for recognizing Python tokens. These are hand +coded versions of the regular expressions originally appearing in Ping's +tokenize module in the Python standard library. + +When run from the command line, this should pretty print the DFA machinery. + +$Id: genPytokenize.py,v 1.1 2003/10/02 17:37:17 jriehl Exp $ +""" + +import autopath +from pypy.interpreter.pyparser.pylexer import * +from pypy.interpreter.pyparser.automata import NonGreedyDFA, DFA, DEFAULT + +def makePyPseudoDFA (): + import string + states = [] + # ____________________________________________________________ + def makeLineCont (): + return chain(states, + newArcPair(states, "\\"), + maybe(states, newArcPair(states, "\r")), + newArcPair(states, "\n")) + # ____________________________________________________________ + # Ignore stuff + def makeWhitespace (): + return any(states, groupStr(states, " \f\t")) + # ____________________________________________________________ + def makeComment (): + return chain(states, + newArcPair(states, "#"), + any(states, notGroupStr(states, "\r\n"))) + # ____________________________________________________________ + #ignore = chain(states, + # makeWhitespace(), + # any(states, chain(states, + # makeLineCont(), + # makeWhitespace())), + # maybe(states, makeComment())) + # ____________________________________________________________ + # Names + name = chain(states, + groupStr(states, string.letters + "_"), + any(states, groupStr(states, + string.letters + string.digits + "_"))) + # ____________________________________________________________ + # Digits + def makeDigits (): + return groupStr(states, "0123456789") + # ____________________________________________________________ + # Integer numbers + hexNumber = chain(states, + newArcPair(states, "0"), + groupStr(states, "xX"), + any(states, groupStr(states, "0123456789abcdefABCDEF")), + maybe(states, groupStr(states, "lL"))) + octNumber = chain(states, + newArcPair(states, "0"), + any(states, groupStr(states, "01234567")), + maybe(states, groupStr(states, "lL"))) + decNumber = chain(states, + groupStr(states, "123456789"), + any(states, makeDigits()), + maybe(states, groupStr(states, "lL"))) + intNumber = group(states, hexNumber, octNumber, decNumber) + # ____________________________________________________________ + # Exponents + def makeExp (): + return chain(states, + groupStr(states, "eE"), + maybe(states, groupStr(states, "+-")), + atleastonce(states, makeDigits())) + # ____________________________________________________________ + # Floating point numbers + def makeFloat (): + pointFloat = chain(states, + group(states, + chain(states, + atleastonce(states, makeDigits()), + newArcPair(states, "."), + any(states, makeDigits())), + chain(states, + newArcPair(states, "."), + atleastonce(states, makeDigits()))), + maybe(states, makeExp())) + expFloat = chain(states, + atleastonce(states, makeDigits()), + makeExp()) + return group(states, pointFloat, expFloat) + # ____________________________________________________________ + # Imaginary numbers + imagNumber = group(states, + chain(states, + atleastonce(states, makeDigits()), + groupStr(states, "jJ")), + chain(states, + makeFloat(), + groupStr(states, "jJ"))) + # ____________________________________________________________ + # Any old number. + number = group(states, imagNumber, makeFloat(), intNumber) + # ____________________________________________________________ + # Funny + operator = group(states, + chain(states, + chainStr(states, "**"), + maybe(states, newArcPair(states, "="))), + chain(states, + chainStr(states, ">>"), + maybe(states, newArcPair(states, "="))), + chain(states, + chainStr(states, "<<"), + maybe(states, newArcPair(states, "="))), + chainStr(states, "<>"), + chainStr(states, "!="), + chain(states, + chainStr(states, "//"), + maybe(states, newArcPair(states, "="))), + chain(states, + groupStr(states, "+-*/%&|^=<>"), + maybe(states, newArcPair(states, "="))), + newArcPair(states, "~")) + bracket = groupStr(states, "[](){}") + special = group(states, + chain(states, + maybe(states, newArcPair(states, "\r")), + newArcPair(states, "\n")), + groupStr(states, "@:;.,`")) + funny = group(states, operator, bracket, special) + # ____________________________________________________________ + def makeStrPrefix (): + return chain(states, + maybe(states, groupStr(states, "uU")), + maybe(states, groupStr(states, "rR"))) + # ____________________________________________________________ + contStr = group(states, + chain(states, + makeStrPrefix(), + newArcPair(states, "'"), + any(states, + notGroupStr(states, "\n'\\")), + any(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT), + any(states, + notGroupStr(states, "\n'\\")))), + group(states, + newArcPair(states, "'"), + makeLineCont())), + chain(states, + makeStrPrefix(), + newArcPair(states, '"'), + any(states, + notGroupStr(states, '\n"\\')), + any(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT), + any(states, + notGroupStr(states, '\n"\\')))), + group(states, + newArcPair(states, '"'), + makeLineCont()))) + triple = chain(states, + makeStrPrefix(), + group(states, + chainStr(states, "'''"), + chainStr(states, '"""'))) + pseudoExtras = group(states, + makeLineCont(), + makeComment(), + triple) + pseudoToken = chain(states, + makeWhitespace(), + group(states, + pseudoExtras, number, funny, contStr, name)) + dfaStates, dfaAccepts = nfaToDfa(states, *pseudoToken) + return DFA(dfaStates, dfaAccepts) + +# ______________________________________________________________________ + +def makePyEndDFAMap (): + states = [] + single = chain(states, + any(states, notGroupStr(states, "'\\")), + any(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT), + any(states, notGroupStr(states, "'\\")))), + newArcPair(states, "'")) + singleDFA = DFA(*nfaToDfa(states, *single)) + states = [] + double = chain(states, + any(states, notGroupStr(states, '"\\')), + any(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT), + any(states, notGroupStr(states, '"\\')))), + newArcPair(states, '"')) + doubleDFA = DFA(*nfaToDfa(states, *double)) + states = [] + single3 = chain(states, + any(states, notGroupStr(states, "'\\")), + any(states, + chain(states, + group(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT)), + chain(states, + newArcPair(states, "'"), + notChainStr(states, "''"))), + any(states, notGroupStr(states, "'\\")))), + chainStr(states, "'''")) + single3DFA = NonGreedyDFA(*nfaToDfa(states, *single3)) + states = [] + double3 = chain(states, + any(states, notGroupStr(states, '"\\')), + any(states, + chain(states, + group(states, + chain(states, + newArcPair(states, "\\"), + newArcPair(states, DEFAULT)), + chain(states, + newArcPair(states, '"'), + notChainStr(states, '""'))), + any(states, notGroupStr(states, '"\\')))), + chainStr(states, '"""')) + double3DFA = NonGreedyDFA(*nfaToDfa(states, *double3)) + map = {"'" : singleDFA, + '"' : doubleDFA, + "r" : None, + "R" : None, + "u" : None, + "U" : None} + for uniPrefix in ("", "u", "U", ): + for rawPrefix in ("", "r", "R"): + prefix = uniPrefix + rawPrefix + map[prefix + "'''"] = single3DFA + map[prefix + '"""'] = double3DFA + return map + +# ______________________________________________________________________ + +def output(name, dfa_class, dfa): + import textwrap + i = 0 + for line in textwrap.wrap(repr(dfa.accepts), width = 50): + if i == 0: + print "accepts =", line + else: + print " ", line + i += 1 + import StringIO + print "states = [" + for numstate, state in enumerate(dfa.states): + print " #", numstate + s = StringIO.StringIO() + i = 0 + for k, v in sorted(state.items()): + i += 1 + if k == '\x00default': + k = "automata.DEFAULT" + else: + k = repr(k) + s.write(k) + s.write('::') + s.write(repr(v)) + if i < len(state): + s.write(', ') + s.write('},') + i = 0 + if len(state) <= 4: + text = [s.getvalue()] + else: + text = textwrap.wrap(s.getvalue(), width=36) + for line in text: + line = line.replace('::', ': ') + if i == 0: + print ' {' + line + else: + print ' ' + line + i += 1 + print " ]" + print "%s = automata.%s(states, accepts)" % (name, dfa_class) + print + +def main (): + pseudoDFA = makePyPseudoDFA() + output("pseudoDFA", "DFA", pseudoDFA) + endDFAMap = makePyEndDFAMap() + output("double3DFA", "NonGreedyDFA", endDFAMap['"""']) + output("single3DFA", "NonGreedyDFA", endDFAMap["'''"]) + output("singleDFA", "DFA", endDFAMap["'"]) + output("doubleDFA", "DFA", endDFAMap['"']) + +# ______________________________________________________________________ + +if __name__ == "__main__": + main() Added: pypy/branch/jitffi/pypy/interpreter/pyparser/pylexer.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/interpreter/pyparser/pylexer.py Tue Oct 12 13:41:51 2010 @@ -0,0 +1,240 @@ +# Used by genpytokenize.py to generate the parser in pytokenize.py +from pypy.interpreter.pyparser.automata import DFA, DEFAULT + +class EMPTY: pass + +def newArcPair (states, transitionLabel): + s1Index = len(states) + s2Index = s1Index + 1 + states.append([(transitionLabel, s2Index)]) + states.append([]) + return s1Index, s2Index + +# ______________________________________________________________________ + +def chain (states, *stateIndexPairs): + if len(stateIndexPairs) > 1: + start, lastFinish = stateIndexPairs[0] + for nStart, nFinish in stateIndexPairs[1:]: + states[lastFinish].append((EMPTY, nStart)) + lastFinish = nFinish + return start, nFinish + else: + return stateIndexPairs[0] + + +# ______________________________________________________________________ + +def chainStr (states, str): + return chain(states, *map(lambda x : newArcPair(states, x), str)) + +# ______________________________________________________________________ + +def notChainStr (states, str): + """XXX I'm not sure this is how it should be done, but I'm going to + try it anyway. Note that for this case, I require only single character + arcs, since I would have to basically invert all accepting states and + non-accepting states of any sub-NFA's. + """ + assert len(str) > 0 + arcs = map(lambda x : newArcPair(states, x), str) + finish = len(states) + states.append([]) + start, lastFinish = arcs[0] + states[start].append((EMPTY, finish)) + for crntStart, crntFinish in arcs[1:]: + states[lastFinish].append((EMPTY, crntStart)) + states[crntStart].append((EMPTY, finish)) + return start, finish + +# ______________________________________________________________________ + +def group (states, *stateIndexPairs): + if len(stateIndexPairs) > 1: + start = len(states) + finish = start + 1 + startList = [] + states.append(startList) + states.append([]) + for eStart, eFinish in stateIndexPairs: + startList.append((EMPTY, eStart)) + states[eFinish].append((EMPTY, finish)) + return start, finish + else: + return stateIndexPairs[0] + +# ______________________________________________________________________ + +def groupStr (states, str): + return group(states, *map(lambda x : newArcPair(states, x), str)) + +# ______________________________________________________________________ + +def notGroup (states, *stateIndexPairs): + """Like group, but will add a DEFAULT transition to a new end state, + causing anything in the group to not match by going to a dead state. + XXX I think this is right... + """ + start, dead = group(states, *stateIndexPairs) + finish = len(states) + states.append([]) + states[start].append((DEFAULT, finish)) + return start, finish + +# ______________________________________________________________________ + +def notGroupStr (states, str): + return notGroup(states, *map(lambda x : newArcPair(states, x), str)) +# ______________________________________________________________________ + +def any (states, *stateIndexPairs): + start, finish = group(states, *stateIndexPairs) + states[finish].append((EMPTY, start)) + return start, start + +# ______________________________________________________________________ + +def maybe (states, *stateIndexPairs): + start, finish = group(states, *stateIndexPairs) + states[start].append((EMPTY, finish)) + return start, finish + +# ______________________________________________________________________ + +def atleastonce (states, *stateIndexPairs): + start, finish = group(states, *stateIndexPairs) + states[finish].append((EMPTY, start)) + return start, finish + +# ______________________________________________________________________ + +def closure (states, start, result = 0L): + if None == result: + result = 0L + if 0 == (result & (1L << start)): + result |= (1L << start) + for label, arrow in states[start]: + if label == EMPTY: + result |= closure(states, arrow, result) + return result + +# ______________________________________________________________________ + +def nfaToDfa (states, start, finish): + tempStates = [] + startClosure = closure(states, start) + crntTempState = [startClosure, [], 0 != (startClosure & (1L << finish))] + tempStates.append(crntTempState) + index = 0 + while index < len(tempStates): + crntTempState = tempStates[index] + crntClosure, crntArcs, crntAccept = crntTempState + for index2 in range(0, len(states)): + if 0 != (crntClosure & (1L << index2)): + for label, nfaArrow in states[index2]: + if label == EMPTY: + continue + foundTempArc = False + for tempArc in crntArcs: + if tempArc[0] == label: + foundTempArc = True + break + if not foundTempArc: + tempArc = [label, -1, 0L] + crntArcs.append(tempArc) + tempArc[2] = closure(states, nfaArrow, tempArc[2]) + for arcIndex in range(0, len(crntArcs)): + label, arrow, targetStates = crntArcs[arcIndex] + targetFound = False + arrow = 0 + for destTempState in tempStates: + if destTempState[0] == targetStates: + targetFound = True + break + arrow += 1 + if not targetFound: + assert arrow == len(tempStates) + newState = [targetStates, [], 0 != (targetStates & + (1L << finish))] + tempStates.append(newState) + crntArcs[arcIndex][1] = arrow + index += 1 + tempStates = simplifyTempDfa(tempStates) + states = finalizeTempDfa(tempStates) + return states + +# ______________________________________________________________________ + +def sameState (s1, s2): + """sameState(s1, s2) + Note: + state := [ nfaclosure : Long, [ arc ], accept : Boolean ] + arc := [ label, arrow : Int, nfaClosure : Long ] + """ + if (len(s1[1]) != len(s2[1])) or (s1[2] != s2[2]): + return False + for arcIndex in range(0, len(s1[1])): + arc1 = s1[1][arcIndex] + arc2 = s2[1][arcIndex] + if arc1[:-1] != arc2[:-1]: + return False + return True + +# ______________________________________________________________________ + +def simplifyTempDfa (tempStates): + """simplifyTempDfa (tempStates) + """ + changes = True + deletedStates = [] + while changes: + changes = False + for i in range(1, len(tempStates)): + if i in deletedStates: + continue + for j in range(0, i): + if j in deletedStates: + continue + if sameState(tempStates[i], tempStates[j]): + deletedStates.append(i) + for k in range(0, len(tempStates)): + if k in deletedStates: + continue + for arc in tempStates[k][1]: + if arc[1] == i: + arc[1] = j + changes = True + break + for stateIndex in deletedStates: + tempStates[stateIndex] = None + return tempStates +# ______________________________________________________________________ + +def finalizeTempDfa (tempStates): + """finalizeTempDfa (tempStates) + + Input domain: + tempState := [ nfaClosure : Long, [ tempArc ], accept : Boolean ] + tempArc := [ label, arrow, nfaClosure ] + + Output domain: + state := [ arcMap, accept : Boolean ] + """ + states = [] + accepts = [] + stateMap = {} + tempIndex = 0 + for tempIndex in range(0, len(tempStates)): + tempState = tempStates[tempIndex] + if None != tempState: + stateMap[tempIndex] = len(states) + states.append({}) + accepts.append(tempState[2]) + for tempIndex in stateMap.keys(): + stateBitset, tempArcs, accepting = tempStates[tempIndex] + newIndex = stateMap[tempIndex] + arcMap = states[newIndex] + for tempArc in tempArcs: + arcMap[tempArc[0]] = stateMap[tempArc[1]] + return states, accepts + Added: pypy/branch/jitffi/pypy/jit/backend/conftest.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/jit/backend/conftest.py Tue Oct 12 13:41:51 2010 @@ -0,0 +1,35 @@ +""" +This conftest adds options used by test/test_random and +x86/test/test_zll_random. +""" +import py, random + +option = py.test.config.option + +def pytest_addoption(parser): + group = parser.getgroup('random test options') + group.addoption('--random-seed', action="store", type="int", + default=random.randrange(0, 10000), + dest="randomseed", + help="choose a fixed random seed") + group.addoption('--backend', action="store", + default='llgraph', + choices=['llgraph', 'x86'], + dest="backend", + help="select the backend to run the functions with") + group.addoption('--block-length', action="store", type="int", + default=30, + dest="block_length", + help="insert up to this many operations in each test") + group.addoption('--n-vars', action="store", type="int", + default=10, + dest="n_vars", + help="supply this many randomly-valued arguments to " + "the function") + group.addoption('--repeat', action="store", type="int", + default=15, + dest="repeat", + help="run the test this many times"), + group.addoption('--output', '-O', action="store", type="str", + default="", dest="output", + help="dump output to a file") Added: pypy/branch/jitffi/pypy/jit/backend/x86/test/test_string.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/jit/backend/x86/test/test_string.py Tue Oct 12 13:41:51 2010 @@ -0,0 +1,9 @@ +import py +from pypy.jit.metainterp.test import test_string +from pypy.jit.backend.x86.test.test_basic import Jit386Mixin + +class TestString(Jit386Mixin, test_string.StringTests): + # for the individual tests see + # ====> ../../../metainterp/test/test_string.py + CALL = 'call' + CALL_PURE = 'call_pure' Added: pypy/branch/jitffi/pypy/jit/metainterp/optimize_nopspec.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimize_nopspec.py Tue Oct 12 13:41:51 2010 @@ -0,0 +1,41 @@ + +from pypy.rlib.debug import debug_start, debug_stop +from pypy.jit.metainterp.optimizeopt import optimize_loop_1, optimize_bridge_1 +from pypy.jit.metainterp.optimizefindnode import PerfectSpecializationFinder +from pypy.jit.metainterp.optimizefindnode import BridgeSpecializationFinder + +def optimize_loop(metainterp_sd, old_loop_tokens, loop): + debug_start("jit-optimize") + try: + return _optimize_loop(metainterp_sd, old_loop_tokens, loop) + finally: + debug_stop("jit-optimize") + +def _optimize_loop(metainterp_sd, old_loop_tokens, loop): + cpu = metainterp_sd.cpu + metainterp_sd.logger_noopt.log_loop(loop.inputargs, loop.operations) + finder = PerfectSpecializationFinder(cpu) + finder.find_nodes_loop(loop, False) + if old_loop_tokens: + return old_loop_tokens[0] + optimize_loop_1(metainterp_sd, loop) + return None + +def optimize_bridge(metainterp_sd, old_loop_tokens, bridge): + debug_start("jit-optimize") + try: + return _optimize_bridge(metainterp_sd, old_loop_tokens, bridge) + finally: + debug_stop("jit-optimize") + +def _optimize_bridge(metainterp_sd, old_loop_tokens, bridge): + cpu = metainterp_sd.cpu + metainterp_sd.logger_noopt.log_loop(bridge.inputargs, bridge.operations) + finder = BridgeSpecializationFinder(cpu) + finder.find_nodes_bridge(bridge) + if old_loop_tokens: + old_loop_token = old_loop_tokens[0] + bridge.operations[-1].setdescr(old_loop_token) # patch jump target + optimize_bridge_1(metainterp_sd, bridge) + return old_loop_token + return None Added: pypy/branch/jitffi/pypy/jit/metainterp/test/test_loop_nopspec.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_loop_nopspec.py Tue Oct 12 13:41:51 2010 @@ -0,0 +1,27 @@ + +from pypy.jit.metainterp.test import test_loop, test_send +from pypy.jit.metainterp.warmspot import ll_meta_interp +from pypy.rlib.jit import OPTIMIZER_NO_PERFECTSPEC +from pypy.jit.metainterp.test.test_basic import LLJitMixin, OOJitMixin + +class LoopNoPSpecTest(test_send.SendTests): + def meta_interp(self, func, args, **kwds): + return ll_meta_interp(func, args, optimizer=OPTIMIZER_NO_PERFECTSPEC, + CPUClass=self.CPUClass, + type_system=self.type_system, + **kwds) + + def check_loops(self, *args, **kwds): + pass + + def check_loop_count(self, count): + pass + + def check_jumps(self, maxcount): + pass + +class TestLLtype(LoopNoPSpecTest, LLJitMixin): + pass + +class TestOOtype(LoopNoPSpecTest, OOJitMixin): + pass Added: pypy/branch/jitffi/pypy/jit/tl/jittest.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/jit/tl/jittest.py Tue Oct 12 13:41:51 2010 @@ -0,0 +1,43 @@ +""" +This file is imported by pypy.translation.driver when running the +target --jittest. Feel free to hack it as needed; it is imported +only after the '---> Checkpoint' fork. +""" + +from pypy.conftest import option +from pypy.rpython.lltypesystem import lltype +from pypy.rpython.llinterp import LLInterpreter +from pypy.rpython.annlowlevel import llstr +from pypy.jit.metainterp import warmspot +from pypy.rlib.jit import OPTIMIZER_FULL + + +ARGS = ["jittest", "100"] + + +def jittest(driver): + graph = driver.translator.graphs[0] + interp = LLInterpreter(driver.translator.rtyper, malloc_check=False) + + def returns_null(T, *args, **kwds): + return lltype.nullptr(T) + interp.heap.malloc_nonmovable = returns_null # XXX + + get_policy = driver.extra['jitpolicy'] + jitpolicy = get_policy(driver) + + from pypy.jit.backend.llgraph.runner import LLtypeCPU + apply_jit(jitpolicy, interp, graph, LLtypeCPU) + + +def apply_jit(policy, interp, graph, CPUClass): + print 'warmspot.jittify_and_run() started...' + option.view = True + LIST = graph.getargs()[0].concretetype + lst = LIST.TO.ll_newlist(len(ARGS)) + for i, arg in enumerate(ARGS): + lst.ll_setitem_fast(i, llstr(arg)) + warmspot.jittify_and_run(interp, graph, [lst], policy=policy, + listops=True, CPUClass=CPUClass, + backendopt=True, inline=True, + optimizer=OPTIMIZER_FULL) Added: pypy/branch/jitffi/pypy/rpython/memory/gc/inspector.py ============================================================================== --- (empty file) +++ pypy/branch/jitffi/pypy/rpython/memory/gc/inspector.py Tue Oct 12 13:41:51 2010 @@ -0,0 +1,200 @@ +""" +Utility RPython functions to inspect objects in the GC. +""" +from pypy.rpython.lltypesystem import lltype, llmemory, rffi +from pypy.rlib.objectmodel import free_non_gc_object +from pypy.rpython.module.ll_os import underscore_on_windows +from pypy.rlib import rposix + +from pypy.rpython.memory.support import AddressDict, get_address_stack + + +# ---------- implementation of pypy.rlib.rgc.get_rpy_roots() ---------- + +def _counting_rpy_root(gc, root): + gc._count_rpy += 1 + +def _do_count_rpy_roots(gc): + gc._count_rpy = 0 + gc.root_walker.walk_roots( + _counting_rpy_root, + _counting_rpy_root, + _counting_rpy_root) + return gc._count_rpy + +def _append_rpy_root(gc, root): + # Can use the gc list, but should not allocate! + # It is essential that the list is not resizable! + lst = gc._list_rpy + index = gc._count_rpy + if index >= len(lst): + raise ValueError + gc._count_rpy = index + 1 + lst[index] = llmemory.cast_adr_to_ptr(root.address[0], llmemory.GCREF) + +def _do_append_rpy_roots(gc, lst): + gc._count_rpy = 0 + gc._list_rpy = lst + gc.root_walker.walk_roots( + _append_rpy_root, + _append_rpy_root, + _append_rpy_root) + gc._list_rpy = None + +def get_rpy_roots(gc): + count = _do_count_rpy_roots(gc) + extra = 16 + while True: + result = [lltype.nullptr(llmemory.GCREF.TO)] * (count + extra) + try: + _do_append_rpy_roots(gc, result) + except ValueError: + extra *= 3 + else: + return result + +# ---------- implementation of pypy.rlib.rgc.get_rpy_referents() ---------- + +def _count_rpy_referent(pointer, gc): + gc._count_rpy += 1 + +def _do_count_rpy_referents(gc, gcref): + gc._count_rpy = 0 + gc.trace(llmemory.cast_ptr_to_adr(gcref), _count_rpy_referent, gc) + return gc._count_rpy + +def _append_rpy_referent(pointer, gc): + # Can use the gc list, but should not allocate! + # It is essential that the list is not resizable! + lst = gc._list_rpy + index = gc._count_rpy + if index >= len(lst): + raise ValueError + gc._count_rpy = index + 1 + lst[index] = llmemory.cast_adr_to_ptr(pointer.address[0], + llmemory.GCREF) + +def _do_append_rpy_referents(gc, gcref, lst): + gc._count_rpy = 0 + gc._list_rpy = lst + gc.trace(llmemory.cast_ptr_to_adr(gcref), _append_rpy_referent, gc) + +def get_rpy_referents(gc, gcref): + count = _do_count_rpy_referents(gc, gcref) + result = [lltype.nullptr(llmemory.GCREF.TO)] * count + _do_append_rpy_referents(gc, gcref, result) + return result + +# ---------- + +def get_rpy_memory_usage(gc, gcref): + return gc.get_size_incl_hash(llmemory.cast_ptr_to_adr(gcref)) + +def get_rpy_type_index(gc, gcref): + typeid = gc.get_type_id(llmemory.cast_ptr_to_adr(gcref)) + return gc.get_member_index(typeid) + +def is_rpy_instance(gc, gcref): + typeid = gc.get_type_id(llmemory.cast_ptr_to_adr(gcref)) + return gc.is_rpython_class(typeid) + +# ---------- + +raw_os_write = rffi.llexternal(underscore_on_windows+'write', + [rffi.INT, llmemory.Address, rffi.SIZE_T], + rffi.SIZE_T, + sandboxsafe=True, _nowrapper=True) + +AddressStack = get_address_stack() + +class HeapDumper: + _alloc_flavor_ = "raw" + BUFSIZE = 8192 # words + + def __init__(self, gc, fd): + self.gc = gc + self.fd = rffi.cast(rffi.INT, fd) + self.writebuffer = lltype.malloc(rffi.LONGP.TO, self.BUFSIZE, + flavor='raw') + self.buf_count = 0 + self.seen = AddressDict() + self.pending = AddressStack() + + def delete(self): + self.seen.delete() + self.pending.delete() + lltype.free(self.writebuffer, flavor='raw') + free_non_gc_object(self) + + def flush(self): + if self.buf_count > 0: + bytes = self.buf_count * rffi.sizeof(rffi.LONG) + count = raw_os_write(self.fd, + rffi.cast(llmemory.Address, self.writebuffer), + rffi.cast(rffi.SIZE_T, bytes)) + if rffi.cast(lltype.Signed, count) != bytes: + raise OSError(rposix.get_errno(), "raw_os_write failed") + self.buf_count = 0 + flush._dont_inline_ = True + + def write(self, value): + x = self.buf_count + self.writebuffer[x] = value + x += 1 + self.buf_count = x + if x == self.BUFSIZE: + self.flush() + write._always_inline_ = True + + def write_marker(self): + self.write(0) + self.write(0) + self.write(0) + self.write(-1) + + def writeobj(self, obj): + gc = self.gc + typeid = gc.get_type_id(obj) + self.write(llmemory.cast_adr_to_int(obj)) + self.write(gc.get_member_index(typeid)) + self.write(gc.get_size_incl_hash(obj)) + gc.trace(obj, self._writeref, None) + self.write(-1) + + def _writeref(self, pointer, _): + obj = pointer.address[0] + self.write(llmemory.cast_adr_to_int(obj)) + self.add(obj) + + def add(self, obj): + if not self.seen.contains(obj): + self.seen.setitem(obj, obj) + self.pending.append(obj) + + def add_roots(self): + self.gc._heap_dumper = self + self.gc.root_walker.walk_roots( + _hd_add_root, + _hd_add_root, + _hd_add_root) + self.gc._heap_dumper = None + pendingroots = self.pending + self.pending = AddressStack() + self.walk(pendingroots) + pendingroots.delete() + self.write_marker() + + def walk(self, pending): + while pending.non_empty(): + self.writeobj(pending.pop()) + +def _hd_add_root(gc, root): + gc._heap_dumper.add(root.address[0]) + +def dump_rpy_heap(gc, fd): + heapdumper = HeapDumper(gc, fd) + heapdumper.add_roots() + heapdumper.walk(heapdumper.pending) + heapdumper.flush() + heapdumper.delete() + return True From cfbolz at codespeak.net Tue Oct 12 14:15:40 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 14:15:40 +0200 (CEST) Subject: [pypy-svn] r77827 - pypy/trunk/pypy/jit/metainterp Message-ID: <20101012121540.F28E3282BF6@codespeak.net> Author: cfbolz Date: Tue Oct 12 14:15:37 2010 New Revision: 77827 Modified: pypy/trunk/pypy/jit/metainterp/optimize_nopspec.py Log: I think this is a reason why perfect specialization needs to still be run. bit annoying Modified: pypy/trunk/pypy/jit/metainterp/optimize_nopspec.py ============================================================================== --- pypy/trunk/pypy/jit/metainterp/optimize_nopspec.py (original) +++ pypy/trunk/pypy/jit/metainterp/optimize_nopspec.py Tue Oct 12 14:15:37 2010 @@ -14,6 +14,9 @@ def _optimize_loop(metainterp_sd, old_loop_tokens, loop): cpu = metainterp_sd.cpu metainterp_sd.logger_noopt.log_loop(loop.inputargs, loop.operations) + # XXX the following lines are probably still needed, to discard invalid + # loops. bit silly to run a full perfect specialization and throw the + # result away. finder = PerfectSpecializationFinder(cpu) finder.find_nodes_loop(loop, False) if old_loop_tokens: @@ -31,6 +34,7 @@ def _optimize_bridge(metainterp_sd, old_loop_tokens, bridge): cpu = metainterp_sd.cpu metainterp_sd.logger_noopt.log_loop(bridge.inputargs, bridge.operations) + # XXX same comment as above applies finder = BridgeSpecializationFinder(cpu) finder.find_nodes_bridge(bridge) if old_loop_tokens: From arigo at codespeak.net Tue Oct 12 14:38:12 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 12 Oct 2010 14:38:12 +0200 (CEST) Subject: [pypy-svn] r77828 - in pypy/trunk/pypy: config doc/config interpreter module/__builtin__ module/__builtin__/test module/_weakref module/cpyext module/pypyjit/test objspace/std objspace/std/test rlib rlib/test Message-ID: <20101012123812.ED9A5282BF6@codespeak.net> Author: arigo Date: Tue Oct 12 14:38:09 2010 New Revision: 77828 Added: pypy/trunk/pypy/doc/config/objspace.std.withmapdict.txt - copied unchanged from r77827, pypy/branch/better-map-instances/pypy/doc/config/objspace.std.withmapdict.txt pypy/trunk/pypy/objspace/std/mapdict.py - copied unchanged from r77827, pypy/branch/better-map-instances/pypy/objspace/std/mapdict.py pypy/trunk/pypy/objspace/std/test/test_mapdict.py - copied unchanged from r77827, pypy/branch/better-map-instances/pypy/objspace/std/test/test_mapdict.py pypy/trunk/pypy/rlib/rerased.py - copied unchanged from r77827, pypy/branch/better-map-instances/pypy/rlib/rerased.py pypy/trunk/pypy/rlib/test/test_rerased.py - copied unchanged from r77827, pypy/branch/better-map-instances/pypy/rlib/test/test_rerased.py Modified: pypy/trunk/pypy/config/pypyoption.py pypy/trunk/pypy/interpreter/baseobjspace.py pypy/trunk/pypy/interpreter/pycode.py pypy/trunk/pypy/interpreter/pyopcode.py pypy/trunk/pypy/interpreter/typedef.py pypy/trunk/pypy/module/__builtin__/interp_classobj.py pypy/trunk/pypy/module/__builtin__/test/test_classobj.py pypy/trunk/pypy/module/_weakref/interp__weakref.py pypy/trunk/pypy/module/cpyext/classobject.py pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py pypy/trunk/pypy/objspace/std/celldict.py pypy/trunk/pypy/objspace/std/dictmultiobject.py pypy/trunk/pypy/objspace/std/objspace.py pypy/trunk/pypy/objspace/std/sharingdict.py pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py pypy/trunk/pypy/objspace/std/test/test_shadowtracking.py pypy/trunk/pypy/objspace/std/typeobject.py Log: Merge branch/better-map-instances. Not enabled by default for the JIT yet; more clean-ups needed first. But it is enabled by default for -Omem translations, experimentally. Modified: pypy/trunk/pypy/config/pypyoption.py ============================================================================== --- pypy/trunk/pypy/config/pypyoption.py (original) +++ pypy/trunk/pypy/config/pypyoption.py Tue Oct 12 14:38:09 2010 @@ -242,6 +242,16 @@ default=False, requires=[("objspace.std.withshadowtracking", False)]), + BoolOption("withmapdict", + "make instances really small but slow without the JIT", + default=False, + requires=[("objspace.std.withshadowtracking", False), + ("objspace.std.withinlineddict", False), + ("objspace.std.withsharingdict", False), + ("objspace.std.getattributeshortcut", True), + ("objspace.std.withtypeversion", True), + ]), + BoolOption("withrangelist", "enable special range list implementation that does not " "actually create the full list until the resulting " @@ -347,7 +357,7 @@ config.objspace.std.suggest(withprebuiltint=True) config.objspace.std.suggest(withrangelist=True) config.objspace.std.suggest(withprebuiltchar=True) - config.objspace.std.suggest(withinlineddict=True) + config.objspace.std.suggest(withmapdict=True) config.objspace.std.suggest(withstrslice=True) config.objspace.std.suggest(withstrjoin=True) # xxx other options? ropes maybe? @@ -363,6 +373,7 @@ # extra optimizations with the JIT if level == 'jit': config.objspace.std.suggest(withcelldict=True) + #config.objspace.std.suggest(withmapdict=True) def enable_allworkingmodules(config): Modified: pypy/trunk/pypy/interpreter/baseobjspace.py ============================================================================== --- pypy/trunk/pypy/interpreter/baseobjspace.py (original) +++ pypy/trunk/pypy/interpreter/baseobjspace.py Tue Oct 12 14:38:09 2010 @@ -168,6 +168,20 @@ def _call_builtin_destructor(self): pass # method overridden in typedef.py + # hooks that the mapdict implementations needs: + def _get_mapdict_map(self): + return None + def _set_mapdict_map(self, map): + raise NotImplementedError + def _mapdict_read_storage(self, index): + raise NotImplementedError + def _mapdict_write_storage(self, index, value): + raise NotImplementedError + def _mapdict_storage_length(self): + raise NotImplementedError + def _set_mapdict_storage_and_map(self, storage, map): + raise NotImplementedError + class Wrappable(W_Root): """A subclass of Wrappable is an internal, interpreter-level class Modified: pypy/trunk/pypy/interpreter/pycode.py ============================================================================== --- pypy/trunk/pypy/interpreter/pycode.py (original) +++ pypy/trunk/pypy/interpreter/pycode.py Tue Oct 12 14:38:09 2010 @@ -117,6 +117,10 @@ self._compute_flatcall() + if self.space.config.objspace.std.withmapdict: + from pypy.objspace.std.mapdict import init_mapdict_cache + init_mapdict_cache(self) + def _freeze_(self): if (self.magic == cpython_magic and '__pypy__' not in sys.builtin_module_names): Modified: pypy/trunk/pypy/interpreter/pyopcode.py ============================================================================== --- pypy/trunk/pypy/interpreter/pyopcode.py (original) +++ pypy/trunk/pypy/interpreter/pyopcode.py Tue Oct 12 14:38:09 2010 @@ -710,9 +710,14 @@ def LOAD_ATTR(self, nameindex, next_instr): "obj.attributename" - w_attributename = self.getname_w(nameindex) w_obj = self.popvalue() - w_value = self.space.getattr(w_obj, w_attributename) + if (self.space.config.objspace.std.withmapdict + and not jit.we_are_jitted()): + from pypy.objspace.std.mapdict import LOAD_ATTR_caching + w_value = LOAD_ATTR_caching(self.getcode(), w_obj, nameindex) + else: + w_attributename = self.getname_w(nameindex) + w_value = self.space.getattr(w_obj, w_attributename) self.pushvalue(w_value) LOAD_ATTR._always_inline_ = True Modified: pypy/trunk/pypy/interpreter/typedef.py ============================================================================== --- pypy/trunk/pypy/interpreter/typedef.py (original) +++ pypy/trunk/pypy/interpreter/typedef.py Tue Oct 12 14:38:09 2010 @@ -133,6 +133,13 @@ typedef = cls.typedef if wants_dict and typedef.hasdict: wants_dict = False + if config.objspace.std.withmapdict and not typedef.hasdict: + # mapdict only works if the type does not already have a dict + if wants_del: + parentcls = get_unique_interplevel_subclass(config, cls, True, True, + False, True) + return _usersubclswithfeature(config, parentcls, "del") + return _usersubclswithfeature(config, cls, "user", "dict", "weakref", "slots") # Forest of if's - see the comment above. if wants_del: if wants_dict: @@ -186,10 +193,20 @@ def add(Proto): for key, value in Proto.__dict__.items(): - if not key.startswith('__') or key == '__del__': + if (not key.startswith('__') and not key.startswith('_mixin_') + or key == '__del__'): + if hasattr(value, "func_name"): + value = func_with_new_name(value, value.func_name) body[key] = value + if (config.objspace.std.withmapdict and "dict" in features): + from pypy.objspace.std.mapdict import BaseMapdictObject, ObjectMixin + add(BaseMapdictObject) + add(ObjectMixin) + features = () + if "user" in features: # generic feature needed by all subcls + class Proto(object): user_overridden_class = True @@ -255,6 +272,9 @@ wantdict = False if wantdict: + base_user_setup = supercls.user_setup.im_func + if "user_setup" in body: + base_user_setup = body["user_setup"] class Proto(object): def getdict(self): return self.w__dict__ @@ -263,11 +283,9 @@ self.w__dict__ = check_new_dictionary(space, w_dict) def user_setup(self, space, w_subtype): - self.space = space - self.w__class__ = w_subtype self.w__dict__ = space.newdict( instance=True, classofinstance=w_subtype) - self.user_setup_slots(w_subtype.nslots) + base_user_setup(self, space, w_subtype) def setclass(self, space, w_subtype): # only used by descr_set___class__ Modified: pypy/trunk/pypy/module/__builtin__/interp_classobj.py ============================================================================== --- pypy/trunk/pypy/module/__builtin__/interp_classobj.py (original) +++ pypy/trunk/pypy/module/__builtin__/interp_classobj.py Tue Oct 12 14:38:09 2010 @@ -2,9 +2,11 @@ from pypy.interpreter.error import OperationError, operationerrfmt from pypy.interpreter.gateway import ObjSpace, W_Root, NoneNotWrapped, applevel from pypy.interpreter.gateway import interp2app, ObjSpace -from pypy.interpreter.typedef import TypeDef, make_weakref_descr +from pypy.interpreter.typedef import TypeDef from pypy.interpreter.argument import Arguments from pypy.interpreter.baseobjspace import Wrappable +from pypy.interpreter.typedef import GetSetProperty, descr_get_dict +from pypy.interpreter.typedef import descr_set_dict from pypy.rlib.rarithmetic import r_uint, intmask from pypy.rlib.objectmodel import compute_identity_hash from pypy.rlib.debug import make_sure_not_resized @@ -57,6 +59,14 @@ self.bases_w = bases self.w_dict = w_dict + def instantiate(self, space): + cache = space.fromcache(Cache) + if self.lookup(space, '__del__') is not None: + w_inst = cache.cls_with_del(space, self) + else: + w_inst = cache.cls_without_del(space, self) + return w_inst + def getdict(self): return self.w_dict @@ -100,15 +110,15 @@ return False @jit.unroll_safe - def lookup(self, space, w_attr): + def lookup(self, space, attr): # returns w_value or interplevel None - w_result = space.finditem(self.w_dict, w_attr) + w_result = space.finditem_str(self.w_dict, attr) if w_result is not None: return w_result for base in self.bases_w: # XXX fix annotation of bases_w to be a list of W_ClassObjects assert isinstance(base, W_ClassObject) - w_result = base.lookup(space, w_attr) + w_result = base.lookup(space, attr) if w_result is not None: return w_result return None @@ -122,7 +132,7 @@ return space.wrap(self.name) elif name == "__bases__": return space.newtuple(self.bases_w) - w_value = self.lookup(space, w_attr) + w_value = self.lookup(space, name) if w_value is None: raise operationerrfmt( space.w_AttributeError, @@ -147,7 +157,7 @@ self.setbases(space, w_value) return elif name == "__del__": - if self.lookup(space, w_attr) is None: + if self.lookup(space, name) is None: msg = ("a __del__ method added to an existing class " "will not be called") space.warn(msg, space.w_RuntimeWarning) @@ -195,13 +205,20 @@ # NOT_RPYTHON return '' % self.name +class Cache: + def __init__(self, space): + from pypy.interpreter.typedef import _usersubclswithfeature + # evil + self.cls_without_del = _usersubclswithfeature( + space.config, W_InstanceObject, "dict", "weakref") + self.cls_with_del = _usersubclswithfeature( + space.config, self.cls_without_del, "del") + + def class_descr_call(space, w_self, __args__): self = space.interp_w(W_ClassObject, w_self) - if self.lookup(space, space.wrap('__del__')) is not None: - w_inst = W_InstanceObjectWithDel(space, self) - else: - w_inst = W_InstanceObject(space, self) - w_init = w_inst.getattr_from_class(space, space.wrap('__init__')) + w_inst = self.instantiate(space) + w_init = w_inst.getattr_from_class(space, '__init__') if w_init is not None: w_result = space.call_args(w_init, __args__) if not space.is_w(w_result, space.w_None): @@ -234,7 +251,7 @@ def make_unary_instance_method(name): def unaryop(self, space): - w_meth = self.getattr(space, space.wrap(name), True) + w_meth = self.getattr(space, name, True) return space.call_function(w_meth) unaryop.func_name = name return unaryop @@ -242,7 +259,7 @@ def make_binary_returning_notimplemented_instance_method(name): def binaryop(self, space, w_other): try: - w_meth = self.getattr(space, space.wrap(name), False) + w_meth = self.getattr(space, name, False) except OperationError, e: if e.match(space, space.w_AttributeError): return space.w_NotImplemented @@ -267,7 +284,7 @@ w_a = self w_b = w_other if w_a is self: - w_meth = self.getattr(space, space.wrap(specialname), False) + w_meth = self.getattr(space, specialname, False) if w_meth is None: return space.w_NotImplemented return space.call_function(w_meth, w_b) @@ -278,7 +295,7 @@ def rbinaryop(self, space, w_other): w_a, w_b = _coerce_helper(space, self, w_other) if w_a is None or w_a is self: - w_meth = self.getattr(space, space.wrap(rspecialname), False) + w_meth = self.getattr(space, rspecialname, False) if w_meth is None: return space.w_NotImplemented return space.call_function(w_meth, w_other) @@ -302,46 +319,34 @@ raise OperationError( space.w_TypeError, space.wrap("instance() first arg must be class")) - if space.is_w(w_dict, space.w_None): - w_dict = None - elif not space.is_true(space.isinstance(w_dict, space.w_dict)): - raise OperationError( - space.w_TypeError, - space.wrap("instance() second arg must be dictionary or None")) - return W_InstanceObject(space, w_class, w_dict) + w_result = w_class.instantiate(space) + if not space.is_w(w_dict, space.w_None): + w_result.setdict(space, w_dict) + return w_result class W_InstanceObject(Wrappable): - def __init__(self, space, w_class, w_dict=None): - if w_dict is None: - w_dict = space.newdict(instance=True) + def __init__(self, space, w_class): + # note that user_setup is overridden by the typedef.py machinery + self.user_setup(space, space.gettypeobject(self.typedef)) assert isinstance(w_class, W_ClassObject) self.w_class = w_class - self.w_dict = w_dict - self.space = space - - def getdict(self): - return self.w_dict - def setdict(self, space, w_dict): - if (w_dict is None or - not space.is_true(space.isinstance(w_dict, space.w_dict))): - raise OperationError( - space.w_TypeError, - space.wrap("__dict__ must be a dictionary object")) - self.w_dict = w_dict + def user_setup(self, space, w_subtype): + self.space = space - def setclass(self, space, w_class): + def set_oldstyle_class(self, space, w_class): if w_class is None or not isinstance(w_class, W_ClassObject): raise OperationError( space.w_TypeError, space.wrap("__class__ must be set to a class")) self.w_class = w_class - def getattr_from_class(self, space, w_name): + def getattr_from_class(self, space, name): # Look up w_name in the class dict, and call its __get__. # This method ignores the instance dict and the __getattr__. # Returns None if not found. - w_value = self.w_class.lookup(space, w_name) + assert isinstance(name, str) + w_value = self.w_class.lookup(space, name) if w_value is None: return None w_descr_get = space.lookup(w_value, '__get__') @@ -349,19 +354,20 @@ return w_value return space.call_function(w_descr_get, w_value, self, self.w_class) - def getattr(self, space, w_name, exc=True): + def getattr(self, space, name, exc=True): # Normal getattr rules: look up w_name in the instance dict, # in the class dict, and then via a call to __getatttr__. - w_result = space.finditem(self.w_dict, w_name) + assert isinstance(name, str) + w_result = self.getdictvalue(space, name) if w_result is not None: return w_result - w_result = self.getattr_from_class(space, w_name) + w_result = self.getattr_from_class(space, name) if w_result is not None: return w_result - w_meth = self.getattr_from_class(space, space.wrap('__getattr__')) + w_meth = self.getattr_from_class(space, '__getattr__') if w_meth is not None: try: - return space.call_function(w_meth, w_name) + return space.call_function(w_meth, space.wrap(name)) except OperationError, e: if not exc and e.match(space, space.w_AttributeError): return None # eat the AttributeError @@ -371,7 +377,7 @@ raise operationerrfmt( space.w_AttributeError, "%s instance has no attribute '%s'", - self.w_class.name, space.str_w(w_name)) + self.w_class.name, name) else: return None @@ -379,44 +385,46 @@ name = space.str_w(w_attr) if len(name) >= 8 and name[0] == '_': if name == "__dict__": - return self.w_dict + return self.getdict() elif name == "__class__": return self.w_class - return self.getattr(space, w_attr) + return self.getattr(space, name) def descr_setattr(self, space, w_name, w_value): name = unwrap_attr(space, w_name) - w_meth = self.getattr_from_class(space, space.wrap('__setattr__')) + w_meth = self.getattr_from_class(space, '__setattr__') if name and name[0] == "_": if name == '__dict__': self.setdict(space, w_value) return if name == '__class__': - self.setclass(space, w_value) + self.set_oldstyle_class(space, w_value) return if name == '__del__' and w_meth is None: - if (not isinstance(self, W_InstanceObjectWithDel) - and space.finditem(self.w_dict, w_name) is None): + cache = space.fromcache(Cache) + if (not isinstance(self, cache.cls_with_del) + and self.getdictvalue(space, '__del__') is None): msg = ("a __del__ method added to an instance " "with no __del__ in the class will not be called") space.warn(msg, space.w_RuntimeWarning) if w_meth is not None: space.call_function(w_meth, w_name, w_value) else: - self.setdictvalue(space, name, w_value) + # bit obscure: appease normalization + self.setdictvalue(space, name, w_value, True) def descr_delattr(self, space, w_name): name = unwrap_attr(space, w_name) if name and name[0] == "_": if name == '__dict__': # use setdict to raise the error - self.setdict(space, None) + self.setdict(space, space.w_None) return elif name == '__class__': - # use setclass to raise the error - self.setclass(space, None) + # use set_oldstyle_class to raise the error + self.set_oldstyle_class(space, None) return - w_meth = self.getattr_from_class(space, space.wrap('__delattr__')) + w_meth = self.getattr_from_class(space, '__delattr__') if w_meth is not None: space.call_function(w_meth, w_name) else: @@ -427,7 +435,7 @@ self.w_class.name, name) def descr_repr(self, space): - w_meth = self.getattr(space, space.wrap('__repr__'), False) + w_meth = self.getattr(space, '__repr__', False) if w_meth is None: w_class = self.w_class mod = w_class.get_module_string(space) @@ -435,19 +443,19 @@ return space.call_function(w_meth) def descr_str(self, space): - w_meth = self.getattr(space, space.wrap('__str__'), False) + w_meth = self.getattr(space, '__str__', False) if w_meth is None: return self.descr_repr(space) return space.call_function(w_meth) def descr_unicode(self, space): - w_meth = self.getattr(space, space.wrap('__unicode__'), False) + w_meth = self.getattr(space, '__unicode__', False) if w_meth is None: return self.descr_str(space) return space.call_function(w_meth) def descr_len(self, space): - w_meth = self.getattr(space, space.wrap('__len__')) + w_meth = self.getattr(space, '__len__') w_result = space.call_function(w_meth) if space.is_true(space.isinstance(w_result, space.w_int)): if space.is_true(space.lt(w_result, space.wrap(0))): @@ -460,22 +468,22 @@ space.wrap("__len__() should return an int")) def descr_getitem(self, space, w_key): - w_meth = self.getattr(space, space.wrap('__getitem__')) + w_meth = self.getattr(space, '__getitem__') return space.call_function(w_meth, w_key) def descr_setitem(self, space, w_key, w_value): - w_meth = self.getattr(space, space.wrap('__setitem__')) + w_meth = self.getattr(space, '__setitem__') space.call_function(w_meth, w_key, w_value) def descr_delitem(self, space, w_key): - w_meth = self.getattr(space, space.wrap('__delitem__')) + w_meth = self.getattr(space, '__delitem__') space.call_function(w_meth, w_key) def descr_iter(self, space): - w_meth = self.getattr(space, space.wrap('__iter__'), False) + w_meth = self.getattr(space, '__iter__', False) if w_meth is not None: return space.call_function(w_meth) - w_meth = self.getattr(space, space.wrap('__getitem__'), False) + w_meth = self.getattr(space, '__getitem__', False) if w_meth is None: raise OperationError( space.w_TypeError, @@ -485,14 +493,14 @@ # don't see the point def descr_getslice(self, space, w_i, w_j): - w_meth = self.getattr(space, space.wrap('__getslice__'), False) + w_meth = self.getattr(space, '__getslice__', False) if w_meth is not None: return space.call_function(w_meth, w_i, w_j) else: return space.getitem(self, space.newslice(w_i, w_j, space.w_None)) def descr_setslice(self, space, w_i, w_j, w_sequence): - w_meth = self.getattr(space, space.wrap('__setslice__'), False) + w_meth = self.getattr(space, '__setslice__', False) if w_meth is not None: space.call_function(w_meth, w_i, w_j, w_sequence) else: @@ -500,20 +508,20 @@ w_sequence) def descr_delslice(self, space, w_i, w_j): - w_meth = self.getattr(space, space.wrap('__delslice__'), False) + w_meth = self.getattr(space, '__delslice__', False) if w_meth is not None: space.call_function(w_meth, w_i, w_j) else: return space.delitem(self, space.newslice(w_i, w_j, space.w_None)) def descr_call(self, space, __args__): - w_meth = self.getattr(space, space.wrap('__call__')) + w_meth = self.getattr(space, '__call__') return space.call_args(w_meth, __args__) def descr_nonzero(self, space): - w_func = self.getattr(space, space.wrap('__nonzero__'), False) + w_func = self.getattr(space, '__nonzero__', False) if w_func is None: - w_func = self.getattr(space, space.wrap('__len__'), False) + w_func = self.getattr(space, '__len__', False) if w_func is None: return space.w_True w_result = space.call_function(w_func) @@ -537,7 +545,7 @@ not isinstance(w_b, W_InstanceObject)): return space.cmp(w_a, w_b) if isinstance(w_a, W_InstanceObject): - w_func = w_a.getattr(space, space.wrap('__cmp__'), False) + w_func = w_a.getattr(space, '__cmp__', False) if w_func is not None: w_res = space.call_function(w_func, w_b) if space.is_w(w_res, space.w_NotImplemented): @@ -556,7 +564,7 @@ return space.wrap(-1) return space.wrap(0) if isinstance(w_b, W_InstanceObject): - w_func = w_b.getattr(space, space.wrap('__cmp__'), False) + w_func = w_b.getattr(space, '__cmp__', False) if w_func is not None: w_res = space.call_function(w_func, w_a) if space.is_w(w_res, space.w_NotImplemented): @@ -577,10 +585,10 @@ return space.w_NotImplemented def descr_hash(self, space): - w_func = self.getattr(space, space.wrap('__hash__'), False) + w_func = self.getattr(space, '__hash__', False) if w_func is None: - w_eq = self.getattr(space, space.wrap('__eq__'), False) - w_cmp = self.getattr(space, space.wrap('__cmp__'), False) + w_eq = self.getattr(space, '__eq__', False) + w_cmp = self.getattr(space, '__cmp__', False) if w_eq is not None or w_cmp is not None: raise OperationError(space.w_TypeError, space.wrap("unhashable instance")) @@ -595,7 +603,7 @@ return w_ret def descr_index(self, space): - w_func = self.getattr(space, space.wrap('__index__'), False) + w_func = self.getattr(space, '__index__', False) if w_func is not None: return space.call_function(w_func) raise OperationError( @@ -603,7 +611,7 @@ space.wrap("object cannot be interpreted as an index")) def descr_contains(self, space, w_obj): - w_func = self.getattr(space, space.wrap('__contains__'), False) + w_func = self.getattr(space, '__contains__', False) if w_func is not None: return space.wrap(space.is_true(space.call_function(w_func, w_obj))) # now do it ourselves @@ -626,7 +634,7 @@ w_a = self w_b = w_other if w_a is self: - w_func = self.getattr(space, space.wrap('__pow__'), False) + w_func = self.getattr(space, '__pow__', False) if w_func is not None: return space.call_function(w_func, w_other) return space.w_NotImplemented @@ -634,7 +642,7 @@ return space.pow(w_a, w_b, space.w_None) else: # CPython also doesn't try coercion in this case - w_func = self.getattr(space, space.wrap('__pow__'), False) + w_func = self.getattr(space, '__pow__', False) if w_func is not None: return space.call_function(w_func, w_other, w_modulo) return space.w_NotImplemented @@ -646,7 +654,7 @@ w_a = self w_b = w_other if w_a is self: - w_func = self.getattr(space, space.wrap('__rpow__'), False) + w_func = self.getattr(space, '__rpow__', False) if w_func is not None: return space.call_function(w_func, w_other) return space.w_NotImplemented @@ -654,13 +662,13 @@ return space.pow(w_b, w_a, space.w_None) else: # CPython also doesn't try coercion in this case - w_func = self.getattr(space, space.wrap('__rpow__'), False) + w_func = self.getattr(space, '__rpow__', False) if w_func is not None: return space.call_function(w_func, w_other, w_modulo) return space.w_NotImplemented def descr_next(self, space): - w_func = self.getattr(space, space.wrap('next'), False) + w_func = self.getattr(space, 'next', False) if w_func is None: raise OperationError(space.w_TypeError, space.wrap("instance has no next() method")) @@ -669,10 +677,9 @@ def descr_del(self, space): # Note that this is called from executioncontext.UserDelAction # via the space.userdel() method. - w_name = space.wrap('__del__') - w_func = space.finditem(self.w_dict, w_name) + w_func = self.getdictvalue(space, '__del__') if w_func is None: - w_func = self.getattr_from_class(space, w_name) + w_func = self.getattr_from_class(space, '__del__') if w_func is not None: space.call_function(w_func) @@ -717,6 +724,14 @@ rmeth, unwrap_spec=["self", ObjSpace, W_Root]) + +def descr_del_dict(space, w_inst): + # use setdict to raise the error + w_inst.setdict(space, space.w_None) + +dict_descr = GetSetProperty(descr_get_dict, descr_set_dict, descr_del_dict) +dict_descr.name = '__dict__' + W_InstanceObject.typedef = TypeDef("instance", __new__ = interp2app(descr_instance_new), __getattribute__ = interp2app(W_InstanceObject.descr_getattribute, @@ -766,12 +781,9 @@ unwrap_spec=['self', ObjSpace, W_Root, W_Root]), next = interp2app(W_InstanceObject.descr_next, unwrap_spec=['self', ObjSpace]), - __weakref__ = make_weakref_descr(W_InstanceObject), __del__ = interp2app(W_InstanceObject.descr_del, unwrap_spec=['self', ObjSpace]), + __dict__ = dict_descr, **rawdict ) - -class W_InstanceObjectWithDel(W_InstanceObject): - def __del__(self): - self._enqueue_for_destruction(self.space) +W_InstanceObject.typedef.acceptable_as_base_class = False Modified: pypy/trunk/pypy/module/__builtin__/test/test_classobj.py ============================================================================== --- pypy/trunk/pypy/module/__builtin__/test/test_classobj.py (original) +++ pypy/trunk/pypy/module/__builtin__/test/test_classobj.py Tue Oct 12 14:38:09 2010 @@ -928,6 +928,31 @@ assert x is b assert y == 5 + def test_cant_subclass_instance(self): + class A: + pass + try: + class B(type(A())): + pass + except TypeError: + pass + else: + assert 0, "should have raised" + + def test_dict_descriptor(self): + import sys + if not hasattr(sys, 'pypy_objspaceclass'): + skip("on CPython old-style instances don't have a __dict__ descriptor") + class A: + pass + a = A() + a.x = 1 + descr = type(a).__dict__['__dict__'] + assert descr.__get__(a) == {'x': 1} + descr.__set__(a, {'x': 2}) + assert a.x == 2 + raises(TypeError, descr.__delete__, a) + class AppTestOldStyleSharing(AppTestOldstyle): def setup_class(cls): @@ -966,3 +991,22 @@ a = 1 b = 2 assert self.is_strdict(A) + +class AppTestOldStyleMapDict(AppTestOldstyle): + def setup_class(cls): + cls.space = gettestobjspace(**{"objspace.std.withmapdict": True}) + if option.runappdirect: + py.test.skip("can only be run on py.py") + def has_mapdict(space, w_inst): + return space.wrap(w_inst._get_mapdict_map() is not None) + cls.w_has_mapdict = cls.space.wrap(gateway.interp2app(has_mapdict)) + + + def test_has_mapdict(self): + class A: + def __init__(self): + self.x = 42 + a = A() + assert a.x == 42 + assert self.has_mapdict(a) + Modified: pypy/trunk/pypy/module/_weakref/interp__weakref.py ============================================================================== --- pypy/trunk/pypy/module/_weakref/interp__weakref.py (original) +++ pypy/trunk/pypy/module/_weakref/interp__weakref.py Tue Oct 12 14:38:09 2010 @@ -7,7 +7,7 @@ import weakref -class WeakrefLifeline(object): +class WeakrefLifeline(W_Root): def __init__(self, space): self.space = space # this is here for W_Root.clear_all_weakrefs() self.refs_weak = [] Modified: pypy/trunk/pypy/module/cpyext/classobject.py ============================================================================== --- pypy/trunk/pypy/module/cpyext/classobject.py (original) +++ pypy/trunk/pypy/module/cpyext/classobject.py Tue Oct 12 14:38:09 2010 @@ -15,16 +15,20 @@ class is the class of new object. The dict parameter will be used as the object's __dict__; if NULL, a new dictionary will be created for the instance.""" - if not PyClass_Check(space, w_class): + if not isinstance(w_class, W_ClassObject): return PyErr_BadInternalCall(space) - return W_InstanceObject(space, w_class, w_dict) + w_result = w_class.instantiate(space) + if w_dict is not None: + w_result.setdict(space, w_dict) + return w_result @cpython_api([PyObject, PyObject], PyObject, error=CANNOT_FAIL) def _PyInstance_Lookup(space, w_instance, w_name): + name = space.str_w(w_name) assert isinstance(w_instance, W_InstanceObject) - w_result = space.finditem(w_instance.w_dict, w_name) + w_result = w_instance.getdictvalue(space, name) if w_result is not None: return w_result - return w_instance.w_class.lookup(space, w_name) + return w_instance.w_class.lookup(space, name) Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py ============================================================================== --- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py (original) +++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py Tue Oct 12 14:38:09 2010 @@ -272,7 +272,7 @@ assert len(ops) == 2 assert not ops[0].get_opnames("call") assert not ops[0].get_opnames("new") - assert len(ops[0].get_opnames("guard")) <= 7 + assert len(ops[0].get_opnames("guard")) <= 2 assert not ops[1] # second LOOKUP_METHOD folded away ops = self.get_by_bytecode("CALL_METHOD") @@ -283,7 +283,7 @@ else: assert not bytecode.get_opnames("call") assert not bytecode.get_opnames("new") - assert len(bytecode.get_opnames("guard")) <= 9 + assert len(bytecode.get_opnames("guard")) <= 6 assert len(ops[1]) < len(ops[0]) ops = self.get_by_bytecode("LOAD_ATTR") @@ -317,8 +317,8 @@ assert len(ops) == 2 assert not ops[0].get_opnames("call") assert not ops[0].get_opnames("new") - assert len(ops[0].get_opnames("guard")) <= 7 - assert len(ops[0].get_opnames("getfield")) < 6 + assert len(ops[0].get_opnames("guard")) <= 2 + assert len(ops[0].get_opnames("getfield")) < 5 assert not ops[1] # second LOOKUP_METHOD folded away def test_default_and_kw(self): @@ -382,7 +382,7 @@ a.x = 2 i = i + a.x return i - ''', 67, + ''', 69, ([20], 20), ([31], 32)) @@ -390,7 +390,7 @@ self.get_by_bytecode("CALL_FUNCTION")) assert not callA.get_opnames("call") assert not callA.get_opnames("new") - assert len(callA.get_opnames("guard")) <= 8 + assert len(callA.get_opnames("guard")) <= 2 assert not callisinstance1.get_opnames("call") assert not callisinstance1.get_opnames("new") assert len(callisinstance1.get_opnames("guard")) <= 2 @@ -742,6 +742,8 @@ '''%(op1, float(a)/4.0, float(b)/4.0, op2), 109, ([], res)) def test_boolrewrite_ptr(self): + # XXX this test is way too imprecise in what it is actually testing + # it should count the number of guards instead compares = ('a == b', 'b == a', 'a != b', 'b != a', 'a == c', 'c != b') for e1 in compares: for e2 in compares: @@ -765,7 +767,7 @@ print print 'Test:', e1, e2, n, res self.run_source(''' - class tst: + class tst(object): pass def main(): a = tst() @@ -847,6 +849,8 @@ ''', 65, ([], 122880)) def test_array_intimg(self): + # XXX this test is way too imprecise in what it is actually testing + # it should count the number of guards instead for tc, maxops in zip('ilILd', (67, 67, 69, 69, 61)): res = 73574560 if tc in 'IL': Modified: pypy/trunk/pypy/objspace/std/celldict.py ============================================================================== --- pypy/trunk/pypy/objspace/std/celldict.py (original) +++ pypy/trunk/pypy/objspace/std/celldict.py Tue Oct 12 14:38:09 2010 @@ -45,7 +45,7 @@ if space.is_w(space.type(w_key), space.w_str): self.impl_setitem_str(self.space.str_w(w_key), w_value) else: - self._as_rdict().setitem(w_key, w_value) + self._as_rdict().impl_fallback_setitem(w_key, w_value) def impl_setitem_str(self, name, w_value, shadows_type=True): self.getcell(name, True).w_value = w_value @@ -66,7 +66,7 @@ elif _is_sane_hash(space, w_key_type): raise KeyError else: - self._as_rdict().delitem(w_key) + self._as_rdict().impl_fallback_delitem(w_key) def impl_length(self): # inefficient, but do we care? @@ -85,7 +85,7 @@ elif _is_sane_hash(space, w_lookup_type): return None else: - return self._as_rdict().getitem(w_lookup) + return self._as_rdict().impl_fallback_getitem(w_lookup) def impl_getitem_str(self, lookup): res = self.getcell(lookup, False) Modified: pypy/trunk/pypy/objspace/std/dictmultiobject.py ============================================================================== --- pypy/trunk/pypy/objspace/std/dictmultiobject.py (original) +++ pypy/trunk/pypy/objspace/std/dictmultiobject.py Tue Oct 12 14:38:09 2010 @@ -102,17 +102,17 @@ else: return None - # _________________________________________________________________ + # _________________________________________________________________ # implementation methods def impl_getitem(self, w_key): #return w_value or None raise NotImplementedError("abstract base class") - def impl_getitem_str(self, w_key): + def impl_getitem_str(self, key): #return w_value or None raise NotImplementedError("abstract base class") - def impl_setitem_str(self, key, w_value, shadows_type=True): + def impl_setitem_str(self, key, w_value, shadows_type=True): raise NotImplementedError("abstract base class") def impl_setitem(self, w_key, w_value): @@ -120,7 +120,7 @@ def impl_delitem(self, w_key): raise NotImplementedError("abstract base class") - + def impl_length(self): raise NotImplementedError("abstract base class") @@ -310,7 +310,7 @@ if space.is_w(space.type(w_key), space.w_str): self.impl_setitem_str(self.space.str_w(w_key), w_value) else: - self._as_rdict().setitem(w_key, w_value) + self._as_rdict().impl_fallback_setitem(w_key, w_value) def impl_setitem_str(self, key, w_value, shadows_type=True): self.content[key] = w_value @@ -324,7 +324,7 @@ elif _is_sane_hash(space, w_key_type): raise KeyError else: - self._as_rdict().delitem(w_key) + self._as_rdict().impl_fallback_delitem(w_key) def impl_length(self): return len(self.content) @@ -344,7 +344,7 @@ elif _is_sane_hash(space, w_lookup_type): return None else: - return self._as_rdict().getitem(w_key) + return self._as_rdict().impl_fallback_getitem(w_key) def impl_iter(self): return StrIteratorImplementation(self.space, self) @@ -414,7 +414,7 @@ StrDictImplementation.impl_setitem_str( self, self.space.str_w(w_key), w_value, False) else: - self._as_rdict().setitem(w_key, w_value) + self._as_rdict().impl_fallback_setitem(w_key, w_value) def impl_shadows_anything(self): return (self._shadows_anything or @@ -446,7 +446,7 @@ elif _is_sane_hash(space, w_key_type): raise KeyError else: - self._as_rdict().delitem(w_key) + self._as_rdict().impl_fallback_delitem(w_key) def impl_get_builtin_indexed(self, i): return self.shadowed[i] Modified: pypy/trunk/pypy/objspace/std/objspace.py ============================================================================== --- pypy/trunk/pypy/objspace/std/objspace.py (original) +++ pypy/trunk/pypy/objspace/std/objspace.py Tue Oct 12 14:38:09 2010 @@ -23,6 +23,7 @@ from pypy.objspace.std.listobject import W_ListObject from pypy.objspace.std.longobject import W_LongObject from pypy.objspace.std.noneobject import W_NoneObject +from pypy.objspace.std.objectobject import W_ObjectObject from pypy.objspace.std.ropeobject import W_RopeObject from pypy.objspace.std.iterobject import W_SeqIterObject from pypy.objspace.std.setobject import W_SetObject, W_FrozensetObject @@ -317,9 +318,14 @@ w_subtype = w_type.check_user_subclass(w_subtype) if cls.typedef.applevel_subclasses_base is not None: cls = cls.typedef.applevel_subclasses_base - subcls = get_unique_interplevel_subclass( - self.config, cls, w_subtype.hasdict, w_subtype.nslots != 0, - w_subtype.needsdel, w_subtype.weakrefable) + if (self.config.objspace.std.withmapdict and cls is W_ObjectObject + and not w_subtype.needsdel): + from pypy.objspace.std.mapdict import get_subclass_of_correct_size + subcls = get_subclass_of_correct_size(self, cls, w_subtype) + else: + subcls = get_unique_interplevel_subclass( + self.config, cls, w_subtype.hasdict, w_subtype.nslots != 0, + w_subtype.needsdel, w_subtype.weakrefable) instance = instantiate(subcls) assert isinstance(instance, cls) instance.user_setup(self, w_subtype) Modified: pypy/trunk/pypy/objspace/std/sharingdict.py ============================================================================== --- pypy/trunk/pypy/objspace/std/sharingdict.py (original) +++ pypy/trunk/pypy/objspace/std/sharingdict.py Tue Oct 12 14:38:09 2010 @@ -71,7 +71,7 @@ elif _is_sane_hash(space, w_lookup_type): return None else: - return self._as_rdict().getitem(w_lookup) + return self._as_rdict().impl_fallback_getitem(w_lookup) def impl_getitem_str(self, lookup): i = self.structure.lookup_position(lookup) @@ -84,7 +84,7 @@ if space.is_w(space.type(w_key), space.w_str): self.impl_setitem_str(self.space.str_w(w_key), w_value) else: - self._as_rdict().setitem(w_key, w_value) + self._as_rdict().impl_fallback_setitem(w_key, w_value) @unroll_safe def impl_setitem_str(self, key, w_value, shadows_type=True): @@ -132,7 +132,7 @@ elif _is_sane_hash(space, w_key_type): raise KeyError else: - self._as_rdict().delitem(w_key) + self._as_rdict().impl_fallback_delitem(w_key) def impl_length(self): return self.structure.length Modified: pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py ============================================================================== --- pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py (original) +++ pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py Tue Oct 12 14:38:09 2010 @@ -602,6 +602,15 @@ classofinstance=classofinstance, from_strdict_shared=from_strdict_shared) + def finditem_str(self, w_dict, s): + return w_dict.getitem_str(s) # assume it's a multidict + + def setitem_str(self, w_dict, s, w_value): + return w_dict.setitem_str(s, w_value) # assume it's a multidict + + def delitem(self, w_dict, w_s): + return w_dict.delitem(w_s) # assume it's a multidict + def allocate_instance(self, cls, type): return object.__new__(cls) @@ -611,7 +620,7 @@ w_StopIteration = StopIteration w_None = None StringObjectCls = FakeString - w_dict = None + w_dict = W_DictMultiObject iter = iter fixedview = list listview = list @@ -687,6 +696,14 @@ assert self.impl.length() == 0 self.check_not_devolved() + def test_clear(self): + self.fill_impl() + assert self.impl.length() == 2 + self.impl.clear() + assert self.impl.length() == 0 + self.check_not_devolved() + + def test_keys(self): self.fill_impl() keys = self.impl.keys() Modified: pypy/trunk/pypy/objspace/std/test/test_shadowtracking.py ============================================================================== --- pypy/trunk/pypy/objspace/std/test/test_shadowtracking.py (original) +++ pypy/trunk/pypy/objspace/std/test/test_shadowtracking.py Tue Oct 12 14:38:09 2010 @@ -3,7 +3,8 @@ class TestShadowTracking(object): def setup_class(cls): - cls.space = gettestobjspace(**{"objspace.std.withshadowtracking": True}) + cls.space = gettestobjspace(**{"objspace.std.withshadowtracking": True, + "objspace.std.withmapdict": False}) def test_simple_shadowing(self): space = self.space Modified: pypy/trunk/pypy/objspace/std/typeobject.py ============================================================================== --- pypy/trunk/pypy/objspace/std/typeobject.py (original) +++ pypy/trunk/pypy/objspace/std/typeobject.py Tue Oct 12 14:38:09 2010 @@ -75,7 +75,9 @@ 'weakrefable', 'hasdict', 'nslots', - 'instancetypedef'] + 'instancetypedef', + 'terminator', + ] # for config.objspace.std.getattributeshortcut # (False is a conservative default, fixed during real usage) @@ -116,6 +118,12 @@ # dict_w of any of the types in the mro changes, or if the mro # itself changes w_self._version_tag = VersionTag() + if space.config.objspace.std.withmapdict: + from pypy.objspace.std.mapdict import DictTerminator, NoDictTerminator + if w_self.hasdict: + w_self.terminator = DictTerminator(space, w_self) + else: + w_self.terminator = NoDictTerminator(space, w_self) def mutated(w_self): space = w_self.space From arigo at codespeak.net Tue Oct 12 14:38:26 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 12 Oct 2010 14:38:26 +0200 (CEST) Subject: [pypy-svn] r77829 - pypy/branch/better-map-instances Message-ID: <20101012123826.643A8282BF7@codespeak.net> Author: arigo Date: Tue Oct 12 14:38:24 2010 New Revision: 77829 Removed: pypy/branch/better-map-instances/ Log: Remove merged branch. From arigo at codespeak.net Tue Oct 12 14:43:05 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 12 Oct 2010 14:43:05 +0200 (CEST) Subject: [pypy-svn] r77830 - in pypy/branch/32ptr-on-64bit/pypy: annotation annotation/test config doc doc/config interpreter interpreter/astcompiler/test interpreter/pyparser jit/backend/test jit/backend/x86 jit/backend/x86/test jit/codewriter jit/codewriter/test jit/metainterp jit/metainterp/optimizeopt jit/metainterp/test jit/tl/spli module/__builtin__ module/__builtin__/test module/_weakref module/cpyext module/gc module/pypyjit/test module/signal module/sys objspace/std objspace/std/test rlib rlib/test rpython rpython/lltypesystem rpython/module rpython/test translator/c/gcc translator/c/gcc/test/elf translator/c/src translator/goal translator/jvm/test translator/oosupport/test_template Message-ID: <20101012124305.E63D8282BF6@codespeak.net> Author: arigo Date: Tue Oct 12 14:42:59 2010 New Revision: 77830 Added: pypy/branch/32ptr-on-64bit/pypy/doc/config/objspace.std.withmapdict.txt - copied unchanged from r77829, pypy/trunk/pypy/doc/config/objspace.std.withmapdict.txt pypy/branch/32ptr-on-64bit/pypy/interpreter/pyparser/autopath.py - copied unchanged from r77829, pypy/trunk/pypy/interpreter/pyparser/autopath.py pypy/branch/32ptr-on-64bit/pypy/interpreter/pyparser/genpytokenize.py - copied unchanged from r77829, pypy/trunk/pypy/interpreter/pyparser/genpytokenize.py pypy/branch/32ptr-on-64bit/pypy/interpreter/pyparser/pylexer.py - copied unchanged from r77829, pypy/trunk/pypy/interpreter/pyparser/pylexer.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py - copied unchanged from r77829, pypy/trunk/pypy/objspace/std/mapdict.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/strbufobject.py - copied unchanged from r77829, pypy/trunk/pypy/objspace/std/strbufobject.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_mapdict.py - copied unchanged from r77829, pypy/trunk/pypy/objspace/std/test/test_mapdict.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_strbufobject.py - copied unchanged from r77829, pypy/trunk/pypy/objspace/std/test/test_strbufobject.py pypy/branch/32ptr-on-64bit/pypy/rlib/rerased.py - copied unchanged from r77829, pypy/trunk/pypy/rlib/rerased.py pypy/branch/32ptr-on-64bit/pypy/rlib/test/test_rerased.py - copied unchanged from r77829, pypy/trunk/pypy/rlib/test/test_rerased.py pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf/track10.s - copied unchanged from r77829, pypy/trunk/pypy/translator/c/gcc/test/elf/track10.s pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf/track11.s - copied unchanged from r77829, pypy/trunk/pypy/translator/c/gcc/test/elf/track11.s Removed: pypy/branch/32ptr-on-64bit/pypy/rpython/rspecialcase.py pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rspecialcase.py Modified: pypy/branch/32ptr-on-64bit/pypy/annotation/policy.py pypy/branch/32ptr-on-64bit/pypy/annotation/test/test_annrpython.py pypy/branch/32ptr-on-64bit/pypy/config/pypyoption.py pypy/branch/32ptr-on-64bit/pypy/doc/docindex.txt pypy/branch/32ptr-on-64bit/pypy/interpreter/astcompiler/test/test_compiler.py pypy/branch/32ptr-on-64bit/pypy/interpreter/baseobjspace.py pypy/branch/32ptr-on-64bit/pypy/interpreter/pycode.py pypy/branch/32ptr-on-64bit/pypy/interpreter/pyframe.py pypy/branch/32ptr-on-64bit/pypy/interpreter/pyopcode.py pypy/branch/32ptr-on-64bit/pypy/interpreter/pyparser/pytokenize.py pypy/branch/32ptr-on-64bit/pypy/interpreter/typedef.py pypy/branch/32ptr-on-64bit/pypy/jit/backend/test/runner_test.py pypy/branch/32ptr-on-64bit/pypy/jit/backend/test/test_ll_random.py pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/assembler.py pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/regalloc.py pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/test/test_runner.py pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/test/test_string.py pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/effectinfo.py pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/jtransform.py pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/test/test_jtransform.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/history.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimize_nopspec.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/intbounds.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/rewrite.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/string.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/resume.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/simple_optimize.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/oparser.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_optimizefindnode.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_optimizeopt.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_resume.py pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_string.py pypy/branch/32ptr-on-64bit/pypy/jit/tl/spli/interpreter.py pypy/branch/32ptr-on-64bit/pypy/module/__builtin__/interp_classobj.py pypy/branch/32ptr-on-64bit/pypy/module/__builtin__/test/test_classobj.py pypy/branch/32ptr-on-64bit/pypy/module/_weakref/interp__weakref.py pypy/branch/32ptr-on-64bit/pypy/module/cpyext/classobject.py pypy/branch/32ptr-on-64bit/pypy/module/gc/referents.py pypy/branch/32ptr-on-64bit/pypy/module/pypyjit/test/test_pypy_c.py pypy/branch/32ptr-on-64bit/pypy/module/signal/interp_signal.py pypy/branch/32ptr-on-64bit/pypy/module/sys/__init__.py pypy/branch/32ptr-on-64bit/pypy/module/sys/state.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/celldict.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/dictmultiobject.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/fake.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/model.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/objspace.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/sharingdict.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/stringobject.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/stringtype.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_dictmultiobject.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_shadowtracking.py pypy/branch/32ptr-on-64bit/pypy/objspace/std/typeobject.py pypy/branch/32ptr-on-64bit/pypy/rlib/_rsocket_rffi.py pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py pypy/branch/32ptr-on-64bit/pypy/rlib/rstring.py pypy/branch/32ptr-on-64bit/pypy/rlib/test/test_rsocket.py pypy/branch/32ptr-on-64bit/pypy/rlib/test/test_rstring.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rbuilder.py pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rclass.py pypy/branch/32ptr-on-64bit/pypy/rpython/module/ll_time.py pypy/branch/32ptr-on-64bit/pypy/rpython/rbuilder.py pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rbuilder.py pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rclass.py pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf/track5.s pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/trackgcroot.py pypy/branch/32ptr-on-64bit/pypy/translator/c/src/signals.h pypy/branch/32ptr-on-64bit/pypy/translator/goal/ann_override.py pypy/branch/32ptr-on-64bit/pypy/translator/jvm/test/test_class.py pypy/branch/32ptr-on-64bit/pypy/translator/oosupport/test_template/class_.py Log: Merge trunk up to r77829. Modified: pypy/branch/32ptr-on-64bit/pypy/annotation/policy.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/annotation/policy.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/annotation/policy.py Tue Oct 12 14:42:59 2010 @@ -1,4 +1,4 @@ -# base annotation policy for overrides and specialization +# base annotation policy for specialization from pypy.annotation.specialize import default_specialize as default from pypy.annotation.specialize import specialize_argvalue, specialize_argtype, specialize_arglistitemtype from pypy.annotation.specialize import memo @@ -41,7 +41,7 @@ if directive is None: return pol.default_specialize - # specialize|override:name[(args)] + # specialize[(args)] directive_parts = directive.split('(', 1) if len(directive_parts) == 1: [name] = directive_parts @@ -60,14 +60,6 @@ except AttributeError: raise AttributeError("%r specialize tag not defined in annotation" "policy %s" % (name, pol)) - if directive.startswith('override:'): - # different signature: override__xyz(*args_s) - if parms: - raise Exception, "override:* specialisations don't support parameters" - def specialize_override(funcdesc, args_s): - funcdesc.overridden = True - return specializer(*args_s) - return specialize_override else: if not parms: return specializer @@ -92,9 +84,5 @@ from pypy.rpython.annlowlevel import LowLevelAnnotatorPolicy return LowLevelAnnotatorPolicy.specialize__ll_and_arg(*args) - def override__ignore(pol, *args): - bk = getbookkeeper() - return bk.immutablevalue(None) - class StrictAnnotatorPolicy(AnnotatorPolicy): allow_someobjects = False Modified: pypy/branch/32ptr-on-64bit/pypy/annotation/test/test_annrpython.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/annotation/test/test_annrpython.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/annotation/test/test_annrpython.py Tue Oct 12 14:42:59 2010 @@ -766,28 +766,6 @@ s = a.build_types(f, [list]) assert s.classdef is a.bookkeeper.getuniqueclassdef(IndexError) # KeyError ignored because l is a list - def test_overrides(self): - excs = [] - def record_exc(e): - """NOT_RPYTHON""" - excs.append(sys.exc_info) - record_exc._annspecialcase_ = "override:record_exc" - def g(): - pass - def f(): - try: - g() - except Exception, e: - record_exc(e) - class MyAnnotatorPolicy(policy.AnnotatorPolicy): - - def override__record_exc(pol, s_e): - return a.bookkeeper.immutablevalue(None) - - a = self.RPythonAnnotator(policy=MyAnnotatorPolicy()) - s = a.build_types(f, []) - assert s.const is None - def test_freeze_protocol(self): class Stuff: def __init__(self, flag): Modified: pypy/branch/32ptr-on-64bit/pypy/config/pypyoption.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/config/pypyoption.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/config/pypyoption.py Tue Oct 12 14:42:59 2010 @@ -198,6 +198,9 @@ BoolOption("withstrslice", "use strings optimized for slicing", default=False), + BoolOption("withstrbuf", "use strings optimized for addition (ver 2)", + default=False), + BoolOption("withprebuiltchar", "use prebuilt single-character string objects", default=False), @@ -210,7 +213,8 @@ BoolOption("withrope", "use ropes as the string implementation", default=False, requires=[("objspace.std.withstrslice", False), - ("objspace.std.withstrjoin", False)], + ("objspace.std.withstrjoin", False), + ("objspace.std.withstrbuf", False)], suggests=[("objspace.std.withprebuiltchar", True), ("objspace.std.sharesmallstr", True)]), @@ -238,6 +242,16 @@ default=False, requires=[("objspace.std.withshadowtracking", False)]), + BoolOption("withmapdict", + "make instances really small but slow without the JIT", + default=False, + requires=[("objspace.std.withshadowtracking", False), + ("objspace.std.withinlineddict", False), + ("objspace.std.withsharingdict", False), + ("objspace.std.getattributeshortcut", True), + ("objspace.std.withtypeversion", True), + ]), + BoolOption("withrangelist", "enable special range list implementation that does not " "actually create the full list until the resulting " @@ -343,7 +357,7 @@ config.objspace.std.suggest(withprebuiltint=True) config.objspace.std.suggest(withrangelist=True) config.objspace.std.suggest(withprebuiltchar=True) - config.objspace.std.suggest(withinlineddict=True) + config.objspace.std.suggest(withmapdict=True) config.objspace.std.suggest(withstrslice=True) config.objspace.std.suggest(withstrjoin=True) # xxx other options? ropes maybe? @@ -359,6 +373,7 @@ # extra optimizations with the JIT if level == 'jit': config.objspace.std.suggest(withcelldict=True) + #config.objspace.std.suggest(withmapdict=True) def enable_allworkingmodules(config): Modified: pypy/branch/32ptr-on-64bit/pypy/doc/docindex.txt ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/doc/docindex.txt (original) +++ pypy/branch/32ptr-on-64bit/pypy/doc/docindex.txt Tue Oct 12 14:42:59 2010 @@ -84,7 +84,7 @@ PyPy's own tests `summary`_, daily updated, run through BuildBot infrastructure. You can also find CPython's compliance tests run with compiled ``pypy-c`` -exeuctables there. +executables there. information dating from early 2007: Modified: pypy/branch/32ptr-on-64bit/pypy/interpreter/astcompiler/test/test_compiler.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/interpreter/astcompiler/test/test_compiler.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/interpreter/astcompiler/test/test_compiler.py Tue Oct 12 14:42:59 2010 @@ -41,7 +41,7 @@ co_expr = compile(evalexpr, '', 'eval') space = self.space pyco_expr = PyCode._from_code(space, co_expr) - w_res = pyco_expr.exec_code(space, w_dict, w_dict) + w_res = pyco_expr.exec_host_bytecode(space, w_dict, w_dict) res = space.str_w(space.repr(w_res)) if not isinstance(expected, float): assert res == repr(expected) Modified: pypy/branch/32ptr-on-64bit/pypy/interpreter/baseobjspace.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/interpreter/baseobjspace.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/interpreter/baseobjspace.py Tue Oct 12 14:42:59 2010 @@ -168,6 +168,20 @@ def _call_builtin_destructor(self): pass # method overridden in typedef.py + # hooks that the mapdict implementations needs: + def _get_mapdict_map(self): + return None + def _set_mapdict_map(self, map): + raise NotImplementedError + def _mapdict_read_storage(self, index): + raise NotImplementedError + def _mapdict_write_storage(self, index, value): + raise NotImplementedError + def _mapdict_storage_length(self): + raise NotImplementedError + def _set_mapdict_storage_and_map(self, storage, map): + raise NotImplementedError + class Wrappable(W_Root): """A subclass of Wrappable is an internal, interpreter-level class Modified: pypy/branch/32ptr-on-64bit/pypy/interpreter/pycode.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/interpreter/pycode.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/interpreter/pycode.py Tue Oct 12 14:42:59 2010 @@ -117,6 +117,10 @@ self._compute_flatcall() + if self.space.config.objspace.std.withmapdict: + from pypy.objspace.std.mapdict import init_mapdict_cache + init_mapdict_cache(self) + def _freeze_(self): if (self.magic == cpython_magic and '__pypy__' not in sys.builtin_module_names): @@ -253,6 +257,12 @@ tuple(self.co_freevars), tuple(self.co_cellvars) ) + def exec_host_bytecode(self, w_dict, w_globals, w_locals): + from pypy.interpreter.pyframe import CPythonFrame + frame = CPythonFrame(self.space, self, w_globals, None) + frame.setdictscope(w_locals) + return frame.run() + def dump(self): """A dis.dis() dump of the code object.""" co = self._to_code() Modified: pypy/branch/32ptr-on-64bit/pypy/interpreter/pyframe.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/interpreter/pyframe.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/interpreter/pyframe.py Tue Oct 12 14:42:59 2010 @@ -13,6 +13,7 @@ from pypy.rlib.rarithmetic import intmask from pypy.rlib import jit, rstack from pypy.tool import stdlib_opcode +from pypy.tool.stdlib_opcode import host_bytecode_spec # Define some opcodes used g = globals() @@ -140,7 +141,8 @@ # the following 'assert' is an annotation hint: it hides from # the annotator all methods that are defined in PyFrame but # overridden in the {,Host}FrameClass subclasses of PyFrame. - assert isinstance(self, self.space.FrameClass) + assert (isinstance(self, self.space.FrameClass) or + not self.space.config.translating) executioncontext = self.space.getexecutioncontext() executioncontext.enter(self) try: @@ -634,6 +636,18 @@ return space.wrap(self.builtin is not space.builtin) return space.w_False +class CPythonFrame(PyFrame): + """ + Execution of host (CPython) opcodes. + """ + + bytecode_spec = host_bytecode_spec + opcode_method_names = host_bytecode_spec.method_names + opcodedesc = host_bytecode_spec.opcodedesc + opdescmap = host_bytecode_spec.opdescmap + HAVE_ARGUMENT = host_bytecode_spec.HAVE_ARGUMENT + + # ____________________________________________________________ def get_block_class(opname): Modified: pypy/branch/32ptr-on-64bit/pypy/interpreter/pyopcode.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/interpreter/pyopcode.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/interpreter/pyopcode.py Tue Oct 12 14:42:59 2010 @@ -710,9 +710,14 @@ def LOAD_ATTR(self, nameindex, next_instr): "obj.attributename" - w_attributename = self.getname_w(nameindex) w_obj = self.popvalue() - w_value = self.space.getattr(w_obj, w_attributename) + if (self.space.config.objspace.std.withmapdict + and not jit.we_are_jitted()): + from pypy.objspace.std.mapdict import LOAD_ATTR_caching + w_value = LOAD_ATTR_caching(self.getcode(), w_obj, nameindex) + else: + w_attributename = self.getname_w(nameindex) + w_value = self.space.getattr(w_obj, w_attributename) self.pushvalue(w_value) LOAD_ATTR._always_inline_ = True Modified: pypy/branch/32ptr-on-64bit/pypy/interpreter/pyparser/pytokenize.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/interpreter/pyparser/pytokenize.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/interpreter/pyparser/pytokenize.py Tue Oct 12 14:42:59 2010 @@ -21,40 +21,42 @@ __all__ = [ "tokenize" ] # ______________________________________________________________________ -# Automatically generated DFA's (with one or two hand tweeks): -pseudoStatesAccepts = [True, True, True, True, True, True, True, True, - True, True, False, True, True, True, False, False, - False, False, True, False, False, True, True, False, - True, False, True, False, True, False, True, False, - False, False, True, False, False, False, True] - -pseudoStates = [ - {'\t': 0, '\n': 13, '\x0c': 0, '\r': 14, ' ': 0, '!': 10, - '"': 16, '#': 18, '%': 12, '&': 12, - "'": 15, '(': 13, ')': 13, '*': 7, - '+': 12, ',': 13, '-': 12, '.': 6, - '/': 11, '0': 4, '1': 5, '2': 5, - '3': 5, '4': 5, '5': 5, '6': 5, - '7': 5, '8': 5, '9': 5, ':': 13, - ';': 13, '<': 9, '=': 12, '>': 8, - '@': 13, 'A': 1, - 'B': 1, 'C': 1, 'D': 1, 'E': 1, - 'F': 1, 'G': 1, 'H': 1, 'I': 1, - 'J': 1, 'K': 1, 'L': 1, 'M': 1, - 'N': 1, 'O': 1, 'P': 1, 'Q': 1, - 'R': 2, 'S': 1, 'T': 1, 'U': 3, - 'V': 1, 'W': 1, 'X': 1, 'Y': 1, - 'Z': 1, '[': 13, '\\': 17, ']': 13, - '^': 12, '_': 1, '`': 13, 'a': 1, - 'b': 1, 'c': 1, 'd': 1, 'e': 1, - 'f': 1, 'g': 1, 'h': 1, 'i': 1, - 'j': 1, 'k': 1, 'l': 1, 'm': 1, - 'n': 1, 'o': 1, 'p': 1, 'q': 1, - 'r': 2, 's': 1, 't': 1, 'u': 3, - 'v': 1, 'w': 1, 'x': 1, 'y': 1, - 'z': 1, '{': 13, '|': 12, '}': 13, - '~': 13}, +# Automatically generated DFA's +accepts = [True, True, True, True, True, True, True, True, + True, True, False, True, True, True, False, False, + False, False, True, True, True, False, True, + False, True, False, True, False, False, True, + False, False, False, False, True, False, False, + False, True] +states = [ + # 0 + {'\t': 0, '\n': 13, '\x0c': 0, + '\r': 14, ' ': 0, '!': 10, '"': 16, + '#': 18, '%': 12, '&': 12, "'": 15, + '(': 13, ')': 13, '*': 7, '+': 12, + ',': 13, '-': 12, '.': 6, '/': 11, + '0': 4, '1': 5, '2': 5, '3': 5, + '4': 5, '5': 5, '6': 5, '7': 5, + '8': 5, '9': 5, ':': 13, ';': 13, + '<': 9, '=': 12, '>': 8, '@': 13, + 'A': 1, 'B': 1, 'C': 1, 'D': 1, + 'E': 1, 'F': 1, 'G': 1, 'H': 1, + 'I': 1, 'J': 1, 'K': 1, 'L': 1, + 'M': 1, 'N': 1, 'O': 1, 'P': 1, + 'Q': 1, 'R': 2, 'S': 1, 'T': 1, + 'U': 3, 'V': 1, 'W': 1, 'X': 1, + 'Y': 1, 'Z': 1, '[': 13, '\\': 17, + ']': 13, '^': 12, '_': 1, '`': 13, + 'a': 1, 'b': 1, 'c': 1, 'd': 1, + 'e': 1, 'f': 1, 'g': 1, 'h': 1, + 'i': 1, 'j': 1, 'k': 1, 'l': 1, + 'm': 1, 'n': 1, 'o': 1, 'p': 1, + 'q': 1, 'r': 2, 's': 1, 't': 1, + 'u': 3, 'v': 1, 'w': 1, 'x': 1, + 'y': 1, 'z': 1, '{': 13, '|': 12, + '}': 13, '~': 13}, + # 1 {'0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, 'A': 1, 'B': 1, @@ -71,7 +73,7 @@ 'p': 1, 'q': 1, 'r': 1, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - + # 2 {'"': 16, "'": 15, '0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, @@ -89,7 +91,7 @@ 'r': 1, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - + # 3 {'"': 16, "'": 15, '0': 1, '1': 1, '2': 1, '3': 1, '4': 1, '5': 1, '6': 1, '7': 1, '8': 1, '9': 1, @@ -107,158 +109,182 @@ 'r': 2, 's': 1, 't': 1, 'u': 1, 'v': 1, 'w': 1, 'x': 1, 'y': 1, 'z': 1}, - - {'.': 24, '0': 22, '1': 22, '2': 22, - '3': 22, '4': 22, '5': 22, '6': 22, - '7': 22, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'L': 13, 'X': 21, 'e': 25, - 'j': 13, 'l': 13, 'x': 21}, - - {'.': 24, '0': 5, '1': 5, '2': 5, + # 4 + {'.': 22, '0': 20, '1': 20, '2': 20, + '3': 20, '4': 20, '5': 20, '6': 20, + '7': 20, '8': 21, '9': 21, 'E': 23, + 'J': 13, 'L': 13, 'X': 19, 'e': 23, + 'j': 13, 'l': 13, 'x': 19}, + # 5 + {'.': 22, '0': 5, '1': 5, '2': 5, '3': 5, '4': 5, '5': 5, '6': 5, - '7': 5, '8': 5, '9': 5, 'E': 25, - 'J': 13, 'L': 13, 'e': 25, 'j': 13, + '7': 5, '8': 5, '9': 5, 'E': 23, + 'J': 13, 'L': 13, 'e': 23, 'j': 13, 'l': 13}, - - {'0': 26, '1': 26, '2': 26, '3': 26, - '4': 26, '5': 26, '6': 26, '7': 26, - '8': 26, '9': 26}, - + # 6 + {'0': 24, '1': 24, '2': 24, '3': 24, + '4': 24, '5': 24, '6': 24, '7': 24, + '8': 24, '9': 24}, + # 7 {'*': 12, '=': 13}, - + # 8 {'=': 13, '>': 12}, - - {'=': 13, '<': 12, '>': 13}, - + # 9 + {'<': 12, '=': 13, '>': 13}, + # 10 {'=': 13}, - - {'=': 13, '/': 12}, - + # 11 + {'/': 12, '=': 13}, + # 12 {'=': 13}, - + # 13 {}, - + # 14 {'\n': 13}, - - {automata.DEFAULT: 19, '\n': 27, '\\': 29, "'": 28}, - - {automata.DEFAULT: 20, '"': 30, '\n': 27, '\\': 31}, - + # 15 + {automata.DEFAULT: 28, '\n': 25, "'": 26, '\\': 27}, + # 16 + {automata.DEFAULT: 31, '\n': 25, '"': 29, '\\': 30}, + # 17 {'\n': 13, '\r': 14}, - - {automata.DEFAULT: 18, '\n': 27, '\r': 27}, - - {automata.DEFAULT: 19, '\n': 27, '\\': 29, "'": 13}, - - {automata.DEFAULT: 20, '"': 13, '\n': 27, '\\': 31}, - - {'0': 21, '1': 21, '2': 21, '3': 21, - '4': 21, '5': 21, '6': 21, '7': 21, - '8': 21, '9': 21, 'A': 21, 'B': 21, - 'C': 21, 'D': 21, 'E': 21, 'F': 21, - 'L': 13, 'a': 21, 'b': 21, 'c': 21, - 'd': 21, 'e': 21, 'f': 21, 'l': 13}, - - {'.': 24, '0': 22, '1': 22, '2': 22, - '3': 22, '4': 22, '5': 22, '6': 22, - '7': 22, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'L': 13, 'e': 25, 'j': 13, + # 18 + {automata.DEFAULT: 18, '\n': 25, '\r': 25}, + # 19 + {'0': 19, '1': 19, '2': 19, '3': 19, + '4': 19, '5': 19, '6': 19, '7': 19, + '8': 19, '9': 19, 'A': 19, 'B': 19, + 'C': 19, 'D': 19, 'E': 19, 'F': 19, + 'L': 13, 'a': 19, 'b': 19, 'c': 19, + 'd': 19, 'e': 19, 'f': 19, 'l': 13}, + # 20 + {'.': 22, '0': 20, '1': 20, '2': 20, + '3': 20, '4': 20, '5': 20, '6': 20, + '7': 20, '8': 21, '9': 21, 'E': 23, + 'J': 13, 'L': 13, 'e': 23, 'j': 13, 'l': 13}, - - {'.': 24, '0': 23, '1': 23, '2': 23, - '3': 23, '4': 23, '5': 23, '6': 23, - '7': 23, '8': 23, '9': 23, 'E': 25, - 'J': 13, 'e': 25, 'j': 13}, - - {'0': 24, '1': 24, '2': 24, '3': 24, - '4': 24, '5': 24, '6': 24, '7': 24, - '8': 24, '9': 24, 'E': 32, 'J': 13, + # 21 + {'.': 22, '0': 21, '1': 21, '2': 21, + '3': 21, '4': 21, '5': 21, '6': 21, + '7': 21, '8': 21, '9': 21, 'E': 23, + 'J': 13, 'e': 23, 'j': 13}, + # 22 + {'0': 22, '1': 22, '2': 22, '3': 22, + '4': 22, '5': 22, '6': 22, '7': 22, + '8': 22, '9': 22, 'E': 32, 'J': 13, 'e': 32, 'j': 13}, - + # 23 {'+': 33, '-': 33, '0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34}, - - {'0': 26, '1': 26, '2': 26, '3': 26, - '4': 26, '5': 26, '6': 26, '7': 26, - '8': 26, '9': 26, 'E': 32, 'J': 13, + # 24 + {'0': 24, '1': 24, '2': 24, '3': 24, + '4': 24, '5': 24, '6': 24, '7': 24, + '8': 24, '9': 24, 'E': 32, 'J': 13, 'e': 32, 'j': 13}, - + # 25 {}, - + # 26 {"'": 13}, - + # 27 {automata.DEFAULT: 35, '\n': 13, '\r': 14}, - + # 28 + {automata.DEFAULT: 28, '\n': 25, "'": 13, '\\': 27}, + # 29 {'"': 13}, - + # 30 {automata.DEFAULT: 36, '\n': 13, '\r': 14}, - + # 31 + {automata.DEFAULT: 31, '\n': 25, '"': 13, '\\': 30}, + # 32 {'+': 37, '-': 37, '0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38}, - - + # 33 {'0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34}, - + # 34 {'0': 34, '1': 34, '2': 34, '3': 34, '4': 34, '5': 34, '6': 34, '7': 34, '8': 34, '9': 34, 'J': 13, 'j': 13}, - - {automata.DEFAULT: 35, '\n': 27, '\\': 29, "'": 13}, - - {automata.DEFAULT: 36, '"': 13, '\n': 27, '\\': 31}, - + # 35 + {automata.DEFAULT: 35, '\n': 25, "'": 13, '\\': 27}, + # 36 + {automata.DEFAULT: 36, '\n': 25, '"': 13, '\\': 30}, + # 37 {'0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38}, - + # 38 {'0': 38, '1': 38, '2': 38, '3': 38, '4': 38, '5': 38, '6': 38, '7': 38, '8': 38, '9': 38, 'J': 13, 'j': 13}, ] +pseudoDFA = automata.DFA(states, accepts) -pseudoDFA = automata.DFA(pseudoStates, pseudoStatesAccepts) - -double3StatesAccepts = [False, False, False, False, False, True] -double3States = [ +accepts = [False, False, False, False, False, True] +states = [ + # 0 {automata.DEFAULT: 0, '"': 1, '\\': 2}, + # 1 {automata.DEFAULT: 4, '"': 3, '\\': 2}, + # 2 {automata.DEFAULT: 4}, + # 3 {automata.DEFAULT: 4, '"': 5, '\\': 2}, + # 4 {automata.DEFAULT: 4, '"': 1, '\\': 2}, + # 5 {automata.DEFAULT: 4, '"': 5, '\\': 2}, ] -double3DFA = automata.NonGreedyDFA(double3States, double3StatesAccepts) +double3DFA = automata.NonGreedyDFA(states, accepts) -single3StatesAccepts = [False, False, False, False, False, True] -single3States = [ - {automata.DEFAULT: 0, '\\': 2, "'": 1}, - {automata.DEFAULT: 4, '\\': 2, "'": 3}, +accepts = [False, False, False, False, False, True] +states = [ + # 0 + {automata.DEFAULT: 0, "'": 1, '\\': 2}, + # 1 + {automata.DEFAULT: 4, "'": 3, '\\': 2}, + # 2 {automata.DEFAULT: 4}, - {automata.DEFAULT: 4, '\\': 2, "'": 5}, - {automata.DEFAULT: 4, '\\': 2, "'": 1}, - {automata.DEFAULT: 4, '\\': 2, "'": 5}, + # 3 + {automata.DEFAULT: 4, "'": 5, '\\': 2}, + # 4 + {automata.DEFAULT: 4, "'": 1, '\\': 2}, + # 5 + {automata.DEFAULT: 4, "'": 5, '\\': 2}, ] -single3DFA = automata.NonGreedyDFA(single3States, single3StatesAccepts) +single3DFA = automata.NonGreedyDFA(states, accepts) -singleStatesAccepts = [False, True, False] -singleStates = [ - {automata.DEFAULT: 0, '\\': 2, "'": 1}, +accepts = [False, True, False, False] +states = [ + # 0 + {automata.DEFAULT: 0, "'": 1, '\\': 2}, + # 1 {}, - {automata.DEFAULT: 0}, + # 2 + {automata.DEFAULT: 3}, + # 3 + {automata.DEFAULT: 3, "'": 1, '\\': 2}, ] -singleDFA = automata.DFA(singleStates, singleStatesAccepts) +singleDFA = automata.DFA(states, accepts) -doubleStatesAccepts = [False, True, False] -doubleStates = [ +accepts = [False, True, False, False] +states = [ + # 0 {automata.DEFAULT: 0, '"': 1, '\\': 2}, + # 1 {}, - {automata.DEFAULT: 0}, + # 2 + {automata.DEFAULT: 3}, + # 3 + {automata.DEFAULT: 3, '"': 1, '\\': 2}, ] -doubleDFA = automata.DFA(doubleStates, doubleStatesAccepts) +doubleDFA = automata.DFA(states, accepts) + + +#_______________________________________________________________________ +# End of automatically generated DFA's endDFAs = {"'" : singleDFA, '"' : doubleDFA, Modified: pypy/branch/32ptr-on-64bit/pypy/interpreter/typedef.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/interpreter/typedef.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/interpreter/typedef.py Tue Oct 12 14:42:59 2010 @@ -133,6 +133,13 @@ typedef = cls.typedef if wants_dict and typedef.hasdict: wants_dict = False + if config.objspace.std.withmapdict and not typedef.hasdict: + # mapdict only works if the type does not already have a dict + if wants_del: + parentcls = get_unique_interplevel_subclass(config, cls, True, True, + False, True) + return _usersubclswithfeature(config, parentcls, "del") + return _usersubclswithfeature(config, cls, "user", "dict", "weakref", "slots") # Forest of if's - see the comment above. if wants_del: if wants_dict: @@ -186,10 +193,20 @@ def add(Proto): for key, value in Proto.__dict__.items(): - if not key.startswith('__') or key == '__del__': + if (not key.startswith('__') and not key.startswith('_mixin_') + or key == '__del__'): + if hasattr(value, "func_name"): + value = func_with_new_name(value, value.func_name) body[key] = value + if (config.objspace.std.withmapdict and "dict" in features): + from pypy.objspace.std.mapdict import BaseMapdictObject, ObjectMixin + add(BaseMapdictObject) + add(ObjectMixin) + features = () + if "user" in features: # generic feature needed by all subcls + class Proto(object): user_overridden_class = True @@ -255,6 +272,9 @@ wantdict = False if wantdict: + base_user_setup = supercls.user_setup.im_func + if "user_setup" in body: + base_user_setup = body["user_setup"] class Proto(object): def getdict(self): return self.w__dict__ @@ -263,11 +283,9 @@ self.w__dict__ = check_new_dictionary(space, w_dict) def user_setup(self, space, w_subtype): - self.space = space - self.w__class__ = w_subtype self.w__dict__ = space.newdict( instance=True, classofinstance=w_subtype) - self.user_setup_slots(w_subtype.nslots) + base_user_setup(self, space, w_subtype) def setclass(self, space, w_subtype): # only used by descr_set___class__ Modified: pypy/branch/32ptr-on-64bit/pypy/jit/backend/test/runner_test.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/backend/test/runner_test.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/backend/test/runner_test.py Tue Oct 12 14:42:59 2010 @@ -833,6 +833,23 @@ length_box], 'void') assert self.look_string(r_box) == "!??cdef?!" + def test_copyunicodecontent(self): + s_box = self.alloc_unicode(u"abcdef") + for s_box in [s_box, s_box.constbox()]: + for srcstart_box in [BoxInt(2), ConstInt(2)]: + for dststart_box in [BoxInt(3), ConstInt(3)]: + for length_box in [BoxInt(4), ConstInt(4)]: + for r_box_is_const in [False, True]: + r_box = self.alloc_unicode(u"!???????!") + if r_box_is_const: + r_box = r_box.constbox() + self.execute_operation(rop.COPYUNICODECONTENT, + [s_box, r_box, + srcstart_box, + dststart_box, + length_box], 'void') + assert self.look_unicode(r_box) == u"!??cdef?!" + def test_do_unicode_basic(self): u = self.cpu.bh_newunicode(5) self.cpu.bh_unicodesetitem(u, 4, 123) @@ -1227,6 +1244,10 @@ u_box = BoxPtr(lltype.cast_opaque_ptr(llmemory.GCREF, u)) return u_box + def look_unicode(self, unicode_box): + u = unicode_box.getref(lltype.Ptr(rstr.UNICODE)) + return u''.join(u.chars) + def test_casts(self): py.test.skip("xxx fix or kill") Modified: pypy/branch/32ptr-on-64bit/pypy/jit/backend/test/test_ll_random.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/backend/test/test_ll_random.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/backend/test/test_ll_random.py Tue Oct 12 14:42:59 2010 @@ -599,7 +599,7 @@ OPERATIONS.append(StrLenOperation(rop.STRLEN)) OPERATIONS.append(UnicodeLenOperation(rop.UNICODELEN)) OPERATIONS.append(CopyStrContentOperation(rop.COPYSTRCONTENT)) - #OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT)) + OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT)) for i in range(2): OPERATIONS.append(GuardClassOperation(rop.GUARD_CLASS)) Modified: pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/assembler.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/assembler.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/assembler.py Tue Oct 12 14:42:59 2010 @@ -244,7 +244,7 @@ f = open_file_as_stream(output_log, "w") for i in range(len(self.loop_run_counters)): name, struct = self.loop_run_counters[i] - f.write(str(struct.i) + " " * (8 - len(str(struct.i))) + name + "\n") + f.write(str(name) + ":" + str(struct.i) + "\n") f.close() def _build_float_constants(self): @@ -401,7 +401,7 @@ if self._debug: struct = lltype.malloc(DEBUG_COUNTER, flavor='raw') struct.i = 0 - self.loop_run_counters.append((funcname, struct)) + self.loop_run_counters.append((len(self.loop_run_counters), struct)) return funcname def patch_jump_for_descr(self, faildescr, adr_new_target): Modified: pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/regalloc.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/regalloc.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/regalloc.py Tue Oct 12 14:42:59 2010 @@ -778,15 +778,11 @@ loc = self.loc(op.getarg(0)) return self._call(op, [loc]) # boehm GC (XXX kill the following code at some point) - ofs_items, itemsize, ofs = symbolic.get_array_token(rstr.UNICODE, self.translate_support_code) - if itemsize == 4: - return self._malloc_varsize(ofs_items, ofs, 2, op.getarg(0), - op.result) - elif itemsize == 2: - return self._malloc_varsize(ofs_items, ofs, 1, op.getarg(0), - op.result) - else: - assert False, itemsize + ofs_items, _, ofs = symbolic.get_array_token(rstr.UNICODE, + self.translate_support_code) + scale = self._get_unicode_item_scale() + return self._malloc_varsize(ofs_items, ofs, scale, op.getarg(0), + op.result) def _malloc_varsize(self, ofs_items, ofs_length, scale, v, res_v): # XXX kill this function at some point @@ -959,6 +955,12 @@ consider_unicodegetitem = consider_strgetitem def consider_copystrcontent(self, op): + self._consider_copystrcontent(op, is_unicode=False) + + def consider_copyunicodecontent(self, op): + self._consider_copystrcontent(op, is_unicode=True) + + def _consider_copystrcontent(self, op, is_unicode): # compute the source address args = op.getarglist() base_loc = self.rm.make_sure_var_in_reg(args[0], args) @@ -970,7 +972,8 @@ srcaddr_box = TempBox() forbidden_vars = [args[1], args[3], args[4], srcaddr_box] srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box, forbidden_vars) - self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc) + self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc, + is_unicode=is_unicode) # compute the destination address base_loc = self.rm.make_sure_var_in_reg(args[1], forbidden_vars) ofs_loc = self.rm.make_sure_var_in_reg(args[3], forbidden_vars) @@ -980,25 +983,57 @@ forbidden_vars = [args[4], srcaddr_box] dstaddr_box = TempBox() dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box, forbidden_vars) - self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc) + self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc, + is_unicode=is_unicode) + # compute the length in bytes + length_box = args[4] + length_loc = self.loc(length_box) + if is_unicode: + self.rm.possibly_free_var(length_box) + forbidden_vars = [srcaddr_box, dstaddr_box] + bytes_box = TempBox() + bytes_loc = self.rm.force_allocate_reg(bytes_box, forbidden_vars) + scale = self._get_unicode_item_scale() + if not (isinstance(length_loc, ImmedLoc) or + isinstance(length_loc, RegLoc)): + self.assembler.mov(length_loc, bytes_loc) + length_loc = bytes_loc + self.assembler.load_effective_addr(length_loc, 0, scale, bytes_loc) + length_box = bytes_box + length_loc = bytes_loc # call memcpy() - length_loc = self.loc(args[4]) self.rm.before_call() self.xrm.before_call() self.assembler._emit_call(imm(self.assembler.memcpy_addr), [dstaddr_loc, srcaddr_loc, length_loc]) - self.rm.possibly_free_var(args[4]) + self.rm.possibly_free_var(length_box) self.rm.possibly_free_var(dstaddr_box) self.rm.possibly_free_var(srcaddr_box) - def _gen_address_inside_string(self, baseloc, ofsloc, resloc): + def _gen_address_inside_string(self, baseloc, ofsloc, resloc, is_unicode): cpu = self.assembler.cpu - ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR, + if is_unicode: + ofs_items, _, _ = symbolic.get_array_token(rstr.UNICODE, + self.translate_support_code) + scale = self._get_unicode_item_scale() + else: + ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR, self.translate_support_code) - assert itemsize == 1 - self.assembler.load_effective_addr(ofsloc, ofs_items, 0, + assert itemsize == 1 + scale = 0 + self.assembler.load_effective_addr(ofsloc, ofs_items, scale, resloc, baseloc) + def _get_unicode_item_scale(self): + _, itemsize, _ = symbolic.get_array_token(rstr.UNICODE, + self.translate_support_code) + if itemsize == 4: + return 2 + elif itemsize == 2: + return 1 + else: + raise AssertionError("bad unicode item size") + def consider_jump(self, op): assembler = self.assembler assert self.jump_target_descr is None Modified: pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/test/test_runner.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/test/test_runner.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/test/test_runner.py Tue Oct 12 14:42:59 2010 @@ -506,8 +506,8 @@ self.cpu.execute_token(ops.token) # check debugging info name, struct = self.cpu.assembler.loop_run_counters[0] - assert name == 'xyz' + assert name == 0 # 'xyz' assert struct.i == 10 self.cpu.finish_once() lines = py.path.local(self.logfile + ".count").readlines() - assert lines[0] == '10 xyz\n' + assert lines[0] == '0:10\n' # '10 xyz\n' Modified: pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/test/test_string.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/test/test_string.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/backend/x86/test/test_string.py Tue Oct 12 14:42:59 2010 @@ -2,8 +2,12 @@ from pypy.jit.metainterp.test import test_string from pypy.jit.backend.x86.test.test_basic import Jit386Mixin -class TestString(Jit386Mixin, test_string.StringTests): +class TestString(Jit386Mixin, test_string.TestLLtype): # for the individual tests see # ====> ../../../metainterp/test/test_string.py - CALL = 'call' - CALL_PURE = 'call_pure' + pass + +class TestUnicode(Jit386Mixin, test_string.TestLLtypeUnicode): + # for the individual tests see + # ====> ../../../metainterp/test/test_string.py + pass Modified: pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/effectinfo.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/effectinfo.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/effectinfo.py Tue Oct 12 14:42:59 2010 @@ -18,19 +18,30 @@ # the 'oopspecindex' field is one of the following values: OS_NONE = 0 # normal case, no oopspec OS_ARRAYCOPY = 1 # "list.ll_arraycopy" - OS_STR_CONCAT = 2 # "stroruni.concat" - OS_UNI_CONCAT = 3 # "stroruni.concat" - OS_STR_SLICE = 4 # "stroruni.slice" - OS_UNI_SLICE = 5 # "stroruni.slice" - OS_STR_EQUAL = 6 # "stroruni.equal" - OS_UNI_EQUAL = 7 # "stroruni.equal" - OS_STREQ_SLICE_CHECKNULL = 8 # s2!=NULL and s1[x:x+length]==s2 - OS_STREQ_SLICE_NONNULL = 9 # s1[x:x+length]==s2 (assert s2!=NULL) - OS_STREQ_SLICE_CHAR = 10 # s1[x:x+length]==char - OS_STREQ_NONNULL = 11 # s1 == s2 (assert s1!=NULL,s2!=NULL) - OS_STREQ_NONNULL_CHAR = 12 # s1 == char (assert s1!=NULL) - OS_STREQ_CHECKNULL_CHAR = 13 # s1!=NULL and s1==char - OS_STREQ_LENGTHOK = 14 # s1 == s2 (assert len(s1)==len(s2)) + OS_STR2UNICODE = 2 # "str.str2unicode" + + OS_STR_CONCAT = 22 # "stroruni.concat" + OS_STR_SLICE = 23 # "stroruni.slice" + OS_STR_EQUAL = 24 # "stroruni.equal" + OS_STREQ_SLICE_CHECKNULL = 25 # s2!=NULL and s1[x:x+length]==s2 + OS_STREQ_SLICE_NONNULL = 26 # s1[x:x+length]==s2 (assert s2!=NULL) + OS_STREQ_SLICE_CHAR = 27 # s1[x:x+length]==char + OS_STREQ_NONNULL = 28 # s1 == s2 (assert s1!=NULL,s2!=NULL) + OS_STREQ_NONNULL_CHAR = 29 # s1 == char (assert s1!=NULL) + OS_STREQ_CHECKNULL_CHAR = 30 # s1!=NULL and s1==char + OS_STREQ_LENGTHOK = 31 # s1 == s2 (assert len(s1)==len(s2)) + + OS_UNI_CONCAT = 42 # + OS_UNI_SLICE = 43 # + OS_UNI_EQUAL = 44 # + OS_UNIEQ_SLICE_CHECKNULL = 45 # + OS_UNIEQ_SLICE_NONNULL = 46 # + OS_UNIEQ_SLICE_CHAR = 47 # + OS_UNIEQ_NONNULL = 48 # the same for unicode + OS_UNIEQ_NONNULL_CHAR = 49 # (must be the same amount as for + OS_UNIEQ_CHECKNULL_CHAR = 50 # STR, in the same order) + OS_UNIEQ_LENGTHOK = 51 # + _OS_offset_uni = OS_UNI_CONCAT - OS_STR_CONCAT def __new__(cls, readonly_descrs_fields, write_descrs_fields, write_descrs_arrays, Modified: pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/jtransform.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/jtransform.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/jtransform.py Tue Oct 12 14:42:59 2010 @@ -316,6 +316,8 @@ prepare = self._handle_list_call elif oopspec_name.startswith('stroruni.'): prepare = self._handle_stroruni_call + elif oopspec_name == 'str.str2unicode': + prepare = self._handle_str2unicode_call elif oopspec_name.startswith('virtual_ref'): prepare = self._handle_virtual_ref_call else: @@ -1106,11 +1108,16 @@ [SoU, SoU], lltype.Signed), ]: + if args[0].concretetype.TO == rstr.UNICODE: + otherindex += EffectInfo._OS_offset_uni self._register_extra_helper(otherindex, othername, argtypes, resulttype) # return self._handle_oopspec_call(op, args, dict[oopspec_name]) + def _handle_str2unicode_call(self, op, oopspec_name, args): + return self._handle_oopspec_call(op, args, EffectInfo.OS_STR2UNICODE) + # ---------- # VirtualRefs. Modified: pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/test/test_jtransform.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/test/test_jtransform.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/codewriter/test/test_jtransform.py Tue Oct 12 14:42:59 2010 @@ -77,7 +77,8 @@ class FakeBuiltinCallControl: def guess_call_kind(self, op): return 'builtin' - def getcalldescr(self, op, oopspecindex): + def getcalldescr(self, op, oopspecindex=None): + assert oopspecindex is not None # in this test return 'calldescr-%d' % oopspecindex def calldescr_canraise(self, calldescr): return False @@ -766,6 +767,24 @@ assert op1.args[3] == ListOfKind('ref', [v1]) assert op1.result == v4 +def test_str2unicode(): + # test that the oopspec is present and correctly transformed + PSTR = lltype.Ptr(rstr.STR) + PUNICODE = lltype.Ptr(rstr.UNICODE) + FUNC = lltype.FuncType([PSTR], PUNICODE) + func = lltype.functionptr(FUNC, 'll_str2unicode', + _callable=rstr.LLHelpers.ll_str2unicode) + v1 = varoftype(PSTR) + v2 = varoftype(PUNICODE) + op = SpaceOperation('direct_call', [const(func), v1], v2) + tr = Transformer(FakeCPU(), FakeBuiltinCallControl()) + op1 = tr.rewrite_operation(op) + assert op1.opname == 'residual_call_r_r' + assert op1.args[0].value == func + assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_STR2UNICODE + assert op1.args[2] == ListOfKind('ref', [v1]) + assert op1.result == v2 + def test_list_ll_arraycopy(): from pypy.rlib.rgc import ll_arraycopy LIST = lltype.GcArray(lltype.Signed) Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/history.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/history.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/history.py Tue Oct 12 14:42:59 2010 @@ -698,6 +698,21 @@ return result _const_ptr_for_string = {} +def get_const_ptr_for_unicode(s): + from pypy.rpython.annlowlevel import llunicode + if not we_are_translated(): + try: + return _const_ptr_for_unicode[s] + except KeyError: + pass + if isinstance(s, str): + s = unicode(s) + result = ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, llunicode(s))) + if not we_are_translated(): + _const_ptr_for_unicode[s] = result + return result +_const_ptr_for_unicode = {} + # ____________________________________________________________ # The TreeLoop class contains a loop or a generalized loop, i.e. a tree Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimize_nopspec.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimize_nopspec.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimize_nopspec.py Tue Oct 12 14:42:59 2010 @@ -14,6 +14,9 @@ def _optimize_loop(metainterp_sd, old_loop_tokens, loop): cpu = metainterp_sd.cpu metainterp_sd.logger_noopt.log_loop(loop.inputargs, loop.operations) + # XXX the following lines are probably still needed, to discard invalid + # loops. bit silly to run a full perfect specialization and throw the + # result away. finder = PerfectSpecializationFinder(cpu) finder.find_nodes_loop(loop, False) if old_loop_tokens: @@ -31,6 +34,7 @@ def _optimize_bridge(metainterp_sd, old_loop_tokens, bridge): cpu = metainterp_sd.cpu metainterp_sd.logger_noopt.log_loop(bridge.inputargs, bridge.operations) + # XXX same comment as above applies finder = BridgeSpecializationFinder(cpu) finder.find_nodes_bridge(bridge) if old_loop_tokens: Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/intbounds.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/intbounds.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/intbounds.py Tue Oct 12 14:42:59 2010 @@ -191,6 +191,7 @@ v1.intbound.make_ge(IntLowerBound(0)) optimize_STRLEN = optimize_ARRAYLEN_GC + optimize_UNICODELEN = optimize_ARRAYLEN_GC def make_int_lt(self, box1, box2): v1 = self.getvalue(box1) Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/rewrite.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/rewrite.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/rewrite.py Tue Oct 12 14:42:59 2010 @@ -3,6 +3,7 @@ from pypy.jit.metainterp.history import ConstInt from pypy.jit.metainterp.optimizeutil import _findall from pypy.jit.metainterp.resoperation import rop, ResOperation +from pypy.jit.codewriter.effectinfo import EffectInfo class OptRewrite(Optimization): """Rewrite operations into equivalent, cheaper operations. @@ -245,6 +246,9 @@ def optimize_CALL_LOOPINVARIANT(self, op): funcvalue = self.getvalue(op.getarg(0)) if not funcvalue.is_constant(): + # XXX this code path is never executed in tests nor in production. + # in fact, it can't even happen since residual_call in codewriter + # expects a compile-time constant self.emit_operation(op) return key = make_hashable_int(op.getarg(0).getint()) @@ -323,8 +327,37 @@ ## return ## self.emit_operation(op) -optimize_ops = _findall(OptRewrite, 'optimize_') - + def optimize_CALL(self, op): + # dispatch based on 'oopspecindex' to a method that handles + # specifically the given oopspec call. For non-oopspec calls, + # oopspecindex is just zero. + effectinfo = op.getdescr().get_extra_info() + if effectinfo is not None: + oopspecindex = effectinfo.oopspecindex + if oopspecindex == EffectInfo.OS_ARRAYCOPY: + if self._optimize_CALL_ARRAYCOPY(op): + return + self.emit_operation(op) + def _optimize_CALL_ARRAYCOPY(self, op): + source_value = self.getvalue(op.getarg(1)) + dest_value = self.getvalue(op.getarg(2)) + source_start_box = self.get_constant_box(op.getarg(3)) + dest_start_box = self.get_constant_box(op.getarg(4)) + length = self.get_constant_box(op.getarg(5)) + if (source_value.is_virtual() and source_start_box and dest_start_box + and length and dest_value.is_virtual()): + # XXX optimize the case where dest value is not virtual, + # but we still can avoid a mess + source_start = source_start_box.getint() + dest_start = dest_start_box.getint() + for index in range(length.getint()): + val = source_value.getitem(index + source_start) + dest_value.setitem(index + dest_start, val) + return True + if length and length.getint() == 0: + return True # 0-length arraycopy + return False + +optimize_ops = _findall(OptRewrite, 'optimize_') - Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/string.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/string.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/optimizeopt/string.py Tue Oct 12 14:42:59 2010 @@ -3,6 +3,7 @@ from pypy.jit.metainterp.history import Box, BoxInt, BoxPtr from pypy.jit.metainterp.history import Const, ConstInt, ConstPtr from pypy.jit.metainterp.history import get_const_ptr_for_string +from pypy.jit.metainterp.history import get_const_ptr_for_unicode from pypy.jit.metainterp.resoperation import rop, ResOperation from pypy.jit.metainterp.optimizeopt import optimizer, virtualize from pypy.jit.metainterp.optimizeopt.optimizer import CONST_0, CONST_1 @@ -11,56 +12,103 @@ from pypy.jit.codewriter.effectinfo import EffectInfo, callinfo_for_oopspec from pypy.jit.codewriter import heaptracker from pypy.rlib.unroll import unrolling_iterable +from pypy.rlib.objectmodel import specialize + + +class StrOrUnicode(object): + def __init__(self, LLTYPE, hlstr, emptystr, chr, + NEWSTR, STRLEN, STRGETITEM, STRSETITEM, COPYSTRCONTENT, + OS_offset): + self.LLTYPE = LLTYPE + self.hlstr = hlstr + self.emptystr = emptystr + self.chr = chr + self.NEWSTR = NEWSTR + self.STRLEN = STRLEN + self.STRGETITEM = STRGETITEM + self.STRSETITEM = STRSETITEM + self.COPYSTRCONTENT = COPYSTRCONTENT + self.OS_offset = OS_offset + + def _freeze_(self): + return True + +mode_string = StrOrUnicode(rstr.STR, annlowlevel.hlstr, '', chr, + rop.NEWSTR, rop.STRLEN, rop.STRGETITEM, + rop.STRSETITEM, rop.COPYSTRCONTENT, 0) +mode_unicode = StrOrUnicode(rstr.UNICODE, annlowlevel.hlunicode, u'', unichr, + rop.NEWUNICODE, rop.UNICODELEN, rop.UNICODEGETITEM, + rop.UNICODESETITEM, rop.COPYUNICODECONTENT, + EffectInfo._OS_offset_uni) + +# ____________________________________________________________ class __extend__(optimizer.OptValue): """New methods added to the base class OptValue for this file.""" - def getstrlen(self, newoperations): - s = self.get_constant_string() - if s is not None: - return ConstInt(len(s)) + def getstrlen(self, newoperations, mode): + if mode is mode_string: + s = self.get_constant_string_spec(mode_string) + if s is not None: + return ConstInt(len(s)) else: - if newoperations is None: - return None - self.ensure_nonnull() - box = self.force_box() - lengthbox = BoxInt() - newoperations.append(ResOperation(rop.STRLEN, [box], lengthbox)) - return lengthbox + s = self.get_constant_string_spec(mode_unicode) + if s is not None: + return ConstInt(len(s)) + if newoperations is None: + return None + self.ensure_nonnull() + box = self.force_box() + lengthbox = BoxInt() + newoperations.append(ResOperation(mode.STRLEN, [box], lengthbox)) + return lengthbox - def get_constant_string(self): + @specialize.arg(1) + def get_constant_string_spec(self, mode): if self.is_constant(): - s = self.box.getref(lltype.Ptr(rstr.STR)) - return annlowlevel.hlstr(s) + s = self.box.getref(lltype.Ptr(mode.LLTYPE)) + return mode.hlstr(s) else: return None - def string_copy_parts(self, newoperations, targetbox, offsetbox): + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): # Copies the pointer-to-string 'self' into the target string # given by 'targetbox', at the specified offset. Returns the offset # at the end of the copy. - lengthbox = self.getstrlen(newoperations) + lengthbox = self.getstrlen(newoperations, mode) srcbox = self.force_box() return copy_str_content(newoperations, srcbox, targetbox, - CONST_0, offsetbox, lengthbox) + CONST_0, offsetbox, lengthbox, mode) class VAbstractStringValue(virtualize.AbstractVirtualValue): - _attrs_ = () + _attrs_ = ('mode',) + + def __init__(self, optimizer, keybox, source_op, mode): + virtualize.AbstractVirtualValue.__init__(self, optimizer, keybox, + source_op) + self.mode = mode def _really_force(self): - s = self.get_constant_string() - if s is not None: - c_s = get_const_ptr_for_string(s) - self.make_constant(c_s) - return + if self.mode is mode_string: + s = self.get_constant_string_spec(mode_string) + if s is not None: + c_s = get_const_ptr_for_string(s) + self.make_constant(c_s) + return + else: + s = self.get_constant_string_spec(mode_unicode) + if s is not None: + c_s = get_const_ptr_for_unicode(s) + self.make_constant(c_s) + return assert self.source_op is not None self.box = box = self.source_op.result newoperations = self.optimizer.newoperations - lengthbox = self.getstrlen(newoperations) - newoperations.append(ResOperation(rop.NEWSTR, [lengthbox], box)) - self.string_copy_parts(newoperations, box, CONST_0) + lengthbox = self.getstrlen(newoperations, self.mode) + newoperations.append(ResOperation(self.mode.NEWSTR, [lengthbox], box)) + self.string_copy_parts(newoperations, box, CONST_0, self.mode) class VStringPlainValue(VAbstractStringValue): @@ -74,7 +122,7 @@ assert 0 <= start <= stop <= len(longerlist) self._chars = longerlist[start:stop] - def getstrlen(self, _): + def getstrlen(self, _, mode): if self._lengthbox is None: self._lengthbox = ConstInt(len(self._chars)) return self._lengthbox @@ -86,18 +134,21 @@ assert isinstance(charvalue, optimizer.OptValue) self._chars[index] = charvalue - def get_constant_string(self): + @specialize.arg(1) + def get_constant_string_spec(self, mode): for c in self._chars: if c is optimizer.CVAL_UNINITIALIZED_ZERO or not c.is_constant(): return None - return ''.join([chr(c.box.getint()) for c in self._chars]) + return mode.emptystr.join([mode.chr(c.box.getint()) + for c in self._chars]) - def string_copy_parts(self, newoperations, targetbox, offsetbox): + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): for i in range(len(self._chars)): charbox = self._chars[i].force_box() - newoperations.append(ResOperation(rop.STRSETITEM, [targetbox, - offsetbox, - charbox], None)) + newoperations.append(ResOperation(mode.STRSETITEM, [targetbox, + offsetbox, + charbox], + None)) offsetbox = _int_add(newoperations, offsetbox, CONST_1) return offsetbox @@ -109,7 +160,7 @@ value.get_args_for_fail(modifier) def _make_virtual(self, modifier): - return modifier.make_vstrplain() + return modifier.make_vstrplain(self.mode is mode_unicode) class VStringConcatValue(VAbstractStringValue): @@ -120,23 +171,24 @@ self.right = right self.lengthbox = lengthbox - def getstrlen(self, _): + def getstrlen(self, _, mode): return self.lengthbox - def get_constant_string(self): - s1 = self.left.get_constant_string() + @specialize.arg(1) + def get_constant_string_spec(self, mode): + s1 = self.left.get_constant_string_spec(mode) if s1 is None: return None - s2 = self.right.get_constant_string() + s2 = self.right.get_constant_string_spec(mode) if s2 is None: return None return s1 + s2 - def string_copy_parts(self, newoperations, targetbox, offsetbox): + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): offsetbox = self.left.string_copy_parts(newoperations, targetbox, - offsetbox) + offsetbox, mode) offsetbox = self.right.string_copy_parts(newoperations, targetbox, - offsetbox) + offsetbox, mode) return offsetbox def get_args_for_fail(self, modifier): @@ -150,7 +202,7 @@ self.right.get_args_for_fail(modifier) def _make_virtual(self, modifier): - return modifier.make_vstrconcat() + return modifier.make_vstrconcat(self.mode is mode_unicode) class VStringSliceValue(VAbstractStringValue): @@ -162,12 +214,13 @@ self.vstart = vstart self.vlength = vlength - def getstrlen(self, _): + def getstrlen(self, _, mode): return self.vlength.force_box() - def get_constant_string(self): + @specialize.arg(1) + def get_constant_string_spec(self, mode): if self.vstart.is_constant() and self.vlength.is_constant(): - s1 = self.vstr.get_constant_string() + s1 = self.vstr.get_constant_string_spec(mode) if s1 is None: return None start = self.vstart.box.getint() @@ -177,12 +230,12 @@ return s1[start : start + length] return None - def string_copy_parts(self, newoperations, targetbox, offsetbox): - lengthbox = self.getstrlen(newoperations) + def string_copy_parts(self, newoperations, targetbox, offsetbox, mode): + lengthbox = self.getstrlen(newoperations, mode) return copy_str_content(newoperations, self.vstr.force_box(), targetbox, self.vstart.force_box(), offsetbox, - lengthbox) + lengthbox, mode) def get_args_for_fail(self, modifier): if self.box is None and not modifier.already_seen_virtual(self.keybox): @@ -195,11 +248,11 @@ self.vlength.get_args_for_fail(modifier) def _make_virtual(self, modifier): - return modifier.make_vstrslice() + return modifier.make_vstrslice(self.mode is mode_unicode) def copy_str_content(newoperations, srcbox, targetbox, - srcoffsetbox, offsetbox, lengthbox): + srcoffsetbox, offsetbox, lengthbox, mode): if isinstance(srcbox, ConstPtr) and isinstance(srcoffsetbox, Const): M = 5 else: @@ -208,17 +261,18 @@ # up to M characters are done "inline", i.e. with STRGETITEM/STRSETITEM # instead of just a COPYSTRCONTENT. for i in range(lengthbox.value): - charbox = _strgetitem(newoperations, srcbox, srcoffsetbox) + charbox = _strgetitem(newoperations, srcbox, srcoffsetbox, mode) srcoffsetbox = _int_add(newoperations, srcoffsetbox, CONST_1) - newoperations.append(ResOperation(rop.STRSETITEM, [targetbox, - offsetbox, - charbox], None)) + newoperations.append(ResOperation(mode.STRSETITEM, [targetbox, + offsetbox, + charbox], + None)) offsetbox = _int_add(newoperations, offsetbox, CONST_1) else: nextoffsetbox = _int_add(newoperations, offsetbox, lengthbox) - op = ResOperation(rop.COPYSTRCONTENT, [srcbox, targetbox, - srcoffsetbox, offsetbox, - lengthbox], None) + op = ResOperation(mode.COPYSTRCONTENT, [srcbox, targetbox, + srcoffsetbox, offsetbox, + lengthbox], None) newoperations.append(op) offsetbox = nextoffsetbox return offsetbox @@ -245,12 +299,16 @@ newoperations.append(ResOperation(rop.INT_SUB, [box1, box2], resbox)) return resbox -def _strgetitem(newoperations, strbox, indexbox): +def _strgetitem(newoperations, strbox, indexbox, mode): if isinstance(strbox, ConstPtr) and isinstance(indexbox, ConstInt): - s = strbox.getref(lltype.Ptr(rstr.STR)) - return ConstInt(ord(s.chars[indexbox.getint()])) + if mode is mode_string: + s = strbox.getref(lltype.Ptr(rstr.STR)) + return ConstInt(ord(s.chars[indexbox.getint()])) + else: + s = strbox.getref(lltype.Ptr(rstr.UNICODE)) + return ConstInt(ord(s.chars[indexbox.getint()])) resbox = BoxInt() - newoperations.append(ResOperation(rop.STRGETITEM, [strbox, indexbox], + newoperations.append(ResOperation(mode.STRGETITEM, [strbox, indexbox], resbox)) return resbox @@ -258,62 +316,34 @@ class OptString(optimizer.Optimization): "Handling of strings and unicodes." - def make_vstring_plain(self, box, source_op=None): - vvalue = VStringPlainValue(self.optimizer, box, source_op) + def make_vstring_plain(self, box, source_op, mode): + vvalue = VStringPlainValue(self.optimizer, box, source_op, mode) self.make_equal_to(box, vvalue) return vvalue - def make_vstring_concat(self, box, source_op=None): - vvalue = VStringConcatValue(self.optimizer, box, source_op) + def make_vstring_concat(self, box, source_op, mode): + vvalue = VStringConcatValue(self.optimizer, box, source_op, mode) self.make_equal_to(box, vvalue) return vvalue - def make_vstring_slice(self, box, source_op=None): - vvalue = VStringSliceValue(self.optimizer, box, source_op) + def make_vstring_slice(self, box, source_op, mode): + vvalue = VStringSliceValue(self.optimizer, box, source_op, mode) self.make_equal_to(box, vvalue) return vvalue - def optimize_CALL(self, op): - # dispatch based on 'oopspecindex' to a method that handles - # specifically the given oopspec call. For non-oopspec calls, - # oopspecindex is just zero. - effectinfo = op.getdescr().get_extra_info() - if effectinfo is not None: - oopspecindex = effectinfo.oopspecindex - for value, meth in opt_call_oopspec_ops: - if oopspecindex == value: - if meth(self, op): - return - self.emit_operation(op) - - def opt_call_oopspec_ARRAYCOPY(self, op): - source_value = self.getvalue(op.getarg(1)) - dest_value = self.getvalue(op.getarg(2)) - source_start_box = self.get_constant_box(op.getarg(3)) - dest_start_box = self.get_constant_box(op.getarg(4)) - length = self.get_constant_box(op.getarg(5)) - if (source_value.is_virtual() and source_start_box and dest_start_box - and length and dest_value.is_virtual()): - # XXX optimize the case where dest value is not virtual, - # but we still can avoid a mess - source_start = source_start_box.getint() - dest_start = dest_start_box.getint() - for index in range(length.getint()): - val = source_value.getitem(index + source_start) - dest_value.setitem(index + dest_start, val) - return True - if length and length.getint() == 0: - return True # 0-length arraycopy - return False - def optimize_NEWSTR(self, op): + self._optimize_NEWSTR(op, mode_string) + def optimize_NEWUNICODE(self, op): + self._optimize_NEWSTR(op, mode_unicode) + + def _optimize_NEWSTR(self, op, mode): length_box = self.get_constant_box(op.getarg(0)) if length_box: # if the original 'op' did not have a ConstInt as argument, # build a new one with the ConstInt argument if not isinstance(op.getarg(0), ConstInt): - op = ResOperation(rop.NEWSTR, [length_box], op.result) - vvalue = self.make_vstring_plain(op.result, op) + op = ResOperation(mode.NEWSTR, [length_box], op.result) + vvalue = self.make_vstring_plain(op.result, op, mode) vvalue.setup(length_box.getint()) else: self.getvalue(op.result).ensure_nonnull() @@ -329,13 +359,20 @@ value.ensure_nonnull() self.emit_operation(op) + optimize_UNICODESETITEM = optimize_STRSETITEM + def optimize_STRGETITEM(self, op): + self._optimize_STRGETITEM(op, mode_string) + def optimize_UNICODEGETITEM(self, op): + self._optimize_STRGETITEM(op, mode_unicode) + + def _optimize_STRGETITEM(self, op, mode): value = self.getvalue(op.getarg(0)) vindex = self.getvalue(op.getarg(1)) - vresult = self.strgetitem(value, vindex) + vresult = self.strgetitem(value, vindex, mode) self.make_equal_to(op.result, vresult) - def strgetitem(self, value, vindex): + def strgetitem(self, value, vindex, mode): value.ensure_nonnull() # if value.is_virtual() and isinstance(value, VStringSliceValue): @@ -350,28 +387,71 @@ return value.getitem(vindex.box.getint()) # resbox = _strgetitem(self.optimizer.newoperations, - value.force_box(),vindex.force_box()) + value.force_box(),vindex.force_box(), mode) return self.getvalue(resbox) def optimize_STRLEN(self, op): + self._optimize_STRLEN(op, mode_string) + def optimize_UNICODELEN(self, op): + self._optimize_STRLEN(op, mode_unicode) + + def _optimize_STRLEN(self, op, mode): value = self.getvalue(op.getarg(0)) - lengthbox = value.getstrlen(self.optimizer.newoperations) + lengthbox = value.getstrlen(self.optimizer.newoperations, mode) self.make_equal_to(op.result, self.getvalue(lengthbox)) - def opt_call_oopspec_STR_CONCAT(self, op): + def optimize_CALL(self, op): + # dispatch based on 'oopspecindex' to a method that handles + # specifically the given oopspec call. For non-oopspec calls, + # oopspecindex is just zero. + effectinfo = op.getdescr().get_extra_info() + if effectinfo is not None: + oopspecindex = effectinfo.oopspecindex + for value, meth in opt_call_oopspec_ops: + if oopspecindex == value: # a match with the OS_STR_xxx + if meth(self, op, mode_string): + return + break + if oopspecindex == value + EffectInfo._OS_offset_uni: + # a match with the OS_UNI_xxx + if meth(self, op, mode_unicode): + return + break + if oopspecindex == EffectInfo.OS_STR2UNICODE: + if self.opt_call_str_STR2UNICODE(op): + return + self.emit_operation(op) + + def opt_call_str_STR2UNICODE(self, op): + # Constant-fold unicode("constant string"). + # More generally, supporting non-constant but virtual cases is + # not obvious, because of the exception UnicodeDecodeError that + # can be raised by ll_str2unicode() + varg = self.getvalue(op.getarg(1)) + s = varg.get_constant_string_spec(mode_string) + if s is None: + return False + try: + u = unicode(s) + except UnicodeDecodeError: + return False + self.make_constant(op.result, get_const_ptr_for_unicode(u)) + return True + + def opt_call_stroruni_STR_CONCAT(self, op, mode): vleft = self.getvalue(op.getarg(1)) vright = self.getvalue(op.getarg(2)) vleft.ensure_nonnull() vright.ensure_nonnull() newoperations = self.optimizer.newoperations - len1box = vleft.getstrlen(newoperations) - len2box = vright.getstrlen(newoperations) + len1box = vleft.getstrlen(newoperations, mode) + len2box = vright.getstrlen(newoperations, mode) lengthbox = _int_add(newoperations, len1box, len2box) - value = self.make_vstring_concat(op.result, op) + value = self.make_vstring_concat(op.result, op, mode) value.setup(vleft, vright, lengthbox) return True - def opt_call_oopspec_STR_SLICE(self, op): + def opt_call_stroruni_STR_SLICE(self, op, mode): newoperations = self.optimizer.newoperations vstr = self.getvalue(op.getarg(1)) vstart = self.getvalue(op.getarg(2)) @@ -380,7 +460,7 @@ if (isinstance(vstr, VStringPlainValue) and vstart.is_constant() and vstop.is_constant()): # slicing with constant bounds of a VStringPlainValue - value = self.make_vstring_plain(op.result, op) + value = self.make_vstring_plain(op.result, op, mode) value.setup_slice(vstr._chars, vstart.box.getint(), vstop.box.getint()) return True @@ -398,16 +478,16 @@ vstart.force_box()) vstart = self.getvalue(startbox) # - value = self.make_vstring_slice(op.result, op) + value = self.make_vstring_slice(op.result, op, mode) value.setup(vstr, vstart, self.getvalue(lengthbox)) return True - def opt_call_oopspec_STR_EQUAL(self, op): + def opt_call_stroruni_STR_EQUAL(self, op, mode): v1 = self.getvalue(op.getarg(1)) v2 = self.getvalue(op.getarg(2)) # - l1box = v1.getstrlen(None) - l2box = v2.getstrlen(None) + l1box = v1.getstrlen(None, mode) + l2box = v2.getstrlen(None, mode) if (l1box is not None and l2box is not None and isinstance(l1box, ConstInt) and isinstance(l2box, ConstInt) and @@ -416,13 +496,13 @@ self.make_constant(op.result, CONST_0) return True # - if self.handle_str_equal_level1(v1, v2, op.result): + if self.handle_str_equal_level1(v1, v2, op.result, mode): return True - if self.handle_str_equal_level1(v2, v1, op.result): + if self.handle_str_equal_level1(v2, v1, op.result, mode): return True - if self.handle_str_equal_level2(v1, v2, op.result): + if self.handle_str_equal_level2(v1, v2, op.result, mode): return True - if self.handle_str_equal_level2(v2, v1, op.result): + if self.handle_str_equal_level2(v2, v1, op.result, mode): return True # if v1.is_nonnull() and v2.is_nonnull(): @@ -434,37 +514,37 @@ else: do = EffectInfo.OS_STREQ_NONNULL self.generate_modified_call(do, [v1.force_box(), - v2.force_box()], op.result) + v2.force_box()], op.result, mode) return True return False - def handle_str_equal_level1(self, v1, v2, resultbox): - l2box = v2.getstrlen(None) + def handle_str_equal_level1(self, v1, v2, resultbox, mode): + l2box = v2.getstrlen(None, mode) if isinstance(l2box, ConstInt): if l2box.value == 0: - lengthbox = v1.getstrlen(self.optimizer.newoperations) + lengthbox = v1.getstrlen(self.optimizer.newoperations, mode) seo = self.optimizer.send_extra_operation seo(ResOperation(rop.INT_EQ, [lengthbox, CONST_0], resultbox)) return True if l2box.value == 1: - l1box = v1.getstrlen(None) + l1box = v1.getstrlen(None, mode) if isinstance(l1box, ConstInt) and l1box.value == 1: # comparing two single chars - vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO) - vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO) + vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO, mode) + vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO, mode) seo = self.optimizer.send_extra_operation seo(ResOperation(rop.INT_EQ, [vchar1.force_box(), vchar2.force_box()], resultbox)) return True if isinstance(v1, VStringSliceValue): - vchar = self.strgetitem(v2, optimizer.CVAL_ZERO) + vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode) do = EffectInfo.OS_STREQ_SLICE_CHAR self.generate_modified_call(do, [v1.vstr.force_box(), v1.vstart.force_box(), v1.vlength.force_box(), vchar.force_box()], - resultbox) + resultbox, mode) return True # if v2.is_null(): @@ -482,17 +562,18 @@ # return False - def handle_str_equal_level2(self, v1, v2, resultbox): - l2box = v2.getstrlen(None) + def handle_str_equal_level2(self, v1, v2, resultbox, mode): + l2box = v2.getstrlen(None, mode) if isinstance(l2box, ConstInt): if l2box.value == 1: - vchar = self.strgetitem(v2, optimizer.CVAL_ZERO) + vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode) if v1.is_nonnull(): do = EffectInfo.OS_STREQ_NONNULL_CHAR else: do = EffectInfo.OS_STREQ_CHECKNULL_CHAR self.generate_modified_call(do, [v1.force_box(), - vchar.force_box()], resultbox) + vchar.force_box()], resultbox, + mode) return True # if v1.is_virtual() and isinstance(v1, VStringSliceValue): @@ -503,11 +584,12 @@ self.generate_modified_call(do, [v1.vstr.force_box(), v1.vstart.force_box(), v1.vlength.force_box(), - v2.force_box()], resultbox) + v2.force_box()], resultbox, mode) return True return False - def generate_modified_call(self, oopspecindex, args, result): + def generate_modified_call(self, oopspecindex, args, result, mode): + oopspecindex += mode.OS_offset calldescr, func = callinfo_for_oopspec(oopspecindex) op = ResOperation(rop.CALL, [ConstInt(func)] + args, result, descr=calldescr) @@ -525,7 +607,7 @@ optimize_ops = _findall(OptString, 'optimize_') def _findall_call_oopspec(): - prefix = 'opt_call_oopspec_' + prefix = 'opt_call_stroruni_' result = [] for name in dir(OptString): if name.startswith(prefix): Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/resume.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/resume.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/resume.py Tue Oct 12 14:42:59 2010 @@ -255,13 +255,19 @@ def make_varray(self, arraydescr): return VArrayInfo(arraydescr) - def make_vstrplain(self): + def make_vstrplain(self, is_unicode=False): + if is_unicode: + return VUniPlainInfo() return VStrPlainInfo() - def make_vstrconcat(self): + def make_vstrconcat(self, is_unicode=False): + if is_unicode: + return VUniConcatInfo() return VStrConcatInfo() - def make_vstrslice(self): + def make_vstrslice(self, is_unicode=False): + if is_unicode: + return VUniSliceInfo() return VStrSliceInfo() def register_virtual_fields(self, virtualbox, fieldboxes): @@ -550,6 +556,60 @@ for i in self.fieldnums: debug_print("\t\t", str(untag(i))) + +class VUniPlainInfo(AbstractVirtualInfo): + """Stands for the unicode string made out of the characters of all + fieldnums.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + length = len(self.fieldnums) + string = decoder.allocate_unicode(length) + decoder.virtuals_cache[index] = string + for i in range(length): + decoder.unicode_setitem(string, i, self.fieldnums[i]) + return string + + def debug_prints(self): + debug_print("\tvuniplaininfo length", len(self.fieldnums)) + + +class VUniConcatInfo(AbstractVirtualInfo): + """Stands for the unicode string made out of the concatenation of two + other unicode strings.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + # xxx for blackhole resuming, this will build all intermediate + # strings and throw them away immediately, which is a bit sub- + # efficient. Not sure we care. + left, right = self.fieldnums + string = decoder.concat_unicodes(left, right) + decoder.virtuals_cache[index] = string + return string + + def debug_prints(self): + debug_print("\tvuniconcatinfo") + for i in self.fieldnums: + debug_print("\t\t", str(untag(i))) + + +class VUniSliceInfo(AbstractVirtualInfo): + """Stands for the unicode string made out of slicing another + unicode string.""" + + @specialize.argtype(1) + def allocate(self, decoder, index): + largerstr, start, length = self.fieldnums + string = decoder.slice_unicode(largerstr, start, length) + decoder.virtuals_cache[index] = string + return string + + def debug_prints(self): + debug_print("\tvunisliceinfo") + for i in self.fieldnums: + debug_print("\t\t", str(untag(i))) + # ____________________________________________________________ class AbstractResumeDataReader(object): @@ -725,6 +785,32 @@ return self.metainterp.execute_and_record_varargs( rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr) + def allocate_unicode(self, length): + return self.metainterp.execute_and_record(rop.NEWUNICODE, + None, ConstInt(length)) + + def unicode_setitem(self, strbox, index, charnum): + charbox = self.decode_box(charnum, INT) + self.metainterp.execute_and_record(rop.UNICODESETITEM, None, + strbox, ConstInt(index), charbox) + + def concat_unicodes(self, str1num, str2num): + calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_CONCAT) + str1box = self.decode_box(str1num, REF) + str2box = self.decode_box(str2num, REF) + return self.metainterp.execute_and_record_varargs( + rop.CALL, [ConstInt(func), str1box, str2box], calldescr) + + def slice_unicode(self, strnum, startnum, lengthnum): + calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_SLICE) + strbox = self.decode_box(strnum, REF) + startbox = self.decode_box(startnum, INT) + lengthbox = self.decode_box(lengthnum, INT) + stopbox = self.metainterp.execute_and_record(rop.INT_ADD, None, + startbox, lengthbox) + return self.metainterp.execute_and_record_varargs( + rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr) + def setfield(self, descr, structbox, fieldnum): if descr.is_pointer_field(): kind = REF @@ -967,6 +1053,31 @@ result = funcptr(str, start, start + length) return lltype.cast_opaque_ptr(llmemory.GCREF, result) + def allocate_unicode(self, length): + return self.cpu.bh_newunicode(length) + + def unicode_setitem(self, str, index, charnum): + char = self.decode_int(charnum) + self.cpu.bh_unicodesetitem(str, index, char) + + def concat_unicodes(self, str1num, str2num): + str1 = self.decode_ref(str1num) + str2 = self.decode_ref(str2num) + str1 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str1) + str2 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str2) + funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_CONCAT) + result = funcptr(str1, str2) + return lltype.cast_opaque_ptr(llmemory.GCREF, result) + + def slice_unicode(self, strnum, startnum, lengthnum): + str = self.decode_ref(strnum) + start = self.decode_int(startnum) + length = self.decode_int(lengthnum) + str = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str) + funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_SLICE) + result = funcptr(str, start, start + length) + return lltype.cast_opaque_ptr(llmemory.GCREF, result) + def setfield(self, descr, struct, fieldnum): if descr.is_pointer_field(): newvalue = self.decode_ref(fieldnum) Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/simple_optimize.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/simple_optimize.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/simple_optimize.py Tue Oct 12 14:42:59 2010 @@ -9,11 +9,13 @@ def transform(op): from pypy.jit.metainterp.history import AbstractDescr - # Rename CALL_PURE to CALL. + # Rename CALL_PURE and CALL_LOOPINVARIANT to CALL. # Simplify the VIRTUAL_REF_* so that they don't show up in the backend. if op.getopnum() == rop.CALL_PURE: op = ResOperation(rop.CALL, op.getarglist()[1:], op.result, op.getdescr()) + elif op.getopnum() == rop.CALL_LOOPINVARIANT: + op = op.copy_and_change(rop.CALL) elif op.getopnum() == rop.VIRTUAL_REF: op = ResOperation(rop.SAME_AS, [op.getarg(0)], op.result) elif op.getopnum() == rop.VIRTUAL_REF_FINISH: Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/oparser.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/oparser.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/oparser.py Tue Oct 12 14:42:59 2010 @@ -5,7 +5,7 @@ from pypy.jit.metainterp.history import TreeLoop, BoxInt, ConstInt,\ ConstObj, ConstPtr, Box, BasicFailDescr, BoxFloat, ConstFloat,\ - LoopToken, get_const_ptr_for_string + LoopToken, get_const_ptr_for_string, get_const_ptr_for_unicode from pypy.jit.metainterp.resoperation import rop, ResOperation, ResOpWithDescr, N_aryOp from pypy.jit.metainterp.typesystem import llhelper from pypy.jit.codewriter.heaptracker import adr2int @@ -158,10 +158,15 @@ except ValueError: if self.is_float(arg): return ConstFloat(float(arg)) - if arg.startswith('"') or arg.startswith("'"): + if (arg.startswith('"') or arg.startswith("'") or + arg.startswith('s"')): # XXX ootype - info = arg.strip("'\"") + info = arg[1:].strip("'\"") return get_const_ptr_for_string(info) + if arg.startswith('u"'): + # XXX ootype + info = arg[1:].strip("'\"") + return get_const_ptr_for_unicode(info) if arg.startswith('ConstClass('): name = arg[len('ConstClass('):-1] return self.get_const(name, 'class') Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_optimizefindnode.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_optimizefindnode.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_optimizefindnode.py Tue Oct 12 14:42:59 2010 @@ -117,33 +117,32 @@ EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE)) arraycopydescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, EffectInfo([], [], [], oopspecindex=EffectInfo.OS_ARRAYCOPY)) - strconcatdescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_CONCAT)) - slicedescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_SLICE)) - strequaldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_EQUAL)) - streq_slice_checknull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_SLICE_CHECKNULL)) - streq_slice_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_SLICE_NONNULL)) - streq_slice_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_SLICE_CHAR)) - streq_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_NONNULL)) - streq_nonnull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_NONNULL_CHAR)) - streq_checknull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_CHECKNULL_CHAR)) - streq_lengthok_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, - EffectInfo([], [], [], - oopspecindex=EffectInfo.OS_STREQ_LENGTHOK)) + + for _name, _os in [ + ('strconcatdescr', 'OS_STR_CONCAT'), + ('strslicedescr', 'OS_STR_SLICE'), + ('strequaldescr', 'OS_STR_EQUAL'), + ('streq_slice_checknull_descr', 'OS_STREQ_SLICE_CHECKNULL'), + ('streq_slice_nonnull_descr', 'OS_STREQ_SLICE_NONNULL'), + ('streq_slice_char_descr', 'OS_STREQ_SLICE_CHAR'), + ('streq_nonnull_descr', 'OS_STREQ_NONNULL'), + ('streq_nonnull_char_descr', 'OS_STREQ_NONNULL_CHAR'), + ('streq_checknull_char_descr', 'OS_STREQ_CHECKNULL_CHAR'), + ('streq_lengthok_descr', 'OS_STREQ_LENGTHOK'), + ]: + _oopspecindex = getattr(EffectInfo, _os) + locals()[_name] = \ + cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=_oopspecindex)) + # + _oopspecindex = getattr(EffectInfo, _os.replace('STR', 'UNI')) + locals()[_name.replace('str', 'unicode')] = \ + cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=_oopspecindex)) + + s2u_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, + EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR2UNICODE)) + # class LoopToken(AbstractDescr): pass Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_optimizeopt.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_optimizeopt.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_optimizeopt.py Tue Oct 12 14:42:59 2010 @@ -3487,7 +3487,7 @@ i0 = strlen(p0) jump(p0) """ - self.optimize_loop(ops, 'Not', expected) + self.optimize_strunicode_loop(ops, 'Not', expected) def test_addsub_const(self): ops = """ @@ -3893,6 +3893,15 @@ """ self.optimize_loop(ops, 'Not, Not', expected) + # ---------- + def optimize_strunicode_loop(self, ops, spectext, optops): + # check with the arguments passed in + self.optimize_loop(ops, spectext, optops) + # check with replacing 'str' with 'unicode' everywhere + self.optimize_loop(ops.replace('str','unicode').replace('s"', 'u"'), + spectext, + optops.replace('str','unicode').replace('s"', 'u"')) + def test_newstr_1(self): ops = """ [i0] @@ -3905,7 +3914,7 @@ [i0] jump(i0) """ - self.optimize_loop(ops, 'Not', expected) + self.optimize_strunicode_loop(ops, 'Not', expected) def test_newstr_2(self): ops = """ @@ -3921,7 +3930,7 @@ [i0, i1] jump(i1, i0) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_concat_1(self): ops = """ @@ -3942,7 +3951,7 @@ copystrcontent(p2, p3, 0, i4, i5) jump(p2, p3) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_concat_vstr2_str(self): ops = """ @@ -3965,7 +3974,7 @@ copystrcontent(p2, p3, 0, 2, i4) jump(i1, i0, p3) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_concat_str_vstr2(self): ops = """ @@ -3989,7 +3998,7 @@ i6 = int_add(i5, 1) # will be killed by the backend jump(i1, i0, p3) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_concat_str_str_str(self): ops = """ @@ -4016,12 +4025,12 @@ copystrcontent(p3, p5, 0, i12b, i3b) jump(p2, p3, p5) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_concat_str_cstr1(self): ops = """ [p2] - p3 = call(0, p2, "x", descr=strconcatdescr) + p3 = call(0, p2, s"x", descr=strconcatdescr) jump(p3) """ expected = """ @@ -4035,28 +4044,28 @@ i5 = int_add(i4, 1) # will be killed by the backend jump(p3) """ - self.optimize_loop(ops, 'Not', expected) + self.optimize_strunicode_loop(ops, 'Not', expected) def test_str_concat_consts(self): ops = """ [] - p1 = same_as("ab") - p2 = same_as("cde") + p1 = same_as(s"ab") + p2 = same_as(s"cde") p3 = call(0, p1, p2, descr=strconcatdescr) escape(p3) jump() """ expected = """ [] - escape("abcde") + escape(s"abcde") jump() """ - self.optimize_loop(ops, '', expected) + self.optimize_strunicode_loop(ops, '', expected) def test_str_slice_1(self): ops = """ [p1, i1, i2] - p2 = call(0, p1, i1, i2, descr=slicedescr) + p2 = call(0, p1, i1, i2, descr=strslicedescr) jump(p2, i1, i2) """ expected = """ @@ -4066,12 +4075,12 @@ copystrcontent(p1, p2, i1, 0, i3) jump(p2, i1, i2) """ - self.optimize_loop(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected) def test_str_slice_2(self): ops = """ [p1, i2] - p2 = call(0, p1, 0, i2, descr=slicedescr) + p2 = call(0, p1, 0, i2, descr=strslicedescr) jump(p2, i2) """ expected = """ @@ -4080,13 +4089,13 @@ copystrcontent(p1, p2, 0, 0, i2) jump(p2, i2) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_slice_3(self): ops = """ [p1, i1, i2, i3, i4] - p2 = call(0, p1, i1, i2, descr=slicedescr) - p3 = call(0, p2, i3, i4, descr=slicedescr) + p2 = call(0, p1, i1, i2, descr=strslicedescr) + p3 = call(0, p2, i3, i4, descr=strslicedescr) jump(p3, i1, i2, i3, i4) """ expected = """ @@ -4098,12 +4107,12 @@ copystrcontent(p1, p3, i6, 0, i5) jump(p3, i1, i2, i3, i4) """ - self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not, Not', expected) def test_str_slice_getitem1(self): ops = """ [p1, i1, i2, i3] - p2 = call(0, p1, i1, i2, descr=slicedescr) + p2 = call(0, p1, i1, i2, descr=strslicedescr) i4 = strgetitem(p2, i3) escape(i4) jump(p1, i1, i2, i3) @@ -4116,7 +4125,7 @@ escape(i4) jump(p1, i1, i2, i3) """ - self.optimize_loop(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected) def test_str_slice_plain(self): ops = """ @@ -4124,7 +4133,7 @@ p1 = newstr(2) strsetitem(p1, 0, i3) strsetitem(p1, 1, i4) - p2 = call(0, p1, 1, 2, descr=slicedescr) + p2 = call(0, p1, 1, 2, descr=strslicedescr) i5 = strgetitem(p2, 0) escape(i5) jump(i3, i4) @@ -4134,12 +4143,12 @@ escape(i4) jump(i3, i4) """ - self.optimize_loop(ops, 'Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not', expected) def test_str_slice_concat(self): ops = """ [p1, i1, i2, p2] - p3 = call(0, p1, i1, i2, descr=slicedescr) + p3 = call(0, p1, i1, i2, descr=strslicedescr) p4 = call(0, p3, p2, descr=strconcatdescr) jump(p4, i1, i2, p2) """ @@ -4155,10 +4164,10 @@ copystrcontent(p2, p4, 0, i3, i4b) jump(p4, i1, i2, p2) """ - self.optimize_loop(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected) # ---------- - def optimize_loop_extradescrs(self, ops, spectext, optops): + def optimize_strunicode_loop_extradescrs(self, ops, spectext, optops): from pypy.jit.metainterp.optimizeopt import string def my_callinfo_for_oopspec(oopspecindex): calldescrtype = type(LLtypeMixin.strequaldescr) @@ -4173,7 +4182,7 @@ saved = string.callinfo_for_oopspec try: string.callinfo_for_oopspec = my_callinfo_for_oopspec - self.optimize_loop(ops, spectext, optops) + self.optimize_strunicode_loop(ops, spectext, optops) finally: string.callinfo_for_oopspec = saved @@ -4184,7 +4193,7 @@ escape(i0) jump(p1, p2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not', ops) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', ops) def test_str_equal_noop2(self): ops = """ @@ -4209,12 +4218,13 @@ escape(i0) jump(p1, p2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not', + expected) def test_str_equal_slice1(self): ops = """ [p1, i1, i2, p3] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) i0 = call(0, p4, p3, descr=strequaldescr) escape(i0) jump(p1, i1, i2, p3) @@ -4226,12 +4236,13 @@ escape(i0) jump(p1, i1, i2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_slice2(self): ops = """ [p1, i1, i2, p3] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) i0 = call(0, p3, p4, descr=strequaldescr) escape(i0) jump(p1, i1, i2, p3) @@ -4243,13 +4254,14 @@ escape(i0) jump(p1, i1, i2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_slice3(self): ops = """ [p1, i1, i2, p3] guard_nonnull(p3) [] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) i0 = call(0, p3, p4, descr=strequaldescr) escape(i0) jump(p1, i1, i2, p3) @@ -4262,13 +4274,14 @@ escape(i0) jump(p1, i1, i2, p3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_slice4(self): ops = """ [p1, i1, i2] - p3 = call(0, p1, i1, i2, descr=slicedescr) - i0 = call(0, p3, "x", descr=strequaldescr) + p3 = call(0, p1, i1, i2, descr=strslicedescr) + i0 = call(0, p3, s"x", descr=strequaldescr) escape(i0) jump(p1, i1, i2) """ @@ -4279,12 +4292,13 @@ escape(i0) jump(p1, i1, i2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not', + expected) def test_str_equal_slice5(self): ops = """ [p1, i1, i2, i3] - p4 = call(0, p1, i1, i2, descr=slicedescr) + p4 = call(0, p1, i1, i2, descr=strslicedescr) p5 = newstr(1) strsetitem(p5, 0, i3) i0 = call(0, p5, p4, descr=strequaldescr) @@ -4298,7 +4312,8 @@ escape(i0) jump(p1, i1, i2, i3) """ - self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not', + expected) def test_str_equal_none1(self): ops = """ @@ -4313,7 +4328,7 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_none2(self): ops = """ @@ -4328,30 +4343,30 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull1(self): ops = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "hello world", descr=strequaldescr) + i0 = call(0, p1, s"hello world", descr=strequaldescr) escape(i0) jump(p1) """ expected = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "hello world", descr=streq_nonnull_descr) + i0 = call(0, p1, s"hello world", descr=streq_nonnull_descr) escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull2(self): ops = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "", descr=strequaldescr) + i0 = call(0, p1, s"", descr=strequaldescr) escape(i0) jump(p1) """ @@ -4363,13 +4378,13 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull3(self): ops = """ [p1] guard_nonnull(p1) [] - i0 = call(0, p1, "x", descr=strequaldescr) + i0 = call(0, p1, s"x", descr=strequaldescr) escape(i0) jump(p1) """ @@ -4380,13 +4395,13 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_nonnull4(self): ops = """ [p1, p2] p4 = call(0, p1, p2, descr=strconcatdescr) - i0 = call(0, "hello world", p4, descr=strequaldescr) + i0 = call(0, s"hello world", p4, descr=strequaldescr) escape(i0) jump(p1, p2) """ @@ -4401,17 +4416,17 @@ i5 = strlen(p2) i6 = int_add(i4, i5) # will be killed by the backend copystrcontent(p2, p4, 0, i4, i5) - i0 = call(0, "hello world", p4, descr=streq_nonnull_descr) + i0 = call(0, s"hello world", p4, descr=streq_nonnull_descr) escape(i0) jump(p1, p2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected) def test_str_equal_chars0(self): ops = """ [i1] p1 = newstr(0) - i0 = call(0, p1, "", descr=strequaldescr) + i0 = call(0, p1, s"", descr=strequaldescr) escape(i0) jump(i1) """ @@ -4420,14 +4435,14 @@ escape(1) jump(i1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_chars1(self): ops = """ [i1] p1 = newstr(1) strsetitem(p1, 0, i1) - i0 = call(0, p1, "x", descr=strequaldescr) + i0 = call(0, p1, s"x", descr=strequaldescr) escape(i0) jump(i1) """ @@ -4437,7 +4452,7 @@ escape(i0) jump(i1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_chars2(self): ops = """ @@ -4445,7 +4460,7 @@ p1 = newstr(2) strsetitem(p1, 0, i1) strsetitem(p1, 1, i2) - i0 = call(0, p1, "xy", descr=strequaldescr) + i0 = call(0, p1, s"xy", descr=strequaldescr) escape(i0) jump(i1, i2) """ @@ -4454,16 +4469,16 @@ p1 = newstr(2) strsetitem(p1, 0, i1) strsetitem(p1, 1, i2) - i0 = call(0, p1, "xy", descr=streq_lengthok_descr) + i0 = call(0, p1, s"xy", descr=streq_lengthok_descr) escape(i0) jump(i1, i2) """ - self.optimize_loop_extradescrs(ops, 'Not, Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected) def test_str_equal_chars3(self): ops = """ [p1] - i0 = call(0, "x", p1, descr=strequaldescr) + i0 = call(0, s"x", p1, descr=strequaldescr) escape(i0) jump(p1) """ @@ -4473,14 +4488,14 @@ escape(i0) jump(p1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) def test_str_equal_lengthmismatch1(self): ops = """ [i1] p1 = newstr(1) strsetitem(p1, 0, i1) - i0 = call(0, "xy", p1, descr=strequaldescr) + i0 = call(0, s"xy", p1, descr=strequaldescr) escape(i0) jump(i1) """ @@ -4489,10 +4504,33 @@ escape(0) jump(i1) """ - self.optimize_loop_extradescrs(ops, 'Not', expected) + self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected) - # XXX unicode operations - # XXX str2unicode + def test_str2unicode_constant(self): + ops = """ + [] + p0 = call(0, "xy", descr=s2u_descr) # string -> unicode + escape(p0) + jump() + """ + expected = """ + [] + escape(u"xy") + jump() + """ + self.optimize_strunicode_loop_extradescrs(ops, '', expected) + + def test_str2unicode_nonconstant(self): + ops = """ + [p0] + p1 = call(0, p0, descr=s2u_descr) # string -> unicode + escape(p1) + jump(p1) + """ + self.optimize_strunicode_loop_extradescrs(ops, 'Not', ops) + # more generally, supporting non-constant but virtual cases is + # not obvious, because of the exception UnicodeDecodeError that + # can be raised by ll_str2unicode() ##class TestOOtype(BaseTestOptimizeOpt, OOtypeMixin): Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_resume.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_resume.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_resume.py Tue Oct 12 14:42:59 2010 @@ -240,6 +240,17 @@ return FakeBuiltObject(strconcat=[left, right]) def slice_string(self, str, start, length): return FakeBuiltObject(strslice=[str, start, length]) + def allocate_unicode(self, length): + return FakeBuiltObject(unistring=[None]*length) + def unicode_setitem(self, unistring, i, fieldnum): + value, tag = untag(fieldnum) + assert tag == TAGINT + assert 0 <= i < len(unistring.unistring) + unistring.unistring[i] = value + def concat_unicodes(self, left, right): + return FakeBuiltObject(uniconcat=[left, right]) + def slice_unicode(self, str, start, length): + return FakeBuiltObject(unislice=[str, start, length]) class FakeBuiltObject(object): def __init__(self, **kwds): @@ -304,6 +315,30 @@ assert reader.force_all_virtuals() == [ FakeBuiltObject(strslice=info.fieldnums)] +def test_vuniplaininfo(): + info = VUniPlainInfo() + info.fieldnums = [tag(60, TAGINT)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(unistring=[60])] + +def test_vuniconcatinfo(): + info = VUniConcatInfo() + info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(uniconcat=info.fieldnums)] + +def test_vunisliceinfo(): + info = VUniSliceInfo() + info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX), tag(30, TAGBOX)] + reader = FakeResumeDataReader() + reader._prepare_virtuals([info]) + assert reader.force_all_virtuals() == [ + FakeBuiltObject(unislice=info.fieldnums)] + # ____________________________________________________________ Modified: pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_string.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_string.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/metainterp/test/test_string.py Tue Oct 12 14:42:59 2010 @@ -6,14 +6,17 @@ class StringTests: + _str, _chr = str, chr + def test_eq_residual(self): + _str = self._str jitdriver = JitDriver(greens = [], reds = ['n', 'i', 's']) - global_s = "hello" + global_s = _str("hello") def f(n, b, s): if b: - s += "ello" + s += _str("ello") else: - s += "allo" + s += _str("allo") i = 0 while n > 0: jitdriver.can_enter_jit(s=s, n=n, i=i) @@ -21,18 +24,19 @@ n -= 1 + (s == global_s) i += 1 return i - res = self.meta_interp(f, [10, True, 'h'], listops=True) + res = self.meta_interp(f, [10, True, _str('h')], listops=True) assert res == 5 self.check_loops(**{self.CALL: 1, self.CALL_PURE: 0}) def test_eq_folded(self): + _str = self._str jitdriver = JitDriver(greens = ['s'], reds = ['n', 'i']) - global_s = "hello" + global_s = _str("hello") def f(n, b, s): if b: - s += "ello" + s += _str("ello") else: - s += "allo" + s += _str("allo") i = 0 while n > 0: jitdriver.can_enter_jit(s=s, n=n, i=i) @@ -40,31 +44,18 @@ n -= 1 + (s == global_s) i += 1 return i - res = self.meta_interp(f, [10, True, 'h'], listops=True) + res = self.meta_interp(f, [10, True, _str('h')], listops=True) assert res == 5 self.check_loops(**{self.CALL: 0, self.CALL_PURE: 0}) def test_newstr(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['n', 'm']) def f(n, m): while True: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - bytecode = 'adlfkj' + chr(n) - res = bytecode[n] - m -= 1 - if m < 0: - return ord(res) - res = self.meta_interp(f, [6, 10]) - assert res == 6 - - def test_newunicode(self): - jitdriver = JitDriver(greens = [], reds = ['n', 'm']) - def f(n, m): - while True: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - bytecode = u'adlfkj' + unichr(n) + bytecode = _str('adlfkj') + _chr(n) res = bytecode[n] m -= 1 if m < 0: @@ -73,95 +64,96 @@ assert res == 6 def test_char2string_pure(self): - for dochr in [chr, ]: #unichr]: - jitdriver = JitDriver(greens = [], reds = ['n']) - @dont_look_inside - def escape(x): - pass - def f(n): - while n > 0: - jitdriver.can_enter_jit(n=n) - jitdriver.jit_merge_point(n=n) - s = dochr(n) - if not we_are_jitted(): - s += s # forces to be a string - if n > 100: - escape(s) - n -= 1 - return 42 - self.meta_interp(f, [6]) - self.check_loops(newstr=0, strsetitem=0, strlen=0, - newunicode=0, unicodesetitem=0, unicodelen=0) + _str, _chr = self._str, self._chr + jitdriver = JitDriver(greens = [], reds = ['n']) + @dont_look_inside + def escape(x): + pass + def f(n): + while n > 0: + jitdriver.can_enter_jit(n=n) + jitdriver.jit_merge_point(n=n) + s = _chr(n) + if not we_are_jitted(): + s += s # forces to be a string + if n > 100: + escape(s) + n -= 1 + return 42 + self.meta_interp(f, [6]) + self.check_loops(newstr=0, strsetitem=0, strlen=0, + newunicode=0, unicodesetitem=0, unicodelen=0) def test_char2string_escape(self): - for dochr in [chr, ]: #unichr]: - jitdriver = JitDriver(greens = [], reds = ['n', 'total']) - @dont_look_inside - def escape(x): - return ord(x[0]) - def f(n): - total = 0 - while n > 0: - jitdriver.can_enter_jit(n=n, total=total) - jitdriver.jit_merge_point(n=n, total=total) - s = dochr(n) - if not we_are_jitted(): - s += s # forces to be a string - total += escape(s) - n -= 1 - return total - res = self.meta_interp(f, [6]) - assert res == 21 + _str, _chr = self._str, self._chr + jitdriver = JitDriver(greens = [], reds = ['n', 'total']) + @dont_look_inside + def escape(x): + return ord(x[0]) + def f(n): + total = 0 + while n > 0: + jitdriver.can_enter_jit(n=n, total=total) + jitdriver.jit_merge_point(n=n, total=total) + s = _chr(n) + if not we_are_jitted(): + s += s # forces to be a string + total += escape(s) + n -= 1 + return total + res = self.meta_interp(f, [6]) + assert res == 21 def test_char2string2char(self): - for dochr in [chr, ]: #unichr]: - jitdriver = JitDriver(greens = [], reds = ['m', 'total']) - def f(m): - total = 0 - while m > 0: - jitdriver.can_enter_jit(m=m, total=total) - jitdriver.jit_merge_point(m=m, total=total) - string = dochr(m) - if m > 100: - string += string # forces to be a string - # read back the character - c = string[0] - total += ord(c) - m -= 1 - return total - res = self.meta_interp(f, [6]) - assert res == 21 - self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0, - newunicode=0, unicodegetitem=0, unicodesetitem=0, - unicodelen=0) + _str, _chr = self._str, self._chr + jitdriver = JitDriver(greens = [], reds = ['m', 'total']) + def f(m): + total = 0 + while m > 0: + jitdriver.can_enter_jit(m=m, total=total) + jitdriver.jit_merge_point(m=m, total=total) + string = _chr(m) + if m > 100: + string += string # forces to be a string + # read back the character + c = string[0] + total += ord(c) + m -= 1 + return total + res = self.meta_interp(f, [6]) + assert res == 21 + self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0, + newunicode=0, unicodegetitem=0, unicodesetitem=0, + unicodelen=0) def test_strconcat_pure(self): - for somestr in ["abc", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - mylist = [somestr+str(i) for i in range(10)] - def f(n, m): - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + mylist[m] - if m > 100: - escape(s) - m -= 1 - return 42 - self.meta_interp(f, [6, 7]) - self.check_loops(newstr=0, strsetitem=0, - newunicode=0, unicodesetitem=0, - call=0, call_pure=0) + _str = self._str + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = [_str("abc") + _str(i) for i in range(10)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + mylist[m] + if m > 100: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=0, strsetitem=0, + newunicode=0, unicodesetitem=0, + call=0, call_pure=0) def test_strconcat_escape_str_str(self): + _str = self._str jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) @@ -171,46 +163,64 @@ m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=0, copystrcontent=2, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=0, copystrcontent=2, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=0, + copyunicodecontent=2, + call=1, call_pure=0) # escape def test_strconcat_escape_str_char(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + chr(m) + s = mylist[n] + _chr(m) escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=1, + copyunicodecontent=1, + call=1, call_pure=0) # escape def test_strconcat_escape_char_str(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = chr(n) + mylist[m] + s = _chr(n) + mylist[m] escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=1, copystrcontent=1, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=1, + copyunicodecontent=1, + call=1, call_pure=0) # escape def test_strconcat_escape_char_char(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): @@ -219,91 +229,132 @@ while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = chr(n) + chr(m) + s = _chr(n) + _chr(m) escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=2, copystrcontent=0, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=2, copystrcontent=0, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=2, + copyunicodecontent=0, + call=1, call_pure=0) # escape def test_strconcat_escape_str_char_str(self): + _str, _chr = self._str, self._chr jitdriver = JitDriver(greens = [], reds = ['m', 'n']) @dont_look_inside def escape(x): pass - mylist = ["somestr"+str(i) for i in range(10)] + mylist = [_str("somestr") + _str(i) for i in range(10)] def f(n, m): while m >= 0: jitdriver.can_enter_jit(m=m, n=n) jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + chr(n) + mylist[m] + s = mylist[n] + _chr(n) + mylist[m] escape(s) m -= 1 return 42 self.meta_interp(f, [6, 7]) - self.check_loops(newstr=1, strsetitem=1, copystrcontent=2, - call=1, call_pure=0) # escape + if _str is str: + self.check_loops(newstr=1, strsetitem=1, copystrcontent=2, + call=1, call_pure=0) # escape + else: + self.check_loops(newunicode=1, unicodesetitem=1, + copyunicodecontent=2, + call=1, call_pure=0) # escape def test_strconcat_guard_fail(self): - for somestr in ["abc", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - mylist = [somestr+str(i) for i in range(12)] - def f(n, m): - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = mylist[n] + mylist[m] - if m & 1: - escape(s) - m -= 1 - return 42 - self.meta_interp(f, [6, 10]) + _str = self._str + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + mylist = [_str("abc") + _str(i) for i in range(12)] + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = mylist[n] + mylist[m] + if m & 1: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 10]) def test_strslice(self): - for somestr in ["abc", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - def f(n, m): - assert n >= 0 - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = "foobarbazetc"[m:n] - if m <= 5: - escape(s) - m -= 1 - return 42 - self.meta_interp(f, [10, 10]) + _str = self._str + longstring = _str("foobarbazetc") + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + def f(n, m): + assert n >= 0 + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = longstring[m:n] + if m <= 5: + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [10, 10]) def test_streq_char(self): - for somestr in ["?abcdefg", ]: #u"def"]: - jitdriver = JitDriver(greens = [], reds = ['m', 'n']) - @dont_look_inside - def escape(x): - pass - def f(n, m): - assert n >= 0 - while m >= 0: - jitdriver.can_enter_jit(m=m, n=n) - jitdriver.jit_merge_point(m=m, n=n) - s = somestr[:m] - escape(s == "?") - m -= 1 - return 42 - self.meta_interp(f, [6, 7]) - self.check_loops(newstr=0, newunicode=0) - - -class TestOOtype(StringTests, OOJitMixin): - CALL = "oosend" - CALL_PURE = "oosend_pure" + _str = self._str + longstring = _str("?abcdefg") + somechar = _str("?") + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + @dont_look_inside + def escape(x): + pass + def f(n, m): + assert n >= 0 + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + s = longstring[:m] + escape(s == somechar) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(newstr=0, newunicode=0) + + +#class TestOOtype(StringTests, OOJitMixin): +# CALL = "oosend" +# CALL_PURE = "oosend_pure" class TestLLtype(StringTests, LLJitMixin): CALL = "call" CALL_PURE = "call_pure" + +class TestLLtypeUnicode(TestLLtype): + _str, _chr = unicode, unichr + + def test_str2unicode(self): + _str = self._str + jitdriver = JitDriver(greens = [], reds = ['m', 'n']) + class Foo: + pass + @dont_look_inside + def escape(x): + assert x == _str("6y") + def f(n, m): + while m >= 0: + jitdriver.can_enter_jit(m=m, n=n) + jitdriver.jit_merge_point(m=m, n=n) + foo = Foo() + foo.y = chr(m) + foo.y = "y" + s = _str(str(n)) + _str(foo.y) + escape(s) + m -= 1 + return 42 + self.meta_interp(f, [6, 7]) + self.check_loops(call=3, # str(), _str(), escape() + newunicode=1, unicodegetitem=0, + unicodesetitem=1, copyunicodecontent=1) Modified: pypy/branch/32ptr-on-64bit/pypy/jit/tl/spli/interpreter.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/jit/tl/spli/interpreter.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/jit/tl/spli/interpreter.py Tue Oct 12 14:42:59 2010 @@ -1,12 +1,14 @@ import os -from pypy.tool import stdlib_opcode as opcode +from pypy.tool import stdlib_opcode from pypy.jit.tl.spli import objects, pycode -from pypy.tool.stdlib_opcode import unrolling_opcode_descs -from pypy.tool.stdlib_opcode import opcode_method_names from pypy.rlib.unroll import unrolling_iterable from pypy.rlib.jit import JitDriver, hint, dont_look_inside from pypy.rlib.objectmodel import we_are_translated +opcode_method_names = stdlib_opcode.host_bytecode_spec.method_names +unrolling_opcode_descs = unrolling_iterable( + stdlib_opcode.host_bytecode_spec.ordered_opdescs) +HAVE_ARGUMENT = stdlib_opcode.host_HAVE_ARGUMENT compare_ops = [ "cmp_lt", # "<" @@ -79,7 +81,7 @@ self.stack_depth = hint(self.stack_depth, promote=True) op = ord(code[instr_index]) instr_index += 1 - if op >= opcode.HAVE_ARGUMENT: + if op >= HAVE_ARGUMENT: low = ord(code[instr_index]) hi = ord(code[instr_index + 1]) oparg = (hi << 8) | low @@ -183,6 +185,12 @@ next_instr += arg return next_instr + def POP_JUMP_IF_FALSE(self, arg, next_instr, code): + w_cond = self.pop() + if not w_cond.is_true(): + next_instr = arg + return next_instr + def JUMP_FORWARD(self, arg, next_instr, code): return next_instr + arg Modified: pypy/branch/32ptr-on-64bit/pypy/module/__builtin__/interp_classobj.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/module/__builtin__/interp_classobj.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/module/__builtin__/interp_classobj.py Tue Oct 12 14:42:59 2010 @@ -2,9 +2,11 @@ from pypy.interpreter.error import OperationError, operationerrfmt from pypy.interpreter.gateway import ObjSpace, W_Root, NoneNotWrapped, applevel from pypy.interpreter.gateway import interp2app, ObjSpace -from pypy.interpreter.typedef import TypeDef, make_weakref_descr +from pypy.interpreter.typedef import TypeDef from pypy.interpreter.argument import Arguments from pypy.interpreter.baseobjspace import Wrappable +from pypy.interpreter.typedef import GetSetProperty, descr_get_dict +from pypy.interpreter.typedef import descr_set_dict from pypy.rlib.rarithmetic import r_uint, intmask from pypy.rlib.objectmodel import compute_identity_hash from pypy.rlib.debug import make_sure_not_resized @@ -57,6 +59,14 @@ self.bases_w = bases self.w_dict = w_dict + def instantiate(self, space): + cache = space.fromcache(Cache) + if self.lookup(space, '__del__') is not None: + w_inst = cache.cls_with_del(space, self) + else: + w_inst = cache.cls_without_del(space, self) + return w_inst + def getdict(self): return self.w_dict @@ -100,15 +110,15 @@ return False @jit.unroll_safe - def lookup(self, space, w_attr): + def lookup(self, space, attr): # returns w_value or interplevel None - w_result = space.finditem(self.w_dict, w_attr) + w_result = space.finditem_str(self.w_dict, attr) if w_result is not None: return w_result for base in self.bases_w: # XXX fix annotation of bases_w to be a list of W_ClassObjects assert isinstance(base, W_ClassObject) - w_result = base.lookup(space, w_attr) + w_result = base.lookup(space, attr) if w_result is not None: return w_result return None @@ -122,7 +132,7 @@ return space.wrap(self.name) elif name == "__bases__": return space.newtuple(self.bases_w) - w_value = self.lookup(space, w_attr) + w_value = self.lookup(space, name) if w_value is None: raise operationerrfmt( space.w_AttributeError, @@ -147,7 +157,7 @@ self.setbases(space, w_value) return elif name == "__del__": - if self.lookup(space, w_attr) is None: + if self.lookup(space, name) is None: msg = ("a __del__ method added to an existing class " "will not be called") space.warn(msg, space.w_RuntimeWarning) @@ -195,13 +205,20 @@ # NOT_RPYTHON return '' % self.name +class Cache: + def __init__(self, space): + from pypy.interpreter.typedef import _usersubclswithfeature + # evil + self.cls_without_del = _usersubclswithfeature( + space.config, W_InstanceObject, "dict", "weakref") + self.cls_with_del = _usersubclswithfeature( + space.config, self.cls_without_del, "del") + + def class_descr_call(space, w_self, __args__): self = space.interp_w(W_ClassObject, w_self) - if self.lookup(space, space.wrap('__del__')) is not None: - w_inst = W_InstanceObjectWithDel(space, self) - else: - w_inst = W_InstanceObject(space, self) - w_init = w_inst.getattr_from_class(space, space.wrap('__init__')) + w_inst = self.instantiate(space) + w_init = w_inst.getattr_from_class(space, '__init__') if w_init is not None: w_result = space.call_args(w_init, __args__) if not space.is_w(w_result, space.w_None): @@ -234,7 +251,7 @@ def make_unary_instance_method(name): def unaryop(self, space): - w_meth = self.getattr(space, space.wrap(name), True) + w_meth = self.getattr(space, name, True) return space.call_function(w_meth) unaryop.func_name = name return unaryop @@ -242,7 +259,7 @@ def make_binary_returning_notimplemented_instance_method(name): def binaryop(self, space, w_other): try: - w_meth = self.getattr(space, space.wrap(name), False) + w_meth = self.getattr(space, name, False) except OperationError, e: if e.match(space, space.w_AttributeError): return space.w_NotImplemented @@ -267,7 +284,7 @@ w_a = self w_b = w_other if w_a is self: - w_meth = self.getattr(space, space.wrap(specialname), False) + w_meth = self.getattr(space, specialname, False) if w_meth is None: return space.w_NotImplemented return space.call_function(w_meth, w_b) @@ -278,7 +295,7 @@ def rbinaryop(self, space, w_other): w_a, w_b = _coerce_helper(space, self, w_other) if w_a is None or w_a is self: - w_meth = self.getattr(space, space.wrap(rspecialname), False) + w_meth = self.getattr(space, rspecialname, False) if w_meth is None: return space.w_NotImplemented return space.call_function(w_meth, w_other) @@ -302,46 +319,34 @@ raise OperationError( space.w_TypeError, space.wrap("instance() first arg must be class")) - if space.is_w(w_dict, space.w_None): - w_dict = None - elif not space.is_true(space.isinstance(w_dict, space.w_dict)): - raise OperationError( - space.w_TypeError, - space.wrap("instance() second arg must be dictionary or None")) - return W_InstanceObject(space, w_class, w_dict) + w_result = w_class.instantiate(space) + if not space.is_w(w_dict, space.w_None): + w_result.setdict(space, w_dict) + return w_result class W_InstanceObject(Wrappable): - def __init__(self, space, w_class, w_dict=None): - if w_dict is None: - w_dict = space.newdict(instance=True) + def __init__(self, space, w_class): + # note that user_setup is overridden by the typedef.py machinery + self.user_setup(space, space.gettypeobject(self.typedef)) assert isinstance(w_class, W_ClassObject) self.w_class = w_class - self.w_dict = w_dict - self.space = space - - def getdict(self): - return self.w_dict - def setdict(self, space, w_dict): - if (w_dict is None or - not space.is_true(space.isinstance(w_dict, space.w_dict))): - raise OperationError( - space.w_TypeError, - space.wrap("__dict__ must be a dictionary object")) - self.w_dict = w_dict + def user_setup(self, space, w_subtype): + self.space = space - def setclass(self, space, w_class): + def set_oldstyle_class(self, space, w_class): if w_class is None or not isinstance(w_class, W_ClassObject): raise OperationError( space.w_TypeError, space.wrap("__class__ must be set to a class")) self.w_class = w_class - def getattr_from_class(self, space, w_name): + def getattr_from_class(self, space, name): # Look up w_name in the class dict, and call its __get__. # This method ignores the instance dict and the __getattr__. # Returns None if not found. - w_value = self.w_class.lookup(space, w_name) + assert isinstance(name, str) + w_value = self.w_class.lookup(space, name) if w_value is None: return None w_descr_get = space.lookup(w_value, '__get__') @@ -349,19 +354,20 @@ return w_value return space.call_function(w_descr_get, w_value, self, self.w_class) - def getattr(self, space, w_name, exc=True): + def getattr(self, space, name, exc=True): # Normal getattr rules: look up w_name in the instance dict, # in the class dict, and then via a call to __getatttr__. - w_result = space.finditem(self.w_dict, w_name) + assert isinstance(name, str) + w_result = self.getdictvalue(space, name) if w_result is not None: return w_result - w_result = self.getattr_from_class(space, w_name) + w_result = self.getattr_from_class(space, name) if w_result is not None: return w_result - w_meth = self.getattr_from_class(space, space.wrap('__getattr__')) + w_meth = self.getattr_from_class(space, '__getattr__') if w_meth is not None: try: - return space.call_function(w_meth, w_name) + return space.call_function(w_meth, space.wrap(name)) except OperationError, e: if not exc and e.match(space, space.w_AttributeError): return None # eat the AttributeError @@ -371,7 +377,7 @@ raise operationerrfmt( space.w_AttributeError, "%s instance has no attribute '%s'", - self.w_class.name, space.str_w(w_name)) + self.w_class.name, name) else: return None @@ -379,44 +385,46 @@ name = space.str_w(w_attr) if len(name) >= 8 and name[0] == '_': if name == "__dict__": - return self.w_dict + return self.getdict() elif name == "__class__": return self.w_class - return self.getattr(space, w_attr) + return self.getattr(space, name) def descr_setattr(self, space, w_name, w_value): name = unwrap_attr(space, w_name) - w_meth = self.getattr_from_class(space, space.wrap('__setattr__')) + w_meth = self.getattr_from_class(space, '__setattr__') if name and name[0] == "_": if name == '__dict__': self.setdict(space, w_value) return if name == '__class__': - self.setclass(space, w_value) + self.set_oldstyle_class(space, w_value) return if name == '__del__' and w_meth is None: - if (not isinstance(self, W_InstanceObjectWithDel) - and space.finditem(self.w_dict, w_name) is None): + cache = space.fromcache(Cache) + if (not isinstance(self, cache.cls_with_del) + and self.getdictvalue(space, '__del__') is None): msg = ("a __del__ method added to an instance " "with no __del__ in the class will not be called") space.warn(msg, space.w_RuntimeWarning) if w_meth is not None: space.call_function(w_meth, w_name, w_value) else: - self.setdictvalue(space, name, w_value) + # bit obscure: appease normalization + self.setdictvalue(space, name, w_value, True) def descr_delattr(self, space, w_name): name = unwrap_attr(space, w_name) if name and name[0] == "_": if name == '__dict__': # use setdict to raise the error - self.setdict(space, None) + self.setdict(space, space.w_None) return elif name == '__class__': - # use setclass to raise the error - self.setclass(space, None) + # use set_oldstyle_class to raise the error + self.set_oldstyle_class(space, None) return - w_meth = self.getattr_from_class(space, space.wrap('__delattr__')) + w_meth = self.getattr_from_class(space, '__delattr__') if w_meth is not None: space.call_function(w_meth, w_name) else: @@ -427,7 +435,7 @@ self.w_class.name, name) def descr_repr(self, space): - w_meth = self.getattr(space, space.wrap('__repr__'), False) + w_meth = self.getattr(space, '__repr__', False) if w_meth is None: w_class = self.w_class mod = w_class.get_module_string(space) @@ -435,19 +443,19 @@ return space.call_function(w_meth) def descr_str(self, space): - w_meth = self.getattr(space, space.wrap('__str__'), False) + w_meth = self.getattr(space, '__str__', False) if w_meth is None: return self.descr_repr(space) return space.call_function(w_meth) def descr_unicode(self, space): - w_meth = self.getattr(space, space.wrap('__unicode__'), False) + w_meth = self.getattr(space, '__unicode__', False) if w_meth is None: return self.descr_str(space) return space.call_function(w_meth) def descr_len(self, space): - w_meth = self.getattr(space, space.wrap('__len__')) + w_meth = self.getattr(space, '__len__') w_result = space.call_function(w_meth) if space.is_true(space.isinstance(w_result, space.w_int)): if space.is_true(space.lt(w_result, space.wrap(0))): @@ -460,22 +468,22 @@ space.wrap("__len__() should return an int")) def descr_getitem(self, space, w_key): - w_meth = self.getattr(space, space.wrap('__getitem__')) + w_meth = self.getattr(space, '__getitem__') return space.call_function(w_meth, w_key) def descr_setitem(self, space, w_key, w_value): - w_meth = self.getattr(space, space.wrap('__setitem__')) + w_meth = self.getattr(space, '__setitem__') space.call_function(w_meth, w_key, w_value) def descr_delitem(self, space, w_key): - w_meth = self.getattr(space, space.wrap('__delitem__')) + w_meth = self.getattr(space, '__delitem__') space.call_function(w_meth, w_key) def descr_iter(self, space): - w_meth = self.getattr(space, space.wrap('__iter__'), False) + w_meth = self.getattr(space, '__iter__', False) if w_meth is not None: return space.call_function(w_meth) - w_meth = self.getattr(space, space.wrap('__getitem__'), False) + w_meth = self.getattr(space, '__getitem__', False) if w_meth is None: raise OperationError( space.w_TypeError, @@ -485,14 +493,14 @@ # don't see the point def descr_getslice(self, space, w_i, w_j): - w_meth = self.getattr(space, space.wrap('__getslice__'), False) + w_meth = self.getattr(space, '__getslice__', False) if w_meth is not None: return space.call_function(w_meth, w_i, w_j) else: return space.getitem(self, space.newslice(w_i, w_j, space.w_None)) def descr_setslice(self, space, w_i, w_j, w_sequence): - w_meth = self.getattr(space, space.wrap('__setslice__'), False) + w_meth = self.getattr(space, '__setslice__', False) if w_meth is not None: space.call_function(w_meth, w_i, w_j, w_sequence) else: @@ -500,20 +508,20 @@ w_sequence) def descr_delslice(self, space, w_i, w_j): - w_meth = self.getattr(space, space.wrap('__delslice__'), False) + w_meth = self.getattr(space, '__delslice__', False) if w_meth is not None: space.call_function(w_meth, w_i, w_j) else: return space.delitem(self, space.newslice(w_i, w_j, space.w_None)) def descr_call(self, space, __args__): - w_meth = self.getattr(space, space.wrap('__call__')) + w_meth = self.getattr(space, '__call__') return space.call_args(w_meth, __args__) def descr_nonzero(self, space): - w_func = self.getattr(space, space.wrap('__nonzero__'), False) + w_func = self.getattr(space, '__nonzero__', False) if w_func is None: - w_func = self.getattr(space, space.wrap('__len__'), False) + w_func = self.getattr(space, '__len__', False) if w_func is None: return space.w_True w_result = space.call_function(w_func) @@ -537,7 +545,7 @@ not isinstance(w_b, W_InstanceObject)): return space.cmp(w_a, w_b) if isinstance(w_a, W_InstanceObject): - w_func = w_a.getattr(space, space.wrap('__cmp__'), False) + w_func = w_a.getattr(space, '__cmp__', False) if w_func is not None: w_res = space.call_function(w_func, w_b) if space.is_w(w_res, space.w_NotImplemented): @@ -556,7 +564,7 @@ return space.wrap(-1) return space.wrap(0) if isinstance(w_b, W_InstanceObject): - w_func = w_b.getattr(space, space.wrap('__cmp__'), False) + w_func = w_b.getattr(space, '__cmp__', False) if w_func is not None: w_res = space.call_function(w_func, w_a) if space.is_w(w_res, space.w_NotImplemented): @@ -577,10 +585,10 @@ return space.w_NotImplemented def descr_hash(self, space): - w_func = self.getattr(space, space.wrap('__hash__'), False) + w_func = self.getattr(space, '__hash__', False) if w_func is None: - w_eq = self.getattr(space, space.wrap('__eq__'), False) - w_cmp = self.getattr(space, space.wrap('__cmp__'), False) + w_eq = self.getattr(space, '__eq__', False) + w_cmp = self.getattr(space, '__cmp__', False) if w_eq is not None or w_cmp is not None: raise OperationError(space.w_TypeError, space.wrap("unhashable instance")) @@ -595,7 +603,7 @@ return w_ret def descr_index(self, space): - w_func = self.getattr(space, space.wrap('__index__'), False) + w_func = self.getattr(space, '__index__', False) if w_func is not None: return space.call_function(w_func) raise OperationError( @@ -603,7 +611,7 @@ space.wrap("object cannot be interpreted as an index")) def descr_contains(self, space, w_obj): - w_func = self.getattr(space, space.wrap('__contains__'), False) + w_func = self.getattr(space, '__contains__', False) if w_func is not None: return space.wrap(space.is_true(space.call_function(w_func, w_obj))) # now do it ourselves @@ -626,7 +634,7 @@ w_a = self w_b = w_other if w_a is self: - w_func = self.getattr(space, space.wrap('__pow__'), False) + w_func = self.getattr(space, '__pow__', False) if w_func is not None: return space.call_function(w_func, w_other) return space.w_NotImplemented @@ -634,7 +642,7 @@ return space.pow(w_a, w_b, space.w_None) else: # CPython also doesn't try coercion in this case - w_func = self.getattr(space, space.wrap('__pow__'), False) + w_func = self.getattr(space, '__pow__', False) if w_func is not None: return space.call_function(w_func, w_other, w_modulo) return space.w_NotImplemented @@ -646,7 +654,7 @@ w_a = self w_b = w_other if w_a is self: - w_func = self.getattr(space, space.wrap('__rpow__'), False) + w_func = self.getattr(space, '__rpow__', False) if w_func is not None: return space.call_function(w_func, w_other) return space.w_NotImplemented @@ -654,13 +662,13 @@ return space.pow(w_b, w_a, space.w_None) else: # CPython also doesn't try coercion in this case - w_func = self.getattr(space, space.wrap('__rpow__'), False) + w_func = self.getattr(space, '__rpow__', False) if w_func is not None: return space.call_function(w_func, w_other, w_modulo) return space.w_NotImplemented def descr_next(self, space): - w_func = self.getattr(space, space.wrap('next'), False) + w_func = self.getattr(space, 'next', False) if w_func is None: raise OperationError(space.w_TypeError, space.wrap("instance has no next() method")) @@ -669,10 +677,9 @@ def descr_del(self, space): # Note that this is called from executioncontext.UserDelAction # via the space.userdel() method. - w_name = space.wrap('__del__') - w_func = space.finditem(self.w_dict, w_name) + w_func = self.getdictvalue(space, '__del__') if w_func is None: - w_func = self.getattr_from_class(space, w_name) + w_func = self.getattr_from_class(space, '__del__') if w_func is not None: space.call_function(w_func) @@ -717,6 +724,14 @@ rmeth, unwrap_spec=["self", ObjSpace, W_Root]) + +def descr_del_dict(space, w_inst): + # use setdict to raise the error + w_inst.setdict(space, space.w_None) + +dict_descr = GetSetProperty(descr_get_dict, descr_set_dict, descr_del_dict) +dict_descr.name = '__dict__' + W_InstanceObject.typedef = TypeDef("instance", __new__ = interp2app(descr_instance_new), __getattribute__ = interp2app(W_InstanceObject.descr_getattribute, @@ -766,12 +781,9 @@ unwrap_spec=['self', ObjSpace, W_Root, W_Root]), next = interp2app(W_InstanceObject.descr_next, unwrap_spec=['self', ObjSpace]), - __weakref__ = make_weakref_descr(W_InstanceObject), __del__ = interp2app(W_InstanceObject.descr_del, unwrap_spec=['self', ObjSpace]), + __dict__ = dict_descr, **rawdict ) - -class W_InstanceObjectWithDel(W_InstanceObject): - def __del__(self): - self._enqueue_for_destruction(self.space) +W_InstanceObject.typedef.acceptable_as_base_class = False Modified: pypy/branch/32ptr-on-64bit/pypy/module/__builtin__/test/test_classobj.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/module/__builtin__/test/test_classobj.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/module/__builtin__/test/test_classobj.py Tue Oct 12 14:42:59 2010 @@ -928,6 +928,31 @@ assert x is b assert y == 5 + def test_cant_subclass_instance(self): + class A: + pass + try: + class B(type(A())): + pass + except TypeError: + pass + else: + assert 0, "should have raised" + + def test_dict_descriptor(self): + import sys + if not hasattr(sys, 'pypy_objspaceclass'): + skip("on CPython old-style instances don't have a __dict__ descriptor") + class A: + pass + a = A() + a.x = 1 + descr = type(a).__dict__['__dict__'] + assert descr.__get__(a) == {'x': 1} + descr.__set__(a, {'x': 2}) + assert a.x == 2 + raises(TypeError, descr.__delete__, a) + class AppTestOldStyleSharing(AppTestOldstyle): def setup_class(cls): @@ -966,3 +991,22 @@ a = 1 b = 2 assert self.is_strdict(A) + +class AppTestOldStyleMapDict(AppTestOldstyle): + def setup_class(cls): + cls.space = gettestobjspace(**{"objspace.std.withmapdict": True}) + if option.runappdirect: + py.test.skip("can only be run on py.py") + def has_mapdict(space, w_inst): + return space.wrap(w_inst._get_mapdict_map() is not None) + cls.w_has_mapdict = cls.space.wrap(gateway.interp2app(has_mapdict)) + + + def test_has_mapdict(self): + class A: + def __init__(self): + self.x = 42 + a = A() + assert a.x == 42 + assert self.has_mapdict(a) + Modified: pypy/branch/32ptr-on-64bit/pypy/module/_weakref/interp__weakref.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/module/_weakref/interp__weakref.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/module/_weakref/interp__weakref.py Tue Oct 12 14:42:59 2010 @@ -7,7 +7,7 @@ import weakref -class WeakrefLifeline(object): +class WeakrefLifeline(W_Root): def __init__(self, space): self.space = space # this is here for W_Root.clear_all_weakrefs() self.refs_weak = [] Modified: pypy/branch/32ptr-on-64bit/pypy/module/cpyext/classobject.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/module/cpyext/classobject.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/module/cpyext/classobject.py Tue Oct 12 14:42:59 2010 @@ -15,16 +15,20 @@ class is the class of new object. The dict parameter will be used as the object's __dict__; if NULL, a new dictionary will be created for the instance.""" - if not PyClass_Check(space, w_class): + if not isinstance(w_class, W_ClassObject): return PyErr_BadInternalCall(space) - return W_InstanceObject(space, w_class, w_dict) + w_result = w_class.instantiate(space) + if w_dict is not None: + w_result.setdict(space, w_dict) + return w_result @cpython_api([PyObject, PyObject], PyObject, error=CANNOT_FAIL) def _PyInstance_Lookup(space, w_instance, w_name): + name = space.str_w(w_name) assert isinstance(w_instance, W_InstanceObject) - w_result = space.finditem(w_instance.w_dict, w_name) + w_result = w_instance.getdictvalue(space, name) if w_result is not None: return w_result - return w_instance.w_class.lookup(space, w_name) + return w_instance.w_class.lookup(space, name) Modified: pypy/branch/32ptr-on-64bit/pypy/module/gc/referents.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/module/gc/referents.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/module/gc/referents.py Tue Oct 12 14:42:59 2010 @@ -15,6 +15,10 @@ def try_cast_gcref_to_w_root(gcref): w_obj = rgc.try_cast_gcref_to_instance(W_Root, gcref) + # Ignore instances of exactly the class W_Root. Seems that + # we get one on 64-bit translations, for some reason. + if w_obj is not None and type(w_obj) is W_Root: + w_obj = None if not we_are_translated() and not hasattr(w_obj, 'typedef'): w_obj = None return w_obj Modified: pypy/branch/32ptr-on-64bit/pypy/module/pypyjit/test/test_pypy_c.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/module/pypyjit/test/test_pypy_c.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/module/pypyjit/test/test_pypy_c.py Tue Oct 12 14:42:59 2010 @@ -272,7 +272,7 @@ assert len(ops) == 2 assert not ops[0].get_opnames("call") assert not ops[0].get_opnames("new") - assert len(ops[0].get_opnames("guard")) <= 7 + assert len(ops[0].get_opnames("guard")) <= 2 assert not ops[1] # second LOOKUP_METHOD folded away ops = self.get_by_bytecode("CALL_METHOD") @@ -283,7 +283,7 @@ else: assert not bytecode.get_opnames("call") assert not bytecode.get_opnames("new") - assert len(bytecode.get_opnames("guard")) <= 9 + assert len(bytecode.get_opnames("guard")) <= 6 assert len(ops[1]) < len(ops[0]) ops = self.get_by_bytecode("LOAD_ATTR") @@ -317,8 +317,8 @@ assert len(ops) == 2 assert not ops[0].get_opnames("call") assert not ops[0].get_opnames("new") - assert len(ops[0].get_opnames("guard")) <= 7 - assert len(ops[0].get_opnames("getfield")) < 6 + assert len(ops[0].get_opnames("guard")) <= 2 + assert len(ops[0].get_opnames("getfield")) < 5 assert not ops[1] # second LOOKUP_METHOD folded away def test_default_and_kw(self): @@ -382,7 +382,7 @@ a.x = 2 i = i + a.x return i - ''', 67, + ''', 69, ([20], 20), ([31], 32)) @@ -390,7 +390,7 @@ self.get_by_bytecode("CALL_FUNCTION")) assert not callA.get_opnames("call") assert not callA.get_opnames("new") - assert len(callA.get_opnames("guard")) <= 8 + assert len(callA.get_opnames("guard")) <= 2 assert not callisinstance1.get_opnames("call") assert not callisinstance1.get_opnames("new") assert len(callisinstance1.get_opnames("guard")) <= 2 @@ -742,6 +742,8 @@ '''%(op1, float(a)/4.0, float(b)/4.0, op2), 109, ([], res)) def test_boolrewrite_ptr(self): + # XXX this test is way too imprecise in what it is actually testing + # it should count the number of guards instead compares = ('a == b', 'b == a', 'a != b', 'b != a', 'a == c', 'c != b') for e1 in compares: for e2 in compares: @@ -765,7 +767,7 @@ print print 'Test:', e1, e2, n, res self.run_source(''' - class tst: + class tst(object): pass def main(): a = tst() @@ -847,6 +849,8 @@ ''', 65, ([], 122880)) def test_array_intimg(self): + # XXX this test is way too imprecise in what it is actually testing + # it should count the number of guards instead for tc, maxops in zip('ilILd', (67, 67, 69, 69, 61)): res = 73574560 if tc in 'IL': Modified: pypy/branch/32ptr-on-64bit/pypy/module/signal/interp_signal.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/module/signal/interp_signal.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/module/signal/interp_signal.py Tue Oct 12 14:42:59 2010 @@ -1,7 +1,6 @@ from pypy.interpreter.error import OperationError from pypy.interpreter.baseobjspace import W_Root, ObjSpace from pypy.interpreter.executioncontext import AsyncAction, AbstractActionFlag -from pypy.rlib.rarithmetic import LONG_BIT, intmask import signal as cpy_signal from pypy.rpython.lltypesystem import lltype, rffi from pypy.translator.tool.cbuild import ExternalCompilationInfo @@ -64,8 +63,8 @@ class CheckSignalAction(AsyncAction): """An action that is automatically invoked when a signal is received.""" - # The C-level signal handler sets the highest bit of pypysig_occurred: - bitmask = intmask(1 << (LONG_BIT-1)) + # The C-level signal handler sets the bit 30 of pypysig_occurred: + bitmask = 1 << 30 def __init__(self, space): AsyncAction.__init__(self, space) Modified: pypy/branch/32ptr-on-64bit/pypy/module/sys/__init__.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/module/sys/__init__.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/module/sys/__init__.py Tue Oct 12 14:42:59 2010 @@ -7,13 +7,15 @@ """Sys Builtin Module. """ def __init__(self, space, w_name): """NOT_RPYTHON""" # because parent __init__ isn't + if space.config.translating: + del self.__class__.interpleveldefs['pypy_getudir'] super(Module, self).__init__(space, w_name) self.checkinterval = 100 self.recursionlimit = 100 self.w_default_encoder = None self.defaultencoding = "ascii" self.filesystemencoding = None - + interpleveldefs = { '__name__' : '(space.wrap("sys"))', '__doc__' : '(space.wrap("PyPy sys module"))', @@ -37,7 +39,7 @@ 'argv' : 'state.get(space).w_argv', 'warnoptions' : 'state.get(space).w_warnoptions', 'builtin_module_names' : 'state.w_None', - 'pypy_getudir' : 'state.pypy_getudir', + 'pypy_getudir' : 'state.pypy_getudir', # not translated 'pypy_initial_path' : 'state.pypy_initial_path', '_getframe' : 'vm._getframe', Modified: pypy/branch/32ptr-on-64bit/pypy/module/sys/state.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/module/sys/state.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/module/sys/state.py Tue Oct 12 14:42:59 2010 @@ -95,15 +95,8 @@ def getio(space): return space.fromcache(IOState) -def _pypy_getudir(space): - """NOT_RPYTHON""" +def pypy_getudir(space): + """NOT_RPYTHON + (should be removed from interpleveldefs before translation)""" from pypy.tool.udir import udir return space.wrap(str(udir)) -_pypy_getudir._annspecialcase_ = "override:ignore" - -# we need the indirection because this function will live in a dictionary with other -# RPYTHON functions and share call sites with them. Better it not be a special-case -# directly. -def pypy_getudir(space): - return _pypy_getudir(space) - Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/celldict.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/celldict.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/celldict.py Tue Oct 12 14:42:59 2010 @@ -45,7 +45,7 @@ if space.is_w(space.type(w_key), space.w_str): self.impl_setitem_str(self.space.str_w(w_key), w_value) else: - self._as_rdict().setitem(w_key, w_value) + self._as_rdict().impl_fallback_setitem(w_key, w_value) def impl_setitem_str(self, name, w_value, shadows_type=True): self.getcell(name, True).w_value = w_value @@ -66,7 +66,7 @@ elif _is_sane_hash(space, w_key_type): raise KeyError else: - self._as_rdict().delitem(w_key) + self._as_rdict().impl_fallback_delitem(w_key) def impl_length(self): # inefficient, but do we care? @@ -85,7 +85,7 @@ elif _is_sane_hash(space, w_lookup_type): return None else: - return self._as_rdict().getitem(w_lookup) + return self._as_rdict().impl_fallback_getitem(w_lookup) def impl_getitem_str(self, lookup): res = self.getcell(lookup, False) Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/dictmultiobject.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/dictmultiobject.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/dictmultiobject.py Tue Oct 12 14:42:59 2010 @@ -102,17 +102,17 @@ else: return None - # _________________________________________________________________ + # _________________________________________________________________ # implementation methods def impl_getitem(self, w_key): #return w_value or None raise NotImplementedError("abstract base class") - def impl_getitem_str(self, w_key): + def impl_getitem_str(self, key): #return w_value or None raise NotImplementedError("abstract base class") - def impl_setitem_str(self, key, w_value, shadows_type=True): + def impl_setitem_str(self, key, w_value, shadows_type=True): raise NotImplementedError("abstract base class") def impl_setitem(self, w_key, w_value): @@ -120,7 +120,7 @@ def impl_delitem(self, w_key): raise NotImplementedError("abstract base class") - + def impl_length(self): raise NotImplementedError("abstract base class") @@ -310,7 +310,7 @@ if space.is_w(space.type(w_key), space.w_str): self.impl_setitem_str(self.space.str_w(w_key), w_value) else: - self._as_rdict().setitem(w_key, w_value) + self._as_rdict().impl_fallback_setitem(w_key, w_value) def impl_setitem_str(self, key, w_value, shadows_type=True): self.content[key] = w_value @@ -324,7 +324,7 @@ elif _is_sane_hash(space, w_key_type): raise KeyError else: - self._as_rdict().delitem(w_key) + self._as_rdict().impl_fallback_delitem(w_key) def impl_length(self): return len(self.content) @@ -344,7 +344,7 @@ elif _is_sane_hash(space, w_lookup_type): return None else: - return self._as_rdict().getitem(w_key) + return self._as_rdict().impl_fallback_getitem(w_key) def impl_iter(self): return StrIteratorImplementation(self.space, self) @@ -414,7 +414,7 @@ StrDictImplementation.impl_setitem_str( self, self.space.str_w(w_key), w_value, False) else: - self._as_rdict().setitem(w_key, w_value) + self._as_rdict().impl_fallback_setitem(w_key, w_value) def impl_shadows_anything(self): return (self._shadows_anything or @@ -446,7 +446,7 @@ elif _is_sane_hash(space, w_key_type): raise KeyError else: - self._as_rdict().delitem(w_key) + self._as_rdict().impl_fallback_delitem(w_key) def impl_get_builtin_indexed(self, i): return self.shadowed[i] Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/fake.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/fake.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/fake.py Tue Oct 12 14:42:59 2010 @@ -21,7 +21,6 @@ #debug_print("faking obj %s" % x) ft = fake_type(type(x)) return ft(space, x) -fake_object._annspecialcase_ = "override:fake_object" import sys @@ -47,7 +46,6 @@ w_exc = space.wrap(exc) w_value = space.wrap(value) raise OperationError, OperationError(w_exc, w_value), tb -wrap_exception._annspecialcase_ = "override:ignore" def fake_type(cpy_type): assert type(cpy_type) is type Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/model.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/model.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/model.py Tue Oct 12 14:42:59 2010 @@ -18,6 +18,7 @@ "withsmallint" : ["smallintobject.W_SmallIntObject"], "withstrslice" : ["strsliceobject.W_StringSliceObject"], "withstrjoin" : ["strjoinobject.W_StringJoinObject"], + "withstrbuf" : ["strbufobject.W_StringBufferObject"], "withrope" : ["ropeobject.W_RopeObject", "ropeobject.W_RopeIterObject"], "withropeunicode": ["ropeunicodeobject.W_RopeUnicodeObject", @@ -75,6 +76,7 @@ from pypy.objspace.std import ropeunicodeobject from pypy.objspace.std import strsliceobject from pypy.objspace.std import strjoinobject + from pypy.objspace.std import strbufobject from pypy.objspace.std import typeobject from pypy.objspace.std import sliceobject from pypy.objspace.std import longobject @@ -222,6 +224,13 @@ (unicodeobject.W_UnicodeObject, strjoinobject.delegate_join2unicode) ] + elif config.objspace.std.withstrbuf: + self.typeorder[strbufobject.W_StringBufferObject] += [ + (stringobject.W_StringObject, + strbufobject.delegate_buf2str), + (unicodeobject.W_UnicodeObject, + strbufobject.delegate_buf2unicode) + ] if config.objspace.std.withrangelist: self.typeorder[rangeobject.W_RangeListObject] += [ (listobject.W_ListObject, Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/objspace.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/objspace.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/objspace.py Tue Oct 12 14:42:59 2010 @@ -23,6 +23,7 @@ from pypy.objspace.std.listobject import W_ListObject from pypy.objspace.std.longobject import W_LongObject from pypy.objspace.std.noneobject import W_NoneObject +from pypy.objspace.std.objectobject import W_ObjectObject from pypy.objspace.std.ropeobject import W_RopeObject from pypy.objspace.std.iterobject import W_SeqIterObject from pypy.objspace.std.setobject import W_SetObject, W_FrozensetObject @@ -242,7 +243,6 @@ w_result = getattr(self, 'w_' + x.__name__) return w_result return None - wrap_exception_cls._annspecialcase_ = "override:wrap_exception_cls" def unwrap(self, w_obj): if isinstance(w_obj, Wrappable): @@ -318,9 +318,14 @@ w_subtype = w_type.check_user_subclass(w_subtype) if cls.typedef.applevel_subclasses_base is not None: cls = cls.typedef.applevel_subclasses_base - subcls = get_unique_interplevel_subclass( - self.config, cls, w_subtype.hasdict, w_subtype.nslots != 0, - w_subtype.needsdel, w_subtype.weakrefable) + if (self.config.objspace.std.withmapdict and cls is W_ObjectObject + and not w_subtype.needsdel): + from pypy.objspace.std.mapdict import get_subclass_of_correct_size + subcls = get_subclass_of_correct_size(self, cls, w_subtype) + else: + subcls = get_unique_interplevel_subclass( + self.config, cls, w_subtype.hasdict, w_subtype.nslots != 0, + w_subtype.needsdel, w_subtype.weakrefable) instance = instantiate(subcls) assert isinstance(instance, cls) instance.user_setup(self, w_subtype) Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/sharingdict.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/sharingdict.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/sharingdict.py Tue Oct 12 14:42:59 2010 @@ -71,7 +71,7 @@ elif _is_sane_hash(space, w_lookup_type): return None else: - return self._as_rdict().getitem(w_lookup) + return self._as_rdict().impl_fallback_getitem(w_lookup) def impl_getitem_str(self, lookup): i = self.structure.lookup_position(lookup) @@ -84,7 +84,7 @@ if space.is_w(space.type(w_key), space.w_str): self.impl_setitem_str(self.space.str_w(w_key), w_value) else: - self._as_rdict().setitem(w_key, w_value) + self._as_rdict().impl_fallback_setitem(w_key, w_value) @unroll_safe def impl_setitem_str(self, key, w_value, shadows_type=True): @@ -132,7 +132,7 @@ elif _is_sane_hash(space, w_key_type): raise KeyError else: - self._as_rdict().delitem(w_key) + self._as_rdict().impl_fallback_delitem(w_key) def impl_length(self): return self.structure.length Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/stringobject.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/stringobject.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/stringobject.py Tue Oct 12 14:42:59 2010 @@ -14,7 +14,7 @@ from pypy.rlib.rstring import StringBuilder, string_repeat from pypy.interpreter.buffer import StringBuffer -from pypy.objspace.std.stringtype import sliced, joined, wrapstr, wrapchar, \ +from pypy.objspace.std.stringtype import sliced, wrapstr, wrapchar, \ stringendswith, stringstartswith, joined2 from pypy.objspace.std.formatting import mod_format Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/stringtype.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/stringtype.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/stringtype.py Tue Oct 12 14:42:59 2010 @@ -55,19 +55,14 @@ return W_StringSliceObject(s, start, stop) return wrapstr(space, s[start:stop]) -def joined(space, strlist): - assert not space.config.objspace.std.withrope - if space.config.objspace.std.withstrjoin: - from pypy.objspace.std.strjoinobject import W_StringJoinObject - return W_StringJoinObject(strlist) - else: - return wrapstr(space, "".join(strlist)) - def joined2(space, str1, str2): assert not space.config.objspace.std.withrope if space.config.objspace.std.withstrjoin: from pypy.objspace.std.strjoinobject import W_StringJoinObject return W_StringJoinObject([str1, str2]) + elif space.config.objspace.std.withstrbuf: + from pypy.objspace.std.strbufobject import joined2 + return joined2(str1, str2) else: return wrapstr(space, str1 + str2) Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_dictmultiobject.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_dictmultiobject.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_dictmultiobject.py Tue Oct 12 14:42:59 2010 @@ -602,6 +602,15 @@ classofinstance=classofinstance, from_strdict_shared=from_strdict_shared) + def finditem_str(self, w_dict, s): + return w_dict.getitem_str(s) # assume it's a multidict + + def setitem_str(self, w_dict, s, w_value): + return w_dict.setitem_str(s, w_value) # assume it's a multidict + + def delitem(self, w_dict, w_s): + return w_dict.delitem(w_s) # assume it's a multidict + def allocate_instance(self, cls, type): return object.__new__(cls) @@ -611,7 +620,7 @@ w_StopIteration = StopIteration w_None = None StringObjectCls = FakeString - w_dict = None + w_dict = W_DictMultiObject iter = iter fixedview = list listview = list @@ -687,6 +696,14 @@ assert self.impl.length() == 0 self.check_not_devolved() + def test_clear(self): + self.fill_impl() + assert self.impl.length() == 2 + self.impl.clear() + assert self.impl.length() == 0 + self.check_not_devolved() + + def test_keys(self): self.fill_impl() keys = self.impl.keys() Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_shadowtracking.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_shadowtracking.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_shadowtracking.py Tue Oct 12 14:42:59 2010 @@ -3,7 +3,8 @@ class TestShadowTracking(object): def setup_class(cls): - cls.space = gettestobjspace(**{"objspace.std.withshadowtracking": True}) + cls.space = gettestobjspace(**{"objspace.std.withshadowtracking": True, + "objspace.std.withmapdict": False}) def test_simple_shadowing(self): space = self.space Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/typeobject.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/objspace/std/typeobject.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/typeobject.py Tue Oct 12 14:42:59 2010 @@ -75,7 +75,9 @@ 'weakrefable', 'hasdict', 'nslots', - 'instancetypedef'] + 'instancetypedef', + 'terminator', + ] # for config.objspace.std.getattributeshortcut # (False is a conservative default, fixed during real usage) @@ -116,6 +118,12 @@ # dict_w of any of the types in the mro changes, or if the mro # itself changes w_self._version_tag = VersionTag() + if space.config.objspace.std.withmapdict: + from pypy.objspace.std.mapdict import DictTerminator, NoDictTerminator + if w_self.hasdict: + w_self.terminator = DictTerminator(space, w_self) + else: + w_self.terminator = NoDictTerminator(space, w_self) def mutated(w_self): space = w_self.space Modified: pypy/branch/32ptr-on-64bit/pypy/rlib/_rsocket_rffi.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rlib/_rsocket_rffi.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rlib/_rsocket_rffi.py Tue Oct 12 14:42:59 2010 @@ -324,10 +324,11 @@ ('sll_hatype', rffi.INT), ('sll_addr', rffi.CFixedArray(rffi.CHAR, 8)), ('sll_halen', rffi.INT)], - ) + ifdef='AF_PACKET') CConfig.ifreq = platform.Struct('struct ifreq', [('ifr_ifindex', rffi.INT), - ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))]) + ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))], + ifdef='AF_PACKET') if _WIN32: CConfig.WSAEVENT = platform.SimpleType('WSAEVENT', rffi.VOIDP) @@ -532,8 +533,9 @@ socketpair_t = rffi.CArray(socketfd_type) socketpair = external('socketpair', [rffi.INT, rffi.INT, rffi.INT, lltype.Ptr(socketpair_t)], rffi.INT) - ioctl = external('ioctl', [socketfd_type, rffi.INT, lltype.Ptr(ifreq)], - rffi.INT) + if ifreq is not None: + ioctl = external('ioctl', [socketfd_type, rffi.INT, lltype.Ptr(ifreq)], + rffi.INT) if _WIN32: ioctlsocket = external('ioctlsocket', Modified: pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rlib/rmmap.py Tue Oct 12 14:42:59 2010 @@ -50,8 +50,8 @@ constant_names = ['MAP_SHARED', 'MAP_PRIVATE', 'PROT_READ', 'PROT_WRITE', 'MS_SYNC'] - opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS', 'MAP_FIXED', - 'PROT_EXEC', + opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS', 'MAP_NORESERVE', + 'MAP_FIXED', 'PROT_EXEC', 'MAP_DENYWRITE', 'MAP_EXECUTABLE'] for name in constant_names: setattr(CConfig, name, rffi_platform.ConstantInteger(name)) Modified: pypy/branch/32ptr-on-64bit/pypy/rlib/rstring.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rlib/rstring.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rlib/rstring.py Tue Oct 12 14:42:59 2010 @@ -54,6 +54,7 @@ self.l = [] def append(self, s): + assert isinstance(s, self._type) self.l.append(s) def append_slice(self, s, start, end): @@ -63,11 +64,16 @@ def append_multiple_char(self, c, times): self.l.append(c * times) + def getlength(self): + return len(self.build()) + class StringBuilder(AbstractStringBuilder): + _type = str def build(self): return "".join(self.l) class UnicodeBuilder(AbstractStringBuilder): + _type = unicode def build(self): return u''.join(self.l) @@ -121,9 +127,12 @@ assert s_times.nonneg return s_None + def method_getlength(self): + return SomeInteger(nonneg=True) + def method_build(self): return SomeString() - + def rtyper_makerepr(self, rtyper): return rtyper.type_system.rbuilder.stringbuilder_repr @@ -146,6 +155,9 @@ assert s_times.nonneg return s_None + def method_getlength(self): + return SomeInteger(nonneg=True) + def method_build(self): return SomeUnicodeString() Modified: pypy/branch/32ptr-on-64bit/pypy/rlib/test/test_rsocket.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rlib/test/test_rsocket.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rlib/test/test_rsocket.py Tue Oct 12 14:42:59 2010 @@ -437,3 +437,31 @@ foo = self.serv.accept() py.test.raises(SocketError, raise_error) +def _test_cond_include(cond): + # Test that _rsocket_rffi is importable even on platforms where + # AF_PACKET or AF_NETLINK is not defined. + import re + from pypy.rlib import _rsocket_rffi + srcfile = _rsocket_rffi.__file__ + if srcfile.lower().endswith('c') or srcfile.lower().endswith('o'): + srcfile = srcfile[:-1] # .pyc => .py + assert srcfile.lower().endswith('.py') + sourcelines = open(srcfile, 'rb').read().splitlines() + found = False + for i, line in enumerate(sourcelines): + line2 = re.sub(r"(\s*COND_HEADER\s*=)", + r"\1'#undef %s\\n'+" % cond, + line) + if line2 != line: + found = True + sourcelines[i] = line2 + assert found + d = {} + sourcelines.append('') + exec '\n'.join(sourcelines) in d + +def test_no_AF_PACKET(): + _test_cond_include('AF_PACKET') + +def test_no_AF_NETLINK(): + _test_cond_include('AF_NETLINK') Modified: pypy/branch/32ptr-on-64bit/pypy/rlib/test/test_rstring.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rlib/test/test_rstring.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rlib/test/test_rstring.py Tue Oct 12 14:42:59 2010 @@ -29,6 +29,7 @@ s = StringBuilder() s.append("a") s.append("abc") + assert s.getlength() == len('aabc') s.append("a") s.append_slice("abc", 1, 2) s.append_multiple_char('d', 4) @@ -39,6 +40,7 @@ s.append(u'a') s.append(u'abc') s.append_slice(u'abcdef', 1, 2) + assert s.getlength() == len('aabcb') s.append_multiple_char('d', 4) assert s.build() == 'aabcbdddd' assert isinstance(s.build(), unicode) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py Tue Oct 12 14:42:59 2010 @@ -69,7 +69,7 @@ PIECESIZE = 0x08000000 PIECES = 10 m = rmmap.mmap(-1, PIECES * PIECESIZE, - rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS, + rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS|rmmap.MAP_NORESERVE, rmmap.PROT_READ|rmmap.PROT_WRITE) m.close = lambda : None # leak instead of giving a spurious # error at CPython's shutdown Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rbuilder.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rbuilder.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rbuilder.py Tue Oct 12 14:42:59 2010 @@ -100,6 +100,10 @@ ll_builder.used = used @staticmethod + def ll_getlength(ll_builder): + return ll_builder.used + + @staticmethod def ll_build(ll_builder): final_size = ll_builder.used assert final_size >= 0 Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rclass.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rclass.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rclass.py Tue Oct 12 14:42:59 2010 @@ -331,10 +331,9 @@ fields['__class__'] = 'typeptr', r, r else: # instance attributes - if llfields is None: - llfields = [] attrs = self.classdef.attrs.items() attrs.sort() + myllfields = [] for name, attrdef in attrs: if not attrdef.readonly: r = self.rtyper.getrepr(attrdef.s_value) @@ -344,7 +343,25 @@ else: internal_r = r fields[name] = mangled_name, r, internal_r - llfields.append((mangled_name, internal_r.lowleveltype)) + myllfields.append((mangled_name, internal_r.lowleveltype)) + + # Sort the instance attributes by decreasing "likely size", + # as reported by rffi.sizeof(), to minimize padding holes in C. + # Fields of the same size are sorted by name (by attrs.sort() + # above) just to minimize randomness. + def keysize((_, T)): + if T is lltype.Void: + return None + from pypy.rpython.lltypesystem.rffi import sizeof + try: + return -sizeof(T) + except StandardError: + return None + myllfields.sort(key = keysize) + if llfields is None: + llfields = myllfields + else: + llfields = llfields + myllfields self.rbase = getinstancerepr(self.rtyper, self.classdef.basedef, self.gcflavor) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/module/ll_time.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/module/ll_time.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/module/ll_time.py Tue Oct 12 14:42:59 2010 @@ -108,7 +108,7 @@ errcode = -1 if self.GETTIMEOFDAY_NO_TZ: - errcode = g_gettimeofday(t) + errcode = c_gettimeofday(t) else: errcode = c_gettimeofday(t, void) Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/rbuilder.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/rbuilder.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/rbuilder.py Tue Oct 12 14:42:59 2010 @@ -36,8 +36,12 @@ hop.exception_cannot_occur() return hop.gendirectcall(self.ll_append_multiple_char, *vlist) + def rtype_method_getlength(self, hop): + vlist = hop.inputargs(self) + hop.exception_cannot_occur() + return hop.gendirectcall(self.ll_getlength, *vlist) + def rtype_method_build(self, hop): vlist = hop.inputargs(self) hop.exception_cannot_occur() return hop.gendirectcall(self.ll_build, *vlist) - Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/rtyper.py Tue Oct 12 14:42:59 2010 @@ -421,7 +421,7 @@ assert noexclink.exitcase is None if pos == "removed": # the exception cannot actually occur at all. - # See for example rspecialcase.rtype_call_specialcase(). + # This is set by calling exception_cannot_occur(). # We just remove all exception links. block.exitswitch = None block.exits = block.exits[:1] @@ -1019,7 +1019,7 @@ from pypy.rpython import rint, rbool, rfloat from pypy.rpython import rrange from pypy.rpython import rstr, rdict, rlist -from pypy.rpython import rclass, rbuiltin, rpbc, rspecialcase +from pypy.rpython import rclass, rbuiltin, rpbc from pypy.rpython import rexternalobj from pypy.rpython import rptr from pypy.rpython import rgeneric Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rbuilder.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rbuilder.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rbuilder.py Tue Oct 12 14:42:59 2010 @@ -1,4 +1,4 @@ - +import py from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin from pypy.rpython.lltypesystem.rbuilder import * from pypy.rpython.annlowlevel import llstr, hlstr @@ -55,8 +55,29 @@ assert res == 'aabcabcdefbuuuu' assert isinstance(res, unicode) + def test_string_getlength(self): + def func(): + s = StringBuilder() + s.append("a") + s.append("abc") + return s.getlength() + res = self.interpret(func, []) + assert res == 4 + + def test_unicode_getlength(self): + def func(): + s = UnicodeBuilder() + s.append(u"a") + s.append(u"abc") + return s.getlength() + res = self.interpret(func, []) + assert res == 4 + class TestLLtype(BaseTestStringBuilder, LLRtypeMixin): pass class TestOOtype(BaseTestStringBuilder, OORtypeMixin): - pass + def test_string_getlength(self): + py.test.skip("getlength(): not implemented on ootype") + def test_unicode_getlength(self): + py.test.skip("getlength(): not implemented on ootype") Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rclass.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rclass.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rclass.py Tue Oct 12 14:42:59 2010 @@ -3,7 +3,7 @@ from pypy.translator.translator import TranslationContext, graphof from pypy.rpython.lltypesystem.lltype import * from pypy.rpython.ootypesystem import ootype -from pypy.rlib.rarithmetic import intmask +from pypy.rlib.rarithmetic import intmask, r_longlong from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin from pypy.objspace.flow.model import summary @@ -1001,6 +1001,40 @@ res = self.interpret(f, [5]) assert res == 0 + def test_order_of_fields(self): + class A(object): + pass + def f(n): + a = A() + a.as_int = n + a.as_char = chr(n) + a.as_unichar = unichr(n) + a.as_double = n + 0.5 + a.as_bool = bool(n) + a.as_void = None + a.as_longlong = r_longlong(n) + a.as_reference = A() + return a + + res = self.interpret(f, [5]) + names = list(typeOf(res).TO._names) + i = names.index('inst_as_int') + c = names.index('inst_as_char') + u = names.index('inst_as_unichar') + d = names.index('inst_as_double') + b = names.index('inst_as_bool') + v = names.index('inst_as_void') + l = names.index('inst_as_longlong') + r = names.index('inst_as_reference') + assert v == 1 # void fields are first + assert sorted([c, b]) == [7, 8] + if sys.maxint == 2147483647: + assert sorted([u, i, r]) == [4, 5, 6] # 32-bit types + assert sorted([d, l]) == [2, 3] # 64-bit types + else: + assert sorted([u]) == [6] # 32-bit types + assert sorted([i, r, d, l]) == [2, 3, 4, 5] # 64-bit types + class TestOOtype(BaseTestRclass, OORtypeMixin): Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf/track5.s ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf/track5.s (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/test/elf/track5.s Tue Oct 12 14:42:59 2010 @@ -44,7 +44,7 @@ addl %eax, %ebx jmp .L1221 .L1227: - call RPyAbort + ;;call RPyAbort cmpl 12(%esi), %ebx jb .L1229 addl $20, %esp Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/trackgcroot.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/trackgcroot.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/gcc/trackgcroot.py Tue Oct 12 14:42:59 2010 @@ -46,6 +46,7 @@ self.findlabels() self.parse_instructions() try: + self.trim_unreachable_instructions() self.find_noncollecting_calls() if not self.list_collecting_call_insns(): return [] @@ -122,6 +123,26 @@ assert label not in self.labels, "duplicate label: %s" % label self.labels[label] = Label(label, lineno) + def trim_unreachable_instructions(self): + reached = set([self.insns[0]]) + prevlen = 0 + while len(reached) > prevlen: + prevlen = len(reached) + for insn in self.insns: + if insn not in reached: + for previnsn in insn.previous_insns: + if previnsn in reached: + # this instruction is reachable too + reached.add(insn) + break + # now kill all unreachable instructions + i = 0 + while i < len(self.insns): + if self.insns[i] in reached: + i += 1 + else: + del self.insns[i] + def find_noncollecting_calls(self): cannot_collect = {} for line in self.lines: @@ -752,7 +773,7 @@ target, = sources if target in self.FUNCTIONS_NOT_RETURNING: - return [InsnStop(target), InsnCannotFollowEsp()] + return [InsnStop(target)] if self.format == 'mingw32' and target == '__alloca': # in functions with large stack requirements, windows # needs a call to _alloca(), to turn reserved pages Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/signals.h ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/c/src/signals.h (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/c/src/signals.h Tue Oct 12 14:42:59 2010 @@ -6,20 +6,6 @@ #include -#ifndef LONG_MAX -#if SIZEOF_LONG == 4 -#define LONG_MAX 0X7FFFFFFFL -#elif SIZEOF_LONG == 8 -#define LONG_MAX 0X7FFFFFFFFFFFFFFFL -#else -#error "could not set LONG_MAX in pyport.h" -#endif -#endif - -#ifndef LONG_MIN -#define LONG_MIN (-LONG_MAX-1) -#endif - #include #ifdef MS_WINDOWS @@ -28,10 +14,6 @@ #include -#ifndef SIG_ERR -#define SIG_ERR ((PyOS_sighandler_t)(-1)) -#endif - #if defined(PYOS_OS2) && !defined(PYCC_GCC) #define NSIG 12 #include @@ -65,11 +47,11 @@ /* utility to poll for signals that arrived */ int pypysig_poll(void); /* => signum or -1 */ -/* When a signal is received, the high bit of pypysig_occurred is set. - After all signals are processed by pypysig_poll(), the high bit is +/* When a signal is received, the bit 30 of pypysig_occurred is set. + After all signals are processed by pypysig_poll(), the bit 30 is cleared again. The variable is exposed and RPython code is free to use the other bits in any way. */ -#define PENDING_SIGNAL_BIT (LONG_MIN) /* high bit */ +#define PENDING_SIGNAL_BIT (1 << 30) /* This is a struct for the JIT. See interp_signal.py. */ struct pypysig_long_struct { long value; Modified: pypy/branch/32ptr-on-64bit/pypy/translator/goal/ann_override.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/goal/ann_override.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/goal/ann_override.py Tue Oct 12 14:42:59 2010 @@ -27,21 +27,6 @@ pol.pypytypes = {} pol.single_space = single_space - #def override__wrap_exception_cls(pol, space, x): - # import pypy.objspace.std.typeobject as typeobject - # clsdef = getbookkeeper().getuniqueclassdef(typeobject.W_TypeObject) - # return annmodel.SomeInstance(clsdef, can_be_None=True) - # - #def override__fake_object(pol, space, x): - # from pypy.interpreter import typedef - # clsdef = getbookkeeper().getuniqueclassdef(typedef.W_Root) - # return annmodel.SomeInstance(clsdef) - # - #def override__cpy_compile(pol, self, source, filename, mode, flags): - # from pypy.interpreter import pycode - # clsdef = getbookkeeper().getuniqueclassdef(pycode.PyCode) - # return annmodel.SomeInstance(clsdef) - def specialize__wrap(pol, funcdesc, args_s): from pypy.interpreter.baseobjspace import Wrappable from pypy.annotation.classdef import ClassDef Modified: pypy/branch/32ptr-on-64bit/pypy/translator/jvm/test/test_class.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/jvm/test/test_class.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/jvm/test/test_class.py Tue Oct 12 14:42:59 2010 @@ -1,6 +1,6 @@ import py from pypy.translator.jvm.test.runtest import JvmTest -from pypy.translator.oosupport.test_template.class_ import BaseTestClass, BaseTestSpecialcase +from pypy.translator.oosupport.test_template.class_ import BaseTestClass class TestJvmClass(JvmTest, BaseTestClass): def test_overridden_classattr_as_defaults(self): @@ -26,6 +26,3 @@ def test_specialize_methods(self): py.test.skip('ABSTRACT METHOD FIX: RE-TEST AFTER MERGE') - -class TestJvmSpecialCase(JvmTest, BaseTestSpecialcase): - pass Modified: pypy/branch/32ptr-on-64bit/pypy/translator/oosupport/test_template/class_.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/translator/oosupport/test_template/class_.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/translator/oosupport/test_template/class_.py Tue Oct 12 14:42:59 2010 @@ -1,6 +1,5 @@ import py from pypy.rpython.test import test_rclass -from pypy.rpython.test.test_rspecialcase import BaseTestRspecialcase class BaseTestClass(test_rclass.TestOOtype): def test_abstract_method(self): @@ -66,6 +65,3 @@ def test_cast_object_mix_null(self): py.test.skip('cannot return ootype.NULL from translated functions') - -class BaseTestSpecialcase(BaseTestRspecialcase): - pass From arigo at codespeak.net Tue Oct 12 14:46:22 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 12 Oct 2010 14:46:22 +0200 (CEST) Subject: [pypy-svn] r77831 - pypy/branch/32ptr-on-64bit/pypy/config Message-ID: <20101012124622.C6A10282BF6@codespeak.net> Author: arigo Date: Tue Oct 12 14:46:21 2010 New Revision: 77831 Modified: pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py Log: Enable --compressptr in -Omem translations. Modified: pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/config/translationoption.py Tue Oct 12 14:46:21 2010 @@ -345,11 +345,11 @@ #level: gc backend optimizations... '0': 'boehm nobackendopt', '1': 'boehm lowinline', - 'size': 'boehm lowinline remove_asserts', - 'mem': DEFL_GC + ' lowinline remove_asserts removetypeptr', + 'size': 'boehm lowinline remove_asserts', + 'mem': DEFL_GC + ' lowinline remove_asserts removetypeptr compressptr', '2': DEFL_GC + ' extraopts', - '3': DEFL_GC + ' extraopts remove_asserts', - 'jit': DEFL_GC + ' extraopts jit', + '3': DEFL_GC + ' extraopts remove_asserts', + 'jit': DEFL_GC + ' extraopts jit', } def final_check_config(config): @@ -393,6 +393,9 @@ raise NotImplementedError("JIT conflicts with stackless for now") elif word == 'removetypeptr': config.translation.suggest(gcremovetypeptr=True) + elif word == 'compressptr': + if IS_64_BITS: + config.translation.suggest(compressptr=True) else: raise ValueError(word) From arigo at codespeak.net Tue Oct 12 14:49:09 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 12 Oct 2010 14:49:09 +0200 (CEST) Subject: [pypy-svn] r77832 - pypy/branch/32ptr-on-64bit/pypy/module/pyexpat Message-ID: <20101012124909.1B0D6282BF6@codespeak.net> Author: arigo Date: Tue Oct 12 14:49:06 2010 New Revision: 77832 Modified: pypy/branch/32ptr-on-64bit/pypy/module/pyexpat/interp_pyexpat.py Log: Fix. Modified: pypy/branch/32ptr-on-64bit/pypy/module/pyexpat/interp_pyexpat.py ============================================================================== --- pypy/branch/32ptr-on-64bit/pypy/module/pyexpat/interp_pyexpat.py (original) +++ pypy/branch/32ptr-on-64bit/pypy/module/pyexpat/interp_pyexpat.py Tue Oct 12 14:49:06 2010 @@ -32,7 +32,7 @@ include_dir='lib', library_dir='win32/bin/release'), ]) -XML_Content_Ptr = lltype.Ptr(lltype.ForwardReference()) +XML_Content_Ptr = lltype.Ptr(lltype.ForwardReference(will_be_varsize=True)) XML_Parser = rffi.COpaquePtr(typedef='XML_Parser') class CConfigure: From afa at codespeak.net Tue Oct 12 15:01:44 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 12 Oct 2010 15:01:44 +0200 (CEST) Subject: [pypy-svn] r77833 - in pypy/branch/fast-forward/pypy/module/_multiprocessing: . test Message-ID: <20101012130144.2EDA0282BF6@codespeak.net> Author: afa Date: Tue Oct 12 15:01:41 2010 New Revision: 77833 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py Log: Implement Semlock._get_value and Semlock._is_zero + some cleanup in exceptions Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py Tue Oct 12 15:01:41 2010 @@ -212,6 +212,9 @@ self.counter += 1 return value +# These functions may raise bare OSError or WindowsError, +# don't forget to wrap them into OperationError + if sys.platform == 'win32': def create_semaphore(space, name, val, max): rwin32.SetLastError(0) @@ -219,8 +222,7 @@ # On Windows we should fail on ERROR_ALREADY_EXISTS err = rwin32.GetLastError() if err != 0: - raise wrap_windowserror( - space, WindowsError(err, "CreateSemaphore")) + raise WindowsError(err, "CreateSemaphore") return handle def semlock_acquire(self, space, block, w_timeout): @@ -239,10 +241,7 @@ full_msecs = int(timeout + 0.5) # check whether we can acquire without blocking - try: - res = rwin32.WaitForSingleObject(self.handle, 0) - except WindowsError, e: - raise wrap_windowserror(space, e) + res = rwin32.WaitForSingleObject(self.handle, 0) if res != rwin32.WAIT_TIMEOUT: return True @@ -255,10 +254,7 @@ # do the wait _ResetEvent(globalState.sigint_event) - try: - res = rwin32.WaitForMultipleObjects(handles, timeout=msecs) - except WindowsError, e: - raise wrap_windowserror(space, e) + res = rwin32.WaitForMultipleObjects(handles, timeout=msecs) if res != rwin32.WAIT_OBJECT_0 + 1: break @@ -290,8 +286,21 @@ space.w_ValueError, space.wrap("semaphore or lock released too many times")) else: - raise wrap_windowserror( - space, WindowsError(err, "ReleaseSemaphore")) + raise WindowsError(err, "ReleaseSemaphore") + + def semlock_getvalue(self, space): + if rwin32.WaitForSingleObject(self.handle, 0) == rwin32.WAIT_TIMEOUT: + return 0 + previous_ptr = lltype.malloc(rffi.LONGP.TO, 1, flavor='raw') + try: + if not _ReleaseSemaphore(self.handle, 1, previous_ptr): + raise rwin32.lastWindowsError("ReleaseSemaphore") + return previous_ptr[0] + 1 + finally: + lltype.free(previous_ptr, flavor='raw') + + def semlock_iszero(self, space): + return semlock_getvalue(self, space) == 0 else: def create_semaphore(space, name, val, max): @@ -334,7 +343,7 @@ continue elif e.errno in (errno.EAGAIN, errno.ETIMEDOUT): return False - raise wrap_oserror(space, e) + raise # XXX PyErr_CheckSignals() return True @@ -372,6 +381,31 @@ sem_post(self.handle) + def semlock_getvalue(self, space): + if HAVE_BROKEN_SEM_GETVALUE: + raise OperationError(space.w_NotImplementedError) + else: + val = sem_getvalue(self.handle) + # some posix implementations use negative numbers to indicate + # the number of waiting threads + if val < 0: + val = 0 + return val + + def semlock_iszero(self, space): + if HAVE_BROKEN_SEM_GETVALUE: + try: + sem_trywait(self.handle) + except OSError, e: + if e.errno != errno.EAGAIN: + raise + return True + else: + sem_post(self.handle) + return False + else: + return semlock_getvalue(self, space) == 0 + class W_SemLock(Wrappable): def __init__(self, handle, kind, maxvalue): @@ -398,6 +432,22 @@ def is_mine(self, space): return space.wrap(self._ismine()) + @unwrap_spec('self', ObjSpace) + def is_zero(self, space): + try: + res = semlock_iszero(self, space) + except OSError, e: + raise wrap_oserror(space, e) + return space.wrap(res) + + @unwrap_spec('self', ObjSpace) + def get_value(self, space): + try: + val = semlock_getvalue(self, space) + except OSError, e: + raise wrap_oserror(space, e) + return space.wrap(val) + @unwrap_spec('self', ObjSpace, bool, W_Root) def acquire(self, space, block=True, w_timeout=None): # check whether we already own the lock @@ -405,7 +455,12 @@ self.count += 1 return space.w_True - if semlock_acquire(self, space, block, w_timeout): + try: + got = semlock_acquire(self, space, block, w_timeout) + except OSError, e: + raise wrap_oserror(space, e) + + if got: self.last_tid = ll_thread.get_ident() self.count += 1 return space.w_True @@ -424,7 +479,10 @@ self.count -= 1 return - semlock_release(self, space) + try: + semlock_release(self, space) + except OSError, e: + raise wrap_oserror(space, e) self.count -= 1 @@ -443,7 +501,10 @@ counter = space.fromcache(CounterState).getCount() name = "/mp%d-%d" % (os.getpid(), counter) - handle = create_semaphore(space, name, value, maxvalue) + try: + handle = create_semaphore(space, name, value, maxvalue) + except OSError, e: + raise wrap_oserror(space, e) self = space.allocate_instance(W_SemLock, w_subtype) self.__init__(handle, kind, maxvalue) @@ -458,6 +519,8 @@ handle = GetSetProperty(W_SemLock.handle_get), _count = interp2app(W_SemLock.get_count), _is_mine = interp2app(W_SemLock.is_mine), + _is_zero = interp2app(W_SemLock.is_zero), + _get_value = interp2app(W_SemLock.get_value), acquire = interp2app(W_SemLock.acquire), release = interp2app(W_SemLock.release), _rebuild = interp2app(W_SemLock.rebuild.im_func, as_classmethod=True), Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py Tue Oct 12 15:01:41 2010 @@ -21,9 +21,13 @@ assert isinstance(sem.handle, int) assert sem._count() == 0 + assert sem._get_value() == 1 + assert sem._is_zero() == False sem.acquire() assert sem._is_mine() assert sem._count() == 1 + assert sem._get_value() == 0 + assert sem._is_zero() == True sem.release() assert sem._count() == 0 From arigo at codespeak.net Tue Oct 12 15:54:09 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 12 Oct 2010 15:54:09 +0200 (CEST) Subject: [pypy-svn] r77834 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012135409.F24FA282BF6@codespeak.net> Author: arigo Date: Tue Oct 12 15:54:05 2010 New Revision: 77834 Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf pypy/extradoc/talk/pepm2011/math.lyx pypy/extradoc/talk/pepm2011/paper.tex Log: Typos. Swap $u$ and $v$ in \texttt{get}, to match the \texttt{set}. Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf ============================================================================== Binary files. No diff available. Modified: pypy/extradoc/talk/pepm2011/math.lyx ============================================================================== --- pypy/extradoc/talk/pepm2011/math.lyx (original) +++ pypy/extradoc/talk/pepm2011/math.lyx Tue Oct 12 15:54:05 2010 @@ -60,7 +60,7 @@ \begin_layout Standard \begin_inset Formula \begin{eqnarray*} -u,v,w & \in & V\mathrm{\,\, are\, Variables\, in\, the\, original\, trace}\\ +u,v & \in & V\mathrm{\,\, are\, Variables\, in\, the\, original\, trace}\\ u^{*},v^{*},w^{*} & \in & V^{*}\,\mathrm{\, are\, Variables\, in\, the\, optimized\, trace}\\ T & \in & \mathfrak{T}\mathrm{\,\, are\, runtime\, types}\\ F & \in & \left\{ L,R\right\} \,\mathrm{\, are\, fields\, of\, objects}\\ @@ -78,7 +78,7 @@ \begin_layout Standard \begin_inset Formula \begin{eqnarray*} & & v=\mathrm{new}(T)\,\,\mathrm{make\, a\, new\, object}\\ - & & v=\mathrm{get}(u,F)\,\,\mathrm{read\, a\, field}\\ + & & u=\mathrm{get}(v,F)\,\,\mathrm{read\, a\, field}\\ & & \mathrm{set}\left(v,F,u\right)\,\,\mathrm{write\, a\, field}\\ & & \mathrm{guard}(v,T)\,\,\mathrm{check\, the\, type\, of\, an\, object}\end{eqnarray*} @@ -206,7 +206,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{\,}{v=\mathtt{get}(u,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto H\left(E\left(u\right)\right)_{F}\right],H}}$ +\begin_inset Formula ${\displaystyle \frac{\,}{u=\mathtt{get}(v,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[u\mapsto H\left(E\left(v\right)\right)_{F}\right],H}}$ \end_inset @@ -385,7 +385,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(u)\in\mathrm{dom}(S)}{v=\mathtt{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto S(E(u))_{F}\right],S}}$ +\begin_inset Formula ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{u=\mathtt{get}(v,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[u\mapsto S(E(v))_{F}\right],S}}$ \end_inset @@ -408,7 +408,7 @@ \begin_inset Text \begin_layout Plain Layout -\begin_inset Formula ${\displaystyle \frac{E(u)\notin\mathrm{dom}(S)\, v^{*}\,\mathrm{fresh}}{v=\mathtt{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle v^{*}=\mathtt{get}(E(u),F)\right\rangle ,E\left[v\mapsto v^{*}\right],S}}$ +\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\, u^{*}\,\mathrm{fresh}}{u=\mathtt{get}(v,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle u^{*}=\mathtt{get}(E(v),F)\right\rangle ,E\left[u\mapsto u^{*}\right],S}}$ \end_inset Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 15:54:05 2010 @@ -542,11 +542,11 @@ } The static object associated with $p_{5}$ would know that it is a -\texttt{BoxedInteger}, and that the \texttt{intval} field contains $i_{4}$, the -one associated with $p_{6}$ would know that its \texttt{intval} field contains -the constant -100. +\texttt{BoxedInteger} whose \texttt{intval} field contains $i_{4}$; the +one associated with $p_{6}$ would know that it is a \texttt{BoxedInteger} +whose \texttt{intval} field contains the constant -100. -The following operations, that use $p_{5}$ and $p_{6}$ could then be +The following operations on $p_{5}$ and $p_{6}$ could then be optimized using that knowledge: \texttt{ @@ -580,13 +580,13 @@ a static object is stored in a globally accessible place, the object needs to actually be allocated, as it might live longer than one iteration of the loop and because the partial evaluator looses track of it. This means that the static -objects needs to be turned into a dynamic one, \ie lifted. This makes it +object needs to be turned into a dynamic one, \ie lifted. This makes it necessary to put operations into the residual code that actually allocate the static object at runtime. This is what happens at the end of the trace in Figure~\ref{fig:unopt-trace}, when the \texttt{jump} operation is hit. The arguments of the jump are at this point static objects. Before the -jump is emitted, they are \emph{lifted}. This means that the optimizers produces code +jump is emitted, they are \emph{lifted}. This means that the optimizer produces code that allocates a new object of the right type and sets its fields to the field values that the static object has (if the static object points to other static objects, those need to be lifted as well) This means that instead of the jump, @@ -617,6 +617,7 @@ The final optimized trace of the example can be seen in Figure~\ref{fig:step1}. +XXX why does it says ``Figure 4.1'' here but ``Figure 4'' in the label? The optimized trace contains only two allocations, instead of the original five, and only three \texttt{guard\_class} operations, from the original seven. @@ -640,7 +641,7 @@ \emph{Object Domains:} $$\setlength\arraycolsep{0.1em} \begin{array}{rcll} - u,v,w & \in & V & \mathrm{\ variables\ in\ trace}\\ + u,v & \in & V & \mathrm{\ variables\ in\ trace}\\ T & \in & \mathfrak{T} & \mathrm{\ runtime\ types}\\ F & \in & \left\{ L,R\right\} & \mathrm{\ fields\ of\ objects}\\ l & \in & L & \mathrm{\ locations\ on\ heap} @@ -675,8 +676,8 @@ variables are locations (i.e.~pointers). Locations are mapped to objects, which are represented by triples of a type $T$, and two locations that represent the fields of the object. When a new object is created, the fields are initialized -to null, but we require that they are immediately initialized to a real -location, otherwise the trace is malformed. +to null, but we require that they are initialized to a real +location before being read, otherwise the trace is malformed. We use some abbreviations when dealing with object triples. To read the type of an object, $\mathrm{type}((T,l_1,l_2))=T$ is used. Reading a field $F$ from an @@ -687,7 +688,7 @@ Figure~\ref{fig:semantics} shows the operational semantics for traces. The interpreter formalized there executes one operation at a time. Its state is -represented by an environment and a heap, which are potentially changed by the +represented by an environment $E$ and a heap $H$, which are potentially changed by the execution of an operation. The environment is a partial function from variables to locations and the heap is a partial function from locations to objects. Note that a variable can never be null in the environment, otherwise the trace would @@ -699,12 +700,12 @@ $E[v\mapsto l]$ denotes the environment which is just like $E$, but maps $v$ to $l$. -The new operation creates a new object $(T,\mathrm{null},\mathrm{null})$, on the +The \texttt{new} operation creates a new object $(T,\mathrm{null},\mathrm{null})$ on the heap under a fresh location $l$ and adds the result variable to the environment, -mapping to the new location $l$. +mapping it to the new location $l$. -The \texttt{get} operation reads a field $F$ out of an object and adds the result -variable to the environment, mapping to the read location. The heap is +The \texttt{get} operation reads a field $F$ out of an object, and adds the result +variable to the environment, mapping it to the read location. The heap is unchanged. The \texttt{set} operation changes field $F$ of an object stored at the location that @@ -771,53 +772,54 @@ The state of the optimizer is stored in an environment $E$ and a \emph{static heap} $S$. The environment is a partial function from variables in the -unoptimized trace to variables in the optimized trace (which are written with a +unoptimized trace $V$ to variables in the optimized trace $V^*$ (which are +themselves written with a $\ ^*$ for clarity). The reason for introducing new variables in the optimized trace is that several variables that appear in the unoptimized trace can turn into the same variables in the optimized trace. Thus the environment of the -optimizer serves a function similar to that of the environment in the semantics. +optimizer serves a function similar to that of the environment in the semantics: sharing. The static heap is a partial function from $V^*$ into the -set of static objects, which are triples of a type, and two elements of $V^*$. -An variable $v^*$ is in the domain of the static heap $S$ as long as the +set of static objects, which are triples of a type and two elements of $V^*$. +A variable $v^*$ is in the domain of the static heap $S$ as long as the optimizer can fully keep track of the object. The image of $v^*$ is what is statically known about the object stored in it, \ie its type and its fields. The fields of objects in the static heap are also elements of $V^*$ (or null, for short periods of time). -When the optimizer sees a new operation, it optimistically removes it and +When the optimizer sees a \texttt{new} operation, it optimistically removes it and assumes that the resulting object can stay static. The optimization for all further operations is split into two cases. One case is for when the involved variables are in the static heap, which means that the operation can be performed at optimization time and removed from the trace. These rules mirror -the execution semantics closely. The other case is that nothing is known about -the variables, which means the operation has to be residualized. +the execution semantics closely. The other case is for when not enough is known about +the variables, and the operation has to be residualized. -If the argument $u$ of a \texttt{get} operation is mapped to something in the static +If the argument $v$ of a \texttt{get} operation is mapped to something in the static heap, the get can be performed at optimization time. Otherwise, the \texttt{get} -operation needs to be emitted. +operation needs to be residualized. If the first argument $v$ to a \texttt{set} operation is mapped to something in the static heap, then the \texttt{set} can performed at optimization time and the static heap -is updated. Otherwise the \texttt{set} operation needs to be emitted. This needs to be -done carefully, because the new value for the field stored in the variable $u$ +updated. Otherwise the \texttt{set} operation needs to be residualized. This needs to be +done carefully, because the new value for the field, from the variable $u$, could itself be static, in which case it needs to be lifted first. -I a \texttt{guard\_class} is performed on a variable that is in the static heap, the type check +If a \texttt{guard\_class} is performed on a variable that is in the static heap, the type check can be performed at optimization time, which means the operation can be removed if the types match. If the type check fails statically or if the object is not -in the static heap, the \texttt{guard\_class} is put into the residual trace. This also needs to +in the static heap, the \texttt{guard\_class} is residualized. This also needs to lift the variable on which the \texttt{guard\_class} is performed. Lifting takes a variable that is potentially in the static heap and makes sure that it is turned into a dynamic variable. This means that operations are -emitted that construct an object that looks like the shape described in the +emitted that construct an object with the shape described in the static heap, and the variable is removed from the static heap. Lifting a static object needs to recursively lift its fields. Some care needs to be taken when lifting a static object, because the structures described by the static heap can be cyclic. To make sure that the same static object is not lifted -twice, the liftfield operation removes it from the static heap \emph{before} +twice, the \texttt{liftfield} operation removes it from the static heap \emph{before} recursively lifting its fields. @@ -1120,7 +1122,7 @@ hint in first js paper by michael franz \cite{mason_chang_efficient_2007} -SPUR, a tracing JIT for C# seems to be able to remove allocations in a similar +SPUR, a tracing JIT for C\# seems to be able to remove allocations in a similar way to the approach described here, as hinted at in the technical report \cite{XXX}. However, no details for the approach and its implementation are given. From cfbolz at codespeak.net Tue Oct 12 15:57:08 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 15:57:08 +0200 (CEST) Subject: [pypy-svn] r77835 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012135708.88AD9282BF6@codespeak.net> Author: cfbolz Date: Tue Oct 12 15:57:07 2010 New Revision: 77835 Modified: pypy/extradoc/talk/pepm2011/paper.bib Log: more papers Modified: pypy/extradoc/talk/pepm2011/paper.bib ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.bib (original) +++ pypy/extradoc/talk/pepm2011/paper.bib Tue Oct 12 15:57:07 2010 @@ -68,6 +68,49 @@ pages = {328--347} }, + at techreport{michael_bebenita_spur:_2010, + title = {{SPUR:} A {Trace-Based} {JIT} Compiler for {CIL}}, + abstract = {Tracing just-in-time compilers {(TJITs)} determine frequently executed traces (hot paths and loops) in running programs and focus their optimization effort by emitting optimized machine code specialized to these traces. Prior work has established this strategy to be especially beneficial for dynamic languages such as {JavaScript,} where the {TJIT} interfaces with the interpreter and produces machine code from the {JavaScript} trace. + +This direct coupling with a {JavaScript} interpreter makes it difficult to harness the power of a {TJIT} for other components that are not written in {JavaScript,} e.g., the {DOM} implementation or the layout engine inside a browser. Furthermore, if a {TJIT} is tied to a particular high-level language interpreter, it is difficult to reuse it for other input languages as the optimizations are likely targeted at specific idioms of the source language. + +To address these issues, we designed and implemented a {TJIT} for Microsoft?s Common Intermediate Language {CIL} (the target language of C\#, {VisualBasic,} F\#, and many other languages). Working on {CIL} enables {TJIT} optimizations for any program compiled to this platform. In addition, to validate that the performance gains of a {TJIT} for {JavaScript} do not depend on specific idioms of {JavaScript} that are lost in the translation to {CIL,} we provide a performance evaluation of our {JavaScript} runtime which translates {JavaScript} to {CIL} and then runs on top of our {CIL} {TJIT.}}, + number = {{MSR-TR-2010-27}}, + institution = {Microsoft Research}, + author = {Michael Bebenita and Florian Brandner and Manuel Fahndrich and Francesco Logozzo and Wolfram Schulte and Nikolai Tillmann and Herman Venter}, + month = mar, + year = {2010}, +}, + + at article{bruno_blanchet_escape_2003, + title = {Escape analysis for Java: Theory and practice}, + volume = {25}, + shorttitle = {Escape analysis for {Java{\textless}sup{\textgreater}TM{\textless}/sup{\textgreater}}}, + url = {http://portal.acm.org/citation.cfm?id=945885.945886}, + doi = {10.1145/945885.945886}, + abstract = {Escape analysis is a static analysis that determines whether the lifetime of data may exceed its static {scope.This} paper first presents the design and correctness proof of an escape analysis for {JavaTM.} This analysis is interprocedural, context sensitive, and as flow-sensitive as the static single assignment form. So, assignments to object fields are analyzed in a flow-insensitive manner. Since Java is an imperative language, the effect of assignments must be precisely determined. This goal is achieved thanks to our technique using two interdependent analyses, one forward, one backward. We introduce a new method to prove the correctness of this analysis, using aliases as an intermediate step. We use integers to represent the escaping parts of values, which leads to a fast and precise {analysis.Our} implementation {[Blanchet} 1999], which applies to the whole Java language, is then presented. Escape analysis is applied to stack allocation and synchronization elimination. In our benchmarks, we stack allocate 13\% to 95\% of data, eliminate more than 20\% of synchronizations on most programs (94\% and 99\% on two examples) and get up to 43\% runtime decrease (21\% on average). Our detailed experimental study on large programs shows that the improvement comes more from the decrease of the garbage collection and allocation times than from improvements on data locality, contrary to what happened for {ML.} This comes from the difference in the garbage collectors.}, + number = {6}, + journal = {{ACM} Trans. Program. Lang. Syst.}, + author = {Bruno Blanchet}, + year = {2003}, + pages = {713--775} +}, + + at inproceedings{chang_tracing_2009, + address = {Washington, {DC,} {USA}}, + title = {Tracing for Web 3.0: Trace Compilation for the Next Generation Web Applications}, + isbn = {978-1-60558-375-4}, + shorttitle = {Tracing for web 3.0}, + url = {http://portal.acm.org/citation.cfm?id=1508293.1508304}, + doi = {10.1145/1508293.1508304}, + abstract = {Today's web applications are pushing the limits of modern web browsers. The emergence of the browser as the platform of choice for rich client-side applications has shifted the use of in-browser {JavaScript} from small scripting programs to large computationally intensive application logic. For many web applications, {JavaScript} performance has become one of the bottlenecks preventing the development of even more interactive client side applications. While traditional just-in-time compilation is successful for statically typed virtual machine based languages like Java, compiling {JavaScript} turns out to be a challenging task. Many {JavaScript} programs and scripts are short-lived, and users expect a responsive browser during page loading. This leaves little time for compilation of {JavaScript} to generate machine code.}, + booktitle = {Proceedings of the 2009 {ACM} {SIGPLAN/SIGOPS} International Conference on Virtual Execution Environments}, + publisher = {{ACM}}, + author = {Mason Chang and Edwin Smith and Rick Reitmaier and Michael Bebenita and Andreas Gal and Christian Wimmer and Brendan Eich and Michael Franz}, + year = {2009}, + pages = {71--80} +}, + @inproceedings{davide_ancona_rpython:_2007, address = {Montreal, Quebec, Canada}, title = {{RPython:} a step towards reconciling dynamically and statically typed {OO} languages}, @@ -128,6 +171,19 @@ pages = {20---34} }, + at article{georges_statistically_2007, + title = {Statistically rigorous java performance evaluation}, + volume = {42}, + url = {http://portal.acm.org/citation.cfm?id=1297105.1297033}, + doi = {10.1145/1297105.1297033}, + abstract = {Java performance is far from being trivial to benchmark because it is affected by various factors such as the Java application, its input, the virtual machine, the garbage collector, the heap size, etc. In addition, non-determinism at run-time causes the execution time of a Java program to differ from run to run. There are a number of sources of non-determinism such as {Just-In-Time} {(JIT)} compilation and optimization in the virtual machine {(VM)} driven by timer-based method sampling, thread scheduling, garbage collection, and various.}, + number = {10}, + journal = {{SIGPLAN} Not.}, + author = {Andy Georges and Dries Buytaert and Lieven Eeckhout}, + year = {2007}, + pages = {57--76}, +}, + @techreport{miranda_context_1999, title = {Context Management in {VisualWorks} 5i}, abstract = {Smalltalk-80 provides a reification of execution state in the form of context objects which represent procedure activation records. Smalltalk-80 also provides full closures with indefinite extent. These features pose interesting implementation challenges because a na?ve implementation entails instantiating context objects on every method activation, but typical Smalltalk-80 programs obey stack discipline for the vast majority of activations. Both software and hardware implementations of Smalltalk-80 have mapped contexts and closure activations to stack frames but not without overhead when compared to traditional stack-based activation and return in ?conventional? languages. We present a new design for contexts and closures that significantly reduces the overall overhead of these features and imposes overhead only in code that actually manipulates execution state in the form of contexts.}, From antocuni at codespeak.net Tue Oct 12 16:08:30 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 12 Oct 2010 16:08:30 +0200 (CEST) Subject: [pypy-svn] r77836 - in pypy/branch/jitffi/pypy: jit/backend/x86/test jit/metainterp/optimizeopt rlib Message-ID: <20101012140830.614AD282BF6@codespeak.net> Author: antocuni Date: Tue Oct 12 16:08:28 2010 New Revision: 77836 Modified: pypy/branch/jitffi/pypy/jit/backend/x86/test/test_ztranslation.py pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: fix translation of fficall optimization Modified: pypy/branch/jitffi/pypy/jit/backend/x86/test/test_ztranslation.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/x86/test/test_ztranslation.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/x86/test/test_ztranslation.py Tue Oct 12 16:08:28 2010 @@ -1,4 +1,4 @@ -import py, os +import py, os, sys from pypy.tool.udir import udir from pypy.rlib.jit import JitDriver, OPTIMIZER_FULL, unroll_parameters from pypy.rlib.jit import PARAMETERS, dont_look_inside @@ -63,8 +63,32 @@ if k - abs(j): raise ValueError if k - abs(-j): raise ValueError return total * 10 + # + from pypy.rpython.lltypesystem import lltype, rffi + from pypy.rlib.libffi import types, CDLL, ArgChain + from pypy.rlib.test.test_libffi import get_libm_name + libm_name = get_libm_name(sys.platform) + jitdriver2 = JitDriver(greens=[], reds = ['i', 'func', 'res', 'x']) + def libffi_stuff(i, j): + lib = CDLL(libm_name) + func = lib.getpointer('fabs', [types.double], types.double) + res = 0.0 + x = float(j) + while i > 0: + jitdriver2.jit_merge_point(i=i, res=res, func=func, x=x) + jitdriver2.can_enter_jit(i=i, res=res, func=func, x=x) + func = hint(func, promote=True) + argchain = ArgChain() + argchain.arg(x) + res = func.call(argchain, rffi.DOUBLE) + i -= 1 + return res + # + def main(i, j): + return f(i, j) + libffi_stuff(i, j) + expected = f(40, -49) res = self.meta_interp(f, [40, -49]) - assert res == f(40, -49) + assert res == expected def test_direct_assembler_call_translates(self): class Thing(object): Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py (original) +++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py Tue Oct 12 16:08:28 2010 @@ -8,6 +8,10 @@ class FuncInfo(object): + argtypes = None + restype = None + descr = None + def __init__(self, funcval, cpu): self.opargs = [] argtypes, restype = self._get_signature(funcval) @@ -36,7 +40,8 @@ llfunc = funcval.box.getref_base() if we_are_translated(): - XXX + func = cast_base_ptr_to_instance(Func, llfunc) + return func.argtypes, func.restype elif getattr(llfunc, '_fake_class', None) is Func: # untranslated return llfunc.argtypes, llfunc.restype Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Tue Oct 12 16:08:28 2010 @@ -156,6 +156,9 @@ class Func(AbstractFuncPtr): _immutable_fields_ = ['funcsym', 'argtypes', 'restype'] + argtypes = [] + restype = None + funcsym = None def __init__(self, name, argtypes, restype, funcsym, flags=FUNCFLAG_CDECL, keepalive=None): From cfbolz at codespeak.net Tue Oct 12 16:14:54 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 16:14:54 +0200 (CEST) Subject: [pypy-svn] r77837 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012141454.A6201282BF6@codespeak.net> Author: cfbolz Date: Tue Oct 12 16:14:52 2010 New Revision: 77837 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: related work Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 16:14:52 2010 @@ -35,6 +35,7 @@ \newcommand\ie{i.e.,\xspace} \newcommand\eg{e.g.,\xspace} +\newcommand{\etal}{\emph{et al.}\xspace} \normalem @@ -1112,7 +1113,32 @@ \section{Related Work} \label{sec:related} -Escape analysis: XXX +There exists a large number of works on escape analysis, which is an program +analysis that tries to find an upper bound for the lifetime of objects allocated +at specific program points +\cite{goldberg_higher_1990,park_escape_1992,choi_escape_1999,bruno_blanchet_escape_2003}. +This information can then be used to decide that certain objects can be +allocated on the stack, because their lifetime does not exceed that of the stack +frame it is allocated in. The difference to our work is that escape analysis is +split into an analysis and an optimization phase. The analysis can be a lot more +complex than our simple one-pass optimization. Also, stack-allocation reduces +garbage-collection pressure but does not optimize away the actual accesses to +the stack-allocated object. In our case, an object is not needed at all any +more. +XXX more papers? + +Chang \etal describe a tracing JIT for JavaScript running on top of a JVM +\cite{mason_chang_efficient_2007}. They mention in passing an approach to +allocation removal that moves the allocation of an object of type 1 out of the +loop to only allocate it once, instead of every iteration. No details are given +for this optimization. The fact that the object is still allocated and needs to +be written to means that only the allocations are optimized away, but not the +reads and writes out of/into the object. + +SPUR, a tracing JIT for C\# seems to be able to remove allocations in a similar +way to the approach described here, as hinted at in the technical report +\cite{XXX}. However, no details for the approach and its implementation are +given. partial evaluation: @@ -1120,13 +1146,6 @@ partially static data structures: kenichi asai's thesis? -hint in first js paper by michael franz \cite{mason_chang_efficient_2007} - -SPUR, a tracing JIT for C\# seems to be able to remove allocations in a similar -way to the approach described here, as hinted at in the technical report -\cite{XXX}. However, no details for the approach and its implementation are -given. - \section{Conclusions} \label{sec:conclusions} From leuschel at codespeak.net Tue Oct 12 16:48:24 2010 From: leuschel at codespeak.net (leuschel at codespeak.net) Date: Tue, 12 Oct 2010 16:48:24 +0200 (CEST) Subject: [pypy-svn] r77838 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012144824.DA91E282BF6@codespeak.net> Author: leuschel Date: Tue Oct 12 16:48:22 2010 New Revision: 77838 Modified: pypy/extradoc/talk/pepm2011/paper.bib pypy/extradoc/talk/pepm2011/paper.tex Log: added refs Modified: pypy/extradoc/talk/pepm2011/paper.bib ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.bib (original) +++ pypy/extradoc/talk/pepm2011/paper.bib Tue Oct 12 16:48:22 2010 @@ -290,3 +290,40 @@ year = {2008}, pages = {123--139} } + + at inproceedings{DBLP:conf/iclp/MazurRJB01, + author = {Nancy Mazur and + Peter Ross and + Gerda Janssens and + Maurice Bruynooghe}, + title = {Practical Aspects for a Working Compile Time Garbage Collection + System for Mercury}, + booktitle = {ICLP}, + year = {2001}, + pages = {105-119}, + ee = {http://link.springer.de/link/service/series/0558/bibs/2237/22370105.htm}, + crossref = {DBLP:conf/iclp/2001}, + bibsource = {DBLP, http://dblp.uni-trier.de} +} + at proceedings{DBLP:conf/iclp/2001, + editor = {Philippe Codognet}, + title = {Logic Programming, 17th International Conference, ICLP 2001, + Paphos, Cyprus, November 26 - December 1, 2001, Proceedings}, + booktitle = {ICLP}, + publisher = {Springer}, + series = {Lecture Notes in Computer Science}, + volume = {2237}, + year = {2001}, + isbn = {3-540-42935-2}, + bibsource = {DBLP, http://dblp.uni-trier.de} +} + + at article{Lloyd:jlp91, + author = {J. W. Lloyd and J. C. Shepherdson}, + title = {Partial Evaluation in Logic Programming}, + journal = strjlp, + year = 1991, + volume = 11, + number = {3\& 4}, + pages = {217--242}, +} \ No newline at end of file Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 16:48:22 2010 @@ -1142,10 +1142,16 @@ partial evaluation: -Prolog +Prolog: + partially static datastructures are already built-in to Prolog and was built-into partial + evaluation from the beginning \cite{Lloyd:jlp91}. + FP partially static data structures: kenichi asai's thesis? +xxx: relation to compile-time garbage collection \cite{DBLP:conf/iclp/MazurRJB01}; separation logic; John Hughes: type specialization + + \section{Conclusions} \label{sec:conclusions} From cfbolz at codespeak.net Tue Oct 12 17:05:47 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 17:05:47 +0200 (CEST) Subject: [pypy-svn] r77839 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012150547.D4F9536C230@codespeak.net> Author: cfbolz Date: Tue Oct 12 17:05:45 2010 New Revision: 77839 Modified: pypy/extradoc/talk/pepm2011/paper.bib pypy/extradoc/talk/pepm2011/paper.tex Log: unify stuff Modified: pypy/extradoc/talk/pepm2011/paper.bib ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.bib (original) +++ pypy/extradoc/talk/pepm2011/paper.bib Tue Oct 12 17:05:45 2010 @@ -96,6 +96,32 @@ pages = {713--775} }, + at incollection{mazur_practical_2001, + series = {Lecture Notes in Computer Science}, + title = {Practical Aspects for a Working Compile Time Garbage Collection System for Mercury}, + volume = {2237}, + url = {http://dx.doi.org/10.1007/3-540-45635-X_15}, + abstract = {Compile-time garbage collection {(CTGC)} is still a very uncommon feature within compilers. In previous work we have developed a compile-time structure reuse system for Mercury, a logic programming language. This system indicates which datastructures can safely be reused at run-time. As preliminary experiments were promising, we have continued this work and have now a working and well performing near-to-ship {CTGC-system} built into the Melbourne Mercury Compiler {(MMC).} In this paper we present the multiple design decisions leading to this system, we report the results of using {CTGC} for a set of benchmarks, including a real-world program, and finally we discuss further possible improvements. Benchmarks show substantial memory savings and a noticeable reduction in execution time.}, + booktitle = {Logic Programming}, + publisher = {Springer Berlin / Heidelberg}, + author = {Nancy Mazur and Peter Ross and Gerda Janssens and Maurice Bruynooghe}, + editor = {Philippe Codognet}, + year = {2001}, + note = {{10.1007/3-540-45635-X\_15}}, + pages = {105--119} +}, + + at article{lloyd_partial_1991, + title = {Partial evaluation in logic programming}, + volume = {11}, + url = {http://portal.acm.org/citation.cfm?id=118495}, + number = {3-4}, + journal = {J. Log. Program.}, + author = {J. W. Lloyd and J. C. Shepherdson}, + year = {1991}, + pages = {217--242} +}, + @inproceedings{chang_tracing_2009, address = {Washington, {DC,} {USA}}, title = {Tracing for Web 3.0: Trace Compilation for the Next Generation Web Applications}, @@ -290,40 +316,3 @@ year = {2008}, pages = {123--139} } - - at inproceedings{DBLP:conf/iclp/MazurRJB01, - author = {Nancy Mazur and - Peter Ross and - Gerda Janssens and - Maurice Bruynooghe}, - title = {Practical Aspects for a Working Compile Time Garbage Collection - System for Mercury}, - booktitle = {ICLP}, - year = {2001}, - pages = {105-119}, - ee = {http://link.springer.de/link/service/series/0558/bibs/2237/22370105.htm}, - crossref = {DBLP:conf/iclp/2001}, - bibsource = {DBLP, http://dblp.uni-trier.de} -} - at proceedings{DBLP:conf/iclp/2001, - editor = {Philippe Codognet}, - title = {Logic Programming, 17th International Conference, ICLP 2001, - Paphos, Cyprus, November 26 - December 1, 2001, Proceedings}, - booktitle = {ICLP}, - publisher = {Springer}, - series = {Lecture Notes in Computer Science}, - volume = {2237}, - year = {2001}, - isbn = {3-540-42935-2}, - bibsource = {DBLP, http://dblp.uni-trier.de} -} - - at article{Lloyd:jlp91, - author = {J. W. Lloyd and J. C. Shepherdson}, - title = {Partial Evaluation in Logic Programming}, - journal = strjlp, - year = 1991, - volume = 11, - number = {3\& 4}, - pages = {217--242}, -} \ No newline at end of file Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 17:05:45 2010 @@ -1143,13 +1143,13 @@ partial evaluation: Prolog: - partially static datastructures are already built-in to Prolog and was built-into partial - evaluation from the beginning \cite{Lloyd:jlp91}. +partially static datastructures are already built-in to Prolog and was built-into partial +evaluation from the beginning \cite{lloyd_partial_1991}. FP partially static data structures: kenichi asai's thesis? -xxx: relation to compile-time garbage collection \cite{DBLP:conf/iclp/MazurRJB01}; separation logic; John Hughes: type specialization +xxx: relation to compile-time garbage collection \cite{mazur_practical_2001}; separation logic; John Hughes: type specialization \section{Conclusions} From antocuni at codespeak.net Tue Oct 12 17:08:22 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Tue, 12 Oct 2010 17:08:22 +0200 (CEST) Subject: [pypy-svn] r77840 - in pypy/branch/jitffi/pypy/rpython: . test Message-ID: <20101012150822.3A9ED36C230@codespeak.net> Author: antocuni Date: Tue Oct 12 17:08:20 2010 New Revision: 77840 Modified: pypy/branch/jitffi/pypy/rpython/rpbc.py pypy/branch/jitffi/pypy/rpython/test/test_rclass.py Log: (arigo, antocuni): fix this corner case Modified: pypy/branch/jitffi/pypy/rpython/rpbc.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/rpbc.py (original) +++ pypy/branch/jitffi/pypy/rpython/rpbc.py Tue Oct 12 17:08:20 2010 @@ -256,6 +256,8 @@ def convert_const(self, value): if isinstance(value, types.MethodType) and value.im_self is None: value = value.im_func # unbound method -> bare function + elif isinstance(value, staticmethod): + value = value.__get__(42) # hackish, get the function wrapped by staticmethod if self.lowleveltype is Void: return None if value is None: Modified: pypy/branch/jitffi/pypy/rpython/test/test_rclass.py ============================================================================== --- pypy/branch/jitffi/pypy/rpython/test/test_rclass.py (original) +++ pypy/branch/jitffi/pypy/rpython/test/test_rclass.py Tue Oct 12 17:08:20 2010 @@ -319,6 +319,17 @@ res = self.interpret(f, []) assert res == 42 + def test_staticmethod2(self): + class A(object): + f = staticmethod(lambda x, y: x*y) + class B(A): + f = staticmethod(lambda x, y: x+y) + def f(): + b = B() + return b.f(6, 7) + res = self.interpret(f, []) + assert res == 13 + def test_is(self): class A: pass class B(A): pass From arigo at codespeak.net Tue Oct 12 17:09:37 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 12 Oct 2010 17:09:37 +0200 (CEST) Subject: [pypy-svn] r77841 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012150937.7E79036C230@codespeak.net> Author: arigo Date: Tue Oct 12 17:09:36 2010 New Revision: 77841 Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf pypy/extradoc/talk/pepm2011/paper.tex Log: Write a small bit of text for "Analysis of the Algorithm". Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf ============================================================================== Binary files. No diff available. Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 17:09:36 2010 @@ -829,13 +829,32 @@ \subsection{Analysis of the Algorithm} \label{sub:analysis} -XXX algorithm is linear in the length of the trace - -XXX Category 2 The optimization of -Section~\ref{sec:statics} deals with them too: the \texttt{new} that creates them and -the field accesses are deferred, until the point where the object escapes. - -XXX optimization particularly effective for chains of operations +While we do not offer a formal proof of it, it should be relatively clear +that the algorithm presented above is sound: it works by delaying (and +often completely removing) some operations. The algorithm runs in a +single pass over the list of operations. We can check that altough +recursively lifting a static object is not a constant-time operation, +the algorithm only takes a total time linear in the length of the trace. +Moreover, it gives the ``best'' possible result within its constrains, +e.g.\ in term of the number of residual operations. The +algorithm itself is not particularly complex or innovative; our focus is +rather that \emph{in the context of tracing JITs} it is possible to find a +simple enough algorithm that still gives the best results. + +Note in particular that objects in category 1 (i.e.\ the ones that do +not escape) are completely removed; moreover, objects in category 2 +(i.e.\ escaping) are still partially dealt with: if such an object +escapes later than its creation point, all the operations inbetween that +involve the object are removed. + +The optimization is particularly effective for chains of operations. +For example, it is typical for an interpreter to generate sequences of +writes-followed-by-reads, where one interpreted opcode writes to some +object's field and the next interpreted opcode reads it back, possibly +dispatching on the type of the object created just before. In the case +of PyPy's Python interpreter, this optimization can even remove the +allocation of all intermediate frames that occur in the interpreter, +corresponding to all calls that have been inlined in the trace. % subsection Analysis of the Algorithm (end) From cfbolz at codespeak.net Tue Oct 12 17:14:32 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 17:14:32 +0200 (CEST) Subject: [pypy-svn] r77842 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012151432.DB764282BF6@codespeak.net> Author: cfbolz Date: Tue Oct 12 17:14:31 2010 New Revision: 77842 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: add missing citations Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 17:14:31 2010 @@ -141,7 +141,7 @@ The most important technique we use to achieve to optimize traces is a straightforward application of partial evaluation. The partial evaluation -performs a form of escape analysis \cite{XXX} on the traces and make some +performs a form of escape analysis \cite{bruno_blanchet_escape_2003} on the traces and make some objects that are allocated in the trace \emph{static}\footnote{These objects are called \emph{virtual} in Psyco \cite{rigo_representation-based_2004}.} which means that they do not occur any more in the optimized trace. This technique is @@ -213,8 +213,8 @@ \cite{bala_dynamo:_2000}. Tracing JITs have then be adapted to be used for a very light-weight Java VM \cite{gal_hotpathvm:_2006} and afterwards used in several implementations of dynamic languages, such as JavaScript -\cite{andreas_gal_trace-based_2009}, Lua \cite{XXX} and now Python (and other -languages) via PyPy. +\cite{andreas_gal_trace-based_2009}, Lua\footnote{\texttt{http://luajit.org/}} +and now Python (and other languages) via PyPy. The core idea of tracing JITs is to focus the optimization effort of the JIT compiler on the hot paths of the core loops of the program and to just use an @@ -1013,7 +1013,7 @@ objects. The JIT will then fill these objects only lazily when they are actually accessed (\eg because a debugger is used). Therefore in the common case, nothing is stored into the frame objects, making the problem of too much escaping go -away. This is a common approach in VM implementations \cite{XXX}, the only +away. This is a common approach in VM implementations \cite{miranda_context_1999}, the only novelty in our approach lays in its generality, because most other JITs are just specifically written for one particular language. @@ -1040,12 +1040,12 @@ framework\footnote{\texttt{http://www.djangoproject.com/}}. \item \textbf{go}: A Monte-Carlo Go AI\footnote{\texttt{http://shed-skin.blogspot.com/2009/07/ disco-elegant-python-go-player.html}}. - \item \textbf{html5lib}: HTML5 parser - \item \textbf{pyflate-fast}: BZ2 decoder - \item \textbf{raytrace-simple}: ray tracer - \item \textbf{richards}: The Richards benchmark \cite{XXX} + \item \textbf{html5lib}: An HTML5 parser. + \item \textbf{pyflate-fast}: A BZ2 decoder. + \item \textbf{raytrace-simple}: A ray tracer. + \item \textbf{richards}: The Richards benchmark. \item \textbf{spambayes}: A Bayesian spam filter\footnote{\texttt{http://spambayes.sourceforge.net/}}. - \item \textbf{telco}: A Python version of the Telco decimal + \item \textbf{telco}: A Python version of the Telco decimal. benchmark\footnote{\texttt{http://speleotrove.com/decimal/telco.html}}, using a pure Python decimal floating point implementation. \item \textbf{twisted\_names}: A DNS server benchmark using the Twisted networking @@ -1090,7 +1090,7 @@ various Python implementations on the benchmarks. As a baseline, we used the standard Python implementation in C, called CPython\footnote{\texttt{http://python.org}}, which uses a bytecode-based -interpreter. Furthermore we compared against Psyco \cite{XXX}, an extension to +interpreter. Furthermore we compared against Psyco \cite{rigo_representation-based_2004}, an extension to CPython which is a just-in-time compiler that produces machine code at run-time. It is not based on traces. Of PyPy's Python interpreter we used three versions, one without a JIT, one including the JIT but not using the allocation removal @@ -1099,7 +1099,7 @@ All benchmarks were run 50 times in the same process, to give the JIT time to produce machine code. The arithmetic mean of the times of the last 30 runs were used as the result. The errors were computed using a confidence interval with a 95\% -confidence level \cite{XXX}. The results are reported in Figure~\ref{fig:times}. +confidence level \cite{georges_statistically_2007}. The results are reported in Figure~\ref{fig:times}. With the optimization turned on, PyPy's Python interpreter outperforms CPython in all benchmarks except spambayes (which heavily relies on regular expression performance). All benchmarks are improved by the allocation removal @@ -1156,8 +1156,8 @@ SPUR, a tracing JIT for C\# seems to be able to remove allocations in a similar way to the approach described here, as hinted at in the technical report -\cite{XXX}. However, no details for the approach and its implementation are -given. +\cite{michael_bebenita_spur:_2010}. However, no details for the approach and its +implementation are given. partial evaluation: From cfbolz at codespeak.net Tue Oct 12 17:17:25 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 17:17:25 +0200 (CEST) Subject: [pypy-svn] r77843 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012151725.5DCBA282BF6@codespeak.net> Author: cfbolz Date: Tue Oct 12 17:17:23 2010 New Revision: 77843 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: use nice macros Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 17:17:23 2010 @@ -836,14 +836,14 @@ recursively lifting a static object is not a constant-time operation, the algorithm only takes a total time linear in the length of the trace. Moreover, it gives the ``best'' possible result within its constrains, -e.g.\ in term of the number of residual operations. The +\eg in term of the number of residual operations. The algorithm itself is not particularly complex or innovative; our focus is rather that \emph{in the context of tracing JITs} it is possible to find a simple enough algorithm that still gives the best results. -Note in particular that objects in category 1 (i.e.\ the ones that do +Note in particular that objects in category 1 (\ie the ones that do not escape) are completely removed; moreover, objects in category 2 -(i.e.\ escaping) are still partially dealt with: if such an object +(\ie escaping) are still partially dealt with: if such an object escapes later than its creation point, all the operations inbetween that involve the object are removed. From cfbolz at codespeak.net Tue Oct 12 17:22:44 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 17:22:44 +0200 (CEST) Subject: [pypy-svn] r77844 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012152244.B20C9282BF6@codespeak.net> Author: cfbolz Date: Tue Oct 12 17:22:43 2010 New Revision: 77844 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: kill the description of the cross-loop optimization to save space :`-( Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 17:22:43 2010 @@ -148,12 +148,10 @@ informally described in Section~\ref{sec:statics}, a more formal description is given in Section~\ref{sec:formal}. -The basic approach of static objects can then be extended to also be used for -type-specializing the traces that are produced by the tracing JIT -(Section~\ref{sec:crossloop}). In Section~\ref{sec:support} we describe some -supporting techniques that are not central to the approach, but are needed to -improve the results. The introduced techniques are evaluated in -Section~\ref{sec:Evaluation} using PyPy's Python interpreter as a case study. +In Section~\ref{sec:support} we describe some supporting techniques that are not +central to the approach, but are needed to improve the results. The introduced +techniques are evaluated in Section~\ref{sec:Evaluation} using PyPy's Python +interpreter as a case study. The contributions of this paper are: @@ -485,10 +483,7 @@ \item Category 4: Objects that live for a while, survive across the jump, and then escape. To these we also count the objects that live across several - jumps and then either escape or stop being used.\footnote{In theory, the - approach of Section~\ref{sec:crossloop} works also for objects that live for - exactly $n>1$ iterations and then don't escape, but we expect this to be a - very rare case, so we do not handle it.} + jumps and then either escape or stop being used. \end{itemize} The objects that are allocated in the example trace in @@ -497,8 +492,8 @@ category 3. The creation of objects in category 1 is removed by the optimization described -in Sections~\ref{sec:statics} and \ref{sec:formal}. We will look at objects in -category 3 in Section~\ref{sec:crossloop}. +in Sections~\ref{sec:statics} and \ref{sec:formal}. Objects in the other +categories are partially optimized by this approach as well. \section{Allocation Removal in Traces} \label{sec:statics} @@ -864,132 +859,6 @@ % section Escape Analysis in a Tracing JIT (end) -\section{Allocation Removal Across Loop Boundaries} -\label{sec:crossloop} - -In the last sections we described how partial evaluation can be used to remove -many of the allocations of short-lived objects and many of the type dispatches -that are present in a non-optimized trace. In this section we will improve the -optimization to also handle more cases. - -The optimization of the last section considered the passing of an object along a -jump to be equivalent to escaping. It was thus treating objects in category 3 -and 4 like those in category 2. - -The improved optimization described in this section will make it possible to deal -better with objects in category 3 and 4. This will have two consequences: on -the one hand, more allocations are removed from the trace (which is clearly -good). As a side-effect of this, the traces will also be type-specialized. - - -%___________________________________________________________________________ - -\subsection{Optimizing Across the Jump} - - -Let's look at the final trace obtained in Section~\ref{sec:statics} for the -example loop. The final trace (Figure~\ref{fig:step1}) was much better than the -original one, because many allocations were removed from it. However, it also -still contained allocations. - -The two new \texttt{BoxedIntegers} stored in $p_{15}$ and $p_{10}$ are passed into -the next iteration of the loop. The next iteration will check that they are -indeed \texttt{BoxedIntegers}, read their \texttt{intval} fields and then not use them -any more. Thus those instances are in category 3. - -In its current state the loop -allocates two \texttt{BoxedIntegers} at the end of every iteration, that then die -very quickly in the next iteration. In addition, the type checks at the start -of the loop are superfluous, at least after the first iteration. - - -The reason why we cannot optimize the remaining allocations away is because -their lifetime crosses the jump. To improve the situation, a little trick is -needed.\footnote{The algorithm that PyPy currently uses is significantly more complex -than the one that is described here. The resulting behaviour is nearly -identical, however, so we will use the simpler version (and plan to switch to -that at some point in the actual implementation).} -The trace in Figure~\ref{fig:step1} represents a loop, i.e. the jump at the end jumps to -the beginning. Where in the loop the jump occurs is arbitrary, since the loop -can only be left via failing guards anyway. Therefore it does not change the -semantics of the loop to put the jump at another point into the trace and we -can move the \texttt{jump} operation just above the allocation of the objects that -appear in the current \texttt{jump}. This needs some care, because the arguments to -\texttt{jump} are all currently live variables, thus they need to be adapted. - -\begin{figure} -\includegraphics{figures/step2.pdf} -\label{fig:step2} -\caption{Shifting the Jump} -\end{figure} - -If we do that for our example trace, the trace looks like in Figure~\ref{fig:step2}. -Now the lifetime of the remaining allocations no longer crosses the jump, and -we can run our partial evaluation a second time, to get the trace in -Figure~\ref{fig:step3}. - -\begin{figure} -\includegraphics{figures/step3.pdf} -\label{fig:step3} -\caption{Removing Allocations a Second Time} -\end{figure} - -This result is now really good. The code performs the same operations than -the original code, but using direct CPU arithmetic and no boxing, as opposed to -the original version which used dynamic dispatching and boxing. - -Looking at the final trace it is also completely clear that specialization has -happened. The trace corresponds to the situation in which the trace was -originally recorded, which happened to be a loop where \texttt{BoxedIntegers} were -used. The now resulting loop does not refer to the \texttt{BoxedInteger} class at -all any more, but it still has the same behaviour. If the original loop had -used \texttt{BoxedFloats}, the final loop would use \texttt{float\_*} operations -everywhere instead (or even be very different, if the object model had -more different classes). - - -%___________________________________________________________________________ - -\subsection{Entering the Loop} - -The approach of placing the \texttt{jump} at some other point in the loop leads to -one additional complication that we glossed over so far. The beginning of the -original loop corresponds to a point in the original program, namely the -\texttt{while} loop in the function \texttt{f} from the last section. - -Now recall that in a VM that uses a tracing JIT, all programs start by being -interpreted. This means that when \texttt{f} is executed by the interpreter, it is -easy to go from the interpreter to the first version of the compiled loop. -After the \texttt{jump} is moved and the escape analysis optimization is applied a -second time, this is no longer easily possible. In particular, the new loop -expects two integers as input arguments, while the old one expected two -instances. - -To make it possible to enter the loop directly from the intepreter, there -needs to be some additional code that enters the loop by taking as input -arguments what is available to the interpreter, i.e. two instances. This -additional code corresponds to one iteration of the loop, which is thus -peeled off \cite{XXX}, see Figure~\ref{fig:step3}. - -\begin{figure} -\includegraphics{figures/step4.pdf} -\label{fig:step3} -\caption{A Way to Enter the Loop From the Interpreter} -\end{figure} - - -%___________________________________________________________________________ - -\subsection{Summary} - -The optimization described in this section can be used to optimize away -allocations in category 3 and improve allocations in category 4, by deferring -them until they are no longer avoidable. A side-effect of these optimizations -is also that the optimized loops are specialized for the types of the variables -that are used inside them. - -% section Allocation Removal Across Loop Boundaries (end) - \section{Supporting Techniques} \label{sec:support} From arigo at codespeak.net Tue Oct 12 17:24:09 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 12 Oct 2010 17:24:09 +0200 (CEST) Subject: [pypy-svn] r77845 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012152409.8C17D282BF6@codespeak.net> Author: arigo Date: Tue Oct 12 17:24:08 2010 New Revision: 77845 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: We plan to have only one conclusion, not many. Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 17:24:08 2010 @@ -1040,7 +1040,7 @@ xxx: relation to compile-time garbage collection \cite{mazur_practical_2001}; separation logic; John Hughes: type specialization -\section{Conclusions} +\section{Conclusion} \label{sec:conclusions} \section*{Acknowledgements} From cfbolz at codespeak.net Tue Oct 12 17:26:33 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 17:26:33 +0200 (CEST) Subject: [pypy-svn] r77846 - in pypy/extradoc/talk/pepm2011: . benchmarks Message-ID: <20101012152633.BE191282BF6@codespeak.net> Author: cfbolz Date: Tue Oct 12 17:26:32 2010 New Revision: 77846 Modified: pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric pypy/extradoc/talk/pepm2011/escape-tracing.pdf pypy/extradoc/talk/pepm2011/paper.tex Log: clean up figures Modified: pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric ============================================================================== Binary files. No diff available. Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf ============================================================================== Binary files. No diff available. Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 17:26:32 2010 @@ -266,7 +266,8 @@ mixed addition) and \texttt{is\_positive}, which returns whether the number is greater than zero. The implementation of \texttt{add} uses classical Smalltalk-like double-dispatching. These classes could be part of the implementation of a very -simple interpreter written in RPython. +simple interpreter written in RPython. The classes can be seen in +Figure~\ref{fig:objmodel}. \begin{figure} \begin{verbatim} @@ -313,6 +314,7 @@ return self.floatval > 0.0 \end{verbatim} \caption{A simple object model} +\label{fig:objmodel} \end{figure} Using these classes to implement arithmetic shows the basic problem that a @@ -409,8 +411,8 @@ jump($p_{15}$, $p_{10}$) \\ \end{tabular} } -\label{fig:unopt-trace} \caption{Unoptimized Trace for the Simple Object Model} +\label{fig:unopt-trace} \end{figure} If the function is executed using the tracing JIT, with \texttt{y} being a @@ -607,14 +609,11 @@ \begin{figure} \includegraphics{figures/step1.pdf} -\label{fig:step1} \caption{Resulting Trace After Allocation Removal} +\label{fig:step1} \end{figure} -The final optimized trace of the example can be seen in -Figure~\ref{fig:step1}. -XXX why does it says ``Figure 4.1'' here but ``Figure 4'' in the label? - +The final optimized trace of the example can be seen in Figure~\ref{fig:step1}. The optimized trace contains only two allocations, instead of the original five, and only three \texttt{guard\_class} operations, from the original seven. @@ -656,8 +655,8 @@ $$ \end{minipage} \end{center} -\label{fig:semantics} \caption{The Operational Semantics of Simplified Traces} +\label{fig:semantics} \end{figure*} In this section we want to give a formal description of the semantics of the @@ -756,8 +755,8 @@ $$ \end{minipage} \end{center} -\label{fig:optimization} \caption{Optimization Rules} +\label{fig:optimization} \end{figure*} To optimize the simple traces from the last section, we use online partial From cfbolz at codespeak.net Tue Oct 12 17:32:58 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 17:32:58 +0200 (CEST) Subject: [pypy-svn] r77847 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012153258.91DCD282BF6@codespeak.net> Author: cfbolz Date: Tue Oct 12 17:32:57 2010 New Revision: 77847 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: actually reference this figure Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 17:32:57 2010 @@ -763,7 +763,8 @@ evaluation. The partial evaluator optimizes one operation of the trace at a time. Every operation in the unoptimized trace is replaced by a list of operations in the optimized trace. This list is empty if the operation -can be optimized away (which hopefully happens often). +can be optimized away (which hopefully happens often). The optimization rules +can be seen in Figure~\ref{fig:optimization}. The state of the optimizer is stored in an environment $E$ and a \emph{static heap} $S$. The environment is a partial function from variables in the From afa at codespeak.net Tue Oct 12 17:36:24 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 12 Oct 2010 17:36:24 +0200 (CEST) Subject: [pypy-svn] r77848 - in pypy/branch/fast-forward/pypy/module/_multiprocessing: . test Message-ID: <20101012153624.79D49282BF6@codespeak.net> Author: afa Date: Tue Oct 12 17:36:23 2010 New Revision: 77848 Added: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_memory.py (contents, props changed) pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_memory.py (contents, props changed) Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/__init__.py Log: Implement _multiprocessing.address_of_buffer(buf) For mmap buffers only! Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/__init__.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/__init__.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/__init__.py Tue Oct 12 17:36:23 2010 @@ -7,6 +7,8 @@ 'Connection' : 'interp_connection.W_FileConnection', 'PipeConnection' : 'interp_connection.W_PipeConnection', 'SemLock' : 'interp_semaphore.W_SemLock', + + 'address_of_buffer' : 'interp_memory.address_of_buffer', } appleveldefs = { Added: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_memory.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_memory.py Tue Oct 12 17:36:23 2010 @@ -0,0 +1,16 @@ +from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root +from pypy.interpreter.gateway import unwrap_spec +from pypy.interpreter.error import OperationError +from pypy.rpython.lltypesystem import rffi +from pypy.module.mmap.interp_mmap import W_MMap + + at unwrap_spec(ObjSpace, W_Root) +def address_of_buffer(space, w_obj): + if space.config.objspace.usemodules.mmap: + mmap = space.interp_w(W_MMap, w_obj) + address = rffi.cast(rffi.SIZE_T, mmap.mmap.data) + return space.newtuple([space.wrap(address), + space.wrap(mmap.mmap.size)]) + else: + raise OperationError(space.w_TypeError, space.wrap( + "cannot get address of buffer")) Added: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_memory.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_memory.py Tue Oct 12 17:36:23 2010 @@ -0,0 +1,40 @@ +from pypy.conftest import gettestobjspace + +class AppTestMemory: + def setup_class(cls): + space = gettestobjspace( + usemodules=('_multiprocessing', 'mmap', '_rawffi')) + cls.space = space + + def test_address_of(self): + import _multiprocessing + raises(TypeError, _multiprocessing.address_of_buffer, None) + raises(TypeError, _multiprocessing.address_of_buffer, "a") + + def test_mmap_address(self): + import mmap + import _multiprocessing + + # This is a bit faster than importing ctypes + import _ctypes + class c_double(_ctypes._SimpleCData): + _type_ = "d" + sizeof_double = _ctypes.sizeof(c_double) + + buf = mmap.mmap(-1, 300) + buf[0:300] = '\0' * 300 + + # Get the address of shared memory + address, length = _multiprocessing.address_of_buffer(buf) + assert length == 300 + + # build a ctypes object from it + var = c_double.from_address(address) + assert buf[0:sizeof_double] == '\0' * sizeof_double + assert var.value == 0 + + # check that both objects share the same memory + var.value = 123.456 + assert buf[0:sizeof_double] != '\0' * sizeof_double + buf[0:sizeof_double] = '\0' * sizeof_double + assert var.value == 0 From arigo at codespeak.net Tue Oct 12 18:26:30 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 12 Oct 2010 18:26:30 +0200 (CEST) Subject: [pypy-svn] r77849 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012162630.8D124282BF6@codespeak.net> Author: arigo Date: Tue Oct 12 18:26:28 2010 New Revision: 77849 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: (cfbolz, arigo) Conclusion. Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 18:26:28 2010 @@ -1041,7 +1041,25 @@ \section{Conclusion} -\label{sec:conclusions} +\label{sec:conclusion} + +In this paper, we used a approach based on partial evaluation to +optimize allocations in the traces of a tracing JIT. In this context a +simple approach to partial evaluation gives good results. This is due +to the fact that the tracing JIT itself is responsible for all control +issues, which are usually the hardest part of partial evaluation: the +tracing JIT selects the parts of the program that are worthwhile to +optimize, and extracts linear paths through them, inlining functions as +necessary. What is left to optimize is only those linear paths. + +We expect a similar result for other optimizations that usually require +a complex analysis phase and are thus normally too slow to use at +runtime. A tracing JIT selects interesting linear paths by itself; +therefore, a naive version of many optimizations on such paths should +give mostly the same results. For example, we experimented with (and +plan to write about) store-load propagation with a very simple alias +analysis. + \section*{Acknowledgements} From arigo at codespeak.net Tue Oct 12 18:42:29 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Tue, 12 Oct 2010 18:42:29 +0200 (CEST) Subject: [pypy-svn] r77850 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012164229.30827282BF6@codespeak.net> Author: arigo Date: Tue Oct 12 18:42:27 2010 New Revision: 77850 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: (cfbolz, arigo) Abstract. Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 18:42:27 2010 @@ -74,6 +74,14 @@ \maketitle \begin{abstract} +The performance of many dynamic language implementations suffers from +high allocation rates and runtime type checks. This makes dynamic +languages less applicable to purely algorithmic problems, despite their +growing popularity. In this paper, we present a simple optimization +based on online partial evaluation to remove object allocations and +runtime type checks in the context of a tracing JIT. We evaluate the +optimization using a Python VM and find that it gives good results for +all our (real-life) benchmarks. \footnote{This research is partially supported by the BMBF funded project PyJIT (nr. 01QE0913B; Eureka Eurostars).} \end{abstract} @@ -1043,7 +1051,7 @@ \section{Conclusion} \label{sec:conclusion} -In this paper, we used a approach based on partial evaluation to +In this paper, we used a approach based on online partial evaluation to optimize allocations in the traces of a tracing JIT. In this context a simple approach to partial evaluation gives good results. This is due to the fact that the tracing JIT itself is responsible for all control From afa at codespeak.net Tue Oct 12 18:56:33 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Tue, 12 Oct 2010 18:56:33 +0200 (CEST) Subject: [pypy-svn] r77851 - pypy/branch/fast-forward/pypy/module/_multiprocessing Message-ID: <20101012165633.23B64282BF6@codespeak.net> Author: afa Date: Tue Oct 12 18:56:31 2010 New Revision: 77851 Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py Log: Add missing constant on Windows Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py (original) +++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_win32.py Tue Oct 12 18:56:31 2010 @@ -16,6 +16,7 @@ NMPWAIT_WAIT_FOREVER ERROR_PIPE_CONNECTED ERROR_SEM_TIMEOUT ERROR_PIPE_BUSY ERROR_NO_SYSTEM_RESOURCES ERROR_BROKEN_PIPE ERROR_MORE_DATA + ERROR_ALREADY_EXISTS """.split() class CConfig: From hakanardo at codespeak.net Tue Oct 12 19:03:56 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Tue, 12 Oct 2010 19:03:56 +0200 (CEST) Subject: [pypy-svn] r77852 - pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt Message-ID: <20101012170356.1EBA2282BF6@codespeak.net> Author: hakanardo Date: Tue Oct 12 19:03:53 2010 New Revision: 77852 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py Log: virtual arrays Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py Tue Oct 12 19:03:53 2010 @@ -203,6 +203,15 @@ def _make_virtual(self, modifier): return modifier.make_varray(self.arraydescr) + def get_forced_boxes(self): + if self.box is None: + boxes = [] + for itemvalue in self._items: + boxes.extend(itemvalue.get_forced_boxes()) + return boxes + else: + return [self.box] + class __extend__(SpecNode): def setup_virtual_node(self, optimizer, box, newinputargs): From cfbolz at codespeak.net Tue Oct 12 19:07:12 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Tue, 12 Oct 2010 19:07:12 +0200 (CEST) Subject: [pypy-svn] r77853 - pypy/extradoc/talk/pepm2011 Message-ID: <20101012170712.3A1A1282BF7@codespeak.net> Author: cfbolz Date: Tue Oct 12 19:07:10 2010 New Revision: 77853 Modified: pypy/extradoc/talk/pepm2011/math.lyx pypy/extradoc/talk/pepm2011/paper.tex Log: re-import math from lyx file Modified: pypy/extradoc/talk/pepm2011/math.lyx ============================================================================== --- pypy/extradoc/talk/pepm2011/math.lyx (original) +++ pypy/extradoc/talk/pepm2011/math.lyx Tue Oct 12 19:07:10 2010 @@ -61,7 +61,7 @@ \begin_layout Standard \begin_inset Formula \begin{eqnarray*} u,v & \in & V\mathrm{\,\, are\, Variables\, in\, the\, original\, trace}\\ -u^{*},v^{*},w^{*} & \in & V^{*}\,\mathrm{\, are\, Variables\, in\, the\, optimized\, trace}\\ +u^{*},v^{*} & \in & V^{*}\,\mathrm{\, are\, Variables\, in\, the\, optimized\, trace}\\ T & \in & \mathfrak{T}\mathrm{\,\, are\, runtime\, types}\\ F & \in & \left\{ L,R\right\} \,\mathrm{\, are\, fields\, of\, objects}\\ l & \in & L\,\mathrm{\, are\, locations\, in\, the\, heap}\end{eqnarray*} Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Tue Oct 12 19:07:10 2010 @@ -632,12 +632,9 @@ \begin{figure*} \begin{center} \begin{tabular}{lcccc} -\emph{new} & ${\displaystyle \frac{l\,\mathrm{fresh}}{v=\mathrm{new}(T),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto l\right],H\left[l\mapsto\left(T,\mathrm{null},\mathrm{null}\right)\right]}}$ & ~~~ & -\emph{guard} & ${\displaystyle \frac{\mathrm{type}(H(E(v))=T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H}}$\tabularnewline[3em] -\emph{get} & ${\displaystyle \frac{\,}{v=\mathrm{get}(u,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto H\left(E\left(u\right)\right)_{F}\right],H}}$ & ~~~ & -& ${\displaystyle \frac{\mathrm{type}(H(E(v))\neq T}{\mathrm{guard}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$\tabularnewline[3em] -\emph{set} & ${\displaystyle \frac{\,}{\mathrm{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H\left[E\left(v\right)\mapsto\left(H\left(E\left(v\right)\right)!_{F}E(u)\right)\right]}}$ & ~~~ & -& \tabularnewline[4em] +\emph{new} & ${\displaystyle \frac{l\,\mathrm{fresh}}{v=\mathtt{new}(T),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[v\mapsto l\right],H\left[l\mapsto\left(T,\mathrm{null},\mathrm{null}\right)\right]}}$ & ~~~ & \emph{guard} & ${\displaystyle \frac{\mathrm{type}(H(E(v))=T}{\mathtt{guard\_class}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H}}$\tabularnewline[3em] +\emph{get} & ${\displaystyle \frac{\,}{u=\mathtt{get}(v,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[u\mapsto H\left(E\left(v\right)\right)_{F}\right],H}}$ & ~~~ & & ${\displaystyle \frac{\mathrm{type}(H(E(v))\neq T}{\mathtt{guard\_class}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$\tabularnewline[3em] +\emph{set} & ${\displaystyle \frac{\,}{\mathtt{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H\left[E\left(v\right)\mapsto\left(H\left(E\left(v\right)\right)!_{F}E(u)\right)\right]}}$ & ~~~ & & \tabularnewline[4em] \end{tabular} \begin{minipage}[b]{7 cm} @@ -728,24 +725,24 @@ \begin{figure*} \begin{center} \begin{tabular}{lc} -\emph{new} & ${\displaystyle \frac{v^{*}\,\mathrm{fresh}}{v=\mathrm{new}(T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto v^{*}\right],S\left[v^{*}\mapsto\left(T,\mathrm{null,null}\right)\right]}}$\tabularnewline[3em] -\emph{get} & ${\displaystyle \frac{E(u)\in\mathrm{dom}(S)}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto S(E(u))_{F}\right],S}}$\tabularnewline[3em] - & ${\displaystyle \frac{E(u)\notin\mathrm{dom}(S)\, v^{*}\,\mathrm{fresh}}{v=\mathrm{get}(u,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle v^{*}=\mathrm{get}(E(u),F)\right\rangle ,E\left[v\mapsto v^{*}\right],S}}$\tabularnewline[3em] -\emph{set} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S\left[E\left(v\right)\mapsto\left(S(E(v))!_{F}E(u)\right)\right]}}$\tabularnewline[3em] - & ${\displaystyle \frac{E(v)\notin\mathrm{dom}\left(S\right),\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathrm{set}\left(E(v),F,E(u)\right)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] -\emph{guard} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$\tabularnewline[3em] - & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathrm{guard}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathrm{guard}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] +\emph{new} & ${\displaystyle \frac{v^{*}\,\mathrm{fresh}}{v=\mathtt{new}(T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto v^{*}\right],S\left[v^{*}\mapsto\left(T,\mathrm{null,null}\right)\right]}}$\tabularnewline[3em] +\emph{get} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{u=\mathtt{get}(v,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[u\mapsto S(E(v))_{F}\right],S}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\, u^{*}\,\mathrm{fresh}}{u=\mathtt{get}(v,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle u^{*}=\mathtt{get}(E(v),F)\right\rangle ,E\left[u\mapsto u^{*}\right],S}}$\tabularnewline[3em] +\emph{set} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{\mathtt{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S\left[E\left(v\right)\mapsto\left(S(E(v))!_{F}E(u)\right)\right]}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(v)\notin\mathrm{dom}\left(S\right),\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathtt{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathtt{set}\left(E(v),F,E(u)\right)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] +\emph{guard} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$\tabularnewline[3em] + & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathtt{guard\_class}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em] \emph{lifting} & ${\displaystyle \frac{v^{*}\notin\mathrm{dom}(S)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle \,\right\rangle ,S}}$\tabularnewline[3em] - & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathrm{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em] - & ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathrm{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathrm{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline[3em] + & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathtt{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em] + & ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathtt{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathtt{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline[3em] \end{tabular} \begin{minipage}[b]{7 cm} \emph{Object Domains:} $$\setlength\arraycolsep{0.1em} \begin{array}{rcll} - u,v,w & \in & V & \mathrm{\ variables\ in\ trace}\\ - u^*,v^*,w^* & \in & V^* & \mathrm{\ variables\ in\ optimized\ trace}\\ + u,v & \in & V & \mathrm{\ variables\ in\ trace}\\ + u^*,v^* & \in & V^* & \mathrm{\ variables\ in\ optimized\ trace}\\ T & \in & \mathfrak{T} & \mathrm{\ runtime\ types}\\ F & \in & \left\{ L,R\right\} & \mathrm{\ fields\ of\ objects}\\ \end{array} From hakanardo at codespeak.net Tue Oct 12 19:51:22 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Tue, 12 Oct 2010 19:51:22 +0200 (CEST) Subject: [pypy-svn] r77854 - pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt Message-ID: <20101012175122.E994F282BF7@codespeak.net> Author: hakanardo Date: Tue Oct 12 19:51:20 2010 New Revision: 77854 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Log: cleanups Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py Tue Oct 12 19:51:20 2010 @@ -209,9 +209,9 @@ # the following assert should always be true for now, # because invalid loops that would fail it are detected # earlier, in optimizefindnode.py. + if realclassbox.same_constant(expectedclassbox): + return raise InvalidLoop - assert realclassbox.same_constant(expectedclassbox) - return emit_operation = True if value.last_guard_index != -1: # there already has been a guard_nonnull or guard_class or Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Tue Oct 12 19:51:20 2010 @@ -19,7 +19,6 @@ if op.getopnum() == rop.JUMP: loop = self.optimizer.loop loop.preamble.operations = self.optimizer.newoperations - print '\n'.join([str(o) for o in loop.preamble.operations]) self.optimizer.newoperations = [] jump_args = op.getarglist() op.initarglist([]) @@ -50,7 +49,6 @@ for a in self.getvalue(arg).get_forced_boxes(): if not isinstance(a, Const): inputargs.append(a) - print "Inputargs: ", inputargs for op in loop_operations: #import pdb; pdb.set_trace() @@ -72,12 +70,10 @@ args.extend(self.getvalue(arg).get_forced_boxes()) newop.initarglist(args + inputargs[len(args):]) - print "P: ", newop current = len(self.optimizer.newoperations) self.emit_operation(newop) for op in self.optimizer.newoperations[current:]: - print "E: ", op if op.is_guard(): op.getdescr().rd_snapshot = None #FIXME: In the right place? args = op.getarglist() @@ -87,7 +83,6 @@ if not isinstance(a, Const) and a in self.optimizer.values: v = self.getvalue(a) if v.fromstart and a not in inputargs: - print "Arg: ", a inputargs.append(a) if op.getopnum() == rop.JUMP: op.initarglist(op.getarglist() + [argmap[a]]) From hakanardo at codespeak.net Tue Oct 12 20:18:09 2010 From: hakanardo at codespeak.net (hakanardo at codespeak.net) Date: Tue, 12 Oct 2010 20:18:09 +0200 (CEST) Subject: [pypy-svn] r77855 - pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt Message-ID: <20101012181809.125C9282BF6@codespeak.net> Author: hakanardo Date: Tue Oct 12 20:18:08 2010 New Revision: 77855 Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Log: translation fixes Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py ============================================================================== --- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py (original) +++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py Tue Oct 12 20:18:08 2010 @@ -59,7 +59,9 @@ argmap[op.result] = newop.result descr = newop.getdescr() if isinstance(descr, ResumeGuardDescr): - op.getdescr().rd_snapshot = None #FIXME: In the right place? + orgdescr = op.getdescr() + assert isinstance(orgdescr, ResumeGuardDescr) + orgdescr.rd_snapshot = None #FIXME: In the right place? descr.rd_numb = None descr.rd_snapshot = self.inline_snapshot(descr.rd_snapshot) @@ -75,7 +77,9 @@ for op in self.optimizer.newoperations[current:]: if op.is_guard(): - op.getdescr().rd_snapshot = None #FIXME: In the right place? + descr = op.getdescr() + assert isinstance(descr, ResumeGuardDescr) + descr.rd_snapshot = None #FIXME: In the right place? args = op.getarglist() if op.is_guard(): args = args + op.getfailargs() @@ -103,7 +107,9 @@ boxes.append(a) else: boxes.append(self.inline_arg(a)) - return Snapshot(self.inline_snapshot(snapshot.prev), boxes) + new_snapshot = Snapshot(self.inline_snapshot(snapshot.prev), boxes[:]) + self.snapshot_map[snapshot] = new_snapshot + return new_snapshot From antocuni at codespeak.net Wed Oct 13 11:12:19 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 13 Oct 2010 11:12:19 +0200 (CEST) Subject: [pypy-svn] r77857 - in pypy/branch/jitffi/pypy: jit/backend/llsupport rlib Message-ID: <20101013091219.2F135282B9E@codespeak.net> Author: antocuni Date: Wed Oct 13 11:12:17 2010 New Revision: 77857 Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py pypy/branch/jitffi/pypy/rlib/clibffi.py pypy/branch/jitffi/pypy/rlib/libffi.py Log: fix translation Modified: pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py ============================================================================== --- pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py (original) +++ pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py Wed Oct 13 11:12:17 2010 @@ -1,3 +1,4 @@ +from pypy.rlib.rarithmetic import intmask from pypy.jit.metainterp import history from pypy.jit.backend.llsupport.descr import DynamicIntCallDescr, NonGcPtrCallDescr,\ FloatCallDescr, VoidCallDescr @@ -12,7 +13,8 @@ return None # ?? arg_classes = ''.join(argkinds) if reskind == history.INT: - return DynamicIntCallDescr(arg_classes, ffi_result.c_size, extrainfo) + size = intmask(ffi_result.c_size) + return DynamicIntCallDescr(arg_classes, size, extrainfo) elif reskind == history.REF: return NonGcPtrCallDescr(arg_classes, extrainfo) elif reskind == history.FLOAT: Modified: pypy/branch/jitffi/pypy/rlib/clibffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/clibffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/clibffi.py Wed Oct 13 11:12:17 2010 @@ -378,6 +378,8 @@ ll_cif = lltype.nullptr(FFI_CIFP.TO) ll_argtypes = lltype.nullptr(FFI_TYPE_PP.TO) + _immutable_fields_ = ['argtypes', 'restype'] + def __init__(self, name, argtypes, restype, flags=FUNCFLAG_CDECL): self.name = name self.argtypes = argtypes Modified: pypy/branch/jitffi/pypy/rlib/libffi.py ============================================================================== --- pypy/branch/jitffi/pypy/rlib/libffi.py (original) +++ pypy/branch/jitffi/pypy/rlib/libffi.py Wed Oct 13 11:12:17 2010 @@ -155,10 +155,10 @@ class Func(AbstractFuncPtr): - _immutable_fields_ = ['funcsym', 'argtypes', 'restype'] + _immutable_fields_ = ['funcsym'] argtypes = [] - restype = None - funcsym = None + restype = lltype.nullptr(clibffi.FFI_TYPE_P.TO) + funcsym = lltype.nullptr(rffi.VOIDP.TO) def __init__(self, name, argtypes, restype, funcsym, flags=FUNCFLAG_CDECL, keepalive=None): From afa at codespeak.net Wed Oct 13 11:46:26 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Wed, 13 Oct 2010 11:46:26 +0200 (CEST) Subject: [pypy-svn] r77858 - in pypy/branch/fast-forward: lib_pypy pypy/config pypy/doc/config pypy/module/_md5 pypy/module/_md5/test pypy/module/_sha pypy/module/_sha/test pypy/module/md5 pypy/module/sha Message-ID: <20101013094626.5A178282BD4@codespeak.net> Author: afa Date: Wed Oct 13 11:46:24 2010 New Revision: 77858 Added: pypy/branch/fast-forward/lib_pypy/_md5.py - copied unchanged from r77812, pypy/branch/fast-forward/lib_pypy/md5.py pypy/branch/fast-forward/lib_pypy/_sha.py - copied unchanged from r77812, pypy/branch/fast-forward/lib_pypy/sha.py pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._md5.txt - copied, changed from r77856, pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules.md5.txt pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._sha.txt - copied, changed from r77856, pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules.sha.txt pypy/branch/fast-forward/pypy/module/_md5/ - copied from r77856, pypy/branch/fast-forward/pypy/module/md5/ pypy/branch/fast-forward/pypy/module/_sha/ - copied from r77856, pypy/branch/fast-forward/pypy/module/sha/ Removed: pypy/branch/fast-forward/lib_pypy/hashlib.py pypy/branch/fast-forward/lib_pypy/md5.py pypy/branch/fast-forward/lib_pypy/sha.py pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules.md5.txt pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules.sha.txt pypy/branch/fast-forward/pypy/module/md5/ pypy/branch/fast-forward/pypy/module/sha/ Modified: pypy/branch/fast-forward/pypy/config/pypyoption.py pypy/branch/fast-forward/pypy/module/_md5/test/test_md5.py pypy/branch/fast-forward/pypy/module/_sha/test/test_sha.py Log: Rename the builtin modules: md5 to _md5, sha to _sha. Now the standard md5.py imports hashlib, which first tries to import _hashlib (based on openSSL) and falls back to _md5. Next step: have a working _hashlib on Windows. Modified: pypy/branch/fast-forward/pypy/config/pypyoption.py ============================================================================== --- pypy/branch/fast-forward/pypy/config/pypyoption.py (original) +++ pypy/branch/fast-forward/pypy/config/pypyoption.py Wed Oct 13 11:46:24 2010 @@ -29,13 +29,13 @@ ["_socket", "unicodedata", "mmap", "fcntl", "rctime" , "select", "zipimport", "_lsprof", "crypt", "signal", "_rawffi", "termios", "zlib", - "struct", "md5", "sha", "bz2", "_minimal_curses", "cStringIO", + "struct", "_md5", "_sha", "bz2", "_minimal_curses", "cStringIO", "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array"] )) working_oo_modules = default_modules.copy() working_oo_modules.update(dict.fromkeys( - ["md5", "sha", "cStringIO", "itertools"] + ["_md5", "_sha", "cStringIO", "itertools"] )) # XXX this should move somewhere else, maybe to platform ("is this posixish" Copied: pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._md5.txt (from r77856, pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules.md5.txt) ============================================================================== --- pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules.md5.txt (original) +++ pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._md5.txt Wed Oct 13 11:46:24 2010 @@ -1,4 +1,4 @@ -Use the built-in 'md5' module. +Use the built-in '_md5' module. This module is expected to be working and is included by default. There is also a pure Python version in lib_pypy which is used if the built-in is disabled, but it is several orders of magnitude Copied: pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._sha.txt (from r77856, pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules.sha.txt) ============================================================================== --- pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules.sha.txt (original) +++ pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._sha.txt Wed Oct 13 11:46:24 2010 @@ -1,4 +1,4 @@ -Use the built-in 'sha' module. +Use the built-in _'sha' module. This module is expected to be working and is included by default. There is also a pure Python version in lib_pypy which is used if the built-in is disabled, but it is several orders of magnitude Modified: pypy/branch/fast-forward/pypy/module/_md5/test/test_md5.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/md5/test/test_md5.py (original) +++ pypy/branch/fast-forward/pypy/module/_md5/test/test_md5.py Wed Oct 13 11:46:24 2010 @@ -13,7 +13,7 @@ Create a space with the md5 module and import it for use by the tests. """ - cls.space = gettestobjspace(usemodules=['md5']) + cls.space = gettestobjspace(usemodules=['_md5']) cls.w_md5 = cls.space.appexec([], """(): import md5 return md5 Modified: pypy/branch/fast-forward/pypy/module/_sha/test/test_sha.py ============================================================================== --- pypy/branch/fast-forward/pypy/module/sha/test/test_sha.py (original) +++ pypy/branch/fast-forward/pypy/module/_sha/test/test_sha.py Wed Oct 13 11:46:24 2010 @@ -13,7 +13,7 @@ Create a space with the sha module and import it for use by the tests. """ - cls.space = gettestobjspace(usemodules=['sha']) + cls.space = gettestobjspace(usemodules=['_sha']) cls.w_sha = cls.space.appexec([], """(): import sha return sha From cfbolz at codespeak.net Wed Oct 13 13:30:17 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 13 Oct 2010 13:30:17 +0200 (CEST) Subject: [pypy-svn] r77859 - pypy/extradoc/talk/pepm2011/benchmarks Message-ID: <20101013113017.44F0C282BD6@codespeak.net> Author: cfbolz Date: Wed Oct 13 13:30:14 2010 New Revision: 77859 Added: pypy/extradoc/talk/pepm2011/benchmarks/psyco.json Modified: pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json Log: new benchmarks on my machine. for some strange reason, pyaes is huuuuugely slow. Modified: pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json ============================================================================== --- pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json (original) +++ pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json Wed Oct 13 13:30:14 2010 @@ -1 +1 @@ -{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [1.3408269882199999, 0.89828705787700003, 0.79922604560900001, 0.78520703315700002, 0.77960705757100002, 0.77884316444400004, 0.78138089180000003, 0.78564095497099995, 0.77933979034400003, 0.78032112121599995, 0.78032398223900001, 0.77994108200099999, 0.78709197044400003, 0.78046298027000005, 0.78100490570100001, 0.78174090385399997, 0.78066301345800004, 0.78935813903800001, 0.78030800819400004, 0.78058695793199995, 0.78065609931900004, 0.81349897384600001, 0.78035616874699998, 0.78057289123499995, 0.77895307540900005, 0.78031492233300004, 0.77951288223299997, 0.78010797500600004, 0.77941989898700004, 0.77966809272799997, 0.77928400039699997, 0.78034520149200004, 0.78004193306000003, 0.77843499183700005, 0.77900505065900005, 0.77978110313399995, 0.77990984916700001, 0.78063511848400002, 0.78277206420900003, 0.78097486496000001, 0.78050684928900005, 0.780658006668, 0.81388092041000004, 0.77942514419599995, 0.77982401847799998, 0.779586076736, 0.77912712097199999, 0.77842688560499995, 0.77977490425100005, 0.77885198593100002], "base_times": [1.96005105972, 1.9576470851900001, 1.96088910103, 1.9608309268999999, 1.9567291736600001, 1.9601850509600001, 1.9592499732999999, 1.95819783211, 1.95975589752, 1.9611279964399999, 1.9579570293399999, 1.96019601822, 1.95961117744, 1.95499491692, 1.9610810279799999, 1.95511603355, 1.9587879180900001, 1.95879602432, 1.95566296577, 1.9595639705700001, 1.95859694481, 1.9552450180100001, 1.95912909508, 1.9604980945599999, 1.95646905899, 1.9599859714500001, 1.96060585976, 1.9573318958299999, 1.95889997482, 1.9600548744199999, 1.9571371078499999, 1.96107792854, 1.9600939750699999, 1.9585239887200001, 1.9604239463799999, 1.95847797394, 1.9600219726599999, 1.9607260227200001, 1.95767593384, 1.9606490135200001, 1.9595348834999999, 1.9582331180600001, 1.95944404602, 1.96064209938, 1.9565119743299999, 1.9607801437400001, 1.9594299793200001, 1.95776391029, 1.95995807648, 1.9593291282700001]}], ["django", "RawResult", {"changed_times": [0.431762933731, 0.375938892365, 0.36341500282299999, 0.367589950562, 0.36318898200999999, 0.36300396919299999, 0.36316084861800002, 0.37451195716899999, 0.36371111869799999, 0.363828897476, 0.36306118965099998, 0.36771798133900002, 0.36362099647500001, 0.37479805946400002, 0.36376190185500001, 0.36297202110299998, 0.36336398124699998, 0.36382389068600002, 0.36317396163900001, 0.37901401519799999, 0.36382102966300001, 0.36297416687, 0.36419606208799998, 0.36393713951099999, 0.36413717269899998, 0.37551498413099998, 0.363807201385, 0.36678910255399999, 0.36428284645100001, 0.363599061966, 0.36372780799900001, 0.37592792511000001, 0.36405706405600002, 0.36310291290300001, 0.36305308341999998, 0.36352086067200001, 0.36760711669899998, 0.37471699714700002, 0.36391615867600002, 0.36355113983199999, 0.36311888694799999, 0.36337804794299999, 0.36314678192099997, 0.37514305114699997, 0.36750411987300002, 0.36400318145799998, 0.36394405364999999, 0.363122940063, 0.36291694641099997, 0.37458801269500003], "base_times": [0.62737584114099998, 0.62728309631300005, 0.627307891846, 0.62758588790900005, 0.62738704681400004, 0.62718105316200001, 0.62660717964199997, 0.62682294845599995, 0.62685489654500004, 0.626901865005, 0.62694001197799998, 0.62680888176000005, 0.62702178955099996, 0.62695503234899996, 0.626930952072, 0.62696814537000001, 0.62787008285500001, 0.62750697135900002, 0.62742996215799995, 0.62795114517200001, 0.62798190116899999, 0.62778282165499999, 0.62753701210000001, 0.62766814231900003, 0.62768888473499995, 0.62764692306500003, 0.627485990524, 0.62803101539600004, 0.62795495986899996, 0.62786889076200003, 0.62758994102499999, 0.62817692756700005, 0.62814188003500004, 0.62749600410499995, 0.62808799743699995, 0.62854099273700004, 0.62779498100300002, 0.62789297103899999, 0.62796211242699995, 0.62824296951299996, 0.62812089920000003, 0.62798500060999995, 0.62708497047400003, 0.62771701812699998, 0.62825989723200004, 0.62763118743900004, 0.62737894058200006, 0.62798690795900003, 0.62775301933299998, 0.627470970154]}], ["fannkuch", "RawResult", {"changed_times": [0.43332195282000002, 0.38041710853600003, 0.37921309471100001, 0.37792515754700001, 0.37811303138699998, 0.37825512886000001, 0.377966880798, 0.378439188004, 0.37887310981799999, 0.37781596183799998, 0.37887811660800003, 0.37811684608500001, 0.37786602973900002, 0.37938904762300002, 0.377534151077, 0.38209295272799998, 0.376986980438, 0.377496004105, 0.37841606140099998, 0.37773180007899998, 0.37749290466300001, 0.37853693962099999, 0.37957501411400002, 0.37821793556200001, 0.37789702415499998, 0.37755322456399998, 0.37845897674599999, 0.37784194946299998, 0.377678155899, 0.37787008285500001, 0.37771797180200001, 0.378336906433, 0.37751984596299998, 0.377940893173, 0.38246393203700002, 0.37781810760500001, 0.37745118141200001, 0.37853002548199999, 0.37841701507600001, 0.378228902817, 0.37974500656100002, 0.37745285034199999, 0.37854695320100001, 0.377611875534, 0.37745094299300003, 0.37820911407500002, 0.378015041351, 0.37732100486800002, 0.37784695625300002, 0.37757682800300002], "base_times": [1.3584520816800001, 1.3565759658800001, 1.3554861545600001, 1.3549780845599999, 1.3569629192399999, 1.35226488113, 1.35883998871, 1.35334992409, 1.3553960323300001, 1.35195589066, 1.3530910015099999, 1.35774278641, 1.3555450439500001, 1.35708403587, 1.3534219265, 1.3573589324999999, 1.3554661273999999, 1.35724401474, 1.3552920818300001, 1.35891509056, 1.35526394844, 1.3548588752699999, 1.3580558300000001, 1.3540120124799999, 1.3573079109199999, 1.3565089702599999, 1.3589961528800001, 1.3558809757200001, 1.35862016678, 1.3561370372799999, 1.3535299301099999, 1.35631299019, 1.35327315331, 1.3568890094799999, 1.3567941188799999, 1.35737490654, 1.35441994667, 1.35872101784, 1.35426902771, 1.3580150604200001, 1.35494399071, 1.3543949127199999, 1.3575069904299999, 1.3546030521400001, 1.3580288887, 1.3550050258599999, 1.35805010796, 1.35595703125, 1.3576319217699999, 1.3546538352999999]}], ["go", "RawResult", {"changed_times": [1.14897704124, 1.2345161438000001, 1.1498041153, 1.29407215118, 1.08174014091, 1.20440006256, 1.1048231124900001, 1.2440271377600001, 1.1091661453199999, 1.04328203201, 1.06029200554, 1.2512080669400001, 1.0826690197, 1.09385991096, 1.1346490383100001, 1.13187813759, 1.0736310482, 1.0886130332899999, 1.04829597473, 1.0568811893500001, 1.0900421142600001, 1.12933301926, 1.0717599391899999, 1.05894494057, 1.03591299057, 1.0868139267000001, 1.04581999779, 1.07085800171, 1.0427441597, 1.1460630893699999, 1.0693988800000001, 1.07336401939, 1.0726578235599999, 1.0731029510500001, 1.0713851451900001, 1.03998804092, 1.0435359477999999, 1.1242880821200001, 1.05398511887, 1.02363014221, 1.04768109322, 1.02518391609, 1.0290699005099999, 1.0308260917700001, 1.02171015739, 1.1216301918, 1.0642161369300001, 1.1219260692599999, 1.04517388344, 1.0866401195499999], "base_times": [0.59255790710400003, 0.599092006683, 0.59348702430699996, 0.59416389465300001, 0.59464716911299997, 0.59416198730500003, 0.59368491172799998, 0.59720110893199996, 0.59490489959699999, 0.59492588043200001, 0.59482812881500002, 0.59362506866499998, 0.59689593315099998, 0.59788894653299995, 0.59597611427300001, 0.59460496902500004, 0.59366416931199995, 0.59722304344199995, 0.59511184692399999, 0.59583401680000003, 0.59516787529000004, 0.59308505058299998, 0.59634900093099996, 0.59311318397500001, 0.59705591201800001, 0.595954179764, 0.59271693229699995, 0.59645009040800001, 0.59529709815999998, 0.59400391578699996, 0.59524512290999998, 0.59631705284100001, 0.59657907485999995, 0.59425806999200004, 0.59539294242899998, 0.59478211402900005, 0.59715604782099996, 0.59750199317899999, 0.596050977707, 0.59527707099899996, 0.59443402290299996, 0.59375095367399999, 0.59820485115099997, 0.59519815444900004, 0.59441995620700006, 0.59474897384600001, 0.59502887725800002, 0.59912514686600005, 0.59507703781099996, 0.59598803520200005]}], ["html5lib", "RawResult", {"changed_times": [23.0328969955, 17.374573946000002, 16.435561180099999, 15.829866170900001, 15.6528949738], "base_times": [8.6721470356000001, 8.7515590190899992, 8.73878312111, 8.7048590183299996, 8.6830830573999993]}], ["meteor-contest", "RawResult", {"changed_times": [0.35940599441499999, 0.29690194129899999, 0.29317784309400002, 0.296035051346, 0.29229211807299998, 0.29228496551499999, 0.31367707252499999, 0.291788816452, 0.29139184951800001, 0.29157185554499998, 0.29212999343899998, 0.291711091995, 0.30185389518700001, 0.29527878761300003, 0.290993928909, 0.29129290580700001, 0.29075789451599998, 0.29058814048800002, 0.29080295562699998, 0.290637969971, 0.29061007499699998, 0.29125404357899998, 0.29032993316700001, 0.29056715965300001, 0.29158401489300001, 0.29034614563, 0.29054307937599999, 0.29032897949199998, 0.29048895835900002, 0.290990114212, 0.30065107345600001, 0.29087710380600001, 0.29015898704499998, 0.29014682769799999, 0.29019498825099999, 0.29038119316099997, 0.291100978851, 0.290424108505, 0.29048800468399999, 0.29063892364499999, 0.29032087326, 0.29016399383500002, 0.29016995429999998, 0.29066991805999998, 0.29026079177899999, 0.29024004936199999, 0.29001903533899998, 0.29036879539499999, 0.29006004333500002, 0.29905819892899999], "base_times": [0.24460601806599999, 0.243553876877, 0.24350500106799999, 0.24308800697300001, 0.24368906021100001, 0.24375700950599999, 0.24353194236799999, 0.24331998825100001, 0.24436902999900001, 0.243896007538, 0.243317127228, 0.24335289001499999, 0.24417495727499999, 0.243915081024, 0.24382185936, 0.24391078949, 0.24390697479199999, 0.244327068329, 0.243896007538, 0.243628978729, 0.24348807334899999, 0.24388504028300001, 0.243416070938, 0.24379396438600001, 0.243729114532, 0.244396924973, 0.243938922882, 0.243566989899, 0.243503093719, 0.243898153305, 0.24379396438600001, 0.24365305900600001, 0.24366998672500001, 0.244452953339, 0.24378490448000001, 0.243889093399, 0.243768930435, 0.24420404434199999, 0.243615865707, 0.24389505386400001, 0.243630170822, 0.244421958923, 0.244014024734, 0.24404406547499999, 0.243921995163, 0.24391913413999999, 0.24434804916399999, 0.24426007270799999, 0.243828058243, 0.24390697479199999]}], ["nbody_modified", "RawResult", {"changed_times": [0.115092039108, 0.087569952011100002, 0.084489107132000002, 0.084752082824699995, 0.084683895111099999, 0.0844850540161, 0.084747791290299995, 0.084717988967900004, 0.0844769477844, 0.084722042083700003, 0.085180044174199998, 0.084502935409500005, 0.084743022918700006, 0.084678888320899995, 0.084486961364700006, 0.084739923477199997, 0.084715127944900007, 0.084480047225999999, 0.084748029708899997, 0.084702014923100005, 0.084467172622699996, 0.085448026657099996, 0.084717988967900004, 0.0844888687134, 0.084763050079300004, 0.084743976592999995, 0.084486961364700006, 0.0847291946411, 0.084710836410500007, 0.084522008895900003, 0.084717035293600002, 0.084691047668499997, 0.084509134292599999, 0.085023880004899993, 0.084743976592999995, 0.084515094757100007, 0.0847249031067, 0.084691047668499997, 0.084517955780000001, 0.084717988967900004, 0.084722995758099995, 0.084486007690400003, 0.084743022918700006, 0.084718227386500006, 0.084471940994299999, 0.085363149642899996, 0.084741115570100001, 0.0845909118652, 0.084722995758099995, 0.086489915847799997], "base_times": [0.40417194366499998, 0.40459799766499999, 0.40421700477599998, 0.40403985977200002, 0.40431380271900003, 0.40496802329999998, 0.40494704246500002, 0.40464997291600002, 0.40360713005100002, 0.40396285057100001, 0.403512001038, 0.403342008591, 0.403450012207, 0.40323090553300001, 0.40359306335400003, 0.40368413925199997, 0.40387392044100001, 0.403959989548, 0.40371799469000003, 0.40425086021399997, 0.40426588058500001, 0.40466904640200002, 0.40393304824800003, 0.40405297279399999, 0.40509009361300002, 0.40615010261500001, 0.40549302101099999, 0.405642986298, 0.40584611892700001, 0.40618610382100001, 0.40610885620100001, 0.40699696540800001, 0.40708398819000002, 0.406311988831, 0.40554094314599998, 0.40626502037000001, 0.40597796440099998, 0.40627288818399998, 0.40671586990399999, 0.40668511390700002, 0.40602588653600002, 0.40550208091700002, 0.40621900558500001, 0.40642905235299998, 0.40591812133799998, 0.40632581710799998, 0.40576291084299998, 0.40567016601599998, 0.40628194809000001, 0.40637803077700002]}], ["pyflate-fast", "RawResult", {"changed_times": [1.8020980358100001, 1.84628176689, 1.78963685036, 1.8260049819899999, 1.8187170028699999, 1.82588410378, 1.78930997849, 1.8257989883400001, 1.82189011574, 1.8041939735400001, 1.81843900681, 1.7986948490100001, 1.8285381794, 1.79674696922, 1.8408238887799999, 1.80955219269, 1.823802948, 1.80054306984, 1.8244709968599999, 1.82194590569, 1.8056621551500001, 1.8262178897900001, 1.7976200580599999, 1.8334770202599999, 1.81172394753, 1.8108351230599999, 1.83649802208, 1.82515788078, 1.80139780045, 1.8085131645200001, 1.81924104691, 1.83042001724, 1.8213908672300001, 1.8187279701200001, 1.81911611557, 1.8179550170900001, 1.8195929527300001, 1.8187839984900001, 1.8185229301500001, 1.85378098488, 1.79992103577, 1.8257040977500001, 1.81010389328, 1.8136050701099999, 1.83390116692, 1.8319008350399999, 1.8119609355899999, 1.81212496758, 1.8113842010500001, 1.8158957958199999], "base_times": [1.9715361595200001, 1.97649312019, 1.9744579792000001, 1.97769284248, 1.9740889072400001, 1.9755351543399999, 1.9799709320100001, 1.97855901718, 1.9788129329699999, 1.98169994354, 1.9795730114000001, 1.9760389328000001, 1.9755880832699999, 1.9830451011700001, 1.97887706757, 1.9769110679599999, 1.9802250862099999, 1.9751098156, 1.97200202942, 1.9791991710700001, 1.9742319583900001, 1.97487902641, 1.98901081085, 1.9792599678, 1.97859692574, 1.97864222527, 1.9701180458100001, 1.9805810451500001, 1.97375798225, 1.97326683998, 1.9741070270500001, 1.9746761321999999, 1.9728059768699999, 1.9711620807600001, 1.97109913826, 1.97114300728, 1.98034501076, 1.9756407737699999, 1.97721195221, 1.97855806351, 1.97502398491, 1.97638607025, 1.9746539592700001, 1.9767880439800001, 1.97603201866, 1.97480201721, 1.9750320911399999, 1.97502207756, 1.9728920459699999, 1.9767138957999999]}], ["raytrace-simple", "RawResult", {"changed_times": [1.1192121505699999, 1.12430500984, 1.14396595955, 1.1181769371000001, 1.12474393845, 1.1357970237699999, 1.1164288520800001, 1.1184740066500001, 1.1876978874199999, 1.12074708939, 1.1192080974600001, 1.11397886276, 1.11682295799, 1.1244900226600001, 1.1138560771899999, 1.1160299778, 1.1417291164400001, 1.11880779266, 1.11914396286, 1.11626505852, 1.24021792412, 1.11648702621, 1.1162600517300001, 1.11813688278, 1.1161379814100001, 1.1181910038, 1.1156840324399999, 1.1143019199399999, 1.1215739250200001, 1.23314094543, 1.1156570911399999, 1.11691904068, 1.1153750419599999, 1.1148521900199999, 1.11209821701, 1.1128268241899999, 1.11406207085, 1.1566548347500001, 1.1125919818900001, 1.1168460845899999, 1.11554002762, 1.1169860362999999, 1.11424398422, 1.11369895935, 1.12529706955, 1.16200399399, 1.11288189888, 1.1190280914299999, 1.1127638816800001, 1.1173341274299999], "base_times": [1.6308507919299999, 1.6061539650000001, 1.6082999706300001, 1.60679507256, 1.6074709892300001, 1.6028199195899999, 1.60836696625, 1.60565710068, 1.60432505608, 1.60986208916, 1.63934397697, 1.6348330974600001, 1.60514497757, 1.60835003853, 1.60388612747, 1.6093668937700001, 1.60202693939, 1.6092281341600001, 1.6030900478400001, 1.62427687645, 1.6077330112499999, 1.6068379879000001, 1.6104369163500001, 1.6036658287000001, 1.60728192329, 1.60446190834, 1.64275884628, 1.6381349563600001, 1.60937809944, 1.60457992554, 1.6079080104800001, 1.6385262012499999, 1.6040019989000001, 1.6066889762900001, 1.6061730384799999, 1.6106119155900001, 1.6066889762900001, 1.6068558692899999, 1.6067490577700001, 1.60826301575, 1.6016521453899999, 1.6092960834500001, 1.6044838428499999, 1.6094570159899999, 1.60569787025, 1.60940694809, 1.61079907417, 1.60953187943, 1.6040878295900001, 1.60577011108]}], ["richards", "RawResult", {"changed_times": [0.188634872437, 0.17832899093599999, 0.17431497573900001, 0.172646045685, 0.17323303222700001, 0.17191815376299999, 0.17156910896300001, 0.171721935272, 0.17107295989999999, 0.17158412933299999, 0.171247959137, 0.17127990722700001, 0.172159910202, 0.171751022339, 0.17034697532699999, 0.1701130867, 0.18102502822899999, 0.21005392074599999, 0.17098188400299999, 0.173319816589, 0.16994905471800001, 0.169959068298, 0.169358968735, 0.17905807495100001, 0.169506072998, 0.16935610771199999, 0.168767929077, 0.16901707649200001, 0.168606996536, 0.16865921020499999, 0.16839098930400001, 0.168842077255, 0.168585777283, 0.168391942978, 0.168664216995, 0.16862487793, 0.17172694206200001, 0.16935086250299999, 0.16864490508999999, 0.16911721229599999, 0.16891694068900001, 0.168562889099, 0.168758869171, 0.16852879524200001, 0.16824889183, 0.16894888877899999, 0.16881513595600001, 0.184386014938, 0.168937921524, 0.168543815613], "base_times": [0.21284794807400001, 0.21313500404399999, 0.21374988555900001, 0.21324205398599999, 0.21453785896300001, 0.21360492706299999, 0.21391510963400001, 0.213798999786, 0.213606119156, 0.21400713920600001, 0.21364998817399999, 0.21356511115999999, 0.214838027954, 0.21462011337299999, 0.21340608596800001, 0.212734937668, 0.21283102035500001, 0.21362400054899999, 0.214515924454, 0.21316099166899999, 0.21444606781, 0.21465396881099999, 0.21373987197899999, 0.213769197464, 0.213486909866, 0.213093996048, 0.21455001831100001, 0.21463489532499999, 0.213505029678, 0.21406984329199999, 0.215165138245, 0.215652942657, 0.21378803253199999, 0.21330499649000001, 0.21377801895099999, 0.214590072632, 0.214936971664, 0.21295380592300001, 0.212911844254, 0.213427066803, 0.21461415290800001, 0.21314406394999999, 0.213227987289, 0.21308803558299999, 0.21376299858100001, 0.21473717689499999, 0.213126897812, 0.212868928909, 0.21446895599400001, 0.21520400047300001]}], ["spambayes", "RawResult", {"changed_times": [0.35268497466999998, 0.48473596572900002, 0.48003888130200001, 0.54478287696799998, 0.58626985549900001, 0.49527096748400001, 0.45211386680600002, 0.43088984489400001, 0.30615997314499999, 0.54539084434499996, 0.397897958755, 0.34469389915499998, 0.34239602088900001, 0.322602987289, 0.68369793891899999, 0.41977381706200001, 0.42178511619600001, 0.33393692970299999, 0.383501052856, 0.358753919601, 0.30524086952200002, 0.38231086731000002, 0.31400895118700001, 0.34354496002200002, 0.30663800239599998, 0.31292319297799998, 0.38818907737699998, 0.30091691017200001, 0.39267206191999998, 0.29696297645600001, 0.372807025909, 0.456650972366, 0.29690003395100001, 0.34635186195399997, 0.29959511756899998, 0.34087109565700002, 0.37458395957899998, 0.306196928024, 0.29220890998799998, 0.35120987892200001, 0.31294512748699999, 0.29141306877099998, 0.31265306472799997, 0.34791302681000003, 0.29072403907799999, 0.28969788551300002, 0.29165005683900003, 0.34921717643700001, 0.38425016403200002, 0.28991580009500001], "base_times": [0.192424058914, 0.192607879639, 0.19253897666899999, 0.19256591796899999, 0.19250607490499999, 0.19241905212400001, 0.19255399704000001, 0.192532062531, 0.19254207611099999, 0.19256997108500001, 0.19251918792700001, 0.19549083709699999, 0.19256782531700001, 0.192401885986, 0.19272398948700001, 0.19264316558799999, 0.19307994842500001, 0.192593097687, 0.19265103340100001, 0.192397117615, 0.192836046219, 0.192632913589, 0.19270300865199999, 0.192532062531, 0.19256210327100001, 0.19262599945100001, 0.19496798515300001, 0.19257903099099999, 0.19252800941500001, 0.19245505332900001, 0.19250011444099999, 0.192535877228, 0.192492008209, 0.192594051361, 0.19248485565199999, 0.192499876022, 0.192586183548, 0.19254899025, 0.192557096481, 0.19252395629899999, 0.192557096481, 0.19257092475900001, 0.195178985596, 0.19255685806299999, 0.192538022995, 0.192622184753, 0.19253182411200001, 0.19251084327699999, 0.19250798225400001, 0.19247317314099999]}], ["spectral-norm", "RawResult", {"changed_times": [0.28835701942399999, 0.227369070053, 0.226712942123, 0.225772857666, 0.224092006683, 0.223258972168, 0.223456859589, 0.22423720359800001, 0.22428393363999999, 0.22225785255399999, 0.222506999969, 0.22108101844799999, 0.221049070358, 0.22128820419299999, 0.221066951752, 0.22229719162, 0.22106409072899999, 0.221334934235, 0.22109699249299999, 0.22112703323399999, 0.22133898734999999, 0.221033096313, 0.221338033676, 0.221124172211, 0.22107505798300001, 0.22130012512200001, 0.22103691101100001, 0.221324920654, 0.221083879471, 0.22106719017000001, 0.22131204605099999, 0.221076965332, 0.22137594223000001, 0.22109699249299999, 0.22108101844799999, 0.22130608558699999, 0.22109103202800001, 0.22142004966699999, 0.22129201889, 0.22182607650799999, 0.22131991386399999, 0.221093893051, 0.22135591506999999, 0.22108507156400001, 0.22105407714799999, 0.22134613990800001, 0.22112011909500001, 0.221364021301, 0.221080064774, 0.221827983856], "base_times": [0.38507890701300002, 0.38513684272799997, 0.38540577888499999, 0.385334968567, 0.38533091545100001, 0.38737297058100001, 0.38601493835400003, 0.38538408279399999, 0.38633489608799998, 0.38616895675700003, 0.38576793670699999, 0.38699293136599999, 0.38566589355499997, 0.38517880439800001, 0.38546013832100001, 0.385420084, 0.38523602485699998, 0.38521504402200002, 0.38548302650499999, 0.385416030884, 0.38633704185500001, 0.38570713996900002, 0.38650393486000001, 0.38614487648000001, 0.385656833649, 0.38680696487400001, 0.38568186759899997, 0.38566184043899998, 0.38578081131000003, 0.38579583167999998, 0.38546395301800002, 0.38546991348300003, 0.38550615310699998, 0.38638687133799998, 0.386216163635, 0.38625407218899999, 0.38584995269799999, 0.38524508476300001, 0.385580062866, 0.38524007797199999, 0.385420084, 0.38597679138199997, 0.38613581657399998, 0.385598182678, 0.38566184043899998, 0.38541483879100003, 0.38540506362900001, 0.38542604446399997, 0.385876178741, 0.38554000854499998]}], ["telco", "RawResult", {"changed_times": [0.69999999999999996, 0.64000000000000001, 0.64000000000000001, 0.66000000000000003, 0.63, 0.65000000000000002, 0.63, 0.62, 0.62, 0.63, 0.63, 0.62, 0.63, 0.62, 0.63, 0.63, 0.63, 0.63, 0.62, 0.62, 0.62, 0.63, 0.63, 0.64000000000000001, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62, 0.63, 0.63, 0.62, 0.62, 0.62, 0.63, 0.62, 0.62, 0.62, 0.63, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62, 0.62], "base_times": [0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004]}], ["twisted_names", "RawResult", {"changed_times": [0.0089047195013357075, 0.0088105726872246704, 0.0087873462214411256, 0.0088809946714031966, 0.0088339222614840993, 0.0088417329796640146, 0.0089928057553956831, 0.009442870632672332, 0.0092336103416435829, 0.0088028169014084511, 0.0088183421516754845, 0.0090991810737033659, 0.0088495575221238937, 0.0087642418930762491, 0.0087642418930762491, 0.0096061479346781949, 0.0088809946714031966, 0.0090826521344232521, 0.0088495575221238937, 0.0087336244541484712, 0.0087642418930762491, 0.0087642418930762491, 0.008771929824561403, 0.0087412587412587419, 0.0096153846153846159, 0.0087642418930762491, 0.008771929824561403, 0.0091743119266055051, 0.0087642418930762491, 0.0090252707581227436, 0.008771929824561403, 0.0087260034904013961, 0.0087489063867016627, 0.0092506938020351526, 0.008771929824561403, 0.0087412587412587419, 0.0087565674255691769, 0.0087412587412587419, 0.0087412587412587419, 0.0087642418930762491, 0.0087412587412587419, 0.0087183958151700082, 0.0087412587412587419, 0.0087183958151700082, 0.0087412587412587419, 0.0087412587412587419, 0.0087642418930762491, 0.0087260034904013961, 0.0095693779904306216, 0.0087642418930762491], "base_times": [0.0065832784726793945, 0.0065832784726793945, 0.006587615283267457, 0.0065703022339027592, 0.0065919578114700065, 0.0065789473684210523, 0.0065789473684210523, 0.0066093853271645738, 0.0066006600660066007, 0.0066137566137566134, 0.0065832784726793945, 0.0065919578114700065, 0.0065963060686015833, 0.0065919578114700065, 0.0066006600660066007, 0.0065789473684210523, 0.0065963060686015833, 0.006587615283267457, 0.006587615283267457, 0.0065789473684210523, 0.0066006600660066007, 0.0065963060686015833, 0.0065789473684210523, 0.0066050198150594455, 0.0066093853271645738, 0.0065832784726793945, 0.0066093853271645738, 0.0065832784726793945, 0.0065746219592373442, 0.0066137566137566134, 0.0065919578114700065, 0.0065789473684210523, 0.0066006600660066007, 0.0066050198150594455, 0.0065832784726793945, 0.0066093853271645738, 0.0065963060686015833, 0.0065832784726793945, 0.0065963060686015833, 0.0065746219592373442, 0.0066050198150594455, 0.0065963060686015833, 0.0066137566137566134, 0.006587615283267457, 0.0066050198150594455, 0.0066137566137566134, 0.006587615283267457, 0.0065963060686015833, 0.006587615283267457, 0.0065746219592373442]}]], "branch": "trunk", "revision": 0} \ No newline at end of file +{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [83.608081817599995, 81.512792825700004, 81.586477994899994, 81.442406892799994, 81.350951910000006, 81.291965007800002, 81.274972915600003, 81.285789966600007, 81.302440881699994, 81.471642970999994, 81.280892133699993, 81.281094074199999, 81.301096916199995, 81.287539005300005, 81.236584901800001, 81.283654928199994, 81.291748046899997, 81.5425839424, 81.236918926200005, 80.983955144899994, 80.997071981399998, 80.963902950299996, 80.967394113500006, 80.967602014500002, 81.2447729111, 81.042499065399994, 81.091964006400005, 81.300304889700001, 81.322211027099996, 81.287165164900003, 81.302358150499998, 81.531341075900002, 81.253934860200005, 81.308516979199993, 81.402919053999995, 81.258775949500006, 81.280184984200005, 81.299252033200005, 81.306777000400004, 81.550753116600006, 81.246633052799993, 81.279681921000005, 81.2769200802, 81.316480874999996, 81.294853925699996, 81.257145166399994, 81.467491149899999, 81.312170982400005, 81.252816200300003, 81.258814811700006], "base_times": [2.80382990837, 2.7956340313000001, 2.8010652065300001, 2.7976541519200002, 2.7950928211199999, 2.8041520118699998, 2.8043060302699998, 2.7999861240400001, 2.8013310432399998, 2.8088479042099999, 2.8007519245100001, 2.8009748458899999, 2.8043870925899999, 2.80448412895, 2.80184221268, 2.80495095253, 2.8023998737300002, 2.8056631088300001, 2.8033299446100002, 2.7996089458500002, 2.8002779483800002, 2.8084959983800002, 2.8017699718500002, 2.8082520961799999, 2.8050520420099998, 2.8028419017799999, 2.7981400489800001, 2.7999629974400002, 2.7998390197799998, 2.8000919818900001, 2.80037903786, 2.79657483101, 2.7981040477799999, 2.8026959896100001, 2.7995419502300001, 2.7954881191299998, 2.7970201969100001, 2.79839706421, 2.7952239513400001, 2.7976598739599998, 2.7945868968999998, 2.7983119487799999, 2.7953550815599999, 2.79377818108, 2.7956900596600001, 2.8036141395600001, 2.79533600807, 2.7930212020899998, 2.7950999736800002, 2.7983438968700001]}], ["django", "RawResult", {"changed_times": [0.94225811958299999, 0.68628406524700003, 0.70013499259900003, 0.68669486045799999, 0.70185899734500001, 0.68997812271100001, 0.70118880271899997, 0.68901705741899999, 0.70103502273600005, 0.68790817260699999, 0.69985699653599998, 0.68816995620700006, 0.73480010032700005, 0.68603014945999996, 0.70323491096500002, 0.68623781204199996, 0.70352911949200003, 0.68471789360000002, 0.70435285568200001, 0.68524813652000005, 0.70314097404499998, 0.68677711486799997, 0.70417714118999997, 0.68562698364300001, 0.70090913772600005, 0.68850708007799999, 0.70100784301800001, 0.68775796890300001, 0.70195102691699995, 0.68907117843599996, 0.70120120048500001, 0.68785810470599995, 0.70091485977199997, 0.68843507766699996, 0.70012712478600003, 0.68629503250099999, 0.70415806770300005, 0.68661308288599998, 0.70444512367199996, 0.68531179428099998, 0.70345211029099997, 0.68501901626600004, 0.704196929932, 0.68532204628000004, 0.70484805107100001, 0.68551492691000004, 0.70121288299600004, 0.68866515159599995, 0.70078611373900002, 0.68853020667999998], "base_times": [0.98770689964299996, 0.98978590965299995, 0.98754811286900002, 0.98878502845799998, 0.98909711837799996, 0.989701032639, 1.0125169754000001, 0.99142193794300004, 0.98840188980099997, 0.99162793159499996, 0.99071907997099995, 0.99076414108300004, 0.98944306373599999, 0.98936295509299998, 0.99107599258400003, 0.99007892608600001, 0.98786211013799996, 0.992697954178, 0.99010705947900002, 0.98694705963100005, 0.98762106895400004, 0.98965787887599999, 0.98832798004199995, 0.98824381828300001, 0.98842883110000002, 0.98763680458100001, 0.99011301994300005, 0.98688387870799998, 0.98969793319699995, 0.98943901061999995, 0.98809719085699999, 0.98793482780499997, 0.98821902275100004, 0.98756289482100001, 0.99030303955099996, 0.98743104934699999, 0.98614192009000001, 0.98893618583700005, 0.98977398872400002, 0.98548102378799995, 0.987926959991, 0.98703289031999997, 0.98901891708400003, 0.98792290687600004, 0.98637413978599997, 0.98895907402000005, 0.98991489410400002, 0.98569011688200003, 0.98801302909900002, 0.99004602432300004]}], ["fannkuch", "RawResult", {"changed_times": [0.65027499198899996, 0.57184004783599995, 0.56566786766099997, 0.56521892547599994, 0.56701707840000004, 0.57264208793600002, 0.56822896003699996, 0.56470012664799996, 0.56810212135299998, 0.56781196594200001, 0.564800024033, 0.56483197212199998, 0.56917119026200003, 0.56767296791099997, 0.56695199012800002, 0.56578302383400003, 0.56515192985499996, 0.57563495635999995, 0.56515789031999997, 0.56409192085299997, 0.56773018837, 0.56481719016999998, 0.56611680984500001, 0.56627202034000002, 0.56611609458900003, 0.56967306137100004, 0.567966938019, 0.56780695915199997, 0.57448220253000004, 0.57842707634000001, 0.56447005271899997, 0.56811785697899997, 0.56486487388600004, 0.56649589538599998, 0.56501197814899995, 0.56486201286299997, 0.56838393211399996, 0.56405186653100003, 0.56555008888199998, 0.56801414489699997, 0.56562018394500002, 0.56669306755100002, 0.56592893600500005, 0.56481790542599997, 0.56828188896200005, 0.56566095352199997, 0.56510496139499999, 0.56719899177599997, 0.56526398658800003, 0.56593322753899999], "base_times": [1.84706997871, 1.8756129741700001, 1.85868191719, 1.8472619056699999, 1.87546992302, 1.8508591651899999, 1.8696298599200001, 1.86460900307, 1.85237002373, 1.87226009369, 1.8722259998299999, 1.8783550262499999, 1.8736140727999999, 1.87361216545, 1.87663817406, 1.8724210262300001, 1.8727889061, 1.8782889842999999, 1.8723690509799999, 1.8713910579699999, 1.8771908283200001, 1.87365293503, 1.87231016159, 1.87713694572, 1.87258410454, 1.87410783768, 1.8863110542299999, 1.87264585495, 1.87151694298, 1.8771209716799999, 1.8736791610700001, 1.8717048168199999, 1.8770380020099999, 1.8727111816399999, 1.87127304077, 1.87773990631, 1.89574599266, 1.87074995041, 1.92922902107, 1.9290540218400001, 1.94486498833, 1.8772900104500001, 1.87182497978, 1.87466216087, 1.8774259090400001, 1.87289023399, 1.87402200699, 1.8763341903699999, 1.8749959468799999, 1.8724749088299999]}], ["go", "RawResult", {"changed_times": [2.3995778560600001, 2.3866791725200001, 2.3967480659499998, 2.5598640441899998, 2.2185318469999999, 2.3486278057100001, 2.3838458061200001, 2.4448201656299999, 2.19373893738, 2.1973280906700001, 2.1919651031499998, 2.5130779743199998, 2.2208979129799999, 2.2443778514899999, 2.3977251052900002, 2.1551439762100002, 2.1595408916499998, 2.23243093491, 2.17227602005, 2.1977610588099998, 2.3166739940599999, 2.2348148822799998, 2.1878528595, 2.1694250106799999, 2.1508538722999999, 2.3760650157900001, 2.1592259406999998, 2.1955871581999999, 2.1416928768200001, 2.2533910274500002, 2.1912820339199999, 2.2604429721799999, 2.2298648357399999, 2.1796870231600001, 2.20620703697, 2.16007018089, 2.1497859954799998, 2.2840979099299998, 2.2118680477099999, 2.1375119686100001, 2.15009498596, 2.1446928978000002, 2.1316239833799999, 2.13385415077, 2.1266629695899999, 2.3893008232100001, 2.1834890842400001, 2.1347019672399998, 2.21711921692, 2.1232719421400001], "base_times": [0.93096089363099999, 0.95089387893699995, 0.93110585212700003, 0.92558383941699995, 0.93034386634800004, 0.92623090744000003, 0.93461894989000005, 0.93881106376599999, 0.92857980728099998, 0.93526601791399999, 0.92670106887799997, 0.93337702751200002, 0.93183684349100004, 0.93682098388699997, 0.92921209335300003, 0.93008303642300005, 0.92924404144299999, 0.93299007415799995, 0.93573307990999999, 0.930922985077, 0.93020296096800004, 0.93374800682100001, 0.93536710739100004, 0.93496298789999999, 0.93623590469399998, 0.93143200874300003, 0.92915606498699999, 0.92980098724399995, 0.94962000846899997, 0.93416810035700004, 0.93491911888099999, 0.92967605590799995, 0.927985906601, 0.934444904327, 0.93304419517500004, 0.93216395378100003, 0.93552398681600002, 0.94005179405200001, 0.93092107772800003, 0.93850493431100002, 0.932507038116, 0.93334603309599995, 0.94100594520600001, 0.93229484558099995, 0.93091201782199995, 0.93197703361499995, 0.938956975937, 0.93856000900299996, 0.93897986412000001, 0.92620086669900004]}], ["html5lib", "RawResult", {"changed_times": [38.3545889854, 29.4800949097, 28.267448186900001, 27.7905261517, 27.673813104600001, 27.295396089600001, 27.640147924400001, 27.573884964000001, 27.469006061599998, 27.410823106799999, 27.163343191100001, 27.421272039400002, 27.498446941400001, 27.377060890199999, 27.862658977500001, 27.3771958351, 27.126015901599999, 27.002305030799999, 27.370562076599999, 27.4734969139, 27.1270251274, 27.124243021000002, 27.327117919900001, 27.165127992599999, 27.1296830177, 27.4667949677, 27.082341194200001, 27.294297933599999, 27.245062112799999, 27.360240936299999, 27.298542022700001, 27.139825105700002, 27.091902017599999, 27.068879127500001, 27.3232738972, 27.168828010599999, 27.2176198959, 27.3889100552, 27.146837949799998, 27.0338580608, 27.022614002200001, 27.345255851699999, 27.146274805099999, 27.0234370232, 27.357608795200001, 27.049774169900001, 27.077157974199999, 27.123055934900002, 27.078799009299999, 27.409001111999999], "base_times": [13.864197969399999, 14.0840089321, 14.0057208538, 14.0325820446, 14.0150129795, 14.1008889675, 14.097244978000001, 14.067807912799999, 14.0274028778, 14.012732029, 13.961874008200001, 14.1108551025, 14.0863909721, 14.0686590672, 14.004349947, 14.185241937600001, 13.9961509705, 14.1126899719, 14.1245219707, 14.086353063600001, 14.048186063799999, 14.0126638412, 13.965003013600001, 14.136936903, 14.125771045700001, 14.0789718628, 14.0491027832, 13.986720800400001, 13.978865861899999, 14.133366823199999, 14.0888400078, 14.118774890899999, 14.0262331963, 14.035698890700001, 13.984946966200001, 14.1640629768, 14.148418188100001, 14.091625928899999, 14.075366020200001, 14.0536768436, 13.9736099243, 14.1238529682, 14.118108987799999, 14.103012085, 14.082417964899999, 14.0419850349, 14.015744209299999, 13.983751058599999, 14.112045049700001, 14.1095650196]}], ["meteor-contest", "RawResult", {"changed_times": [0.54956293106099996, 0.44127202034000002, 0.43488597869899998, 0.43912196159400002, 0.43368196487400001, 0.43549299240099998, 0.46642994880700001, 0.43648004531899998, 0.43979382514999998, 0.43357920646699999, 0.43448400497400003, 0.43302893638599999, 0.44884395599400001, 0.43243002891499999, 0.43497920036299997, 0.43333315849300003, 0.43120193481399999, 0.44183921813999999, 0.43201804161099999, 0.43070816993700001, 0.43093800544700001, 0.43396782875099998, 0.431432962418, 0.43151283264200002, 0.432732105255, 0.43281292915300001, 0.43130493164099998, 0.43159103393600001, 0.442057132721, 0.43084406852700002, 0.44665694236800002, 0.43070578575099999, 0.43876791000400001, 0.42976713180499998, 0.43031406402599998, 0.43287897110000001, 0.43102192878700002, 0.43873286247299997, 0.42987489700300002, 0.43340301513700003, 0.42970800399800002, 0.43067002296399998, 0.43263816833500002, 0.43107914924599999, 0.42996597289999999, 0.429845809937, 0.433667898178, 0.43897414207500002, 0.43040704727200002, 0.43301296234100001], "base_times": [0.34740400314300002, 0.34649682045000002, 0.34654498100300002, 0.34828615188599998, 0.34688496589700002, 0.34619617462199997, 0.348422050476, 0.35294890403700002, 0.346195220947, 0.34620308876, 0.34614801406899998, 0.34765100479099997, 0.34694099426300001, 0.34586310386699998, 0.34622812271100001, 0.34607410430899999, 0.348037004471, 0.34679603576700002, 0.34619593620299999, 0.34622883796699999, 0.34815096855200001, 0.34621715545699999, 0.347053050995, 0.34751987457299999, 0.34610795974699998, 0.34874510765099997, 0.34702110290499999, 0.35155105590800001, 0.34627103805499998, 0.34825491905200001, 0.34611105918899998, 0.34621214866599997, 0.34627509117100003, 0.34635615348799997, 0.34847092628499998, 0.34634494781500003, 0.34610104560900001, 0.34599280357399997, 0.34852099418600002, 0.34607791900599999, 0.34652614593499997, 0.34637212753300001, 0.348030090332, 0.346480131149, 0.34624314308199999, 0.346033096313, 0.346461057663, 0.348530054092, 0.34677815437300002, 0.34670209884600001]}], ["nbody_modified", "RawResult", {"changed_times": [0.178552150726, 0.14297604560900001, 0.135512828827, 0.13514518737799999, 0.13505196571399999, 0.13496088981599999, 0.135854959488, 0.13503122329700001, 0.13507986068700001, 0.13504290580700001, 0.13506603241000001, 0.13503408432, 0.13780403137200001, 0.13503003120400001, 0.13493585586500001, 0.13506293296800001, 0.135177135468, 0.13551998138400001, 0.134822845459, 0.13526797294599999, 0.134998083115, 0.13494396209699999, 0.13502788543700001, 0.13709807395900001, 0.135847091675, 0.135095119476, 0.13512802124000001, 0.13511490821800001, 0.13506793975799999, 0.135057926178, 0.135121107101, 0.135377883911, 0.13559198379500001, 0.13520693779000001, 0.13507509231600001, 0.138010978699, 0.13512301445, 0.13525795936599999, 0.13526701927199999, 0.135272979736, 0.13510107994100001, 0.13522982597399999, 0.13502717018099999, 0.13525414466899999, 0.135074853897, 0.13518810272199999, 0.13801503181499999, 0.13605713844299999, 0.134965896606, 0.13791918754599999], "base_times": [0.60831594467200001, 0.60184097290000005, 0.60403108596800004, 0.604898929596, 0.59930682182299999, 0.59702301025399995, 0.60113716125500005, 0.59830880165099998, 0.60461401939399995, 0.59961390495300004, 0.60151195526099999, 0.60288000106799999, 0.60938596725500005, 0.60743498802200002, 0.600023984909, 0.60061693191499999, 0.60342597961400002, 0.59932994842499998, 0.60085797309900002, 0.60232901573200004, 0.59961009025599998, 0.65427803993199996, 0.60101699829099997, 0.60087990760800003, 0.60391497612, 0.60592889785799997, 0.60827589034999996, 0.60652494430500004, 0.60713696479799995, 0.61633515358000002, 0.60946607589699997, 0.61303901672399996, 0.61002016067499998, 0.61310410499599999, 0.61289215087899995, 0.61635208129899999, 0.62035799026500005, 0.61284279823300003, 0.61309885978699996, 0.61907792091400005, 0.61475586891199996, 0.61362600326500005, 0.61063408851599998, 0.61360001564, 0.61051511764499999, 0.61368680000300002, 0.60794496536300002, 0.61147809028599998, 0.61695909500099999, 0.61682605743399999]}], ["pyflate-fast", "RawResult", {"changed_times": [3.3121449947400001, 3.32391285896, 3.3113570213300001, 3.3125739097600002, 3.3533909320799999, 3.27320981026, 3.3294079303699999, 3.3954648971600001, 3.36747694016, 3.3318870067600002, 3.3314881324800001, 3.29482102394, 3.2987878322599999, 3.3235919475600002, 3.2715950012200001, 3.2692580223099998, 3.3049309253699999, 3.2853808403000002, 3.2795870304100001, 3.2983829975100001, 3.25447106361, 3.3078680038499999, 3.2952280044600002, 3.27434897423, 3.2544469833399998, 3.3341419696800001, 3.2960839271500002, 3.2732009887700002, 3.2589120864900001, 3.3169569969200001, 3.2971358299300002, 3.2613229751600001, 3.3093330860100001, 3.25996899605, 3.3377301692999999, 3.2606930732700001, 3.2845060825300001, 3.2952270507799999, 3.2635328769699998, 3.2907948494000001, 3.2753908634200002, 3.29530191422, 3.2842710018200001, 3.2528989314999999, 3.2978489399000002, 3.3096721172299999, 3.2890150547000001, 3.2512168884300001, 3.3030450344100002, 3.2920761108400001], "base_times": [3.18242001534, 3.1863389015200001, 3.1912939548499999, 3.1865100860600002, 3.1824879646299999, 3.19247794151, 3.1979899406399999, 3.1937181949600002, 3.1884422302200002, 3.20042586327, 3.18729782104, 3.1916460990900002, 3.1865410804700001, 3.1863729953800002, 3.1931619644200002, 3.1834938526199998, 3.2051031589500001, 3.1961328983300001, 3.19024610519, 3.1944649219499999, 3.19651508331, 3.2056679725600001, 3.1969439983400001, 3.1953899860399999, 3.1878740787500002, 3.1908388137800001, 3.1916890144300001, 3.1885077953300001, 3.1937019824999999, 3.1973328590399999, 3.1935818195299999, 3.1918790340399998, 3.18938112259, 3.1975688934300002, 3.1952290534999999, 3.1982200145699999, 3.18593597412, 3.1900849342300002, 3.20125794411, 3.1916630268100001, 3.2036459445999999, 3.20056295395, 3.1956481933599998, 3.1894419193300001, 3.2185740470900002, 3.1986999511700001, 3.1960220336899998, 3.1938240528100001, 3.1854040622699999, 3.1919748783099999]}], ["raytrace-simple", "RawResult", {"changed_times": [2.7798359394099998, 2.7791428566, 2.7640888690900001, 2.76987791061, 2.8119649887099998, 2.7948009967799998, 2.8166298866299999, 2.7788488864900001, 2.8109869957, 2.7679738998399999, 2.8182439804100001, 2.8162288665799999, 2.8095500469200001, 2.7651660442399999, 2.76630306244, 2.81266593933, 2.80415391922, 2.76372790337, 2.7615659236900001, 2.7600309848800002, 2.9178488254500001, 2.76185297966, 2.7615509033199999, 2.7574489116700001, 2.7583611011500002, 2.7616539001499998, 2.8071138858800002, 2.7628870010400002, 2.7632458209999999, 2.9184291362799999, 2.7783088684099999, 2.7595660686499999, 2.7599799633000002, 2.7592060565900001, 2.7621810436200001, 2.8037729263300002, 2.7557120323199999, 2.75506186485, 2.7569091320000001, 2.7592680454299998, 2.7995710372899998, 2.7594900131200002, 2.7773869037600001, 2.7546079158799999, 2.7564780712100001, 2.8319108486200002, 2.75832509995, 2.7571489810899998, 2.7593541145299998, 2.7733960151699999], "base_times": [2.66753315926, 2.6737060546899998, 2.67894601822, 2.6753239631699999, 2.6803069114700002, 2.6736009120899999, 2.6738917827600002, 2.67639303207, 2.6885011196100002, 2.6775891780899999, 2.67288613319, 2.6740000248000002, 2.6716890335099999, 2.6766500472999999, 2.6777601242100002, 2.6722629070299999, 2.67888998985, 2.6715672016099998, 2.6804389953599999, 2.6803979873700001, 2.6752359867100002, 2.67298698425, 2.67721605301, 2.6766111850700001, 2.67558693886, 2.6749620437599999, 2.67335510254, 2.67534399033, 2.6777729988100001, 2.67326784134, 2.67556595802, 2.6889100074800001, 2.6720960140200001, 2.6758561134300001, 2.6755039691900002, 2.6716270446800001, 2.6724560260799999, 2.6727318763699999, 2.6774148941, 2.6764769554100001, 2.6750671863600002, 2.6714379787400002, 2.7435669899000001, 2.7485840320600001, 2.70751786232, 2.7092530727400002, 2.67544198036, 2.6766531467400001, 2.6786830425299999, 2.67903494835]}], ["richards", "RawResult", {"changed_times": [0.42485690116899999, 0.391672134399, 0.38697290420500002, 0.38978099823000001, 0.38653588295000002, 0.38539409637499999, 0.38544106483500001, 0.38723397254899999, 0.38484597206100002, 0.38460087776200003, 0.38481712341300001, 0.387125015259, 0.38511180877700002, 0.38335013389599998, 0.38253307342499998, 0.38541388511699998, 0.40138697624199998, 0.41167879104600003, 0.385834932327, 0.4046189785, 0.38262510299699998, 0.39745593071000002, 0.38132977485699998, 0.38368082046500002, 0.380796909332, 0.37990593910199999, 0.38024783134500001, 0.38323211669899998, 0.379715919495, 0.38007807731600002, 0.37966990470899997, 0.40578198432899998, 0.38037204742399999, 0.38033199310299998, 0.38059902191200001, 0.382125139236, 0.38005089759799998, 0.38041114807100002, 0.37975502014200002, 0.41739606857299999, 0.37962198257399998, 0.37924098968499997, 0.40164279937699998, 0.38241791725199997, 0.37951016426099998, 0.37923693656899998, 0.37933707237199998, 0.38282299041700002, 0.37891888618500003, 0.37944197654700001], "base_times": [0.34984207153300001, 0.35352921485900002, 0.35030293464700002, 0.35039401054399999, 0.35356593132000003, 0.35617804527300001, 0.35045003890999998, 0.35419392585800002, 0.34917712211599999, 0.35496902465800001, 0.34898185729999998, 0.35016894340499999, 0.34672594070399998, 0.35246014595000003, 0.34967398643499997, 0.35236191749599999, 0.34780907630899999, 0.35169386863699997, 0.35305500030499998, 0.35085582733199999, 0.35157895088199997, 0.348843097687, 0.355578899384, 0.35684394836400002, 0.35460305213900001, 0.34866809845000002, 0.34962701797500001, 0.35786795616099998, 0.35232281684900002, 0.34995484352099998, 0.357528924942, 0.35109806060799997, 0.353901147842, 0.34876418113699997, 0.34823799133299999, 0.34872198104899998, 0.35193896293600002, 0.34771203994799998, 0.34916996955899998, 0.34796500205999997, 0.34642791748000001, 0.35339903831500002, 0.34859085083000002, 0.35140800476099998, 0.34648203849800002, 0.355483055115, 0.353637933731, 0.34880208969100002, 0.37337613105799999, 0.35136389732399997]}], ["spambayes", "RawResult", {"changed_times": [0.59816980361899996, 0.78452110290499999, 0.83023285865800001, 0.93411684036300002, 0.69184017181400004, 1.10234284401, 0.74256396293600002, 0.71328186988800002, 0.57437109947200005, 0.86799788474999995, 0.67963004112199998, 0.58452200889600003, 0.58005404472400002, 0.52808690071099995, 1.1698250770600001, 0.69769787788399995, 0.64825582504299994, 0.694402217865, 0.63659095764200002, 0.58709597587600004, 0.52931094169600001, 0.65094089508099995, 0.54797601699800003, 0.52007412910499995, 0.61390900611900001, 0.545217990875, 0.66334605216999998, 0.54549193382299999, 0.67743301391599997, 0.51848697662400001, 0.71530890464800001, 0.76569509506199995, 0.52243494987500005, 0.51943516731299999, 0.61838197708099996, 0.60029411315900005, 0.64244699478099998, 0.53688502311700004, 0.51747083663899995, 0.51864504814099999, 0.64398002624499995, 0.516402006149, 0.54355907440200002, 0.60826396942100003, 0.51326608657799999, 0.509041070938, 0.61415004730199996, 0.592803001404, 0.50892901420600001, 0.60751390457200005], "base_times": [0.29755282402, 0.29553508758500002, 0.29539799690200003, 0.29539704322799998, 0.29553294181799999, 0.29756402969399998, 0.295476913452, 0.29511308670000003, 0.29523301124599999, 0.294858932495, 0.29885101318399998, 0.29543805122400002, 0.29549407959000001, 0.29555702209500001, 0.295747041702, 0.29760813713099998, 0.29595708847000002, 0.29637312889099998, 0.29531884193399999, 0.295172929764, 0.29555201530500003, 0.29735279083299998, 0.29529094696000002, 0.29522705078099998, 0.29555082321199999, 0.29535007476800001, 0.29763197898900001, 0.29533004760699999, 0.29603099823000001, 0.29527783393899998, 0.29541301727300001, 0.29754710197399997, 0.29553294181799999, 0.295480012894, 0.295413970947, 0.29505491256700001, 0.29747200012199998, 0.29603290557900003, 0.29523515701300002, 0.295083999634, 0.29551792144799999, 0.29732418060299998, 0.29565906524699997, 0.29551196098299998, 0.29589390754700001, 0.295162916183, 0.29713106155399999, 0.29647493362400001, 0.295417070389, 0.29561495780899999]}], ["spectral-norm", "RawResult", {"changed_times": [0.46351408958399998, 0.36286616325400001, 0.36123991012599999, 0.35890722274800002, 0.35978198051499999, 0.35527896881100002, 0.35585403442399999, 0.35739207267799999, 0.35975408554100002, 0.35413813590999998, 0.35580110549900001, 0.35268497466999998, 0.364890098572, 0.35489296913099999, 0.35201191902200002, 0.354170084, 0.35294985771199999, 0.35513281822199999, 0.35184288024900001, 0.35136795043899999, 0.35138392448400002, 0.35504508018499997, 0.35184502601599998, 0.35189914703399999, 0.35134291648900001, 0.35165905952499998, 0.35476517677300001, 0.352794885635, 0.35161089897199999, 0.35130691528300001, 0.355042934418, 0.35176992416399999, 0.35218715667700001, 0.353415966034, 0.35497093200699997, 0.35284900665300001, 0.351359128952, 0.35298585891700002, 0.35289096832299999, 0.35544204711900002, 0.35241413116499998, 0.35347700118999997, 0.35199999809299998, 0.35367488861099999, 0.35371303558299999, 0.35233402252200002, 0.35166788101200003, 0.35387802124000001, 0.37293601036099999, 0.35278296470600001], "base_times": [0.50352501869199995, 0.47929906845100001, 0.48261594772299998, 0.47830605506899998, 0.48235297203100003, 0.48249197006200001, 0.48067307472199999, 0.48097801208500002, 0.47920393943799999, 0.47869086265600003, 0.48005580902099998, 0.47897410392799999, 0.48137307167100002, 0.483394145966, 0.47975897788999999, 0.47675085067700002, 0.48147702217100002, 0.47802019119299999, 0.482943058014, 0.47830295562699998, 0.48056387901300002, 0.47623491287199998, 0.47782897949199998, 0.47907996177700002, 0.48068904876700003, 0.47843909263599999, 0.48215293884299998, 0.478461027145, 0.48303794860799998, 0.47899913787800003, 0.478845119476, 0.480139970779, 0.48173093795799998, 0.47723102569600001, 0.47754788398699999, 0.47700595855700001, 0.48208713531500003, 0.47760915756200001, 0.480431079865, 0.48341894149800002, 0.47852706909199999, 0.47913908958399998, 0.48555207252499999, 0.490929841995, 0.47841882705700001, 0.48059010505700001, 0.47776508331299999, 0.47688412666300001, 0.48040413856500003, 0.47939014434799998]}], ["telco", "RawResult", {"changed_times": [1.452091, 1.324082, 1.3400840000000001, 1.4160889999999999, 1.3320829999999999, 1.3800859999999999, 1.280081, 1.304081, 1.304082, 1.300081, 1.308082, 1.2840800000000001, 1.308082, 1.296081, 1.3240829999999999, 1.308081, 1.284081, 1.324082, 1.288081, 1.300081, 1.284081, 1.328082, 1.3200829999999999, 1.288081, 1.300081, 1.2880799999999999, 1.288081, 1.308081, 1.288081, 1.308082, 1.284081, 1.304081, 1.2800800000000001, 1.304081, 1.288081, 1.304081, 1.288081, 1.300081, 1.288081, 1.2920799999999999, 1.312082, 1.284081, 1.308082, 1.2800800000000001, 1.304081, 1.288081, 1.2960799999999999, 1.288081, 1.304081, 1.284081], "base_times": [1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.2, 1.22, 1.2, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.2, 1.2, 1.21, 1.21, 1.21]}], ["twisted_names", "RawResult", {"changed_times": [0.022883295194508008, 0.020242914979757085, 0.024330900243309004, 0.02004008016032064, 0.020080321285140562, 0.018832391713747645, 0.017793594306049824, 0.018484288354898338, 0.017953321364452424, 0.017825311942959002, 0.017793594306049824, 0.019801980198019802, 0.017825311942959002, 0.017667844522968199, 0.017761989342806393, 0.017761989342806393, 0.018691588785046728, 0.018214936247723135, 0.017636684303350969, 0.021231422505307854, 0.017699115044247787, 0.018518518518518517, 0.017793594306049824, 0.017699115044247787, 0.018050541516245487, 0.017730496453900711, 0.01937984496124031, 0.01824817518248175, 0.017761989342806393, 0.018181818181818181, 0.017574692442882251, 0.017953321364452424, 0.017574692442882251, 0.017482517482517484, 0.017574692442882251, 0.017574692442882251, 0.017513134851138354, 0.020161290322580645, 0.017543859649122806, 0.017482517482517484, 0.018796992481203006, 0.017482517482517484, 0.017421602787456445, 0.017513134851138354, 0.017574692442882251, 0.017513134851138354, 0.017513134851138354, 0.020161290322580645, 0.017574692442882251, 0.018761726078799251], "base_times": [0.0095693779904306216, 0.0095147478591817315, 0.0095877277085330784, 0.0095602294455066923, 0.0095877277085330784, 0.0096339113680154135, 0.0095693779904306216, 0.0095602294455066923, 0.0095602294455066923, 0.0095785440613026813, 0.0095510983763132766, 0.0095785440613026813, 0.0095969289827255271, 0.0095969289827255271, 0.0096246390760346481, 0.0095602294455066923, 0.0095969289827255271, 0.0095785440613026813, 0.0095785440613026813, 0.0095602294455066923, 0.0096153846153846159, 0.0095602294455066923, 0.0096618357487922701, 0.0095693779904306216, 0.009727626459143969, 0.0095693779904306216, 0.0095602294455066923, 0.0095419847328244278, 0.0095693779904306216, 0.0095785440613026813, 0.0095693779904306216, 0.0095510983763132766, 0.0095969289827255271, 0.0095602294455066923, 0.0095785440613026813, 0.0095877277085330784, 0.0095693779904306216, 0.0095419847328244278, 0.0095785440613026813, 0.0095238095238095247, 0.0095693779904306216, 0.0095693779904306216, 0.0095877277085330784, 0.0095693779904306216, 0.0095877277085330784, 0.0095969289827255271, 0.0095785440613026813, 0.0095693779904306216, 0.0095602294455066923, 0.0095785440613026813]}]], "branch": "trunk", "revision": 0} \ No newline at end of file Modified: pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json ============================================================================== --- pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json (original) +++ pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json Wed Oct 13 13:30:14 2010 @@ -1 +1 @@ -{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [0.53373098373399996, 0.23460102081299999, 0.15513491630599999, 0.15454292297399999, 0.15753602981600001, 0.16903495788600001, 0.16105413436900001, 0.15707802772500001, 0.15180110931400001, 0.152679920197, 0.15186595916699999, 0.15010499954199999, 0.15148997306799999, 0.15008211135899999, 0.15097498893700001, 0.152936935425, 0.15074682235699999, 0.148921012878, 0.15586805343599999, 0.15136885643, 0.150803089142, 0.14815711975099999, 0.144705057144, 0.145576000214, 0.162219047546, 0.145064115524, 0.14615988731400001, 0.14499902725200001, 0.14503312110899999, 0.145503044128, 0.14562892913799999, 0.15218400955200001, 0.15338587760899999, 0.153208971024, 0.15358805656399999, 0.15331101417500001, 0.14960598945600001, 0.151103019714, 0.15276718139600001, 0.15110111236599999, 0.150665998459, 0.14848804473900001, 0.151926040649, 0.15147399902299999, 0.199955940247, 0.14494299888600001, 0.14402890205400001, 0.14464497566199999, 0.14744687080400001, 0.148157835007], "base_times": [1.98487401009, 1.9759330749499999, 1.97832417488, 1.9764730930300001, 1.97814917564, 1.98089694977, 1.9775922298399999, 1.9774131774899999, 1.98496413231, 1.97159814835, 1.98496413231, 1.97353196144, 1.9848189353900001, 1.97257804871, 1.9910008907300001, 1.9682059288, 1.9917528629300001, 1.9724640846299999, 1.98371481895, 1.9713909625999999, 1.9869208335899999, 1.96859908104, 1.9883351326000001, 1.9759950637799999, 1.9792518615700001, 1.9805219173399999, 1.9756710529299999, 1.9734799861900001, 2.0159261226699998, 1.99893116951, 1.97291398048, 1.9838719368, 1.9677410125699999, 1.9859578609499999, 1.96171092987, 1.9678678512600001, 1.97432017326, 1.97341108322, 1.9648609161399999, 1.96110486984, 1.9707100391400001, 1.98760914803, 1.9710340499900001, 1.9824509620699999, 1.9752931594800001, 1.9807510375999999, 1.9799480438199999, 1.9826321601900001, 1.9798321723900001, 1.9828879833199999]}], ["django", "RawResult", {"changed_times": [0.150973081589, 0.123874902725, 0.12347698211700001, 0.12420392036400001, 0.12366008758499999, 0.123784065247, 0.13101482391399999, 0.123060941696, 0.123810052872, 0.12381196022, 0.123774051666, 0.123774051666, 0.124183893204, 0.12363195419299999, 0.130721092224, 0.121907949448, 0.12308216095000001, 0.12296700477600001, 0.123407840729, 0.123026132584, 0.12340188026399999, 0.123024940491, 0.130163908005, 0.12311792373700001, 0.123579978943, 0.123372077942, 0.123338937759, 0.123032093048, 0.12360286712599999, 0.123323917389, 0.130560159683, 0.123554944992, 0.123425006866, 0.12345290184, 0.12413501739500001, 0.12303304672199999, 0.123439073563, 0.123630046844, 0.130742788315, 0.123475074768, 0.123498916626, 0.123980045319, 0.12337899208100001, 0.123532056808, 0.12364602088899999, 0.123553037643, 0.130734920502, 0.123599052429, 0.12362909317, 0.123425960541], "base_times": [0.634876966476, 0.63119101524400001, 0.64540290832500002, 0.63067197799700003, 0.63386702537499995, 0.63004899025000005, 0.63024806976299996, 0.63103890418999997, 0.63627886772200004, 0.63133406639099998, 0.63078498840300001, 0.63178682327299995, 0.63143706321699999, 0.63687920570400003, 0.63120698928800001, 0.631051063538, 0.630378007889, 0.63146805763199998, 0.63601493835400003, 0.63147497177099998, 0.63133502006499997, 0.63147711753799995, 0.63075184822100006, 0.63706302642799995, 0.63112902641299995, 0.63136720657300005, 0.631887197495, 0.63646221160899996, 0.63175106048600005, 0.63184499740599998, 0.63149690628099997, 0.63135409355200001, 0.63489508628799995, 0.63233184814499999, 0.63151216506999996, 0.629364967346, 0.62923312187199998, 0.633197069168, 0.63016796112100004, 0.62957406043999997, 0.62918901443499997, 0.63133692741400005, 0.63574194908100001, 0.63390803337099999, 0.63227581977799996, 0.63121104240399994, 0.62836694717399999, 0.63247489929199996, 0.63313794136000001, 0.63352489471399998]}], ["fannkuch", "RawResult", {"changed_times": [0.357728004456, 0.31800484657299999, 0.30672502517700001, 0.32194614410400002, 0.31080985069299999, 0.30235886573800003, 0.30262207984900003, 0.30578708648699998, 0.31529092788700003, 0.31280684471100001, 0.31020498275800001, 0.31131386756899998, 0.31377482414199998, 0.31479120254499998, 0.31179499626200002, 0.30305600166300001, 0.30441617965700002, 0.30722284316999998, 0.301214933395, 0.30310297012300003, 0.301802873611, 0.311623096466, 0.31810212135299998, 0.31204795837400001, 0.312577962875, 0.31050682067899998, 0.31150794029200002, 0.31205916404700001, 0.30325889587400001, 0.30611991882299999, 0.30642008781399999, 0.301440000534, 0.304536104202, 0.30922794341999998, 0.313621044159, 0.31280899047900002, 0.31117606163, 0.311715126038, 0.30853891372699999, 0.31230807304399999, 0.30691790580700001, 0.30679011344899998, 0.30811786651599998, 0.301835775375, 0.30178809165999998, 0.30465817451499999, 0.31420707702599998, 0.313306093216, 0.31133508682299998, 0.31083703041100003], "base_times": [1.36353611946, 1.35942316055, 1.3685638904599999, 1.3592638969399999, 1.36338400841, 1.3641269207, 1.36004590988, 1.3620519638099999, 1.36313199997, 1.36007881165, 1.3650290966, 1.3646898269700001, 1.35792398453, 1.36864614487, 1.36348605156, 1.36114597321, 1.3651762008699999, 1.36486387253, 1.3583400249499999, 1.36733984947, 1.3646521568300001, 1.3632168769799999, 1.36906194687, 1.3584411144299999, 1.36989593506, 1.3611299991600001, 1.36594104767, 1.4054760932899999, 1.36255788803, 1.36518597603, 1.3638501167299999, 1.3670859336900001, 1.3645708560900001, 1.3667361736300001, 1.36373782158, 1.36704611778, 1.3673570156099999, 1.3680310249300001, 1.3642778396599999, 1.36619615555, 1.36799502373, 1.3654208183300001, 1.3667390346499999, 1.35933709145, 1.36695981026, 1.3632080554999999, 1.36811590195, 1.3632729053499999, 1.35716891289, 1.3655180931099999]}], ["go", "RawResult", {"changed_times": [0.205440998077, 0.237554788589, 0.178432941437, 0.22178697586099999, 0.15453481674200001, 0.184565067291, 0.181967973709, 0.20851802825900001, 0.16120600700400001, 0.12751483917199999, 0.126490831375, 0.21313405036899999, 0.13336992263799999, 0.13810706138600001, 0.17555379867599999, 0.117103099823, 0.120506048203, 0.124084234238, 0.12555694580099999, 0.13820600509600001, 0.15432786941500001, 0.12550187110899999, 0.12009692192099999, 0.126209020615, 0.120643138885, 0.12826299667400001, 0.125062942505, 0.14115309715300001, 0.112189769745, 0.12329006195099999, 0.119536876678, 0.115894079208, 0.13150215148899999, 0.120622158051, 0.133524894714, 0.10953402519200001, 0.110867977142, 0.120260953903, 0.128885030746, 0.11038398742699999, 0.113687992096, 0.105720996857, 0.110606908798, 0.106270074844, 0.12149906158400001, 0.146656036377, 0.118952989578, 0.108425855637, 0.126958847046, 0.116177082062], "base_times": [0.59508895874000001, 0.59899997711200004, 0.59387707710299997, 0.59602403640699997, 0.59553503990199996, 0.593573093414, 0.59433388710000001, 0.598137140274, 0.59456205368000004, 0.59513306617700001, 0.59438300132800004, 0.59516787529000004, 0.59493899345400003, 0.59641695022600005, 0.59443712234500001, 0.59466099739099998, 0.59525895118700001, 0.597331047058, 0.59680700302099998, 0.59574007988, 0.59577918052699996, 0.59496688842800005, 0.595326185226, 0.59544396400499999, 0.59744405746499996, 0.59654498100300002, 0.595510959625, 0.59514212608299999, 0.59548306465099998, 0.59657096862799996, 0.59839987754799995, 0.59662580490100003, 0.59663081169099996, 0.59636092186, 0.59646797180199995, 0.59581613540599998, 0.59876608848599999, 0.59532284736600005, 0.59693384170499997, 0.59681200981100002, 0.59692192077600004, 0.59601807594300005, 0.59875106811500001, 0.59596300125099999, 0.59473204612700004, 0.59547209739700002, 0.59548902511599999, 0.59676194190999998, 0.59593701362600005, 0.59788584709199999]}], ["html5lib", "RawResult", {"changed_times": [9.1950540542599999, 6.5574679374700002, 6.13121509552, 5.92117595673, 5.8948459625199998], "base_times": [8.6243629455599997, 8.6805830001800004, 8.6462841033900002, 8.6504099368999992, 8.6197869777699996]}], ["meteor-contest", "RawResult", {"changed_times": [0.32466602325400001, 0.28066992759699999, 0.27618789672900002, 0.27726387977599998, 0.27565312385599999, 0.27732300758400003, 0.283158063889, 0.27601695060699999, 0.27488493919399998, 0.27476692199699998, 0.274809122086, 0.27458906173699998, 0.28050112724300003, 0.27524900436400002, 0.27467489242600002, 0.273777008057, 0.27420091628999999, 0.27381587028499998, 0.27388405799900001, 0.27448701858500002, 0.27409100532500003, 0.27384519576999999, 0.27392005920399998, 0.27450394630399999, 0.27359294891399999, 0.27404904365499999, 0.27379584312400002, 0.27414393425, 0.27374601364099999, 0.27368783950800002, 0.279189825058, 0.27381300926199997, 0.27382278442399999, 0.27362394332899997, 0.27394199371299999, 0.27363896369899998, 0.27471089363099999, 0.27363920211800002, 0.273921966553, 0.273852109909, 0.27355313301099998, 0.27437615394600001, 0.27359104156500003, 0.27364397049, 0.27369093895000002, 0.27492690086400001, 0.27363181114200003, 0.273891925812, 0.27367687225300003, 0.27414608001700003], "base_times": [0.24080085754399999, 0.24055409431499999, 0.24092507362400001, 0.240586042404, 0.24052286148099999, 0.24070692062400001, 0.24118590354899999, 0.24084687232999999, 0.240713119507, 0.24066591262799999, 0.24061107635500001, 0.24103999137900001, 0.240550041199, 0.24071502685500001, 0.240919113159, 0.24124288559000001, 0.24059915542599999, 0.24063205719, 0.240550994873, 0.240890026093, 0.24065899848899999, 0.24082207679699999, 0.24078011512799999, 0.241106033325, 0.240442991257, 0.24064207076999999, 0.24046015739400001, 0.24054288864100001, 0.240490198135, 0.24042892456100001, 0.24063515663099999, 0.24120903015100001, 0.24073195457499999, 0.240746021271, 0.24080610275299999, 0.240445137024, 0.241003990173, 0.24048995971699999, 0.240598201752, 0.24078392982499999, 0.24111199379000001, 0.24032306671100001, 0.24051117896999999, 0.24035692215000001, 0.24025416374200001, 0.239776849747, 0.24031782150299999, 0.24069786071800001, 0.24129700660699999, 0.241048812866]}], ["nbody_modified", "RawResult", {"changed_times": [0.088541984558100001, 0.072235107421899994, 0.068952083587599994, 0.068921089172400002, 0.069000959396399997, 0.068970203399700003, 0.069295883178700002, 0.069005012512199995, 0.069174051284800003, 0.068920135497999996, 0.068984031677200006, 0.068948984146099998, 0.068947076797500007, 0.068919181823699993, 0.068933963775600002, 0.069010972976700002, 0.0692219734192, 0.068889141082800004, 0.068933963775600002, 0.068947076797500007, 0.068988084793099994, 0.069360971450800002, 0.068981885909999999, 0.069058895111099999, 0.069154977798499995, 0.069015026092500001, 0.068929910659800003, 0.068998813629200004, 0.068931102752699994, 0.068953037262, 0.068956851959199997, 0.068850040435799995, 0.069129943847700007, 0.068831920623800003, 0.068848848342900004, 0.069072008132900001, 0.068994045257600001, 0.068930149078400005, 0.068916082382199997, 0.068859100341799998, 0.069105863571199994, 0.068810939788799999, 0.068866014480599994, 0.068874120712299994, 0.0688879489899, 0.068877935409500005, 0.068838119506799994, 0.068907022476199994, 0.069243192672700002, 0.069992065429700004], "base_times": [0.430072069168, 0.43174719810500001, 0.43013787269600001, 0.42945313453700001, 0.42914891242999997, 0.43041586875900001, 0.42992877960199999, 0.43064594268799999, 0.431170940399, 0.431824922562, 0.43092703819299999, 0.43053197860699999, 0.42869901657100001, 0.43168401718100002, 0.43105721473699998, 0.43112492561299998, 0.43205499649000001, 0.43006110191300001, 0.42961001396199999, 0.428696155548, 0.429335832596, 0.42900800704999997, 0.429177999496, 0.42947387695299999, 0.42899489402800001, 0.42581796646100001, 0.42579293251, 0.426750898361, 0.42619681358299999, 0.42690300941499998, 0.426344156265, 0.425684928894, 0.42617511749300002, 0.425367116928, 0.42452383041399999, 0.42517709732100001, 0.42424011230499997, 0.42376208305399998, 0.42331886291499998, 0.42367100715599998, 0.423776865005, 0.42354416847199999, 0.42360591888400001, 0.42415189742999998, 0.423532009125, 0.42389607429499998, 0.42379999160800003, 0.42320799827599997, 0.42313408851599998, 0.42342805862400001]}], ["pyflate-fast", "RawResult", {"changed_times": [1.0659608840899999, 1.05365204811, 1.06374502182, 1.0581080913500001, 1.06380200386, 1.0373589992500001, 1.05517578125, 1.0548541545900001, 1.07799100876, 1.0740821361499999, 1.0693590640999999, 1.04684495926, 1.0749399662000001, 1.07159519196, 1.0657529830900001, 1.0503039359999999, 1.0730760097500001, 1.06034016609, 1.06997489929, 1.05047512054, 1.0729389190700001, 1.06514906883, 1.0397839546200001, 1.0728981494900001, 1.0665550231900001, 1.0433421134900001, 1.07183599472, 1.0667510032700001, 1.0408699512499999, 1.0743548870099999, 1.07203793526, 1.0681369304699999, 1.0525979995700001, 1.0704171657599999, 1.0463750362399999, 1.07289981842, 1.0518350601199999, 1.06989216805, 1.044672966, 1.0753920078300001, 1.0531618595100001, 1.0723462104799999, 1.0446758270300001, 1.07318806648, 1.0539751052899999, 1.08029985428, 1.04665017128, 1.0725378990200001, 1.05208015442, 1.0732369422900001], "base_times": [2.0108699798599998, 2.0109808445000001, 2.0061230659499998, 2.0109241008800001, 2.0078430175799999, 2.0112669467900002, 2.0139169692999999, 2.0103061199200001, 2.00638103485, 2.01138401031, 2.0138700008399999, 2.0058629512800001, 2.0086250305200002, 2.0138199329400002, 2.0063400268599998, 2.0082821846000001, 2.0112380981400002, 2.0076570510899998, 2.0080878734600001, 2.0108051300000001, 2.00607514381, 2.0063180923499999, 2.0096981525399999, 2.0101280212399999, 2.0155758857700001, 2.0102498531299999, 2.0053129196200001, 2.0141990184799998, 2.0114769935600001, 2.0126900672899999, 2.0084600448600001, 2.01082491875, 2.0114600658400001, 2.0073368549300001, 2.0094199180599999, 2.00728797913, 2.0094258785200001, 2.0149018764500002, 2.0094230175000001, 2.0109641551999999, 2.01358914375, 2.01244997978, 2.0089750289900001, 2.0090889930700002, 2.0079500675199999, 2.0080699920699998, 2.01176691055, 2.0092239379899999, 2.0114538669600002, 2.0104010105099999]}], ["raytrace-simple", "RawResult", {"changed_times": [0.104208946228, 0.106266021729, 0.098669052123999998, 0.091032028198199999, 0.091249942779500007, 0.118623018265, 0.089195013046299995, 0.088937997818000006, 0.097711086273200007, 0.097254991531400006, 0.090123891830399996, 0.089380025863599999, 0.11132407188399999, 0.088888168335000003, 0.089291095733600001, 0.088446140289299993, 0.113281011581, 0.088557004928599997, 0.088536024093599994, 0.087917804717999995, 0.114279031754, 0.096406936645499999, 0.087327957153299995, 0.087787151336699995, 0.10245609283400001, 0.086941003799399993, 0.087326049804700004, 0.087269783020000005, 0.088474035263100007, 0.116209030151, 0.087454080581699997, 0.101906776428, 0.086088895797700002, 0.086314916610699996, 0.085654973983799995, 0.086269855499299997, 0.085838079452499993, 0.086305856704700007, 0.087749958038300005, 0.14508390426600001, 0.086570978164699997, 0.085652112960799998, 0.108924865723, 0.086509943008399995, 0.086097002029400002, 0.10036110878, 0.099938154220599998, 0.085299968719500005, 0.085878849029500007, 0.0852539539337], "base_times": [1.6054999828300001, 1.6075158119199999, 1.6068139076200001, 1.6222910880999999, 1.61292099953, 1.60913705826, 1.6109960079200001, 1.62233877182, 1.6038889884900001, 1.60813307762, 1.6408219337500001, 1.60450601578, 1.6084368228899999, 1.6058139801, 1.6087529659299999, 1.6044671535499999, 1.60560894012, 1.6072280406999999, 1.6039071083100001, 1.6084260940599999, 1.6079659462, 1.6057059764899999, 1.6081471443199999, 1.60494303703, 1.60676002502, 1.60675907135, 1.6066009998299999, 1.60767817497, 1.6090710163099999, 1.6052989959699999, 1.6084320545199999, 1.61109900475, 1.6051030158999999, 1.60873317719, 1.6393661499000001, 1.6057238578799999, 1.6092879772199999, 1.6042079925499999, 1.62158799171, 1.6056730747200001, 1.60550308228, 1.60018992424, 1.62533283234, 1.60493087769, 1.61037397385, 1.6037578582800001, 1.6093821525600001, 1.60478901863, 1.63921904564, 1.60729408264]}], ["richards", "RawResult", {"changed_times": [0.0228841304779, 0.0150561332703, 0.0134420394897, 0.013730049133300001, 0.0136449337006, 0.0136260986328, 0.0137310028076, 0.0131878852844, 0.0136740207672, 0.013035058975199999, 0.0134780406952, 0.0130591392517, 0.012836933135999999, 0.0136439800262, 0.0124011039734, 0.013193130493200001, 0.0125820636749, 0.012362003326400001, 0.016746997833300001, 0.0124850273132, 0.0123841762543, 0.012007951736499999, 0.012148857116699999, 0.011607885360699999, 0.0119562149048, 0.0117020606995, 0.0120451450348, 0.0118389129639, 0.011528015136699999, 0.012020111084, 0.011619091033900001, 0.012064933776900001, 0.0116050243378, 0.0118329524994, 0.012051820755, 0.011452913284300001, 0.0119879245758, 0.0114850997925, 0.0118889808655, 0.0117588043213, 0.0115189552307, 0.0119571685791, 0.0116050243378, 0.0119731426239, 0.0115461349487, 0.0121710300446, 0.011741161346400001, 0.011747121810899999, 0.0118410587311, 0.0113570690155], "base_times": [0.21546196937600001, 0.21623301506000001, 0.21627402305599999, 0.216693162918, 0.21707105636599999, 0.21611881256099999, 0.21635699272200001, 0.21637606620800001, 0.21546506881700001, 0.21522092819200001, 0.21543502807600001, 0.21676111221300001, 0.21627402305599999, 0.21640300750700001, 0.21558499336199999, 0.21569085121199999, 0.21543717384300001, 0.21592593193099999, 0.21752214431799999, 0.21556806564299999, 0.21569395065300001, 0.21604013443, 0.21652293205299999, 0.21636700630200001, 0.215793848038, 0.215296030045, 0.215675830841, 0.21583795547500001, 0.21543502807600001, 0.216253995895, 0.21705412864699999, 0.21611094474799999, 0.216734886169, 0.21633100509600001, 0.218207120895, 0.21751618385300001, 0.21754693985000001, 0.21495008468599999, 0.21610498428300001, 0.216495037079, 0.217024087906, 0.21569108963, 0.21534991264299999, 0.215554952621, 0.21538805961599999, 0.21663308143599999, 0.215176820755, 0.21531486511199999, 0.21609115600600001, 0.216979026794]}], ["spambayes", "RawResult", {"changed_times": [0.28751707076999999, 0.34429097175599999, 0.327184915543, 0.30877208709699999, 0.30719304084799998, 0.29175710678099998, 0.30372905731200001, 0.307572126389, 0.24438500404399999, 0.303647994995, 0.273859024048, 0.278746128082, 0.23960399627699999, 0.26048088073699999, 0.33833193778999998, 0.26831603050199998, 0.28914999961900001, 0.239518880844, 0.24775385856599999, 0.24561405181900001, 0.25214099884000002, 0.25770306587199998, 0.25043201446500002, 0.22554898262, 0.24968504905700001, 0.23150801658600001, 0.25834798812900001, 0.22896385192900001, 0.24762606620800001, 0.24714899063099999, 0.26752901077300001, 0.290445804596, 0.21951293945299999, 0.241242170334, 0.22318506240800001, 0.25629496574400001, 0.22942090034500001, 0.24148082733199999, 0.21564912795999999, 0.23730015754700001, 0.21533489227300001, 0.23749184608500001, 0.22082400322000001, 0.24251198768599999, 0.214025020599, 0.235796928406, 0.227761983871, 0.23491716384899999, 0.22989392280599999, 0.23521780967700001], "base_times": [0.19353604316699999, 0.19359993934600001, 0.19369316101100001, 0.19351387023899999, 0.193557977676, 0.19352483749400001, 0.19358801841699999, 0.19363498687700001, 0.19352602958699999, 0.193531990051, 0.193488121033, 0.193665027618, 0.19356894493099999, 0.19348406791700001, 0.19356298446699999, 0.19351005554199999, 0.19349193573000001, 0.19352388381999999, 0.193593978882, 0.193593978882, 0.19361186027499999, 0.193543195724, 0.19364786148099999, 0.19361186027499999, 0.193534135818, 0.19354987144499999, 0.19374704361, 0.193845033646, 0.19361901283300001, 0.19356989860500001, 0.19359111785899999, 0.19361281394999999, 0.19363498687700001, 0.19360589981099999, 0.193613052368, 0.193671941757, 0.19370293617199999, 0.193685054779, 0.19372391700700001, 0.193596124649, 0.193512916565, 0.193508148193, 0.193747997284, 0.193646907806, 0.193574905396, 0.19359993934600001, 0.193615198135, 0.19353985786399999, 0.19352793693500001, 0.19359207153300001]}], ["spectral-norm", "RawResult", {"changed_times": [0.078598976135299994, 0.036157846450800002, 0.035543203353900002, 0.035031080245999997, 0.033834934234600003, 0.0332310199738, 0.033292055129999998, 0.0338280200958, 0.033846855163599997, 0.033032894134500002, 0.032781124114999997, 0.032047986984300003, 0.031976938247700003, 0.031943082809400003, 0.031916141510000003, 0.032538175582900002, 0.032065868377700001, 0.032016992569000001, 0.0319721698761, 0.031663894653300002, 0.031998872757000002, 0.032014846801800001, 0.031949996948199999, 0.031929016113299999, 0.031960010528600001, 0.032012939453099999, 0.031993150711100003, 0.031938076019300002, 0.031931161880499999, 0.031976938247700003, 0.032008171081500003, 0.031945943832400001, 0.031930923461899997, 0.031975030899000001, 0.032008886337300001, 0.031981945037799997, 0.031640052795399998, 0.031987905502299997, 0.031998872757000002, 0.032404184341400001, 0.031841039657599997, 0.031826019287100001, 0.0318641662598, 0.031888961792000001, 0.0318200588226, 0.031824111938500002, 0.031880140304599999, 0.031901121139499999, 0.031842947006200002, 0.032522916793799997], "base_times": [0.38017416000400001, 0.37997889518700001, 0.38001012802099998, 0.38086199760400002, 0.38064312934900002, 0.38048291206399998, 0.38021111488300002, 0.38219904899599999, 0.38127899169899998, 0.38254404067999997, 0.38078594207799998, 0.380933046341, 0.38103008270299998, 0.38071012496899997, 0.38104891777, 0.38050794601400001, 0.38025808334400002, 0.38140201568600002, 0.38133311271699999, 0.38049197197000001, 0.38048505783100001, 0.38087606430100002, 0.380648136139, 0.38163900375400001, 0.38113594055200001, 0.38124513626099998, 0.38089895248400002, 0.38078594207799998, 0.38048291206399998, 0.38178706169100002, 0.38335895538300002, 0.38262677192700001, 0.38049507141099997, 0.38140916824299997, 0.380529880524, 0.381772994995, 0.38075399398799997, 0.381628990173, 0.380851984024, 0.38191199302700002, 0.38043808937099999, 0.38140487670899997, 0.38037204742399999, 0.38166618347199999, 0.38078689575199998, 0.38159108161900002, 0.38048791885400002, 0.381910085678, 0.38064503669700001, 0.38182401657100001]}], ["telco", "RawResult", {"changed_times": [0.20000000000000001, 0.17999999999999999, 0.17000000000000001, 0.17000000000000001, 0.16, 0.17000000000000001, 0.16, 0.16, 0.16, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.16, 0.16, 0.16, 0.14999999999999999, 0.16, 0.16, 0.16, 0.14999999999999999, 0.16, 0.16, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999, 0.14999999999999999, 0.16, 0.14999999999999999, 0.16, 0.16, 0.14999999999999999], "base_times": [0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.78000000000000003, 0.79000000000000004, 0.78000000000000003, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.78000000000000003, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.79000000000000004, 0.80000000000000004, 0.78000000000000003, 0.79000000000000004]}], ["twisted_names", "RawResult", {"changed_times": [0.0043215211754537601, 0.0047438330170777986, 0.0043215211754537601, 0.0043421623968736434, 0.0043365134431916736, 0.0043763676148796497, 0.0042936882782310002, 0.0042789901583226361, 0.0046641791044776115, 0.0042716787697565147, 0.0043859649122807015, 0.0042735042735042739, 0.0042992261392949269, 0.0042863266180882984, 0.0042955326460481103, 0.0046860356138706651, 0.0043047783039173483, 0.004269854824935952, 0.0042936882782310002, 0.0042826552462526769, 0.0043103448275862068, 0.0042844901456726651, 0.0047169811320754715, 0.004268032437046522, 0.0042936882782310002, 0.0043029259896729772, 0.0042808219178082189, 0.0042808219178082189, 0.0042863266180882984, 0.0047103155911446069, 0.0042753313381787093, 0.0043122035360068992, 0.0042716787697565147, 0.0042973785990545769, 0.0042918454935622317, 0.0043122035360068992, 0.0046728971962616819, 0.0042863266180882984, 0.0042789901583226361, 0.0042973785990545769, 0.0043084877208099956, 0.0043047783039173483, 0.0042771599657827203, 0.0043066322136089581, 0.0047551117451260106, 0.0042918454935622317, 0.0042844901456726651, 0.0042973785990545769, 0.0042973785990545769, 0.0043066322136089581], "base_times": [0.0066225165562913907, 0.0066401062416998674, 0.0066401062416998674, 0.0066137566137566134, 0.006648936170212766, 0.0066445182724252493, 0.0066401062416998674, 0.0066533599467731202, 0.0066357000663570011, 0.006648936170212766, 0.0066401062416998674, 0.0066225165562913907, 0.0066401062416998674, 0.0066357000663570011, 0.0066312997347480109, 0.0066312997347480109, 0.0066181336863004635, 0.0066533599467731202, 0.0066181336863004635, 0.0066312997347480109, 0.0066137566137566134, 0.0066357000663570011, 0.0066225165562913907, 0.006648936170212766, 0.0066312997347480109, 0.0066577896138482022, 0.0066577896138482022, 0.0066445182724252493, 0.0066401062416998674, 0.006648936170212766, 0.0066269052352551355, 0.0066445182724252493, 0.0066533599467731202, 0.0066533599467731202, 0.0066533599467731202, 0.0066225165562913907, 0.0066401062416998674, 0.0066269052352551355, 0.0066269052352551355, 0.0066225165562913907, 0.0066445182724252493, 0.0066269052352551355, 0.0066401062416998674, 0.0066137566137566134, 0.0066445182724252493, 0.0066357000663570011, 0.0066357000663570011, 0.0066312997347480109, 0.0066357000663570011, 0.0066137566137566134]}]], "branch": "trunk", "revision": 0} \ No newline at end of file +{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [19.594973802599998, 17.836122035999999, 17.7611591816, 17.7673540115, 17.797179937399999, 17.701630115499999, 17.701794147499999, 17.726331234, 17.7549710274, 17.7046818733, 17.715419054000002, 17.7220032215, 17.708485126500001, 17.698208808899999, 17.943202018699999, 17.7065188885, 17.7018699646, 17.723783969900001, 17.703621864300001, 17.702537059800001, 17.698129892299999, 17.717091083500002, 17.693710088700001, 17.698023796099999, 17.790216922799999, 17.7049281597, 17.7118620872, 17.7670118809, 17.7513260841, 17.712387800199998, 17.703951120399999, 17.7405331135, 17.704728841800002, 17.7038481236, 17.719988107700001, 17.7051708698, 17.701725959800001, 17.709410905799999, 17.716867923700001, 17.694391012200001, 17.700695037799999, 17.712723970399999, 17.692533969900001, 17.705747842800001, 17.712710142100001, 17.758448123899999, 17.701386928600002, 17.700757980300001, 18.095870017999999, 17.7208271027], "base_times": [2.76247382164, 2.75780797005, 2.7526109218600001, 2.7581448554999999, 2.7579169273400002, 2.7591660022700002, 2.7544469833399998, 2.75570702553, 2.7566180229200001, 2.7799978256200002, 2.75436115265, 2.7584359645799998, 2.7715408801999999, 2.7583799362199999, 2.75467610359, 2.7834770679499998, 2.7558917999300001, 2.7597181797000001, 2.75486803055, 2.7581379413599998, 2.7532849311800001, 2.7636089324999999, 2.75440192223, 2.7553980350499998, 2.7562680244400002, 2.7588288784000001, 2.75431799889, 2.7562849521600001, 2.7566390037500001, 2.75684499741, 2.7588860988600001, 2.7639617919899999, 2.758425951, 2.7611510753599999, 2.75679397583, 2.7544021606400002, 2.7587559223200002, 2.7592377662700001, 2.7555589675899999, 2.7590520381900001, 2.75563788414, 2.76086902618, 2.7588081359899999, 2.7565140724199999, 2.75627803802, 2.76230502129, 2.7579100132000001, 2.75721096992, 2.7558379173300001, 2.7606220245399999]}], ["django", "RawResult", {"changed_times": [0.30002903938300002, 0.25580286979700001, 0.261374950409, 0.26095104217499998, 0.25220704078700001, 0.26073884963999999, 0.26132106780999997, 0.254949092865, 0.26201701164199998, 0.26217198371900002, 0.25196504592899999, 0.26127791404700001, 0.26208996772799997, 0.254974842072, 0.261093854904, 0.261404037476, 0.25236296653700002, 0.26157402992200002, 0.26323699951200003, 0.25301718711900001, 0.26132202148400002, 0.26150584220899997, 0.25210714340200002, 0.26118707656899998, 0.26585507392899999, 0.25383305549599999, 0.26143312454200002, 0.26261687278700002, 0.267751932144, 0.26117897033699999, 0.264419078827, 0.252134799957, 0.26171708107000002, 0.26141405105600002, 0.25201201438900001, 0.261445045471, 0.26490807533299998, 0.25207686424300002, 0.26115393638599999, 0.26161694526700002, 0.25278902053800001, 0.261168003082, 0.26426982879599997, 0.25209307670600001, 0.261046171188, 0.26155900955200001, 0.25206208229100002, 0.261226177216, 0.26494908332799999, 0.252925872803], "base_times": [0.995049953461, 0.99419713020300005, 0.99113988876299997, 0.99470710754400005, 0.99212503433200006, 0.99265909194900004, 0.99358582496600001, 0.99400091171299998, 0.99305796623200004, 0.99362897872900002, 0.99429702758799998, 0.99161601066600003, 0.99425101280200001, 0.99299907684300004, 0.99139285087600004, 0.99515914917000003, 0.99177312850999999, 0.99377703666700001, 0.99326705932600001, 0.99304509162900001, 0.99348807334900002, 0.99273705482499996, 0.99258708953899999, 0.99477910995499996, 0.99403500556900004, 0.99051785469099995, 0.994111061096, 0.99377894401599998, 0.99141192436199999, 0.99447512626599999, 0.99370288848900001, 0.99225115776100004, 0.99282193183900003, 0.99233508110000002, 0.99293303489700002, 0.99670696258500002, 0.99345684051500005, 0.99477696418799999, 0.99382996559100001, 0.99122619628899999, 0.99395608901999999, 0.99324393272400002, 0.99102210998499995, 0.99499702453600003, 0.99375295639000005, 0.99156498909000002, 0.99277806282000003, 0.99424910545300005, 0.99084305763199998, 0.993386983871]}], ["fannkuch", "RawResult", {"changed_times": [0.52755904197699999, 0.47701597213699998, 0.46741414070100001, 0.47383809089700002, 0.46101403236400001, 0.46737408638, 0.47243213653600002, 0.47586488723800002, 0.46479701995799999, 0.46248412132299999, 0.46604609489400001, 0.46004605293299999, 0.46376204490700002, 0.46708703041100003, 0.47270798683199999, 0.472360134125, 0.47210407257100001, 0.47691607475300002, 0.47826886177099998, 0.47204399108900003, 0.47201299667399998, 0.46304917335500001, 0.46217513084400003, 0.46784305572500001, 0.460357189178, 0.46357607841499998, 0.46536898613, 0.46894693374599999, 0.46510815620399998, 0.46110606193499998, 0.46555995941200001, 0.46238088607799999, 0.46605896949800002, 0.46549701690700002, 0.45882105827300002, 0.45827507972699999, 0.473033905029, 0.47556495666499998, 0.460822105408, 0.47130393981899998, 0.476540803909, 0.46522593498199999, 0.46266603469799999, 0.47138500213599999, 0.46661305427600003, 0.46587181091300001, 0.47512102127099998, 0.47284221649199998, 0.47128200530999997, 0.47180795669600001], "base_times": [1.9883148670199999, 1.9792110919999999, 1.9909551143599999, 1.98928499222, 1.97898578644, 1.98664617538, 1.9864010810899999, 1.98417901993, 1.9872028827699999, 1.98194694519, 1.9878189563799999, 1.98948693275, 1.9811220169099999, 1.9890899658200001, 1.9894771575900001, 1.9801352024100001, 1.9894909858700001, 1.9891271591199999, 1.98107600212, 1.99568986893, 1.9895131587999999, 1.9803609847999999, 2.0069239139600001, 1.9916350841499999, 1.98022890091, 1.98729896545, 1.99073004723, 1.98082304001, 1.98834180832, 1.9909420013400001, 1.9793639183, 1.98839306831, 1.9902248382600001, 1.9822490215299999, 1.98810195923, 1.99047279358, 1.98221516609, 1.98903012276, 1.9900078773500001, 1.9820079803499999, 1.9892179966000001, 1.9895989894899999, 1.9800350665999999, 1.9895751476300001, 1.9891831874799999, 1.9804949760399999, 1.9896249771100001, 1.9898300170900001, 1.9807391166699999, 1.9895370006599999]}], ["go", "RawResult", {"changed_times": [0.46677207946799998, 0.51466798782300005, 0.388464927673, 0.50362396240200002, 0.36921596527099998, 0.41929197311400002, 0.37606406211900001, 0.428333997726, 0.36696696281399999, 0.33521819114700002, 0.35271787643399999, 0.47167301177999998, 0.36826205253599997, 0.33157110214199997, 0.39287805557299998, 0.30100107192999997, 0.31953597068799999, 0.34765505790700002, 0.31619405746500001, 0.32286810874900002, 0.37090992927600003, 0.328006982803, 0.31113696098299998, 0.355150938034, 0.30218696594200001, 0.31915521621699999, 0.31008601188700002, 0.34765815734900002, 0.30721187591600002, 0.32940602302600003, 0.30662703514099998, 0.33826303482100001, 0.29518413543700001, 0.32855200767499998, 0.37890100479099997, 0.295953989029, 0.28935694694500003, 0.32658910751300002, 0.31018304824800003, 0.32643103599500001, 0.28621220588700003, 0.29142880439800001, 0.28649687766999998, 0.301717042923, 0.288222074509, 0.353245973587, 0.33032798767100002, 0.285399913788, 0.29600501060500001, 0.28842616081200001], "base_times": [0.94673204422000001, 0.94645810127300001, 0.94507908821099995, 0.944802045822, 0.941561937332, 0.944491147995, 0.94291400909400003, 0.95224094390900005, 0.94410514831500003, 0.943855047226, 0.94802284240699997, 0.94546699523900002, 0.94773101806600002, 0.94944882392899999, 0.94337701797499995, 0.94636011123700003, 0.94657802581799999, 0.94187402725199998, 0.94828701019299999, 0.94231510162400001, 0.94628596305799995, 0.94881415367099997, 0.94279098510699999, 0.94889903068500003, 0.95280098915099998, 0.94382286071800003, 0.94861888885500001, 0.94221186637900001, 0.94600486755400004, 0.94160294532800004, 0.94753003120400003, 0.94430899620099995, 0.94843101501500005, 0.96007490158099995, 0.94707822799700003, 0.94710588455199995, 0.95057606697100006, 0.94280695915199997, 0.94800686836199999, 0.94509601593000003, 0.96038508415199997, 0.94446587562600004, 0.94845819473299997, 0.94734406471300003, 0.94510412216200002, 0.94387602806100002, 0.94524908065799995, 0.94992303848299997, 0.94564318656900004, 0.94311594963099998]}], ["html5lib", "RawResult", {"changed_times": [15.457784891099999, 11.3277301788, 10.600764036199999, 10.4216821194, 10.2325599194, 10.2733259201, 10.3756709099, 10.156955957399999, 10.101744890200001, 10.133589983, 10.206784963600001, 10.1694169044, 10.0392251015, 10.155544042600001, 10.1885080338, 10.058653831499999, 10.2810080051, 10.2360329628, 10.039163112600001, 10.1998898983, 10.119593143499999, 10.1641438007, 10.0413920879, 10.107867002500001, 10.135586977000001, 10.1675360203, 10.1493079662, 10.060842990899999, 10.178154945399999, 10.1074330807, 10.170939922300001, 10.047384023699999, 10.0127818584, 10.050449132900001, 10.161669015899999, 10.037492990500001, 10.0900728703, 10.190579891200001, 9.9983761310600006, 10.1199560165, 10.003987073899999, 10.119729042099999, 10.1512379646, 10.054825067499999, 9.9258110523200003, 10.0976631641, 10.0940468311, 10.122640132900001, 9.9850409030900007, 10.099215984300001], "base_times": [13.800020933200001, 13.988759994500001, 13.945523977300001, 13.9515860081, 13.8968560696, 14.0439591408, 14.0257151127, 13.9940800667, 13.977200985, 13.965268135100001, 13.8857018948, 14.0525372028, 14.022065877899999, 14.0086979866, 13.961042880999999, 13.921565771099999, 13.924687862400001, 14.0379269123, 14.256644964199999, 13.9928328991, 14.0013000965, 13.9456880093, 13.914921999000001, 14.061727047, 14.0501580238, 14.0101730824, 13.9787597656, 13.921334981899999, 13.889928102500001, 14.0468928814, 14.0285110474, 14.0453250408, 13.9559979439, 13.9637241364, 13.9317610264, 14.0579519272, 14.0324690342, 14.0096049309, 13.9675040245, 13.964776992799999, 13.924336910199999, 14.0519120693, 14.0543570518, 14.0083789825, 13.9841370583, 13.954766988799999, 13.9033069611, 13.8949439526, 14.038803100599999, 14.0201179981]}], ["meteor-contest", "RawResult", {"changed_times": [0.47674798965499998, 0.403456926346, 0.39809894561800002, 0.40778803825400001, 0.39824509620699999, 0.39766597747799998, 0.40741491317700002, 0.39641904830899999, 0.39668798446699999, 0.39379596710199999, 0.39647078514099998, 0.393444061279, 0.40611696243299999, 0.39961814880399998, 0.39285993576, 0.39224314689599998, 0.39388489723199999, 0.39147710800199997, 0.39165306091300001, 0.39033508300800002, 0.39437389373800003, 0.39175200462300003, 0.391242027283, 0.39103412628200002, 0.39511799812300002, 0.398463010788, 0.39204406738300002, 0.39300107955899999, 0.39334487914999999, 0.391426801682, 0.39687800407399998, 0.39431095123299997, 0.39070391654999997, 0.39106798172000001, 0.39058089256299999, 0.39391088485699999, 0.39834094047500002, 0.39057397842399999, 0.39198994636500001, 0.39368200302099998, 0.39073801040599998, 0.39122891426099998, 0.39064598083500002, 0.39390277862500001, 0.390840053558, 0.39128398895299998, 0.39098405838, 0.39335107803300001, 0.39786505699199998, 0.39080500602700002], "base_times": [0.348359107971, 0.34692883491499998, 0.346405982971, 0.34667205810500001, 0.34701514244100001, 0.34753990173299998, 0.346569061279, 0.34635281562800002, 0.346190929413, 0.34796619415300001, 0.34590101242100002, 0.34643101692200001, 0.34670686721799998, 0.34775090217600002, 0.34620094299300003, 0.34622883796699999, 0.34623694419899997, 0.34779596328700002, 0.348699092865, 0.34642219543500002, 0.347030878067, 0.34581995010400002, 0.347792863846, 0.34766483306899998, 0.34653806686400002, 0.34683513641399999, 0.34936118125900001, 0.34839010238599999, 0.346399068832, 0.34670805931100002, 0.34576678276099998, 0.348170042038, 0.34661507606500003, 0.34650683402999999, 0.34621381759600001, 0.34878396987900001, 0.34638595581100001, 0.34623980522199999, 0.34593510627700003, 0.34620404243500003, 0.34790611267100002, 0.346742153168, 0.34607505798299998, 0.34633517265300001, 0.34812307357799999, 0.34604907035799998, 0.34677386283900002, 0.34667706489599998, 0.34659886360199998, 0.34881806373599999]}], ["nbody_modified", "RawResult", {"changed_times": [0.12817502021800001, 0.114922046661, 0.105006933212, 0.10339498519900001, 0.104254007339, 0.103782892227, 0.103751897812, 0.105894088745, 0.104180812836, 0.103867053986, 0.103599071503, 0.103564023972, 0.103793859482, 0.103470087051, 0.103558063507, 0.103418111801, 0.10340905189500001, 0.10347700119, 0.103801012039, 0.10350394249, 0.103738069534, 0.10368895530699999, 0.106331110001, 0.103569984436, 0.10441899299600001, 0.103644132614, 0.103709936142, 0.10339307785, 0.103735923767, 0.10362410545300001, 0.103760004044, 0.103440999985, 0.10381603240999999, 0.103622913361, 0.10393595695500001, 0.103415966034, 0.103775978088, 0.106086969376, 0.103862047195, 0.103460073471, 0.103792905807, 0.10356497764600001, 0.103854894638, 0.103960990906, 0.103703975677, 0.103446006775, 0.10406708717300001, 0.103525876999, 0.103811979294, 0.10510087013199999], "base_times": [0.64283800125099999, 0.64096808433499997, 0.64577603340099998, 0.64423298835800002, 0.64046311378499998, 0.63754677772500001, 0.63747310638400001, 0.63725614547700005, 0.63658809661899995, 0.64228487014799995, 0.64188194274900001, 0.64234304428099998, 0.63921594619800004, 0.63971495628399999, 0.64251685142500004, 0.64014887809800003, 0.64212799072299998, 0.64027500152600003, 0.63313889503499998, 0.63153290748599999, 0.63076686859099995, 0.63370203971899997, 0.63236403465299995, 0.63347291946399997, 0.63246107101399995, 0.63299584388700003, 0.63348007202099998, 0.63026380538899995, 0.632724046707, 0.63243818283099995, 0.63531398773199999, 0.63819217681899998, 0.63529610633900002, 0.63803815841699996, 0.63671016693100002, 0.64081311225899995, 0.63594484329199996, 0.64017796516400005, 0.64193677902199997, 0.64040708541900004, 0.64205718040500004, 0.64145493507399998, 0.640824079514, 0.65274095535300003, 0.64189887046799998, 0.64417505264300001, 0.64191007614100004, 0.640285015106, 0.63908886909499996, 0.64495897293100002]}], ["pyflate-fast", "RawResult", {"changed_times": [1.83701682091, 1.8073658943199999, 1.8406269550300001, 1.82018709183, 1.8337500095399999, 1.8074929714200001, 1.84810090065, 1.8087329864499999, 1.8338890075700001, 1.81683397293, 1.78763699532, 1.79593896866, 1.81039500237, 1.7986569404599999, 1.8007159233100001, 1.8014991283399999, 1.7972960472099999, 1.80028700829, 1.8007249832200001, 1.85811805725, 1.7931230068199999, 1.82364487648, 1.7935469150500001, 1.82844901085, 1.7920200824700001, 1.82318806648, 1.7929878234900001, 1.82861304283, 1.79402518272, 1.82314896584, 1.79180502892, 1.8225588798500001, 1.80868387222, 1.80998492241, 1.8257219791399999, 1.8084599971799999, 1.81027412415, 1.80907607079, 1.80926918983, 1.82390213013, 1.8229339122799999, 1.9030389785799999, 1.9522318839999999, 1.8275589942899999, 1.81771707535, 1.8227140903500001, 1.8171408176399999, 1.8353171348599999, 1.8287210464500001, 1.83105301857], "base_times": [3.1601369381, 3.17563796043, 3.1705720424699999, 3.17048382759, 3.1654319763199998, 3.1680698394800002, 3.1681051254299999, 3.1718590259599999, 3.1830987930300001, 3.1775770187400001, 3.1700208187099999, 3.1612432002999999, 3.15727186203, 3.1732730865500001, 3.1751198768600002, 3.1624131202700001, 3.1771841049199998, 3.1661839485200001, 3.16439390182, 3.1774570941900002, 3.1796469688400002, 3.16518807411, 3.1729710102099999, 3.1790311336500001, 3.17372894287, 3.1641249656700001, 3.1653139591200001, 3.16444206238, 3.1757740974400002, 3.16623592377, 3.1762218475299999, 3.1651339530899998, 3.1656432151799998, 3.1724681854200001, 3.1735718250299998, 3.1667320728299999, 3.1666581630700001, 3.1662278175399998, 3.1711161136600001, 3.1634638309500001, 3.1635270118699998, 3.1630849838300001, 3.1776990890499999, 3.17032909393, 3.1747660636899999, 3.1702299117999999, 3.17133522034, 3.1701278686499998, 3.1603660583500002, 3.1653149128]}], ["raytrace-simple", "RawResult", {"changed_times": [0.659919023514, 0.66626596450800002, 0.65816497802700002, 0.65398693084699999, 0.63991713523899996, 0.66545104980500003, 0.65175795555100002, 0.63982892036399996, 0.66220998764000005, 0.65392279624899996, 0.63932204246500002, 0.65032505989099998, 0.65231609344499997, 0.63614416122399997, 0.65278100967399999, 0.63543891906700001, 0.65301513671900002, 0.647172212601, 0.63230013847400002, 0.64931488037100005, 0.689571857452, 0.63732600212099999, 0.67306900024399996, 0.63961911201499999, 0.650694131851, 0.63388586044299999, 0.65063095092800005, 0.64781498909000002, 0.63850116729700002, 0.65920114517200001, 0.68630504608200005, 0.63646793365499998, 0.64802885055500004, 0.64920401573200004, 0.63225889205899999, 0.65038490295399998, 0.63108992576599998, 0.64513182640099997, 0.63317394256600001, 0.64602303504900005, 0.64737510681199995, 0.63001990318300005, 0.64657092094400004, 0.63210487365699997, 0.645840883255, 0.65991592407199995, 0.62948513030999997, 0.65090513229400004, 0.63588213920600001, 0.66070103645300005], "base_times": [2.6739389896399999, 2.6792252063799999, 2.67714691162, 2.6755440235100001, 2.6839830875400001, 2.67896699905, 2.6759178638500001, 2.6771140098599999, 2.6831061840100001, 2.6781611442600002, 2.6843791008000002, 2.67555809021, 2.67742395401, 2.68145084381, 2.6856698989900001, 2.6754090786, 2.6820809841200002, 2.7008218765300001, 2.6860649585699998, 2.68976688385, 2.7091708183300001, 2.6831979751600001, 2.6825587749499999, 2.7229101657900001, 2.7227280139899999, 2.7074718475299999, 2.7032160758999999, 2.69513916969, 2.7023360729200001, 2.7511570453599998, 2.6899399757400002, 2.6838908195500002, 2.7002210617100002, 2.8071360588099998, 2.6863989830000001, 2.6864531040199999, 2.69359779358, 2.7453310489699998, 2.7076761722599998, 3.06150817871, 2.7332241535200001, 2.7483820915199999, 2.7639210224199999, 3.4124541282699998, 2.7048859596299999, 2.6892578601800001, 2.6912610530899999, 2.6821250915500001, 2.68154597282, 2.6890358924900002]}], ["richards", "RawResult", {"changed_times": [0.12052297592199999, 0.093158006668099994, 0.091630935668900002, 0.091804027557399998, 0.091670036315899997, 0.091012954711900004, 0.091180086135899993, 0.102787017822, 0.090846061706500003, 0.090148925781199998, 0.093178987503099997, 0.090260028839100004, 0.090500116348300005, 0.098414897918700006, 0.090541839599599994, 0.090975999832200005, 0.090515136718800002, 0.088860034942600002, 0.089382886886599996, 0.087617158889799998, 0.087574005127000004, 0.088225841522200005, 0.087996006012, 0.087512969970700002, 0.0879878997803, 0.096507072448699993, 0.087088108062699995, 0.090749025344800005, 0.087268829345700002, 0.087903976440400003, 0.087549924850500005, 0.087819099426300007, 0.087373971939100001, 0.087810039520300004, 0.087095975875899997, 0.0871419906616, 0.088514089584400005, 0.087375879287700006, 0.086944818496699994, 0.087454080581699997, 0.087461948394799996, 0.097897052764900003, 0.087609052658099998, 0.087203979492200004, 0.089391946792599999, 0.088086128234900002, 0.086940050125100005, 0.087315082550000006, 0.086781978607199994, 0.086982011794999994], "base_times": [0.35359501838700003, 0.35667300224300003, 0.347419977188, 0.35246205329899999, 0.35532999038700003, 0.35530591011000001, 0.35107088088999999, 0.35549902915999998, 0.35110902786300002, 0.35400104522699999, 0.35261011123699998, 0.35541200637800002, 0.35351705551099999, 0.35617089271500002, 0.34958410263099998, 0.35414409637499999, 0.35661983489999999, 0.35693001747100001, 0.35174703598000001, 0.34876704215999998, 0.35491299629200002, 0.35479497909500002, 0.35416102409400002, 0.355372905731, 0.35764288902300001, 0.35487508773799997, 0.35283422470100001, 0.35428977012599999, 0.35132908821100001, 0.35751414299000001, 0.362582206726, 0.35600781440700002, 0.35644507408100001, 0.35086202621500001, 0.35400605201699997, 0.34989190101599998, 0.35055398940999999, 0.35048794746400003, 0.35601019859299998, 0.35249996185299998, 0.35182499885599999, 0.35495114326499999, 0.35285496711699998, 0.355787038803, 0.35039997100800002, 0.35504388809199999, 0.35427594184900002, 0.35611510276800001, 0.35387182235699999, 0.34967398643499997]}], ["spambayes", "RawResult", {"changed_times": [0.48457980155899999, 0.56999397277800001, 0.53518199920700005, 0.54824614524799997, 0.49303603172299998, 0.58462905883799998, 0.51067399978600003, 0.49086999893200001, 0.48435497283899998, 0.42737793922400003, 0.494954109192, 0.43986797332799998, 0.40522003173799997, 0.40721011161800003, 0.57795190811200003, 0.48255205154399999, 0.462779998779, 0.44051218032799999, 0.39301896095299999, 0.39432501792899999, 0.39888882637, 0.43233299255399998, 0.42077803611800002, 0.39932012558000002, 0.40041995048500001, 0.409831047058, 0.44888401031500003, 0.40377283096299998, 0.39395999908399998, 0.40184497833299998, 0.41734480857799999, 0.49758601188700002, 0.39052391052199997, 0.392160177231, 0.39913892746000001, 0.42745113372799998, 0.36878705024699998, 0.38831090927099998, 0.37554693222000002, 0.40881800651599998, 0.37994885444600002, 0.37922286987300002, 0.39221501350400001, 0.39279913902300001, 0.34234404563900001, 0.37785792350800002, 0.38006091117899998, 0.37980985641499998, 0.375820875168, 0.345314979553], "base_times": [0.29905700683600001, 0.298835992813, 0.29825806617700001, 0.29831504821799998, 0.30105614662199998, 0.298439979553, 0.29832100868200001, 0.29835200309799997, 0.29867696762099999, 0.29994201660199998, 0.29847502708399998, 0.29867196083100001, 0.29891991615300001, 0.29807305336000001, 0.30059790611300002, 0.298420906067, 0.29828691482500003, 0.29897713661199998, 0.29811811447100001, 0.300410985947, 0.29847979545600001, 0.2982878685, 0.29861903190599998, 0.29877710342399999, 0.30105996131899998, 0.29816794395399998, 0.29835987091100002, 0.29860401153600002, 0.2985060215, 0.30007410049400002, 0.29955601692200001, 0.299430847168, 0.29854393005399998, 0.29915118217499997, 0.29966092109699999, 0.29980802536000001, 0.29794311523400002, 0.29930615425099999, 0.298702001572, 0.29842782020600001, 0.30247402191200001, 0.29896807670600001, 0.298580169678, 0.29869008064300001, 0.299312114716, 0.30094408988999999, 0.29845499992399999, 0.29868721962, 0.29998612403899999, 0.29916310310400002]}], ["spectral-norm", "RawResult", {"changed_times": [0.18530607223500001, 0.118788957596, 0.11839389801, 0.117673158646, 0.115185976028, 0.115034103394, 0.114280939102, 0.115108966827, 0.115477085114, 0.114944934845, 0.11555695533800001, 0.111644983292, 0.111333847046, 0.111480951309, 0.11155700683600001, 0.11224794387799999, 0.112046003342, 0.111447095871, 0.11138176918000001, 0.11152100563, 0.111184120178, 0.11187982559200001, 0.111309051514, 0.115062952042, 0.111434936523, 0.111573934555, 0.111452102661, 0.128764867783, 0.11133313179, 0.11140704155, 0.11155796051, 0.111167907715, 0.111421108246, 0.11097002029399999, 0.111446857452, 0.110933065414, 0.110732078552, 0.113990068436, 0.11116909980799999, 0.111049890518, 0.11179399490399999, 0.11199402809099999, 0.110781908035, 0.111202001572, 0.111080884933, 0.110934972763, 0.111124992371, 0.110868930817, 0.11123108863800001, 0.112289190292], "base_times": [0.48088502883899997, 0.47923994064300002, 0.48160886764499999, 0.47763085365300001, 0.47884607315099997, 0.48438310623199998, 0.48113894462599999, 0.47794389724699998, 0.47896790504499998, 0.48052501678499998, 0.47819399833699999, 0.47891688346900002, 0.48100805282600001, 0.48024415969799999, 0.47854995727499999, 0.47899198532100001, 0.47972202301, 0.47723197937, 0.48147606849699998, 0.47865796089200002, 0.47799110412599999, 0.47740817070000002, 0.47738909721400002, 0.47732496261599999, 0.47762703895600001, 0.48322010040300001, 0.47945380210900002, 0.47645592689499999, 0.479950904846, 0.47760391235400002, 0.47802209854099997, 0.47923922538800001, 0.477582931519, 0.47682905197100001, 0.47848796844500002, 0.47821402549699998, 0.47620606422400003, 0.48374009132399998, 0.481714963913, 0.47922301292399999, 0.47735190391499999, 0.48569798469499997, 0.47899794578600002, 0.477490901947, 0.47891783714300001, 0.47670793533299999, 0.47655487060500001, 0.47783589363099999, 0.47823095321699999, 0.47739887237500001]}], ["telco", "RawResult", {"changed_times": [0.56803599999999999, 0.51203200000000004, 0.51603200000000005, 0.54403400000000002, 0.51203200000000004, 0.53603400000000001, 0.492031, 0.496031, 0.49203000000000002, 0.51203200000000004, 0.496031, 0.492031, 0.500031, 0.492031, 0.50403100000000001, 0.50403200000000004, 0.496031, 0.492031, 0.49203000000000002, 0.50403200000000004, 0.492031, 0.492031, 0.51203200000000004, 0.49203000000000002, 0.49203200000000002, 0.50403100000000001, 0.492031, 0.49203000000000002, 0.50403200000000004, 0.48803000000000002, 0.496031, 0.50403100000000001, 0.48803099999999999, 0.492031, 0.50003200000000003, 0.49203000000000002, 0.492031, 0.496031, 0.492031, 0.492031, 0.500031, 0.496031, 0.492031, 0.500031, 0.492031, 0.48803099999999999, 0.50403100000000001, 0.48803099999999999, 0.492031, 0.500031], "base_times": [1.22, 1.2, 1.21, 1.21, 1.2, 1.21, 1.22, 1.2, 1.2, 1.22, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.22, 1.21, 1.21, 1.21, 1.2, 1.2, 1.22, 1.2, 1.21, 1.21, 1.2, 1.2, 1.21, 1.21, 1.2, 1.22, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.2, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21]}], ["twisted_names", "RawResult", {"changed_times": [0.0074571215510812828, 0.0072046109510086453, 0.0071581961345740875, 0.0076745970836531079, 0.007215007215007215, 0.007446016381236039, 0.0072674418604651162, 0.0073909830007390983, 0.007320644216691069, 0.007246376811594203, 0.007446016381236039, 0.0071684587813620072, 0.0073691967575534268, 0.0070372976776917661, 0.0071530758226037196, 0.0074239049740163323, 0.0070721357850070717, 0.0070077084793272598, 0.0074794315632011965, 0.007052186177715092, 0.0075815011372251705, 0.0074850299401197605, 0.0069979006298110571, 0.0071174377224199285, 0.0075187969924812026, 0.0072833211944646759, 0.007763975155279503, 0.006939625260235947, 0.006939625260235947, 0.007331378299120235, 0.006954102920723227, 0.006954102920723227, 0.006939625260235947, 0.0073421439060205578, 0.0069156293222683261, 0.0068917987594762234, 0.0073746312684365781, 0.0069444444444444441, 0.006920415224913495, 0.0073637702503681884, 0.0069492703266157054, 0.0069300069300069298, 0.0070571630204657732, 0.0069108500345542506, 0.0068917987594762234, 0.0069348127600554789, 0.0073046018991964941, 0.0069637883008356544, 0.006954102920723227, 0.0073583517292126564], "base_times": [0.0096246390760346481, 0.0095877277085330784, 0.0095419847328244278, 0.0095877277085330784, 0.0095602294455066923, 0.0095785440613026813, 0.0095693779904306216, 0.0095510983763132766, 0.0095419847328244278, 0.0099009900990099011, 0.0095693779904306216, 0.0095419847328244278, 0.0096061479346781949, 0.0095693779904306216, 0.0095510983763132766, 0.0095969289827255271, 0.0095785440613026813, 0.0095602294455066923, 0.0095785440613026813, 0.0095969289827255271, 0.0095785440613026813, 0.0096061479346781949, 0.0095602294455066923, 0.0095877277085330784, 0.0095877277085330784, 0.0095602294455066923, 0.0095877277085330784, 0.0095510983763132766, 0.0095328884652049577, 0.0095877277085330784, 0.0096061479346781949, 0.0095693779904306216, 0.0096061479346781949, 0.0095785440613026813, 0.0095693779904306216, 0.0095693779904306216, 0.0095785440613026813, 0.0095510983763132766, 0.0095877277085330784, 0.0095877277085330784, 0.0095785440613026813, 0.0095785440613026813, 0.0095969289827255271, 0.0095693779904306216, 0.0095877277085330784, 0.0095602294455066923, 0.0095419847328244278, 0.0095877277085330784, 0.0095785440613026813, 0.0095785440613026813]}]], "branch": "trunk", "revision": 0} \ No newline at end of file Added: pypy/extradoc/talk/pepm2011/benchmarks/psyco.json ============================================================================== --- (empty file) +++ pypy/extradoc/talk/pepm2011/benchmarks/psyco.json Wed Oct 13 13:30:14 2010 @@ -0,0 +1 @@ +{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [0.24086999893200001, 0.068964004516599994, 0.067575216293300003, 0.0673339366913, 0.067570924758900003, 0.067430019378699996, 0.067543029785200007, 0.067275047302200006, 0.067908048629800002, 0.069012880325299994, 0.072937965393100004, 0.067466974258399995, 0.067596912383999994, 0.067381858825699997, 0.068427085876499996, 0.067336082458499993, 0.067768096923799995, 0.067370176315299998, 0.067598104476899998, 0.067315101623500004, 0.068055152893100004, 0.067442893981899996, 0.067678928375199995, 0.067797183990499998, 0.072264909744299999, 0.067381858825699997, 0.068480014800999994, 0.067888975143400004, 0.072933912277200003, 0.067322969436600003, 0.067459106445299996, 0.0673580169678, 0.067704916000400003, 0.067392826080300006, 0.067551851272599994, 0.067342996597300003, 0.067787170410200007, 0.067278146743800005, 0.067821979522699999, 0.0673339366913, 0.067644119262700003, 0.067193984985399996, 0.067703962325999997, 0.067339181900000003, 0.067665100097700007, 0.067229032516500004, 0.067646026611299995, 0.067302942276000005, 0.067595005035400002, 0.067299127578700005], "base_times": [2.99749398232, 2.9997680187200002, 2.9963829517399998, 2.9968049526199998, 2.9959740638699999, 2.9977769851699998, 2.99551987648, 2.9970090389299999, 2.99462604523, 2.9960889816299998, 2.9945168495200001, 2.9982719421400001, 2.9949910640700002, 2.9980101585400001, 2.9997220039400001, 3.0011110305800002, 2.9947130680099998, 2.9989519119299999, 2.9928319454199999, 2.9970700740799998, 2.9944999218000001, 3.0004448890700002, 2.9967379570000001, 2.99714899063, 2.9953291416200001, 2.99760389328, 3.0011970996900001, 2.9919300079300002, 2.9997839927699999, 2.9920029640200001, 3.00179791451, 2.9937269687699999, 2.99896001816, 3.0404541492499999, 3.1035900116000001, 3.0030460357700002, 2.9972250461600001, 2.9991810321800001, 2.99679613113, 2.9955308437300001, 2.99544000626, 2.99763393402, 2.9966008663200001, 2.99617195129, 2.9960741996800002, 2.99449300766, 2.9985389709499999, 2.9971520900700002, 2.9952371120499999, 2.9977967739100002]}], ["django", "RawResult", {"changed_times": [0.91241788864100004, 0.90872383117699995, 0.92016100883499996, 0.89697694778399994, 0.91372919082600002, 0.90660500526400001, 0.90812492370599995, 0.92440700530999997, 0.90569281578100003, 0.92590403556800005, 0.91092205047599994, 0.89118409156800005, 0.91204190254200002, 0.91047716140699997, 0.898652076721, 0.90288305282600001, 0.90163207054100003, 0.91061997413600004, 0.91161584854099997, 0.91608476638799996, 0.933578968048, 0.91335201263400001, 0.93553304672199999, 0.92455196380600002, 0.92451906204199996, 0.91092705726599998, 0.90619492530800005, 0.90510320663499999, 0.92645692825299997, 0.90040588378899999, 0.92053794860799998, 0.92685198783900002, 0.906805038452, 0.89644694328300001, 0.92246198654199996, 0.89942693710300003, 0.91932320594799999, 0.90771389007600001, 0.91687607765199997, 0.89139580726599998, 0.93094897270200005, 0.90798401832599995, 0.90613412857099995, 0.92559409141500004, 0.89797496795700005, 0.90965700149499995, 0.90096592903100003, 0.91928386688200003, 0.90714716911299997, 0.91119408607499996], "base_times": [0.98527598381000003, 0.98649215698199999, 0.98675608634900003, 0.98700690269500002, 0.98524284362799996, 0.98649191856399998, 0.98681497573900001, 0.99080705642699995, 0.98529195785500001, 0.98713397979700002, 0.98623895645100002, 0.98769807815599997, 0.984663009644, 0.98703598976100004, 0.98637104034400003, 0.98750090599100004, 0.98651099205000004, 0.98755002021799998, 0.98614001274100005, 0.98810005188000005, 0.98580408096299998, 0.987119913101, 0.98672580719000003, 0.99214601516699996, 0.98567104339599998, 0.99467992782600001, 0.98890900611900001, 0.99003291130100002, 0.988044977188, 0.98867988586400002, 0.98798298835800002, 0.988909959793, 0.98743391036999995, 0.98779296875, 0.98777294158899998, 0.99035811424300002, 0.98490500450100005, 0.98681116104099997, 0.98640084266700001, 0.98687696456899998, 0.98529601097099995, 0.98615694046000002, 0.98579502105700001, 0.98772692680399998, 0.98597598075899995, 0.98605084419300004, 0.98599410057100001, 0.98663592338600004, 0.98617005348200004, 0.98613810539199998]}], ["fannkuch", "RawResult", {"changed_times": [2.0868110656700001, 0.95944404602099997, 0.94198584556599996, 0.94393110275299996, 0.94214797019999996, 0.944802045822, 0.94370889663699997, 0.94486498832700005, 0.94305491447400003, 0.94709801673899996, 0.94345903396599995, 0.94617295265199997, 0.94330191612199998, 0.94647693633999996, 0.94317007064799996, 0.94647479057299999, 0.94380092620800005, 0.94613718986499995, 0.94333314895599996, 0.946511983871, 0.94391989707900004, 0.94642496108999996, 0.94316697120699999, 0.94675111770599996, 0.943277835846, 0.94598197936999995, 0.94335103035000001, 0.94561409950300002, 0.94404387474100004, 0.94574284553499999, 0.94397807121300004, 0.94565796852100004, 0.944031953812, 0.94567799568199995, 0.947828054428, 0.94561696052599997, 0.94435000419600001, 0.94418907165499999, 0.94202494621300004, 0.94604516029399999, 0.94264101982100001, 0.94473910331699995, 0.94210100174, 0.944357156754, 0.94217014312699998, 0.94522094726600003, 0.94141697883599995, 0.94716501235999995, 0.94124603271499996, 0.94444704055799999], "base_times": [1.9176969528200001, 1.9202299117999999, 1.9235260486600001, 1.9171619415300001, 1.92679715157, 1.91401004791, 1.90943288803, 1.92051100731, 1.9146890640300001, 1.9129071235699999, 1.9229171276100001, 1.9258570671099999, 1.91273188591, 1.9221320152300001, 1.9251158237499999, 1.9122190475500001, 1.9216961860699999, 1.92365598679, 1.91404604912, 1.9210500717200001, 1.9241530895200001, 1.9115061759900001, 1.9218940734900001, 1.9283528327899999, 1.91030097008, 1.9220068454699999, 1.9232158660900001, 1.9122979641, 1.92231798172, 1.923869133, 1.9122321605699999, 1.9220750331900001, 1.9251158237499999, 1.91487503052, 1.9224019050600001, 1.9240219593000001, 1.9117150306699999, 1.9220988750500001, 1.92301011086, 1.9119341373400001, 1.9218909740400001, 1.9251511097, 1.91221499443, 1.92299008369, 1.9239330291700001, 1.9131119251299999, 1.9216499328600001, 1.9235699176800001, 1.9122760295900001, 1.92265415192]}], ["go", "RawResult", {"changed_times": [0.44916915893600001, 0.45678114891100002, 0.45058012008699999, 0.44536614417999998, 0.44658303260799997, 0.44329500198400001, 0.45158696174599999, 0.44648003578200002, 0.44638085365300001, 0.44618391990700002, 0.44584703445399998, 0.446818113327, 0.45075201988199998, 0.44904303550699998, 0.446666002274, 0.44573283195500002, 0.444730997086, 0.444979906082, 0.44973683357200001, 0.44689011573800003, 0.44517397880600001, 0.44268488884000001, 0.445667982101, 0.44369792938199998, 0.45045089721699999, 0.445962905884, 0.44611811637900001, 0.44507789611800003, 0.44434905052200002, 0.44469594955399999, 0.45005702972400002, 0.44598793983500001, 0.44651508331299999, 0.444850206375, 0.44458484649699997, 0.44302582740800001, 0.44856619834900002, 0.44638204574599999, 0.44514584541300001, 0.44602012634299998, 0.44548392295799999, 0.44940400123599999, 0.445560932159, 0.44690299034100001, 0.44452691078200002, 0.44488596916200002, 0.445541858673, 0.44877910613999999, 0.44740295410199998, 0.44541597366300001], "base_times": [0.93293118476900005, 0.93541598319999997, 0.93207883834799998, 0.95246386528000004, 0.93352389335599995, 0.93661594390900005, 0.93178510665900005, 0.94123816490199996, 0.93654799461399996, 0.934614181519, 0.93224811553999998, 0.93286204338099998, 0.93584704399100005, 0.938571929932, 0.93282914161700003, 0.93698310852099997, 0.93200492858899997, 0.931174039841, 0.93693208694499996, 0.93402409553499999, 0.93588519096400002, 0.93242597579999997, 0.932387828827, 0.935797929764, 0.93831300735500001, 0.93114900589000005, 0.93581485748299997, 0.93277502059899997, 0.93401098251299997, 0.92857909202599997, 0.94130086898800003, 0.93222689628599997, 0.93134689331099996, 0.93450593948399996, 0.93423318862899996, 0.93199896812399996, 0.93972015380899998, 0.93394899368299999, 0.93780016899100005, 0.93434810638400001, 0.93742513656600002, 0.93885803222700004, 0.93848896026600004, 0.93989205360399997, 0.93258023262, 0.93085002899199998, 0.93369913101199997, 0.93829298019399998, 0.93431401252699997, 0.931654930115]}], ["html5lib", "RawResult", {"changed_times": [17.196844101, 17.448256015799998, 17.206418991100001, 17.204179048499999, 17.097077131300001, 17.348690986600001, 17.4630849361, 17.426965951900002, 17.369714975400001, 17.3314421177, 17.309705018999999, 17.472054958299999, 17.4677460194, 17.627110004399999, 17.309695005399998, 17.354887962300001, 17.301031827900001, 17.5074188709, 17.399863004699998, 17.361423969299999, 17.425707817100001, 17.253575086600001, 17.3923199177, 17.507417201999999, 17.436900854099999, 17.491322994200001, 17.48346591, 17.486616849899999, 17.415644884100001, 17.466309070600001, 17.537580013300001, 17.4069769382, 17.421916008, 17.242964983, 17.146942853900001, 17.3513269424, 17.455329894999998, 17.4232571125, 17.383727073700001, 17.250620126699999, 17.350507974599999, 17.440768003500001, 17.459242820699998, 17.337256908400001, 17.392287969600002, 17.413850068999999, 17.198677063000002, 17.520031213799999, 17.558635950100001, 17.296360015899999], "base_times": [13.754509925800001, 13.942677021, 13.855687856699999, 13.845515966400001, 13.847697973300001, 13.9333691597, 13.9368638992, 13.9086000919, 13.875296115899999, 13.853040933600001, 13.8222680092, 13.9552228451, 13.938019990900001, 13.8921999931, 13.853141069399999, 13.839344024700001, 13.7949540615, 13.9310882092, 13.953261852300001, 13.9194419384, 13.9146289825, 13.8520970345, 13.809508085299999, 13.9844410419, 13.966258049, 13.913993120200001, 13.868940114999999, 13.825479984299999, 13.8008880615, 13.973448038100001, 13.9451079369, 13.9387960434, 13.8836650848, 13.8697099686, 13.8277349472, 13.9671690464, 13.9775838852, 13.907384157199999, 13.9473381042, 13.8945560455, 13.9592449665, 13.9714989662, 13.942196846, 13.9209439754, 13.902752876299999, 13.866322994200001, 13.8447039127, 13.8183758259, 13.9480679035, 13.9563720226]}], ["meteor-contest", "RawResult", {"changed_times": [0.22766804695100001, 0.216835021973, 0.214914083481, 0.215392827988, 0.21503186225900001, 0.214871883392, 0.214845895767, 0.21476101875299999, 0.21559500694299999, 0.21447587013200001, 0.21465492248500001, 0.21474695205700001, 0.21479892730700001, 0.214986801147, 0.21476602554300001, 0.214766979218, 0.21477794647199999, 0.21497702598599999, 0.21569609642000001, 0.21472907066300001, 0.214810848236, 0.21513891220100001, 0.215142965317, 0.215198040009, 0.215027093887, 0.21522903442399999, 0.215254068375, 0.216848134995, 0.215054035187, 0.21492290496800001, 0.21528983116100001, 0.21517300605799999, 0.215394020081, 0.216041088104, 0.21514177322399999, 0.21530199050900001, 0.215008974075, 0.21482992172199999, 0.21607804298399999, 0.21591496467599999, 0.216004133224, 0.21603202819799999, 0.216161966324, 0.21632885932900001, 0.21636295318599999, 0.216923952103, 0.21622109413099999, 0.216312885284, 0.21637821197500001, 0.21630501747100001], "base_times": [0.34524512290999998, 0.34548401832600001, 0.34518408775300002, 0.34471106529200002, 0.34508490562400002, 0.34438800811800002, 0.34416389465300001, 0.34460997581500002, 0.34558606147799997, 0.34469914436299998, 0.34492015838599999, 0.34501504898099999, 0.34493398666399999, 0.34555506706200001, 0.34438300132799998, 0.34434390068100001, 0.34486389160199998, 0.34501290321400002, 0.34506106376599999, 0.34580302238499999, 0.34507203102099998, 0.34458112716700001, 0.34469914436299998, 0.34462499618499998, 0.34500002861000001, 0.34571313858000002, 0.34487700462300003, 0.34526395797699999, 0.34495615959199999, 0.34477305412300002, 0.34441876411400002, 0.34517693519600001, 0.34503078460699999, 0.34493207931499997, 0.34471702575699997, 0.34489202499400001, 0.34555888175999999, 0.34510707855200001, 0.344701051712, 0.344593048096, 0.34483194351200003, 0.345004081726, 0.34531402587900001, 0.34469985961900002, 0.34493589401199998, 0.34466695785500001, 0.34443211555499997, 0.34493017196699999, 0.34593892097500001, 0.34549784660299998]}], ["nbody_modified", "RawResult", {"changed_times": [0.64423489570600001, 0.25821900367700001, 0.25713014602700002, 0.25906801223800002, 0.25766491889999998, 0.25762987136799997, 0.25773000717200001, 0.25788497924800002, 0.25927114486699998, 0.25772809982299999, 0.257742166519, 0.25765109062199998, 0.25812292099, 0.258606910706, 0.25697898864699997, 0.25673794746400003, 0.256858110428, 0.25740098953200002, 0.25779008865399999, 0.25714015960699999, 0.25731110572799998, 0.25716495513900001, 0.25704193115200002, 0.25666093826300002, 0.25757193565399999, 0.25681495666499998, 0.25823497772199999, 0.25659799575800002, 0.25655198097199999, 0.25701594352700002, 0.25630092620799999, 0.256217956543, 0.256570100784, 0.25721597671500002, 0.25766992569000002, 0.257313013077, 0.25665688514700002, 0.25670599937400002, 0.25656700134299998, 0.25662589073199998, 0.25648212432900003, 0.25651192665099998, 0.25721383094799999, 0.25626397132899997, 0.25642418861400001, 0.25654816627499999, 0.25621700286900001, 0.255903959274, 0.25610113143899998, 0.25687408447299998], "base_times": [0.606163978577, 0.60316300392199995, 0.61118078231799999, 0.60724902153000004, 0.603479146957, 0.600904226303, 0.60342693328899999, 0.60212087631199995, 0.60914492607100001, 0.60772490501400001, 0.60486197471600001, 0.60462021827699997, 0.60468196868900004, 0.602827072144, 0.60509490966799995, 0.60543298721299998, 0.605483055115, 0.60416388511700003, 0.60476493835400003, 0.60451793670700005, 0.60463404655499997, 0.60426592826799996, 0.6060090065, 0.60466003418000003, 0.60642600059499996, 0.60611486434899997, 0.60610508918799999, 0.60284781456000003, 0.60367608070400003, 0.603693008423, 0.60347199439999999, 0.60303306579600002, 0.60393404960599995, 0.60278487205499998, 0.60334587097199999, 0.60244584083599995, 0.60166096687299997, 0.60149288177500004, 0.60218286514300001, 0.60322499275200003, 0.60127496719399998, 0.601732969284, 0.60190510749799997, 0.60083103179899999, 0.59951996803300001, 0.59986305236799997, 0.59898996353099998, 0.59933304786700003, 0.59813284873999994, 0.59744501113899995]}], ["pyflate-fast", "RawResult", {"changed_times": [1.2742009162900001, 1.2612369060499999, 1.28401517868, 1.28018116951, 1.2790110111199999, 1.2954878807100001, 1.28488206863, 1.28994083405, 1.27809095383, 1.2824020385699999, 1.2826840877500001, 1.2910630702999999, 1.2841398716000001, 1.2740240097, 1.27432203293, 1.2724800109900001, 1.2929151058199999, 1.27352023125, 1.2630290985099999, 1.2715139389000001, 1.2828531265300001, 1.30801796913, 1.2725479602800001, 1.28313684464, 1.2696249485, 1.2616369724300001, 1.2793409824399999, 1.2647218704200001, 1.2773220539100001, 1.26942896843, 1.2831149101299999, 1.2660479545600001, 1.27586102486, 1.27837896347, 1.28248405457, 1.2833318710299999, 1.2803418636299999, 1.2794771194500001, 1.2777268886599999, 1.2776129245800001, 1.2947490215299999, 1.28232908249, 1.28013205528, 1.2718667984000001, 1.27652716637, 1.2749559879300001, 1.27654790878, 1.27595591545, 1.2849490642500001, 1.27388691902], "base_times": [3.1746640205399999, 3.1804258823399998, 3.18864703178, 3.1822848320000001, 3.18055701256, 3.1810128688799999, 3.1893451213800001, 3.17621803284, 3.1973741054499998, 3.1734278202100001, 3.2152090072599999, 3.1938381195100001, 3.17906212807, 3.1824810504899999, 3.1831138133999999, 3.1812291145299998, 3.1951298713699998, 3.18023300171, 3.1814811229700002, 3.18463993073, 3.1928250789599999, 3.1773390770000001, 3.18154597282, 3.1812200546299998, 3.19782304764, 3.1794288158400001, 3.18299722672, 3.1785600185399998, 3.18470001221, 3.1789801120800001, 3.1828451156600002, 3.1784930229200001, 3.19799184799, 3.1770670414, 3.1846189498899999, 3.1840870380399999, 3.1804630756400001, 3.1793830394699998, 3.1800920963300001, 3.1793489456200001, 3.1926109790799999, 3.1778960228000002, 3.19325399399, 3.1858689784999998, 3.2012858390800001, 3.1838262081100002, 3.1853210926100002, 3.17996501923, 3.1940989494299998, 3.1812150478399999]}], ["raytrace-simple", "RawResult", {"changed_times": [1.07620501518, 1.0725319385500001, 1.07604408264, 1.07081604004, 1.0746629238100001, 1.06848597527, 1.07144999504, 1.07353186607, 1.07191300392, 1.06764793396, 1.0666182041200001, 1.0714840888999999, 1.0744469165799999, 1.0714991092699999, 1.0746941566499999, 1.06847810745, 1.0670039653800001, 1.07264709473, 1.0731511116000001, 1.062759161, 1.0754990577700001, 1.0738348960899999, 1.0749199390399999, 1.0727381706200001, 1.0736379623400001, 1.06875705719, 1.0730288028699999, 1.0691349506400001, 1.0735499858899999, 1.0720009803799999, 1.0738818645499999, 1.07100605965, 1.06647109985, 1.0699331760399999, 1.07295703888, 1.0721671581300001, 1.0731370449099999, 1.0647718906400001, 1.06833195686, 1.0762178897900001, 1.0769939422599999, 1.0778059959399999, 1.0763518810299999, 1.07419896126, 1.0745780467999999, 1.0709280967699999, 1.07163715363, 1.0731480121600001, 1.07552099228, 1.0727770328499999], "base_times": [2.6755640506699998, 2.6818940639500002, 2.6747961044299999, 2.6823041439100002, 2.6822581291200001, 2.67866396904, 2.67972493172, 2.6763191223099998, 2.6772010326400002, 2.6844789981799999, 2.67685413361, 2.67394709587, 2.6753950119000001, 2.6806318759900001, 2.6780338287399998, 2.6775889396700001, 2.6776020526900002, 2.6761560440099998, 2.6816561222100002, 2.68350195885, 2.6748669147499999, 2.6816148757899998, 2.6792290210699998, 2.6795959472700002, 2.68182587624, 2.68277406693, 2.6740851402299999, 2.72062897682, 2.77934503555, 2.6858270168299998, 2.6787958145099999, 2.6758110523199998, 2.67273712158, 2.6781260967299998, 2.6828680038499999, 2.6729729175600001, 2.6761100292200002, 2.67716908455, 2.6765320301100002, 2.6807610988600001, 2.6783590316799999, 2.6721069812799998, 2.67397713661, 2.6773550510400002, 2.6771290302300002, 2.67722797394, 2.6763949394200002, 2.6750438213300001, 2.6793069839500001, 2.68012309074]}], ["richards", "RawResult", {"changed_times": [0.065301895141599994, 0.063172817230200004, 0.063138961792000001, 0.064099073409999999, 0.063661098480200004, 0.063364028930700006, 0.063547134399400004, 0.063157796859699994, 0.063546895980800003, 0.063879013061499998, 0.063356876373299995, 0.063307046890300006, 0.063264131545999996, 0.065795183181799999, 0.0646030902863, 0.064000844955399996, 0.063102006912199998, 0.063642024993899995, 0.063217878341700007, 0.063143968582200005, 0.062988042831399998, 0.063710927963299996, 0.064306974411000001, 0.063272953033399998, 0.063179016113299999, 0.063326120376600001, 0.064442873001100007, 0.063086986541700002, 0.063616991043099994, 0.063778877258300004, 0.063082933425900004, 0.063648223877000004, 0.063886880874599997, 0.063105106353799997, 0.063647985458400003, 0.064046144485500001, 0.063460111617999998, 0.064031124115000004, 0.063258886337299994, 0.0633749961853, 0.063982963562000006, 0.063292026519799996, 0.063495874404899996, 0.062860965728799997, 0.063717842102100006, 0.063158988952599998, 0.062774181366000006, 0.063338041305499998, 0.063010931014999993, 0.063432931900000003], "base_times": [0.35280799865700002, 0.359627008438, 0.35187888145399998, 0.36519289016700002, 0.35657286644000002, 0.35461711883500002, 0.35608792304999998, 0.356234073639, 0.35131311416599997, 0.35823798179600003, 0.35196018219000003, 0.34958600997900002, 0.35142707824699998, 0.35373997688300002, 0.34900689125099998, 0.35184597969100001, 0.35277295112599999, 0.35084080696100001, 0.35085892677300001, 0.353019952774, 0.35483789444000002, 0.35198116302499999, 0.35653901100199997, 0.34980392456100001, 0.35202193260199999, 0.35297513008100001, 0.35106587410000001, 0.35060191154499998, 0.34863805770900003, 0.35603594779999997, 0.35384202003499998, 0.35121297836299997, 0.35444617271399997, 0.348870038986, 0.35179209709199999, 0.35253095626800002, 0.35237503051800001, 0.34470391273500001, 0.35610294341999998, 0.35717201232899998, 0.35300779342700001, 0.35652518272400002, 0.35759401321399997, 0.35532307624800002, 0.35166001319899998, 0.354029893875, 0.352816104889, 0.35459423065200002, 0.35481095314, 0.35345506668100002]}], ["spambayes", "RawResult", {"changed_times": [0.34045600891099997, 0.32780194282500003, 0.34443902969399998, 0.34438800811800002, 0.33345198631299999, 0.35003399848900002, 0.34026598930399998, 0.33723402023299998, 0.34488701820399997, 0.34427690506000003, 0.33687591552700002, 0.33324790000900001, 0.33831787109400002, 0.34821605682399998, 0.348307132721, 0.33934497833299998, 0.33219695091200002, 0.33459782600400001, 0.33166790008500002, 0.33012008666999998, 0.33208990097000002, 0.33377885818500003, 0.35483503341700001, 0.33462214469899998, 0.32909703254700001, 0.34683609008799998, 0.343517065048, 0.328886985779, 0.364034891129, 0.33743596076999999, 0.331935882568, 0.34463405609100001, 0.35129594802899999, 0.33779311180100002, 0.337424039841, 0.33314609527599998, 0.33569407463099998, 0.34027504921000001, 0.33753299713099999, 0.34884500503499999, 0.34054994583100001, 0.33529496192899999, 0.33340597152700002, 0.32456183433500002, 0.32919597625699998, 0.34403300285299998, 0.34574103355399999, 0.34348893165599997, 0.33085107803300001, 0.329581975937], "base_times": [0.29761099815399999, 0.29877209663400001, 0.29939198494000002, 0.29858589172400002, 0.29960298538199998, 0.30084013938900001, 0.298014163971, 0.298666000366, 0.298691987991, 0.29817199707000003, 0.29828095436099999, 0.298875808716, 0.29877400398300002, 0.29845499992399999, 0.29812312126200002, 0.299697160721, 0.29821205139200002, 0.29817700386000001, 0.29894804954499998, 0.29830908775300002, 0.29822707176199997, 0.29805707931499997, 0.298221111298, 0.29915213584900002, 0.29897093772900002, 0.30103397369399998, 0.29833221435500001, 0.29866790771500001, 0.29847502708399998, 0.298280000687, 0.29843211174000001, 0.29984307289099998, 0.29922294616700001, 0.29815006256100002, 0.298325061798, 0.29904103279100003, 0.29848790168799999, 0.29830098152200002, 0.29914808273299998, 0.29844117164599998, 0.29853296279899999, 0.29871106147799997, 0.29861211776699997, 0.29853105545000003, 0.29837203025800002, 0.29879808425900001, 0.29828119277999998, 0.29866003990200002, 0.29857087135299998, 0.29899382591200002]}], ["spectral-norm", "RawResult", {"changed_times": [0.17074894905099999, 0.14435315132099999, 0.137468099594, 0.13734507560699999, 0.14394092559800001, 0.13739204406700001, 0.13723492622399999, 0.14594006538400001, 0.13758420944200001, 0.13743281364400001, 0.143836975098, 0.13292717933699999, 0.137432098389, 0.146055936813, 0.13755106925999999, 0.13504004478500001, 0.146221160889, 0.137521028519, 0.13721990585300001, 0.14478611946100001, 0.136807203293, 0.13522505760199999, 0.14593815803499999, 0.13738512992900001, 0.13760805130000001, 0.14592599868799999, 0.13768291473399999, 0.134978055954, 0.14392304420499999, 0.13763713836700001, 0.13788318633999999, 0.14610290527299999, 0.137641906738, 0.137439966202, 0.145768880844, 0.13380408287000001, 0.136070013046, 0.14630413055399999, 0.13757896423300001, 0.137543916702, 0.145984888077, 0.13747000694299999, 0.13757491111799999, 0.145990848541, 0.13560295105, 0.13743591308600001, 0.14435505867000001, 0.137537956238, 0.137521028519, 0.14630484581], "base_times": [0.49145984649699997, 0.49088001251199997, 0.49142503738400001, 0.49289107322699999, 0.49142193794299999, 0.49155712127700002, 0.49165201187099999, 0.489893913269, 0.49077796936000001, 0.49144983291599997, 0.49010205268899998, 0.49105095863300002, 0.48979687690700002, 0.49000597000099999, 0.49139714240999999, 0.49049901962300002, 0.49198389053300001, 0.49077105522199999, 0.49398016929600003, 0.49002695083600001, 0.49011707305899999, 0.489161968231, 0.49115204811099999, 0.48943805694600001, 0.48959302902200003, 0.48984408378599997, 0.494458913803, 0.48965787887599999, 0.49176192283600001, 0.49097299575800002, 0.49315404891999998, 0.48959803581200001, 0.49069213867200001, 0.49154210090599998, 0.49045801162699998, 0.490023136139, 0.48951697349500001, 0.48991894722000001, 0.49031996726999999, 0.49174714088400001, 0.48993802070600001, 0.49052405357399997, 0.49143004417399999, 0.48952603340099998, 0.48930788040200002, 0.48916006088300001, 0.49066591262800002, 0.48851394653300001, 0.49009609222400002, 0.488929033279]}], ["telco", "RawResult", {"changed_times": [0.73999999999999999, 0.71999999999999997, 0.71999999999999997, 0.72999999999999998, 0.72999999999999998, 0.71999999999999997, 0.72999999999999998, 0.72999999999999998, 0.72999999999999998, 0.71999999999999997, 0.71999999999999997, 0.72999999999999998, 0.75, 0.75, 0.71999999999999997, 0.73999999999999999, 0.72999999999999998, 0.73999999999999999, 0.72999999999999998, 0.71999999999999997, 0.72999999999999998, 0.71999999999999997, 0.71999999999999997, 0.72999999999999998, 0.72999999999999998, 0.71999999999999997, 0.72999999999999998, 0.71999999999999997, 0.72999999999999998, 0.72999999999999998, 0.73999999999999999, 0.72999999999999998, 0.73999999999999999, 0.73999999999999999, 0.71999999999999997, 0.71999999999999997, 0.72999999999999998, 0.72999999999999998, 0.72999999999999998, 0.72999999999999998, 0.72999999999999998, 0.72999999999999998, 0.72999999999999998, 0.72999999999999998, 0.73999999999999999, 0.71999999999999997, 0.72999999999999998, 0.75, 0.73999999999999999, 0.72999999999999998], "base_times": [1.2, 1.22, 1.2, 1.2, 1.2, 1.2, 1.21, 1.2, 1.2, 1.2, 1.2, 1.2, 1.2, 1.21, 1.21, 1.21, 1.22, 1.2, 1.2, 1.21, 1.2, 1.2, 1.2, 1.21, 1.2, 1.2, 1.2, 1.21, 1.2, 1.2, 1.2, 1.21, 1.2, 1.21, 1.1899999999999999, 1.21, 1.2, 1.2, 1.2, 1.2, 1.2, 1.21, 1.1899999999999999, 1.2, 1.2, 1.21, 1.2, 1.2, 1.2, 1.21]}], ["twisted_names", "RawResult", {"changed_times": [0.010416666666666666, 0.010395010395010396, 0.010427528675703858, 0.010384215991692628, 0.010460251046025104, 0.01040582726326743, 0.010438413361169102, 0.010438413361169102, 0.010384215991692628, 0.010438413361169102, 0.010449320794148381, 0.010416666666666666, 0.01040582726326743, 0.010416666666666666, 0.010384215991692628, 0.010416666666666666, 0.010416666666666666, 0.010395010395010396, 0.010427528675703858, 0.010384215991692628, 0.010449320794148381, 0.010438413361169102, 0.010416666666666666, 0.010449320794148381, 0.010460251046025104, 0.01040582726326743, 0.010460251046025104, 0.010438413361169102, 0.010395010395010396, 0.01040582726326743, 0.010427528675703858, 0.01040582726326743, 0.010416666666666666, 0.010416666666666666, 0.010438413361169102, 0.010449320794148381, 0.01040582726326743, 0.010449320794148381, 0.01040582726326743, 0.010427528675703858, 0.010416666666666666, 0.01040582726326743, 0.01040582726326743, 0.010427528675703858, 0.010460251046025104, 0.010438413361169102, 0.010449320794148381, 0.010416666666666666, 0.010384215991692628, 0.010438413361169102], "base_times": [0.0095693779904306216, 0.0095785440613026813, 0.0095969289827255271, 0.0095969289827255271, 0.0095969289827255271, 0.0095785440613026813, 0.0095419847328244278, 0.0095602294455066923, 0.0095602294455066923, 0.0095693779904306216, 0.0096153846153846159, 0.0096061479346781949, 0.0095785440613026813, 0.0095969289827255271, 0.0095602294455066923, 0.0096061479346781949, 0.0095877277085330784, 0.0095693779904306216, 0.0095877277085330784, 0.0095510983763132766, 0.0095877277085330784, 0.0095969289827255271, 0.0095693779904306216, 0.0095602294455066923, 0.0095785440613026813, 0.0095877277085330784, 0.0095969289827255271, 0.0095602294455066923, 0.0096061479346781949, 0.0095602294455066923, 0.0095877277085330784, 0.0096246390760346481, 0.0095785440613026813, 0.0095877277085330784, 0.0096153846153846159, 0.0095785440613026813, 0.0095510983763132766, 0.0095785440613026813, 0.0095969289827255271, 0.0095969289827255271, 0.0096246390760346481, 0.0095877277085330784, 0.0096153846153846159, 0.0096153846153846159, 0.0095785440613026813, 0.0095785440613026813, 0.0096061479346781949, 0.0095693779904306216, 0.0096153846153846159, 0.0096246390760346481]}]], "branch": "trunk", "revision": 0} \ No newline at end of file From antocuni at codespeak.net Wed Oct 13 13:45:16 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 13 Oct 2010 13:45:16 +0200 (CEST) Subject: [pypy-svn] r77860 - in pypy/extradoc/talk/pepm2011: . figures Message-ID: <20101013114516.BEEEB282BD6@codespeak.net> Author: antocuni Date: Wed Oct 13 13:45:15 2010 New Revision: 77860 Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf pypy/extradoc/talk/pepm2011/figures/step1.pdf pypy/extradoc/talk/pepm2011/figures/step2.pdf pypy/extradoc/talk/pepm2011/paper.bib pypy/extradoc/talk/pepm2011/paper.tex Log: fix some typos, add few XXXs, improve sentences here and there Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf ============================================================================== Binary files. No diff available. Modified: pypy/extradoc/talk/pepm2011/figures/step1.pdf ============================================================================== Binary files. No diff available. Modified: pypy/extradoc/talk/pepm2011/figures/step2.pdf ============================================================================== Binary files. No diff available. Modified: pypy/extradoc/talk/pepm2011/paper.bib ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.bib (original) +++ pypy/extradoc/talk/pepm2011/paper.bib Wed Oct 13 13:45:15 2010 @@ -240,6 +240,7 @@ pages = {18--25} }, + @techreport{mason_chang_efficient_2007, title = {Efficient {Just-In-Time} Execution of Dynamically Typed Languages Via Code Specialization Using Precise Runtime Type Inference}, @@ -316,3 +317,13 @@ year = {2008}, pages = {123--139} } + + + at phdthesis{cuni_python_cli_2010, + author = {Antonio Cuni}, + title = {High performance implementation of {Python} for {CLI/.NET} with + {JIT} compiler generation for dynamic languages.}, + school = {Dipartimento di Informatica e Scienze dell'Informazione, University of Genova}, + note = {Technical Report {DISI-TH-2010-05}}, + year = {2010}, +}, Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 13 13:45:15 2010 @@ -28,6 +28,7 @@ \newcommand\arigo[1]{\nb{AR}{#1}} \newcommand\fijal[1]{\nb{FIJAL}{#1}} \newcommand\david[1]{\nb{DAVID}{#1}} +\newcommand\anto[1]{\nb{ANTO}{#1}} \newcommand\reva[1]{\nb{Reviewer 1}{#1}} \newcommand\revb[1]{\nb{Reviewer 2}{#1}} \newcommand\revc[1]{\nb{Reviewer 3}{#1}} @@ -130,13 +131,13 @@ A recently popular approach to implementing just-in-time compilers for dynamic languages is that of a tracing JIT. A tracing JIT works by observing the running -program and recording linear execution traces, which are then turned into +program and recording its hot parts into linear execution traces, which are then turned into machine code. One reason for the popularity of tracing JITs is their relative simplicity. They can often be added to an interpreter and a lot of the infrastructure of an interpreter can be reused. They give some important optimizations like inlining and constant-folding for free. A tracing JIT always -produces linear pieces of code, which makes many optimizations that are usually -hard in a compiler simpler, such as register allocation. +produces linear pieces of code, which simplifies many optimizations that are usually +hard in a compiler, such as register allocation. The usage of a tracing JIT can remove the overhead of bytecode dispatch and that of the interpreter data structures. In this paper we want to present an approach @@ -198,7 +199,7 @@ \cite{carl_friedrich_bolz_back_2008} and a GameBoy emulator \cite{bruni_pygirl:_2009}. -The feature that makes PyPy more than a compiler with a runtime system is it's +The feature that makes PyPy more than a compiler with a runtime system is its support for automated JIT compiler generation \cite{bolz_tracing_2009}. During the translation to C, PyPy's tools can generate a just-in-time compiler for the language that the interpreter is implementing. This process is mostly @@ -206,7 +207,13 @@ source-code hints. Mostly-automatically generating a JIT compiler has many advantages over writing one manually, which is an error-prone and tedious process. By construction, the generated JIT has the same semantics as the interpreter. -Many optimizations can benefit all languages implemented as an interpreter in RPython. +Many optimizations can benefit all languages implemented as an interpreter in RPython. + +Moreover, thanks to the internal design of the JIT generator, it is very easy +to add new \emph{backends} for producing the actual machine code, in addition +to the original backend for the Intel \emph{x86} architecture. Examples of +additional JIT backends are the one for Intel \emph{x86-64} and an +experimental one for the CLI .NET Virtual Machine \cite{cuni_python_cli_2010}. The JIT that is produced by PyPy's JIT generator is a \emph{tracing JIT compiler}, a concept which we now explain in more details. @@ -233,7 +240,7 @@ interpreter records all operations that it is executing while running one iteration of the hot loop. This history of executed operations of one loop is called a \emph{trace}. Because the trace corresponds to one iteration of a loop, -it always ends with a jump to its own beginning. The trace also contains all +it always ends with a jump to its own beginning \anto{this is not true: what if we trace the last iteration?}. The trace also contains all operations that are performed in functions that were called in the loop, thus a tracing JIT automatically performs inlining. @@ -264,7 +271,9 @@ If one specific guard fails often enough, the tracing JIT will generate a new trace that starts exactly at the position of the failing guard. The existing assembler is patched to jump to the new trace when the guard fails -\cite{andreas_gal_incremental_2006}. +\cite{andreas_gal_incremental_2006}. This approach guarantees that all the +hot paths in the program will eventually be traced and compiled into efficient +code. \subsection{Running Example} @@ -356,13 +365,14 @@ call to \texttt{add}. These method calls need to check the type of the involved objects repeatedly and redundantly. In addition, a lot of objects are created when executing that loop, many of these objects do not survive for very long. -The actual computation that is performed by \texttt{f} is simply a number of +The actual computation that is performed by \texttt{f} is simply a sequence of float or integer additions. \begin{figure} \texttt{ \begin{tabular}{l} +\# XXX: maybe we should specify that $p_{0}$, $p_{1}$ corresponds to y and res \# arguments to the trace: $p_{0}$, $p_{1}$ \\ \# inside f: res.add(y) \\ guard\_class($p_{1}$, BoxedInteger) \\ @@ -533,7 +543,8 @@ \texttt{get} from such an object, the result is read from the shape description, and the operation is also removed. Equivalently, a \texttt{guard\_class} on a variable that has a shape description can be removed -as well, because the shape description stores the type. +as well, because the shape description stores the type and thus the result of +the guard is statically known. In the example from last section, the following operations would produce two static objects, and be completely removed from the optimized trace: @@ -551,6 +562,8 @@ \texttt{BoxedInteger} whose \texttt{intval} field contains $i_{4}$; the one associated with $p_{6}$ would know that it is a \texttt{BoxedInteger} whose \texttt{intval} field contains the constant -100. +\anto{this works only because we use SSI and thus the value of $i_{4}$ never +changes. However, SSI is not explained anywhere in the paper} The following operations on $p_{5}$ and $p_{6}$ could then be optimized using that knowledge: @@ -595,7 +608,7 @@ jump is emitted, they are \emph{lifted}. This means that the optimizer produces code that allocates a new object of the right type and sets its fields to the field values that the static object has (if the static object points to other static -objects, those need to be lifted as well) This means that instead of the jump, +objects, those need to be lifted as well, recursively) This means that instead of the jump, the following operations are emitted: \texttt{ @@ -613,7 +626,9 @@ the objects are still allocated at the end. However, the optimization was still worthwhile even in this case, because some operations that have been performed on the lifted static objects have been removed (some \texttt{get} operations -and \texttt{guard\_class} operations). +and \texttt{guard\_class} operations). Moreover, in real life example usually +the loops are more complex and contain more objects of type 1, thus this +technique is more effective. \begin{figure} \includegraphics{figures/step1.pdf} @@ -797,7 +812,7 @@ the variables, and the operation has to be residualized. If the argument $v$ of a \texttt{get} operation is mapped to something in the static -heap, the get can be performed at optimization time. Otherwise, the \texttt{get} +heap, the \texttt{get} can be performed at optimization time. Otherwise, the \texttt{get} operation needs to be residualized. If the first argument $v$ to a \texttt{set} operation is mapped to something in the @@ -919,14 +934,14 @@ \item \textbf{raytrace-simple}: A ray tracer. \item \textbf{richards}: The Richards benchmark. \item \textbf{spambayes}: A Bayesian spam filter\footnote{\texttt{http://spambayes.sourceforge.net/}}. - \item \textbf{telco}: A Python version of the Telco decimal. + \item \textbf{telco}: A Python version of the Telco decimal benchmark\footnote{\texttt{http://speleotrove.com/decimal/telco.html}}, using a pure Python decimal floating point implementation. \item \textbf{twisted\_names}: A DNS server benchmark using the Twisted networking framework\footnote{\texttt{http://twistedmatrix.com/}}. \end{itemize} -We evaluate the allocation removal algorithm along two lines: First we want to +We evaluate the allocation removal algorithm along two lines: first we want to know how many allocations could be optimized away. On the other hand, we want to know how much the run times of the benchmarks is improved. @@ -935,6 +950,7 @@ seen in Figure~\ref{fig:numops}. The optimization removes as many as XXX and as little as XXX percent of allocation operations in the benchmarks. All benchmarks taken together, the optimization removes XXX percent of allocation operations. +\anto{Actually, we can only know how many operations we removed from the traces, not from the actual execution} \begin{figure*} \begin{tabular}{lrrrrrrrrrrrrrrrrrrrrrrr} @@ -966,7 +982,7 @@ CPython\footnote{\texttt{http://python.org}}, which uses a bytecode-based interpreter. Furthermore we compared against Psyco \cite{rigo_representation-based_2004}, an extension to CPython which is a just-in-time compiler that produces machine code at run-time. -It is not based on traces. Of PyPy's Python interpreter we used three versions, +It is not based on traces. Finally, we used three versions of PyPy's Python interpreter: one without a JIT, one including the JIT but not using the allocation removal optimization, and one using the allocation removal optimizations. From afa at codespeak.net Wed Oct 13 14:03:21 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Wed, 13 Oct 2010 14:03:21 +0200 (CEST) Subject: [pypy-svn] r77861 - in pypy/branch/fast-forward/pypy/module/_hashlib: . test Message-ID: <20101013120321.2343C282BDB@codespeak.net> Author: afa Date: Wed Oct 13 14:03:19 2010 New Revision: 77861 Added: pypy/branch/fast-forward/pypy/module/_hashlib/ (props changed) pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py (contents, props changed) pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py (contents, props changed) pypy/branch/fast-forward/pypy/module/_hashlib/test/ (props changed) pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py (contents, props changed) Log: Start a _hashlib module. Added: pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py Wed Oct 13 14:03:19 2010 @@ -0,0 +1,15 @@ +from pypy.interpreter.mixedmodule import MixedModule +from pypy.module._hashlib import interp_hashlib + + +class Module(MixedModule): + interpleveldefs = { + 'new' : 'interp_hashlib.new', + 'HASH': 'interp_hashlib.W_Hash', + } + + appleveldefs = { + } + + for name in interp_hashlib.algorithms: + interpleveldefs[name] = getattr(interp_hashlib, 'new_' + name) Added: pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py Wed Oct 13 14:03:19 2010 @@ -0,0 +1,60 @@ +from pypy.interpreter.gateway import unwrap_spec, interp2app +from pypy.interpreter.typedef import TypeDef +from pypy.tool.sourcetools import func_renamer +from pypy.interpreter.baseobjspace import Wrappable, W_Root, ObjSpace +from pypy.rpython.lltypesystem import lltype, rffi +from pypy.rlib import ropenssl + +algorithms = ('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512') + +class W_Hash(Wrappable): + def __init__(self, name): + self.name = name + self.ctx = lltype.malloc(ropenssl.EVP_MD_CTX.TO, flavor='raw') + + @unwrap_spec('self', ObjSpace, str, str) + def descr_init(self, space, name, buffer): + digest = ropenssl.EVT_get_digestbyname(name) + if not digest: + raise OperationError(space.w_ValueError, + space.wrap("unknown hash function")) + ropenssl.EVP_DigestInit(self.ctx, digest) + + if buffer: + self._hash(buffer) + + @unwrap_spec('self', ObjSpace) + def descr_repr(self, space): + return space.wrap("<%s HASH object @ 0x%x>" % (self.name, id(self))) + + @unwrap_spec('self', ObjSpace, str) + def update(self, space, buffer): + self._hash(buffer) + + def _hash(self, buffer): + buf = rffi.str2charp(buffer) + try: + ropenssl.EVP_DigestUpdate(self.ctx, buf, len(buffer)) + finally: + rffi.free_charp(buf) + +W_Hash.typedef = TypeDef( + 'HASH', + __init__=interp2app(W_Hash.descr_init), + __repr__=interp2app(W_Hash.descr_repr), + update=interp2app(W_Hash.update), + ) + + at unwrap_spec(ObjSpace, str, str) +def new(space, method, string=''): + w_hash = W_Hash(method) + w_hash.update(space, string) + return space.wrap(w_hash) + +# shortcut functions +for name in algorithms: + newname = 'new_%s' % (name,) + @func_renamer(newname) + def new_hash(w_string=''): + return _new(name, w_string) + globals()[newname] = new_hash Added: pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py Wed Oct 13 14:03:19 2010 @@ -0,0 +1,53 @@ +import py +from pypy.conftest import gettestobjspace + +class AppTestHashlib: + def setup_class(cls): + cls.space = gettestobjspace(usemodules=['_hashlib']) + + def test_simple(self): + import _hashlib + assert isinstance(_hashlib.new('md5'), _hashlib.HASH) + + def test_attributes(self): + import _hashlib + for name, expected_size in {'md5': 16, + 'sha1': 20, + 'sha224': 28, + 'sha256': 32, + 'sha384': 48, + 'sha512': 64, + }.items(): + h = hashlib.new(name) + assert h.digest_size == expected_size + assert h.digestsize == expected_size + # + h.update('abc') + h2 = h.copy() + h.update('def') + digest = h.digest() + hexdigest = h.hexdigest() + h2.update('d') + h2.update('ef') + assert digest == h.digest() + assert hexdigest == h.hexdigest() + + # also test the pure Python implementation + h = hashlib.__get_builtin_constructor(name)('') + assert h.digest_size == expected_size + assert h.digestsize == expected_size + # + h.update('abc') + h2 = h.copy() + h.update('def') + digest = h.digest() + hexdigest = h.hexdigest() + h2.update('d') + h2.update('ef') + assert digest == h.digest() + assert hexdigest == h.hexdigest() + + def test_unicode(self): + import _hashlib + assert isinstance(hashlib.new('sha1', u'xxx'), _hashlib.HASH) + From afa at codespeak.net Wed Oct 13 14:04:56 2010 From: afa at codespeak.net (afa at codespeak.net) Date: Wed, 13 Oct 2010 14:04:56 +0200 (CEST) Subject: [pypy-svn] r77862 - pypy/branch/fast-forward/pypy/rlib Message-ID: <20101013120456.0C6D1282BDC@codespeak.net> Author: afa Date: Wed Oct 13 14:04:55 2010 New Revision: 77862 Added: pypy/branch/fast-forward/pypy/rlib/ropenssl.py (contents, props changed) Log: Add missing file Added: pypy/branch/fast-forward/pypy/rlib/ropenssl.py ============================================================================== --- (empty file) +++ pypy/branch/fast-forward/pypy/rlib/ropenssl.py Wed Oct 13 14:04:55 2010 @@ -0,0 +1,195 @@ +from pypy.rpython.lltypesystem import rffi, lltype +from pypy.rpython.tool import rffi_platform +from pypy.translator.platform import platform +from pypy.translator.tool.cbuild import ExternalCompilationInfo + +import sys + +if sys.platform == 'win32' and platform.name != 'mingw32': + libraries = ['libeay32', 'ssleay32', 'user32', 'advapi32', 'gdi32'] + includes = [ + # ssl.h includes winsock.h, which will conflict with our own + # need of winsock2. Remove this when separate compilation is + # available... + 'winsock2.h', + # wincrypt.h defines X509_NAME, include it here + # so that openssl/ssl.h can repair this nonsense. + 'wincrypt.h', + 'openssl/ssl.h', + 'openssl/err.h'] +else: + libraries = ['ssl', 'crypto'] + includes = ['openssl/ssl.h', 'openssl/err.h'] + +eci = ExternalCompilationInfo( + libraries = libraries, + includes = includes, + export_symbols = ['SSL_load_error_strings'], + ) + +eci = rffi_platform.configure_external_library( + 'openssl', eci, + [dict(prefix='openssl-', + include_dir='inc32', library_dir='out32'), + ]) + +## user defined constants +X509_NAME_MAXLEN = 256 +## # these mirror ssl.h +PY_SSL_ERROR_NONE, PY_SSL_ERROR_SSL = 0, 1 +PY_SSL_ERROR_WANT_READ, PY_SSL_ERROR_WANT_WRITE = 2, 3 +PY_SSL_ERROR_WANT_X509_LOOKUP = 4 +PY_SSL_ERROR_SYSCALL = 5 # look at error stack/return value/errno +PY_SSL_ERROR_ZERO_RETURN, PY_SSL_ERROR_WANT_CONNECT = 6, 7 +# start of non ssl.h errorcodes +PY_SSL_ERROR_EOF = 8 # special case of SSL_ERROR_SYSCALL +PY_SSL_ERROR_INVALID_ERROR_CODE = 9 + +PY_SSL_CERT_NONE, PY_SSL_CERT_OPTIONAL, PY_SSL_CERT_REQUIRED = 0, 1, 2 + +(PY_SSL_VERSION_SSL2, PY_SSL_VERSION_SSL3, + PY_SSL_VERSION_SSL23, PY_SSL_VERSION_TLS1) = range(4) + +SOCKET_IS_NONBLOCKING, SOCKET_IS_BLOCKING = 0, 1 +SOCKET_HAS_TIMED_OUT, SOCKET_HAS_BEEN_CLOSED = 2, 3 +SOCKET_TOO_LARGE_FOR_SELECT, SOCKET_OPERATION_OK = 4, 5 + +# WinSock does not use a bitmask in select, and uses +# socket handles greater than FD_SETSIZE +if sys.platform == 'win32': + MAX_FD_SIZE = None +else: + from pypy.rlib._rsocket_rffi import FD_SETSIZE as MAX_FD_SIZE + +HAVE_RPOLL = True # Even win32 has rpoll.poll + +class CConfig: + _compilation_info_ = eci + + OPENSSL_VERSION_NUMBER = rffi_platform.ConstantInteger( + "OPENSSL_VERSION_NUMBER") + SSLEAY_VERSION = rffi_platform.DefinedConstantString( + "SSLEAY_VERSION", "SSLeay_version(SSLEAY_VERSION)") + SSL_FILETYPE_PEM = rffi_platform.ConstantInteger("SSL_FILETYPE_PEM") + SSL_OP_ALL = rffi_platform.ConstantInteger("SSL_OP_ALL") + SSL_VERIFY_NONE = rffi_platform.ConstantInteger("SSL_VERIFY_NONE") + SSL_ERROR_WANT_READ = rffi_platform.ConstantInteger( + "SSL_ERROR_WANT_READ") + SSL_ERROR_WANT_WRITE = rffi_platform.ConstantInteger( + "SSL_ERROR_WANT_WRITE") + SSL_ERROR_ZERO_RETURN = rffi_platform.ConstantInteger( + "SSL_ERROR_ZERO_RETURN") + SSL_ERROR_WANT_X509_LOOKUP = rffi_platform.ConstantInteger( + "SSL_ERROR_WANT_X509_LOOKUP") + SSL_ERROR_WANT_CONNECT = rffi_platform.ConstantInteger( + "SSL_ERROR_WANT_CONNECT") + SSL_ERROR_SYSCALL = rffi_platform.ConstantInteger("SSL_ERROR_SYSCALL") + SSL_ERROR_SSL = rffi_platform.ConstantInteger("SSL_ERROR_SSL") + SSL_CTRL_OPTIONS = rffi_platform.ConstantInteger("SSL_CTRL_OPTIONS") + BIO_C_SET_NBIO = rffi_platform.ConstantInteger("BIO_C_SET_NBIO") + +for k, v in rffi_platform.configure(CConfig).items(): + globals()[k] = v + +# opaque structures +SSL_METHOD = rffi.COpaquePtr('SSL_METHOD') +SSL_CTX = rffi.COpaquePtr('SSL_CTX') +SSL = rffi.COpaquePtr('SSL') +BIO = rffi.COpaquePtr('BIO') +X509 = rffi.COpaquePtr('X509') +X509_NAME = rffi.COpaquePtr('X509_NAME') + +HAVE_OPENSSL_RAND = OPENSSL_VERSION_NUMBER >= 0x0090500f + +constants = {} +constants["SSL_ERROR_ZERO_RETURN"] = PY_SSL_ERROR_ZERO_RETURN +constants["SSL_ERROR_WANT_READ"] = PY_SSL_ERROR_WANT_READ +constants["SSL_ERROR_WANT_WRITE"] = PY_SSL_ERROR_WANT_WRITE +constants["SSL_ERROR_WANT_X509_LOOKUP"] = PY_SSL_ERROR_WANT_X509_LOOKUP +constants["SSL_ERROR_SYSCALL"] = PY_SSL_ERROR_SYSCALL +constants["SSL_ERROR_SSL"] = PY_SSL_ERROR_SSL +constants["SSL_ERROR_WANT_CONNECT"] = PY_SSL_ERROR_WANT_CONNECT +constants["SSL_ERROR_EOF"] = PY_SSL_ERROR_EOF +constants["SSL_ERROR_INVALID_ERROR_CODE"] = PY_SSL_ERROR_INVALID_ERROR_CODE + +constants["CERT_NONE"] = PY_SSL_CERT_NONE +constants["CERT_OPTIONAL"] = PY_SSL_CERT_OPTIONAL +constants["CERT_REQUIRED"] = PY_SSL_CERT_REQUIRED + +constants["PROTOCOL_SSLv2"] = PY_SSL_VERSION_SSL2 +constants["PROTOCOL_SSLv3"] = PY_SSL_VERSION_SSL3 +constants["PROTOCOL_SSLv23"] = PY_SSL_VERSION_SSL23 +constants["PROTOCOL_TLSv1"] = PY_SSL_VERSION_TLS1 + +constants["OPENSSL_VERSION_NUMBER"] = OPENSSL_VERSION_NUMBER +ver = OPENSSL_VERSION_NUMBER +ver, status = divmod(ver, 16) +ver, patch = divmod(ver, 256) +ver, fix = divmod(ver, 256) +ver, minor = divmod(ver, 256) +ver, major = divmod(ver, 256) +constants["OPENSSL_VERSION_INFO"] = (major, minor, fix, patch, status) +constants["OPENSSL_VERSION"] = SSLEAY_VERSION + +def external(name, argtypes, restype, **kw): + kw['compilation_info'] = eci + return rffi.llexternal( + name, argtypes, restype, **kw) + +def ssl_external(name, argtypes, restype, **kw): + globals()['libssl_' + name] = external( + name, argtypes, restype, **kw) + +ssl_external('SSL_load_error_strings', [], lltype.Void) +ssl_external('SSL_library_init', [], rffi.INT) +if HAVE_OPENSSL_RAND: + ssl_external('RAND_add', [rffi.CCHARP, rffi.INT, rffi.DOUBLE], lltype.Void) + ssl_external('RAND_status', [], rffi.INT) + ssl_external('RAND_egd', [rffi.CCHARP], rffi.INT) +ssl_external('SSL_CTX_new', [SSL_METHOD], SSL_CTX) +ssl_external('SSLv23_method', [], SSL_METHOD) +ssl_external('SSL_CTX_use_PrivateKey_file', [SSL_CTX, rffi.CCHARP, rffi.INT], rffi.INT) +ssl_external('SSL_CTX_use_certificate_chain_file', [SSL_CTX, rffi.CCHARP], rffi.INT) +ssl_external('SSL_CTX_ctrl', [SSL_CTX, rffi.INT, rffi.INT, rffi.VOIDP], rffi.INT) +ssl_external('SSL_CTX_set_verify', [SSL_CTX, rffi.INT, rffi.VOIDP], lltype.Void) +ssl_external('SSL_new', [SSL_CTX], SSL) +ssl_external('SSL_set_fd', [SSL, rffi.INT], rffi.INT) +ssl_external('BIO_ctrl', [BIO, rffi.INT, rffi.INT, rffi.VOIDP], rffi.INT) +ssl_external('SSL_get_rbio', [SSL], BIO) +ssl_external('SSL_get_wbio', [SSL], BIO) +ssl_external('SSL_set_connect_state', [SSL], lltype.Void) +ssl_external('SSL_connect', [SSL], rffi.INT) +ssl_external('SSL_get_error', [SSL, rffi.INT], rffi.INT) + +ssl_external('ERR_get_error', [], rffi.INT) +ssl_external('ERR_error_string', [rffi.ULONG, rffi.CCHARP], rffi.CCHARP) +ssl_external('SSL_get_peer_certificate', [SSL], X509) +ssl_external('X509_get_subject_name', [X509], X509_NAME) +ssl_external('X509_get_issuer_name', [X509], X509_NAME) +ssl_external('X509_NAME_oneline', [X509_NAME, rffi.CCHARP, rffi.INT], rffi.CCHARP) +ssl_external('X509_free', [X509], lltype.Void) +ssl_external('SSL_free', [SSL], lltype.Void) +ssl_external('SSL_CTX_free', [SSL_CTX], lltype.Void) +ssl_external('SSL_write', [SSL, rffi.CCHARP, rffi.INT], rffi.INT) +ssl_external('SSL_pending', [SSL], rffi.INT) +ssl_external('SSL_read', [SSL, rffi.CCHARP, rffi.INT], rffi.INT) + +ssl_external('SSL_read', [SSL, rffi.CCHARP, rffi.INT], rffi.INT) + +EVP_MD_CTX = rffi.COpaquePtr('EVP_MD_CTX') +EVP_MD = rffi.COpaquePtr('EVP_MD') + +EVP_get_digestbyname = external( + 'EVP_get_digestbyname', + [rffi.CCHARP], EVP_MD) +EVP_DigestInit = external( + 'EVP_DigestInit', + [EVP_MD_CTX, EVP_MD], rffi.INT) +EVP_DigestUpdate = external( + 'EVP_DigestUpdate', + [EVP_MD_CTX, rffi.CCHARP, rffi.SIZE_T], rffi.INT) + +def _init_ssl(): + libssl_SSL_load_error_strings() + libssl_SSL_library_init() + From arigo at codespeak.net Wed Oct 13 14:10:59 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 13 Oct 2010 14:10:59 +0200 (CEST) Subject: [pypy-svn] r77863 - pypy/trunk/pypy/doc/config Message-ID: <20101013121059.E8BDE282BE0@codespeak.net> Author: arigo Date: Wed Oct 13 14:10:58 2010 New Revision: 77863 Added: pypy/trunk/pypy/doc/config/objspace.std.withstrbuf.txt Log: Add documentation. Added: pypy/trunk/pypy/doc/config/objspace.std.withstrbuf.txt ============================================================================== --- (empty file) +++ pypy/trunk/pypy/doc/config/objspace.std.withstrbuf.txt Wed Oct 13 14:10:58 2010 @@ -0,0 +1,4 @@ +Enable "string buffer" objects. + +Similar to "string join" objects, but using a StringBuilder to represent +a string built by repeated application of ``+=``. From arigo at codespeak.net Wed Oct 13 14:14:53 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 13 Oct 2010 14:14:53 +0200 (CEST) Subject: [pypy-svn] r77864 - pypy/trunk/pypy/rlib/test Message-ID: <20101013121453.B2FC4282BE7@codespeak.net> Author: arigo Date: Wed Oct 13 14:14:52 2010 New Revision: 77864 Modified: pypy/trunk/pypy/rlib/test/test_runicode.py Log: Fix. Modified: pypy/trunk/pypy/rlib/test/test_runicode.py ============================================================================== --- pypy/trunk/pypy/rlib/test/test_runicode.py (original) +++ pypy/trunk/pypy/rlib/test/test_runicode.py Wed Oct 13 14:14:52 2010 @@ -76,7 +76,7 @@ assert start == startingpos assert stop == endingpos return u"42424242", stop - return "", endingpos + return u"", endingpos decoder = self.getdecoder(encoding) if addstuff: s += "some rest in ascii" From cfbolz at codespeak.net Wed Oct 13 14:20:05 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 13 Oct 2010 14:20:05 +0200 (CEST) Subject: [pypy-svn] r77865 - pypy/extradoc/talk/pepm2011 Message-ID: <20101013122005.187A6282BE7@codespeak.net> Author: cfbolz Date: Wed Oct 13 14:20:03 2010 New Revision: 77865 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: fix a few of the fixes/remarks Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 13 14:20:03 2010 @@ -207,7 +207,7 @@ source-code hints. Mostly-automatically generating a JIT compiler has many advantages over writing one manually, which is an error-prone and tedious process. By construction, the generated JIT has the same semantics as the interpreter. -Many optimizations can benefit all languages implemented as an interpreter in RPython. +Many optimizations can benefit all languages implemented as an interpreter in RPython. Moreover, thanks to the internal design of the JIT generator, it is very easy to add new \emph{backends} for producing the actual machine code, in addition @@ -240,7 +240,7 @@ interpreter records all operations that it is executing while running one iteration of the hot loop. This history of executed operations of one loop is called a \emph{trace}. Because the trace corresponds to one iteration of a loop, -it always ends with a jump to its own beginning \anto{this is not true: what if we trace the last iteration?}. The trace also contains all +it always ends with a jump to its own beginning. The trace also contains all operations that are performed in functions that were called in the loop, thus a tracing JIT automatically performs inlining. @@ -371,8 +371,8 @@ \begin{figure} \texttt{ -\begin{tabular}{l} -\# XXX: maybe we should specify that $p_{0}$, $p_{1}$ corresponds to y and res +\begin{tabular}{l} +\# XXX: maybe we should specify that $p_{0}$, $p_{1}$ corresponds to y and res \\ \# arguments to the trace: $p_{0}$, $p_{1}$ \\ \# inside f: res.add(y) \\ guard\_class($p_{1}$, BoxedInteger) \\ @@ -437,7 +437,8 @@ \texttt{BoxedInteger}, the produced trace looks like Figure~\ref{fig:unopt-trace}. The operations in the trace are shown indented to correspond to the stack level of the function that contains the traced -operation. The trace also shows the inefficiencies of \texttt{f} clearly, if one +operation. The trace is in single-assignment form, meaning that each variable is +assigned to exactly once. The trace also shows the inefficiencies of \texttt{f} clearly, if one looks at the number of \texttt{new} (corresponding to object creation), \texttt{set/get} (corresponding to attribute reads/writes) and \texttt{guard\_class} operations (corresponding to method calls). @@ -448,7 +449,7 @@ into the trace. The method calls are always preceded by a \texttt{guard\_class} operation, to check that the class of the receiver is the same as the one that was observed during tracing.\footnote{\texttt{guard\_class} performs a precise -class check, not checking for subclasses} These guards make the trace specific +class check, not checking for subclasses.} These guards make the trace specific to the situation where \texttt{y} is really a \texttt{BoxedInteger}, it can already be said to be specialized for \texttt{BoxedIntegers}. When the trace is turned into machine code and then executed with \texttt{BoxedFloats}, the @@ -543,14 +544,14 @@ \texttt{get} from such an object, the result is read from the shape description, and the operation is also removed. Equivalently, a \texttt{guard\_class} on a variable that has a shape description can be removed -as well, because the shape description stores the type and thus the result of -the guard is statically known. +as well, because the shape description stores the type and thus the outcome of +the type check che guard does is statically known. In the example from last section, the following operations would produce two static objects, and be completely removed from the optimized trace: \texttt{ -\begin{tabular}{l} +\begin{tabular}{l} $p_{5}$ = new(BoxedInteger) \\ set($p_{5}$, intval, $i_{4}$) \\ $p_{6}$ = new(BoxedInteger) \\ @@ -562,14 +563,12 @@ \texttt{BoxedInteger} whose \texttt{intval} field contains $i_{4}$; the one associated with $p_{6}$ would know that it is a \texttt{BoxedInteger} whose \texttt{intval} field contains the constant -100. -\anto{this works only because we use SSI and thus the value of $i_{4}$ never -changes. However, SSI is not explained anywhere in the paper} The following operations on $p_{5}$ and $p_{6}$ could then be optimized using that knowledge: \texttt{ -\begin{tabular}{l} +\begin{tabular}{l} guard\_class($p_{5}$, BoxedInteger) \\ $i_{7}$ = get($p_{5}$, intval) \\ \# inside BoxedInteger.add \\ @@ -586,11 +585,11 @@ remaining operation in the optimized trace would be: \texttt{ -\begin{tabular}{l} +\begin{tabular}{l} $i_{9}$ = int\_add($i_{4}$, -100) \\ \end{tabular} } - + The rest of the trace is optimized similarly. So far we have only described what happens when static objects are used in @@ -612,7 +611,7 @@ the following operations are emitted: \texttt{ -\begin{tabular}{l} +\begin{tabular}{l} $p_{15}$ = new(BoxedInteger) \\ set($p_{15}$, intval, $i_{14}$) \\ $p_{10}$ = new(BoxedInteger) \\ @@ -626,9 +625,7 @@ the objects are still allocated at the end. However, the optimization was still worthwhile even in this case, because some operations that have been performed on the lifted static objects have been removed (some \texttt{get} operations -and \texttt{guard\_class} operations). Moreover, in real life example usually -the loops are more complex and contain more objects of type 1, thus this -technique is more effective. +and \texttt{guard\_class} operations). \begin{figure} \includegraphics{figures/step1.pdf} @@ -948,9 +945,9 @@ For the former we counted the occurring operations in all generated traces before and after the optimization phase for all benchmarks. The results can be seen in Figure~\ref{fig:numops}. The optimization removes as many as XXX and as -little as XXX percent of allocation operations in the benchmarks. All benchmarks -taken together, the optimization removes XXX percent of allocation operations. -\anto{Actually, we can only know how many operations we removed from the traces, not from the actual execution} +little as XXX percent of allocation operations in the traces of the benchmarks. +All benchmarks taken together, the optimization removes XXX percent of +allocation operations. \begin{figure*} \begin{tabular}{lrrrrrrrrrrrrrrrrrrrrrrr} From cfbolz at codespeak.net Wed Oct 13 14:38:46 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 13 Oct 2010 14:38:46 +0200 (CEST) Subject: [pypy-svn] r77866 - pypy/extradoc/talk/pepm2011 Message-ID: <20101013123846.98380282BE9@codespeak.net> Author: cfbolz Date: Wed Oct 13 14:38:39 2010 New Revision: 77866 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: move code around, another fix Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 13 14:38:39 2010 @@ -329,8 +329,16 @@ return BoxedFloat(floatother + self.floatval) def is_positive(self): return self.floatval > 0.0 + + +def f(y): + res = BoxedInteger(0) + while y.is_positive(): + res = res.add(y).add(BoxedInteger(-100)) + y = y.add(BoxedInteger(-1)) + return res \end{verbatim} -\caption{A simple object model} +\caption{A Simple Object Model and an Example Function Using it} \label{fig:objmodel} \end{figure} @@ -342,24 +350,16 @@ implement the numeric tower needs two method calls per arithmetic operation, which is costly due to the method dispatch. -To understand the problems more directly, let us consider a simple function -that uses the object model: +To understand the problems more directly, let us consider the simple function +\texttt{f} that uses the object model (see the bottom of +Figure~\ref{fig:objmodel}). XXX this is not an RPython interpreter; put a reference to the previous paper to show how we deal with an interpreted piece of code and remove the interpretation overhead, turning it into basically something equivalent to the example here, which is the start of the present paper. -\begin{verbatim} -def f(y): - res = BoxedInteger(0) - while y.is_positive(): - res = res.add(y).add(BoxedInteger(-100)) - y = y.add(BoxedInteger(-1)) - return res -\end{verbatim} - -The loop iterates \texttt{y} times, and computes something in the process. +The loop in \texttt{f} iterates \texttt{y} times, and computes something in the process. Simply running this function is slow, because there are lots of virtual method calls inside the loop, one for each \texttt{is\_positive} and even two for each call to \texttt{add}. These method calls need to check the type of the involved @@ -372,7 +372,6 @@ \begin{figure} \texttt{ \begin{tabular}{l} -\# XXX: maybe we should specify that $p_{0}$, $p_{1}$ corresponds to y and res \\ \# arguments to the trace: $p_{0}$, $p_{1}$ \\ \# inside f: res.add(y) \\ guard\_class($p_{1}$, BoxedInteger) \\ @@ -438,7 +437,10 @@ Figure~\ref{fig:unopt-trace}. The operations in the trace are shown indented to correspond to the stack level of the function that contains the traced operation. The trace is in single-assignment form, meaning that each variable is -assigned to exactly once. The trace also shows the inefficiencies of \texttt{f} clearly, if one +assigned to exactly once. The arguments $p_0$ and $p_1$ of the loop correspond +to the live variables \texttt{y} and \texttt{res} in the original function. + +The trace shows the inefficiencies of \texttt{f} clearly, if one looks at the number of \texttt{new} (corresponding to object creation), \texttt{set/get} (corresponding to attribute reads/writes) and \texttt{guard\_class} operations (corresponding to method calls). From antocuni at codespeak.net Wed Oct 13 14:39:28 2010 From: antocuni at codespeak.net (antocuni at codespeak.net) Date: Wed, 13 Oct 2010 14:39:28 +0200 (CEST) Subject: [pypy-svn] r77867 - pypy/extradoc/talk/pepm2011 Message-ID: <20101013123928.CDEB5282BEA@codespeak.net> Author: antocuni Date: Wed Oct 13 14:39:26 2010 New Revision: 77867 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: typo Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 13 14:39:26 2010 @@ -547,7 +547,7 @@ description, and the operation is also removed. Equivalently, a \texttt{guard\_class} on a variable that has a shape description can be removed as well, because the shape description stores the type and thus the outcome of -the type check che guard does is statically known. +the type check the guard does is statically known. In the example from last section, the following operations would produce two static objects, and be completely removed from the optimized trace: From arigo at codespeak.net Wed Oct 13 14:41:01 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 13 Oct 2010 14:41:01 +0200 (CEST) Subject: [pypy-svn] r77868 - pypy/trunk/pypy/module/pypyjit/test Message-ID: <20101013124101.8BE10282BEB@codespeak.net> Author: arigo Date: Wed Oct 13 14:40:57 2010 New Revision: 77868 Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py Log: Broken during the merge of better-map-instances, by the fact that I disabled withmapdict in Ojit translations for now. Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py ============================================================================== --- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py (original) +++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py Wed Oct 13 14:40:57 2010 @@ -272,7 +272,7 @@ assert len(ops) == 2 assert not ops[0].get_opnames("call") assert not ops[0].get_opnames("new") - assert len(ops[0].get_opnames("guard")) <= 2 + assert len(ops[0].get_opnames("guard")) <= 3 # we get 2 withmapdict assert not ops[1] # second LOOKUP_METHOD folded away ops = self.get_by_bytecode("CALL_METHOD") @@ -317,8 +317,8 @@ assert len(ops) == 2 assert not ops[0].get_opnames("call") assert not ops[0].get_opnames("new") - assert len(ops[0].get_opnames("guard")) <= 2 - assert len(ops[0].get_opnames("getfield")) < 5 + assert len(ops[0].get_opnames("guard")) <= 3 # we get 2 withmapdict + assert len(ops[0].get_opnames("getfield")) <= 5 # we get <5 withmapdict assert not ops[1] # second LOOKUP_METHOD folded away def test_default_and_kw(self): From cfbolz at codespeak.net Wed Oct 13 14:49:57 2010 From: cfbolz at codespeak.net (cfbolz at codespeak.net) Date: Wed, 13 Oct 2010 14:49:57 +0200 (CEST) Subject: [pypy-svn] r77869 - pypy/extradoc/talk/pepm2011 Message-ID: <20101013124957.CDDC3282BF0@codespeak.net> Author: cfbolz Date: Wed Oct 13 14:49:55 2010 New Revision: 77869 Modified: pypy/extradoc/talk/pepm2011/paper.tex Log: (antocuni, cfbolz) a minimal fix for one of antos remarks that I just killed Modified: pypy/extradoc/talk/pepm2011/paper.tex ============================================================================== --- pypy/extradoc/talk/pepm2011/paper.tex (original) +++ pypy/extradoc/talk/pepm2011/paper.tex Wed Oct 13 14:49:55 2010 @@ -237,7 +237,7 @@ some light-weight profiling at the same time. This profiling is used to identify the hot loops of the program. If a hot loop is found in that way, the interpreter enters a special \emph{tracing mode}. In this tracing mode, the -interpreter records all operations that it is executing while running one +interpreter tries to record all operations that it is executing while running one iteration of the hot loop. This history of executed operations of one loop is called a \emph{trace}. Because the trace corresponds to one iteration of a loop, it always ends with a jump to its own beginning. The trace also contains all From arigo at codespeak.net Wed Oct 13 15:18:31 2010 From: arigo at codespeak.net (arigo at codespeak.net) Date: Wed, 13 Oct 2010 15:18:31 +0200 (CEST) Subject: [pypy-svn] r77870 - in pypy/extradoc/pypy.org: . source Message-ID: <20101013131831.B9AAA282BD4@codespeak.net> Author: arigo Date: Wed Oct 13 15:18:27 2010 New Revision: 77870 Removed: pypy/extradoc/pypy.org/source/TODO Modified: pypy/extradoc/pypy.org/download.html Log: - Remove the outdated TODO. - Update download.html from download.txt. Modified: pypy/extradoc/pypy.org/download.html ============================================================================== --- pypy/extradoc/pypy.org/download.html (original) +++ pypy/extradoc/pypy.org/download.html Wed Oct 13 15:18:27 2010 @@ -164,6 +164,8 @@ The trunk version contains fixes to support MinGW32. More information on our dev site.

+

  • Make sure you installed the dependencies. See the list here.

    +

  • Enter the goal directory:

     cd pypy-trunk/pypy/translator/goal


From cfbolz at codespeak.net  Wed Oct 13 15:22:15 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Wed, 13 Oct 2010 15:22:15 +0200 (CEST)
Subject: [pypy-svn] r77871 - pypy/extradoc/talk/pepm2011/benchmarks
Message-ID: <20101013132215.55375282BDA@codespeak.net>

Author: cfbolz
Date: Wed Oct 13 15:22:13 2010
New Revision: 77871

Added:
   pypy/extradoc/talk/pepm2011/benchmarks/twisted_names.trace.bz2   (contents, props changed)
Removed:
   pypy/extradoc/talk/pepm2011/benchmarks/twisted-names.trace.bz2
Modified:
   pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric
   pypy/extradoc/talk/pepm2011/benchmarks/crypto_pyaes.py.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/django.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/fannkuch.py.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/go.py.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/html5lib.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/meteor-contest.py.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/nbody_modified.py.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/pyflate-fast.py.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/raytrace-simple.py.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/richards.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/spambayes.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/spectral-norm.py.trace.bz2
   pypy/extradoc/talk/pepm2011/benchmarks/telco.py.trace.bz2
Log:
redo the traces. the pyaes one will have to be redone.


Modified: pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/crypto_pyaes.py.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/django.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/fannkuch.py.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/go.py.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/html5lib.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/meteor-contest.py.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/nbody_modified.py.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/pyflate-fast.py.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/raytrace-simple.py.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/richards.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/spambayes.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/spectral-norm.py.trace.bz2
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/telco.py.trace.bz2
==============================================================================
Binary files. No diff available.

Added: pypy/extradoc/talk/pepm2011/benchmarks/twisted_names.trace.bz2
==============================================================================
Binary file. No diff available.


From leuschel at codespeak.net  Wed Oct 13 15:31:22 2010
From: leuschel at codespeak.net (leuschel at codespeak.net)
Date: Wed, 13 Oct 2010 15:31:22 +0200 (CEST)
Subject: [pypy-svn] r77872 - pypy/extradoc/talk/pepm2011
Message-ID: <20101013133122.CD52D282BDA@codespeak.net>

Author: leuschel
Date: Wed Oct 13 15:31:20 2010
New Revision: 77872

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
some changes to Sect 1


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Wed Oct 13 15:31:20 2010
@@ -131,7 +131,7 @@
 
 A recently popular approach to implementing just-in-time compilers for dynamic
 languages is that of a tracing JIT. A tracing JIT works by observing the running
-program and recording its hot parts into linear execution traces, which are then turned into
+program and recording its hot spots into linear execution traces, which are then turned into
 machine code. One reason for the popularity of tracing JITs is their relative
 simplicity. They can often be added to an interpreter and a lot of the
 infrastructure of an interpreter can be reused. They give some important
@@ -140,16 +140,16 @@
 hard in a compiler, such as register allocation.
 
 The usage of a tracing JIT can remove the overhead of bytecode dispatch and that
-of the interpreter data structures. In this paper we want to present an approach
-how an optimization can be added to a tracing JIT that also removes some of the
+of the interpreter data structures. In this paper we want to present a new
+optimization that can be added to a tracing JIT that further removes some of the
 overhead more closely associated to dynamic languages, such as boxing overhead
 and type dispatching. Our experimental platform is the PyPy project, which is an
 environment for implementing dynamic programming languages. PyPy and tracing
 JITs are described in more detail in Section~\ref{sec:Background}.
 Section~\ref{sec:lifetimes} analyzes the problem to be solved more closely.
 
-The most important technique we use to achieve to optimize traces is a
-straightforward application of partial evaluation. The partial evaluation
+The core of our trace optimization technique can be
+viewed as partial evaluation: the partial evaluation
 performs a form of escape analysis \cite{bruno_blanchet_escape_2003} on the traces and make some
 objects that are allocated in the trace \emph{static}\footnote{These objects are
 called \emph{virtual} in Psyco \cite{rigo_representation-based_2004}.} which


From arigo at codespeak.net  Wed Oct 13 15:35:34 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 13 Oct 2010 15:35:34 +0200 (CEST)
Subject: [pypy-svn] r77873 - in pypy/trunk/pypy/jit/codewriter: . test
Message-ID: <20101013133534.0AA61282BE0@codespeak.net>

Author: arigo
Date: Wed Oct 13 15:35:33 2010
New Revision: 77873

Modified:
   pypy/trunk/pypy/jit/codewriter/jtransform.py
   pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py
Log:
Test and fix: optimized unicode equality functions that take a single
char as argument were incorrectly typed like the string equality function,
and took a lltype.Char instead of lltype.UniChar.


Modified: pypy/trunk/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/jtransform.py	Wed Oct 13 15:35:33 2010
@@ -1057,28 +1057,32 @@
                             [c_func] + [varoftype(T) for T in argtypes],
                             varoftype(resulttype))
         calldescr = self.callcontrol.getcalldescr(op, oopspecindex)
-        func = heaptracker.adr2int(
-            llmemory.cast_ptr_to_adr(c_func.value))
+        if isinstance(c_func.value, str):    # in tests only
+            func = c_func.value
+        else:
+            func = heaptracker.adr2int(
+                llmemory.cast_ptr_to_adr(c_func.value))
         _callinfo_for_oopspec[oopspecindex] = calldescr, func
 
     def _handle_stroruni_call(self, op, oopspec_name, args):
-        if args[0].concretetype.TO == rstr.STR:
+        SoU = args[0].concretetype     # Ptr(STR) or Ptr(UNICODE)
+        if SoU.TO == rstr.STR:
             dict = {"stroruni.concat": EffectInfo.OS_STR_CONCAT,
                     "stroruni.slice":  EffectInfo.OS_STR_SLICE,
                     "stroruni.equal":  EffectInfo.OS_STR_EQUAL,
                     }
-        elif args[0].concretetype.TO == rstr.UNICODE:
+            CHR = lltype.Char
+        elif SoU.TO == rstr.UNICODE:
             dict = {"stroruni.concat": EffectInfo.OS_UNI_CONCAT,
                     "stroruni.slice":  EffectInfo.OS_UNI_SLICE,
                     "stroruni.equal":  EffectInfo.OS_UNI_EQUAL,
                     }
+            CHR = lltype.UniChar
         else:
             assert 0, "args[0].concretetype must be STR or UNICODE"
         #
         if oopspec_name == "stroruni.equal":
-            SoU = args[0].concretetype     # Ptr(STR) or Ptr(UNICODE)
             for otherindex, othername, argtypes, resulttype in [
-
                 (EffectInfo.OS_STREQ_SLICE_CHECKNULL,
                      "str.eq_slice_checknull",
                      [SoU, lltype.Signed, lltype.Signed, SoU],
@@ -1089,7 +1093,7 @@
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_SLICE_CHAR,
                      "str.eq_slice_char",
-                     [SoU, lltype.Signed, lltype.Signed, lltype.Char],
+                     [SoU, lltype.Signed, lltype.Signed, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_NONNULL,
                      "str.eq_nonnull",
@@ -1097,11 +1101,11 @@
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_NONNULL_CHAR,
                      "str.eq_nonnull_char",
-                     [SoU, lltype.Char],
+                     [SoU, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_CHECKNULL_CHAR,
                      "str.eq_checknull_char",
-                     [SoU, lltype.Char],
+                     [SoU, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_LENGTHOK,
                      "str.eq_lengthok",

Modified: pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py	Wed Oct 13 15:35:33 2010
@@ -79,6 +79,30 @@
         return 'builtin'
     def getcalldescr(self, op, oopspecindex=None):
         assert oopspecindex is not None    # in this test
+        EI = effectinfo.EffectInfo
+        if oopspecindex != EI.OS_ARRAYCOPY:
+            PSTR = lltype.Ptr(rstr.STR)
+            PUNICODE = lltype.Ptr(rstr.UNICODE)
+            INT = lltype.Signed
+            UNICHAR = lltype.UniChar
+            argtypes = {
+             EI.OS_STR2UNICODE:([PSTR], PUNICODE),
+             EI.OS_STR_CONCAT: ([PSTR, PSTR], PSTR),
+             EI.OS_STR_SLICE:  ([PSTR, INT, INT], PSTR),
+             EI.OS_UNI_CONCAT: ([PUNICODE, PUNICODE], PUNICODE),
+             EI.OS_UNI_SLICE:  ([PUNICODE, INT, INT], PUNICODE),
+             EI.OS_UNI_EQUAL:  ([PUNICODE, PUNICODE], lltype.Bool),
+             EI.OS_UNIEQ_SLICE_CHECKNULL:([PUNICODE, INT, INT, PUNICODE], INT),
+             EI.OS_UNIEQ_SLICE_NONNULL:  ([PUNICODE, INT, INT, PUNICODE], INT),
+             EI.OS_UNIEQ_SLICE_CHAR:     ([PUNICODE, INT, INT, UNICHAR], INT),
+             EI.OS_UNIEQ_NONNULL:        ([PUNICODE, PUNICODE], INT),
+             EI.OS_UNIEQ_NONNULL_CHAR:   ([PUNICODE, UNICHAR], INT),
+             EI.OS_UNIEQ_CHECKNULL_CHAR: ([PUNICODE, UNICHAR], INT),
+             EI.OS_UNIEQ_LENGTHOK:       ([PUNICODE, PUNICODE], INT),
+            }
+            argtypes = argtypes[oopspecindex]
+            assert argtypes[0] == [v.concretetype for v in op.args[1:]]
+            assert argtypes[1] == op.result.concretetype
         return 'calldescr-%d' % oopspecindex
     def calldescr_canraise(self, calldescr):
         return False
@@ -785,6 +809,28 @@
     assert op1.args[2] == ListOfKind('ref', [v1])
     assert op1.result == v2
 
+def test_unicode_eq_checknull_char():
+    # test that the oopspec is present and correctly transformed
+    PUNICODE = lltype.Ptr(rstr.UNICODE)
+    FUNC = lltype.FuncType([PUNICODE, PUNICODE], lltype.Bool)
+    func = lltype.functionptr(FUNC, 'll_streq',
+                              _callable=rstr.LLHelpers.ll_streq)
+    v1 = varoftype(PUNICODE)
+    v2 = varoftype(PUNICODE)
+    v3 = varoftype(lltype.Bool)
+    op = SpaceOperation('direct_call', [const(func), v1, v2], v3)
+    tr = Transformer(FakeCPU(), FakeBuiltinCallControl())
+    op1 = tr.rewrite_operation(op)
+    assert op1.opname == 'residual_call_r_i'
+    assert op1.args[0].value == func
+    assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_UNI_EQUAL
+    assert op1.args[2] == ListOfKind('ref', [v1, v2])
+    assert op1.result == v3
+    # test that the OS_UNIEQ_* functions are registered
+    cifo = effectinfo._callinfo_for_oopspec
+    assert effectinfo.EffectInfo.OS_UNIEQ_SLICE_NONNULL in cifo
+    assert effectinfo.EffectInfo.OS_UNIEQ_CHECKNULL_CHAR in cifo
+
 def test_list_ll_arraycopy():
     from pypy.rlib.rgc import ll_arraycopy
     LIST = lltype.GcArray(lltype.Signed)


From afa at codespeak.net  Wed Oct 13 15:53:09 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 13 Oct 2010 15:53:09 +0200 (CEST)
Subject: [pypy-svn] r77874 - in pypy/branch/fast-forward/pypy:
	module/_hashlib module/_hashlib/test rlib
Message-ID: <20101013135309.D6C80282BE0@codespeak.net>

Author: afa
Date: Wed Oct 13 15:53:07 2010
New Revision: 77874

Modified:
   pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py
   pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py
   pypy/branch/fast-forward/pypy/rlib/ropenssl.py
Log:
Implement the _hashlib module


Modified: pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py	Wed Oct 13 15:53:07 2010
@@ -1,9 +1,10 @@
 from pypy.interpreter.gateway import unwrap_spec, interp2app
-from pypy.interpreter.typedef import TypeDef
+from pypy.interpreter.typedef import TypeDef, GetSetProperty
 from pypy.tool.sourcetools import func_renamer
 from pypy.interpreter.baseobjspace import Wrappable, W_Root, ObjSpace
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.rlib import ropenssl
+from pypy.rlib.rstring import StringBuilder
 
 algorithms = ('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512')
 
@@ -12,37 +13,105 @@
         self.name = name
         self.ctx = lltype.malloc(ropenssl.EVP_MD_CTX.TO, flavor='raw')
 
-    @unwrap_spec('self', ObjSpace, str, str)
-    def descr_init(self, space, name, buffer):
-        digest = ropenssl.EVT_get_digestbyname(name)
+        digest = ropenssl.EVP_get_digestbyname(name)
         if not digest:
             raise OperationError(space.w_ValueError,
                                  space.wrap("unknown hash function"))
         ropenssl.EVP_DigestInit(self.ctx, digest)
 
-        if buffer:
-            self._hash(buffer)
-
     @unwrap_spec('self', ObjSpace)
     def descr_repr(self, space):
         return space.wrap("<%s HASH object @ 0x%x>" % (self.name, id(self)))
 
     @unwrap_spec('self', ObjSpace, str)
     def update(self, space, buffer):
-        self._hash(buffer)
-
-    def _hash(self, buffer):
         buf = rffi.str2charp(buffer)
         try:
             ropenssl.EVP_DigestUpdate(self.ctx, buf, len(buffer))
         finally:
             rffi.free_charp(buf)
 
+    @unwrap_spec('self', ObjSpace)
+    def copy(self, space):
+        "Return a copy of the hash object."
+        w_hash = W_Hash(self.name)
+        ropenssl.EVP_MD_CTX_copy(w_hash.ctx, self.ctx)
+        return w_hash
+
+    @unwrap_spec('self', ObjSpace)
+    def digest(self, space):
+        "Return the digest value as a string of binary data."
+        digest = self._digest(space)
+        return space.wrap(digest)
+
+    @unwrap_spec('self', ObjSpace)
+    def hexdigest(self, space):
+        "Return the digest value as a string of hexadecimal digits."
+        digest = self._digest(space)
+        hexdigits = '0123456789abcdef'
+        result = StringBuilder(self._digest_size() * 2)
+        for c in digest:
+            result.append(hexdigits[(ord(c) >> 4) & 0xf])
+            result.append(hexdigits[ ord(c)       & 0xf])
+        return space.wrap(result.build())
+
+    def get_digest_size(space, self):
+        return space.wrap(self._digest_size())
+
+    def get_block_size(space, self):
+        return space.wrap(self._block_size())
+
+    def _digest(self, space):
+        ctx = self.copy(space).ctx
+        digest_size = self._digest_size()
+        digest = lltype.malloc(rffi.CCHARP.TO, digest_size, flavor='raw')
+
+        try:
+            ropenssl.EVP_DigestFinal(ctx, digest, None)
+            return rffi.charp2strn(digest, digest_size)
+        finally:
+            lltype.free(digest, flavor='raw')
+
+
+    def _digest_size(self):
+        # XXX This isn't the nicest way, but the EVP_MD_size OpenSSL
+        # XXX function is defined as a C macro on OS X and would be
+        # XXX significantly harder to implement in another way.
+        # Values are digest sizes in bytes
+        return {
+            'md5': 16,
+            'sha1': 20,
+            'sha224': 28,
+            'sha256': 32,
+            'sha384': 48,
+            'sha512': 64,
+            }.get(self.name, 0)
+
+    def _block_size(self):
+        # XXX This isn't the nicest way, but the EVP_MD_CTX_block_size
+        # XXX OpenSSL function is defined as a C macro on some systems
+        # XXX and would be significantly harder to implement in
+        # XXX another way.
+        return {
+            'md5':     64,
+            'sha1':    64,
+            'sha224':  64,
+            'sha256':  64,
+            'sha384': 128,
+            'sha512': 128,
+            }.get(self.name, 0)
+
 W_Hash.typedef = TypeDef(
     'HASH',
-    __init__=interp2app(W_Hash.descr_init),
     __repr__=interp2app(W_Hash.descr_repr),
     update=interp2app(W_Hash.update),
+    copy=interp2app(W_Hash.copy),
+    digest=interp2app(W_Hash.digest),
+    hexdigest=interp2app(W_Hash.hexdigest),
+    #
+    digest_size=GetSetProperty(W_Hash.get_digest_size),
+    digestsize=GetSetProperty(W_Hash.get_digest_size),
+    block_size=GetSetProperty(W_Hash.get_block_size),
     )
 
 @unwrap_spec(ObjSpace, str, str)

Modified: pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py	Wed Oct 13 15:53:07 2010
@@ -10,7 +10,7 @@
         assert isinstance(_hashlib.new('md5'), _hashlib.HASH)
 
     def test_attributes(self):
-        import _hashlib
+        import hashlib
         for name, expected_size in {'md5': 16,
                                     'sha1': 20,
                                     'sha224': 28,
@@ -29,8 +29,12 @@
             hexdigest = h.hexdigest()
             h2.update('d')
             h2.update('ef')
-            assert digest == h.digest()
-            assert hexdigest == h.hexdigest()
+            assert digest    == h2.digest()
+            assert hexdigest == h2.hexdigest()
+            assert len(digest)    == h.digest_size
+            assert len(hexdigest) == h.digest_size * 2
+            c_digest    = digest
+            c_hexdigest = hexdigest
 
             # also test the pure Python implementation
             h = hashlib.__get_builtin_constructor(name)('')
@@ -44,10 +48,14 @@
             hexdigest = h.hexdigest()
             h2.update('d')
             h2.update('ef')
-            assert digest == h.digest()
-            assert hexdigest == h.hexdigest()
+            assert digest    == h2.digest()
+            assert hexdigest == h2.hexdigest()
+
+            # compare both implementations
+            assert c_digest    == digest
+            assert c_hexdigest == hexdigest
 
     def test_unicode(self):
         import _hashlib
-        assert isinstance(hashlib.new('sha1', u'xxx'), _hashlib.HASH)
+        assert isinstance(_hashlib.new('sha1', u'xxx'), _hashlib.HASH)
 

Modified: pypy/branch/fast-forward/pypy/rlib/ropenssl.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/ropenssl.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/ropenssl.py	Wed Oct 13 15:53:07 2010
@@ -16,10 +16,12 @@
         # so that openssl/ssl.h can repair this nonsense.
         'wincrypt.h',
         'openssl/ssl.h',
-        'openssl/err.h']
+        'openssl/err.h',
+        'openssl/evp.h']
 else:
     libraries = ['ssl', 'crypto']
-    includes = ['openssl/ssl.h', 'openssl/err.h']
+    includes = ['openssl/ssl.h', 'openssl/err.h',
+                'openssl/evp.h']
 
 eci = ExternalCompilationInfo(
     libraries = libraries,
@@ -176,7 +178,7 @@
 
 ssl_external('SSL_read', [SSL, rffi.CCHARP, rffi.INT], rffi.INT)
 
-EVP_MD_CTX = rffi.COpaquePtr('EVP_MD_CTX')
+EVP_MD_CTX = rffi.COpaquePtr('EVP_MD_CTX', compilation_info=eci)
 EVP_MD     = rffi.COpaquePtr('EVP_MD')
 
 EVP_get_digestbyname = external(
@@ -188,6 +190,13 @@
 EVP_DigestUpdate = external(
     'EVP_DigestUpdate',
     [EVP_MD_CTX, rffi.CCHARP, rffi.SIZE_T], rffi.INT)
+EVP_DigestFinal = external(
+    'EVP_DigestFinal',
+    [EVP_MD_CTX, rffi.CCHARP, rffi.VOIDP], rffi.INT)
+EVP_MD_CTX_copy = external(
+    'EVP_MD_CTX_copy', [EVP_MD_CTX, EVP_MD_CTX], rffi.INT)
+EVP_MD_CTX_cleanup = external(
+    'EVP_MD_CTX_cleanup', [EVP_MD_CTX], rffi.INT)
 
 def _init_ssl():
     libssl_SSL_load_error_strings()


From arigo at codespeak.net  Wed Oct 13 16:03:30 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 13 Oct 2010 16:03:30 +0200 (CEST)
Subject: [pypy-svn] r77875 -
	pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem
Message-ID: <20101013140330.33688282BE7@codespeak.net>

Author: arigo
Date: Wed Oct 13 16:03:28 2010
New Revision: 77875

Modified:
   pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py
Log:
Fix.


Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py	Wed Oct 13 16:03:28 2010
@@ -1158,6 +1158,8 @@
     _TYPE = llmemory.Address
 
     def __init__(self, void_p):
+        if isinstance(void_p, _lladdress):
+            void_p = void_p.void_p
         if isinstance(void_p, (int, long)):
             void_p = ctypes.c_void_p(void_p)
         self.void_p = void_p
@@ -1170,6 +1172,8 @@
         return '<_lladdress %s>' % (self.void_p,)
 
     def __eq__(self, other):
+        if isinstance(other, _lladdress):
+            other = other.intval
         if not isinstance(other, (int, long)):
             other = cast_adr_to_int(other)
         return intmask(other) == self.intval


From leuschel at codespeak.net  Wed Oct 13 16:05:30 2010
From: leuschel at codespeak.net (leuschel at codespeak.net)
Date: Wed, 13 Oct 2010 16:05:30 +0200 (CEST)
Subject: [pypy-svn] r77876 - pypy/extradoc/talk/pepm2011
Message-ID: <20101013140530.7E8BA282BEA@codespeak.net>

Author: leuschel
Date: Wed Oct 13 16:05:28 2010
New Revision: 77876

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
some more changes


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Wed Oct 13 16:05:28 2010
@@ -179,8 +179,8 @@
 
 The work described in this paper was done in the context of the PyPy project
 \cite{armin_rigo_pypys_2006}. PyPy is an environment where dynamic languages can
-be implemented in a simple yet efficient way. The approach taken when
-implementing a language with PyPy is to write an \emph{interpreter}
+be implemented in a simple yet efficient way. 
+When implementing a language with PyPy one writes an \emph{interpreter}
 for the language in
 \emph{RPython} \cite{davide_ancona_rpython:_2007}. RPython ("restricted Python")
 is a subset of Python chosen in such a way that type inference becomes
@@ -191,7 +191,7 @@
 aspects of the final VM are woven into the generated code during the translation
 to C.
 
-A number of languages have been implemented with PyPy. The project was started
+A number of languages have been implemented with PyPy. The project was initiated
 to get a better Python implementation, which inspired the name of the project
 and is still the main focus of development. In addition a number of other
 languages were implemented, among them a Prolog interpreter
@@ -203,7 +203,7 @@
 support for automated JIT compiler generation \cite{bolz_tracing_2009}. During
 the translation to C, PyPy's tools can generate a just-in-time compiler for the
 language that the interpreter is implementing. This process is mostly
-automatic; it only needs to be guided by the language implementer by a small number of
+automatic; it only needs to be guided by the language implementer using a small number of
 source-code hints. Mostly-automatically generating a JIT compiler has many advantages
 over writing one manually, which is an error-prone and tedious process.
 By construction, the generated JIT has the same semantics as the interpreter.
@@ -214,7 +214,7 @@
 to the original backend for the Intel \emph{x86} architecture.  Examples of
 additional JIT backends are the one for Intel \emph{x86-64} and an
 experimental one for the CLI .NET Virtual Machine \cite{cuni_python_cli_2010}.
-The JIT that is produced by PyPy's JIT generator is a \emph{tracing JIT
+PyPy's JIT generator generates a \emph{tracing JIT
 compiler}, a concept which we now explain in more details.
 
 \subsection{Tracing JIT Compilers}
@@ -245,7 +245,7 @@
 tracing JIT automatically performs inlining.
 
 This trace of operations is then the basis of the generated code. The trace is
-optimized in some ways, and then turned into machine code. Both optimizations
+first optimized, and then turned into machine code. Both optimizations
 and machine code generation is simple, because the traces are linear. This
 linearity makes many optimizations a lot more tractable, and the inlining that
 happens gives the optimizations automatically more context to work with.
@@ -277,17 +277,20 @@
 
 \subsection{Running Example}
 
-For the purpose of this paper, we are going to use a very simple object
+For the purpose of this paper, we are going to use a tiny interpreter for a dynamic language with
+ a very simple object
 model, that just supports an integer and a float type. The objects support only
 two operations, \texttt{add}, which adds two objects (promoting ints to floats in a
 mixed addition) and \texttt{is\_positive}, which returns whether the number is greater
 than zero. The implementation of \texttt{add} uses classical Smalltalk-like
-double-dispatching. These classes could be part of the implementation of a very
-simple interpreter written in RPython. The classes can be seen in
-Figure~\ref{fig:objmodel}.
+double-dispatching.
+%These classes could be part of the implementation of a very
+%simple interpreter written in RPython.
+The classes can be seen in
+Figure~\ref{fig:objmodel} (written in RPython).
 
-\begin{figure}
-\begin{verbatim}
+\newcommand{\ignore}[1]{} % {{\tt \small ignore(#1)}}
+\ignore{
 class Base(object):
     def add(self, other):
         """ add self to other """
@@ -303,6 +306,10 @@
     def is_positive(self):
         """ returns whether self is positive """
         raise NotImplementedError("abstract base")
+        }
+\begin{figure}
+\begin{verbatim}
+class Base(object):
 
 class BoxedInteger(Base):
     def __init__(self, intval):
@@ -338,7 +345,8 @@
         y = y.add(BoxedInteger(-1))
     return res
 \end{verbatim}
-\caption{A Simple Object Model and an Example Function Using it}
+\caption{An ``interpreter'' for a tiny Dynamic Language written in RPython}
+%\caption{A Simple Object Model and an Example Function Using it}
 \label{fig:objmodel}
 \end{figure}
 
@@ -350,7 +358,8 @@
 implement the numeric tower needs two method calls per arithmetic operation,
 which is costly due to the method dispatch.
 
-To understand the problems more directly, let us consider the simple function
+To understand the problems more directly, let us consider the simple
+interpreter function
 \texttt{f} that uses the object model (see the bottom of
 Figure~\ref{fig:objmodel}).
 
@@ -434,12 +443,18 @@
 
 If the function is executed using the tracing JIT, with \texttt{y} being a
 \texttt{BoxedInteger}, the produced trace looks like
-Figure~\ref{fig:unopt-trace}. The operations in the trace are shown indented to
+Figure~\ref{fig:unopt-trace} (lines starting with the hash ``\#'' are comments).
+
+XXX in which language is the trace written in ? still RPython ?
+
+The operations in the trace are shown indented to
 correspond to the stack level of the function that contains the traced
 operation. The trace is in single-assignment form, meaning that each variable is
 assigned to exactly once. The arguments $p_0$ and $p_1$ of the loop correspond
 to the live variables \texttt{y} and \texttt{res} in the original function.
 
+XXX explain set and get + int_add briefly
+
 The trace shows the inefficiencies of \texttt{f} clearly, if one
 looks at the number of \texttt{new} (corresponding to object creation),
 \texttt{set/get} (corresponding to attribute reads/writes) and


From arigo at codespeak.net  Wed Oct 13 16:12:18 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 13 Oct 2010 16:12:18 +0200 (CEST)
Subject: [pypy-svn] r77877 -
	pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem
Message-ID: <20101013141218.9E3DD282BEC@codespeak.net>

Author: arigo
Date: Wed Oct 13 16:12:17 2010
New Revision: 77877

Modified:
   pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py
Log:
More fixes.


Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/ll2ctypes.py	Wed Oct 13 16:12:17 2010
@@ -587,6 +587,8 @@
         res = intmask(cobj.value)
         _int2obj[res] = llobj.adr.ptr._obj
         return res
+    if isinstance(llobj, _lladdress):
+        llobj = llobj.intval
     if isinstance(llobj, llmemory.fakeaddress):
         llobj = llobj.ptr or 0
 
@@ -1192,6 +1194,8 @@
     _name = "_llgcopaque"
 
     def __init__(self, void_p):
+        if isinstance(void_p, _lladdress):
+            void_p = void_p.intval
         if isinstance(void_p, (int, long)):
             self.intval = intmask(void_p)
         else:


From arigo at codespeak.net  Wed Oct 13 16:32:44 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 13 Oct 2010 16:32:44 +0200 (CEST)
Subject: [pypy-svn] r77879 -
	pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem
Message-ID: <20101013143244.24BF8282BF7@codespeak.net>

Author: arigo
Date: Wed Oct 13 16:32:42 2010
New Revision: 77879

Modified:
   pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py
   pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rtupletype.py
Log:
Fix for jit.metainterp.test.test_dict.  Reverts the signature of
ll_kvi() in rdict.py to what it was, and in case of 'dict.items()',
fish for the correct tuple type manually from inside the function.


Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rdict.py	Wed Oct 13 16:32:42 2010
@@ -275,14 +275,10 @@
 
     def _rtype_method_kvi(self, hop, ll_func):
         v_dic, = hop.inputargs(self)
-        s_list = hop.s_result
-        s_value = s_list.listdef.listitem.s_value
-        r_value = hop.rtyper.getrepr(s_value)
-        cLISTITEM = hop.inputconst(lltype.Void, r_value.lowleveltype)
         r_list = hop.r_result
         cLIST = hop.inputconst(lltype.Void, r_list.lowleveltype.TO)
         hop.exception_cannot_occur()
-        return hop.gendirectcall(ll_func, cLIST, cLISTITEM, v_dic)
+        return hop.gendirectcall(ll_func, cLIST, v_dic)
 
     def rtype_method_keys(self, hop):
         return self._rtype_method_kvi(hop, ll_dict_keys)
@@ -782,7 +778,17 @@
         return v
 
 def _make_ll_keys_values_items(kind):
-    def ll_kvi(LIST, LISTITEM, dic):
+    from pypy.rpython.lltypesystem.rtupletype import TUPLE_TYPE_2
+    #
+    def ll_kvi(LIST, dic):
+        DICT = lltype.typeOf(dic).TO
+        if kind == 'items':
+            LISTITEM = TUPLE_TYPE_2(DICT.KEY, DICT.VALUE)
+        elif kind == 'keys':
+            LISTITEM = DICT.KEY
+        elif kind == 'values':
+            LISTITEM = DICT.VALUE
+        #
         res = LIST.ll_newlist(dic.num_items)
         entries = dic.entries
         dlen = len(entries)
@@ -807,6 +813,7 @@
             i += 1
         assert p == res.ll_length()
         return res
+    #
     ll_kvi.oopspec = 'dict.%s(dic)' % kind
     return ll_kvi
 

Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rtupletype.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rtupletype.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rtupletype.py	Wed Oct 13 16:32:42 2010
@@ -13,3 +13,7 @@
         kwds = {'hints': {'immutable': True,
                           'noidentity': True}}
         return Ptr(GcStruct('tuple%d' % len(field_lltypes), *fields, **kwds))
+
+def TUPLE_TYPE_2(type1, type2):    # hack for annotation
+    return TUPLE_TYPE([type1, type2])
+TUPLE_TYPE_2._annspecialcase_ = 'specialize:memo'


From arigo at codespeak.net  Wed Oct 13 16:34:03 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 13 Oct 2010 16:34:03 +0200 (CEST)
Subject: [pypy-svn] r77880 -
	pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test
Message-ID: <20101013143403.D93BB282BFA@codespeak.net>

Author: arigo
Date: Wed Oct 13 16:34:02 2010
New Revision: 77880

Modified:
   pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gctypelayout.py
Log:
Fix test.


Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gctypelayout.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gctypelayout.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/rpython/memory/test/test_gctypelayout.py	Wed Oct 13 16:34:02 2010
@@ -38,9 +38,10 @@
         assert len(offsets_to_gc_pointers(T)) == c
 
 def test_hiddenptr32():
-    from pypy.rpython.lltypesystem.llmemory import HIDDENPTR32
+    from pypy.rpython.lltypesystem.llmemory import HiddenGcRef32
     from pypy.rpython.lltypesystem.llmemory import has_odd_value_marker
-    T = lltype.GcStruct('T', ('foo', lltype.Ptr(GC_S3)), ('bar', HIDDENPTR32))
+    T = lltype.GcStruct('T', ('foo', lltype.Ptr(GC_S3)),
+                             ('bar', HiddenGcRef32))
     ofs = offsets_to_gc_pointers(T)
     assert len(ofs) == 2
     assert not has_odd_value_marker(ofs[0])


From arigo at codespeak.net  Wed Oct 13 16:43:38 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 13 Oct 2010 16:43:38 +0200 (CEST)
Subject: [pypy-svn] r77881 - pypy/branch/32ptr-on-64bit/pypy/doc/config
Message-ID: <20101013144338.8E071282BFB@codespeak.net>

Author: arigo
Date: Wed Oct 13 16:43:37 2010
New Revision: 77881

Added:
   pypy/branch/32ptr-on-64bit/pypy/doc/config/translation.compressptr.txt
Log:
Documentation.


Added: pypy/branch/32ptr-on-64bit/pypy/doc/config/translation.compressptr.txt
==============================================================================
--- (empty file)
+++ pypy/branch/32ptr-on-64bit/pypy/doc/config/translation.compressptr.txt	Wed Oct 13 16:43:37 2010
@@ -0,0 +1,7 @@
+On 64-bit platforms, "compress" some pointers to 32 bits.
+
+Pointers to GC-managed, fixed-size objects are compressed.
+This saves potentially a lot of memory.  The drawback is that
+it will refuse to allocate more than 32GB of RAM for fixed-size
+objects.  (The program can use more RAM in total, e.g. with big
+strings, which are not fixed-size objects.)


From antocuni at codespeak.net  Wed Oct 13 17:12:22 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Wed, 13 Oct 2010 17:12:22 +0200 (CEST)
Subject: [pypy-svn] r77882 - pypy/extradoc/talk/pepm2011
Message-ID: <20101013151222.65363282BFC@codespeak.net>

Author: antocuni
Date: Wed Oct 13 17:12:20 2010
New Revision: 77882

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
make the code nicer


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Wed Oct 13 17:12:20 2010
@@ -10,6 +10,27 @@
 \usepackage{amsmath}
 \usepackage{amsfonts}
 \usepackage[utf8]{inputenc}
+\usepackage{setspace}
+
+\usepackage{listings}
+
+\usepackage[T1]{fontenc}
+\usepackage{beramono}
+
+
+\definecolor{gray}{rgb}{0.3,0.3,0.3}
+
+\lstset{
+  basicstyle=\setstretch{1.1}\ttfamily\footnotesize,
+  language=Python,
+  keywordstyle=\bfseries,
+  stringstyle=\color{blue},
+  commentstyle=\color{gray}\textit,
+  fancyvrb=true,
+  showstringspaces=false,
+  keywords={def,while,if,elif,return,class,get,set,new,guard_class}
+}
+
 
 \newboolean{showcomments}
 \setboolean{showcomments}{true}
@@ -255,10 +276,10 @@
 To make sure that the trace is maintaining the correct semantics, it contains a
 \emph{guard} at all places where the execution could have diverged from the
 path. Those guards check the assumptions under which execution can stay on the
-trace. As an example, if a loop contains an \texttt{if} statement, the trace
+trace. As an example, if a loop contains an \lstinline{if} statement, the trace
 will contain the execution of one of the paths only, which is the path that was
 taken during the production of the trace. The trace will also contain a guard
-that checks that the condition of the \texttt{if} statement is true, because if
+that checks that the condition of the \lstinline{if} statement is true, because if
 it isn't, the rest of the trace is not valid.
 
 When generating machine code, every guard is be turned into a quick check to
@@ -280,9 +301,9 @@
 For the purpose of this paper, we are going to use a tiny interpreter for a dynamic language with
  a very simple object
 model, that just supports an integer and a float type. The objects support only
-two operations, \texttt{add}, which adds two objects (promoting ints to floats in a
-mixed addition) and \texttt{is\_positive}, which returns whether the number is greater
-than zero. The implementation of \texttt{add} uses classical Smalltalk-like
+two operations, \lstinline{add}, which adds two objects (promoting ints to floats in a
+mixed addition) and \lstinline{is_positive}, which returns whether the number is greater
+than zero. The implementation of \lstinline{add} uses classical Smalltalk-like
 double-dispatching.
 %These classes could be part of the implementation of a very
 %simple interpreter written in RPython.
@@ -308,43 +329,52 @@
         raise NotImplementedError("abstract base")
         }
 \begin{figure}
-\begin{verbatim}
+\begin{lstlisting}[mathescape]
 class Base(object):
+      ...
 
 class BoxedInteger(Base):
-    def __init__(self, intval):
-        self.intval = intval
-    def add(self, other):
-        return other.add__int(self.intval)
-    def add__int(self, intother):
-        return BoxedInteger(intother + self.intval)
-    def add__float(self, floatother):
-        floatvalue = floatother + float(self.intval)
-        return BoxedFloat(floatvalue)
-    def is_positive(self):
-        return self.intval > 0
+   def __init__(self, intval):
+      self.intval = intval
+
+   def add(self, other):
+      return other.add__int(self.intval)
+
+   def add__int(self, intother):
+      return BoxedInteger(intother + self.intval)
+
+   def add__float(self, floatother):
+      floatvalue = floatother + float(self.intval)
+      return BoxedFloat(floatvalue)
+
+   def is_positive(self):
+      return self.intval > 0
 
 class BoxedFloat(Base):
-    def __init__(self, floatval):
-        self.floatval = floatval
-    def add(self, other):
-        return other.add__float(self.floatval)
-    def add__int(self, intother):
-        floatvalue = float(intother) + self.floatval
-        return BoxedFloat(floatvalue)
-    def add__float(self, floatother):
-        return BoxedFloat(floatother + self.floatval)
-    def is_positive(self):
-        return self.floatval > 0.0
+   def __init__(self, floatval):
+      self.floatval = floatval
+
+   def add(self, other):
+      return other.add__float(self.floatval)
+
+   def add__int(self, intother):
+      floatvalue = float(intother) + self.floatval
+      return BoxedFloat(floatvalue)
+
+   def add__float(self, floatother):
+      return BoxedFloat(floatother + self.floatval)
+
+   def is_positive(self):
+      return self.floatval > 0.0
 
 
 def f(y):
-    res = BoxedInteger(0)
-    while y.is_positive():
-        res = res.add(y).add(BoxedInteger(-100))
-        y = y.add(BoxedInteger(-1))
-    return res
-\end{verbatim}
+   res = BoxedInteger(0)
+   while y.is_positive():
+      res = res.add(y).add(BoxedInteger(-100))
+      y = y.add(BoxedInteger(-1))
+   return res
+\end{lstlisting}
 \caption{An ``interpreter'' for a tiny Dynamic Language written in RPython}
 %\caption{A Simple Object Model and an Example Function Using it}
 \label{fig:objmodel}
@@ -352,7 +382,7 @@
 
 Using these classes to implement arithmetic shows the basic problem that a
 dynamic language implementation has. All the numbers are instances of either
-\texttt{BoxedInteger} or \texttt{BoxedFloat}, thus they consume space on the
+\lstinline{BoxedInteger} or \lstinline{BoxedFloat}, thus they consume space on the
 heap. Performing many arithmetic operations produces lots of garbage quickly,
 thus putting pressure on the garbage collector. Using double dispatching to
 implement the numeric tower needs two method calls per arithmetic operation,
@@ -360,7 +390,7 @@
 
 To understand the problems more directly, let us consider the simple
 interpreter function
-\texttt{f} that uses the object model (see the bottom of
+\lstinline{f} that uses the object model (see the bottom of
 Figure~\ref{fig:objmodel}).
 
 XXX this is not an RPython interpreter; put a reference to the previous
@@ -368,81 +398,80 @@
 the interpretation overhead, turning it into basically something
 equivalent to the example here, which is the start of the present paper.
 
-The loop in \texttt{f} iterates \texttt{y} times, and computes something in the process.
+The loop in \lstinline{f} iterates \lstinline{y} times, and computes something in the process.
 Simply running this function is slow, because there are lots of virtual method
-calls inside the loop, one for each \texttt{is\_positive} and even two for each
-call to \texttt{add}. These method calls need to check the type of the involved
+calls inside the loop, one for each \lstinline{is_positive} and even two for each
+call to \lstinline{add}. These method calls need to check the type of the involved
 objects repeatedly and redundantly. In addition, a lot of objects are created
 when executing that loop, many of these objects do not survive for very long.
-The actual computation that is performed by \texttt{f} is simply a sequence of
+The actual computation that is performed by \lstinline{f} is simply a sequence of
 float or integer additions.
 
 
 \begin{figure}
-\texttt{
-\begin{tabular}{l}
-\# arguments to the trace: $p_{0}$, $p_{1}$ \\
-\# inside f: res.add(y) \\
-guard\_class($p_{1}$, BoxedInteger) \\
-~~~~\# inside BoxedInteger.add \\
-~~~~$i_{2}$ = get($p_{1}$, intval) \\
-~~~~guard\_class($p_{0}$, BoxedInteger) \\
-~~~~~~~~\# inside BoxedInteger.add\_\_int \\
-~~~~~~~~$i_{3}$ = get($p_{0}$, intval) \\
-~~~~~~~~$i_{4}$ = int\_add($i_{2}$, $i_{3}$) \\
-~~~~~~~~$p_{5}$ = new(BoxedInteger) \\
-~~~~~~~~~~~~\# inside BoxedInteger.\_\_init\_\_ \\
-~~~~~~~~~~~~set($p_{5}$, intval, $i_{4}$) \\
-\# inside f: BoxedInteger(-100)  \\
-$p_{6}$ = new(BoxedInteger) \\
-~~~~\# inside BoxedInteger.\_\_init\_\_ \\
-~~~~set($p_{6}$, intval, -100) \\
-~\\
-\# inside f: .add(BoxedInteger(-100)) \\
-guard\_class($p_{5}$, BoxedInteger) \\
-~~~~\# inside BoxedInteger.add \\
-~~~~$i_{7}$ = get($p_{5}$, intval) \\
-~~~~guard\_class($p_{6}$, BoxedInteger) \\
-~~~~~~~~\# inside BoxedInteger.add\_\_int \\
-~~~~~~~~$i_{8}$ = get($p_{6}$, intval) \\
-~~~~~~~~$i_{9}$ = int\_add($i_{7}$, $i_{8}$) \\
-~~~~~~~~$p_{10}$ = new(BoxedInteger) \\
-~~~~~~~~~~~~\# inside BoxedInteger.\_\_init\_\_ \\
-~~~~~~~~~~~~set($p_{10}$, intval, $i_{9}$) \\
-~\\
-\# inside f: BoxedInteger(-1) \\
-$p_{11}$ = new(BoxedInteger) \\
-~~~~\# inside BoxedInteger.\_\_init\_\_ \\
-~~~~set($p_{11}$, intval, -1) \\
-~\\
-\# inside f: y.add(BoxedInteger(-1)) \\
-guard\_class($p_{0}$, BoxedInteger) \\
-~~~~\# inside BoxedInteger.add \\
-~~~~$i_{12}$ = get($p_{0}$, intval) \\
-~~~~guard\_class($p_{11}$, BoxedInteger) \\
-~~~~~~~~\# inside BoxedInteger.add\_\_int \\
-~~~~~~~~$i_{13}$ = get($p_{11}$, intval) \\
-~~~~~~~~$i_{14}$ = int\_add($i_{12}$, $i_{13}$) \\
-~~~~~~~~$p_{15}$ = new(BoxedInteger) \\
-~~~~~~~~~~~~\# inside BoxedInteger.\_\_init\_\_ \\
-~~~~~~~~~~~~set($p_{15}$, intval, $i_{14}$) \\
-~\\
-\# inside f: y.is\_positive() \\
-guard\_class($p_{15}$, BoxedInteger) \\
-~~~~\# inside BoxedInteger.is\_positive \\
-~~~~$i_{16}$ = get($p_{15}$, intval) \\
-~~~~$i_{17}$ = int\_gt($i_{16}$, 0) \\
-\# inside f \\
-guard\_true($i_{17}$) \\
-jump($p_{15}$, $p_{10}$) \\
-\end{tabular}
-}
+\begin{lstlisting}[mathescape]
+# arguments to the trace: $p_{0}$, $p_{1}$
+# inside f: res.add(y)
+guard_class($p_{1}$, BoxedInteger)
+    # inside BoxedInteger.add
+    $i_{2}$ = get($p_{1}$, intval)
+    guard_class($p_{0}$, BoxedInteger)
+        # inside BoxedInteger.add__int
+        $i_{3}$ = get($p_{0}$, intval)
+        $i_{4}$ = int_add($i_{2}$, $i_{3}$)
+        $p_{5}$ = new(BoxedInteger)
+            # inside BoxedInteger.__init__
+            set($p_{5}$, intval, $i_{4}$)
+
+# inside f: BoxedInteger(-100) 
+$p_{6}$ = new(BoxedInteger)
+    # inside BoxedInteger.__init__
+    set($p_{6}$, intval, -100)
+
+# inside f: .add(BoxedInteger(-100))
+guard_class($p_{5}$, BoxedInteger)
+    # inside BoxedInteger.add
+    $i_{7}$ = get($p_{5}$, intval)
+    guard_class($p_{6}$, BoxedInteger)
+        # inside BoxedInteger.add__int
+        $i_{8}$ = get($p_{6}$, intval)
+        $i_{9}$ = int_add($i_{7}$, $i_{8}$)
+        $p_{10}$ = new(BoxedInteger)
+            # inside BoxedInteger.__init__
+            set($p_{10}$, intval, $i_{9}$)
+
+# inside f: BoxedInteger(-1)
+$p_{11}$ = new(BoxedInteger)
+    # inside BoxedInteger.__init__
+    set($p_{11}$, intval, -1)
+
+# inside f: y.add(BoxedInteger(-1))
+guard_class($p_{0}$, BoxedInteger)
+    # inside BoxedInteger.add
+    $i_{12}$ = get($p_{0}$, intval)
+    guard_class($p_{11}$, BoxedInteger)
+        # inside BoxedInteger.add__int
+        $i_{13}$ = get($p_{11}$, intval)
+        $i_{14}$ = int_add($i_{12}$, $i_{13}$)
+        $p_{15}$ = new(BoxedInteger)
+            # inside BoxedInteger.__init__
+            set($p_{15}$, intval, $i_{14}$)
+
+# inside f: y.is_positive()
+guard_class($p_{15}$, BoxedInteger)
+    # inside BoxedInteger.is_positive
+    $i_{16}$ = get($p_{15}$, intval)
+    $i_{17}$ = int_gt($i_{16}$, 0)
+# inside f
+guard_true($i_{17}$)
+jump($p_{15}$, $p_{10}$)
+\end{lstlisting}
 \caption{Unoptimized Trace for the Simple Object Model}
 \label{fig:unopt-trace}
 \end{figure}
 
-If the function is executed using the tracing JIT, with \texttt{y} being a
-\texttt{BoxedInteger}, the produced trace looks like
+If the function is executed using the tracing JIT, with \lstinline{y} being a
+\lstinline{BoxedInteger}, the produced trace looks like
 Figure~\ref{fig:unopt-trace} (lines starting with the hash ``\#'' are comments).
 
 XXX in which language is the trace written in ? still RPython ?
@@ -451,26 +480,26 @@
 correspond to the stack level of the function that contains the traced
 operation. The trace is in single-assignment form, meaning that each variable is
 assigned to exactly once. The arguments $p_0$ and $p_1$ of the loop correspond
-to the live variables \texttt{y} and \texttt{res} in the original function.
+to the live variables \lstinline{y} and \lstinline{res} in the original function.
 
-XXX explain set and get + int_add briefly
+XXX explain set and get + int\_add briefly
 
-The trace shows the inefficiencies of \texttt{f} clearly, if one
-looks at the number of \texttt{new} (corresponding to object creation),
-\texttt{set/get} (corresponding to attribute reads/writes) and
-\texttt{guard\_class} operations (corresponding to method calls).
+The trace shows the inefficiencies of \lstinline{f} clearly, if one
+looks at the number of \lstinline{new} (corresponding to object creation),
+\lstinline{set/get} (corresponding to attribute reads/writes) and
+\lstinline{guard_class} operations (corresponding to method calls).
 In the rest of the paper we will see how this trace can be optimized using
 partial evaluation.
 
-Note how the functions that are called by \texttt{f} are automatically inlined
-into the trace. The method calls are always preceded by a \texttt{guard\_class}
+Note how the functions that are called by \lstinline{f} are automatically inlined
+into the trace. The method calls are always preceded by a \lstinline{guard_class}
 operation, to check that the class of the receiver is the same as the one that
-was observed during tracing.\footnote{\texttt{guard\_class} performs a precise
+was observed during tracing.\footnote{\lstinline{guard_class} performs a precise
 class check, not checking for subclasses.} These guards make the trace specific
-to the situation where \texttt{y} is really a \texttt{BoxedInteger}, it can
-already be said to be specialized for \texttt{BoxedIntegers}. When the trace is
-turned into machine code and then executed with \texttt{BoxedFloats}, the
-first \texttt{guard\_class} instruction will fail and execution will continue
+to the situation where \lstinline{y} is really a \lstinline{BoxedInteger}, it can
+already be said to be specialized for \lstinline{BoxedIntegers}. When the trace is
+turned into machine code and then executed with \lstinline{BoxedFloats}, the
+first \lstinline{guard_class} instruction will fail and execution will continue
 using the interpreter.
 
 
@@ -499,15 +528,15 @@
 executed until one of the guards in the trace fails, and the execution is
 aborted and interpretation resumes.
 
-Some of the operations within this trace are \texttt{new} operations, which each
+Some of the operations within this trace are \lstinline{new} operations, which each
 create a new instance of some class. These instances are used for a while, e.g.
 by calling methods on them (which are inlined into the trace), reading and
 writing their fields. Some of these instances \emph{escape}, which means that
 they are stored in some globally accessible place or are passed into a
 non-inlined function via a residual call.
 
-Together with the \texttt{new} operations, the figure shows the lifetimes of the
-created objects. The objects that are created within a trace using \texttt{new}
+Together with the \lstinline{new} operations, the figure shows the lifetimes of the
+created objects. The objects that are created within a trace using \lstinline{new}
 fall into one of several categories:
 
 \begin{itemize}
@@ -550,62 +579,57 @@
 but it is only used to optimized operations within a trace. XXX mention Prolog.
 
 The partial evaluation works by walking the trace from beginning to end.
-Whenever a \texttt{new} operation is seen, the operation is removed and a static
+Whenever a \lstinline{new} operation is seen, the operation is removed and a static
 object is constructed and associated with the variable that would have stored
-the result of \texttt{new}. The static object describes the shape of the
+the result of \lstinline{new}. The static object describes the shape of the
 original object, \eg where the values that would be stored in the fields of the
 allocated object come from, as well as the type of the object. Whenever the
-optimizer sees a \texttt{set} that writes into such an object, that shape
+optimizer sees a \lstinline{set} that writes into such an object, that shape
 description is updated and the operation can be removed, which means that the
 operation was done at partial evaluation time. When the optimizer encounters a
-\texttt{get} from such an object, the result is read from the shape
+\lstinline{get} from such an object, the result is read from the shape
 description, and the operation is also removed. Equivalently, a
-\texttt{guard\_class} on a variable that has a shape description can be removed
+\lstinline{guard_class} on a variable that has a shape description can be removed
 as well, because the shape description stores the type and thus the outcome of
 the type check the guard does is statically known.
 
 In the example from last section, the following operations would produce two
 static objects, and be completely removed from the optimized trace:
 
-\texttt{
-\begin{tabular}{l}
-$p_{5}$ = new(BoxedInteger) \\
-set($p_{5}$, intval, $i_{4}$) \\
-$p_{6}$ = new(BoxedInteger) \\
-set($p_{6}$, intval, -100) \\
-\end{tabular}
-}
+\begin{lstlisting}[mathescape,xleftmargin=20pt]
+$p_{5}$ = new(BoxedInteger)
+set($p_{5}$, intval, $i_{4}$)
+$p_{6}$ = new(BoxedInteger)
+set($p_{6}$, intval, -100)
+\end{lstlisting}
+
 
 The static object associated with $p_{5}$ would know that it is a
-\texttt{BoxedInteger} whose \texttt{intval} field contains $i_{4}$; the
-one associated with $p_{6}$ would know that it is a \texttt{BoxedInteger}
-whose \texttt{intval} field contains the constant -100.
+\lstinline{BoxedInteger} whose \lstinline{intval} field contains $i_{4}$; the
+one associated with $p_{6}$ would know that it is a \lstinline{BoxedInteger}
+whose \lstinline{intval} field contains the constant -100.
 
 The following operations on $p_{5}$ and $p_{6}$ could then be
 optimized using that knowledge:
 
-\texttt{
-\begin{tabular}{l}
-guard\_class($p_{5}$, BoxedInteger) \\
-$i_{7}$ = get($p_{5}$, intval) \\
-\# inside BoxedInteger.add \\
-guard\_class($p_{6}$, BoxedInteger) \\
-\# inside BoxedInteger.add\_\_int \\
-$i_{8}$ = get($p_{6}$, intval) \\
-$i_{9}$ = int\_add($i_{7}$, $i_{8}$) \\
-\end{tabular}
-}
+\begin{lstlisting}[mathescape,xleftmargin=20pt]
+guard_class($p_{5}$, BoxedInteger)
+$i_{7}$ = get($p_{5}$, intval)
+# inside BoxedInteger.add
+guard_class($p_{6}$, BoxedInteger)
+# inside BoxedInteger.add__int
+$i_{8}$ = get($p_{6}$, intval)
+$i_{9}$ = int_add($i_{7}$, $i_{8}$)
+\end{lstlisting}
 
-The \texttt{guard\_class} operations can be removed, because the classes of $p_{5}$ and
-$p_{6}$ are known to be \texttt{BoxedInteger}. The \texttt{get} operations can be removed
+The \lstinline{guard_class} operations can be removed, because the classes of $p_{5}$ and
+$p_{6}$ are known to be \lstinline{BoxedInteger}. The \lstinline{get} operations can be removed
 and $i_{7}$ and $i_{8}$ are just replaced by $i_{4}$ and -100. Thus the only
 remaining operation in the optimized trace would be:
 
-\texttt{
-\begin{tabular}{l}
-$i_{9}$ = int\_add($i_{4}$, -100) \\
-\end{tabular}
-}
+\begin{lstlisting}[mathescape,xleftmargin=20pt]
+$i_{9}$ = int_add($i_{4}$, -100)
+\end{lstlisting}
 
 The rest of the trace is optimized similarly.
 
@@ -619,7 +643,7 @@
 necessary to put operations into the residual code that actually allocate the
 static object at runtime.
 
-This is what happens at the end of the trace in Figure~\ref{fig:unopt-trace}, when the \texttt{jump} operation
+This is what happens at the end of the trace in Figure~\ref{fig:unopt-trace}, when the \lstinline{jump} operation
 is hit. The arguments of the jump are at this point static objects. Before the
 jump is emitted, they are \emph{lifted}. This means that the optimizer produces code
 that allocates a new object of the right type and sets its fields to the field
@@ -627,22 +651,20 @@
 objects, those need to be lifted as well, recursively) This means that instead of the jump,
 the following operations are emitted:
 
-\texttt{
-\begin{tabular}{l}
-$p_{15}$ = new(BoxedInteger) \\
-set($p_{15}$, intval, $i_{14}$) \\
-$p_{10}$ = new(BoxedInteger) \\
-set($p_{10}$, intval, $i_{9}$) \\
-jump($p_{15}$, $p_{10}$) \\
-\end{tabular}
-}
+\begin{lstlisting}[mathescape,xleftmargin=20pt]
+$p_{15}$ = new(BoxedInteger)
+set($p_{15}$, intval, $i_{14}$)
+$p_{10}$ = new(BoxedInteger)
+set($p_{10}$, intval, $i_{9}$)
+jump($p_{15}$, $p_{10}$)
+\end{lstlisting}
 
 Note how the operations for creating these two instances have been moved down the
 trace. It looks like for these operations we actually didn't win much, because
 the objects are still allocated at the end. However, the optimization was still
 worthwhile even in this case, because some operations that have been performed
-on the lifted static objects have been removed (some \texttt{get} operations
-and \texttt{guard\_class} operations).
+on the lifted static objects have been removed (some \lstinline{get} operations
+and \lstinline{guard_class} operations).
 
 \begin{figure}
 \includegraphics{figures/step1.pdf}
@@ -652,7 +674,7 @@
 
 The final optimized trace of the example can be seen in Figure~\ref{fig:step1}.
 The optimized trace contains only two allocations, instead of the original five,
-and only three \texttt{guard\_class} operations, from the original seven.
+and only three \lstinline{guard_class} operations, from the original seven.
 
 \section{Formal Description of the Algorithm}
 \label{sec:formal}
@@ -699,9 +721,9 @@
 as those are the only ones that are actually optimized. Without loss of
 generality we also consider only objects with two fields in this section.
 
-Traces are lists of operations. The operations considered here are \texttt{new} (to make
-a new object), \texttt{get} (to read a field out of an object), \texttt{set} (to write a field
-into an object) and \texttt{guard\_class} (to check the type of an object). The values of all
+Traces are lists of operations. The operations considered here are \lstinline{new} (to make
+a new object), \lstinline{get} (to read a field out of an object), \lstinline{set} (to write a field
+into an object) and \lstinline{guard_class} (to check the type of an object). The values of all
 variables are locations (i.e.~pointers). Locations are mapped to objects, which
 are represented by triples of a type $T$, and two locations that represent the
 fields of the object. When a new object is created, the fields are initialized
@@ -729,19 +751,19 @@
 $E[v\mapsto l]$ denotes the environment which is just like $E$, but maps $v$ to
 $l$.
 
-The \texttt{new} operation creates a new object $(T,\mathrm{null},\mathrm{null})$ on the
+The \lstinline{new} operation creates a new object $(T,\mathrm{null},\mathrm{null})$ on the
 heap under a fresh location $l$ and adds the result variable to the environment,
 mapping it to the new location $l$.
 
-The \texttt{get} operation reads a field $F$ out of an object, and adds the result
+The \lstinline{get} operation reads a field $F$ out of an object, and adds the result
 variable to the environment, mapping it to the read location. The heap is
 unchanged.
 
-The \texttt{set} operation changes field $F$ of an object stored at the location that
+The \lstinline{set} operation changes field $F$ of an object stored at the location that
 variable $v$ maps to. The new value of the field is the location in variable
 $u$. The environment is unchanged.
 
-The \texttt{guard\_class} operation is used to check whether the object stored at the location
+The \lstinline{guard_class} operation is used to check whether the object stored at the location
 that variable $v$ maps to is of type $T$. If that is the case, then execution
 continues without changing heap and environment. Otherwise, execution is
 stopped.
@@ -817,7 +839,7 @@
 fields of objects in the static heap are also elements of $V^*$ (or null, for
 short periods of time).
 
-When the optimizer sees a \texttt{new} operation, it optimistically removes it and
+When the optimizer sees a \lstinline{new} operation, it optimistically removes it and
 assumes that the resulting object can stay static. The optimization for all
 further operations is split into two cases. One case is for when the
 involved variables are in the static heap, which means that the operation can be
@@ -825,21 +847,21 @@
 the execution semantics closely. The other case is for when not enough is known about
 the variables, and the operation has to be residualized.
 
-If the argument $v$ of a \texttt{get} operation is mapped to something in the static
-heap, the \texttt{get} can be performed at optimization time. Otherwise, the \texttt{get}
+If the argument $v$ of a \lstinline{get} operation is mapped to something in the static
+heap, the \lstinline{get} can be performed at optimization time. Otherwise, the \lstinline{get}
 operation needs to be residualized.
 
-If the first argument $v$ to a \texttt{set} operation is mapped to something in the
-static heap, then the \texttt{set} can performed at optimization time and the static heap
-updated. Otherwise the \texttt{set} operation needs to be residualized. This needs to be
+If the first argument $v$ to a \lstinline{set} operation is mapped to something in the
+static heap, then the \lstinline{set} can performed at optimization time and the static heap
+updated. Otherwise the \lstinline{set} operation needs to be residualized. This needs to be
 done carefully, because the new value for the field, from the variable $u$,
 could itself be static, in which case it needs to be lifted first.
 
-If a \texttt{guard\_class} is performed on a variable that is in the static heap, the type check
+If a \lstinline{guard_class} is performed on a variable that is in the static heap, the type check
 can be performed at optimization time, which means the operation can be removed
 if the types match. If the type check fails statically or if the object is not
-in the static heap, the \texttt{guard\_class} is residualized. This also needs to
-lift the variable on which the \texttt{guard\_class} is performed.
+in the static heap, the \lstinline{guard_class} is residualized. This also needs to
+lift the variable on which the \lstinline{guard_class} is performed.
 
 Lifting takes a variable that is potentially in the static heap and makes sure
 that it is turned into a dynamic variable. This means that operations are
@@ -849,7 +871,7 @@
 Lifting a static object needs to recursively lift its fields. Some care needs to
 be taken when lifting a static object, because the structures described by the
 static heap can be cyclic. To make sure that the same static object is not lifted
-twice, the \texttt{liftfield} operation removes it from the static heap \emph{before}
+twice, the \lstinline{liftfield} operation removes it from the static heap \emph{before}
 recursively lifting its fields.
 
 


From afa at codespeak.net  Wed Oct 13 17:19:08 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 13 Oct 2010 17:19:08 +0200 (CEST)
Subject: [pypy-svn] r77883 - in pypy/branch/fast-forward: lib_pypy
	lib_pypy/ctypes_config_cache lib_pypy/pypy_test
	pypy/module/_ssl pypy/rlib
Message-ID: <20101013151908.B535E282BFD@codespeak.net>

Author: afa
Date: Wed Oct 13 17:19:07 2010
New Revision: 77883

Removed:
   pypy/branch/fast-forward/lib_pypy/_hashlib.py
   pypy/branch/fast-forward/lib_pypy/ctypes_config_cache/hashlib.ctc.py
   pypy/branch/fast-forward/lib_pypy/pypy_test/test_hashlib.py
Modified:
   pypy/branch/fast-forward/pypy/module/_ssl/__init__.py
   pypy/branch/fast-forward/pypy/module/_ssl/interp_ssl.py
   pypy/branch/fast-forward/pypy/rlib/ropenssl.py
Log:
Remove the ctypes-based _hashlib,
and have _ssl use the new rlib.ropenssl functions


Modified: pypy/branch/fast-forward/pypy/module/_ssl/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_ssl/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_ssl/__init__.py	Wed Oct 13 17:19:07 2010
@@ -26,5 +26,5 @@
         super(Module, cls).buildloaders()
 
     def startup(self, space):
-        from pypy.module._ssl.interp_ssl import _init_ssl
-        _init_ssl()
+        from pypy.rlib.ropenssl import init_ssl
+        init_ssl()

Modified: pypy/branch/fast-forward/pypy/module/_ssl/interp_ssl.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_ssl/interp_ssl.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_ssl/interp_ssl.py	Wed Oct 13 17:19:07 2010
@@ -3,42 +3,12 @@
 from pypy.interpreter.baseobjspace import W_Root, ObjSpace, Wrappable
 from pypy.interpreter.typedef import TypeDef
 from pypy.interpreter.gateway import interp2app
-from pypy.rpython.tool import rffi_platform
-from pypy.translator.platform import platform
-from pypy.translator.tool.cbuild import ExternalCompilationInfo
 
 from pypy.rlib import rpoll
+from pypy.rlib.ropenssl import *
 
 import sys
 
-if sys.platform == 'win32' and platform.name != 'mingw32':
-    libraries = ['libeay32', 'ssleay32', 'user32', 'advapi32', 'gdi32']
-    includes = [
-        # ssl.h includes winsock.h, which will conflict with our own
-        # need of winsock2.  Remove this when separate compilation is
-        # available...
-        'winsock2.h',
-        # wincrypt.h defines X509_NAME, include it here
-        # so that openssl/ssl.h can repair this nonsense.
-        'wincrypt.h',
-        'openssl/ssl.h',
-        'openssl/err.h']
-else:
-    libraries = ['ssl', 'crypto']
-    includes = ['openssl/ssl.h', 'openssl/err.h']
-
-eci = ExternalCompilationInfo(
-    libraries = libraries,
-    includes = includes,
-    export_symbols = ['SSL_load_error_strings'],
-    )
-
-eci = rffi_platform.configure_external_library(
-    'openssl', eci,
-    [dict(prefix='openssl-',
-          include_dir='inc32', library_dir='out32'),
-     ])
-
 ## user defined constants
 X509_NAME_MAXLEN = 256
 ## # these mirror ssl.h
@@ -60,53 +30,8 @@
 SOCKET_HAS_TIMED_OUT, SOCKET_HAS_BEEN_CLOSED = 2, 3
 SOCKET_TOO_LARGE_FOR_SELECT, SOCKET_OPERATION_OK = 4, 5
 
-# WinSock does not use a bitmask in select, and uses
-# socket handles greater than FD_SETSIZE
-if sys.platform == 'win32':
-    MAX_FD_SIZE = None
-else:
-    from pypy.rlib._rsocket_rffi import FD_SETSIZE as MAX_FD_SIZE
-
 HAVE_RPOLL = True  # Even win32 has rpoll.poll
 
-class CConfig:
-    _compilation_info_ = eci
-
-    OPENSSL_VERSION_NUMBER = rffi_platform.ConstantInteger(
-        "OPENSSL_VERSION_NUMBER")
-    SSLEAY_VERSION = rffi_platform.DefinedConstantString(
-        "SSLEAY_VERSION", "SSLeay_version(SSLEAY_VERSION)")
-    SSL_FILETYPE_PEM = rffi_platform.ConstantInteger("SSL_FILETYPE_PEM")
-    SSL_OP_ALL = rffi_platform.ConstantInteger("SSL_OP_ALL")
-    SSL_VERIFY_NONE = rffi_platform.ConstantInteger("SSL_VERIFY_NONE")
-    SSL_ERROR_WANT_READ = rffi_platform.ConstantInteger(
-        "SSL_ERROR_WANT_READ")
-    SSL_ERROR_WANT_WRITE = rffi_platform.ConstantInteger(
-        "SSL_ERROR_WANT_WRITE")
-    SSL_ERROR_ZERO_RETURN = rffi_platform.ConstantInteger(
-        "SSL_ERROR_ZERO_RETURN")
-    SSL_ERROR_WANT_X509_LOOKUP = rffi_platform.ConstantInteger(
-        "SSL_ERROR_WANT_X509_LOOKUP")
-    SSL_ERROR_WANT_CONNECT = rffi_platform.ConstantInteger(
-        "SSL_ERROR_WANT_CONNECT")
-    SSL_ERROR_SYSCALL = rffi_platform.ConstantInteger("SSL_ERROR_SYSCALL")
-    SSL_ERROR_SSL = rffi_platform.ConstantInteger("SSL_ERROR_SSL")
-    SSL_CTRL_OPTIONS = rffi_platform.ConstantInteger("SSL_CTRL_OPTIONS")
-    BIO_C_SET_NBIO = rffi_platform.ConstantInteger("BIO_C_SET_NBIO")
-
-for k, v in rffi_platform.configure(CConfig).items():
-    globals()[k] = v
-
-# opaque structures
-SSL_METHOD = rffi.COpaquePtr('SSL_METHOD')
-SSL_CTX = rffi.COpaquePtr('SSL_CTX')
-SSL = rffi.COpaquePtr('SSL')
-BIO = rffi.COpaquePtr('BIO')
-X509 = rffi.COpaquePtr('X509')
-X509_NAME = rffi.COpaquePtr('X509_NAME')
-
-HAVE_OPENSSL_RAND = OPENSSL_VERSION_NUMBER >= 0x0090500f
-
 constants = {}
 constants["SSL_ERROR_ZERO_RETURN"] = PY_SSL_ERROR_ZERO_RETURN
 constants["SSL_ERROR_WANT_READ"] = PY_SSL_ERROR_WANT_READ
@@ -137,54 +62,11 @@
 constants["OPENSSL_VERSION_INFO"] = (major, minor, fix, patch, status)
 constants["OPENSSL_VERSION"] = SSLEAY_VERSION
 
-def ssl_external(name, argtypes, restype, **kw):
-    kw['compilation_info'] = eci
-    globals()['libssl_' + name] = rffi.llexternal(
-        name, argtypes, restype, **kw)
-
-ssl_external('SSL_load_error_strings', [], lltype.Void)
-ssl_external('SSL_library_init', [], rffi.INT)
-if HAVE_OPENSSL_RAND:
-    ssl_external('RAND_add', [rffi.CCHARP, rffi.INT, rffi.DOUBLE], lltype.Void)
-    ssl_external('RAND_status', [], rffi.INT)
-    ssl_external('RAND_egd', [rffi.CCHARP], rffi.INT)
-ssl_external('SSL_CTX_new', [SSL_METHOD], SSL_CTX)
-ssl_external('SSLv23_method', [], SSL_METHOD)
-ssl_external('SSL_CTX_use_PrivateKey_file', [SSL_CTX, rffi.CCHARP, rffi.INT], rffi.INT)
-ssl_external('SSL_CTX_use_certificate_chain_file', [SSL_CTX, rffi.CCHARP], rffi.INT)
-ssl_external('SSL_CTX_ctrl', [SSL_CTX, rffi.INT, rffi.INT, rffi.VOIDP], rffi.INT)
-ssl_external('SSL_CTX_set_verify', [SSL_CTX, rffi.INT, rffi.VOIDP], lltype.Void)
-ssl_external('SSL_new', [SSL_CTX], SSL)
-ssl_external('SSL_set_fd', [SSL, rffi.INT], rffi.INT)
-ssl_external('BIO_ctrl', [BIO, rffi.INT, rffi.INT, rffi.VOIDP], rffi.INT)
-ssl_external('SSL_get_rbio', [SSL], BIO)
-ssl_external('SSL_get_wbio', [SSL], BIO)
-ssl_external('SSL_set_connect_state', [SSL], lltype.Void)
-ssl_external('SSL_connect', [SSL], rffi.INT)
-ssl_external('SSL_get_error', [SSL, rffi.INT], rffi.INT)
-
-ssl_external('ERR_get_error', [], rffi.INT)
-ssl_external('ERR_error_string', [rffi.ULONG, rffi.CCHARP], rffi.CCHARP)
-ssl_external('SSL_get_peer_certificate', [SSL], X509)
-ssl_external('X509_get_subject_name', [X509], X509_NAME)
-ssl_external('X509_get_issuer_name', [X509], X509_NAME)
-ssl_external('X509_NAME_oneline', [X509_NAME, rffi.CCHARP, rffi.INT], rffi.CCHARP)
-ssl_external('X509_free', [X509], lltype.Void)
-ssl_external('SSL_free', [SSL], lltype.Void)
-ssl_external('SSL_CTX_free', [SSL_CTX], lltype.Void)
-ssl_external('SSL_write', [SSL, rffi.CCHARP, rffi.INT], rffi.INT)
-ssl_external('SSL_pending', [SSL], rffi.INT)
-ssl_external('SSL_read', [SSL, rffi.CCHARP, rffi.INT], rffi.INT)
-
 def ssl_error(space, msg):
     w_module = space.getbuiltinmodule('_ssl')
     w_exception = space.getattr(w_module, space.wrap('SSLError'))
     return OperationError(w_exception, space.wrap(msg))
 
-def _init_ssl():
-    libssl_SSL_load_error_strings()
-    libssl_SSL_library_init()
-
 if HAVE_OPENSSL_RAND:
     # helper routines for seeding the SSL PRNG
     def RAND_add(space, string, entropy):

Modified: pypy/branch/fast-forward/pypy/rlib/ropenssl.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/ropenssl.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/ropenssl.py	Wed Oct 13 17:19:07 2010
@@ -35,27 +35,6 @@
           include_dir='inc32', library_dir='out32'),
      ])
 
-## user defined constants
-X509_NAME_MAXLEN = 256
-## # these mirror ssl.h
-PY_SSL_ERROR_NONE, PY_SSL_ERROR_SSL = 0, 1
-PY_SSL_ERROR_WANT_READ, PY_SSL_ERROR_WANT_WRITE = 2, 3
-PY_SSL_ERROR_WANT_X509_LOOKUP = 4
-PY_SSL_ERROR_SYSCALL = 5 # look at error stack/return value/errno
-PY_SSL_ERROR_ZERO_RETURN, PY_SSL_ERROR_WANT_CONNECT = 6, 7
-# start of non ssl.h errorcodes
-PY_SSL_ERROR_EOF = 8 # special case of SSL_ERROR_SYSCALL
-PY_SSL_ERROR_INVALID_ERROR_CODE = 9
-
-PY_SSL_CERT_NONE, PY_SSL_CERT_OPTIONAL, PY_SSL_CERT_REQUIRED = 0, 1, 2
-
-(PY_SSL_VERSION_SSL2, PY_SSL_VERSION_SSL3,
- PY_SSL_VERSION_SSL23, PY_SSL_VERSION_TLS1) = range(4)
-
-SOCKET_IS_NONBLOCKING, SOCKET_IS_BLOCKING = 0, 1
-SOCKET_HAS_TIMED_OUT, SOCKET_HAS_BEEN_CLOSED = 2, 3
-SOCKET_TOO_LARGE_FOR_SELECT, SOCKET_OPERATION_OK = 4, 5
-
 # WinSock does not use a bitmask in select, and uses
 # socket handles greater than FD_SETSIZE
 if sys.platform == 'win32':
@@ -63,8 +42,6 @@
 else:
     from pypy.rlib._rsocket_rffi import FD_SETSIZE as MAX_FD_SIZE
 
-HAVE_RPOLL = True  # Even win32 has rpoll.poll
-
 class CConfig:
     _compilation_info_ = eci
 
@@ -103,36 +80,6 @@
 
 HAVE_OPENSSL_RAND = OPENSSL_VERSION_NUMBER >= 0x0090500f
 
-constants = {}
-constants["SSL_ERROR_ZERO_RETURN"] = PY_SSL_ERROR_ZERO_RETURN
-constants["SSL_ERROR_WANT_READ"] = PY_SSL_ERROR_WANT_READ
-constants["SSL_ERROR_WANT_WRITE"] = PY_SSL_ERROR_WANT_WRITE
-constants["SSL_ERROR_WANT_X509_LOOKUP"] = PY_SSL_ERROR_WANT_X509_LOOKUP
-constants["SSL_ERROR_SYSCALL"] = PY_SSL_ERROR_SYSCALL
-constants["SSL_ERROR_SSL"] = PY_SSL_ERROR_SSL
-constants["SSL_ERROR_WANT_CONNECT"] = PY_SSL_ERROR_WANT_CONNECT
-constants["SSL_ERROR_EOF"] = PY_SSL_ERROR_EOF
-constants["SSL_ERROR_INVALID_ERROR_CODE"] = PY_SSL_ERROR_INVALID_ERROR_CODE
-
-constants["CERT_NONE"]     = PY_SSL_CERT_NONE
-constants["CERT_OPTIONAL"] = PY_SSL_CERT_OPTIONAL
-constants["CERT_REQUIRED"] = PY_SSL_CERT_REQUIRED
-
-constants["PROTOCOL_SSLv2"]  = PY_SSL_VERSION_SSL2
-constants["PROTOCOL_SSLv3"]  = PY_SSL_VERSION_SSL3
-constants["PROTOCOL_SSLv23"] = PY_SSL_VERSION_SSL23
-constants["PROTOCOL_TLSv1"]  = PY_SSL_VERSION_TLS1
-
-constants["OPENSSL_VERSION_NUMBER"] = OPENSSL_VERSION_NUMBER
-ver = OPENSSL_VERSION_NUMBER
-ver, status = divmod(ver, 16)
-ver, patch  = divmod(ver, 256)
-ver, fix    = divmod(ver, 256)
-ver, minor  = divmod(ver, 256)
-ver, major  = divmod(ver, 256)
-constants["OPENSSL_VERSION_INFO"] = (major, minor, fix, patch, status)
-constants["OPENSSL_VERSION"] = SSLEAY_VERSION
-
 def external(name, argtypes, restype, **kw):
     kw['compilation_info'] = eci
     return rffi.llexternal(
@@ -198,7 +145,7 @@
 EVP_MD_CTX_cleanup = external(
     'EVP_MD_CTX_cleanup', [EVP_MD_CTX], rffi.INT)
 
-def _init_ssl():
+def init_ssl():
     libssl_SSL_load_error_strings()
     libssl_SSL_library_init()
 


From antocuni at codespeak.net  Wed Oct 13 17:19:30 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Wed, 13 Oct 2010 17:19:30 +0200 (CEST)
Subject: [pypy-svn] r77884 - pypy/extradoc/talk/pepm2011
Message-ID: <20101013151930.8947D282BFE@codespeak.net>

Author: antocuni
Date: Wed Oct 13 17:19:28 2010
New Revision: 77884

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
don't highlight get/set/new/guard_class in bold


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Wed Oct 13 17:19:28 2010
@@ -28,7 +28,7 @@
   commentstyle=\color{gray}\textit,
   fancyvrb=true,
   showstringspaces=false,
-  keywords={def,while,if,elif,return,class,get,set,new,guard_class}
+  %keywords={def,while,if,elif,return,class,get,set,new,guard_class}
 }
 
 


From arigo at codespeak.net  Wed Oct 13 17:22:16 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 13 Oct 2010 17:22:16 +0200 (CEST)
Subject: [pypy-svn] r77885 - pypy/trunk/pypy/module/gc
Message-ID: <20101013152216.603A9282BFF@codespeak.net>

Author: arigo
Date: Wed Oct 13 17:22:14 2010
New Revision: 77885

Modified:
   pypy/trunk/pypy/module/gc/referents.py
Log:
Yet another attempt.


Modified: pypy/trunk/pypy/module/gc/referents.py
==============================================================================
--- pypy/trunk/pypy/module/gc/referents.py	(original)
+++ pypy/trunk/pypy/module/gc/referents.py	Wed Oct 13 17:22:14 2010
@@ -15,12 +15,16 @@
 
 def try_cast_gcref_to_w_root(gcref):
     w_obj = rgc.try_cast_gcref_to_instance(W_Root, gcref)
-    # Ignore instances of exactly the class W_Root.  Seems that
-    # we get one on 64-bit translations, for some reason.
-    if w_obj is not None and type(w_obj) is W_Root:
-        w_obj = None
-    if not we_are_translated() and not hasattr(w_obj, 'typedef'):
-        w_obj = None
+    # Ignore the instances of W_Root that are not really valid as Python
+    # objects.  There is e.g. WeakrefLifeline in module/_weakref that
+    # inherits from W_Root for internal reasons.  Such instances don't
+    # have a typedef at all (or have a null typedef after translation).
+    if not we_are_translated():
+        if not hasattr(w_obj, 'typedef'):
+            return None
+    else:
+        if w_obj is None or not w_obj.typedef:
+            return None
     return w_obj
 
 def wrap(space, gcref):


From cfbolz at codespeak.net  Wed Oct 13 17:29:56 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Wed, 13 Oct 2010 17:29:56 +0200 (CEST)
Subject: [pypy-svn] r77886 - in pypy/extradoc/talk/pepm2011: . benchmarks
Message-ID: <20101013152956.0A082282C00@codespeak.net>

Author: cfbolz
Date: Wed Oct 13 17:29:53 2010
New Revision: 77886

Modified:
   pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric
   pypy/extradoc/talk/pepm2011/paper.bib
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
Try to solve the XXXs of Michael. Add a new figure with the operation numbers.


Modified: pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.bib
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.bib	(original)
+++ pypy/extradoc/talk/pepm2011/paper.bib	Wed Oct 13 17:29:53 2010
@@ -122,6 +122,14 @@
 	pages = {217--242}
 },
 
+ at phdthesis{cuni_high_2010,
+	title = {High performance implementation of Python for {CLI/.NET} with {JIT} compiler generation for dynamic languages.},
+	school = {Dipartimento di Informatica e Scienze {dell'Informazione,} University of Genova},
+	author = {Antonio Cuni},
+	year = {2010},
+	note = {Technical Report {DISI-TH-2010-05}}
+},
+
 @inproceedings{chang_tracing_2009,
 	address = {Washington, {DC,} {USA}},
 	title = {Tracing for Web 3.0: Trace Compilation for the Next Generation Web Applications},
@@ -240,7 +248,6 @@
 	pages = {18--25}
 },
 
-
 @techreport{mason_chang_efficient_2007,
 	title = {Efficient {Just-In-Time} Execution of Dynamically Typed Languages
 Via Code Specialization Using Precise Runtime Type Inference},
@@ -317,13 +324,3 @@
 	year = {2008},
 	pages = {123--139}
 }
-
-
- at phdthesis{cuni_python_cli_2010,
-	author = {Antonio Cuni},
-	title = {High performance implementation of {Python} for {CLI/.NET} with
-                  {JIT} compiler generation for dynamic languages.},
-	school = {Dipartimento di Informatica e Scienze dell'Informazione, University of Genova},
-	note = {Technical Report {DISI-TH-2010-05}},
-	year = {2010},
-},

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Wed Oct 13 17:29:53 2010
@@ -54,6 +54,7 @@
 \newcommand\revb[1]{\nb{Reviewer 2}{#1}}
 \newcommand\revc[1]{\nb{Reviewer 3}{#1}}
 \newcommand{\commentout}[1]{}
+\newcommand{\ignore}[1]{} % {{\tt \small ignore(#1)}}
 
 \newcommand\ie{i.e.,\xspace}
 \newcommand\eg{e.g.,\xspace}
@@ -234,7 +235,7 @@
 to add new \emph{backends} for producing the actual machine code, in addition
 to the original backend for the Intel \emph{x86} architecture.  Examples of
 additional JIT backends are the one for Intel \emph{x86-64} and an
-experimental one for the CLI .NET Virtual Machine \cite{cuni_python_cli_2010}.
+experimental one for the CLI .NET Virtual Machine \cite{cuni_high_2010}.
 PyPy's JIT generator generates a \emph{tracing JIT
 compiler}, a concept which we now explain in more details.
 
@@ -266,7 +267,7 @@
 tracing JIT automatically performs inlining.
 
 This trace of operations is then the basis of the generated code. The trace is
-first optimized, and then turned into machine code. Both optimizations
+first optimized, and then turned into machine code. Both optimization
 and machine code generation is simple, because the traces are linear. This
 linearity makes many optimizations a lot more tractable, and the inlining that
 happens gives the optimizations automatically more context to work with.
@@ -310,28 +311,10 @@
 The classes can be seen in
 Figure~\ref{fig:objmodel} (written in RPython).
 
-\newcommand{\ignore}[1]{} % {{\tt \small ignore(#1)}}
-\ignore{
-class Base(object):
-    def add(self, other):
-        """ add self to other """
-        raise NotImplementedError("abstract base")
-    def add__int(self, intother):
-        """ add intother to self,
-            where intother is an integer """
-        raise NotImplementedError("abstract base")
-    def add__float(self, floatother):
-        """ add floatother to self,
-            where floatother is a float """
-        raise NotImplementedError("abstract base")
-    def is_positive(self):
-        """ returns whether self is positive """
-        raise NotImplementedError("abstract base")
-        }
 \begin{figure}
 \begin{lstlisting}[mathescape]
 class Base(object):
-      ...
+   pass
 
 class BoxedInteger(Base):
    def __init__(self, intval):
@@ -375,8 +358,7 @@
       y = y.add(BoxedInteger(-1))
    return res
 \end{lstlisting}
-\caption{An ``interpreter'' for a tiny Dynamic Language written in RPython}
-%\caption{A Simple Object Model and an Example Function Using it}
+\caption{An ``Interpreter'' for a Tiny Dynamic Language Written in RPython}
 \label{fig:objmodel}
 \end{figure}
 
@@ -389,15 +371,9 @@
 which is costly due to the method dispatch.
 
 To understand the problems more directly, let us consider the simple
-interpreter function
-\lstinline{f} that uses the object model (see the bottom of
+interpreter function \lstinline{f} that uses the object model (see the bottom of
 Figure~\ref{fig:objmodel}).
 
-XXX this is not an RPython interpreter; put a reference to the previous
-paper to show how we deal with an interpreted piece of code and remove
-the interpretation overhead, turning it into basically something
-equivalent to the example here, which is the start of the present paper.
-
 The loop in \lstinline{f} iterates \lstinline{y} times, and computes something in the process.
 Simply running this function is slow, because there are lots of virtual method
 calls inside the loop, one for each \lstinline{is_positive} and even two for each
@@ -474,22 +450,24 @@
 \lstinline{BoxedInteger}, the produced trace looks like
 Figure~\ref{fig:unopt-trace} (lines starting with the hash ``\#'' are comments).
 
-XXX in which language is the trace written in ? still RPython ?
-
 The operations in the trace are shown indented to
 correspond to the stack level of the function that contains the traced
 operation. The trace is in single-assignment form, meaning that each variable is
 assigned to exactly once. The arguments $p_0$ and $p_1$ of the loop correspond
 to the live variables \lstinline{y} and \lstinline{res} in the original function.
 
-XXX explain set and get + int\_add briefly
+The operations in the trace correspond to the operations in the RPython program
+in Figure~\ref{fig:objmodel}:
 
-The trace shows the inefficiencies of \lstinline{f} clearly, if one
-looks at the number of \lstinline{new} (corresponding to object creation),
-\lstinline{set/get} (corresponding to attribute reads/writes) and
-\lstinline{guard_class} operations (corresponding to method calls).
-In the rest of the paper we will see how this trace can be optimized using
-partial evaluation.
+\begin{itemize}
+    \item \lstinline{new} corresponds to object creation.
+    \item \lstinline{get} correspond to attribute reads.
+    \item \lstinline{set} correspond to attribute writes.
+    \item \lstinline{guard\_class} correspond to method calls and are followed by
+    the trace of the called method.
+    \item \lstinline{int\_add} and \lstinline{int\_get} are integer addition and
+    comparison (``greater than''), respectively.
+\end{itemize}
 
 Note how the functions that are called by \lstinline{f} are automatically inlined
 into the trace. The method calls are always preceded by a \lstinline{guard_class}
@@ -502,6 +480,10 @@
 first \lstinline{guard_class} instruction will fail and execution will continue
 using the interpreter.
 
+The trace shows the inefficiencies of \lstinline{f} clearly, if one looks at the
+number of \lstinline{new}, \lstinline{set/get} and \lstinline{guard\_class} operations.
+In the rest of the paper we will see how this trace can be optimized using
+partial evaluation.
 
 \section{Object Lifetimes in a Tracing JIT}
 \label{sec:lifetimes}
@@ -576,7 +558,7 @@
 This is a process that is usually called \emph{escape analysis}. In this paper we will
 perform escape analysis by using partial evaluation. The partial evalution is a
 bit peculiar in that there are not actually any constant arguments to the trace,
-but it is only used to optimized operations within a trace. XXX mention Prolog.
+but it is only used to optimized operations within a trace.
 
 The partial evaluation works by walking the trace from beginning to end.
 Whenever a \lstinline{new} operation is seen, the operation is removed and a static
@@ -989,25 +971,29 @@
 allocation operations.
 
 \begin{figure*}
-\begin{tabular}{lrrrrrrrrrrrrrrrrrrrrrrr}
-	&\#loops	&new	&	&	&get	&	&	&set	&	&	&guard	&	&	&rest	&	&	&	&	&	&total	&	&\\
-crypto\_pyaes	&75	&1663	&446	&27\%	&18842	&2127	&11\%	&14552	&2020	&14\%	&8952	&235	&3\%	&53339	&7794	&15\%	&44009	&4828	&11\%	&97348	&12622	&13\%\\
-django	&53	&453	&155	&34\%	&11970	&1457	&12\%	&8738	&991	&11\%	&4093	&275	&7\%	&33988	&4076	&12\%	&25254	&2878	&11\%	&59242	&6954	&12\%\\
-fannkuch	&40	&161	&76	&47\%	&574	&306	&53\%	&278	&139	&50\%	&1121	&202	&18\%	&2581	&1685	&65\%	&2134	&723	&34\%	&4715	&2408	&51\%\\
-go	&510	&5604	&1651	&29\%	&112403	&12612	&11\%	&88439	&11915	&13\%	&51621	&3194	&6\%	&300831	&30834	&10\%	&258067	&29372	&11\%	&558898	&60206	&11\%\\
-html5lib	&416	&5939	&1793	&30\%	&241402	&15863	&7\%	&178723	&13960	&8\%	&58486	&3037	&5\%	&682654	&40341	&6\%	&484550	&34653	&7\%	&1167204	&74994	&6\%\\
-meteor-contest	&56	&221	&141	&64\%	&2326	&606	&26\%	&2047	&645	&32\%	&1040	&169	&16\%	&7018	&2250	&32\%	&5634	&1561	&28\%	&12652	&3811	&30\%\\
-nbody	&10	&84	&48	&57\%	&287	&159	&55\%	&121	&74	&61\%	&411	&90	&22\%	&986	&737	&75\%	&903	&371	&41\%	&1889	&1108	&59\%\\
-pyflate-fast	&161	&1878	&723	&38\%	&22815	&2592	&11\%	&18415	&3690	&20\%	&8291	&628	&8\%	&64649	&11327	&18\%	&51399	&7633	&15\%	&116048	&18960	&16\%\\
-raytrace-simple	&117	&1906	&359	&19\%	&52995	&2202	&4\%	&39518	&1823	&5\%	&13451	&447	&3\%	&140723	&6533	&5\%	&107870	&4831	&4\%	&248593	&11364	&5\%\\
-richards	&79	&375	&191	&51\%	&27507	&2095	&8\%	&21217	&2004	&9\%	&3972	&332	&8\%	&76751	&6171	&8\%	&53071	&4622	&9\%	&129822	&10793	&8\%\\
-spambayes	&267	&2413	&505	&21\%	&53327	&3033	&6\%	&40938	&2428	&6\%	&19343	&718	&4\%	&143575	&10322	&7\%	&116021	&6684	&6\%	&259596	&17006	&7\%\\
-spectral-norm	&35	&266	&88	&33\%	&3004	&398	&13\%	&2544	&414	&16\%	&1084	&78	&7\%	&4803	&1184	&25\%	&6898	&978	&14\%	&11701	&2162	&18\%\\
-telco	&42	&660	&70	&11\%	&20716	&424	&2\%	&14444	&608	&4\%	&6574	&56	&1\%	&53674	&1224	&2\%	&42394	&1158	&3\%	&96068	&2382	&2\%\\
-twisted-names	&201	&2706	&317	&12\%	&64438	&2795	&4\%	&47960	&1968	&4\%	&24091	&555	&2\%	&171582	&6828	&4\%	&139195	&5635	&4\%	&310777	&12463	&4\%\\
-total	&2062	&24329	&6563	&27\%	&632606	&46669	&7\%	&477934	&42679	&9\%	&202530	&10016	&5\%	&1737154	&131306	&8\%	&1337399	&105927	&8\%	&3074553	&237233	&8\%\\
+\begin{tabular}{|l||r|rr|rr|rr|rr|}
+\hline
+	&num loops	&new	&removed	&get/set	&removed &guard &removed	&all ops	&removed\\
+\hline
+crypto\_pyaes	&78	&3088	&50\%	&57148	&25\%	&9055	&95\%	&137189	&80\%\\
+django	&51	&673	&54\%	&19318	&18\%	&3876	&93\%	&55682	&85\%\\
+fannkuch	&43	&171	&49\%	&886	&63\%	&1159	&81\%	&4935	&45\%\\
+go	&517	&12234	&76\%	&200842	&21\%	&53138	&90\%	&568542	&84\%\\
+html5lib	&498	&14432	&68\%	&503390	&11\%	&71592	&94\%	&1405780	&91\%\\
+meteor-contest	&59	&277	&36\%	&4402	&31\%	&1078	&83\%	&12862	&68\%\\
+nbody	&13	&96	&38\%	&443	&69\%	&449	&78\%	&2107	&38\%\\
+pyflate-fast	&162	&2278	&55\%	&39126	&20\%	&8194	&92\%	&112857	&80\%\\
+raytrace-simple	&120	&3118	&59\%	&91982	&15\%	&13572	&95\%	&247436	&89\%\\
+richards	&87	&844	&4\%	&49875	&22\%	&4130	&91\%	&133898	&83\%\\
+spambayes	&314	&5608	&79\%	&117002	&11\%	&25313	&94\%	&324125	&90\%\\
+spectral-norm	&38	&360	&64\%	&5553	&20\%	&1122	&92\%	&11878	&77\%\\
+telco	&46	&1257	&90\%	&37470	&3\%	&6644	&99\%	&98590	&97\%\\
+twisted-names	&214	&5273	&84\%	&100010	&10\%	&23247	&96\%	&279667	&92\%\\
+\hline
+total	&2240	&49709	&70\%	&1227447	&14\%	&222569	&93\%	&3395548	&89\%\\
+\hline
 \end{tabular}
-\caption{Number of Operations Before and After Optimization}
+\caption{Number of Operations and Percentage Removed By Optimization}
 \label{fig:numops}
 \end{figure*}
 


From cfbolz at codespeak.net  Wed Oct 13 17:46:00 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Wed, 13 Oct 2010 17:46:00 +0200 (CEST)
Subject: [pypy-svn] r77887 - pypy/extradoc/talk/pepm2011
Message-ID: <20101013154600.4F9BF282BDA@codespeak.net>

Author: cfbolz
Date: Wed Oct 13 17:45:58 2010
New Revision: 77887

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
improve benchmark section


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Wed Oct 13 17:45:58 2010
@@ -963,12 +963,30 @@
 know how many allocations could be optimized away. On the other hand, we want
 to know how much the run times of the benchmarks is improved.
 
-For the former we counted the occurring operations in all generated traces
-before and after the optimization phase for all benchmarks. The results can be
-seen in Figure~\ref{fig:numops}. The optimization removes as many as XXX and as
-little as XXX percent of allocation operations in the traces of the benchmarks.
-All benchmarks taken together, the optimization removes XXX percent of
-allocation operations.
+The benchmarks were run on an otherwise idle Intel Core2 Duo P8400 processor
+with 2.26 GHz and 3072 KB of cache on a machine with 3GB RAM running Linux
+2.6.35. We compared the performance of various Python implementations on the
+benchmarks. As a baseline, we used the standard Python implementation in C,
+CPython 2.6.6\footnote{\texttt{http://python.org}}, which uses a bytecode-based
+interpreter. Furthermore we compared against Psyco 1.6
+\cite{rigo_representation-based_2004}, an extension to CPython which is a
+just-in-time compiler that produces machine code at run-time. It is not based
+on traces. Finally, we used two versions of PyPy's Python interpreter (revision
+77823 of SVN trunk\footnote{\texttt{https://codespeak.net/svn/pypy/trunk}}): one
+including the JIT but not optimizing the traces, and one using the allocation
+removal optimizations (as well as some minor other optimizations, such as
+constant folding).
+
+As the first step, we counted the occurring operations in all generated traces
+before and after the optimization phase for all benchmarks. The resulting
+numbers can be
+seen in Figure~\ref{fig:numops}. The optimization removes as many as 90\% and as
+little as 4\% percent of allocation operations in the traces of the benchmarks.
+All benchmarks taken together, the optimization removes 70\% percent of
+allocation operations. The numbers look similar for reading and writing of
+attributes. There are even more \lstinline{guard} operations that are removed,
+however there is an additional optimization that removes guards, so not all the
+removed guards are an effect of the optimization described here.
 
 \begin{figure*}
 \begin{tabular}{|l||r|rr|rr|rr|rr|}
@@ -997,24 +1015,14 @@
 \label{fig:numops}
 \end{figure*}
 
-In addition to the count of operations we also performed time measurements. The
-machine the benchmarks were run on is XXX. We compared the performance of
-various Python implementations on the benchmarks. As a baseline, we used the
-standard Python implementation in C, called
-CPython\footnote{\texttt{http://python.org}}, which uses a bytecode-based
-interpreter. Furthermore we compared against Psyco \cite{rigo_representation-based_2004}, an extension to
-CPython which is a just-in-time compiler that produces machine code at run-time.
-It is not based on traces. Finally, we used three versions of PyPy's Python interpreter:
-one without a JIT, one including the JIT but not using the allocation removal
-optimization, and one using the allocation removal optimizations.
-
+In addition to the count of operations we also performed time measurements.
 All benchmarks were run 50 times in the same process, to give the JIT time to produce machine
 code. The arithmetic mean of the times of the last 30 runs were used as the
 result. The errors were computed using a confidence interval with a 95\%
 confidence level \cite{georges_statistically_2007}. The results are reported in Figure~\ref{fig:times}.
 With the optimization turned on, PyPy's Python interpreter outperforms CPython
 in all benchmarks except spambayes (which heavily relies on regular expression
-performance). All benchmarks are improved by the allocation removal
+performance XXX). All benchmarks are improved by the allocation removal
 optimization, some as much as XXX. XXX Psyco
 
 XXX runtimes of the algorithm somehow?


From afa at codespeak.net  Wed Oct 13 19:08:08 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 13 Oct 2010 19:08:08 +0200 (CEST)
Subject: [pypy-svn] r77888 - in pypy/branch/fast-forward/pypy: config
	module/_hashlib
Message-ID: <20101013170808.6385C282BDC@codespeak.net>

Author: afa
Date: Wed Oct 13 19:08:05 2010
New Revision: 77888

Modified:
   pypy/branch/fast-forward/pypy/config/pypyoption.py
   pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py
   pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py
Log:
Various translation fixes + enable _hashlib by default


Modified: pypy/branch/fast-forward/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/fast-forward/pypy/config/pypyoption.py	(original)
+++ pypy/branch/fast-forward/pypy/config/pypyoption.py	Wed Oct 13 19:08:05 2010
@@ -28,8 +28,8 @@
 working_modules.update(dict.fromkeys(
     ["_socket", "unicodedata", "mmap", "fcntl",
       "rctime" , "select", "zipimport", "_lsprof",
-     "crypt", "signal", "_rawffi", "termios", "zlib",
-     "struct", "_md5", "_sha", "bz2", "_minimal_curses", "cStringIO",
+     "crypt", "signal", "_rawffi", "termios", "zlib", "bz2",
+     "struct", "_hashlib", "_md5", "_sha", "_minimal_curses", "cStringIO",
      "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array"]
 ))
 

Modified: pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py	Wed Oct 13 19:08:05 2010
@@ -1,5 +1,5 @@
 from pypy.interpreter.mixedmodule import MixedModule
-from pypy.module._hashlib import interp_hashlib
+from pypy.module._hashlib.interp_hashlib import algorithms
 
 
 class Module(MixedModule):
@@ -11,5 +11,5 @@
     appleveldefs = {
         }
 
-    for name in interp_hashlib.algorithms:
-        interpleveldefs[name] = getattr(interp_hashlib, 'new_' + name)
+    for name in algorithms:
+        interpleveldefs[name] = 'interp_hashlib.new_%s' % (name,)

Modified: pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py	Wed Oct 13 19:08:05 2010
@@ -1,5 +1,6 @@
 from pypy.interpreter.gateway import unwrap_spec, interp2app
 from pypy.interpreter.typedef import TypeDef, GetSetProperty
+from pypy.interpreter.error import OperationError
 from pypy.tool.sourcetools import func_renamer
 from pypy.interpreter.baseobjspace import Wrappable, W_Root, ObjSpace
 from pypy.rpython.lltypesystem import lltype, rffi
@@ -9,7 +10,7 @@
 algorithms = ('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512')
 
 class W_Hash(Wrappable):
-    def __init__(self, name):
+    def __init__(self, space, name):
         self.name = name
         self.ctx = lltype.malloc(ropenssl.EVP_MD_CTX.TO, flavor='raw')
 
@@ -21,7 +22,9 @@
 
     @unwrap_spec('self', ObjSpace)
     def descr_repr(self, space):
-        return space.wrap("<%s HASH object @ 0x%x>" % (self.name, id(self)))
+        addrstring = self.getaddrstring(space)
+        return space.wrap("<%s HASH object at 0x%s>" % (
+            self.name, addrstring))
 
     @unwrap_spec('self', ObjSpace, str)
     def update(self, space, buffer):
@@ -34,7 +37,7 @@
     @unwrap_spec('self', ObjSpace)
     def copy(self, space):
         "Return a copy of the hash object."
-        w_hash = W_Hash(self.name)
+        w_hash = W_Hash(space, self.name)
         ropenssl.EVP_MD_CTX_copy(w_hash.ctx, self.ctx)
         return w_hash
 
@@ -115,8 +118,8 @@
     )
 
 @unwrap_spec(ObjSpace, str, str)
-def new(space, method, string=''):
-    w_hash = W_Hash(method)
+def new(space, name, string=''):
+    w_hash = W_Hash(space, name)
     w_hash.update(space, string)
     return space.wrap(w_hash)
 
@@ -124,6 +127,7 @@
 for name in algorithms:
     newname = 'new_%s' % (name,)
     @func_renamer(newname)
-    def new_hash(w_string=''):
-        return _new(name, w_string)
+    @unwrap_spec(ObjSpace, str)
+    def new_hash(space, string=''):
+        return new(space, name, string)
     globals()[newname] = new_hash


From fijal at codespeak.net  Wed Oct 13 20:20:09 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Wed, 13 Oct 2010 20:20:09 +0200 (CEST)
Subject: [pypy-svn] r77891 - pypy/trunk/pypy/jit/tool
Message-ID: <20101013182009.0954C282BDA@codespeak.net>

Author: fijal
Date: Wed Oct 13 20:20:07 2010
New Revision: 77891

Modified:
   pypy/trunk/pypy/jit/tool/traceviewer.py
Log:
Disable counts for now (format has changed)


Modified: pypy/trunk/pypy/jit/tool/traceviewer.py
==============================================================================
--- pypy/trunk/pypy/jit/tool/traceviewer.py	(original)
+++ pypy/trunk/pypy/jit/tool/traceviewer.py	Wed Oct 13 20:20:07 2010
@@ -253,10 +253,10 @@
 def main(loopfile, use_threshold, view=True):
     countname = py.path.local(loopfile + '.count')
     if countname.check():
-        counts = [re.split('( 20 and use_threshold:
             counts.threshold = l[-20]


From cfbolz at codespeak.net  Wed Oct 13 22:14:31 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Wed, 13 Oct 2010 22:14:31 +0200 (CEST)
Subject: [pypy-svn] r77892 - pypy/extradoc/talk/pepm2011/benchmarks
Message-ID: <20101013201431.EA3D9282BDA@codespeak.net>

Author: cfbolz
Date: Wed Oct 13 22:14:28 2010
New Revision: 77892

Modified:
   pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric
   pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json
   pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json
Log:
hopefully final benchmark numbers


Modified: pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json
==============================================================================
--- pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json	(original)
+++ pypy/extradoc/talk/pepm2011/benchmarks/jit-noops.json	Wed Oct 13 22:14:28 2010
@@ -1 +1 @@
-{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [83.608081817599995, 81.512792825700004, 81.586477994899994, 81.442406892799994, 81.350951910000006, 81.291965007800002, 81.274972915600003, 81.285789966600007, 81.302440881699994, 81.471642970999994, 81.280892133699993, 81.281094074199999, 81.301096916199995, 81.287539005300005, 81.236584901800001, 81.283654928199994, 81.291748046899997, 81.5425839424, 81.236918926200005, 80.983955144899994, 80.997071981399998, 80.963902950299996, 80.967394113500006, 80.967602014500002, 81.2447729111, 81.042499065399994, 81.091964006400005, 81.300304889700001, 81.322211027099996, 81.287165164900003, 81.302358150499998, 81.531341075900002, 81.253934860200005, 81.308516979199993, 81.402919053999995, 81.258775949500006, 81.280184984200005, 81.299252033200005, 81.306777000400004, 81.550753116600006, 81.246633052799993, 81.279681921000005, 81.2769200802, 81.316480874999996, 81.294853925699996, 81.257145166399994, 81.467491149899999, 81.312170982400005, 81.252816200300003, 81.258814811700006], "base_times": [2.80382990837, 2.7956340313000001, 2.8010652065300001, 2.7976541519200002, 2.7950928211199999, 2.8041520118699998, 2.8043060302699998, 2.7999861240400001, 2.8013310432399998, 2.8088479042099999, 2.8007519245100001, 2.8009748458899999, 2.8043870925899999, 2.80448412895, 2.80184221268, 2.80495095253, 2.8023998737300002, 2.8056631088300001, 2.8033299446100002, 2.7996089458500002, 2.8002779483800002, 2.8084959983800002, 2.8017699718500002, 2.8082520961799999, 2.8050520420099998, 2.8028419017799999, 2.7981400489800001, 2.7999629974400002, 2.7998390197799998, 2.8000919818900001, 2.80037903786, 2.79657483101, 2.7981040477799999, 2.8026959896100001, 2.7995419502300001, 2.7954881191299998, 2.7970201969100001, 2.79839706421, 2.7952239513400001, 2.7976598739599998, 2.7945868968999998, 2.7983119487799999, 2.7953550815599999, 2.79377818108, 2.7956900596600001, 2.8036141395600001, 2.79533600807, 2.7930212020899998, 2.7950999736800002, 2.7983438968700001]}], ["django", "RawResult", {"changed_times": [0.94225811958299999, 0.68628406524700003, 0.70013499259900003, 0.68669486045799999, 0.70185899734500001, 0.68997812271100001, 0.70118880271899997, 0.68901705741899999, 0.70103502273600005, 0.68790817260699999, 0.69985699653599998, 0.68816995620700006, 0.73480010032700005, 0.68603014945999996, 0.70323491096500002, 0.68623781204199996, 0.70352911949200003, 0.68471789360000002, 0.70435285568200001, 0.68524813652000005, 0.70314097404499998, 0.68677711486799997, 0.70417714118999997, 0.68562698364300001, 0.70090913772600005, 0.68850708007799999, 0.70100784301800001, 0.68775796890300001, 0.70195102691699995, 0.68907117843599996, 0.70120120048500001, 0.68785810470599995, 0.70091485977199997, 0.68843507766699996, 0.70012712478600003, 0.68629503250099999, 0.70415806770300005, 0.68661308288599998, 0.70444512367199996, 0.68531179428099998, 0.70345211029099997, 0.68501901626600004, 0.704196929932, 0.68532204628000004, 0.70484805107100001, 0.68551492691000004, 0.70121288299600004, 0.68866515159599995, 0.70078611373900002, 0.68853020667999998], "base_times": [0.98770689964299996, 0.98978590965299995, 0.98754811286900002, 0.98878502845799998, 0.98909711837799996, 0.989701032639, 1.0125169754000001, 0.99142193794300004, 0.98840188980099997, 0.99162793159499996, 0.99071907997099995, 0.99076414108300004, 0.98944306373599999, 0.98936295509299998, 0.99107599258400003, 0.99007892608600001, 0.98786211013799996, 0.992697954178, 0.99010705947900002, 0.98694705963100005, 0.98762106895400004, 0.98965787887599999, 0.98832798004199995, 0.98824381828300001, 0.98842883110000002, 0.98763680458100001, 0.99011301994300005, 0.98688387870799998, 0.98969793319699995, 0.98943901061999995, 0.98809719085699999, 0.98793482780499997, 0.98821902275100004, 0.98756289482100001, 0.99030303955099996, 0.98743104934699999, 0.98614192009000001, 0.98893618583700005, 0.98977398872400002, 0.98548102378799995, 0.987926959991, 0.98703289031999997, 0.98901891708400003, 0.98792290687600004, 0.98637413978599997, 0.98895907402000005, 0.98991489410400002, 0.98569011688200003, 0.98801302909900002, 0.99004602432300004]}], ["fannkuch", "RawResult", {"changed_times": [0.65027499198899996, 0.57184004783599995, 0.56566786766099997, 0.56521892547599994, 0.56701707840000004, 0.57264208793600002, 0.56822896003699996, 0.56470012664799996, 0.56810212135299998, 0.56781196594200001, 0.564800024033, 0.56483197212199998, 0.56917119026200003, 0.56767296791099997, 0.56695199012800002, 0.56578302383400003, 0.56515192985499996, 0.57563495635999995, 0.56515789031999997, 0.56409192085299997, 0.56773018837, 0.56481719016999998, 0.56611680984500001, 0.56627202034000002, 0.56611609458900003, 0.56967306137100004, 0.567966938019, 0.56780695915199997, 0.57448220253000004, 0.57842707634000001, 0.56447005271899997, 0.56811785697899997, 0.56486487388600004, 0.56649589538599998, 0.56501197814899995, 0.56486201286299997, 0.56838393211399996, 0.56405186653100003, 0.56555008888199998, 0.56801414489699997, 0.56562018394500002, 0.56669306755100002, 0.56592893600500005, 0.56481790542599997, 0.56828188896200005, 0.56566095352199997, 0.56510496139499999, 0.56719899177599997, 0.56526398658800003, 0.56593322753899999], "base_times": [1.84706997871, 1.8756129741700001, 1.85868191719, 1.8472619056699999, 1.87546992302, 1.8508591651899999, 1.8696298599200001, 1.86460900307, 1.85237002373, 1.87226009369, 1.8722259998299999, 1.8783550262499999, 1.8736140727999999, 1.87361216545, 1.87663817406, 1.8724210262300001, 1.8727889061, 1.8782889842999999, 1.8723690509799999, 1.8713910579699999, 1.8771908283200001, 1.87365293503, 1.87231016159, 1.87713694572, 1.87258410454, 1.87410783768, 1.8863110542299999, 1.87264585495, 1.87151694298, 1.8771209716799999, 1.8736791610700001, 1.8717048168199999, 1.8770380020099999, 1.8727111816399999, 1.87127304077, 1.87773990631, 1.89574599266, 1.87074995041, 1.92922902107, 1.9290540218400001, 1.94486498833, 1.8772900104500001, 1.87182497978, 1.87466216087, 1.8774259090400001, 1.87289023399, 1.87402200699, 1.8763341903699999, 1.8749959468799999, 1.8724749088299999]}], ["go", "RawResult", {"changed_times": [2.3995778560600001, 2.3866791725200001, 2.3967480659499998, 2.5598640441899998, 2.2185318469999999, 2.3486278057100001, 2.3838458061200001, 2.4448201656299999, 2.19373893738, 2.1973280906700001, 2.1919651031499998, 2.5130779743199998, 2.2208979129799999, 2.2443778514899999, 2.3977251052900002, 2.1551439762100002, 2.1595408916499998, 2.23243093491, 2.17227602005, 2.1977610588099998, 2.3166739940599999, 2.2348148822799998, 2.1878528595, 2.1694250106799999, 2.1508538722999999, 2.3760650157900001, 2.1592259406999998, 2.1955871581999999, 2.1416928768200001, 2.2533910274500002, 2.1912820339199999, 2.2604429721799999, 2.2298648357399999, 2.1796870231600001, 2.20620703697, 2.16007018089, 2.1497859954799998, 2.2840979099299998, 2.2118680477099999, 2.1375119686100001, 2.15009498596, 2.1446928978000002, 2.1316239833799999, 2.13385415077, 2.1266629695899999, 2.3893008232100001, 2.1834890842400001, 2.1347019672399998, 2.21711921692, 2.1232719421400001], "base_times": [0.93096089363099999, 0.95089387893699995, 0.93110585212700003, 0.92558383941699995, 0.93034386634800004, 0.92623090744000003, 0.93461894989000005, 0.93881106376599999, 0.92857980728099998, 0.93526601791399999, 0.92670106887799997, 0.93337702751200002, 0.93183684349100004, 0.93682098388699997, 0.92921209335300003, 0.93008303642300005, 0.92924404144299999, 0.93299007415799995, 0.93573307990999999, 0.930922985077, 0.93020296096800004, 0.93374800682100001, 0.93536710739100004, 0.93496298789999999, 0.93623590469399998, 0.93143200874300003, 0.92915606498699999, 0.92980098724399995, 0.94962000846899997, 0.93416810035700004, 0.93491911888099999, 0.92967605590799995, 0.927985906601, 0.934444904327, 0.93304419517500004, 0.93216395378100003, 0.93552398681600002, 0.94005179405200001, 0.93092107772800003, 0.93850493431100002, 0.932507038116, 0.93334603309599995, 0.94100594520600001, 0.93229484558099995, 0.93091201782199995, 0.93197703361499995, 0.938956975937, 0.93856000900299996, 0.93897986412000001, 0.92620086669900004]}], ["html5lib", "RawResult", {"changed_times": [38.3545889854, 29.4800949097, 28.267448186900001, 27.7905261517, 27.673813104600001, 27.295396089600001, 27.640147924400001, 27.573884964000001, 27.469006061599998, 27.410823106799999, 27.163343191100001, 27.421272039400002, 27.498446941400001, 27.377060890199999, 27.862658977500001, 27.3771958351, 27.126015901599999, 27.002305030799999, 27.370562076599999, 27.4734969139, 27.1270251274, 27.124243021000002, 27.327117919900001, 27.165127992599999, 27.1296830177, 27.4667949677, 27.082341194200001, 27.294297933599999, 27.245062112799999, 27.360240936299999, 27.298542022700001, 27.139825105700002, 27.091902017599999, 27.068879127500001, 27.3232738972, 27.168828010599999, 27.2176198959, 27.3889100552, 27.146837949799998, 27.0338580608, 27.022614002200001, 27.345255851699999, 27.146274805099999, 27.0234370232, 27.357608795200001, 27.049774169900001, 27.077157974199999, 27.123055934900002, 27.078799009299999, 27.409001111999999], "base_times": [13.864197969399999, 14.0840089321, 14.0057208538, 14.0325820446, 14.0150129795, 14.1008889675, 14.097244978000001, 14.067807912799999, 14.0274028778, 14.012732029, 13.961874008200001, 14.1108551025, 14.0863909721, 14.0686590672, 14.004349947, 14.185241937600001, 13.9961509705, 14.1126899719, 14.1245219707, 14.086353063600001, 14.048186063799999, 14.0126638412, 13.965003013600001, 14.136936903, 14.125771045700001, 14.0789718628, 14.0491027832, 13.986720800400001, 13.978865861899999, 14.133366823199999, 14.0888400078, 14.118774890899999, 14.0262331963, 14.035698890700001, 13.984946966200001, 14.1640629768, 14.148418188100001, 14.091625928899999, 14.075366020200001, 14.0536768436, 13.9736099243, 14.1238529682, 14.118108987799999, 14.103012085, 14.082417964899999, 14.0419850349, 14.015744209299999, 13.983751058599999, 14.112045049700001, 14.1095650196]}], ["meteor-contest", "RawResult", {"changed_times": [0.54956293106099996, 0.44127202034000002, 0.43488597869899998, 0.43912196159400002, 0.43368196487400001, 0.43549299240099998, 0.46642994880700001, 0.43648004531899998, 0.43979382514999998, 0.43357920646699999, 0.43448400497400003, 0.43302893638599999, 0.44884395599400001, 0.43243002891499999, 0.43497920036299997, 0.43333315849300003, 0.43120193481399999, 0.44183921813999999, 0.43201804161099999, 0.43070816993700001, 0.43093800544700001, 0.43396782875099998, 0.431432962418, 0.43151283264200002, 0.432732105255, 0.43281292915300001, 0.43130493164099998, 0.43159103393600001, 0.442057132721, 0.43084406852700002, 0.44665694236800002, 0.43070578575099999, 0.43876791000400001, 0.42976713180499998, 0.43031406402599998, 0.43287897110000001, 0.43102192878700002, 0.43873286247299997, 0.42987489700300002, 0.43340301513700003, 0.42970800399800002, 0.43067002296399998, 0.43263816833500002, 0.43107914924599999, 0.42996597289999999, 0.429845809937, 0.433667898178, 0.43897414207500002, 0.43040704727200002, 0.43301296234100001], "base_times": [0.34740400314300002, 0.34649682045000002, 0.34654498100300002, 0.34828615188599998, 0.34688496589700002, 0.34619617462199997, 0.348422050476, 0.35294890403700002, 0.346195220947, 0.34620308876, 0.34614801406899998, 0.34765100479099997, 0.34694099426300001, 0.34586310386699998, 0.34622812271100001, 0.34607410430899999, 0.348037004471, 0.34679603576700002, 0.34619593620299999, 0.34622883796699999, 0.34815096855200001, 0.34621715545699999, 0.347053050995, 0.34751987457299999, 0.34610795974699998, 0.34874510765099997, 0.34702110290499999, 0.35155105590800001, 0.34627103805499998, 0.34825491905200001, 0.34611105918899998, 0.34621214866599997, 0.34627509117100003, 0.34635615348799997, 0.34847092628499998, 0.34634494781500003, 0.34610104560900001, 0.34599280357399997, 0.34852099418600002, 0.34607791900599999, 0.34652614593499997, 0.34637212753300001, 0.348030090332, 0.346480131149, 0.34624314308199999, 0.346033096313, 0.346461057663, 0.348530054092, 0.34677815437300002, 0.34670209884600001]}], ["nbody_modified", "RawResult", {"changed_times": [0.178552150726, 0.14297604560900001, 0.135512828827, 0.13514518737799999, 0.13505196571399999, 0.13496088981599999, 0.135854959488, 0.13503122329700001, 0.13507986068700001, 0.13504290580700001, 0.13506603241000001, 0.13503408432, 0.13780403137200001, 0.13503003120400001, 0.13493585586500001, 0.13506293296800001, 0.135177135468, 0.13551998138400001, 0.134822845459, 0.13526797294599999, 0.134998083115, 0.13494396209699999, 0.13502788543700001, 0.13709807395900001, 0.135847091675, 0.135095119476, 0.13512802124000001, 0.13511490821800001, 0.13506793975799999, 0.135057926178, 0.135121107101, 0.135377883911, 0.13559198379500001, 0.13520693779000001, 0.13507509231600001, 0.138010978699, 0.13512301445, 0.13525795936599999, 0.13526701927199999, 0.135272979736, 0.13510107994100001, 0.13522982597399999, 0.13502717018099999, 0.13525414466899999, 0.135074853897, 0.13518810272199999, 0.13801503181499999, 0.13605713844299999, 0.134965896606, 0.13791918754599999], "base_times": [0.60831594467200001, 0.60184097290000005, 0.60403108596800004, 0.604898929596, 0.59930682182299999, 0.59702301025399995, 0.60113716125500005, 0.59830880165099998, 0.60461401939399995, 0.59961390495300004, 0.60151195526099999, 0.60288000106799999, 0.60938596725500005, 0.60743498802200002, 0.600023984909, 0.60061693191499999, 0.60342597961400002, 0.59932994842499998, 0.60085797309900002, 0.60232901573200004, 0.59961009025599998, 0.65427803993199996, 0.60101699829099997, 0.60087990760800003, 0.60391497612, 0.60592889785799997, 0.60827589034999996, 0.60652494430500004, 0.60713696479799995, 0.61633515358000002, 0.60946607589699997, 0.61303901672399996, 0.61002016067499998, 0.61310410499599999, 0.61289215087899995, 0.61635208129899999, 0.62035799026500005, 0.61284279823300003, 0.61309885978699996, 0.61907792091400005, 0.61475586891199996, 0.61362600326500005, 0.61063408851599998, 0.61360001564, 0.61051511764499999, 0.61368680000300002, 0.60794496536300002, 0.61147809028599998, 0.61695909500099999, 0.61682605743399999]}], ["pyflate-fast", "RawResult", {"changed_times": [3.3121449947400001, 3.32391285896, 3.3113570213300001, 3.3125739097600002, 3.3533909320799999, 3.27320981026, 3.3294079303699999, 3.3954648971600001, 3.36747694016, 3.3318870067600002, 3.3314881324800001, 3.29482102394, 3.2987878322599999, 3.3235919475600002, 3.2715950012200001, 3.2692580223099998, 3.3049309253699999, 3.2853808403000002, 3.2795870304100001, 3.2983829975100001, 3.25447106361, 3.3078680038499999, 3.2952280044600002, 3.27434897423, 3.2544469833399998, 3.3341419696800001, 3.2960839271500002, 3.2732009887700002, 3.2589120864900001, 3.3169569969200001, 3.2971358299300002, 3.2613229751600001, 3.3093330860100001, 3.25996899605, 3.3377301692999999, 3.2606930732700001, 3.2845060825300001, 3.2952270507799999, 3.2635328769699998, 3.2907948494000001, 3.2753908634200002, 3.29530191422, 3.2842710018200001, 3.2528989314999999, 3.2978489399000002, 3.3096721172299999, 3.2890150547000001, 3.2512168884300001, 3.3030450344100002, 3.2920761108400001], "base_times": [3.18242001534, 3.1863389015200001, 3.1912939548499999, 3.1865100860600002, 3.1824879646299999, 3.19247794151, 3.1979899406399999, 3.1937181949600002, 3.1884422302200002, 3.20042586327, 3.18729782104, 3.1916460990900002, 3.1865410804700001, 3.1863729953800002, 3.1931619644200002, 3.1834938526199998, 3.2051031589500001, 3.1961328983300001, 3.19024610519, 3.1944649219499999, 3.19651508331, 3.2056679725600001, 3.1969439983400001, 3.1953899860399999, 3.1878740787500002, 3.1908388137800001, 3.1916890144300001, 3.1885077953300001, 3.1937019824999999, 3.1973328590399999, 3.1935818195299999, 3.1918790340399998, 3.18938112259, 3.1975688934300002, 3.1952290534999999, 3.1982200145699999, 3.18593597412, 3.1900849342300002, 3.20125794411, 3.1916630268100001, 3.2036459445999999, 3.20056295395, 3.1956481933599998, 3.1894419193300001, 3.2185740470900002, 3.1986999511700001, 3.1960220336899998, 3.1938240528100001, 3.1854040622699999, 3.1919748783099999]}], ["raytrace-simple", "RawResult", {"changed_times": [2.7798359394099998, 2.7791428566, 2.7640888690900001, 2.76987791061, 2.8119649887099998, 2.7948009967799998, 2.8166298866299999, 2.7788488864900001, 2.8109869957, 2.7679738998399999, 2.8182439804100001, 2.8162288665799999, 2.8095500469200001, 2.7651660442399999, 2.76630306244, 2.81266593933, 2.80415391922, 2.76372790337, 2.7615659236900001, 2.7600309848800002, 2.9178488254500001, 2.76185297966, 2.7615509033199999, 2.7574489116700001, 2.7583611011500002, 2.7616539001499998, 2.8071138858800002, 2.7628870010400002, 2.7632458209999999, 2.9184291362799999, 2.7783088684099999, 2.7595660686499999, 2.7599799633000002, 2.7592060565900001, 2.7621810436200001, 2.8037729263300002, 2.7557120323199999, 2.75506186485, 2.7569091320000001, 2.7592680454299998, 2.7995710372899998, 2.7594900131200002, 2.7773869037600001, 2.7546079158799999, 2.7564780712100001, 2.8319108486200002, 2.75832509995, 2.7571489810899998, 2.7593541145299998, 2.7733960151699999], "base_times": [2.66753315926, 2.6737060546899998, 2.67894601822, 2.6753239631699999, 2.6803069114700002, 2.6736009120899999, 2.6738917827600002, 2.67639303207, 2.6885011196100002, 2.6775891780899999, 2.67288613319, 2.6740000248000002, 2.6716890335099999, 2.6766500472999999, 2.6777601242100002, 2.6722629070299999, 2.67888998985, 2.6715672016099998, 2.6804389953599999, 2.6803979873700001, 2.6752359867100002, 2.67298698425, 2.67721605301, 2.6766111850700001, 2.67558693886, 2.6749620437599999, 2.67335510254, 2.67534399033, 2.6777729988100001, 2.67326784134, 2.67556595802, 2.6889100074800001, 2.6720960140200001, 2.6758561134300001, 2.6755039691900002, 2.6716270446800001, 2.6724560260799999, 2.6727318763699999, 2.6774148941, 2.6764769554100001, 2.6750671863600002, 2.6714379787400002, 2.7435669899000001, 2.7485840320600001, 2.70751786232, 2.7092530727400002, 2.67544198036, 2.6766531467400001, 2.6786830425299999, 2.67903494835]}], ["richards", "RawResult", {"changed_times": [0.42485690116899999, 0.391672134399, 0.38697290420500002, 0.38978099823000001, 0.38653588295000002, 0.38539409637499999, 0.38544106483500001, 0.38723397254899999, 0.38484597206100002, 0.38460087776200003, 0.38481712341300001, 0.387125015259, 0.38511180877700002, 0.38335013389599998, 0.38253307342499998, 0.38541388511699998, 0.40138697624199998, 0.41167879104600003, 0.385834932327, 0.4046189785, 0.38262510299699998, 0.39745593071000002, 0.38132977485699998, 0.38368082046500002, 0.380796909332, 0.37990593910199999, 0.38024783134500001, 0.38323211669899998, 0.379715919495, 0.38007807731600002, 0.37966990470899997, 0.40578198432899998, 0.38037204742399999, 0.38033199310299998, 0.38059902191200001, 0.382125139236, 0.38005089759799998, 0.38041114807100002, 0.37975502014200002, 0.41739606857299999, 0.37962198257399998, 0.37924098968499997, 0.40164279937699998, 0.38241791725199997, 0.37951016426099998, 0.37923693656899998, 0.37933707237199998, 0.38282299041700002, 0.37891888618500003, 0.37944197654700001], "base_times": [0.34984207153300001, 0.35352921485900002, 0.35030293464700002, 0.35039401054399999, 0.35356593132000003, 0.35617804527300001, 0.35045003890999998, 0.35419392585800002, 0.34917712211599999, 0.35496902465800001, 0.34898185729999998, 0.35016894340499999, 0.34672594070399998, 0.35246014595000003, 0.34967398643499997, 0.35236191749599999, 0.34780907630899999, 0.35169386863699997, 0.35305500030499998, 0.35085582733199999, 0.35157895088199997, 0.348843097687, 0.355578899384, 0.35684394836400002, 0.35460305213900001, 0.34866809845000002, 0.34962701797500001, 0.35786795616099998, 0.35232281684900002, 0.34995484352099998, 0.357528924942, 0.35109806060799997, 0.353901147842, 0.34876418113699997, 0.34823799133299999, 0.34872198104899998, 0.35193896293600002, 0.34771203994799998, 0.34916996955899998, 0.34796500205999997, 0.34642791748000001, 0.35339903831500002, 0.34859085083000002, 0.35140800476099998, 0.34648203849800002, 0.355483055115, 0.353637933731, 0.34880208969100002, 0.37337613105799999, 0.35136389732399997]}], ["spambayes", "RawResult", {"changed_times": [0.59816980361899996, 0.78452110290499999, 0.83023285865800001, 0.93411684036300002, 0.69184017181400004, 1.10234284401, 0.74256396293600002, 0.71328186988800002, 0.57437109947200005, 0.86799788474999995, 0.67963004112199998, 0.58452200889600003, 0.58005404472400002, 0.52808690071099995, 1.1698250770600001, 0.69769787788399995, 0.64825582504299994, 0.694402217865, 0.63659095764200002, 0.58709597587600004, 0.52931094169600001, 0.65094089508099995, 0.54797601699800003, 0.52007412910499995, 0.61390900611900001, 0.545217990875, 0.66334605216999998, 0.54549193382299999, 0.67743301391599997, 0.51848697662400001, 0.71530890464800001, 0.76569509506199995, 0.52243494987500005, 0.51943516731299999, 0.61838197708099996, 0.60029411315900005, 0.64244699478099998, 0.53688502311700004, 0.51747083663899995, 0.51864504814099999, 0.64398002624499995, 0.516402006149, 0.54355907440200002, 0.60826396942100003, 0.51326608657799999, 0.509041070938, 0.61415004730199996, 0.592803001404, 0.50892901420600001, 0.60751390457200005], "base_times": [0.29755282402, 0.29553508758500002, 0.29539799690200003, 0.29539704322799998, 0.29553294181799999, 0.29756402969399998, 0.295476913452, 0.29511308670000003, 0.29523301124599999, 0.294858932495, 0.29885101318399998, 0.29543805122400002, 0.29549407959000001, 0.29555702209500001, 0.295747041702, 0.29760813713099998, 0.29595708847000002, 0.29637312889099998, 0.29531884193399999, 0.295172929764, 0.29555201530500003, 0.29735279083299998, 0.29529094696000002, 0.29522705078099998, 0.29555082321199999, 0.29535007476800001, 0.29763197898900001, 0.29533004760699999, 0.29603099823000001, 0.29527783393899998, 0.29541301727300001, 0.29754710197399997, 0.29553294181799999, 0.295480012894, 0.295413970947, 0.29505491256700001, 0.29747200012199998, 0.29603290557900003, 0.29523515701300002, 0.295083999634, 0.29551792144799999, 0.29732418060299998, 0.29565906524699997, 0.29551196098299998, 0.29589390754700001, 0.295162916183, 0.29713106155399999, 0.29647493362400001, 0.295417070389, 0.29561495780899999]}], ["spectral-norm", "RawResult", {"changed_times": [0.46351408958399998, 0.36286616325400001, 0.36123991012599999, 0.35890722274800002, 0.35978198051499999, 0.35527896881100002, 0.35585403442399999, 0.35739207267799999, 0.35975408554100002, 0.35413813590999998, 0.35580110549900001, 0.35268497466999998, 0.364890098572, 0.35489296913099999, 0.35201191902200002, 0.354170084, 0.35294985771199999, 0.35513281822199999, 0.35184288024900001, 0.35136795043899999, 0.35138392448400002, 0.35504508018499997, 0.35184502601599998, 0.35189914703399999, 0.35134291648900001, 0.35165905952499998, 0.35476517677300001, 0.352794885635, 0.35161089897199999, 0.35130691528300001, 0.355042934418, 0.35176992416399999, 0.35218715667700001, 0.353415966034, 0.35497093200699997, 0.35284900665300001, 0.351359128952, 0.35298585891700002, 0.35289096832299999, 0.35544204711900002, 0.35241413116499998, 0.35347700118999997, 0.35199999809299998, 0.35367488861099999, 0.35371303558299999, 0.35233402252200002, 0.35166788101200003, 0.35387802124000001, 0.37293601036099999, 0.35278296470600001], "base_times": [0.50352501869199995, 0.47929906845100001, 0.48261594772299998, 0.47830605506899998, 0.48235297203100003, 0.48249197006200001, 0.48067307472199999, 0.48097801208500002, 0.47920393943799999, 0.47869086265600003, 0.48005580902099998, 0.47897410392799999, 0.48137307167100002, 0.483394145966, 0.47975897788999999, 0.47675085067700002, 0.48147702217100002, 0.47802019119299999, 0.482943058014, 0.47830295562699998, 0.48056387901300002, 0.47623491287199998, 0.47782897949199998, 0.47907996177700002, 0.48068904876700003, 0.47843909263599999, 0.48215293884299998, 0.478461027145, 0.48303794860799998, 0.47899913787800003, 0.478845119476, 0.480139970779, 0.48173093795799998, 0.47723102569600001, 0.47754788398699999, 0.47700595855700001, 0.48208713531500003, 0.47760915756200001, 0.480431079865, 0.48341894149800002, 0.47852706909199999, 0.47913908958399998, 0.48555207252499999, 0.490929841995, 0.47841882705700001, 0.48059010505700001, 0.47776508331299999, 0.47688412666300001, 0.48040413856500003, 0.47939014434799998]}], ["telco", "RawResult", {"changed_times": [1.452091, 1.324082, 1.3400840000000001, 1.4160889999999999, 1.3320829999999999, 1.3800859999999999, 1.280081, 1.304081, 1.304082, 1.300081, 1.308082, 1.2840800000000001, 1.308082, 1.296081, 1.3240829999999999, 1.308081, 1.284081, 1.324082, 1.288081, 1.300081, 1.284081, 1.328082, 1.3200829999999999, 1.288081, 1.300081, 1.2880799999999999, 1.288081, 1.308081, 1.288081, 1.308082, 1.284081, 1.304081, 1.2800800000000001, 1.304081, 1.288081, 1.304081, 1.288081, 1.300081, 1.288081, 1.2920799999999999, 1.312082, 1.284081, 1.308082, 1.2800800000000001, 1.304081, 1.288081, 1.2960799999999999, 1.288081, 1.304081, 1.284081], "base_times": [1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.2, 1.22, 1.2, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.2, 1.2, 1.21, 1.21, 1.21]}], ["twisted_names", "RawResult", {"changed_times": [0.022883295194508008, 0.020242914979757085, 0.024330900243309004, 0.02004008016032064, 0.020080321285140562, 0.018832391713747645, 0.017793594306049824, 0.018484288354898338, 0.017953321364452424, 0.017825311942959002, 0.017793594306049824, 0.019801980198019802, 0.017825311942959002, 0.017667844522968199, 0.017761989342806393, 0.017761989342806393, 0.018691588785046728, 0.018214936247723135, 0.017636684303350969, 0.021231422505307854, 0.017699115044247787, 0.018518518518518517, 0.017793594306049824, 0.017699115044247787, 0.018050541516245487, 0.017730496453900711, 0.01937984496124031, 0.01824817518248175, 0.017761989342806393, 0.018181818181818181, 0.017574692442882251, 0.017953321364452424, 0.017574692442882251, 0.017482517482517484, 0.017574692442882251, 0.017574692442882251, 0.017513134851138354, 0.020161290322580645, 0.017543859649122806, 0.017482517482517484, 0.018796992481203006, 0.017482517482517484, 0.017421602787456445, 0.017513134851138354, 0.017574692442882251, 0.017513134851138354, 0.017513134851138354, 0.020161290322580645, 0.017574692442882251, 0.018761726078799251], "base_times": [0.0095693779904306216, 0.0095147478591817315, 0.0095877277085330784, 0.0095602294455066923, 0.0095877277085330784, 0.0096339113680154135, 0.0095693779904306216, 0.0095602294455066923, 0.0095602294455066923, 0.0095785440613026813, 0.0095510983763132766, 0.0095785440613026813, 0.0095969289827255271, 0.0095969289827255271, 0.0096246390760346481, 0.0095602294455066923, 0.0095969289827255271, 0.0095785440613026813, 0.0095785440613026813, 0.0095602294455066923, 0.0096153846153846159, 0.0095602294455066923, 0.0096618357487922701, 0.0095693779904306216, 0.009727626459143969, 0.0095693779904306216, 0.0095602294455066923, 0.0095419847328244278, 0.0095693779904306216, 0.0095785440613026813, 0.0095693779904306216, 0.0095510983763132766, 0.0095969289827255271, 0.0095602294455066923, 0.0095785440613026813, 0.0095877277085330784, 0.0095693779904306216, 0.0095419847328244278, 0.0095785440613026813, 0.0095238095238095247, 0.0095693779904306216, 0.0095693779904306216, 0.0095877277085330784, 0.0095693779904306216, 0.0095877277085330784, 0.0095969289827255271, 0.0095785440613026813, 0.0095693779904306216, 0.0095602294455066923, 0.0095785440613026813]}]], "branch": "trunk", "revision": 0}
\ No newline at end of file
+{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [2.6510879993400001, 1.9344520568800001, 1.73843193054, 1.7308630943300001, 1.7928988933600001, 1.68379402161, 1.6968870163000001, 1.75850701332, 1.7397270202599999, 1.69991493225, 1.6527018547100001, 1.6478400230400001, 1.6623890399900001, 1.6820631027199999, 1.6487820148500001, 1.65380096436, 1.65068292618, 1.64833211899, 1.66016221046, 1.6446061134300001, 1.64752292633, 1.68245911598, 1.6478810310400001, 1.6496109962500001, 1.6509699821499999, 1.6476500034299999, 1.6489810943600001, 1.6475958824200001, 1.64714002609, 1.6492788791699999, 1.6498639583600001, 1.68475699425, 1.6480910778, 1.64762187004, 1.6511659622199999, 1.64651894569, 1.64819908142, 1.64752602577, 1.64961481094, 1.64846897125, 1.64679908752, 1.68699193001, 1.6472530365, 1.6470811367, 1.6470909118699999, 1.6486299037900001, 1.6494009494799999, 1.64831399918, 1.64873719215, 1.6486461162599999], "base_times": [2.80382990837, 2.7956340313000001, 2.8010652065300001, 2.7976541519200002, 2.7950928211199999, 2.8041520118699998, 2.8043060302699998, 2.7999861240400001, 2.8013310432399998, 2.8088479042099999, 2.8007519245100001, 2.8009748458899999, 2.8043870925899999, 2.80448412895, 2.80184221268, 2.80495095253, 2.8023998737300002, 2.8056631088300001, 2.8033299446100002, 2.7996089458500002, 2.8002779483800002, 2.8084959983800002, 2.8017699718500002, 2.8082520961799999, 2.8050520420099998, 2.8028419017799999, 2.7981400489800001, 2.7999629974400002, 2.7998390197799998, 2.8000919818900001, 2.80037903786, 2.79657483101, 2.7981040477799999, 2.8026959896100001, 2.7995419502300001, 2.7954881191299998, 2.7970201969100001, 2.79839706421, 2.7952239513400001, 2.7976598739599998, 2.7945868968999998, 2.7983119487799999, 2.7953550815599999, 2.79377818108, 2.7956900596600001, 2.8036141395600001, 2.79533600807, 2.7930212020899998, 2.7950999736800002, 2.7983438968700001]}], ["django", "RawResult", {"changed_times": [0.94225811958299999, 0.68628406524700003, 0.70013499259900003, 0.68669486045799999, 0.70185899734500001, 0.68997812271100001, 0.70118880271899997, 0.68901705741899999, 0.70103502273600005, 0.68790817260699999, 0.69985699653599998, 0.68816995620700006, 0.73480010032700005, 0.68603014945999996, 0.70323491096500002, 0.68623781204199996, 0.70352911949200003, 0.68471789360000002, 0.70435285568200001, 0.68524813652000005, 0.70314097404499998, 0.68677711486799997, 0.70417714118999997, 0.68562698364300001, 0.70090913772600005, 0.68850708007799999, 0.70100784301800001, 0.68775796890300001, 0.70195102691699995, 0.68907117843599996, 0.70120120048500001, 0.68785810470599995, 0.70091485977199997, 0.68843507766699996, 0.70012712478600003, 0.68629503250099999, 0.70415806770300005, 0.68661308288599998, 0.70444512367199996, 0.68531179428099998, 0.70345211029099997, 0.68501901626600004, 0.704196929932, 0.68532204628000004, 0.70484805107100001, 0.68551492691000004, 0.70121288299600004, 0.68866515159599995, 0.70078611373900002, 0.68853020667999998], "base_times": [0.98770689964299996, 0.98978590965299995, 0.98754811286900002, 0.98878502845799998, 0.98909711837799996, 0.989701032639, 1.0125169754000001, 0.99142193794300004, 0.98840188980099997, 0.99162793159499996, 0.99071907997099995, 0.99076414108300004, 0.98944306373599999, 0.98936295509299998, 0.99107599258400003, 0.99007892608600001, 0.98786211013799996, 0.992697954178, 0.99010705947900002, 0.98694705963100005, 0.98762106895400004, 0.98965787887599999, 0.98832798004199995, 0.98824381828300001, 0.98842883110000002, 0.98763680458100001, 0.99011301994300005, 0.98688387870799998, 0.98969793319699995, 0.98943901061999995, 0.98809719085699999, 0.98793482780499997, 0.98821902275100004, 0.98756289482100001, 0.99030303955099996, 0.98743104934699999, 0.98614192009000001, 0.98893618583700005, 0.98977398872400002, 0.98548102378799995, 0.987926959991, 0.98703289031999997, 0.98901891708400003, 0.98792290687600004, 0.98637413978599997, 0.98895907402000005, 0.98991489410400002, 0.98569011688200003, 0.98801302909900002, 0.99004602432300004]}], ["fannkuch", "RawResult", {"changed_times": [0.65027499198899996, 0.57184004783599995, 0.56566786766099997, 0.56521892547599994, 0.56701707840000004, 0.57264208793600002, 0.56822896003699996, 0.56470012664799996, 0.56810212135299998, 0.56781196594200001, 0.564800024033, 0.56483197212199998, 0.56917119026200003, 0.56767296791099997, 0.56695199012800002, 0.56578302383400003, 0.56515192985499996, 0.57563495635999995, 0.56515789031999997, 0.56409192085299997, 0.56773018837, 0.56481719016999998, 0.56611680984500001, 0.56627202034000002, 0.56611609458900003, 0.56967306137100004, 0.567966938019, 0.56780695915199997, 0.57448220253000004, 0.57842707634000001, 0.56447005271899997, 0.56811785697899997, 0.56486487388600004, 0.56649589538599998, 0.56501197814899995, 0.56486201286299997, 0.56838393211399996, 0.56405186653100003, 0.56555008888199998, 0.56801414489699997, 0.56562018394500002, 0.56669306755100002, 0.56592893600500005, 0.56481790542599997, 0.56828188896200005, 0.56566095352199997, 0.56510496139499999, 0.56719899177599997, 0.56526398658800003, 0.56593322753899999], "base_times": [1.84706997871, 1.8756129741700001, 1.85868191719, 1.8472619056699999, 1.87546992302, 1.8508591651899999, 1.8696298599200001, 1.86460900307, 1.85237002373, 1.87226009369, 1.8722259998299999, 1.8783550262499999, 1.8736140727999999, 1.87361216545, 1.87663817406, 1.8724210262300001, 1.8727889061, 1.8782889842999999, 1.8723690509799999, 1.8713910579699999, 1.8771908283200001, 1.87365293503, 1.87231016159, 1.87713694572, 1.87258410454, 1.87410783768, 1.8863110542299999, 1.87264585495, 1.87151694298, 1.8771209716799999, 1.8736791610700001, 1.8717048168199999, 1.8770380020099999, 1.8727111816399999, 1.87127304077, 1.87773990631, 1.89574599266, 1.87074995041, 1.92922902107, 1.9290540218400001, 1.94486498833, 1.8772900104500001, 1.87182497978, 1.87466216087, 1.8774259090400001, 1.87289023399, 1.87402200699, 1.8763341903699999, 1.8749959468799999, 1.8724749088299999]}], ["go", "RawResult", {"changed_times": [2.3995778560600001, 2.3866791725200001, 2.3967480659499998, 2.5598640441899998, 2.2185318469999999, 2.3486278057100001, 2.3838458061200001, 2.4448201656299999, 2.19373893738, 2.1973280906700001, 2.1919651031499998, 2.5130779743199998, 2.2208979129799999, 2.2443778514899999, 2.3977251052900002, 2.1551439762100002, 2.1595408916499998, 2.23243093491, 2.17227602005, 2.1977610588099998, 2.3166739940599999, 2.2348148822799998, 2.1878528595, 2.1694250106799999, 2.1508538722999999, 2.3760650157900001, 2.1592259406999998, 2.1955871581999999, 2.1416928768200001, 2.2533910274500002, 2.1912820339199999, 2.2604429721799999, 2.2298648357399999, 2.1796870231600001, 2.20620703697, 2.16007018089, 2.1497859954799998, 2.2840979099299998, 2.2118680477099999, 2.1375119686100001, 2.15009498596, 2.1446928978000002, 2.1316239833799999, 2.13385415077, 2.1266629695899999, 2.3893008232100001, 2.1834890842400001, 2.1347019672399998, 2.21711921692, 2.1232719421400001], "base_times": [0.93096089363099999, 0.95089387893699995, 0.93110585212700003, 0.92558383941699995, 0.93034386634800004, 0.92623090744000003, 0.93461894989000005, 0.93881106376599999, 0.92857980728099998, 0.93526601791399999, 0.92670106887799997, 0.93337702751200002, 0.93183684349100004, 0.93682098388699997, 0.92921209335300003, 0.93008303642300005, 0.92924404144299999, 0.93299007415799995, 0.93573307990999999, 0.930922985077, 0.93020296096800004, 0.93374800682100001, 0.93536710739100004, 0.93496298789999999, 0.93623590469399998, 0.93143200874300003, 0.92915606498699999, 0.92980098724399995, 0.94962000846899997, 0.93416810035700004, 0.93491911888099999, 0.92967605590799995, 0.927985906601, 0.934444904327, 0.93304419517500004, 0.93216395378100003, 0.93552398681600002, 0.94005179405200001, 0.93092107772800003, 0.93850493431100002, 0.932507038116, 0.93334603309599995, 0.94100594520600001, 0.93229484558099995, 0.93091201782199995, 0.93197703361499995, 0.938956975937, 0.93856000900299996, 0.93897986412000001, 0.92620086669900004]}], ["html5lib", "RawResult", {"changed_times": [38.3545889854, 29.4800949097, 28.267448186900001, 27.7905261517, 27.673813104600001, 27.295396089600001, 27.640147924400001, 27.573884964000001, 27.469006061599998, 27.410823106799999, 27.163343191100001, 27.421272039400002, 27.498446941400001, 27.377060890199999, 27.862658977500001, 27.3771958351, 27.126015901599999, 27.002305030799999, 27.370562076599999, 27.4734969139, 27.1270251274, 27.124243021000002, 27.327117919900001, 27.165127992599999, 27.1296830177, 27.4667949677, 27.082341194200001, 27.294297933599999, 27.245062112799999, 27.360240936299999, 27.298542022700001, 27.139825105700002, 27.091902017599999, 27.068879127500001, 27.3232738972, 27.168828010599999, 27.2176198959, 27.3889100552, 27.146837949799998, 27.0338580608, 27.022614002200001, 27.345255851699999, 27.146274805099999, 27.0234370232, 27.357608795200001, 27.049774169900001, 27.077157974199999, 27.123055934900002, 27.078799009299999, 27.409001111999999], "base_times": [13.864197969399999, 14.0840089321, 14.0057208538, 14.0325820446, 14.0150129795, 14.1008889675, 14.097244978000001, 14.067807912799999, 14.0274028778, 14.012732029, 13.961874008200001, 14.1108551025, 14.0863909721, 14.0686590672, 14.004349947, 14.185241937600001, 13.9961509705, 14.1126899719, 14.1245219707, 14.086353063600001, 14.048186063799999, 14.0126638412, 13.965003013600001, 14.136936903, 14.125771045700001, 14.0789718628, 14.0491027832, 13.986720800400001, 13.978865861899999, 14.133366823199999, 14.0888400078, 14.118774890899999, 14.0262331963, 14.035698890700001, 13.984946966200001, 14.1640629768, 14.148418188100001, 14.091625928899999, 14.075366020200001, 14.0536768436, 13.9736099243, 14.1238529682, 14.118108987799999, 14.103012085, 14.082417964899999, 14.0419850349, 14.015744209299999, 13.983751058599999, 14.112045049700001, 14.1095650196]}], ["meteor-contest", "RawResult", {"changed_times": [0.54956293106099996, 0.44127202034000002, 0.43488597869899998, 0.43912196159400002, 0.43368196487400001, 0.43549299240099998, 0.46642994880700001, 0.43648004531899998, 0.43979382514999998, 0.43357920646699999, 0.43448400497400003, 0.43302893638599999, 0.44884395599400001, 0.43243002891499999, 0.43497920036299997, 0.43333315849300003, 0.43120193481399999, 0.44183921813999999, 0.43201804161099999, 0.43070816993700001, 0.43093800544700001, 0.43396782875099998, 0.431432962418, 0.43151283264200002, 0.432732105255, 0.43281292915300001, 0.43130493164099998, 0.43159103393600001, 0.442057132721, 0.43084406852700002, 0.44665694236800002, 0.43070578575099999, 0.43876791000400001, 0.42976713180499998, 0.43031406402599998, 0.43287897110000001, 0.43102192878700002, 0.43873286247299997, 0.42987489700300002, 0.43340301513700003, 0.42970800399800002, 0.43067002296399998, 0.43263816833500002, 0.43107914924599999, 0.42996597289999999, 0.429845809937, 0.433667898178, 0.43897414207500002, 0.43040704727200002, 0.43301296234100001], "base_times": [0.34740400314300002, 0.34649682045000002, 0.34654498100300002, 0.34828615188599998, 0.34688496589700002, 0.34619617462199997, 0.348422050476, 0.35294890403700002, 0.346195220947, 0.34620308876, 0.34614801406899998, 0.34765100479099997, 0.34694099426300001, 0.34586310386699998, 0.34622812271100001, 0.34607410430899999, 0.348037004471, 0.34679603576700002, 0.34619593620299999, 0.34622883796699999, 0.34815096855200001, 0.34621715545699999, 0.347053050995, 0.34751987457299999, 0.34610795974699998, 0.34874510765099997, 0.34702110290499999, 0.35155105590800001, 0.34627103805499998, 0.34825491905200001, 0.34611105918899998, 0.34621214866599997, 0.34627509117100003, 0.34635615348799997, 0.34847092628499998, 0.34634494781500003, 0.34610104560900001, 0.34599280357399997, 0.34852099418600002, 0.34607791900599999, 0.34652614593499997, 0.34637212753300001, 0.348030090332, 0.346480131149, 0.34624314308199999, 0.346033096313, 0.346461057663, 0.348530054092, 0.34677815437300002, 0.34670209884600001]}], ["nbody_modified", "RawResult", {"changed_times": [0.178552150726, 0.14297604560900001, 0.135512828827, 0.13514518737799999, 0.13505196571399999, 0.13496088981599999, 0.135854959488, 0.13503122329700001, 0.13507986068700001, 0.13504290580700001, 0.13506603241000001, 0.13503408432, 0.13780403137200001, 0.13503003120400001, 0.13493585586500001, 0.13506293296800001, 0.135177135468, 0.13551998138400001, 0.134822845459, 0.13526797294599999, 0.134998083115, 0.13494396209699999, 0.13502788543700001, 0.13709807395900001, 0.135847091675, 0.135095119476, 0.13512802124000001, 0.13511490821800001, 0.13506793975799999, 0.135057926178, 0.135121107101, 0.135377883911, 0.13559198379500001, 0.13520693779000001, 0.13507509231600001, 0.138010978699, 0.13512301445, 0.13525795936599999, 0.13526701927199999, 0.135272979736, 0.13510107994100001, 0.13522982597399999, 0.13502717018099999, 0.13525414466899999, 0.135074853897, 0.13518810272199999, 0.13801503181499999, 0.13605713844299999, 0.134965896606, 0.13791918754599999], "base_times": [0.60831594467200001, 0.60184097290000005, 0.60403108596800004, 0.604898929596, 0.59930682182299999, 0.59702301025399995, 0.60113716125500005, 0.59830880165099998, 0.60461401939399995, 0.59961390495300004, 0.60151195526099999, 0.60288000106799999, 0.60938596725500005, 0.60743498802200002, 0.600023984909, 0.60061693191499999, 0.60342597961400002, 0.59932994842499998, 0.60085797309900002, 0.60232901573200004, 0.59961009025599998, 0.65427803993199996, 0.60101699829099997, 0.60087990760800003, 0.60391497612, 0.60592889785799997, 0.60827589034999996, 0.60652494430500004, 0.60713696479799995, 0.61633515358000002, 0.60946607589699997, 0.61303901672399996, 0.61002016067499998, 0.61310410499599999, 0.61289215087899995, 0.61635208129899999, 0.62035799026500005, 0.61284279823300003, 0.61309885978699996, 0.61907792091400005, 0.61475586891199996, 0.61362600326500005, 0.61063408851599998, 0.61360001564, 0.61051511764499999, 0.61368680000300002, 0.60794496536300002, 0.61147809028599998, 0.61695909500099999, 0.61682605743399999]}], ["pyflate-fast", "RawResult", {"changed_times": [3.3121449947400001, 3.32391285896, 3.3113570213300001, 3.3125739097600002, 3.3533909320799999, 3.27320981026, 3.3294079303699999, 3.3954648971600001, 3.36747694016, 3.3318870067600002, 3.3314881324800001, 3.29482102394, 3.2987878322599999, 3.3235919475600002, 3.2715950012200001, 3.2692580223099998, 3.3049309253699999, 3.2853808403000002, 3.2795870304100001, 3.2983829975100001, 3.25447106361, 3.3078680038499999, 3.2952280044600002, 3.27434897423, 3.2544469833399998, 3.3341419696800001, 3.2960839271500002, 3.2732009887700002, 3.2589120864900001, 3.3169569969200001, 3.2971358299300002, 3.2613229751600001, 3.3093330860100001, 3.25996899605, 3.3377301692999999, 3.2606930732700001, 3.2845060825300001, 3.2952270507799999, 3.2635328769699998, 3.2907948494000001, 3.2753908634200002, 3.29530191422, 3.2842710018200001, 3.2528989314999999, 3.2978489399000002, 3.3096721172299999, 3.2890150547000001, 3.2512168884300001, 3.3030450344100002, 3.2920761108400001], "base_times": [3.18242001534, 3.1863389015200001, 3.1912939548499999, 3.1865100860600002, 3.1824879646299999, 3.19247794151, 3.1979899406399999, 3.1937181949600002, 3.1884422302200002, 3.20042586327, 3.18729782104, 3.1916460990900002, 3.1865410804700001, 3.1863729953800002, 3.1931619644200002, 3.1834938526199998, 3.2051031589500001, 3.1961328983300001, 3.19024610519, 3.1944649219499999, 3.19651508331, 3.2056679725600001, 3.1969439983400001, 3.1953899860399999, 3.1878740787500002, 3.1908388137800001, 3.1916890144300001, 3.1885077953300001, 3.1937019824999999, 3.1973328590399999, 3.1935818195299999, 3.1918790340399998, 3.18938112259, 3.1975688934300002, 3.1952290534999999, 3.1982200145699999, 3.18593597412, 3.1900849342300002, 3.20125794411, 3.1916630268100001, 3.2036459445999999, 3.20056295395, 3.1956481933599998, 3.1894419193300001, 3.2185740470900002, 3.1986999511700001, 3.1960220336899998, 3.1938240528100001, 3.1854040622699999, 3.1919748783099999]}], ["raytrace-simple", "RawResult", {"changed_times": [2.7798359394099998, 2.7791428566, 2.7640888690900001, 2.76987791061, 2.8119649887099998, 2.7948009967799998, 2.8166298866299999, 2.7788488864900001, 2.8109869957, 2.7679738998399999, 2.8182439804100001, 2.8162288665799999, 2.8095500469200001, 2.7651660442399999, 2.76630306244, 2.81266593933, 2.80415391922, 2.76372790337, 2.7615659236900001, 2.7600309848800002, 2.9178488254500001, 2.76185297966, 2.7615509033199999, 2.7574489116700001, 2.7583611011500002, 2.7616539001499998, 2.8071138858800002, 2.7628870010400002, 2.7632458209999999, 2.9184291362799999, 2.7783088684099999, 2.7595660686499999, 2.7599799633000002, 2.7592060565900001, 2.7621810436200001, 2.8037729263300002, 2.7557120323199999, 2.75506186485, 2.7569091320000001, 2.7592680454299998, 2.7995710372899998, 2.7594900131200002, 2.7773869037600001, 2.7546079158799999, 2.7564780712100001, 2.8319108486200002, 2.75832509995, 2.7571489810899998, 2.7593541145299998, 2.7733960151699999], "base_times": [2.66753315926, 2.6737060546899998, 2.67894601822, 2.6753239631699999, 2.6803069114700002, 2.6736009120899999, 2.6738917827600002, 2.67639303207, 2.6885011196100002, 2.6775891780899999, 2.67288613319, 2.6740000248000002, 2.6716890335099999, 2.6766500472999999, 2.6777601242100002, 2.6722629070299999, 2.67888998985, 2.6715672016099998, 2.6804389953599999, 2.6803979873700001, 2.6752359867100002, 2.67298698425, 2.67721605301, 2.6766111850700001, 2.67558693886, 2.6749620437599999, 2.67335510254, 2.67534399033, 2.6777729988100001, 2.67326784134, 2.67556595802, 2.6889100074800001, 2.6720960140200001, 2.6758561134300001, 2.6755039691900002, 2.6716270446800001, 2.6724560260799999, 2.6727318763699999, 2.6774148941, 2.6764769554100001, 2.6750671863600002, 2.6714379787400002, 2.7435669899000001, 2.7485840320600001, 2.70751786232, 2.7092530727400002, 2.67544198036, 2.6766531467400001, 2.6786830425299999, 2.67903494835]}], ["richards", "RawResult", {"changed_times": [0.42485690116899999, 0.391672134399, 0.38697290420500002, 0.38978099823000001, 0.38653588295000002, 0.38539409637499999, 0.38544106483500001, 0.38723397254899999, 0.38484597206100002, 0.38460087776200003, 0.38481712341300001, 0.387125015259, 0.38511180877700002, 0.38335013389599998, 0.38253307342499998, 0.38541388511699998, 0.40138697624199998, 0.41167879104600003, 0.385834932327, 0.4046189785, 0.38262510299699998, 0.39745593071000002, 0.38132977485699998, 0.38368082046500002, 0.380796909332, 0.37990593910199999, 0.38024783134500001, 0.38323211669899998, 0.379715919495, 0.38007807731600002, 0.37966990470899997, 0.40578198432899998, 0.38037204742399999, 0.38033199310299998, 0.38059902191200001, 0.382125139236, 0.38005089759799998, 0.38041114807100002, 0.37975502014200002, 0.41739606857299999, 0.37962198257399998, 0.37924098968499997, 0.40164279937699998, 0.38241791725199997, 0.37951016426099998, 0.37923693656899998, 0.37933707237199998, 0.38282299041700002, 0.37891888618500003, 0.37944197654700001], "base_times": [0.34984207153300001, 0.35352921485900002, 0.35030293464700002, 0.35039401054399999, 0.35356593132000003, 0.35617804527300001, 0.35045003890999998, 0.35419392585800002, 0.34917712211599999, 0.35496902465800001, 0.34898185729999998, 0.35016894340499999, 0.34672594070399998, 0.35246014595000003, 0.34967398643499997, 0.35236191749599999, 0.34780907630899999, 0.35169386863699997, 0.35305500030499998, 0.35085582733199999, 0.35157895088199997, 0.348843097687, 0.355578899384, 0.35684394836400002, 0.35460305213900001, 0.34866809845000002, 0.34962701797500001, 0.35786795616099998, 0.35232281684900002, 0.34995484352099998, 0.357528924942, 0.35109806060799997, 0.353901147842, 0.34876418113699997, 0.34823799133299999, 0.34872198104899998, 0.35193896293600002, 0.34771203994799998, 0.34916996955899998, 0.34796500205999997, 0.34642791748000001, 0.35339903831500002, 0.34859085083000002, 0.35140800476099998, 0.34648203849800002, 0.355483055115, 0.353637933731, 0.34880208969100002, 0.37337613105799999, 0.35136389732399997]}], ["spambayes", "RawResult", {"changed_times": [0.59816980361899996, 0.78452110290499999, 0.83023285865800001, 0.93411684036300002, 0.69184017181400004, 1.10234284401, 0.74256396293600002, 0.71328186988800002, 0.57437109947200005, 0.86799788474999995, 0.67963004112199998, 0.58452200889600003, 0.58005404472400002, 0.52808690071099995, 1.1698250770600001, 0.69769787788399995, 0.64825582504299994, 0.694402217865, 0.63659095764200002, 0.58709597587600004, 0.52931094169600001, 0.65094089508099995, 0.54797601699800003, 0.52007412910499995, 0.61390900611900001, 0.545217990875, 0.66334605216999998, 0.54549193382299999, 0.67743301391599997, 0.51848697662400001, 0.71530890464800001, 0.76569509506199995, 0.52243494987500005, 0.51943516731299999, 0.61838197708099996, 0.60029411315900005, 0.64244699478099998, 0.53688502311700004, 0.51747083663899995, 0.51864504814099999, 0.64398002624499995, 0.516402006149, 0.54355907440200002, 0.60826396942100003, 0.51326608657799999, 0.509041070938, 0.61415004730199996, 0.592803001404, 0.50892901420600001, 0.60751390457200005], "base_times": [0.29755282402, 0.29553508758500002, 0.29539799690200003, 0.29539704322799998, 0.29553294181799999, 0.29756402969399998, 0.295476913452, 0.29511308670000003, 0.29523301124599999, 0.294858932495, 0.29885101318399998, 0.29543805122400002, 0.29549407959000001, 0.29555702209500001, 0.295747041702, 0.29760813713099998, 0.29595708847000002, 0.29637312889099998, 0.29531884193399999, 0.295172929764, 0.29555201530500003, 0.29735279083299998, 0.29529094696000002, 0.29522705078099998, 0.29555082321199999, 0.29535007476800001, 0.29763197898900001, 0.29533004760699999, 0.29603099823000001, 0.29527783393899998, 0.29541301727300001, 0.29754710197399997, 0.29553294181799999, 0.295480012894, 0.295413970947, 0.29505491256700001, 0.29747200012199998, 0.29603290557900003, 0.29523515701300002, 0.295083999634, 0.29551792144799999, 0.29732418060299998, 0.29565906524699997, 0.29551196098299998, 0.29589390754700001, 0.295162916183, 0.29713106155399999, 0.29647493362400001, 0.295417070389, 0.29561495780899999]}], ["spectral-norm", "RawResult", {"changed_times": [0.46351408958399998, 0.36286616325400001, 0.36123991012599999, 0.35890722274800002, 0.35978198051499999, 0.35527896881100002, 0.35585403442399999, 0.35739207267799999, 0.35975408554100002, 0.35413813590999998, 0.35580110549900001, 0.35268497466999998, 0.364890098572, 0.35489296913099999, 0.35201191902200002, 0.354170084, 0.35294985771199999, 0.35513281822199999, 0.35184288024900001, 0.35136795043899999, 0.35138392448400002, 0.35504508018499997, 0.35184502601599998, 0.35189914703399999, 0.35134291648900001, 0.35165905952499998, 0.35476517677300001, 0.352794885635, 0.35161089897199999, 0.35130691528300001, 0.355042934418, 0.35176992416399999, 0.35218715667700001, 0.353415966034, 0.35497093200699997, 0.35284900665300001, 0.351359128952, 0.35298585891700002, 0.35289096832299999, 0.35544204711900002, 0.35241413116499998, 0.35347700118999997, 0.35199999809299998, 0.35367488861099999, 0.35371303558299999, 0.35233402252200002, 0.35166788101200003, 0.35387802124000001, 0.37293601036099999, 0.35278296470600001], "base_times": [0.50352501869199995, 0.47929906845100001, 0.48261594772299998, 0.47830605506899998, 0.48235297203100003, 0.48249197006200001, 0.48067307472199999, 0.48097801208500002, 0.47920393943799999, 0.47869086265600003, 0.48005580902099998, 0.47897410392799999, 0.48137307167100002, 0.483394145966, 0.47975897788999999, 0.47675085067700002, 0.48147702217100002, 0.47802019119299999, 0.482943058014, 0.47830295562699998, 0.48056387901300002, 0.47623491287199998, 0.47782897949199998, 0.47907996177700002, 0.48068904876700003, 0.47843909263599999, 0.48215293884299998, 0.478461027145, 0.48303794860799998, 0.47899913787800003, 0.478845119476, 0.480139970779, 0.48173093795799998, 0.47723102569600001, 0.47754788398699999, 0.47700595855700001, 0.48208713531500003, 0.47760915756200001, 0.480431079865, 0.48341894149800002, 0.47852706909199999, 0.47913908958399998, 0.48555207252499999, 0.490929841995, 0.47841882705700001, 0.48059010505700001, 0.47776508331299999, 0.47688412666300001, 0.48040413856500003, 0.47939014434799998]}], ["telco", "RawResult", {"changed_times": [1.452091, 1.324082, 1.3400840000000001, 1.4160889999999999, 1.3320829999999999, 1.3800859999999999, 1.280081, 1.304081, 1.304082, 1.300081, 1.308082, 1.2840800000000001, 1.308082, 1.296081, 1.3240829999999999, 1.308081, 1.284081, 1.324082, 1.288081, 1.300081, 1.284081, 1.328082, 1.3200829999999999, 1.288081, 1.300081, 1.2880799999999999, 1.288081, 1.308081, 1.288081, 1.308082, 1.284081, 1.304081, 1.2800800000000001, 1.304081, 1.288081, 1.304081, 1.288081, 1.300081, 1.288081, 1.2920799999999999, 1.312082, 1.284081, 1.308082, 1.2800800000000001, 1.304081, 1.288081, 1.2960799999999999, 1.288081, 1.304081, 1.284081], "base_times": [1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.2, 1.22, 1.2, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.2, 1.2, 1.21, 1.21, 1.21]}], ["twisted_names", "RawResult", {"changed_times": [0.022883295194508008, 0.020242914979757085, 0.024330900243309004, 0.02004008016032064, 0.020080321285140562, 0.018832391713747645, 0.017793594306049824, 0.018484288354898338, 0.017953321364452424, 0.017825311942959002, 0.017793594306049824, 0.019801980198019802, 0.017825311942959002, 0.017667844522968199, 0.017761989342806393, 0.017761989342806393, 0.018691588785046728, 0.018214936247723135, 0.017636684303350969, 0.021231422505307854, 0.017699115044247787, 0.018518518518518517, 0.017793594306049824, 0.017699115044247787, 0.018050541516245487, 0.017730496453900711, 0.01937984496124031, 0.01824817518248175, 0.017761989342806393, 0.018181818181818181, 0.017574692442882251, 0.017953321364452424, 0.017574692442882251, 0.017482517482517484, 0.017574692442882251, 0.017574692442882251, 0.017513134851138354, 0.020161290322580645, 0.017543859649122806, 0.017482517482517484, 0.018796992481203006, 0.017482517482517484, 0.017421602787456445, 0.017513134851138354, 0.017574692442882251, 0.017513134851138354, 0.017513134851138354, 0.020161290322580645, 0.017574692442882251, 0.018761726078799251], "base_times": [0.0095693779904306216, 0.0095147478591817315, 0.0095877277085330784, 0.0095602294455066923, 0.0095877277085330784, 0.0096339113680154135, 0.0095693779904306216, 0.0095602294455066923, 0.0095602294455066923, 0.0095785440613026813, 0.0095510983763132766, 0.0095785440613026813, 0.0095969289827255271, 0.0095969289827255271, 0.0096246390760346481, 0.0095602294455066923, 0.0095969289827255271, 0.0095785440613026813, 0.0095785440613026813, 0.0095602294455066923, 0.0096153846153846159, 0.0095602294455066923, 0.0096618357487922701, 0.0095693779904306216, 0.009727626459143969, 0.0095693779904306216, 0.0095602294455066923, 0.0095419847328244278, 0.0095693779904306216, 0.0095785440613026813, 0.0095693779904306216, 0.0095510983763132766, 0.0095969289827255271, 0.0095602294455066923, 0.0095785440613026813, 0.0095877277085330784, 0.0095693779904306216, 0.0095419847328244278, 0.0095785440613026813, 0.0095238095238095247, 0.0095693779904306216, 0.0095693779904306216, 0.0095877277085330784, 0.0095693779904306216, 0.0095877277085330784, 0.0095969289827255271, 0.0095785440613026813, 0.0095693779904306216, 0.0095602294455066923, 0.0095785440613026813]}]], "branch": "trunk", "revision": 0}

Modified: pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json
==============================================================================
--- pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json	(original)
+++ pypy/extradoc/talk/pepm2011/benchmarks/jit-nopspec.json	Wed Oct 13 22:14:28 2010
@@ -1 +1 @@
-{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [19.594973802599998, 17.836122035999999, 17.7611591816, 17.7673540115, 17.797179937399999, 17.701630115499999, 17.701794147499999, 17.726331234, 17.7549710274, 17.7046818733, 17.715419054000002, 17.7220032215, 17.708485126500001, 17.698208808899999, 17.943202018699999, 17.7065188885, 17.7018699646, 17.723783969900001, 17.703621864300001, 17.702537059800001, 17.698129892299999, 17.717091083500002, 17.693710088700001, 17.698023796099999, 17.790216922799999, 17.7049281597, 17.7118620872, 17.7670118809, 17.7513260841, 17.712387800199998, 17.703951120399999, 17.7405331135, 17.704728841800002, 17.7038481236, 17.719988107700001, 17.7051708698, 17.701725959800001, 17.709410905799999, 17.716867923700001, 17.694391012200001, 17.700695037799999, 17.712723970399999, 17.692533969900001, 17.705747842800001, 17.712710142100001, 17.758448123899999, 17.701386928600002, 17.700757980300001, 18.095870017999999, 17.7208271027], "base_times": [2.76247382164, 2.75780797005, 2.7526109218600001, 2.7581448554999999, 2.7579169273400002, 2.7591660022700002, 2.7544469833399998, 2.75570702553, 2.7566180229200001, 2.7799978256200002, 2.75436115265, 2.7584359645799998, 2.7715408801999999, 2.7583799362199999, 2.75467610359, 2.7834770679499998, 2.7558917999300001, 2.7597181797000001, 2.75486803055, 2.7581379413599998, 2.7532849311800001, 2.7636089324999999, 2.75440192223, 2.7553980350499998, 2.7562680244400002, 2.7588288784000001, 2.75431799889, 2.7562849521600001, 2.7566390037500001, 2.75684499741, 2.7588860988600001, 2.7639617919899999, 2.758425951, 2.7611510753599999, 2.75679397583, 2.7544021606400002, 2.7587559223200002, 2.7592377662700001, 2.7555589675899999, 2.7590520381900001, 2.75563788414, 2.76086902618, 2.7588081359899999, 2.7565140724199999, 2.75627803802, 2.76230502129, 2.7579100132000001, 2.75721096992, 2.7558379173300001, 2.7606220245399999]}], ["django", "RawResult", {"changed_times": [0.30002903938300002, 0.25580286979700001, 0.261374950409, 0.26095104217499998, 0.25220704078700001, 0.26073884963999999, 0.26132106780999997, 0.254949092865, 0.26201701164199998, 0.26217198371900002, 0.25196504592899999, 0.26127791404700001, 0.26208996772799997, 0.254974842072, 0.261093854904, 0.261404037476, 0.25236296653700002, 0.26157402992200002, 0.26323699951200003, 0.25301718711900001, 0.26132202148400002, 0.26150584220899997, 0.25210714340200002, 0.26118707656899998, 0.26585507392899999, 0.25383305549599999, 0.26143312454200002, 0.26261687278700002, 0.267751932144, 0.26117897033699999, 0.264419078827, 0.252134799957, 0.26171708107000002, 0.26141405105600002, 0.25201201438900001, 0.261445045471, 0.26490807533299998, 0.25207686424300002, 0.26115393638599999, 0.26161694526700002, 0.25278902053800001, 0.261168003082, 0.26426982879599997, 0.25209307670600001, 0.261046171188, 0.26155900955200001, 0.25206208229100002, 0.261226177216, 0.26494908332799999, 0.252925872803], "base_times": [0.995049953461, 0.99419713020300005, 0.99113988876299997, 0.99470710754400005, 0.99212503433200006, 0.99265909194900004, 0.99358582496600001, 0.99400091171299998, 0.99305796623200004, 0.99362897872900002, 0.99429702758799998, 0.99161601066600003, 0.99425101280200001, 0.99299907684300004, 0.99139285087600004, 0.99515914917000003, 0.99177312850999999, 0.99377703666700001, 0.99326705932600001, 0.99304509162900001, 0.99348807334900002, 0.99273705482499996, 0.99258708953899999, 0.99477910995499996, 0.99403500556900004, 0.99051785469099995, 0.994111061096, 0.99377894401599998, 0.99141192436199999, 0.99447512626599999, 0.99370288848900001, 0.99225115776100004, 0.99282193183900003, 0.99233508110000002, 0.99293303489700002, 0.99670696258500002, 0.99345684051500005, 0.99477696418799999, 0.99382996559100001, 0.99122619628899999, 0.99395608901999999, 0.99324393272400002, 0.99102210998499995, 0.99499702453600003, 0.99375295639000005, 0.99156498909000002, 0.99277806282000003, 0.99424910545300005, 0.99084305763199998, 0.993386983871]}], ["fannkuch", "RawResult", {"changed_times": [0.52755904197699999, 0.47701597213699998, 0.46741414070100001, 0.47383809089700002, 0.46101403236400001, 0.46737408638, 0.47243213653600002, 0.47586488723800002, 0.46479701995799999, 0.46248412132299999, 0.46604609489400001, 0.46004605293299999, 0.46376204490700002, 0.46708703041100003, 0.47270798683199999, 0.472360134125, 0.47210407257100001, 0.47691607475300002, 0.47826886177099998, 0.47204399108900003, 0.47201299667399998, 0.46304917335500001, 0.46217513084400003, 0.46784305572500001, 0.460357189178, 0.46357607841499998, 0.46536898613, 0.46894693374599999, 0.46510815620399998, 0.46110606193499998, 0.46555995941200001, 0.46238088607799999, 0.46605896949800002, 0.46549701690700002, 0.45882105827300002, 0.45827507972699999, 0.473033905029, 0.47556495666499998, 0.460822105408, 0.47130393981899998, 0.476540803909, 0.46522593498199999, 0.46266603469799999, 0.47138500213599999, 0.46661305427600003, 0.46587181091300001, 0.47512102127099998, 0.47284221649199998, 0.47128200530999997, 0.47180795669600001], "base_times": [1.9883148670199999, 1.9792110919999999, 1.9909551143599999, 1.98928499222, 1.97898578644, 1.98664617538, 1.9864010810899999, 1.98417901993, 1.9872028827699999, 1.98194694519, 1.9878189563799999, 1.98948693275, 1.9811220169099999, 1.9890899658200001, 1.9894771575900001, 1.9801352024100001, 1.9894909858700001, 1.9891271591199999, 1.98107600212, 1.99568986893, 1.9895131587999999, 1.9803609847999999, 2.0069239139600001, 1.9916350841499999, 1.98022890091, 1.98729896545, 1.99073004723, 1.98082304001, 1.98834180832, 1.9909420013400001, 1.9793639183, 1.98839306831, 1.9902248382600001, 1.9822490215299999, 1.98810195923, 1.99047279358, 1.98221516609, 1.98903012276, 1.9900078773500001, 1.9820079803499999, 1.9892179966000001, 1.9895989894899999, 1.9800350665999999, 1.9895751476300001, 1.9891831874799999, 1.9804949760399999, 1.9896249771100001, 1.9898300170900001, 1.9807391166699999, 1.9895370006599999]}], ["go", "RawResult", {"changed_times": [0.46677207946799998, 0.51466798782300005, 0.388464927673, 0.50362396240200002, 0.36921596527099998, 0.41929197311400002, 0.37606406211900001, 0.428333997726, 0.36696696281399999, 0.33521819114700002, 0.35271787643399999, 0.47167301177999998, 0.36826205253599997, 0.33157110214199997, 0.39287805557299998, 0.30100107192999997, 0.31953597068799999, 0.34765505790700002, 0.31619405746500001, 0.32286810874900002, 0.37090992927600003, 0.328006982803, 0.31113696098299998, 0.355150938034, 0.30218696594200001, 0.31915521621699999, 0.31008601188700002, 0.34765815734900002, 0.30721187591600002, 0.32940602302600003, 0.30662703514099998, 0.33826303482100001, 0.29518413543700001, 0.32855200767499998, 0.37890100479099997, 0.295953989029, 0.28935694694500003, 0.32658910751300002, 0.31018304824800003, 0.32643103599500001, 0.28621220588700003, 0.29142880439800001, 0.28649687766999998, 0.301717042923, 0.288222074509, 0.353245973587, 0.33032798767100002, 0.285399913788, 0.29600501060500001, 0.28842616081200001], "base_times": [0.94673204422000001, 0.94645810127300001, 0.94507908821099995, 0.944802045822, 0.941561937332, 0.944491147995, 0.94291400909400003, 0.95224094390900005, 0.94410514831500003, 0.943855047226, 0.94802284240699997, 0.94546699523900002, 0.94773101806600002, 0.94944882392899999, 0.94337701797499995, 0.94636011123700003, 0.94657802581799999, 0.94187402725199998, 0.94828701019299999, 0.94231510162400001, 0.94628596305799995, 0.94881415367099997, 0.94279098510699999, 0.94889903068500003, 0.95280098915099998, 0.94382286071800003, 0.94861888885500001, 0.94221186637900001, 0.94600486755400004, 0.94160294532800004, 0.94753003120400003, 0.94430899620099995, 0.94843101501500005, 0.96007490158099995, 0.94707822799700003, 0.94710588455199995, 0.95057606697100006, 0.94280695915199997, 0.94800686836199999, 0.94509601593000003, 0.96038508415199997, 0.94446587562600004, 0.94845819473299997, 0.94734406471300003, 0.94510412216200002, 0.94387602806100002, 0.94524908065799995, 0.94992303848299997, 0.94564318656900004, 0.94311594963099998]}], ["html5lib", "RawResult", {"changed_times": [15.457784891099999, 11.3277301788, 10.600764036199999, 10.4216821194, 10.2325599194, 10.2733259201, 10.3756709099, 10.156955957399999, 10.101744890200001, 10.133589983, 10.206784963600001, 10.1694169044, 10.0392251015, 10.155544042600001, 10.1885080338, 10.058653831499999, 10.2810080051, 10.2360329628, 10.039163112600001, 10.1998898983, 10.119593143499999, 10.1641438007, 10.0413920879, 10.107867002500001, 10.135586977000001, 10.1675360203, 10.1493079662, 10.060842990899999, 10.178154945399999, 10.1074330807, 10.170939922300001, 10.047384023699999, 10.0127818584, 10.050449132900001, 10.161669015899999, 10.037492990500001, 10.0900728703, 10.190579891200001, 9.9983761310600006, 10.1199560165, 10.003987073899999, 10.119729042099999, 10.1512379646, 10.054825067499999, 9.9258110523200003, 10.0976631641, 10.0940468311, 10.122640132900001, 9.9850409030900007, 10.099215984300001], "base_times": [13.800020933200001, 13.988759994500001, 13.945523977300001, 13.9515860081, 13.8968560696, 14.0439591408, 14.0257151127, 13.9940800667, 13.977200985, 13.965268135100001, 13.8857018948, 14.0525372028, 14.022065877899999, 14.0086979866, 13.961042880999999, 13.921565771099999, 13.924687862400001, 14.0379269123, 14.256644964199999, 13.9928328991, 14.0013000965, 13.9456880093, 13.914921999000001, 14.061727047, 14.0501580238, 14.0101730824, 13.9787597656, 13.921334981899999, 13.889928102500001, 14.0468928814, 14.0285110474, 14.0453250408, 13.9559979439, 13.9637241364, 13.9317610264, 14.0579519272, 14.0324690342, 14.0096049309, 13.9675040245, 13.964776992799999, 13.924336910199999, 14.0519120693, 14.0543570518, 14.0083789825, 13.9841370583, 13.954766988799999, 13.9033069611, 13.8949439526, 14.038803100599999, 14.0201179981]}], ["meteor-contest", "RawResult", {"changed_times": [0.47674798965499998, 0.403456926346, 0.39809894561800002, 0.40778803825400001, 0.39824509620699999, 0.39766597747799998, 0.40741491317700002, 0.39641904830899999, 0.39668798446699999, 0.39379596710199999, 0.39647078514099998, 0.393444061279, 0.40611696243299999, 0.39961814880399998, 0.39285993576, 0.39224314689599998, 0.39388489723199999, 0.39147710800199997, 0.39165306091300001, 0.39033508300800002, 0.39437389373800003, 0.39175200462300003, 0.391242027283, 0.39103412628200002, 0.39511799812300002, 0.398463010788, 0.39204406738300002, 0.39300107955899999, 0.39334487914999999, 0.391426801682, 0.39687800407399998, 0.39431095123299997, 0.39070391654999997, 0.39106798172000001, 0.39058089256299999, 0.39391088485699999, 0.39834094047500002, 0.39057397842399999, 0.39198994636500001, 0.39368200302099998, 0.39073801040599998, 0.39122891426099998, 0.39064598083500002, 0.39390277862500001, 0.390840053558, 0.39128398895299998, 0.39098405838, 0.39335107803300001, 0.39786505699199998, 0.39080500602700002], "base_times": [0.348359107971, 0.34692883491499998, 0.346405982971, 0.34667205810500001, 0.34701514244100001, 0.34753990173299998, 0.346569061279, 0.34635281562800002, 0.346190929413, 0.34796619415300001, 0.34590101242100002, 0.34643101692200001, 0.34670686721799998, 0.34775090217600002, 0.34620094299300003, 0.34622883796699999, 0.34623694419899997, 0.34779596328700002, 0.348699092865, 0.34642219543500002, 0.347030878067, 0.34581995010400002, 0.347792863846, 0.34766483306899998, 0.34653806686400002, 0.34683513641399999, 0.34936118125900001, 0.34839010238599999, 0.346399068832, 0.34670805931100002, 0.34576678276099998, 0.348170042038, 0.34661507606500003, 0.34650683402999999, 0.34621381759600001, 0.34878396987900001, 0.34638595581100001, 0.34623980522199999, 0.34593510627700003, 0.34620404243500003, 0.34790611267100002, 0.346742153168, 0.34607505798299998, 0.34633517265300001, 0.34812307357799999, 0.34604907035799998, 0.34677386283900002, 0.34667706489599998, 0.34659886360199998, 0.34881806373599999]}], ["nbody_modified", "RawResult", {"changed_times": [0.12817502021800001, 0.114922046661, 0.105006933212, 0.10339498519900001, 0.104254007339, 0.103782892227, 0.103751897812, 0.105894088745, 0.104180812836, 0.103867053986, 0.103599071503, 0.103564023972, 0.103793859482, 0.103470087051, 0.103558063507, 0.103418111801, 0.10340905189500001, 0.10347700119, 0.103801012039, 0.10350394249, 0.103738069534, 0.10368895530699999, 0.106331110001, 0.103569984436, 0.10441899299600001, 0.103644132614, 0.103709936142, 0.10339307785, 0.103735923767, 0.10362410545300001, 0.103760004044, 0.103440999985, 0.10381603240999999, 0.103622913361, 0.10393595695500001, 0.103415966034, 0.103775978088, 0.106086969376, 0.103862047195, 0.103460073471, 0.103792905807, 0.10356497764600001, 0.103854894638, 0.103960990906, 0.103703975677, 0.103446006775, 0.10406708717300001, 0.103525876999, 0.103811979294, 0.10510087013199999], "base_times": [0.64283800125099999, 0.64096808433499997, 0.64577603340099998, 0.64423298835800002, 0.64046311378499998, 0.63754677772500001, 0.63747310638400001, 0.63725614547700005, 0.63658809661899995, 0.64228487014799995, 0.64188194274900001, 0.64234304428099998, 0.63921594619800004, 0.63971495628399999, 0.64251685142500004, 0.64014887809800003, 0.64212799072299998, 0.64027500152600003, 0.63313889503499998, 0.63153290748599999, 0.63076686859099995, 0.63370203971899997, 0.63236403465299995, 0.63347291946399997, 0.63246107101399995, 0.63299584388700003, 0.63348007202099998, 0.63026380538899995, 0.632724046707, 0.63243818283099995, 0.63531398773199999, 0.63819217681899998, 0.63529610633900002, 0.63803815841699996, 0.63671016693100002, 0.64081311225899995, 0.63594484329199996, 0.64017796516400005, 0.64193677902199997, 0.64040708541900004, 0.64205718040500004, 0.64145493507399998, 0.640824079514, 0.65274095535300003, 0.64189887046799998, 0.64417505264300001, 0.64191007614100004, 0.640285015106, 0.63908886909499996, 0.64495897293100002]}], ["pyflate-fast", "RawResult", {"changed_times": [1.83701682091, 1.8073658943199999, 1.8406269550300001, 1.82018709183, 1.8337500095399999, 1.8074929714200001, 1.84810090065, 1.8087329864499999, 1.8338890075700001, 1.81683397293, 1.78763699532, 1.79593896866, 1.81039500237, 1.7986569404599999, 1.8007159233100001, 1.8014991283399999, 1.7972960472099999, 1.80028700829, 1.8007249832200001, 1.85811805725, 1.7931230068199999, 1.82364487648, 1.7935469150500001, 1.82844901085, 1.7920200824700001, 1.82318806648, 1.7929878234900001, 1.82861304283, 1.79402518272, 1.82314896584, 1.79180502892, 1.8225588798500001, 1.80868387222, 1.80998492241, 1.8257219791399999, 1.8084599971799999, 1.81027412415, 1.80907607079, 1.80926918983, 1.82390213013, 1.8229339122799999, 1.9030389785799999, 1.9522318839999999, 1.8275589942899999, 1.81771707535, 1.8227140903500001, 1.8171408176399999, 1.8353171348599999, 1.8287210464500001, 1.83105301857], "base_times": [3.1601369381, 3.17563796043, 3.1705720424699999, 3.17048382759, 3.1654319763199998, 3.1680698394800002, 3.1681051254299999, 3.1718590259599999, 3.1830987930300001, 3.1775770187400001, 3.1700208187099999, 3.1612432002999999, 3.15727186203, 3.1732730865500001, 3.1751198768600002, 3.1624131202700001, 3.1771841049199998, 3.1661839485200001, 3.16439390182, 3.1774570941900002, 3.1796469688400002, 3.16518807411, 3.1729710102099999, 3.1790311336500001, 3.17372894287, 3.1641249656700001, 3.1653139591200001, 3.16444206238, 3.1757740974400002, 3.16623592377, 3.1762218475299999, 3.1651339530899998, 3.1656432151799998, 3.1724681854200001, 3.1735718250299998, 3.1667320728299999, 3.1666581630700001, 3.1662278175399998, 3.1711161136600001, 3.1634638309500001, 3.1635270118699998, 3.1630849838300001, 3.1776990890499999, 3.17032909393, 3.1747660636899999, 3.1702299117999999, 3.17133522034, 3.1701278686499998, 3.1603660583500002, 3.1653149128]}], ["raytrace-simple", "RawResult", {"changed_times": [0.659919023514, 0.66626596450800002, 0.65816497802700002, 0.65398693084699999, 0.63991713523899996, 0.66545104980500003, 0.65175795555100002, 0.63982892036399996, 0.66220998764000005, 0.65392279624899996, 0.63932204246500002, 0.65032505989099998, 0.65231609344499997, 0.63614416122399997, 0.65278100967399999, 0.63543891906700001, 0.65301513671900002, 0.647172212601, 0.63230013847400002, 0.64931488037100005, 0.689571857452, 0.63732600212099999, 0.67306900024399996, 0.63961911201499999, 0.650694131851, 0.63388586044299999, 0.65063095092800005, 0.64781498909000002, 0.63850116729700002, 0.65920114517200001, 0.68630504608200005, 0.63646793365499998, 0.64802885055500004, 0.64920401573200004, 0.63225889205899999, 0.65038490295399998, 0.63108992576599998, 0.64513182640099997, 0.63317394256600001, 0.64602303504900005, 0.64737510681199995, 0.63001990318300005, 0.64657092094400004, 0.63210487365699997, 0.645840883255, 0.65991592407199995, 0.62948513030999997, 0.65090513229400004, 0.63588213920600001, 0.66070103645300005], "base_times": [2.6739389896399999, 2.6792252063799999, 2.67714691162, 2.6755440235100001, 2.6839830875400001, 2.67896699905, 2.6759178638500001, 2.6771140098599999, 2.6831061840100001, 2.6781611442600002, 2.6843791008000002, 2.67555809021, 2.67742395401, 2.68145084381, 2.6856698989900001, 2.6754090786, 2.6820809841200002, 2.7008218765300001, 2.6860649585699998, 2.68976688385, 2.7091708183300001, 2.6831979751600001, 2.6825587749499999, 2.7229101657900001, 2.7227280139899999, 2.7074718475299999, 2.7032160758999999, 2.69513916969, 2.7023360729200001, 2.7511570453599998, 2.6899399757400002, 2.6838908195500002, 2.7002210617100002, 2.8071360588099998, 2.6863989830000001, 2.6864531040199999, 2.69359779358, 2.7453310489699998, 2.7076761722599998, 3.06150817871, 2.7332241535200001, 2.7483820915199999, 2.7639210224199999, 3.4124541282699998, 2.7048859596299999, 2.6892578601800001, 2.6912610530899999, 2.6821250915500001, 2.68154597282, 2.6890358924900002]}], ["richards", "RawResult", {"changed_times": [0.12052297592199999, 0.093158006668099994, 0.091630935668900002, 0.091804027557399998, 0.091670036315899997, 0.091012954711900004, 0.091180086135899993, 0.102787017822, 0.090846061706500003, 0.090148925781199998, 0.093178987503099997, 0.090260028839100004, 0.090500116348300005, 0.098414897918700006, 0.090541839599599994, 0.090975999832200005, 0.090515136718800002, 0.088860034942600002, 0.089382886886599996, 0.087617158889799998, 0.087574005127000004, 0.088225841522200005, 0.087996006012, 0.087512969970700002, 0.0879878997803, 0.096507072448699993, 0.087088108062699995, 0.090749025344800005, 0.087268829345700002, 0.087903976440400003, 0.087549924850500005, 0.087819099426300007, 0.087373971939100001, 0.087810039520300004, 0.087095975875899997, 0.0871419906616, 0.088514089584400005, 0.087375879287700006, 0.086944818496699994, 0.087454080581699997, 0.087461948394799996, 0.097897052764900003, 0.087609052658099998, 0.087203979492200004, 0.089391946792599999, 0.088086128234900002, 0.086940050125100005, 0.087315082550000006, 0.086781978607199994, 0.086982011794999994], "base_times": [0.35359501838700003, 0.35667300224300003, 0.347419977188, 0.35246205329899999, 0.35532999038700003, 0.35530591011000001, 0.35107088088999999, 0.35549902915999998, 0.35110902786300002, 0.35400104522699999, 0.35261011123699998, 0.35541200637800002, 0.35351705551099999, 0.35617089271500002, 0.34958410263099998, 0.35414409637499999, 0.35661983489999999, 0.35693001747100001, 0.35174703598000001, 0.34876704215999998, 0.35491299629200002, 0.35479497909500002, 0.35416102409400002, 0.355372905731, 0.35764288902300001, 0.35487508773799997, 0.35283422470100001, 0.35428977012599999, 0.35132908821100001, 0.35751414299000001, 0.362582206726, 0.35600781440700002, 0.35644507408100001, 0.35086202621500001, 0.35400605201699997, 0.34989190101599998, 0.35055398940999999, 0.35048794746400003, 0.35601019859299998, 0.35249996185299998, 0.35182499885599999, 0.35495114326499999, 0.35285496711699998, 0.355787038803, 0.35039997100800002, 0.35504388809199999, 0.35427594184900002, 0.35611510276800001, 0.35387182235699999, 0.34967398643499997]}], ["spambayes", "RawResult", {"changed_times": [0.48457980155899999, 0.56999397277800001, 0.53518199920700005, 0.54824614524799997, 0.49303603172299998, 0.58462905883799998, 0.51067399978600003, 0.49086999893200001, 0.48435497283899998, 0.42737793922400003, 0.494954109192, 0.43986797332799998, 0.40522003173799997, 0.40721011161800003, 0.57795190811200003, 0.48255205154399999, 0.462779998779, 0.44051218032799999, 0.39301896095299999, 0.39432501792899999, 0.39888882637, 0.43233299255399998, 0.42077803611800002, 0.39932012558000002, 0.40041995048500001, 0.409831047058, 0.44888401031500003, 0.40377283096299998, 0.39395999908399998, 0.40184497833299998, 0.41734480857799999, 0.49758601188700002, 0.39052391052199997, 0.392160177231, 0.39913892746000001, 0.42745113372799998, 0.36878705024699998, 0.38831090927099998, 0.37554693222000002, 0.40881800651599998, 0.37994885444600002, 0.37922286987300002, 0.39221501350400001, 0.39279913902300001, 0.34234404563900001, 0.37785792350800002, 0.38006091117899998, 0.37980985641499998, 0.375820875168, 0.345314979553], "base_times": [0.29905700683600001, 0.298835992813, 0.29825806617700001, 0.29831504821799998, 0.30105614662199998, 0.298439979553, 0.29832100868200001, 0.29835200309799997, 0.29867696762099999, 0.29994201660199998, 0.29847502708399998, 0.29867196083100001, 0.29891991615300001, 0.29807305336000001, 0.30059790611300002, 0.298420906067, 0.29828691482500003, 0.29897713661199998, 0.29811811447100001, 0.300410985947, 0.29847979545600001, 0.2982878685, 0.29861903190599998, 0.29877710342399999, 0.30105996131899998, 0.29816794395399998, 0.29835987091100002, 0.29860401153600002, 0.2985060215, 0.30007410049400002, 0.29955601692200001, 0.299430847168, 0.29854393005399998, 0.29915118217499997, 0.29966092109699999, 0.29980802536000001, 0.29794311523400002, 0.29930615425099999, 0.298702001572, 0.29842782020600001, 0.30247402191200001, 0.29896807670600001, 0.298580169678, 0.29869008064300001, 0.299312114716, 0.30094408988999999, 0.29845499992399999, 0.29868721962, 0.29998612403899999, 0.29916310310400002]}], ["spectral-norm", "RawResult", {"changed_times": [0.18530607223500001, 0.118788957596, 0.11839389801, 0.117673158646, 0.115185976028, 0.115034103394, 0.114280939102, 0.115108966827, 0.115477085114, 0.114944934845, 0.11555695533800001, 0.111644983292, 0.111333847046, 0.111480951309, 0.11155700683600001, 0.11224794387799999, 0.112046003342, 0.111447095871, 0.11138176918000001, 0.11152100563, 0.111184120178, 0.11187982559200001, 0.111309051514, 0.115062952042, 0.111434936523, 0.111573934555, 0.111452102661, 0.128764867783, 0.11133313179, 0.11140704155, 0.11155796051, 0.111167907715, 0.111421108246, 0.11097002029399999, 0.111446857452, 0.110933065414, 0.110732078552, 0.113990068436, 0.11116909980799999, 0.111049890518, 0.11179399490399999, 0.11199402809099999, 0.110781908035, 0.111202001572, 0.111080884933, 0.110934972763, 0.111124992371, 0.110868930817, 0.11123108863800001, 0.112289190292], "base_times": [0.48088502883899997, 0.47923994064300002, 0.48160886764499999, 0.47763085365300001, 0.47884607315099997, 0.48438310623199998, 0.48113894462599999, 0.47794389724699998, 0.47896790504499998, 0.48052501678499998, 0.47819399833699999, 0.47891688346900002, 0.48100805282600001, 0.48024415969799999, 0.47854995727499999, 0.47899198532100001, 0.47972202301, 0.47723197937, 0.48147606849699998, 0.47865796089200002, 0.47799110412599999, 0.47740817070000002, 0.47738909721400002, 0.47732496261599999, 0.47762703895600001, 0.48322010040300001, 0.47945380210900002, 0.47645592689499999, 0.479950904846, 0.47760391235400002, 0.47802209854099997, 0.47923922538800001, 0.477582931519, 0.47682905197100001, 0.47848796844500002, 0.47821402549699998, 0.47620606422400003, 0.48374009132399998, 0.481714963913, 0.47922301292399999, 0.47735190391499999, 0.48569798469499997, 0.47899794578600002, 0.477490901947, 0.47891783714300001, 0.47670793533299999, 0.47655487060500001, 0.47783589363099999, 0.47823095321699999, 0.47739887237500001]}], ["telco", "RawResult", {"changed_times": [0.56803599999999999, 0.51203200000000004, 0.51603200000000005, 0.54403400000000002, 0.51203200000000004, 0.53603400000000001, 0.492031, 0.496031, 0.49203000000000002, 0.51203200000000004, 0.496031, 0.492031, 0.500031, 0.492031, 0.50403100000000001, 0.50403200000000004, 0.496031, 0.492031, 0.49203000000000002, 0.50403200000000004, 0.492031, 0.492031, 0.51203200000000004, 0.49203000000000002, 0.49203200000000002, 0.50403100000000001, 0.492031, 0.49203000000000002, 0.50403200000000004, 0.48803000000000002, 0.496031, 0.50403100000000001, 0.48803099999999999, 0.492031, 0.50003200000000003, 0.49203000000000002, 0.492031, 0.496031, 0.492031, 0.492031, 0.500031, 0.496031, 0.492031, 0.500031, 0.492031, 0.48803099999999999, 0.50403100000000001, 0.48803099999999999, 0.492031, 0.500031], "base_times": [1.22, 1.2, 1.21, 1.21, 1.2, 1.21, 1.22, 1.2, 1.2, 1.22, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.22, 1.21, 1.21, 1.21, 1.2, 1.2, 1.22, 1.2, 1.21, 1.21, 1.2, 1.2, 1.21, 1.21, 1.2, 1.22, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.2, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21]}], ["twisted_names", "RawResult", {"changed_times": [0.0074571215510812828, 0.0072046109510086453, 0.0071581961345740875, 0.0076745970836531079, 0.007215007215007215, 0.007446016381236039, 0.0072674418604651162, 0.0073909830007390983, 0.007320644216691069, 0.007246376811594203, 0.007446016381236039, 0.0071684587813620072, 0.0073691967575534268, 0.0070372976776917661, 0.0071530758226037196, 0.0074239049740163323, 0.0070721357850070717, 0.0070077084793272598, 0.0074794315632011965, 0.007052186177715092, 0.0075815011372251705, 0.0074850299401197605, 0.0069979006298110571, 0.0071174377224199285, 0.0075187969924812026, 0.0072833211944646759, 0.007763975155279503, 0.006939625260235947, 0.006939625260235947, 0.007331378299120235, 0.006954102920723227, 0.006954102920723227, 0.006939625260235947, 0.0073421439060205578, 0.0069156293222683261, 0.0068917987594762234, 0.0073746312684365781, 0.0069444444444444441, 0.006920415224913495, 0.0073637702503681884, 0.0069492703266157054, 0.0069300069300069298, 0.0070571630204657732, 0.0069108500345542506, 0.0068917987594762234, 0.0069348127600554789, 0.0073046018991964941, 0.0069637883008356544, 0.006954102920723227, 0.0073583517292126564], "base_times": [0.0096246390760346481, 0.0095877277085330784, 0.0095419847328244278, 0.0095877277085330784, 0.0095602294455066923, 0.0095785440613026813, 0.0095693779904306216, 0.0095510983763132766, 0.0095419847328244278, 0.0099009900990099011, 0.0095693779904306216, 0.0095419847328244278, 0.0096061479346781949, 0.0095693779904306216, 0.0095510983763132766, 0.0095969289827255271, 0.0095785440613026813, 0.0095602294455066923, 0.0095785440613026813, 0.0095969289827255271, 0.0095785440613026813, 0.0096061479346781949, 0.0095602294455066923, 0.0095877277085330784, 0.0095877277085330784, 0.0095602294455066923, 0.0095877277085330784, 0.0095510983763132766, 0.0095328884652049577, 0.0095877277085330784, 0.0096061479346781949, 0.0095693779904306216, 0.0096061479346781949, 0.0095785440613026813, 0.0095693779904306216, 0.0095693779904306216, 0.0095785440613026813, 0.0095510983763132766, 0.0095877277085330784, 0.0095877277085330784, 0.0095785440613026813, 0.0095785440613026813, 0.0095969289827255271, 0.0095693779904306216, 0.0095877277085330784, 0.0095602294455066923, 0.0095419847328244278, 0.0095877277085330784, 0.0095785440613026813, 0.0095785440613026813]}]], "branch": "trunk", "revision": 0}
\ No newline at end of file
+{"options": "", "results": [["crypto_pyaes", "RawResult", {"changed_times": [0.82904410362199998, 0.356333971024, 0.27036023139999998, 0.26120996475199998, 0.26673913001999999, 0.27246308326699997, 0.25749182701099999, 0.25644612312300002, 0.26470899581899998, 0.29816889762900001, 0.27409410476700002, 0.25924706459000002, 0.26701903343200001, 0.256891012192, 0.26991915702800001, 0.33074879646299998, 0.26318502426099999, 0.26682686805700001, 0.25480794906600002, 0.258436918259, 0.25471806526200003, 0.27756595611599999, 0.27438092231799999, 0.25547194480899998, 0.29641008377099998, 0.286099910736, 0.25961518287699997, 0.27814412116999998, 0.258821964264, 0.255064964294, 0.28223800659199999, 0.31574010849, 0.27696394920299999, 0.25558805465700002, 0.25136613845799999, 0.25414109229999998, 0.25539398193399998, 0.25389981269799999, 0.26006698608399997, 0.27383089065600003, 0.28252100944500003, 0.28729081153899999, 0.254612207413, 0.25178408622699999, 0.247875213623, 0.248955011368, 0.25435090064999999, 0.277958154678, 0.25217485427899999, 0.27278900146500001], "base_times": [2.76247382164, 2.75780797005, 2.7526109218600001, 2.7581448554999999, 2.7579169273400002, 2.7591660022700002, 2.7544469833399998, 2.75570702553, 2.7566180229200001, 2.7799978256200002, 2.75436115265, 2.7584359645799998, 2.7715408801999999, 2.7583799362199999, 2.75467610359, 2.7834770679499998, 2.7558917999300001, 2.7597181797000001, 2.75486803055, 2.7581379413599998, 2.7532849311800001, 2.7636089324999999, 2.75440192223, 2.7553980350499998, 2.7562680244400002, 2.7588288784000001, 2.75431799889, 2.7562849521600001, 2.7566390037500001, 2.75684499741, 2.7588860988600001, 2.7639617919899999, 2.758425951, 2.7611510753599999, 2.75679397583, 2.7544021606400002, 2.7587559223200002, 2.7592377662700001, 2.7555589675899999, 2.7590520381900001, 2.75563788414, 2.76086902618, 2.7588081359899999, 2.7565140724199999, 2.75627803802, 2.76230502129, 2.7579100132000001, 2.75721096992, 2.7558379173300001, 2.7606220245399999]}], ["django", "RawResult", {"changed_times": [0.30002903938300002, 0.25580286979700001, 0.261374950409, 0.26095104217499998, 0.25220704078700001, 0.26073884963999999, 0.26132106780999997, 0.254949092865, 0.26201701164199998, 0.26217198371900002, 0.25196504592899999, 0.26127791404700001, 0.26208996772799997, 0.254974842072, 0.261093854904, 0.261404037476, 0.25236296653700002, 0.26157402992200002, 0.26323699951200003, 0.25301718711900001, 0.26132202148400002, 0.26150584220899997, 0.25210714340200002, 0.26118707656899998, 0.26585507392899999, 0.25383305549599999, 0.26143312454200002, 0.26261687278700002, 0.267751932144, 0.26117897033699999, 0.264419078827, 0.252134799957, 0.26171708107000002, 0.26141405105600002, 0.25201201438900001, 0.261445045471, 0.26490807533299998, 0.25207686424300002, 0.26115393638599999, 0.26161694526700002, 0.25278902053800001, 0.261168003082, 0.26426982879599997, 0.25209307670600001, 0.261046171188, 0.26155900955200001, 0.25206208229100002, 0.261226177216, 0.26494908332799999, 0.252925872803], "base_times": [0.995049953461, 0.99419713020300005, 0.99113988876299997, 0.99470710754400005, 0.99212503433200006, 0.99265909194900004, 0.99358582496600001, 0.99400091171299998, 0.99305796623200004, 0.99362897872900002, 0.99429702758799998, 0.99161601066600003, 0.99425101280200001, 0.99299907684300004, 0.99139285087600004, 0.99515914917000003, 0.99177312850999999, 0.99377703666700001, 0.99326705932600001, 0.99304509162900001, 0.99348807334900002, 0.99273705482499996, 0.99258708953899999, 0.99477910995499996, 0.99403500556900004, 0.99051785469099995, 0.994111061096, 0.99377894401599998, 0.99141192436199999, 0.99447512626599999, 0.99370288848900001, 0.99225115776100004, 0.99282193183900003, 0.99233508110000002, 0.99293303489700002, 0.99670696258500002, 0.99345684051500005, 0.99477696418799999, 0.99382996559100001, 0.99122619628899999, 0.99395608901999999, 0.99324393272400002, 0.99102210998499995, 0.99499702453600003, 0.99375295639000005, 0.99156498909000002, 0.99277806282000003, 0.99424910545300005, 0.99084305763199998, 0.993386983871]}], ["fannkuch", "RawResult", {"changed_times": [0.52755904197699999, 0.47701597213699998, 0.46741414070100001, 0.47383809089700002, 0.46101403236400001, 0.46737408638, 0.47243213653600002, 0.47586488723800002, 0.46479701995799999, 0.46248412132299999, 0.46604609489400001, 0.46004605293299999, 0.46376204490700002, 0.46708703041100003, 0.47270798683199999, 0.472360134125, 0.47210407257100001, 0.47691607475300002, 0.47826886177099998, 0.47204399108900003, 0.47201299667399998, 0.46304917335500001, 0.46217513084400003, 0.46784305572500001, 0.460357189178, 0.46357607841499998, 0.46536898613, 0.46894693374599999, 0.46510815620399998, 0.46110606193499998, 0.46555995941200001, 0.46238088607799999, 0.46605896949800002, 0.46549701690700002, 0.45882105827300002, 0.45827507972699999, 0.473033905029, 0.47556495666499998, 0.460822105408, 0.47130393981899998, 0.476540803909, 0.46522593498199999, 0.46266603469799999, 0.47138500213599999, 0.46661305427600003, 0.46587181091300001, 0.47512102127099998, 0.47284221649199998, 0.47128200530999997, 0.47180795669600001], "base_times": [1.9883148670199999, 1.9792110919999999, 1.9909551143599999, 1.98928499222, 1.97898578644, 1.98664617538, 1.9864010810899999, 1.98417901993, 1.9872028827699999, 1.98194694519, 1.9878189563799999, 1.98948693275, 1.9811220169099999, 1.9890899658200001, 1.9894771575900001, 1.9801352024100001, 1.9894909858700001, 1.9891271591199999, 1.98107600212, 1.99568986893, 1.9895131587999999, 1.9803609847999999, 2.0069239139600001, 1.9916350841499999, 1.98022890091, 1.98729896545, 1.99073004723, 1.98082304001, 1.98834180832, 1.9909420013400001, 1.9793639183, 1.98839306831, 1.9902248382600001, 1.9822490215299999, 1.98810195923, 1.99047279358, 1.98221516609, 1.98903012276, 1.9900078773500001, 1.9820079803499999, 1.9892179966000001, 1.9895989894899999, 1.9800350665999999, 1.9895751476300001, 1.9891831874799999, 1.9804949760399999, 1.9896249771100001, 1.9898300170900001, 1.9807391166699999, 1.9895370006599999]}], ["go", "RawResult", {"changed_times": [0.46677207946799998, 0.51466798782300005, 0.388464927673, 0.50362396240200002, 0.36921596527099998, 0.41929197311400002, 0.37606406211900001, 0.428333997726, 0.36696696281399999, 0.33521819114700002, 0.35271787643399999, 0.47167301177999998, 0.36826205253599997, 0.33157110214199997, 0.39287805557299998, 0.30100107192999997, 0.31953597068799999, 0.34765505790700002, 0.31619405746500001, 0.32286810874900002, 0.37090992927600003, 0.328006982803, 0.31113696098299998, 0.355150938034, 0.30218696594200001, 0.31915521621699999, 0.31008601188700002, 0.34765815734900002, 0.30721187591600002, 0.32940602302600003, 0.30662703514099998, 0.33826303482100001, 0.29518413543700001, 0.32855200767499998, 0.37890100479099997, 0.295953989029, 0.28935694694500003, 0.32658910751300002, 0.31018304824800003, 0.32643103599500001, 0.28621220588700003, 0.29142880439800001, 0.28649687766999998, 0.301717042923, 0.288222074509, 0.353245973587, 0.33032798767100002, 0.285399913788, 0.29600501060500001, 0.28842616081200001], "base_times": [0.94673204422000001, 0.94645810127300001, 0.94507908821099995, 0.944802045822, 0.941561937332, 0.944491147995, 0.94291400909400003, 0.95224094390900005, 0.94410514831500003, 0.943855047226, 0.94802284240699997, 0.94546699523900002, 0.94773101806600002, 0.94944882392899999, 0.94337701797499995, 0.94636011123700003, 0.94657802581799999, 0.94187402725199998, 0.94828701019299999, 0.94231510162400001, 0.94628596305799995, 0.94881415367099997, 0.94279098510699999, 0.94889903068500003, 0.95280098915099998, 0.94382286071800003, 0.94861888885500001, 0.94221186637900001, 0.94600486755400004, 0.94160294532800004, 0.94753003120400003, 0.94430899620099995, 0.94843101501500005, 0.96007490158099995, 0.94707822799700003, 0.94710588455199995, 0.95057606697100006, 0.94280695915199997, 0.94800686836199999, 0.94509601593000003, 0.96038508415199997, 0.94446587562600004, 0.94845819473299997, 0.94734406471300003, 0.94510412216200002, 0.94387602806100002, 0.94524908065799995, 0.94992303848299997, 0.94564318656900004, 0.94311594963099998]}], ["html5lib", "RawResult", {"changed_times": [15.457784891099999, 11.3277301788, 10.600764036199999, 10.4216821194, 10.2325599194, 10.2733259201, 10.3756709099, 10.156955957399999, 10.101744890200001, 10.133589983, 10.206784963600001, 10.1694169044, 10.0392251015, 10.155544042600001, 10.1885080338, 10.058653831499999, 10.2810080051, 10.2360329628, 10.039163112600001, 10.1998898983, 10.119593143499999, 10.1641438007, 10.0413920879, 10.107867002500001, 10.135586977000001, 10.1675360203, 10.1493079662, 10.060842990899999, 10.178154945399999, 10.1074330807, 10.170939922300001, 10.047384023699999, 10.0127818584, 10.050449132900001, 10.161669015899999, 10.037492990500001, 10.0900728703, 10.190579891200001, 9.9983761310600006, 10.1199560165, 10.003987073899999, 10.119729042099999, 10.1512379646, 10.054825067499999, 9.9258110523200003, 10.0976631641, 10.0940468311, 10.122640132900001, 9.9850409030900007, 10.099215984300001], "base_times": [13.800020933200001, 13.988759994500001, 13.945523977300001, 13.9515860081, 13.8968560696, 14.0439591408, 14.0257151127, 13.9940800667, 13.977200985, 13.965268135100001, 13.8857018948, 14.0525372028, 14.022065877899999, 14.0086979866, 13.961042880999999, 13.921565771099999, 13.924687862400001, 14.0379269123, 14.256644964199999, 13.9928328991, 14.0013000965, 13.9456880093, 13.914921999000001, 14.061727047, 14.0501580238, 14.0101730824, 13.9787597656, 13.921334981899999, 13.889928102500001, 14.0468928814, 14.0285110474, 14.0453250408, 13.9559979439, 13.9637241364, 13.9317610264, 14.0579519272, 14.0324690342, 14.0096049309, 13.9675040245, 13.964776992799999, 13.924336910199999, 14.0519120693, 14.0543570518, 14.0083789825, 13.9841370583, 13.954766988799999, 13.9033069611, 13.8949439526, 14.038803100599999, 14.0201179981]}], ["meteor-contest", "RawResult", {"changed_times": [0.47674798965499998, 0.403456926346, 0.39809894561800002, 0.40778803825400001, 0.39824509620699999, 0.39766597747799998, 0.40741491317700002, 0.39641904830899999, 0.39668798446699999, 0.39379596710199999, 0.39647078514099998, 0.393444061279, 0.40611696243299999, 0.39961814880399998, 0.39285993576, 0.39224314689599998, 0.39388489723199999, 0.39147710800199997, 0.39165306091300001, 0.39033508300800002, 0.39437389373800003, 0.39175200462300003, 0.391242027283, 0.39103412628200002, 0.39511799812300002, 0.398463010788, 0.39204406738300002, 0.39300107955899999, 0.39334487914999999, 0.391426801682, 0.39687800407399998, 0.39431095123299997, 0.39070391654999997, 0.39106798172000001, 0.39058089256299999, 0.39391088485699999, 0.39834094047500002, 0.39057397842399999, 0.39198994636500001, 0.39368200302099998, 0.39073801040599998, 0.39122891426099998, 0.39064598083500002, 0.39390277862500001, 0.390840053558, 0.39128398895299998, 0.39098405838, 0.39335107803300001, 0.39786505699199998, 0.39080500602700002], "base_times": [0.348359107971, 0.34692883491499998, 0.346405982971, 0.34667205810500001, 0.34701514244100001, 0.34753990173299998, 0.346569061279, 0.34635281562800002, 0.346190929413, 0.34796619415300001, 0.34590101242100002, 0.34643101692200001, 0.34670686721799998, 0.34775090217600002, 0.34620094299300003, 0.34622883796699999, 0.34623694419899997, 0.34779596328700002, 0.348699092865, 0.34642219543500002, 0.347030878067, 0.34581995010400002, 0.347792863846, 0.34766483306899998, 0.34653806686400002, 0.34683513641399999, 0.34936118125900001, 0.34839010238599999, 0.346399068832, 0.34670805931100002, 0.34576678276099998, 0.348170042038, 0.34661507606500003, 0.34650683402999999, 0.34621381759600001, 0.34878396987900001, 0.34638595581100001, 0.34623980522199999, 0.34593510627700003, 0.34620404243500003, 0.34790611267100002, 0.346742153168, 0.34607505798299998, 0.34633517265300001, 0.34812307357799999, 0.34604907035799998, 0.34677386283900002, 0.34667706489599998, 0.34659886360199998, 0.34881806373599999]}], ["nbody_modified", "RawResult", {"changed_times": [0.12817502021800001, 0.114922046661, 0.105006933212, 0.10339498519900001, 0.104254007339, 0.103782892227, 0.103751897812, 0.105894088745, 0.104180812836, 0.103867053986, 0.103599071503, 0.103564023972, 0.103793859482, 0.103470087051, 0.103558063507, 0.103418111801, 0.10340905189500001, 0.10347700119, 0.103801012039, 0.10350394249, 0.103738069534, 0.10368895530699999, 0.106331110001, 0.103569984436, 0.10441899299600001, 0.103644132614, 0.103709936142, 0.10339307785, 0.103735923767, 0.10362410545300001, 0.103760004044, 0.103440999985, 0.10381603240999999, 0.103622913361, 0.10393595695500001, 0.103415966034, 0.103775978088, 0.106086969376, 0.103862047195, 0.103460073471, 0.103792905807, 0.10356497764600001, 0.103854894638, 0.103960990906, 0.103703975677, 0.103446006775, 0.10406708717300001, 0.103525876999, 0.103811979294, 0.10510087013199999], "base_times": [0.64283800125099999, 0.64096808433499997, 0.64577603340099998, 0.64423298835800002, 0.64046311378499998, 0.63754677772500001, 0.63747310638400001, 0.63725614547700005, 0.63658809661899995, 0.64228487014799995, 0.64188194274900001, 0.64234304428099998, 0.63921594619800004, 0.63971495628399999, 0.64251685142500004, 0.64014887809800003, 0.64212799072299998, 0.64027500152600003, 0.63313889503499998, 0.63153290748599999, 0.63076686859099995, 0.63370203971899997, 0.63236403465299995, 0.63347291946399997, 0.63246107101399995, 0.63299584388700003, 0.63348007202099998, 0.63026380538899995, 0.632724046707, 0.63243818283099995, 0.63531398773199999, 0.63819217681899998, 0.63529610633900002, 0.63803815841699996, 0.63671016693100002, 0.64081311225899995, 0.63594484329199996, 0.64017796516400005, 0.64193677902199997, 0.64040708541900004, 0.64205718040500004, 0.64145493507399998, 0.640824079514, 0.65274095535300003, 0.64189887046799998, 0.64417505264300001, 0.64191007614100004, 0.640285015106, 0.63908886909499996, 0.64495897293100002]}], ["pyflate-fast", "RawResult", {"changed_times": [1.83701682091, 1.8073658943199999, 1.8406269550300001, 1.82018709183, 1.8337500095399999, 1.8074929714200001, 1.84810090065, 1.8087329864499999, 1.8338890075700001, 1.81683397293, 1.78763699532, 1.79593896866, 1.81039500237, 1.7986569404599999, 1.8007159233100001, 1.8014991283399999, 1.7972960472099999, 1.80028700829, 1.8007249832200001, 1.85811805725, 1.7931230068199999, 1.82364487648, 1.7935469150500001, 1.82844901085, 1.7920200824700001, 1.82318806648, 1.7929878234900001, 1.82861304283, 1.79402518272, 1.82314896584, 1.79180502892, 1.8225588798500001, 1.80868387222, 1.80998492241, 1.8257219791399999, 1.8084599971799999, 1.81027412415, 1.80907607079, 1.80926918983, 1.82390213013, 1.8229339122799999, 1.9030389785799999, 1.9522318839999999, 1.8275589942899999, 1.81771707535, 1.8227140903500001, 1.8171408176399999, 1.8353171348599999, 1.8287210464500001, 1.83105301857], "base_times": [3.1601369381, 3.17563796043, 3.1705720424699999, 3.17048382759, 3.1654319763199998, 3.1680698394800002, 3.1681051254299999, 3.1718590259599999, 3.1830987930300001, 3.1775770187400001, 3.1700208187099999, 3.1612432002999999, 3.15727186203, 3.1732730865500001, 3.1751198768600002, 3.1624131202700001, 3.1771841049199998, 3.1661839485200001, 3.16439390182, 3.1774570941900002, 3.1796469688400002, 3.16518807411, 3.1729710102099999, 3.1790311336500001, 3.17372894287, 3.1641249656700001, 3.1653139591200001, 3.16444206238, 3.1757740974400002, 3.16623592377, 3.1762218475299999, 3.1651339530899998, 3.1656432151799998, 3.1724681854200001, 3.1735718250299998, 3.1667320728299999, 3.1666581630700001, 3.1662278175399998, 3.1711161136600001, 3.1634638309500001, 3.1635270118699998, 3.1630849838300001, 3.1776990890499999, 3.17032909393, 3.1747660636899999, 3.1702299117999999, 3.17133522034, 3.1701278686499998, 3.1603660583500002, 3.1653149128]}], ["raytrace-simple", "RawResult", {"changed_times": [0.659919023514, 0.66626596450800002, 0.65816497802700002, 0.65398693084699999, 0.63991713523899996, 0.66545104980500003, 0.65175795555100002, 0.63982892036399996, 0.66220998764000005, 0.65392279624899996, 0.63932204246500002, 0.65032505989099998, 0.65231609344499997, 0.63614416122399997, 0.65278100967399999, 0.63543891906700001, 0.65301513671900002, 0.647172212601, 0.63230013847400002, 0.64931488037100005, 0.689571857452, 0.63732600212099999, 0.67306900024399996, 0.63961911201499999, 0.650694131851, 0.63388586044299999, 0.65063095092800005, 0.64781498909000002, 0.63850116729700002, 0.65920114517200001, 0.68630504608200005, 0.63646793365499998, 0.64802885055500004, 0.64920401573200004, 0.63225889205899999, 0.65038490295399998, 0.63108992576599998, 0.64513182640099997, 0.63317394256600001, 0.64602303504900005, 0.64737510681199995, 0.63001990318300005, 0.64657092094400004, 0.63210487365699997, 0.645840883255, 0.65991592407199995, 0.62948513030999997, 0.65090513229400004, 0.63588213920600001, 0.66070103645300005], "base_times": [2.6739389896399999, 2.6792252063799999, 2.67714691162, 2.6755440235100001, 2.6839830875400001, 2.67896699905, 2.6759178638500001, 2.6771140098599999, 2.6831061840100001, 2.6781611442600002, 2.6843791008000002, 2.67555809021, 2.67742395401, 2.68145084381, 2.6856698989900001, 2.6754090786, 2.6820809841200002, 2.7008218765300001, 2.6860649585699998, 2.68976688385, 2.7091708183300001, 2.6831979751600001, 2.6825587749499999, 2.7229101657900001, 2.7227280139899999, 2.7074718475299999, 2.7032160758999999, 2.69513916969, 2.7023360729200001, 2.7511570453599998, 2.6899399757400002, 2.6838908195500002, 2.7002210617100002, 2.8071360588099998, 2.6863989830000001, 2.6864531040199999, 2.69359779358, 2.7453310489699998, 2.7076761722599998, 3.06150817871, 2.7332241535200001, 2.7483820915199999, 2.7639210224199999, 3.4124541282699998, 2.7048859596299999, 2.6892578601800001, 2.6912610530899999, 2.6821250915500001, 2.68154597282, 2.6890358924900002]}], ["richards", "RawResult", {"changed_times": [0.12052297592199999, 0.093158006668099994, 0.091630935668900002, 0.091804027557399998, 0.091670036315899997, 0.091012954711900004, 0.091180086135899993, 0.102787017822, 0.090846061706500003, 0.090148925781199998, 0.093178987503099997, 0.090260028839100004, 0.090500116348300005, 0.098414897918700006, 0.090541839599599994, 0.090975999832200005, 0.090515136718800002, 0.088860034942600002, 0.089382886886599996, 0.087617158889799998, 0.087574005127000004, 0.088225841522200005, 0.087996006012, 0.087512969970700002, 0.0879878997803, 0.096507072448699993, 0.087088108062699995, 0.090749025344800005, 0.087268829345700002, 0.087903976440400003, 0.087549924850500005, 0.087819099426300007, 0.087373971939100001, 0.087810039520300004, 0.087095975875899997, 0.0871419906616, 0.088514089584400005, 0.087375879287700006, 0.086944818496699994, 0.087454080581699997, 0.087461948394799996, 0.097897052764900003, 0.087609052658099998, 0.087203979492200004, 0.089391946792599999, 0.088086128234900002, 0.086940050125100005, 0.087315082550000006, 0.086781978607199994, 0.086982011794999994], "base_times": [0.35359501838700003, 0.35667300224300003, 0.347419977188, 0.35246205329899999, 0.35532999038700003, 0.35530591011000001, 0.35107088088999999, 0.35549902915999998, 0.35110902786300002, 0.35400104522699999, 0.35261011123699998, 0.35541200637800002, 0.35351705551099999, 0.35617089271500002, 0.34958410263099998, 0.35414409637499999, 0.35661983489999999, 0.35693001747100001, 0.35174703598000001, 0.34876704215999998, 0.35491299629200002, 0.35479497909500002, 0.35416102409400002, 0.355372905731, 0.35764288902300001, 0.35487508773799997, 0.35283422470100001, 0.35428977012599999, 0.35132908821100001, 0.35751414299000001, 0.362582206726, 0.35600781440700002, 0.35644507408100001, 0.35086202621500001, 0.35400605201699997, 0.34989190101599998, 0.35055398940999999, 0.35048794746400003, 0.35601019859299998, 0.35249996185299998, 0.35182499885599999, 0.35495114326499999, 0.35285496711699998, 0.355787038803, 0.35039997100800002, 0.35504388809199999, 0.35427594184900002, 0.35611510276800001, 0.35387182235699999, 0.34967398643499997]}], ["spambayes", "RawResult", {"changed_times": [0.48457980155899999, 0.56999397277800001, 0.53518199920700005, 0.54824614524799997, 0.49303603172299998, 0.58462905883799998, 0.51067399978600003, 0.49086999893200001, 0.48435497283899998, 0.42737793922400003, 0.494954109192, 0.43986797332799998, 0.40522003173799997, 0.40721011161800003, 0.57795190811200003, 0.48255205154399999, 0.462779998779, 0.44051218032799999, 0.39301896095299999, 0.39432501792899999, 0.39888882637, 0.43233299255399998, 0.42077803611800002, 0.39932012558000002, 0.40041995048500001, 0.409831047058, 0.44888401031500003, 0.40377283096299998, 0.39395999908399998, 0.40184497833299998, 0.41734480857799999, 0.49758601188700002, 0.39052391052199997, 0.392160177231, 0.39913892746000001, 0.42745113372799998, 0.36878705024699998, 0.38831090927099998, 0.37554693222000002, 0.40881800651599998, 0.37994885444600002, 0.37922286987300002, 0.39221501350400001, 0.39279913902300001, 0.34234404563900001, 0.37785792350800002, 0.38006091117899998, 0.37980985641499998, 0.375820875168, 0.345314979553], "base_times": [0.29905700683600001, 0.298835992813, 0.29825806617700001, 0.29831504821799998, 0.30105614662199998, 0.298439979553, 0.29832100868200001, 0.29835200309799997, 0.29867696762099999, 0.29994201660199998, 0.29847502708399998, 0.29867196083100001, 0.29891991615300001, 0.29807305336000001, 0.30059790611300002, 0.298420906067, 0.29828691482500003, 0.29897713661199998, 0.29811811447100001, 0.300410985947, 0.29847979545600001, 0.2982878685, 0.29861903190599998, 0.29877710342399999, 0.30105996131899998, 0.29816794395399998, 0.29835987091100002, 0.29860401153600002, 0.2985060215, 0.30007410049400002, 0.29955601692200001, 0.299430847168, 0.29854393005399998, 0.29915118217499997, 0.29966092109699999, 0.29980802536000001, 0.29794311523400002, 0.29930615425099999, 0.298702001572, 0.29842782020600001, 0.30247402191200001, 0.29896807670600001, 0.298580169678, 0.29869008064300001, 0.299312114716, 0.30094408988999999, 0.29845499992399999, 0.29868721962, 0.29998612403899999, 0.29916310310400002]}], ["spectral-norm", "RawResult", {"changed_times": [0.18530607223500001, 0.118788957596, 0.11839389801, 0.117673158646, 0.115185976028, 0.115034103394, 0.114280939102, 0.115108966827, 0.115477085114, 0.114944934845, 0.11555695533800001, 0.111644983292, 0.111333847046, 0.111480951309, 0.11155700683600001, 0.11224794387799999, 0.112046003342, 0.111447095871, 0.11138176918000001, 0.11152100563, 0.111184120178, 0.11187982559200001, 0.111309051514, 0.115062952042, 0.111434936523, 0.111573934555, 0.111452102661, 0.128764867783, 0.11133313179, 0.11140704155, 0.11155796051, 0.111167907715, 0.111421108246, 0.11097002029399999, 0.111446857452, 0.110933065414, 0.110732078552, 0.113990068436, 0.11116909980799999, 0.111049890518, 0.11179399490399999, 0.11199402809099999, 0.110781908035, 0.111202001572, 0.111080884933, 0.110934972763, 0.111124992371, 0.110868930817, 0.11123108863800001, 0.112289190292], "base_times": [0.48088502883899997, 0.47923994064300002, 0.48160886764499999, 0.47763085365300001, 0.47884607315099997, 0.48438310623199998, 0.48113894462599999, 0.47794389724699998, 0.47896790504499998, 0.48052501678499998, 0.47819399833699999, 0.47891688346900002, 0.48100805282600001, 0.48024415969799999, 0.47854995727499999, 0.47899198532100001, 0.47972202301, 0.47723197937, 0.48147606849699998, 0.47865796089200002, 0.47799110412599999, 0.47740817070000002, 0.47738909721400002, 0.47732496261599999, 0.47762703895600001, 0.48322010040300001, 0.47945380210900002, 0.47645592689499999, 0.479950904846, 0.47760391235400002, 0.47802209854099997, 0.47923922538800001, 0.477582931519, 0.47682905197100001, 0.47848796844500002, 0.47821402549699998, 0.47620606422400003, 0.48374009132399998, 0.481714963913, 0.47922301292399999, 0.47735190391499999, 0.48569798469499997, 0.47899794578600002, 0.477490901947, 0.47891783714300001, 0.47670793533299999, 0.47655487060500001, 0.47783589363099999, 0.47823095321699999, 0.47739887237500001]}], ["telco", "RawResult", {"changed_times": [0.56803599999999999, 0.51203200000000004, 0.51603200000000005, 0.54403400000000002, 0.51203200000000004, 0.53603400000000001, 0.492031, 0.496031, 0.49203000000000002, 0.51203200000000004, 0.496031, 0.492031, 0.500031, 0.492031, 0.50403100000000001, 0.50403200000000004, 0.496031, 0.492031, 0.49203000000000002, 0.50403200000000004, 0.492031, 0.492031, 0.51203200000000004, 0.49203000000000002, 0.49203200000000002, 0.50403100000000001, 0.492031, 0.49203000000000002, 0.50403200000000004, 0.48803000000000002, 0.496031, 0.50403100000000001, 0.48803099999999999, 0.492031, 0.50003200000000003, 0.49203000000000002, 0.492031, 0.496031, 0.492031, 0.492031, 0.500031, 0.496031, 0.492031, 0.500031, 0.492031, 0.48803099999999999, 0.50403100000000001, 0.48803099999999999, 0.492031, 0.500031], "base_times": [1.22, 1.2, 1.21, 1.21, 1.2, 1.21, 1.22, 1.2, 1.2, 1.22, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21, 1.2, 1.21, 1.22, 1.21, 1.21, 1.21, 1.2, 1.2, 1.22, 1.2, 1.21, 1.21, 1.2, 1.2, 1.21, 1.21, 1.2, 1.22, 1.21, 1.21, 1.2, 1.21, 1.21, 1.21, 1.21, 1.2, 1.2, 1.21, 1.21, 1.21, 1.21, 1.21, 1.21]}], ["twisted_names", "RawResult", {"changed_times": [0.0074571215510812828, 0.0072046109510086453, 0.0071581961345740875, 0.0076745970836531079, 0.007215007215007215, 0.007446016381236039, 0.0072674418604651162, 0.0073909830007390983, 0.007320644216691069, 0.007246376811594203, 0.007446016381236039, 0.0071684587813620072, 0.0073691967575534268, 0.0070372976776917661, 0.0071530758226037196, 0.0074239049740163323, 0.0070721357850070717, 0.0070077084793272598, 0.0074794315632011965, 0.007052186177715092, 0.0075815011372251705, 0.0074850299401197605, 0.0069979006298110571, 0.0071174377224199285, 0.0075187969924812026, 0.0072833211944646759, 0.007763975155279503, 0.006939625260235947, 0.006939625260235947, 0.007331378299120235, 0.006954102920723227, 0.006954102920723227, 0.006939625260235947, 0.0073421439060205578, 0.0069156293222683261, 0.0068917987594762234, 0.0073746312684365781, 0.0069444444444444441, 0.006920415224913495, 0.0073637702503681884, 0.0069492703266157054, 0.0069300069300069298, 0.0070571630204657732, 0.0069108500345542506, 0.0068917987594762234, 0.0069348127600554789, 0.0073046018991964941, 0.0069637883008356544, 0.006954102920723227, 0.0073583517292126564], "base_times": [0.0096246390760346481, 0.0095877277085330784, 0.0095419847328244278, 0.0095877277085330784, 0.0095602294455066923, 0.0095785440613026813, 0.0095693779904306216, 0.0095510983763132766, 0.0095419847328244278, 0.0099009900990099011, 0.0095693779904306216, 0.0095419847328244278, 0.0096061479346781949, 0.0095693779904306216, 0.0095510983763132766, 0.0095969289827255271, 0.0095785440613026813, 0.0095602294455066923, 0.0095785440613026813, 0.0095969289827255271, 0.0095785440613026813, 0.0096061479346781949, 0.0095602294455066923, 0.0095877277085330784, 0.0095877277085330784, 0.0095602294455066923, 0.0095877277085330784, 0.0095510983763132766, 0.0095328884652049577, 0.0095877277085330784, 0.0096061479346781949, 0.0095693779904306216, 0.0096061479346781949, 0.0095785440613026813, 0.0095693779904306216, 0.0095693779904306216, 0.0095785440613026813, 0.0095510983763132766, 0.0095877277085330784, 0.0095877277085330784, 0.0095785440613026813, 0.0095785440613026813, 0.0095969289827255271, 0.0095693779904306216, 0.0095877277085330784, 0.0095602294455066923, 0.0095419847328244278, 0.0095877277085330784, 0.0095785440613026813, 0.0095785440613026813]}]], "branch": "trunk", "revision": 0}


From cfbolz at codespeak.net  Wed Oct 13 22:32:07 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Wed, 13 Oct 2010 22:32:07 +0200 (CEST)
Subject: [pypy-svn] r77893 - pypy/extradoc/talk/pepm2011
Message-ID: <20101013203207.C9ABA282BDB@codespeak.net>

Author: cfbolz
Date: Wed Oct 13 22:32:06 2010
New Revision: 77893

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
re-import benchmark times, fix one XXX


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Wed Oct 13 22:32:06 2010
@@ -989,6 +989,7 @@
 removed guards are an effect of the optimization described here.
 
 \begin{figure*}
+\begin{center}
 \begin{tabular}{|l||r|rr|rr|rr|rr|}
 \hline
 	&num loops	&new	&removed	&get/set	&removed &guard &removed	&all ops	&removed\\
@@ -1011,6 +1012,7 @@
 total	&2240	&49709	&70\%	&1227447	&14\%	&222569	&93\%	&3395548	&89\%\\
 \hline
 \end{tabular}
+\end{center}
 \caption{Number of Operations and Percentage Removed By Optimization}
 \label{fig:numops}
 \end{figure*}
@@ -1022,29 +1024,35 @@
 confidence level \cite{georges_statistically_2007}. The results are reported in Figure~\ref{fig:times}.
 With the optimization turned on, PyPy's Python interpreter outperforms CPython
 in all benchmarks except spambayes (which heavily relies on regular expression
-performance XXX). All benchmarks are improved by the allocation removal
-optimization, some as much as XXX. XXX Psyco
+performance and thus is not helped much by our Python JIT) and meteor-contest.
+All benchmarks are improved by the allocation removal optimization, some as much
+as XXX. XXX Psyco
 
 XXX runtimes of the algorithm somehow?
 
 \begin{figure*}
-\begin{tabular}{lrrrr}
-		&CPython	&JIT no optimizations	&JIT allocation removal	&JIT full\\
-crypto\_pyaes	&1958.40 $\pm$ 0.32	&782.14 $\pm$ 3.08	&150.86 $\pm$ 3.62	&125.68 $\pm$ 3.02\\
-django		&630.91 $\pm$ 0.13	&365.90 $\pm$ 1.57	&124.41 $\pm$ 0.88	&117.36 $\pm$ 0.91\\
-fannkuch	&1344.15 $\pm$ 0.36	&378.18 $\pm$ 0.36	&309.04 $\pm$ 1.56	&299.12 $\pm$ 0.25\\
-go		&595.31 $\pm$ 0.51	&1067.26 $\pm$ 12.20	&121.76 $\pm$ 4.15	&125.57 $\pm$ 3.90\\
-html5lib	&8678.67 $\pm$ 23.37	&17665.16 $\pm$ 2695.45	&6739.95 $\pm$ 1225.25	&6699.24 $\pm$ 1295.92\\
-meteor-contest	&241.53 $\pm$ 0.09	&291.11 $\pm$ 0.86	&274.09 $\pm$ 0.37	&272.77 $\pm$ 0.30\\
-nbody		&396.55 $\pm$ 0.24	&84.77 $\pm$ 0.14	&69.01 $\pm$ 0.08	&68.71 $\pm$ 0.09\\
-pyflate-fast	&1991.33 $\pm$ 1.90	&1818.70 $\pm$ 4.30	&1062.03 $\pm$ 4.56	&1052.02 $\pm$ 3.24\\
-raytrace-simple	&1598.84 $\pm$ 1.07	&1126.93 $\pm$ 11.42	&93.45 $\pm$ 4.75	&92.17 $\pm$ 4.15\\
-richards	&216.47 $\pm$ 0.20	&169.84 $\pm$ 1.21	&11.81 $\pm$ 0.09	&11.90 $\pm$ 0.29\\
-spambayes	&193.30 $\pm$ 0.03	&331.37 $\pm$ 14.93	&238.83 $\pm$ 6.10	&237.23 $\pm$ 5.97\\
-spectral-norm	&375.91 $\pm$ 0.08	&221.24 $\pm$ 0.07	&31.96 $\pm$ 0.06	&28.79 $\pm$ 0.09\\
-telco		&789.67 $\pm$ 1.14	&622.67 $\pm$ 1.86	&155.33 $\pm$ 1.82	&154.67 $\pm$ 1.82\\
-twisted\_names	&6.61 $\pm$ 0.00	&8.85 $\pm$ 0.09	&4.35 $\pm$ 0.05 &4.35 $\pm$ 0.04\\
+\begin{center}
+\begin{tabular}{|l||r|r|r|r|}
+\hline
+	&Cpython	&Psyco	&PyPy w/o optimizations	&PyPy w/ optimizations \\
+\hline
+crypto\_pyaes	&2757.80 $\pm$ 0.98	&67.90 $\pm$ 0.47	&1652.00 $\pm$ 4.00 &266.86 $\pm$ 5.94 \\
+django	&993.19 $\pm$ 0.50	&913.51 $\pm$ 4.22	&694.73 $\pm$ 2.86 &259.53 $\pm$ 1.79 \\
+fannkuch	&1987.22 $\pm$ 2.02	&944.44 $\pm$ 0.61	&566.99 $\pm$ 1.06 &466.87 $\pm$ 1.85 \\
+go	&947.21 $\pm$ 1.58	&445.96 $\pm$ 0.68	&2197.71 $\pm$ 25.21 &316.15 $\pm$ 9.33 \\
+html5lib	&13987.12 $\pm$ 19.51	&17398.25 $\pm$ 36.50	&27194.45 $\pm$ 46.62	&10092.19 $\pm$ 23.50 \\
+meteor-contest	&346.98 $\pm$ 0.35	&215.66 $\pm$ 0.23	&433.04 $\pm$ 1.45 &392.85 $\pm$ 0.87 \\
+nbody\_modified	&637.90 $\pm$ 1.82	&256.78 $\pm$ 0.18	&135.55 $\pm$ 0.33 &103.93 $\pm$ 0.25 \\
+pyflate-fast	&3169.35 $\pm$ 1.89	&1278.16 $\pm$ 3.13	&3285.89 $\pm$ 8.51	&1822.36 $\pm$ 11.52 \\
+raytrace-simple	&2744.60 $\pm$ 51.72	&1072.66 $\pm$ 1.08	&2778.27 $\pm$ 15.13	&647.24 $\pm$ 5.44 \\
+richards	&354.06 $\pm$ 1.00	&63.48 $\pm$ 0.15	&383.93 $\pm$ 3.28 &88.32 $\pm$ 0.91 \\
+spambayes	&299.16 $\pm$ 0.35	&338.68 $\pm$ 3.14	&580.90 $\pm$ 24.68 &397.37 $\pm$ 10.60 \\
+spectral-norm	&478.63 $\pm$ 0.80	&139.83 $\pm$ 1.54	&353.51 $\pm$ 1.39 &112.10 $\pm$ 1.17 \\
+telco	&1207.67 $\pm$ 2.03	&730.00 $\pm$ 2.66	&1296.08 $\pm$ 4.37 &495.23 $\pm$ 2.14 \\
+twisted\_names	&9.58 $\pm$ 0.01	&10.43 $\pm$ 0.01	&17.99 $\pm$ 0.27	&7.13 $\pm$ 0.09 \\
+\hline
 \end{tabular}
+\end{center}
 \caption{Benchmark Times}
 \label{fig:times}
 \end{figure*}


From cfbolz at codespeak.net  Wed Oct 13 22:41:51 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Wed, 13 Oct 2010 22:41:51 +0200 (CEST)
Subject: [pypy-svn] r77894 - pypy/extradoc/talk/pepm2011
Message-ID: <20101013204151.1A148282B9E@codespeak.net>

Author: cfbolz
Date: Wed Oct 13 22:41:48 2010
New Revision: 77894

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
give a unit


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Wed Oct 13 22:41:48 2010
@@ -1053,7 +1053,7 @@
 \hline
 \end{tabular}
 \end{center}
-\caption{Benchmark Times}
+\caption{Benchmark Times in Milliseconds}
 \label{fig:times}
 \end{figure*}
 


From afa at codespeak.net  Thu Oct 14 00:59:28 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 00:59:28 +0200 (CEST)
Subject: [pypy-svn] r77895 - in
	pypy/branch/fast-forward/pypy/module/_multiprocessing: . test
Message-ID: <20101013225928.DBAB4282B9E@codespeak.net>

Author: afa
Date: Thu Oct 14 00:59:27 2010
New Revision: 77895

Modified:
   pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py
   pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py
Log:
Test and fix with recursive mutex


Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py	Thu Oct 14 00:59:27 2010
@@ -354,6 +354,7 @@
 
     def semlock_release(self, space):
         if self.kind == RECURSIVE_MUTEX:
+            sem_post(self.handle)
             return
         if HAVE_BROKEN_SEM_GETVALUE:
             # We will only check properly the maxvalue == 1 case

Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py	Thu Oct 14 00:59:27 2010
@@ -7,6 +7,7 @@
         space = gettestobjspace(usemodules=('_multiprocessing', 'thread'))
         cls.space = space
         cls.w_SEMAPHORE = space.wrap(SEMAPHORE)
+        cls.w_RECURSIVE = space.wrap(RECURSIVE_MUTEX)
 
     def test_semaphore(self):
         from _multiprocessing import SemLock
@@ -31,6 +32,26 @@
         sem.release()
         assert sem._count() == 0
 
+    def test_recursive(self):
+        from _multiprocessing import SemLock
+        kind = self.RECURSIVE
+        value = 1
+        maxvalue = 1
+        sem = SemLock(kind, value, maxvalue)
+
+        sem.acquire()
+        sem.release()
+        assert sem._count() == 0
+        sem.acquire()
+        sem.release()
+
+        # now recursively
+        sem.acquire()
+        sem.acquire()
+        assert sem._count() == 2
+        sem.release()
+        sem.release()
+
     def test_semaphore_wait(self):
         from _multiprocessing import SemLock
         kind = self.SEMAPHORE


From afa at codespeak.net  Thu Oct 14 08:52:11 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 08:52:11 +0200 (CEST)
Subject: [pypy-svn] r77896 - in pypy/branch/fast-forward/pypy/objspace/std:
	. test
Message-ID: <20101014065211.3E58B282BE3@codespeak.net>

Author: afa
Date: Thu Oct 14 08:52:06 2010
New Revision: 77896

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
Log:
Enhance bytearray constructor


Modified: pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py	Thu Oct 14 08:52:06 2010
@@ -1,6 +1,7 @@
 import sys
 from pypy.interpreter import gateway
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
+from pypy.interpreter.error import OperationError
 from pypy.objspace.std.register_all import register_all
 from pypy.objspace.std.stdtypedef import StdTypeDef, SMM
 
@@ -14,15 +15,66 @@
     str_join, str_split, str_rsplit, str_partition, str_rpartition,
     str_splitlines)
 
- at gateway.unwrap_spec(ObjSpace, W_Root, W_Root, W_Root, W_Root)
-def descr__new__(space, w_bytearraytype,
-                 w_source='', w_encoding=None, w_errors=None):
+def _getbytevalue(space, w_value):
+    if space.isinstance_w(w_value, space.w_str):
+        string = space.str_w(w_value)
+        if len(string) != 1:
+            raise OperationError(space.w_ValueError, space.wrap(
+                "string must be of size 1"))
+        return string[0]
+
+    value = space.getindex_w(w_value, None)
+    if not 0 <= value < 256:
+        # this includes the OverflowError in case the long is too large
+        raise OperationError(space.w_ValueError, space.wrap(
+            "byte must be in range(0, 256)"))
+    return chr(value)
+
+def new_bytearray(space, w_bytearraytype, data):
     from pypy.objspace.std.bytearrayobject import W_BytearrayObject
-    data = [c for c in space.str_w(w_source)]
     w_obj = space.allocate_instance(W_BytearrayObject, w_bytearraytype)
     W_BytearrayObject.__init__(w_obj, data)
     return w_obj
 
+ at gateway.unwrap_spec(ObjSpace, W_Root, W_Root, W_Root, W_Root)
+def descr__new__(space, w_bytearraytype,
+                 w_source='', w_encoding=None, w_errors=None):
+
+    data = []
+    # String-like argument
+    try:
+        string = space.str_w(w_source)
+    except OperationError, e:
+        if not e.match(space, space.w_TypeError):
+            raise
+    else:
+        data = [c for c in string]
+        return new_bytearray(space, w_bytearraytype, data)
+
+    # Is it an int?
+    try:
+        count = space.int_w(w_source)
+    except OperationError, e:
+        if not e.match(space, space.w_TypeError):
+            raise
+    else:
+        data = ['\0'] * count
+        return new_bytearray(space, w_bytearraytype, data)
+
+    # sequence of bytes
+    w_iter = space.iter(w_source)
+    while True:
+        try:
+            w_item = space.next(w_iter)
+        except OperationError, e:
+            if not e.match(space, space.w_StopIteration):
+                raise
+            break
+        value = _getbytevalue(space, w_item)
+        data.append(value)
+
+    return new_bytearray(space, w_bytearraytype, data)
+
 # ____________________________________________________________
 
 bytearray_typedef = StdTypeDef("bytearray",

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	Thu Oct 14 08:52:06 2010
@@ -5,6 +5,16 @@
         assert type(b) is bytearray
         assert b.__class__ is bytearray
 
+    def test_constructor(self):
+        assert bytearray() == ""
+        assert bytearray('abc') == "abc"
+        assert bytearray(['a', 'b', 'c']) == "abc"
+        assert bytearray([65, 66, 67]) == "ABC"
+        assert bytearray(5) == '\0' * 5
+        raises(ValueError, bytearray, ['a', 'bc'])
+        raises(ValueError, bytearray, [65, -3])
+        raises(TypeError, bytearray, [65.0])
+
     def test_len(self):
         b = bytearray('test')
         assert len(b) == 4


From cfbolz at codespeak.net  Thu Oct 14 11:18:33 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 11:18:33 +0200 (CEST)
Subject: [pypy-svn] r77898 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014091833.1F507282BE3@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 11:18:31 2010
New Revision: 77898

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
two things from samuele, regen pdf


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 11:18:31 2010
@@ -469,8 +469,7 @@
     comparison (``greater than''), respectively.
 \end{itemize}
 
-Note how the functions that are called by \lstinline{f} are automatically inlined
-into the trace. The method calls are always preceded by a \lstinline{guard_class}
+The method calls in the trace are always preceded by a \lstinline{guard_class}
 operation, to check that the class of the receiver is the same as the one that
 was observed during tracing.\footnote{\lstinline{guard_class} performs a precise
 class check, not checking for subclasses.} These guards make the trace specific
@@ -549,6 +548,9 @@
 
 \subsection{Static Objects}
 
+XXX add a footnote saying that "static" is meant in the sense of PE, not static
+allocation
+
 The main insight to improve the code shown in the last section is that objects
 in category 1 don't survive very long -- they are used only inside the loop and
 nobody else in the program stores a reference to them. The idea for improving


From cfbolz at codespeak.net  Thu Oct 14 11:20:39 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 11:20:39 +0200 (CEST)
Subject: [pypy-svn] r77899 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014092039.95A20282BE3@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 11:20:38 2010
New Revision: 77899

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
another comment by samuele


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 11:20:38 2010
@@ -484,6 +484,8 @@
 In the rest of the paper we will see how this trace can be optimized using
 partial evaluation.
 
+XXX add comment that guards are not only slow, but also use lots of memory
+
 \section{Object Lifetimes in a Tracing JIT}
 \label{sec:lifetimes}
 


From fijal at codespeak.net  Thu Oct 14 11:32:17 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Thu, 14 Oct 2010 11:32:17 +0200 (CEST)
Subject: [pypy-svn] r77900 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014093217.A6F83282BE3@codespeak.net>

Author: fijal
Date: Thu Oct 14 11:32:16 2010
New Revision: 77900

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
Add a comment


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 11:32:16 2010
@@ -134,6 +134,10 @@
 dispatching. Those are problems that are usually not present or at least less
 severe in statically typed languages.
 
+XXX [fijal] I would say that the other goal of the JIT in case of dynamic
+languages is to compile only a common scenario and leave a guard (instead
+of proving that something will never ever happen)
+
 Boxing of primitive types is necessary because dynamic languages need to be able to handle
 all objects, even integers, floats, booleans etc. in the same way as user-defined
 instances. Thus those primitive types are usually \emph{boxed}, i.e. a small


From arigo at codespeak.net  Thu Oct 14 11:35:56 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 11:35:56 +0200 (CEST)
Subject: [pypy-svn] r77901 - in pypy/branch/32ptr-on-64bit/pypy/rpython:
	lltypesystem test
Message-ID: <20101014093556.1536C282BE3@codespeak.net>

Author: arigo
Date: Thu Oct 14 11:35:54 2010
New Revision: 77901

Modified:
   pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py
   pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py
Log:
Test and fix: GCREF may point to a var-sized object, so it cannot be compressed.


Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/rpython/lltypesystem/rcompressed.py	Thu Oct 14 11:35:54 2010
@@ -14,6 +14,8 @@
     # mmap()-within-32GB-of-RAM.
     if baserepr.lowleveltype.TO._is_varsize():
         return baserepr
+    if baserepr.lowleveltype == llmemory.GCREF:
+        return baserepr    # a GCREF may contain a non-fixed-size object
     try:
         comprmgr = rtyper.compressed_gcref_manager
     except AttributeError:

Modified: pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/rpython/test/test_rcompressed.py	Thu Oct 14 11:35:54 2010
@@ -167,6 +167,12 @@
         assert er is r_L
         assert ir.lowleveltype == llmemory.HiddenGcRef32
 
+    def test_rerased(self):
+        from pypy.rlib.rerased import ErasedRepr
+        r_E = ErasedRepr(self.rtyper)
+        er, ir = rmodel.externalvsinternal(self. rtyper, r_E)
+        assert er is ir is r_E
+
 
 class TestLLtype64(MixinCompressed64, test_rclass.TestLLtype):
 


From arigo at codespeak.net  Thu Oct 14 11:36:31 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 11:36:31 +0200 (CEST)
Subject: [pypy-svn] r77902 - pypy/branch/32ptr-on-64bit/pypy/translator/c/src
Message-ID: <20101014093631.ABFED282BE3@codespeak.net>

Author: arigo
Date: Thu Oct 14 11:36:30 2010
New Revision: 77902

Modified:
   pypy/branch/32ptr-on-64bit/pypy/translator/c/src/linuxmemchk.c
Log:
Update the limit to a recently-useful number.


Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/linuxmemchk.c
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/translator/c/src/linuxmemchk.c	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/translator/c/src/linuxmemchk.c	Thu Oct 14 11:36:30 2010
@@ -7,7 +7,7 @@
 #include 
 #define PAGESIZE 4096
 #ifndef MALLOC_BIGBUFFER
-# define MALLOC_BIGBUFFER   (PAGESIZE*32768)   /* 128MB */
+# define MALLOC_BIGBUFFER   (PAGESIZE*32768*14)   /* 1.75GB */
 #endif
 
 


From arigo at codespeak.net  Thu Oct 14 11:38:26 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 11:38:26 +0200 (CEST)
Subject: [pypy-svn] r77903 - in
	pypy/branch/32ptr-on-64bit/pypy/objspace/std: . test
Message-ID: <20101014093826.A0F61282BE3@codespeak.net>

Author: arigo
Date: Thu Oct 14 11:38:25 2010
New Revision: 77903

Modified:
   pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py
   pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_mapdict.py
Log:
Update mapdicts to only use rerased for the last field,
which allows all other fields to be compressed.


Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py	Thu Oct 14 11:38:25 2010
@@ -413,11 +413,14 @@
     rangen = unroll.unrolling_iterable(range(n))
     nmin1 = n - 1
     rangenmin1 = unroll.unrolling_iterable(range(nmin1))
+    #
     class subcls(ObjectMixin, BaseMapdictObject, supercls):
+        _nmin1 = nmin1
+        for _i in rangenmin1:
+            locals()["_value%s" % _i] = None
+        locals()["_value%s" % nmin1] = rerased.erase(None)
+
         def _init_empty(self, map):
-            from pypy.rlib.debug import make_sure_not_resized
-            for i in rangen:
-                setattr(self, "_value%s" % i, rerased.erase(None))
             self.map = map
 
         def _has_storage_list(self):
@@ -430,22 +433,21 @@
         def _mapdict_read_storage(self, index):
             for i in rangenmin1:
                 if index == i:
-                    erased = getattr(self, "_value%s" % i)
-                    return rerased.unerase(erased, W_Root)
+                    return getattr(self, "_value%s" % i)
             if self._has_storage_list():
                 return self._mapdict_get_storage_list()[index - nmin1]
             erased = getattr(self, "_value%s" % nmin1)
             return rerased.unerase(erased, W_Root)
 
         def _mapdict_write_storage(self, index, value):
-            erased = rerased.erase(value)
             for i in rangenmin1:
                 if index == i:
-                    setattr(self, "_value%s" % i, erased)
+                    setattr(self, "_value%s" % i, value)
                     return
             if self._has_storage_list():
                 self._mapdict_get_storage_list()[index - nmin1] = value
                 return
+            erased = rerased.erase(value)
             setattr(self, "_value%s" % nmin1, erased)
 
         def _mapdict_storage_length(self):
@@ -458,10 +460,10 @@
             len_storage = len(storage)
             for i in rangenmin1:
                 if i < len_storage:
-                    erased = rerased.erase(storage[i])
+                    value = storage[i]
                 else:
-                    erased = rerased.erase(None)
-                setattr(self, "_value%s" % i, erased)
+                    value = None
+                setattr(self, "_value%s" % i, value)
             has_storage_list = self._has_storage_list()
             if len_storage < n:
                 assert not has_storage_list

Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_mapdict.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_mapdict.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_mapdict.py	Thu Oct 14 11:38:25 2010
@@ -372,12 +372,18 @@
         obj = objectcls()
         obj.user_setup(space, cls)
         obj.setdictvalue(space, "a", w1)
-        assert rerased.unerase(obj._value0, W_Root) is w1
+        if objectcls._nmin1 == 0:
+            assert rerased.unerase(obj._value0, W_Root) is w1
+        else:
+            assert obj._value0 is w1
         assert obj.getdictvalue(space, "a") is w1
         assert obj.getdictvalue(space, "b") is None
         assert obj.getdictvalue(space, "c") is None
         obj.setdictvalue(space, "a", w2)
-        assert rerased.unerase(obj._value0, W_Root) is w2
+        if objectcls._nmin1 == 0:
+            assert rerased.unerase(obj._value0, W_Root) is w2
+        else:
+            assert obj._value0 is w2
         assert obj.getdictvalue(space, "a") == w2
         assert obj.getdictvalue(space, "b") is None
         assert obj.getdictvalue(space, "c") is None
@@ -395,7 +401,10 @@
 
         res = obj.deldictvalue(space, "a")
         assert res
-        assert rerased.unerase(obj._value0, W_Root) is w4
+        if objectcls._nmin1 == 0:
+            assert rerased.unerase(obj._value0, W_Root) is w4
+        else:
+            assert obj._value0 is w4
         assert obj.getdictvalue(space, "a") is None
         assert obj.getdictvalue(space, "b") is w4
         assert obj.getdictvalue(space, "c") is None


From arigo at codespeak.net  Thu Oct 14 11:40:19 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 11:40:19 +0200 (CEST)
Subject: [pypy-svn] r77904 - in
	pypy/branch/32ptr-on-64bit/pypy/objspace/std: . test
Message-ID: <20101014094019.ABFB2282BE3@codespeak.net>

Author: arigo
Date: Thu Oct 14 11:40:18 2010
New Revision: 77904

Modified:
   pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py
   pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_mapdict.py
Log:
Upgrade the previous hack by not using 'rerased' at all in the compressptr
case.  Instead, point to a small W_Root subclass that itself contains the
pointer to the list.


Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py	Thu Oct 14 11:40:18 2010
@@ -359,8 +359,8 @@
         assert isinstance(weakreflifeline, WeakrefLifeline)
         self._get_mapdict_map().write(self, ("weakref", SPECIAL), weakreflifeline)
 
-class ObjectMixin(object):
-    _mixin_ = True
+class Object(BaseMapdictObject, W_Root):
+    # mainly for tests
     def _init_empty(self, map):
         from pypy.rlib.debug import make_sure_not_resized
         self.map = map
@@ -376,18 +376,13 @@
         self.storage = storage
         self.map = map
 
-class Object(ObjectMixin, BaseMapdictObject, W_Root):
-    pass # mainly for tests
-
 def get_subclass_of_correct_size(space, cls, w_type):
     assert space.config.objspace.std.withmapdict
     map = w_type.terminator
     classes = memo_get_subclass_of_correct_size(space, cls)
     size = map.size_estimate()
-    if not size:
-        size = 1
     try:
-        return classes[size - 1]
+        return classes[size]
     except IndexError:
         return classes[-1]
 get_subclass_of_correct_size._annspecialcase_ = "specialize:arg(1)"
@@ -401,24 +396,48 @@
     except KeyError:
         assert not hasattr(supercls, "__del__")
         result = []
-        for i in range(1, NUM_SUBCLASSES+1):
-            result.append(_make_subclass_size_n(supercls, i))
+        if space.config.translation.compressptr:
+            # with 'compressptr', there are two issues: first we only need
+            # half as many classes, because there is no point in having a
+            # class that contains an odd number of hiddengcrefs (there is
+            # one too for the map, so there is no point in having an even
+            # number of attributes); and we avoid using rlib.rerased because
+            # such an erased pointer cannot be compressed, as it might point
+            # to a list.
+            for i in range(0, NUM_SUBCLASSES // 2 + 1):
+                subcls = _make_subclass_size_n(supercls, i * 2 + 1,
+                                               use_erased=False)
+                result.append(subcls)
+                result.append(subcls)
+        else:
+            # common case
+            for i in range(1, NUM_SUBCLASSES+1):
+                result.append(_make_subclass_size_n(supercls, i,
+                                                    use_erased=True))
+            result.insert(0, result[0])
         _subclass_cache[key] = result
         return result
 memo_get_subclass_of_correct_size._annspecialcase_ = "specialize:memo"
 _subclass_cache = {}
 
-def _make_subclass_size_n(supercls, n):
-    from pypy.rlib import unroll, rerased
+def _make_subclass_size_n(supercls, n, use_erased):
+    from pypy.rlib import unroll
     rangen = unroll.unrolling_iterable(range(n))
     nmin1 = n - 1
     rangenmin1 = unroll.unrolling_iterable(range(nmin1))
+    if use_erased:
+        from pypy.rlib import rerased
+        erase = rerased.erase
+        unerase = rerased.unerase
+    else:
+        erase = lambda x: x
+        unerase = lambda x, t: x
     #
-    class subcls(ObjectMixin, BaseMapdictObject, supercls):
+    class subcls(BaseMapdictObject, supercls):
         _nmin1 = nmin1
         for _i in rangenmin1:
             locals()["_value%s" % _i] = None
-        locals()["_value%s" % nmin1] = rerased.erase(None)
+        locals()["_value%s" % nmin1] = erase(None)
 
         def _init_empty(self, map):
             self.map = map
@@ -428,7 +447,11 @@
 
         def _mapdict_get_storage_list(self):
             erased = getattr(self, "_value%s" % nmin1)
-            return rerased.unerase_fixedsizelist(erased, W_Root)
+            if use_erased:
+                return rerased.unerase_fixedsizelist(erased, W_Root)
+            else:
+                assert isinstance(erased, ExtraAttributes)
+                return erased.storage
 
         def _mapdict_read_storage(self, index):
             for i in rangenmin1:
@@ -437,7 +460,7 @@
             if self._has_storage_list():
                 return self._mapdict_get_storage_list()[index - nmin1]
             erased = getattr(self, "_value%s" % nmin1)
-            return rerased.unerase(erased, W_Root)
+            return unerase(erased, W_Root)
 
         def _mapdict_write_storage(self, index, value):
             for i in rangenmin1:
@@ -447,7 +470,7 @@
             if self._has_storage_list():
                 self._mapdict_get_storage_list()[index - nmin1] = value
                 return
-            erased = rerased.erase(value)
+            erased = erase(value)
             setattr(self, "_value%s" % nmin1, erased)
 
         def _mapdict_storage_length(self):
@@ -467,25 +490,33 @@
             has_storage_list = self._has_storage_list()
             if len_storage < n:
                 assert not has_storage_list
-                erased = rerased.erase(None)
+                erased = erase(None)
             elif len_storage == n:
                 assert not has_storage_list
-                erased = rerased.erase(storage[nmin1])
+                erased = erase(storage[nmin1])
             elif not has_storage_list:
                 # storage is longer than self.map.length() only due to
                 # overallocation
-                erased = rerased.erase(storage[nmin1])
+                erased = erase(storage[nmin1])
                 # in theory, we should be ultra-paranoid and check all entries,
                 # but checking just one should catch most problems anyway:
                 assert storage[n] is None
             else:
                 storage_list = storage[nmin1:]
-                erased = rerased.erase_fixedsizelist(storage_list, W_Root)
+                if use_erased:
+                    erased = rerased.erase_fixedsizelist(storage_list, W_Root)
+                else:
+                    erased = ExtraAttributes(storage_list)
             setattr(self, "_value%s" % nmin1, erased)
 
     subcls.__name__ = supercls.__name__ + "Size%s" % n
     return subcls
 
+class ExtraAttributes(W_Root):
+    def __init__(self, storage):
+        from pypy.rlib.debug import make_sure_not_resized
+        self.storage = make_sure_not_resized(storage)
+
 # ____________________________________________________________
 # dict implementation
 

Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_mapdict.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_mapdict.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/test/test_mapdict.py	Thu Oct 14 11:40:18 2010
@@ -1,6 +1,7 @@
 from pypy.conftest import gettestobjspace, option
 from pypy.objspace.std.test.test_dictmultiobject import FakeSpace, W_DictMultiObject
 from pypy.objspace.std.mapdict import *
+from pypy.objspace.std.mapdict import _subclass_cache
 
 space = FakeSpace()
 
@@ -357,9 +358,15 @@
 # check specialized classes
 
 
-def test_specialized_class():
+def test_specialized_class(compressptr=False):
     from pypy.objspace.std.objectobject import W_ObjectObject
     from pypy.rlib import rerased
+    space = FakeSpace()
+    class translation:
+        pass
+    translation.compressptr = compressptr
+    space.config.translation = translation
+    _subclass_cache.clear()
     classes = memo_get_subclass_of_correct_size(space, W_ObjectObject)
     w1 = W_Root()
     w2 = W_Root()
@@ -372,7 +379,7 @@
         obj = objectcls()
         obj.user_setup(space, cls)
         obj.setdictvalue(space, "a", w1)
-        if objectcls._nmin1 == 0:
+        if objectcls._nmin1 == 0 and not compressptr:
             assert rerased.unerase(obj._value0, W_Root) is w1
         else:
             assert obj._value0 is w1
@@ -380,7 +387,7 @@
         assert obj.getdictvalue(space, "b") is None
         assert obj.getdictvalue(space, "c") is None
         obj.setdictvalue(space, "a", w2)
-        if objectcls._nmin1 == 0:
+        if objectcls._nmin1 == 0 and not compressptr:
             assert rerased.unerase(obj._value0, W_Root) is w2
         else:
             assert obj._value0 is w2
@@ -401,7 +408,7 @@
 
         res = obj.deldictvalue(space, "a")
         assert res
-        if objectcls._nmin1 == 0:
+        if objectcls._nmin1 == 0 and not compressptr:
             assert rerased.unerase(obj._value0, W_Root) is w4
         else:
             assert obj._value0 is w4
@@ -417,6 +424,9 @@
         assert obj2.getdictvalue(space, "b") is w6
         assert obj2.map is abmap
 
+def test_specialized_class_compressptr():
+    test_specialized_class(compressptr=True)
+
 # ___________________________________________________________
 # integration tests
 


From afa at codespeak.net  Thu Oct 14 11:42:03 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 11:42:03 +0200 (CEST)
Subject: [pypy-svn] r77905 - in
	pypy/branch/fast-forward/pypy/module/_hashlib: . test
Message-ID: <20101014094203.0699F282BE3@codespeak.net>

Author: afa
Date: Thu Oct 14 11:42:02 2010
New Revision: 77905

Modified:
   pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py
   pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py
Log:
Hash value was truncated to the first \0 :-(
Test and fix.


Modified: pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py	Thu Oct 14 11:42:02 2010
@@ -71,7 +71,7 @@
 
         try:
             ropenssl.EVP_DigestFinal(ctx, digest, None)
-            return rffi.charp2strn(digest, digest_size)
+            return rffi.charpsize2str(digest, digest_size)
         finally:
             lltype.free(digest, flavor='raw')
 

Modified: pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py	Thu Oct 14 11:42:02 2010
@@ -8,6 +8,7 @@
     def test_simple(self):
         import _hashlib
         assert isinstance(_hashlib.new('md5'), _hashlib.HASH)
+        assert len(_hashlib.new('md5').hexdigest()) == 32
 
     def test_attributes(self):
         import hashlib


From arigo at codespeak.net  Thu Oct 14 11:43:47 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 11:43:47 +0200 (CEST)
Subject: [pypy-svn] r77906 - pypy/branch/32ptr-on-64bit/pypy/objspace/std
Message-ID: <20101014094347.A086E282BE3@codespeak.net>

Author: arigo
Date: Thu Oct 14 11:43:46 2010
New Revision: 77906

Modified:
   pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py
Log:
Revert this part of the change.  ObjectMixin is needed
by interpreter/typedef.py.


Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py	Thu Oct 14 11:43:46 2010
@@ -359,8 +359,8 @@
         assert isinstance(weakreflifeline, WeakrefLifeline)
         self._get_mapdict_map().write(self, ("weakref", SPECIAL), weakreflifeline)
 
-class Object(BaseMapdictObject, W_Root):
-    # mainly for tests
+class ObjectMixin(object):
+    _mixin_ = True
     def _init_empty(self, map):
         from pypy.rlib.debug import make_sure_not_resized
         self.map = map
@@ -376,6 +376,9 @@
         self.storage = storage
         self.map = map
 
+class Object(ObjectMixin, BaseMapdictObject, W_Root):
+    pass # mainly for tests
+
 def get_subclass_of_correct_size(space, cls, w_type):
     assert space.config.objspace.std.withmapdict
     map = w_type.terminator


From antocuni at codespeak.net  Thu Oct 14 11:47:43 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 14 Oct 2010 11:47:43 +0200 (CEST)
Subject: [pypy-svn] r77907 - in pypy/branch/jitffi/pypy/module/_ffi: . test
Message-ID: <20101014094743.F10CF282BE3@codespeak.net>

Author: antocuni
Date: Thu Oct 14 11:47:42 2010
New Revision: 77907

Modified:
   pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py
   pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py
Log:
add a way to the the actual address of the underlying function. Useful for testing


Modified: pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py	(original)
+++ pypy/branch/jitffi/pypy/module/_ffi/interp_ffi.py	Thu Oct 14 11:47:42 2010
@@ -96,9 +96,17 @@
             assert voidres is None
             return space.w_None
 
+    @unwrap_spec('self', ObjSpace)
+    def getaddr(self, space):
+        """
+        Return the physical address in memory of the function
+        """
+        return space.wrap(rffi.cast(rffi.LONG, self.func.funcsym))
+
 W_FuncPtr.typedef = TypeDef(
     'FuncPtr',
-    __call__ = interp2app(W_FuncPtr.call)
+    __call__ = interp2app(W_FuncPtr.call),
+    getaddr = interp2app(W_FuncPtr.getaddr),
     )
 
 

Modified: pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py	(original)
+++ pypy/branch/jitffi/pypy/module/_ffi/test/test__ffi.py	Thu Oct 14 11:47:42 2010
@@ -32,13 +32,19 @@
 
     
     def setup_class(cls):
-        from pypy.rlib.libffi import get_libc_name
+        from pypy.rpython.lltypesystem import rffi
+        from pypy.rlib.libffi import get_libc_name, CDLL, types
         from pypy.rlib.test.test_libffi import get_libm_name
         space = gettestobjspace(usemodules=('_ffi',))
         cls.space = space
         cls.w_libfoo_name = space.wrap(cls.prepare_c_example())
         cls.w_libc_name = space.wrap(get_libc_name())
-        cls.w_libm_name = space.wrap(get_libm_name(sys.platform))
+        libm_name = get_libm_name(sys.platform)
+        cls.w_libm_name = space.wrap(libm_name)
+        libm = CDLL(libm_name)
+        pow = libm.getpointer('pow', [], types.void)
+        pow_addr = rffi.cast(rffi.LONG, pow.funcsym)
+        cls.w_pow_addr = space.wrap(pow_addr)
 
     def test_libload(self):
         import _ffi
@@ -58,6 +64,12 @@
         libm = CDLL(self.libm_name)
         pow = libm.getfunc('pow', [types.double, types.double], types.double)
         assert pow(2, 3) == 8
+
+    def test_getaddr(self):
+        from _ffi import CDLL, types
+        libm = CDLL(self.libm_name)
+        pow = libm.getfunc('pow', [types.double, types.double], types.double)
+        assert pow.getaddr() == self.pow_addr
         
     def test_int_args(self):
         """


From arigo at codespeak.net  Thu Oct 14 11:57:09 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 11:57:09 +0200 (CEST)
Subject: [pypy-svn] r77908 - pypy/branch/32ptr-on-64bit/pypy/objspace/std
Message-ID: <20101014095709.47FC4282BE3@codespeak.net>

Author: arigo
Date: Thu Oct 14 11:57:07 2010
New Revision: 77908

Modified:
   pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py
Log:
Flow graph fix.


Modified: pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/objspace/std/mapdict.py	Thu Oct 14 11:57:07 2010
@@ -433,6 +433,7 @@
         erase = rerased.erase
         unerase = rerased.unerase
     else:
+        rerased = None    # don't use in that case
         erase = lambda x: x
         unerase = lambda x, t: x
     #


From afa at codespeak.net  Thu Oct 14 12:04:09 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 12:04:09 +0200 (CEST)
Subject: [pypy-svn] r77909 - in pypy/branch/fast-forward/pypy:
	module/_hashlib rlib
Message-ID: <20101014100409.925C4282BE3@codespeak.net>

Author: afa
Date: Thu Oct 14 12:04:07 2010
New Revision: 77909

Modified:
   pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py
   pypy/branch/fast-forward/pypy/rlib/ropenssl.py
Log:
"OpenSSL_add_all_digests()" is neeeded to call other hashlib functions,
difficult to test because CPython does it already in the same process.


Modified: pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py	Thu Oct 14 12:04:07 2010
@@ -13,3 +13,7 @@
 
     for name in algorithms:
         interpleveldefs[name] = 'interp_hashlib.new_%s' % (name,)
+
+    def startup(self, space):
+        from pypy.rlib.ropenssl import init_digests
+        init_digests()

Modified: pypy/branch/fast-forward/pypy/rlib/ropenssl.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/ropenssl.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/ropenssl.py	Thu Oct 14 12:04:07 2010
@@ -128,6 +128,8 @@
 EVP_MD_CTX = rffi.COpaquePtr('EVP_MD_CTX', compilation_info=eci)
 EVP_MD     = rffi.COpaquePtr('EVP_MD')
 
+OpenSSL_add_all_digests = external(
+    'OpenSSL_add_all_digests', [], lltype.Void)
 EVP_get_digestbyname = external(
     'EVP_get_digestbyname',
     [rffi.CCHARP], EVP_MD)
@@ -149,3 +151,5 @@
     libssl_SSL_load_error_strings()
     libssl_SSL_library_init()
 
+def init_digests():
+    OpenSSL_add_all_digests()


From afa at codespeak.net  Thu Oct 14 12:05:57 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 12:05:57 +0200 (CEST)
Subject: [pypy-svn] r77910 - in pypy/branch/fast-forward: lib_pypy
	lib_pypy/ctypes_config_cache/test lib_pypy/pypy_test
	pypy/module/_md5 pypy/module/_sha
Message-ID: <20101014100557.B9256282BE3@codespeak.net>

Author: afa
Date: Thu Oct 14 12:05:55 2010
New Revision: 77910

Modified:
   pypy/branch/fast-forward/lib_pypy/_md5.py
   pypy/branch/fast-forward/lib_pypy/_sha.py
   pypy/branch/fast-forward/lib_pypy/ctypes_config_cache/test/test_cache.py
   pypy/branch/fast-forward/lib_pypy/pypy_test/test_md5_extra.py
   pypy/branch/fast-forward/lib_pypy/pypy_test/test_sha_extra.py
   pypy/branch/fast-forward/pypy/module/_md5/__init__.py
   pypy/branch/fast-forward/pypy/module/_md5/interp_md5.py
   pypy/branch/fast-forward/pypy/module/_sha/__init__.py
   pypy/branch/fast-forward/pypy/module/_sha/interp_sha.py
Log:
Line up md5 and sha modules with CPython.
Should fix a warning in hmac.py

+ fix tests after rename.


Modified: pypy/branch/fast-forward/lib_pypy/_md5.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/_md5.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/_md5.py	Thu Oct 14 12:05:55 2010
@@ -116,6 +116,7 @@
     "An implementation of the MD5 hash function in pure Python."
 
     digest_size = digestsize = 16
+    block_size = 64
 
     def __init__(self):
         "Initialisation."
@@ -369,24 +370,10 @@
 
 # ======================================================================
 # Mimic Python top-level functions from standard library API
-# for consistency with the md5 module of the standard library.
+# for consistency with the _md5 module of the standard library.
 # ======================================================================
 
-digest_size = digestsize = 16
-blocksize = 1
-
-def new(arg=None):
-    """Return a new md5 crypto object.
-
-    If arg is present, the method call update(arg) is made.
-    """
-
-    crypto = MD5Type()
-    if arg:
-        crypto.update(arg)
-
-    return crypto
-
+digest_size = 16
 
 def md5(arg=None):
     """Same as new().

Modified: pypy/branch/fast-forward/lib_pypy/_sha.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/_sha.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/_sha.py	Thu Oct 14 12:05:55 2010
@@ -118,6 +118,7 @@
     "An implementation of the MD5 hash function in pure Python."
 
     digest_size = digestsize = 20
+    block_size = 1
 
     def __init__(self):
         "Initialisation."
@@ -328,13 +329,14 @@
 
 # ======================================================================
 # Mimic Python top-level functions from standard library API
-# for consistency with the md5 module of the standard library.
+# for consistency with the _sha module of the standard library.
 # ======================================================================
 
 # These are mandatory variables in the module. They have constant values
 # in the SHA standard.
 
-digest_size = digestsize = 20
+digest_size = 20
+digestsize = 20
 blocksize = 1
 
 def new(arg=None):

Modified: pypy/branch/fast-forward/lib_pypy/ctypes_config_cache/test/test_cache.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/ctypes_config_cache/test/test_cache.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/ctypes_config_cache/test/test_cache.py	Thu Oct 14 12:05:55 2010
@@ -27,10 +27,6 @@
     d = run('syslog.ctc.py', '_syslog_cache.py')
     assert 'LOG_NOTICE' in d
 
-def test_hashlib():
-    d = run('hashlib.ctc.py', '_hashlib_cache.py')
-    assert hasattr(d['EVP_MD_CTX'], 'digest')
-
 def test_resource():
     d = run('resource.ctc.py', '_resource_cache.py')
     assert 'RLIM_NLIMITS' in d

Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_md5_extra.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/pypy_test/test_md5_extra.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_md5_extra.py	Thu Oct 14 12:05:55 2010
@@ -6,7 +6,7 @@
 
 from __future__ import absolute_import
 import md5                              # CPython's implementation in C.
-from .. import md5 as pymd5  
+from .. import _md5 as pymd5  
 
 
 # Helpers...

Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_sha_extra.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/pypy_test/test_sha_extra.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_sha_extra.py	Thu Oct 14 12:05:55 2010
@@ -4,7 +4,7 @@
 # Publication 180-1, Secure Hash Standard,  1995 April 17
 # http://www.itl.nist.gov/div897/pubs/fip180-1.htm
 from __future__ import absolute_import
-from .. import sha 
+from .. import _sha 
 
 class TestSHA: 
     def check(self, data, digest):

Modified: pypy/branch/fast-forward/pypy/module/_md5/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_md5/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_md5/__init__.py	Thu Oct 14 12:05:55 2010
@@ -19,12 +19,9 @@
 far using the digest() method."""
 
     interpleveldefs = {
-        'md5': 'interp_md5.W_MD5',
         'new': 'interp_md5.W_MD5',
         'MD5Type': 'interp_md5.W_MD5',
         'digest_size': 'space.wrap(16)',
-        'digestsize': 'space.wrap(16)',
-        'blocksize': 'space.wrap(1)',
         }
 
     appleveldefs = {

Modified: pypy/branch/fast-forward/pypy/module/_md5/interp_md5.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_md5/interp_md5.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_md5/interp_md5.py	Thu Oct 14 12:05:55 2010
@@ -49,6 +49,7 @@
     copy      = interp2app(W_MD5.copy_w, unwrap_spec=['self']),
     digest_size = 16,
     digestsize = 16,
+    block_size = 64,
     __doc__   = """md5(arg) -> return new md5 object.
 
 If arg is present, the method call update(arg) is made.""")

Modified: pypy/branch/fast-forward/pypy/module/_sha/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_sha/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_sha/__init__.py	Thu Oct 14 12:05:55 2010
@@ -19,7 +19,6 @@
 bits instead of MD5's 128 bits."""
 
     interpleveldefs = {
-        'sha': 'interp_sha.W_SHA',
         'new': 'interp_sha.W_SHA',
         'SHAType': 'interp_sha.W_SHA',
         'blocksize': 'space.wrap(1)',

Modified: pypy/branch/fast-forward/pypy/module/_sha/interp_sha.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_sha/interp_sha.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_sha/interp_sha.py	Thu Oct 14 12:05:55 2010
@@ -49,6 +49,7 @@
     copy      = interp2app(W_SHA.copy_w, unwrap_spec=['self']),
     digest_size = 20,
     digestsize = 20,
+    block_size = 64,
     __doc__   = """sha(arg) -> return new sha object.
 
 If arg is present, the method call update(arg) is made.""")


From pedronis at codespeak.net  Thu Oct 14 12:19:53 2010
From: pedronis at codespeak.net (pedronis at codespeak.net)
Date: Thu, 14 Oct 2010 12:19:53 +0200 (CEST)
Subject: [pypy-svn] r77911 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014101953.BD83A282BE3@codespeak.net>

Author: pedronis
Date: Thu Oct 14 12:19:49 2010
New Revision: 77911

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
the escaping is not needed

Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 12:19:49 2010
@@ -467,9 +467,9 @@
     \item \lstinline{new} corresponds to object creation.
     \item \lstinline{get} correspond to attribute reads.
     \item \lstinline{set} correspond to attribute writes.
-    \item \lstinline{guard\_class} correspond to method calls and are followed by
+    \item \lstinline{guard_class} correspond to method calls and are followed by
     the trace of the called method.
-    \item \lstinline{int\_add} and \lstinline{int\_get} are integer addition and
+    \item \lstinline{int_add} and \lstinline{int_get} are integer addition and
     comparison (``greater than''), respectively.
 \end{itemize}
 
@@ -484,7 +484,7 @@
 using the interpreter.
 
 The trace shows the inefficiencies of \lstinline{f} clearly, if one looks at the
-number of \lstinline{new}, \lstinline{set/get} and \lstinline{guard\_class} operations.
+number of \lstinline{new}, \lstinline{set/get} and \lstinline{guard_class} operations.
 In the rest of the paper we will see how this trace can be optimized using
 partial evaluation.
 


From pedronis at codespeak.net  Thu Oct 14 12:21:07 2010
From: pedronis at codespeak.net (pedronis at codespeak.net)
Date: Thu, 14 Oct 2010 12:21:07 +0200 (CEST)
Subject: [pypy-svn] r77912 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014102107.00169282BE3@codespeak.net>

Author: pedronis
Date: Thu Oct 14 12:21:06 2010
New Revision: 77912

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
typo+tweak

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 12:21:06 2010
@@ -1110,8 +1110,8 @@
 \section{Conclusion}
 \label{sec:conclusion}
 
-In this paper, we used a approach based on online partial evaluation to
-optimize allocations in the traces of a tracing JIT.  In this context a
+In this paper, we used an approach based on online partial evaluation to
+optimize away allocations in the traces of a tracing JIT.  In this context a
 simple approach to partial evaluation gives good results.  This is due
 to the fact that the tracing JIT itself is responsible for all control
 issues, which are usually the hardest part of partial evaluation: the


From pedronis at codespeak.net  Thu Oct 14 12:34:01 2010
From: pedronis at codespeak.net (pedronis at codespeak.net)
Date: Thu, 14 Oct 2010 12:34:01 +0200 (CEST)
Subject: [pypy-svn] r77913 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014103401.E92B3282BE3@codespeak.net>

Author: pedronis
Date: Thu Oct 14 12:34:00 2010
New Revision: 77913

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
Log:
regen pdf

Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.


From afa at codespeak.net  Thu Oct 14 12:59:48 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 12:59:48 +0200 (CEST)
Subject: [pypy-svn] r77914 - in
	pypy/branch/fast-forward/pypy/module/_multiprocessing: . test
Message-ID: <20101014105948.EF014282BE9@codespeak.net>

Author: afa
Date: Thu Oct 14 12:59:47 2010
New Revision: 77914

Modified:
   pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py
   pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py
Log:
Add context manager to _multiprocessing.SemLock


Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_multiprocessing/interp_semaphore.py	Thu Oct 14 12:59:47 2010
@@ -1,6 +1,6 @@
 from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root
 from pypy.interpreter.typedef import TypeDef, GetSetProperty
-from pypy.interpreter.gateway import interp2app, unwrap_spec
+from pypy.interpreter.gateway import interp2app, Arguments, unwrap_spec
 from pypy.interpreter.error import wrap_oserror, OperationError
 from pypy.rpython.lltypesystem import rffi, lltype
 from pypy.rlib.rarithmetic import r_uint
@@ -493,6 +493,14 @@
         self.__init__(handle_w(space, w_handle), kind, maxvalue)
         return space.wrap(self)
 
+    @unwrap_spec('self', ObjSpace)
+    def enter(self, space):
+        return self.acquire(space, w_timeout=space.w_None)
+
+    @unwrap_spec('self', ObjSpace, Arguments)
+    def exit(self, space, __args__):
+        self.release(space)
+
 @unwrap_spec(ObjSpace, W_Root, int, int, int)
 def descr_new(space, w_subtype, kind, value, maxvalue):
     if kind != RECURSIVE_MUTEX and kind != SEMAPHORE:
@@ -525,5 +533,7 @@
     acquire = interp2app(W_SemLock.acquire),
     release = interp2app(W_SemLock.release),
     _rebuild = interp2app(W_SemLock.rebuild.im_func, as_classmethod=True),
+    __enter__=interp2app(W_SemLock.enter),
+    __exit__=interp2app(W_SemLock.exit),
     SEM_VALUE_MAX=SEM_VALUE_MAX,
     )

Modified: pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_multiprocessing/test/test_semaphore.py	Thu Oct 14 12:59:47 2010
@@ -71,3 +71,14 @@
 
         sem2 = SemLock._rebuild(sem.handle, kind, value)
         assert sem.handle == sem2.handle
+
+    def test_semaphore_contextmanager(self):
+        from _multiprocessing import SemLock
+        kind = self.SEMAPHORE
+        value = 1
+        maxvalue = 1
+        sem = SemLock(kind, value, maxvalue)
+
+        with sem:
+            assert sem._count() == 1
+        assert sem._count() == 0


From antocuni at codespeak.net  Thu Oct 14 13:34:33 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 14 Oct 2010 13:34:33 +0200 (CEST)
Subject: [pypy-svn] r77915 - pypy/branch/jitffi/pypy/module/pypyjit/test
Message-ID: <20101014113433.0B8F3282BE9@codespeak.net>

Author: antocuni
Date: Thu Oct 14 13:34:32 2010
New Revision: 77915

Modified:
   pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py
Log:
add a test to check that the applevel _ffi call is optimized


Modified: pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py	Thu Oct 14 13:34:32 2010
@@ -79,8 +79,11 @@
 
 
 class PyPyCJITTests(object):
-    def run_source(self, source, expected_max_ops, *testcases):
+    def run_source(self, source, expected_max_ops, *testcases, **kwds):
         assert isinstance(expected_max_ops, int)
+        threshold = kwds.pop('threshold', 3)
+        if kwds:
+            raise TypeError, 'Unsupported keyword arguments: %s' % kwds.keys()
         source = py.code.Source(source)
         filepath = self.tmpdir.join('case%d.py' % self.counter)
         logfilepath = filepath.new(ext='.log')
@@ -92,7 +95,7 @@
             import sys
             try: # make the file runnable by CPython
                 import pypyjit
-                pypyjit.set_param(threshold=3)
+                pypyjit.set_param(threshold=%d)
             except ImportError:
                 pass
 
@@ -102,7 +105,7 @@
                 print >> sys.stderr, 'got:', repr(result)
                 assert result == expected
                 assert type(result) is type(expected)
-        """)
+        """ % threshold)
         for testcase in testcases * 2:
             print >> f, "check(%r, %r)" % testcase
         print >> f, "print 'OK :-)'"
@@ -123,6 +126,7 @@
         if self.total_ops > expected_max_ops:
             assert 0, "too many operations: got %d, expected maximum %d" % (
                 self.total_ops, expected_max_ops)
+        return result
 
     def parse_loops(self, opslogfile):
         from pypy.jit.metainterp.test.oparser import parse
@@ -1130,6 +1134,36 @@
             return sa
         ''', 88, ([], 1997001))
 
+    def test__ffi_call(self):
+        from pypy.rlib.test.test_libffi import get_libm_name
+        libm_name = get_libm_name(sys.platform)
+        out = self.run_source('''
+        def main():
+            from _ffi import CDLL, types
+            libm = CDLL('%(libm_name)s')
+            pow = libm.getfunc('pow', [types.double, types.double],
+                               types.double)
+            print pow.getaddr()
+            i = 0
+            res = 0
+            while i < 2000:
+                res += pow(2, 3)
+                i += 1
+            return res
+        ''' % locals(),
+                              70, ([], 8.0*2000), threshold=1000)
+        pow_addr = int(out.splitlines()[0])
+        ops = self.get_by_bytecode('CALL_FUNCTION')
+        assert len(ops) == 2 # we get two loops, because of specialization
+        op = ops[0]
+        # XXX: there should be a guard_not_forced
+        assert op[-1].getopname() == 'guard_no_exception'
+        call = op[-2]
+        assert call.getopname() == 'call'
+        assert call.getarg(0).value == pow_addr
+        assert call.getarg(1).value == 2.0
+        assert call.getarg(2).value == 3.0
+
     # test_circular
 
 class AppTestJIT(PyPyCJITTests):


From afa at codespeak.net  Thu Oct 14 13:34:44 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 13:34:44 +0200 (CEST)
Subject: [pypy-svn] r77916 -
	pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests
Message-ID: <20101014113444.77ADD282BEA@codespeak.net>

Author: afa
Date: Thu Oct 14 13:34:42 2010
New Revision: 77916

Modified:
   pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_array.py
Log:
Typo in a test


Modified: pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_array.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_array.py	(original)
+++ pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_array.py	Thu Oct 14 13:34:42 2010
@@ -82,11 +82,11 @@
 
         na = numarray(1, 2, 3, 4, 5)
         values = [i for i in na]
-        assert values, [1, 2, 3, 4 == 5]
+        assert values == [1, 2, 3, 4, 5]
 
         na = numarray(*map(c_int, (1, 2, 3, 4, 5)))
         values = [i for i in na]
-        assert values, [1, 2, 3, 4 == 5]
+        assert values == [1, 2, 3, 4, 5]
 
     def test_classcache(self):
         assert not ARRAY(c_int, 3) is ARRAY(c_int, 4)


From afa at codespeak.net  Thu Oct 14 13:43:03 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 13:43:03 +0200 (CEST)
Subject: [pypy-svn] r77917 - in pypy/branch/fast-forward: lib_pypy/_ctypes
	pypy/module/test_lib_pypy/ctypes_tests
Message-ID: <20101014114303.5A9E4282BEA@codespeak.net>

Author: afa
Date: Thu Oct 14 13:43:01 2010
New Revision: 77917

Modified:
   pypy/branch/fast-forward/lib_pypy/_ctypes/array.py
   pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_array.py
Log:
Error in slicing ctypes arrays: a[:9999] would fail.
Test and fix.


Modified: pypy/branch/fast-forward/lib_pypy/_ctypes/array.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/_ctypes/array.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/_ctypes/array.py	Thu Oct 14 13:43:01 2010
@@ -109,8 +109,18 @@
 def array_get_slice_params(self, index):
     if index.step is not None:
         raise TypeError("3 arg slices not supported (for no reason)")
-    start = index.start or 0
-    stop = index.stop or self._length_
+    if index.start is not None:
+        start = index.start
+        if start < 0:
+            start = 0
+    else:
+        start = 0
+    if index.stop is not None:
+        stop = index.stop
+        if stop > self._length_:
+            stop = self._length_
+    else:
+        stop = self._length_
     return start, stop
 
 def array_slice_setitem(self, index, value):

Modified: pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_array.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_array.py	(original)
+++ pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_array.py	Thu Oct 14 13:43:01 2010
@@ -88,6 +88,16 @@
         values = [i for i in na]
         assert values == [1, 2, 3, 4, 5]
 
+    def test_slice(self):
+        values = range(5)
+        numarray = c_int * 5
+
+        na = numarray(*(c_int(x) for x in values))
+
+        assert list(na[0:0]) == []
+        assert list(na[:])   == values
+        assert list(na[:10]) == values
+
     def test_classcache(self):
         assert not ARRAY(c_int, 3) is ARRAY(c_int, 4)
         assert ARRAY(c_int, 3) is ARRAY(c_int, 3)


From leuschel at codespeak.net  Thu Oct 14 13:52:46 2010
From: leuschel at codespeak.net (leuschel at codespeak.net)
Date: Thu, 14 Oct 2010 13:52:46 +0200 (CEST)
Subject: [pypy-svn] r77918 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014115246.71B22282BEA@codespeak.net>

Author: leuschel
Date: Thu Oct 14 13:52:42 2010
New Revision: 77918

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
some changes to sect. 4


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 13:52:42 2010
@@ -560,15 +560,15 @@
 The main insight to improve the code shown in the last section is that objects
 in category 1 don't survive very long -- they are used only inside the loop and
 nobody else in the program stores a reference to them. The idea for improving
-the code is thus to analyze which objects fall in category 1 and may thus
-not be allocated at all.
+the code is thus to analyze which objects fall in category 1 and thus do
+not have to be allocated at all.
 
 This is a process that is usually called \emph{escape analysis}. In this paper we will
 perform escape analysis by using partial evaluation. The partial evalution is a
-bit peculiar in that there are not actually any constant arguments to the trace,
-but it is only used to optimized operations within a trace.
+bit peculiar in that it receives no static input arguments for the trace,
+but it is only used to optimize operations within a trace.
 
-The partial evaluation works by walking the trace from beginning to end.
+The partial evaluation works by traversing the trace from beginning to end.
 Whenever a \lstinline{new} operation is seen, the operation is removed and a static
 object is constructed and associated with the variable that would have stored
 the result of \lstinline{new}. The static object describes the shape of the
@@ -583,8 +583,9 @@
 as well, because the shape description stores the type and thus the outcome of
 the type check the guard does is statically known.
 
-In the example from last section, the following operations would produce two
-static objects, and be completely removed from the optimized trace:
+In the example from last section, the following operations in the upper half
+of Fig.~\ref{fig:unopt-trace} would produce two
+static objects, and would thus be completely removed from the optimized trace:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 $p_{5}$ = new(BoxedInteger)
@@ -599,7 +600,8 @@
 one associated with $p_{6}$ would know that it is a \lstinline{BoxedInteger}
 whose \lstinline{intval} field contains the constant -100.
 
-The following operations on $p_{5}$ and $p_{6}$ could then be
+The subsequent operations in Fig.~\ref{fig:unopt-trace},
+ which use $p_{5}$ and $p_{6}$, could then be
 optimized using that knowledge:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
@@ -612,8 +614,8 @@
 $i_{9}$ = int_add($i_{7}$, $i_{8}$)
 \end{lstlisting}
 
-The \lstinline{guard_class} operations can be removed, because the classes of $p_{5}$ and
-$p_{6}$ are known to be \lstinline{BoxedInteger}. The \lstinline{get} operations can be removed
+First, the \lstinline{guard_class} operations can be removed, because the classes of $p_{5}$ and
+$p_{6}$ are known to be \lstinline{BoxedInteger}. Second, the \lstinline{get} operations can be removed
 and $i_{7}$ and $i_{8}$ are just replaced by $i_{4}$ and -100. Thus the only
 remaining operation in the optimized trace would be:
 
@@ -621,24 +623,24 @@
 $i_{9}$ = int_add($i_{4}$, -100)
 \end{lstlisting}
 
-The rest of the trace is optimized similarly.
+The rest of the trace from Fig.~\ref{fig:unopt-trace} is optimized similarly.
 
-So far we have only described what happens when static objects are used in
-operations that read and write their fields and in guards. When the static
-object is used in any other operation, it cannot stay static. For example, when
-a static object is stored in a globally accessible place, the object needs to
-actually be allocated, as it might live longer than one iteration of the loop
+So far we have only described what happens when static objects are used in guards and in
+operations that read and write fields. When the static
+object is used in any other operation, it cannot remain static. For example, when
+a static object is stored in a globally accessible place, the object has to
+be allocated, as it might live longer than one iteration of the loop
 and because the partial evaluator looses track of it. This means that the static
 object needs to be turned into a dynamic one, \ie lifted. This makes it
-necessary to put operations into the residual code that actually allocate the
+necessary to put operations into the residual code that allocate the
 static object at runtime.
 
-This is what happens at the end of the trace in Figure~\ref{fig:unopt-trace}, when the \lstinline{jump} operation
+This is what happens at the end of the trace in Fig.~\ref{fig:unopt-trace}, when the \lstinline{jump} operation
 is hit. The arguments of the jump are at this point static objects. Before the
 jump is emitted, they are \emph{lifted}. This means that the optimizer produces code
 that allocates a new object of the right type and sets its fields to the field
 values that the static object has (if the static object points to other static
-objects, those need to be lifted as well, recursively) This means that instead of the jump,
+objects, those need to be lifted as well, recursively). This means that instead of a simple jump,
 the following operations are emitted:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
@@ -649,10 +651,11 @@
 jump($p_{15}$, $p_{10}$)
 \end{lstlisting}
 
-Note how the operations for creating these two instances have been moved down the
-trace. It looks like for these operations we actually didn't win much, because
-the objects are still allocated at the end. However, the optimization was still
-worthwhile even in this case, because some operations that have been performed
+Observe how the operations for creating these two instances have been moved to later point in the
+trace. 
+At first sight, it may look like for these operations we didn't gain much, as
+the objects are still allocated in the end. However, our optimizations were still
+worthwhile, because some operations that have been performed
 on the lifted static objects have been removed (some \lstinline{get} operations
 and \lstinline{guard_class} operations).
 


From cfbolz at codespeak.net  Thu Oct 14 14:06:34 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 14:06:34 +0200 (CEST)
Subject: [pypy-svn] r77919 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014120634.6A8CF282BEA@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 14:06:32 2010
New Revision: 77919

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
try to have a marginally acceptable author block


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 14:06:32 2010
@@ -1,4 +1,4 @@
-\documentclass{sigplanconf}
+\documentclass[preprint]{sigplanconf}
 
 \usepackage{ifthen}
 \usepackage{fancyvrb}
@@ -72,16 +72,23 @@
 \begin{tabular}{c}}
 %
 \begin{document}
-\conferenceinfo{PEPM'11,} {XXX}
-\CopyrightYear{XXX}
-\copyrightdata{XXX}
+\conferenceinfo{PEPM}{'11, Austin, USA}
+\CopyrightYear{2011}
+\copyrightdata{[to be supplied]}
 
 \title{Allocation Removal by Partial Evaluation in a Tracing JIT}
 
-\authorinfo{Carl Friedrich Bolz \and Antonio Cuni \and Maciej Fija?kowski \and Michael Leuschel \and Samuele Pedroni \and Armin Rigo}
-           {Heinrich-Heine-Universit?t D?sseldorf, STUPS Group, Germany XXX}
+\authorinfo{Carl Friedrich Bolz$^a$ \and Antonio Cuni$^a$ \and Maciej Fija?kowski$^b$ \and Michael Leuschel$^a$ \and \\
+            Samuele Pedroni$^c$ \and Armin Rigo$^a$}
+           {$^a$Heinrich-Heine-Universit?t D?sseldorf, STUPS Group, Germany
+
+            $^b$merlinux GmbH, Hildesheim, Germany
+
+            $^c$Open End, G?teborg, Sweden
+           }
            {cfbolz at gmx.de, anto.cuni at gmail.com, fijal at merlinux.eu,
            leuschel at cs.uni-duesseldorf.de, samuele.pedroni at gmail.com, arigo at tunes.org}
+%           {Heinrich-Heine-Universit?t D?sseldorf, STUPS Group, Germany} \and {merlinux GmbH, Hildesheim, Germany} \and {Open End, G?teborg, Sweden}
 
 %\numberofauthors{3}
 %\author{
@@ -1133,8 +1140,8 @@
 
 \section*{Acknowledgements}
 
-The authors would like to thank Stefan Hallerstede and Thomas Stiehl for
-fruitful discussions during the writing of the paper.
+The authors would like to thank Stefan Hallerstede, David Schneider and Thomas
+Stiehl for fruitful discussions during the writing of the paper.
 
 \bibliographystyle{abbrv}
 \bibliography{paper}


From cfbolz at codespeak.net  Thu Oct 14 14:07:58 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 14:07:58 +0200 (CEST)
Subject: [pypy-svn] r77920 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014120758.7176B282BEA@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 14:07:56 2010
New Revision: 77920

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
better email addresses


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 14:07:56 2010
@@ -86,9 +86,8 @@
 
             $^c$Open End, G?teborg, Sweden
            }
-           {cfbolz at gmx.de, anto.cuni at gmail.com, fijal at merlinux.eu,
-           leuschel at cs.uni-duesseldorf.de, samuele.pedroni at gmail.com, arigo at tunes.org}
-%           {Heinrich-Heine-Universit?t D?sseldorf, STUPS Group, Germany} \and {merlinux GmbH, Hildesheim, Germany} \and {Open End, G?teborg, Sweden}
+           {cfbolz at gmx.de \and anto.cuni at gmail.com \and fijal at merlinux.eu,
+           leuschel at cs.uni-duesseldorf.de \and samuele.pedroni at gmail.com \and arigo at tunes.org}
 
 %\numberofauthors{3}
 %\author{


From arigo at codespeak.net  Thu Oct 14 14:20:37 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 14:20:37 +0200 (CEST)
Subject: [pypy-svn] r77921 - pypy/branch/32ptr-on-64bit/pypy/translator/c/src
Message-ID: <20101014122037.6AC55282BEA@codespeak.net>

Author: arigo
Date: Thu Oct 14 14:20:35 2010
New Revision: 77921

Modified:
   pypy/branch/32ptr-on-64bit/pypy/translator/c/src/stack.h
Log:
When compiling on gcc with -O0, we really need more stack.
Thanks antocuni for finding out the proper #ifdefs.


Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/src/stack.h
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/translator/c/src/stack.h	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/translator/c/src/stack.h	Thu Oct 14 14:20:35 2010
@@ -3,7 +3,11 @@
  /***  C header subsection: stack operations               ***/
 
 #ifndef MAX_STACK_SIZE
+#  if defined(__GNUC__) && !defined(__OPTIMIZE__)
+#    define MAX_STACK_SIZE (7 << 20)    /* 7 MB */
+#  else
 #    define MAX_STACK_SIZE (3 << 18)    /* 768 kb */
+#  endif
 #endif
 
 /* This include must be done in any case to initialise


From cfbolz at codespeak.net  Thu Oct 14 14:24:13 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 14:24:13 +0200 (CEST)
Subject: [pypy-svn] r77922 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014122413.7CCC4282BEA@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 14:24:12 2010
New Revision: 77922

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
fix an XXX, add a new one


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 14:24:12 2010
@@ -183,8 +183,7 @@
 The core of our trace optimization technique can be
 viewed as partial evaluation: the partial evaluation
 performs a form of escape analysis \cite{bruno_blanchet_escape_2003} on the traces and make some
-objects that are allocated in the trace \emph{static}\footnote{These objects are
-called \emph{virtual} in Psyco \cite{rigo_representation-based_2004}.} which
+objects that are allocated in the trace \emph{static} which
 means that they do not occur any more in the optimized trace. This technique is
 informally described in Section~\ref{sec:statics}, a more formal description is
 given in Section~\ref{sec:formal}.
@@ -560,9 +559,6 @@
 
 \subsection{Static Objects}
 
-XXX add a footnote saying that "static" is meant in the sense of PE, not static
-allocation
-
 The main insight to improve the code shown in the last section is that objects
 in category 1 don't survive very long -- they are used only inside the loop and
 nobody else in the program stores a reference to them. The idea for improving
@@ -575,8 +571,11 @@
 but it is only used to optimize operations within a trace.
 
 The partial evaluation works by traversing the trace from beginning to end.
-Whenever a \lstinline{new} operation is seen, the operation is removed and a static
-object is constructed and associated with the variable that would have stored
+Whenever a \lstinline{new} operation is seen, the operation is removed and a \empth{static
+object}\footnote{Here ``static'' is meant in the sense of partial
+evaluation, \ie known at partial evaluation time, not in the sense of static
+allocation or static method.} is constructed and associated with the variable
+that would have stored
 the result of \lstinline{new}. The static object describes the shape of the
 original object, \eg where the values that would be stored in the fields of the
 allocated object come from, as well as the type of the object. Whenever the
@@ -1115,6 +1114,9 @@
 
 xxx: relation to compile-time garbage collection \cite{mazur_practical_2001};  separation logic; John Hughes: type specialization
 
+XXX Psyco \cite{rigo_representation-based_2004}
+
+XXX cite SELF paper for the type propagation effects
 
 \section{Conclusion}
 \label{sec:conclusion}


From david at codespeak.net  Thu Oct 14 14:39:11 2010
From: david at codespeak.net (david at codespeak.net)
Date: Thu, 14 Oct 2010 14:39:11 +0200 (CEST)
Subject: [pypy-svn] r77923 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101014123911.52F33282BEA@codespeak.net>

Author: david
Date: Thu Oct 14 14:39:09 2010
New Revision: 77923

Added:
   pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py
Log:
Add ARM regalloc


Added: pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py
==============================================================================
--- (empty file)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py	Thu Oct 14 14:39:09 2010
@@ -0,0 +1,34 @@
+from pypy.jit.backend.llsupport.regalloc import FrameManager, \
+        RegisterManager, compute_vars_longevity
+from pypy.jit.backend.arm import registers as r
+
+class ARMRegisterManager(RegisterManager):
+    all_regs              = r.all_regs
+    box_types             = None       # or a list of acceptable types
+    no_lower_byte_regs    = r.all_regs
+    save_around_call_regs = all_regs
+
+    def __init__(self, longevity, frame_manager=None, assembler=None):
+        RegisterManager.__init__(self, longevity, frame_manager, assembler)
+
+class ARMFrameManager(FrameManager):
+    @staticmethod
+    def frame_pos(loc, type):
+        pass
+
+#class RegAlloc(object):
+#    def __init__(self, assembler, translate_support_code=False):
+#        self.assembler = assembler
+#        self.translate_support_code = translate_support_code
+#        self.fm = None
+#
+#    def _prepare(self, inputargs, operations):
+#        longevity = compute_vars_longevity(inputargs, operations)
+#        self.rm = ARMRegisterManager(longevity, self.fm)
+#
+#    def prepare_loop(self, inputargs, operations, looptoken):
+#        self._prepare(inputargs, operations)
+#
+#    def force_allocate_reg(self, v, forbidden_vars=[], selected_reg=None,
+#                           need_lower_byte=False):
+#        return self.rm.force_allocate_reg(v, forbidden_vars, selected_reg, need_lower_byte)


From fijal at codespeak.net  Thu Oct 14 14:39:57 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Thu, 14 Oct 2010 14:39:57 +0200 (CEST)
Subject: [pypy-svn] r77924 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014123957.BA05D5080B@codespeak.net>

Author: fijal
Date: Thu Oct 14 14:39:56 2010
New Revision: 77924

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
Two more comments


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 14:39:56 2010
@@ -931,6 +931,9 @@
 because every time an object is stored into a local variable, it is stored into
 the frame-object, which makes it escape.
 
+XXX [fijal] Well, in fact that's *a lot* of support for debugger from the VM.
+The support is just generic instead of specific.
+
 This problem is solved by making it possible to the interpreter author to add
 some hints into the source code to declare instances of one class as frame
 objects. The JIT will then fill these objects only lazily when they are actually
@@ -952,6 +955,9 @@
 benchmarks we used are small-to-medium Python programs, some synthetic
 benchmarks, some real applications.
 
+XXX [fijal] note that some of those have cpython-specific hacks removed or
+old versions used (nbody mostly I think).
+
 Some of them are from the Computer Language Benchmark
 Game\footnote{\texttt{http://shootout.alioth.debian.org/}}: \textbf{fannkuch},
 \textbf{nbody}, \textbf{meteor-contest}, \textbf{spectral-norm}.


From cfbolz at codespeak.net  Thu Oct 14 14:43:50 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 14:43:50 +0200 (CEST)
Subject: [pypy-svn] r77925 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014124350.7AD0C282BEA@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 14:43:49 2010
New Revision: 77925

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
fix another an XXX and a typo


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 14:43:49 2010
@@ -488,13 +488,17 @@
 first \lstinline{guard_class} instruction will fail and execution will continue
 using the interpreter.
 
-The trace shows the inefficiencies of \lstinline{f} clearly, if one looks at the
-number of \lstinline{new}, \lstinline{set/get} and \lstinline{guard_class} operations.
+The trace shows the inefficiencies of \lstinline{f} clearly, if one looks at
+the number of \lstinline{new}, \lstinline{set/get} and \lstinline{guard_class}
+operations. The number of \lstinline{guard_class} operation is particularly
+problematic, not only because of the time it takes to run them. All guards also
+have additional information attached that makes it possible to return to the
+interpreter, should the guard fail. This means that many guard operations also
+lead to a memory problem.
+
 In the rest of the paper we will see how this trace can be optimized using
 partial evaluation.
 
-XXX add comment that guards are not only slow, but also use lots of memory
-
 \section{Object Lifetimes in a Tracing JIT}
 \label{sec:lifetimes}
 
@@ -571,7 +575,7 @@
 but it is only used to optimize operations within a trace.
 
 The partial evaluation works by traversing the trace from beginning to end.
-Whenever a \lstinline{new} operation is seen, the operation is removed and a \empth{static
+Whenever a \lstinline{new} operation is seen, the operation is removed and a \emph{static
 object}\footnote{Here ``static'' is meant in the sense of partial
 evaluation, \ie known at partial evaluation time, not in the sense of static
 allocation or static method.} is constructed and associated with the variable


From david at codespeak.net  Thu Oct 14 14:45:47 2010
From: david at codespeak.net (david at codespeak.net)
Date: Thu, 14 Oct 2010 14:45:47 +0200 (CEST)
Subject: [pypy-svn] r77926 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . test
Message-ID: <20101014124547.7CB4136C231@codespeak.net>

Author: david
Date: Thu Oct 14 14:45:45 2010
New Revision: 77926

Added:
   pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
Log:
Refactor parts of the codebuilder, build load and store instructions from a table


Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Thu Oct 14 14:45:45 2010
@@ -1,6 +1,7 @@
 import conditions as cond
 from pypy.rlib.rmmap import alloc
 from pypy.rpython.lltypesystem import lltype, rffi
+from pypy.jit.backend.arm.instruction_builder import define_instructions
 
 class ARMv7Builder(object):
 
@@ -8,21 +9,6 @@
         self._data = alloc(1024)
         self._pos = 0
 
-    def LDR_ri(self, rt, rn, imm=0, cond=cond.AL):
-        #  XXX W and P bits are not encoded yet
-        p = 1
-        w = 0
-        u, imm = self._encode_imm(imm)
-        self.write32(cond << 28
-                        | 0x1 << 26
-                        | (p & 0x1) << 24
-                        | (u & 0x1) << 23
-                        | (w & 0x1) << 21
-                        | 0x1 << 20
-                        | (rn & 0xF) << 16
-                        | (rt & 0xF) << 12
-                        | (imm & 0xFFF))
-
     def ADD_ri(self, rt, rn, imm, cond=cond.AL):
         # XXX S bit
         self.write32(cond << 28
@@ -58,20 +44,6 @@
                     | (rt & 0xF) << 12
                     | (imm & 0xFFF))
 
-    def STR_ri(self, rt, rn, imm=0, cond=cond.AL):
-        #  XXX W and P bits are not encoded yet
-        p = 1
-        w = 0
-        u, imm = self._encode_imm(imm)
-        self.write32(cond << 28
-                    | 0x1 << 26
-                    | (p & 0x1) << 24
-                    | (u & 0x1) << 23
-                    | (w & 0x1) << 21
-                    | (rn & 0xF) << 16
-                    | (rt & 0xF) << 12
-                    | (imm & 0xFFF))
-
     def ASR_ri(self, rd, rm, imm=0, cond=cond.AL, s=0):
         self.write32(cond << 28
                     | 0xD << 21
@@ -139,4 +111,4 @@
     def curraddr(self):
         return self.baseaddr() + self._pos
 
-
+define_instructions(ARMv7Builder)

Added: pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
==============================================================================
--- (empty file)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	Thu Oct 14 14:45:45 2010
@@ -0,0 +1,43 @@
+from pypy.jit.backend.arm import conditions as cond
+from pypy.jit.backend.arm import instructions
+
+def define_load_store_func(target, name, table):
+    #  XXX W and P bits are not encoded yet
+    n = (0x1 << 26
+        | (table['A'] & 0x1) << 25
+        | (table['op1'] & 0x1F) << 20
+        | (table['B'] & 0x1) << 4)
+    if table['imm']:
+        def f(self, rt, rn, imm=0, cond=cond.AL):
+            p = 1
+            w = 0
+            u, imm = self._encode_imm(imm)
+            self.write32(n
+                    | cond << 28
+                    | (p & 0x1) <<  24
+                    | (u & 0x1) << 23
+                    | (w & 0x1) << 21
+                    | (rn & 0xFF) << 16
+                    | (rt & 0xFF) << 12
+                    | (imm & 0xFFF))
+    else:
+        def f(self, rt, rn, rm, imm=0, cond=cond.AL, s=0, shifttype=0):
+            p = 1
+            w = 0
+            u, imm = self._encode_imm(imm)
+            self.write32(n
+                        | cond << 28
+                        | (p & 0x1) <<  24
+                        | (u & 0x1) << 23
+                        | (w & 0x1) << 21
+                        | (rn & 0xFF) << 16
+                        | (rt & 0xFF) << 12
+                        | (imm & 0x1F) << 7
+                        | (shifttype & 0x3) << 5
+                        | (rm & 0xFF))
+
+    setattr(target, name, f)
+
+def define_instructions(target):
+    for key, val in instructions.load_store.iteritems():
+        define_load_store_func(target, key, val)

Added: pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
==============================================================================
--- (empty file)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	Thu Oct 14 14:45:45 2010
@@ -0,0 +1,12 @@
+# XXX ensure b value is as expected
+# XXX add not assertions for op1
+load_store = {
+    'STR_ri': {'A':0, 'op1': 0x0, 'B': 0, 'imm': True},
+    'STR_rr': {'A':1, 'op1': 0x0, 'B': 0, 'imm': False},
+    'LDR_ri': {'A':0, 'op1': 0x1, 'B': 0, 'imm': True},
+    'LDR_rr': {'A':1, 'op1': 0x1, 'B': 0, 'imm': False},
+    'STRB_ri': {'A':0, 'op1': 0x4, 'B': 0, 'imm': True},
+    'STRB_rr': {'A':1, 'op1': 0x4, 'B': 0, 'imm': False},
+    'LDRB_ri': {'A':0, 'op1': 0x5, 'B': 0, 'imm': True},
+    'LDRB_rr': {'A':1, 'op1': 0x5, 'B': 0, 'imm': False},
+}

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	Thu Oct 14 14:45:45 2010
@@ -1,5 +1,6 @@
 from pypy.jit.backend.arm import registers as r
 from pypy.jit.backend.arm import codebuilder
+from pypy.jit.backend.arm import instructions
 from pypy.jit.backend.arm.test.support import requires_arm_as
 from gen import assemble
 import py
@@ -15,8 +16,11 @@
 
     def hexdump(self):
         return ''.join(self.buffer)
+class ASMTest(object):
+    def assert_equal(self, asm):
+        assert self.cb.hexdump() == assemble(asm)
 
-class TestInstrCodeBuilder(object):
+class TestInstrCodeBuilder(ASMTest):
     def setup_method(self, ffuu_method):
         self.cb = CodeBuilder()
 
@@ -110,6 +114,31 @@
         self.cb.CMP(r.r3, 123)
         self.assert_equal('CMP r3, #123')
 
-    def assert_equal(self, asm):
-        assert self.cb.hexdump() == assemble(asm)
 
+class TestInstrCodeBuilderForGeneratedInstr(ASMTest):
+    def setup_method(self, ffuu_method):
+        self.cb = CodeBuilder()
+
+def build_tests():
+    for key, value in instructions.load_store.iteritems():
+        if value['imm']:
+            f = gen_test_imm_func
+        else:
+            f = gen_test_reg_func
+        test = f(key, value)
+    setattr(TestInstrCodeBuilderForGeneratedInstr, 'test_%s' % key, test)
+
+def gen_test_imm_func(name, table):
+    def f(self):
+        func = getattr(self.cb, name)
+        func(r.r3, r.r7, 23)
+        self.assert_equal('%s r3, [r7, #23]' % name[:name.index('_')])
+    return f
+
+def gen_test_reg_func(name, table):
+    def f(self):
+        func = getattr(self.cb, name)
+        func(r.r3, r.r7, r.r12)
+        self.assert_equal('%s r3, [r7, r12]' % name[:name.index('_')])
+    return f
+build_tests()


From antocuni at codespeak.net  Thu Oct 14 14:56:22 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 14 Oct 2010 14:56:22 +0200 (CEST)
Subject: [pypy-svn] r77927 - pypy/branch/jitffi/pypy/jit/metainterp/test
Message-ID: <20101014125622.DBEAE36C231@codespeak.net>

Author: antocuni
Date: Thu Oct 14 14:56:21 2010
New Revision: 77927

Added:
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py   (contents, props changed)
Modified:
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
move ffi tests to their own file


Added: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
==============================================================================
--- (empty file)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	Thu Oct 14 14:56:21 2010
@@ -0,0 +1,81 @@
+from pypy.rpython.lltypesystem import llmemory
+from pypy.rlib.libffi import Func, types
+from pypy.jit.metainterp.history import AbstractDescr
+from pypy.jit.codewriter.effectinfo import EffectInfo
+from pypy.jit.metainterp.test.test_optimizeopt import BaseTestOptimizeOpt, LLtypeMixin
+
+class MyCallDescr(AbstractDescr):
+    """
+    Fake calldescr to be used inside the tests.
+
+    The particularity is that it provides an __eq__ method, so that it
+    comparses by value by comparing the arg_types and typeinfo fields, so you
+    can check that the signature of a call is really what you want.
+    """
+
+    def __init__(self, arg_types, typeinfo):
+        self.arg_types = arg_types
+        self.typeinfo = typeinfo   # return type
+
+    def __eq__(self, other):
+        return self.arg_types == other.arg_types and self.typeinfo == other.typeinfo
+
+class FakeLLObject(object):
+
+    def __init__(self, **kwds):
+        self.__dict__.update(kwds)
+        self._TYPE = llmemory.GCREF
+
+    def _identityhash(self):
+        return id(self)
+
+
+class TestFfiCall(BaseTestOptimizeOpt, LLtypeMixin):
+
+    class namespace:
+        cpu = LLtypeMixin.cpu
+        FUNC = LLtypeMixin.FUNC
+        int_float__int = MyCallDescr('if', 'i')
+        funcptr = FakeLLObject()
+        func = FakeLLObject(_fake_class=Func,
+                            argtypes=[types.sint, types.double],
+                            restype=types.sint)
+        #
+        def calldescr(cpu, FUNC, oopspecindex):
+            einfo = EffectInfo([], [], [], oopspecindex=oopspecindex)
+            return cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, einfo)
+        #
+        libffi_prepare =  calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_PREPARE)
+        libffi_push_arg = calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_PUSH_ARG)
+        libffi_call =     calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_CALL)
+    
+    namespace = namespace.__dict__
+
+    def test_ffi_call_opt(self):
+        ops = """
+        [i0, f1]
+        call(0, ConstPtr(func),             descr=libffi_prepare)
+        call(0, ConstPtr(func), i0,         descr=libffi_push_arg)
+        call(0, ConstPtr(func), f1,         descr=libffi_push_arg)
+        i3 = call(0, ConstPtr(func), 12345, descr=libffi_call)
+        jump(i3, f1)
+        """
+        expected = """
+        [i0, f1]
+        i3 = call(12345, i0, f1, descr=int_float__int)
+        jump(i3, f1)
+        """
+        loop = self.optimize_loop(ops, 'Not, Not', expected)
+
+    def test_ffi_call_nonconst(self):
+        ops = """
+        [i0, f1, p2]
+        call(0, p2,             descr=libffi_prepare)
+        call(0, p2, i0,         descr=libffi_push_arg)
+        call(0, p2, f1,         descr=libffi_push_arg)
+        i3 = call(0, p2, 12345, descr=libffi_call)
+        jump(i3, f1, p2)
+        """
+        expected = ops
+        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py	Thu Oct 14 14:56:21 2010
@@ -4499,87 +4499,6 @@
     # XXX str2unicode
 
 
-# ------------------------------------------------ 
-from pypy.rpython.lltypesystem import llmemory
-from pypy.rlib.libffi import Func, types
-from pypy.jit.metainterp.history import AbstractDescr
-from pypy.jit.codewriter.effectinfo import EffectInfo
-
-class MyCallDescr(AbstractDescr):
-    """
-    Fake calldescr to be used inside the tests.
-
-    The particularity is that it provides an __eq__ method, so that it
-    comparses by value by comparing the arg_types and typeinfo fields, so you
-    can check that the signature of a call is really what you want.
-    """
-
-    def __init__(self, arg_types, typeinfo):
-        self.arg_types = arg_types
-        self.typeinfo = typeinfo   # return type
-
-    def __eq__(self, other):
-        return self.arg_types == other.arg_types and self.typeinfo == other.typeinfo
-
-class FakeLLObject(object):
-
-    def __init__(self, **kwds):
-        self.__dict__.update(kwds)
-        self._TYPE = llmemory.GCREF
-
-    def _identityhash(self):
-        return id(self)
-
-
-class TestFfiCall(BaseTestOptimizeOpt, LLtypeMixin):
-
-    class namespace:
-        cpu = LLtypeMixin.cpu
-        FUNC = LLtypeMixin.FUNC
-        int_float__int = MyCallDescr('if', 'i')
-        funcptr = FakeLLObject()
-        func = FakeLLObject(_fake_class=Func,
-                            argtypes=[types.sint, types.double],
-                            restype=types.sint)
-        #
-        def calldescr(cpu, FUNC, oopspecindex):
-            einfo = EffectInfo([], [], [], oopspecindex=oopspecindex)
-            return cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, einfo)
-        #
-        libffi_prepare =  calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_PREPARE)
-        libffi_push_arg = calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_PUSH_ARG)
-        libffi_call =     calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_CALL)
-    
-    namespace = namespace.__dict__
-
-    def test_ffi_call_opt(self):
-        ops = """
-        [i0, f1]
-        call(0, ConstPtr(func),             descr=libffi_prepare)
-        call(0, ConstPtr(func), i0,         descr=libffi_push_arg)
-        call(0, ConstPtr(func), f1,         descr=libffi_push_arg)
-        i3 = call(0, ConstPtr(func), 12345, descr=libffi_call)
-        jump(i3, f1)
-        """
-        expected = """
-        [i0, f1]
-        i3 = call(12345, i0, f1, descr=int_float__int)
-        jump(i3, f1)
-        """
-        loop = self.optimize_loop(ops, 'Not, Not', expected)
-
-    def test_ffi_call_nonconst(self):
-        ops = """
-        [i0, f1, p2]
-        call(0, p2,             descr=libffi_prepare)
-        call(0, p2, i0,         descr=libffi_push_arg)
-        call(0, p2, f1,         descr=libffi_push_arg)
-        i3 = call(0, p2, 12345, descr=libffi_call)
-        jump(i3, f1, p2)
-        """
-        expected = ops
-        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
-
 
 
 ##class TestOOtype(OptimizeOptTest, OOtypeMixin):


From arigo at codespeak.net  Thu Oct 14 14:58:42 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 14:58:42 +0200 (CEST)
Subject: [pypy-svn] r77928 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014125842.B371E5080B@codespeak.net>

Author: arigo
Date: Thu Oct 14 14:58:41 2010
New Revision: 77928

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
Add the word "modern" here, and a footnote.


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 14:58:41 2010
@@ -157,8 +157,11 @@
 applicable to the objects at hand when doing a generic operation on them. An
 example would be the addition of two objects: The addition needs to check what
 the concrete objects that should be added are, and choose the implementation
-that is fitting for them. Type dispatching is a very common operation in a
-dynamic language because no types are known at compile time, so all operations
+that is fitting for them. Type dispatching is a very common operation in
+modern\footnote{For languages in the LISP family, basic arithmetic operations
+are typically not overloaded; even in Smalltalk, type dispatching is much
+simpler than in Python or JavaScript.}
+dynamic languages because no types are known at compile time, so all operations
 need it.
 
 A recently popular approach to implementing just-in-time compilers for dynamic


From arigo at codespeak.net  Thu Oct 14 15:09:36 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 15:09:36 +0200 (CEST)
Subject: [pypy-svn] r77929 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014130936.02B80282BEB@codespeak.net>

Author: arigo
Date: Thu Oct 14 15:09:34 2010
New Revision: 77929

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
Typos.


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 15:09:34 2010
@@ -169,7 +169,7 @@
 program and recording its hot spots into linear execution traces, which are then turned into
 machine code. One reason for the popularity of tracing JITs is their relative
 simplicity. They can often be added to an interpreter and a lot of the
-infrastructure of an interpreter can be reused. They give some important
+infrastructure of the interpreter can be reused. They give some important
 optimizations like inlining and constant-folding for free. A tracing JIT always
 produces linear pieces of code, which simplifies many optimizations that are usually
 hard in a compiler, such as register allocation.
@@ -186,9 +186,9 @@
 The core of our trace optimization technique can be
 viewed as partial evaluation: the partial evaluation
 performs a form of escape analysis \cite{bruno_blanchet_escape_2003} on the traces and make some
-objects that are allocated in the trace \emph{static} which
+objects that are allocated in the trace \emph{static,} which
 means that they do not occur any more in the optimized trace. This technique is
-informally described in Section~\ref{sec:statics}, a more formal description is
+informally described in Section~\ref{sec:statics}; a more formal description is
 given in Section~\ref{sec:formal}.
 
 In Section~\ref{sec:support} we describe some supporting techniques that are not
@@ -997,11 +997,12 @@
 2.6.35. We compared the performance of various Python implementations on the
 benchmarks. As a baseline, we used the standard Python implementation in C,
 CPython 2.6.6\footnote{\texttt{http://python.org}}, which uses a bytecode-based
-interpreter. Furthermore we compared against Psyco 1.6
-\cite{rigo_representation-based_2004}, an extension to CPython which is a
+interpreter. Furthermore we compared against
+Psyco\cite{rigo_representation-based_2004} 1.6,
+an extension to CPython which is a
 just-in-time compiler that produces machine code at run-time. It is not based
 on traces. Finally, we used two versions of PyPy's Python interpreter (revision
-77823 of SVN trunk\footnote{\texttt{https://codespeak.net/svn/pypy/trunk}}): one
+77823 of SVN trunk\footnote{\texttt{http://codespeak.net/svn/pypy/trunk}}): one
 including the JIT but not optimizing the traces, and one using the allocation
 removal optimizations (as well as some minor other optimizations, such as
 constant folding).
@@ -1063,7 +1064,7 @@
 \begin{center}
 \begin{tabular}{|l||r|r|r|r|}
 \hline
-	&Cpython	&Psyco	&PyPy w/o optimizations	&PyPy w/ optimizations \\
+	&CPython	&Psyco	&PyPy w/o optimizations	&PyPy w/ optimizations \\
 \hline
 crypto\_pyaes	&2757.80 $\pm$ 0.98	&67.90 $\pm$ 0.47	&1652.00 $\pm$ 4.00 &266.86 $\pm$ 5.94 \\
 django	&993.19 $\pm$ 0.50	&913.51 $\pm$ 4.22	&694.73 $\pm$ 2.86 &259.53 $\pm$ 1.79 \\


From arigo at codespeak.net  Thu Oct 14 15:22:01 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 15:22:01 +0200 (CEST)
Subject: [pypy-svn] r77930 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014132201.8608836C231@codespeak.net>

Author: arigo
Date: Thu Oct 14 15:22:00 2010
New Revision: 77930

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
A short paragraph about Psyco in Related Work.


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 15:22:00 2010
@@ -1117,6 +1117,12 @@
 \cite{michael_bebenita_spur:_2010}. However, no details for the approach and its
 implementation are given.
 
+Psyco \cite{rigo_representation-based_2004} is a (non-tracing) JIT for Python
+that fully implements allocation removal (the \emph{static} objects are called
+\emph{virtual} objects).  It is a hand-written extension module for CPython.
+Historically, PyPy's JIT can be seen as some successor of Psyco for a
+general context.
+
 partial evaluation:
 
 Prolog:
@@ -1128,8 +1134,6 @@
 
 xxx: relation to compile-time garbage collection \cite{mazur_practical_2001};  separation logic; John Hughes: type specialization
 
-XXX Psyco \cite{rigo_representation-based_2004}
-
 XXX cite SELF paper for the type propagation effects
 
 \section{Conclusion}


From antocuni at codespeak.net  Thu Oct 14 15:23:12 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 14 Oct 2010 15:23:12 +0200 (CEST)
Subject: [pypy-svn] r77931 - in pypy/branch/jitffi/pypy: jit/codewriter
	jit/metainterp/optimizeopt jit/metainterp/test rlib/test
Message-ID: <20101014132312.F3339282BEA@codespeak.net>

Author: antocuni
Date: Thu Oct 14 15:23:11 2010
New Revision: 77931

Modified:
   pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
   pypy/branch/jitffi/pypy/rlib/test/test_libffi.py
Log:
attach the correct effectinfo to the libffi oopspec, and adapt the optimizer
and the tests: now the ffi call is done through CALL_MAY_FORCE, not simply CALL.

It might force in case of callbacks



Modified: pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py	Thu Oct 14 15:23:11 2010
@@ -1030,8 +1030,10 @@
     # ----------
     # Strings and Unicodes.
 
-    def _handle_oopspec_call(self, op, args, oopspecindex):
+    def _handle_oopspec_call(self, op, args, oopspecindex, extraeffect=None):
         calldescr = self.callcontrol.getcalldescr(op, oopspecindex)
+        if extraeffect:
+            calldescr.get_extra_info().extraeffect = extraeffect
         if isinstance(op.args[0].value, str):
             pass  # for tests only
         else:
@@ -1129,13 +1131,16 @@
     def _handle_libffi_call(self, op, oopspec_name, args):
         if oopspec_name == 'libffi_prepare_call':
             oopspecindex = EffectInfo.OS_LIBFFI_PREPARE
+            extraeffect = EffectInfo.EF_CANNOT_RAISE
         elif oopspec_name.startswith('libffi_push_'):
             oopspecindex = EffectInfo.OS_LIBFFI_PUSH_ARG
+            extraeffect = EffectInfo.EF_CANNOT_RAISE
         elif oopspec_name.startswith('libffi_call_'):
             oopspecindex = EffectInfo.OS_LIBFFI_CALL
+            extraeffect = EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE
         else:
             assert False, 'unsupported oopspec: %s' % oopspec_name
-        return self._handle_oopspec_call(op, args, oopspecindex)
+        return self._handle_oopspec_call(op, args, oopspecindex, extraeffect)
 
     def rewrite_op_jit_force_virtual(self, op):
         return self._do_builtin_call(op)

Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	Thu Oct 14 15:23:11 2010
@@ -88,6 +88,8 @@
             # normal case
             self.emit_operation(op)
 
+    optimize_CALL_MAY_FORCE = optimize_CALL
+
     def _get_funcval(self, op):
         funcval = self.getvalue(op.getarg(1))
         if not funcval.is_constant():
@@ -113,7 +115,7 @@
         for push_op in info.opargs:
             argval = self.getvalue(push_op.getarg(2))
             arglist.append(argval.force_box())
-        newop = ResOperation(rop.CALL, arglist, op.result, descr=info.descr)
+        newop = ResOperation(rop.CALL_MAY_FORCE, arglist, op.result, descr=info.descr)
         del self.func_infos[funcval]
         return newop
 

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	Thu Oct 14 15:23:11 2010
@@ -57,12 +57,16 @@
         call(0, ConstPtr(func),             descr=libffi_prepare)
         call(0, ConstPtr(func), i0,         descr=libffi_push_arg)
         call(0, ConstPtr(func), f1,         descr=libffi_push_arg)
-        i3 = call(0, ConstPtr(func), 12345, descr=libffi_call)
+        i3 = call_may_force(0, ConstPtr(func), 12345, descr=libffi_call)
+        guard_not_forced() []
+        guard_no_exception() []
         jump(i3, f1)
         """
         expected = """
         [i0, f1]
-        i3 = call(12345, i0, f1, descr=int_float__int)
+        i3 = call_may_force(12345, i0, f1, descr=int_float__int)
+        guard_not_forced() []
+        guard_no_exception() []
         jump(i3, f1)
         """
         loop = self.optimize_loop(ops, 'Not, Not', expected)

Modified: pypy/branch/jitffi/pypy/rlib/test/test_libffi.py
==============================================================================
--- pypy/branch/jitffi/pypy/rlib/test/test_libffi.py	(original)
+++ pypy/branch/jitffi/pypy/rlib/test/test_libffi.py	Thu Oct 14 15:23:11 2010
@@ -133,8 +133,9 @@
         res = self.call(func, [38, 4.2], rffi.LONG)
         assert res == 42
         self.check_loops({
-                'call': 1,
+                'call_may_force': 1,
                 'guard_no_exception': 1,
+                'guard_not_forced': 1,
                 'int_add': 1,
                 'int_lt': 1,
                 'guard_true': 1,
@@ -145,7 +146,7 @@
         func = (libm, 'pow', [types.double, types.double], types.double)
         res = self.call(func, [2.0, 3.0], rffi.DOUBLE, init_result=0.0)
         assert res == 8.0
-        self.check_loops(call=1)
+        self.check_loops(call_may_force=1, guard_no_exception=1, guard_not_forced=1)
 
     def test_cast_result(self):
         """
@@ -158,7 +159,7 @@
         func = (libfoo, 'cast_to_uchar_and_ovf', [types.sint], types.uchar)
         res = self.call(func, [0], rffi.UCHAR)
         assert res == 200
-        self.check_loops(call=1)
+        self.check_loops(call_may_force=1, guard_no_exception=1, guard_not_forced=1)
 
     def test_cast_argument(self):
         """


From pedronis at codespeak.net  Thu Oct 14 15:27:08 2010
From: pedronis at codespeak.net (pedronis at codespeak.net)
Date: Thu, 14 Oct 2010 15:27:08 +0200 (CEST)
Subject: [pypy-svn] r77932 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014132708.F302E5080B@codespeak.net>

Author: pedronis
Date: Thu Oct 14 15:27:07 2010
New Revision: 77932

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
expand for clarification

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 15:27:07 2010
@@ -928,27 +928,30 @@
 
 \cfbolz{probably can be cut in case of space problems}
 
-One problem to the successful application of the allocation removal techniques
-described in the previous sections is the presence of frame-introspection
-features in many dynamic languages. Languages such as Python and Smalltalk allow
-the programmer to get access to the frames object that the interpreter uses to
-store local variables. This is a useful feature, as makes the implementation of
-a debugger possible in Python without needing much support from the VM level. On
-the other hand, it severely hinders the effectiveness of allocation removal,
-because every time an object is stored into a local variable, it is stored into
-the frame-object, which makes it escape.
-
-XXX [fijal] Well, in fact that's *a lot* of support for debugger from the VM.
-The support is just generic instead of specific.
-
-This problem is solved by making it possible to the interpreter author to add
-some hints into the source code to declare instances of one class as frame
-objects. The JIT will then fill these objects only lazily when they are actually
-accessed (\eg because a debugger is used). Therefore in the common case, nothing
-is stored into the frame objects, making the problem of too much escaping go
-away. This is a common approach in VM implementations \cite{miranda_context_1999}, the only
-novelty in our approach lays in its generality, because most other JITs are just
-specifically written for one particular language.
+One problem to the successful application of the allocation removal
+techniques described in the previous sections is the presence of
+frame-introspection features in many dynamic languages. Languages such
+as Python and Smalltalk allow the programmer to get access to the
+frames object that the interpreter uses to store local variables. This
+is a useful feature, as makes the implementation of a debugger
+possible in Python without needing much explicit support from the VM
+level. On the other hand, it severely hinders the effectiveness of
+allocation removal, because every time an object is stored into a
+local variable, it is stored into the frame-object, which makes it
+escape.
+
+This problem is solved by making it possible to the interpreter author
+to add some hints into the source code to declare instances of one
+class as frame objects needing special treatment. The JIT will then
+fill these objects only lazily when they are actually accessed (\eg
+because a debugger is used) using values from JIT-emitted code runtime
+locations (typically the CPU stack). Therefore in the common case,
+nothing is stored into the frame objects, making the problem of too
+much escaping go away. This is a common approach in VM implementations
+\cite{miranda_context_1999}, the only novelty in our approach lays in
+its generality, all the delicate support code for this is generated, because
+most other JITs are just specifically written for one particular
+language.
 
 % subsection Virtualizables (end)
 


From antocuni at codespeak.net  Thu Oct 14 15:32:23 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 14 Oct 2010 15:32:23 +0200 (CEST)
Subject: [pypy-svn] r77933 - pypy/branch/jitffi/pypy/jit/metainterp/test
Message-ID: <20101014133223.A97D9282BEA@codespeak.net>

Author: antocuni
Date: Thu Oct 14 15:32:22 2010
New Revision: 77933

Modified:
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
Log:
update this test too


Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	Thu Oct 14 15:32:22 2010
@@ -77,7 +77,9 @@
         call(0, p2,             descr=libffi_prepare)
         call(0, p2, i0,         descr=libffi_push_arg)
         call(0, p2, f1,         descr=libffi_push_arg)
-        i3 = call(0, p2, 12345, descr=libffi_call)
+        i3 = call_may_force(0, p2, 12345, descr=libffi_call)
+        guard_not_forced() []
+        guard_no_exception() []
         jump(i3, f1, p2)
         """
         expected = ops


From cfbolz at codespeak.net  Thu Oct 14 15:37:30 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 15:37:30 +0200 (CEST)
Subject: [pypy-svn] r77934 - in pypy/extradoc/talk/pepm2011: . benchmarks
Message-ID: <20101014133730.6995336E076@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 15:37:28 2010
New Revision: 77934

Modified:
   pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
(cfbolz, arigo for some of it):

    - add URLs for PyPy
    - add factors to the timing table, to read it more easily
    - change psyco description
    - some fixes in the virtualizables section


Modified: pypy/extradoc/talk/pepm2011/benchmarks/bench.gnumeric
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 15:37:28 2010
@@ -211,7 +211,7 @@
 \subsection{PyPy}
 \label{sub:PyPy}
 
-The work described in this paper was done in the context of the PyPy project
+The work described in this paper was done in the context of the PyPy project\footnote{\texttt{http://pypy.org}}
 \cite{armin_rigo_pypys_2006}. PyPy is an environment where dynamic languages can
 be implemented in a simple yet efficient way. 
 When implementing a language with PyPy one writes an \emph{interpreter}
@@ -933,7 +933,7 @@
 frame-introspection features in many dynamic languages. Languages such
 as Python and Smalltalk allow the programmer to get access to the
 frames object that the interpreter uses to store local variables. This
-is a useful feature, as makes the implementation of a debugger
+is a useful feature, as it makes the implementation of a debugger
 possible in Python without needing much explicit support from the VM
 level. On the other hand, it severely hinders the effectiveness of
 allocation removal, because every time an object is stored into a
@@ -947,12 +947,14 @@
 because a debugger is used) using values from JIT-emitted code runtime
 locations (typically the CPU stack). Therefore in the common case,
 nothing is stored into the frame objects, making the problem of too
-much escaping go away. This is a common approach in VM implementations
-\cite{miranda_context_1999}, the only novelty in our approach lays in
-its generality, all the delicate support code for this is generated, because
-most other JITs are just specifically written for one particular
+much escaping go away. This special handling of frame objects is a common approach in VM implementations
+\cite{miranda_context_1999}. The only novelty in our approach lays in
+its generality: all the delicate support code for this is generated, as opposed to
+most other JITs, which are just specifically written for one particular
 language.
 
+XXX one of the JS tracing JITs does this as well
+
 % subsection Virtualizables (end)
 
 % section Supporting Techniques (end)
@@ -963,7 +965,10 @@
 To evaluate the effectiveness of our allocation removal algorithm, we look at
 the effectiveness when used in the tracing JIT of PyPy's Python interpreter. The
 benchmarks we used are small-to-medium Python programs, some synthetic
-benchmarks, some real applications.
+benchmarks, some real applications.\footnote{All the source code of the
+benchmarks can be found at \texttt{http://codespeak.net/svn/pypy/benchmarks/}.
+There is also a website that monitors PyPy's performance nightly at
+\texttt{http://speed.pypy.org/}.}
 
 XXX [fijal] note that some of those have cpython-specific hacks removed or
 old versions used (nbody mostly I think).
@@ -991,6 +996,7 @@
     framework\footnote{\texttt{http://twistedmatrix.com/}}.
 \end{itemize}
 
+
 We evaluate the allocation removal algorithm along two lines: first we want to
 know how many allocations could be optimized away. On the other hand, we want
 to know how much the run times of the benchmarks is improved.
@@ -1025,7 +1031,7 @@
 \begin{center}
 \begin{tabular}{|l||r|rr|rr|rr|rr|}
 \hline
-	&num loops	&new	&removed	&get/set	&removed &guard &removed	&all ops	&removed\\
+	&num loops      &new    &removed        &get/set        &removed &guard &removed        &all ops        &removed\\
 \hline
 crypto\_pyaes	&78	&3088	&50\%	&57148	&25\%	&9055	&95\%	&137189	&80\%\\
 django	&51	&673	&54\%	&19318	&18\%	&3876	&93\%	&55682	&85\%\\
@@ -1061,32 +1067,30 @@
 All benchmarks are improved by the allocation removal optimization, some as much
 as XXX. XXX Psyco
 
-XXX runtimes of the algorithm somehow?
-
 \begin{figure*}
 \begin{center}
-\begin{tabular}{|l||r|r|r|r|}
+\begin{tabular}{|l||r|r||r|r||r|r||r|r|}
 \hline
-	&CPython	&Psyco	&PyPy w/o optimizations	&PyPy w/ optimizations \\
+	&CPython [ms]	& $\times$ &Psyco [ms]	& $\times$ &PyPy w/o optim. [ms] & $\times$ &PyPy w/ optim. [ms]& $\times$ \\
 \hline
-crypto\_pyaes	&2757.80 $\pm$ 0.98	&67.90 $\pm$ 0.47	&1652.00 $\pm$ 4.00 &266.86 $\pm$ 5.94 \\
-django	&993.19 $\pm$ 0.50	&913.51 $\pm$ 4.22	&694.73 $\pm$ 2.86 &259.53 $\pm$ 1.79 \\
-fannkuch	&1987.22 $\pm$ 2.02	&944.44 $\pm$ 0.61	&566.99 $\pm$ 1.06 &466.87 $\pm$ 1.85 \\
-go	&947.21 $\pm$ 1.58	&445.96 $\pm$ 0.68	&2197.71 $\pm$ 25.21 &316.15 $\pm$ 9.33 \\
-html5lib	&13987.12 $\pm$ 19.51	&17398.25 $\pm$ 36.50	&27194.45 $\pm$ 46.62	&10092.19 $\pm$ 23.50 \\
-meteor-contest	&346.98 $\pm$ 0.35	&215.66 $\pm$ 0.23	&433.04 $\pm$ 1.45 &392.85 $\pm$ 0.87 \\
-nbody\_modified	&637.90 $\pm$ 1.82	&256.78 $\pm$ 0.18	&135.55 $\pm$ 0.33 &103.93 $\pm$ 0.25 \\
-pyflate-fast	&3169.35 $\pm$ 1.89	&1278.16 $\pm$ 3.13	&3285.89 $\pm$ 8.51	&1822.36 $\pm$ 11.52 \\
-raytrace-simple	&2744.60 $\pm$ 51.72	&1072.66 $\pm$ 1.08	&2778.27 $\pm$ 15.13	&647.24 $\pm$ 5.44 \\
-richards	&354.06 $\pm$ 1.00	&63.48 $\pm$ 0.15	&383.93 $\pm$ 3.28 &88.32 $\pm$ 0.91 \\
-spambayes	&299.16 $\pm$ 0.35	&338.68 $\pm$ 3.14	&580.90 $\pm$ 24.68 &397.37 $\pm$ 10.60 \\
-spectral-norm	&478.63 $\pm$ 0.80	&139.83 $\pm$ 1.54	&353.51 $\pm$ 1.39 &112.10 $\pm$ 1.17 \\
-telco	&1207.67 $\pm$ 2.03	&730.00 $\pm$ 2.66	&1296.08 $\pm$ 4.37 &495.23 $\pm$ 2.14 \\
-twisted\_names	&9.58 $\pm$ 0.01	&10.43 $\pm$ 0.01	&17.99 $\pm$ 0.27	&7.13 $\pm$ 0.09 \\
+crypto\_pyaes	&2757.80 $\pm$ 0.98	&10.33	&67.90 $\pm$ 0.47	&0.25	&1652.00 $\pm$ 4.00	&6.19	&266.86 $\pm$ 5.94	&1.00\\
+django	&993.19 $\pm$ 0.50	&3.83	&913.51 $\pm$ 4.22	&3.52	&694.73 $\pm$ 2.86	&2.68	&259.53 $\pm$ 1.79	&1.00\\
+fannkuch	&1987.22 $\pm$ 2.02	&4.26	&944.44 $\pm$ 0.61	&2.02	&566.99 $\pm$ 1.06	&1.21	&466.87 $\pm$ 1.85	&1.00\\
+go	&947.21 $\pm$ 1.58	&3.00	&445.96 $\pm$ 0.68	&1.41	&2197.71 $\pm$ 25.21	&6.95	&316.15 $\pm$ 9.33	&1.00\\
+html5lib	&13987.12 $\pm$ 19.51	&1.39	&17398.25 $\pm$ 36.50	&1.72	&27194.45 $\pm$ 46.62	&2.69	&10092.19 $\pm$ 23.50	&1.00\\
+meteor-contest	&346.98 $\pm$ 0.35	&0.88	&215.66 $\pm$ 0.23	&0.55	&433.04 $\pm$ 1.45	&1.10	&392.85 $\pm$ 0.87	&1.00\\
+nbody\_modified	&637.90 $\pm$ 1.82	&6.14	&256.78 $\pm$ 0.18	&2.47	&135.55 $\pm$ 0.33	&1.30	&103.93 $\pm$ 0.25	&1.00\\
+pyflate-fast	&3169.35 $\pm$ 1.89	&1.74	&1278.16 $\pm$ 3.13	&0.70	&3285.89 $\pm$ 8.51	&1.80	&1822.36 $\pm$ 11.52	&1.00\\
+raytrace-simple	&2744.60 $\pm$ 51.72	&4.24	&1072.66 $\pm$ 1.08	&1.66	&2778.27 $\pm$ 15.13	&4.29	&647.24 $\pm$ 5.44	&1.00\\
+richards	&354.06 $\pm$ 1.00	&4.01	&63.48 $\pm$ 0.15	&0.72	&383.93 $\pm$ 3.28	&4.35	&88.32 $\pm$ 0.91	&1.00\\
+spambayes	&299.16 $\pm$ 0.35	&0.75	&338.68 $\pm$ 3.14	&0.85	&580.90 $\pm$ 24.68	&1.46	&397.37 $\pm$ 10.60	&1.00\\
+spectral-norm	&478.63 $\pm$ 0.80	&4.27	&139.83 $\pm$ 1.54	&1.25	&353.51 $\pm$ 1.39	&3.15	&112.10 $\pm$ 1.17	&1.00\\
+telco	&1207.67 $\pm$ 2.03	&2.44	&730.00 $\pm$ 2.66	&1.47	&1296.08 $\pm$ 4.37	&2.62	&495.23 $\pm$ 2.14	&1.00\\
+twisted\_names	&9.58 $\pm$ 0.01	&1.34	&10.43 $\pm$ 0.01	&1.46	&17.99 $\pm$ 0.27	&2.52	&7.13 $\pm$ 0.09	&1.00\\
 \hline
 \end{tabular}
 \end{center}
-\caption{Benchmark Times in Milliseconds}
+\caption{Benchmark Times in Milliseconds, Together With Factor Over PyPy With Optimizations}
 \label{fig:times}
 \end{figure*}
 
@@ -1121,10 +1125,11 @@
 implementation are given.
 
 Psyco \cite{rigo_representation-based_2004} is a (non-tracing) JIT for Python
-that fully implements allocation removal (the \emph{static} objects are called
-\emph{virtual} objects).  It is a hand-written extension module for CPython.
-Historically, PyPy's JIT can be seen as some successor of Psyco for a
-general context.
+that implements a more ad-hoc version of the allocation removal described here.
+Our static objects could be related to what are called \emph{virtual} objects
+in Psyco.  It is a hand-written extension module for CPython. Historically,
+PyPy's JIT can be seen as some successor of Psyco for a general context (one of
+the authors of this paper is the author of Psyco).
 
 partial evaluation:
 


From antocuni at codespeak.net  Thu Oct 14 15:47:30 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 14 Oct 2010 15:47:30 +0200 (CEST)
Subject: [pypy-svn] r77935 - pypy/branch/jitffi/pypy/jit/metainterp/test
Message-ID: <20101014134730.E4FA3282BEC@codespeak.net>

Author: antocuni
Date: Thu Oct 14 15:47:29 2010
New Revision: 77935

Modified:
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
try to use the full terminal width when displaying the oplists side-by-side 


Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py	Thu Oct 14 15:47:29 2010
@@ -132,14 +132,20 @@
 # ____________________________________________________________
 
 def equaloplists(oplist1, oplist2, strict_fail_args=True, remap={}):
-    print '-'*20, 'Comparing lists', '-'*20
+    # try to use the full width of the terminal to display the list
+    # unfortunately, does not work with the default capture method of py.test
+    # (which is fd), you you need to use either -s or --capture=sys, else you
+    # get the standard 80 columns width
+    totwidth = py.io.get_terminal_width()
+    width = totwidth / 2 - 1
+    print ' Comparing lists '.center(totwidth, '-')
     for op1, op2 in zip(oplist1, oplist2):
         txt1 = str(op1)
         txt2 = str(op2)
         while txt1 or txt2:
-            print '%-39s| %s' % (txt1[:39], txt2[:39])
-            txt1 = txt1[39:]
-            txt2 = txt2[39:]
+            print '%s| %s' % (txt1[:width].ljust(width), txt2[:width])
+            txt1 = txt1[width:]
+            txt2 = txt2[width:]
         assert op1.getopnum() == op2.getopnum()
         assert op1.numargs() == op2.numargs()
         for i in range(op1.numargs()):


From pedronis at codespeak.net  Thu Oct 14 15:56:33 2010
From: pedronis at codespeak.net (pedronis at codespeak.net)
Date: Thu, 14 Oct 2010 15:56:33 +0200 (CEST)
Subject: [pypy-svn] r77936 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014135633.A408636E076@codespeak.net>

Author: pedronis
Date: Thu Oct 14 15:56:31 2010
New Revision: 77936

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
clarifications

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 15:56:31 2010
@@ -930,15 +930,15 @@
 
 One problem to the successful application of the allocation removal
 techniques described in the previous sections is the presence of
-frame-introspection features in many dynamic languages. Languages such
-as Python and Smalltalk allow the programmer to get access to the
-frames object that the interpreter uses to store local variables. This
-is a useful feature, as it makes the implementation of a debugger
-possible in Python without needing much explicit support from the VM
-level. On the other hand, it severely hinders the effectiveness of
-allocation removal, because every time an object is stored into a
-local variable, it is stored into the frame-object, which makes it
-escape.
+frame-introspection features in many dynamic languages and the fact
+that frames are heap allocated. Languages such as Python and Smalltalk
+allow the programmer to get access to the frames object that the
+interpreter uses to store local variables. This is a useful feature,
+as it makes the implementation of a debugger possible in Python
+without needing much explicit support from the VM level. On the other
+hand, it severely hinders the effectiveness of allocation removal,
+because every time an object is stored into a local variable, it is
+stored into the frame-object, which makes it escape.
 
 This problem is solved by making it possible to the interpreter author
 to add some hints into the source code to declare instances of one
@@ -1147,14 +1147,15 @@
 \section{Conclusion}
 \label{sec:conclusion}
 
-In this paper, we used an approach based on online partial evaluation to
-optimize away allocations in the traces of a tracing JIT.  In this context a
-simple approach to partial evaluation gives good results.  This is due
-to the fact that the tracing JIT itself is responsible for all control
-issues, which are usually the hardest part of partial evaluation: the
-tracing JIT selects the parts of the program that are worthwhile to
-optimize, and extracts linear paths through them, inlining functions as
-necessary.  What is left to optimize is only those linear paths.
+In this paper, we used an approach based on online partial evaluation
+to optimize away allocations and type guards in the traces of a
+tracing JIT.  In this context a simple approach to partial evaluation
+gives good results.  This is due to the fact that the tracing JIT
+itself is responsible for all control issues, which are usually the
+hardest part of partial evaluation: the tracing JIT selects the parts
+of the program that are worthwhile to optimize, and extracts linear
+paths through them, inlining functions as necessary.  What is left to
+optimize is only those linear paths.
 
 We expect a similar result for other optimizations that usually require
 a complex analysis phase and are thus normally too slow to use at


From cfbolz at codespeak.net  Thu Oct 14 16:06:18 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 16:06:18 +0200 (CEST)
Subject: [pypy-svn] r77937 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014140618.6E577282BEA@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 16:06:14 2010
New Revision: 77937

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
fix XXX, rename section


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 16:06:14 2010
@@ -86,7 +86,7 @@
 
             $^c$Open End, G?teborg, Sweden
            }
-           {cfbolz at gmx.de \and anto.cuni at gmail.com \and fijal at merlinux.eu,
+           {cfbolz at gmx.de \and anto.cuni at gmail.com \and fijal at merlinux.eu \and
            leuschel at cs.uni-duesseldorf.de \and samuele.pedroni at gmail.com \and arigo at tunes.org}
 
 %\numberofauthors{3}
@@ -191,8 +191,8 @@
 informally described in Section~\ref{sec:statics}; a more formal description is
 given in Section~\ref{sec:formal}.
 
-In Section~\ref{sec:support} we describe some supporting techniques that are not
-central to the approach, but are needed to improve the results. The introduced
+In Section~\ref{sec:frames} we describe some supporting techniques that are not
+central to the approach, but are essential to improve the results. The introduced
 techniques are evaluated in Section~\ref{sec:Evaluation} using PyPy's Python
 interpreter as a case study.
 
@@ -920,11 +920,8 @@
 
 % section Escape Analysis in a Tracing JIT (end)
 
-\section{Supporting Techniques}
-\label{sec:support}
-
-\subsection{Virtualizables}
-\label{sub:Virtualizables}
+\section{Taming Frame Objects}
+\label{sec:frames}
 
 \cfbolz{probably can be cut in case of space problems}
 
@@ -955,9 +952,7 @@
 
 XXX one of the JS tracing JITs does this as well
 
-% subsection Virtualizables (end)
-
-% section Supporting Techniques (end)
+% section Taming Frame Objects (end)
 
 \section{Evaluation}
 \label{sec:Evaluation}
@@ -1060,12 +1055,18 @@
 All benchmarks were run 50 times in the same process, to give the JIT time to produce machine
 code. The arithmetic mean of the times of the last 30 runs were used as the
 result. The errors were computed using a confidence interval with a 95\%
-confidence level \cite{georges_statistically_2007}. The results are reported in Figure~\ref{fig:times}.
+confidence level \cite{georges_statistically_2007}. The results are reported in
+Figure~\ref{fig:times}. In addition to the run times the table also reports the
+speedup that PyPy with optimization turned on achieves.
 With the optimization turned on, PyPy's Python interpreter outperforms CPython
 in all benchmarks except spambayes (which heavily relies on regular expression
 performance and thus is not helped much by our Python JIT) and meteor-contest.
-All benchmarks are improved by the allocation removal optimization, some as much
-as XXX. XXX Psyco
+All benchmarks are improved by the allocation removal optimization, by at least
+20\% and by as much as a factor of 6.95.
+
+Psyco is able to outperform PyPy's JIT in five out of 14 benchmarks. We hope to
+overtake Psyco (which is no longer being actively developped) by adding some
+further optimizations.
 
 \begin{figure*}
 \begin{center}


From cfbolz at codespeak.net  Thu Oct 14 16:07:07 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 16:07:07 +0200 (CEST)
Subject: [pypy-svn] r77938 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014140707.9F9E0282BEC@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 16:07:06 2010
New Revision: 77938

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
I think now that there is the explicit link to the source code of the
benchmarks, this XXX can go


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 16:07:06 2010
@@ -965,9 +965,6 @@
 There is also a website that monitors PyPy's performance nightly at
 \texttt{http://speed.pypy.org/}.}
 
-XXX [fijal] note that some of those have cpython-specific hacks removed or
-old versions used (nbody mostly I think).
-
 Some of them are from the Computer Language Benchmark
 Game\footnote{\texttt{http://shootout.alioth.debian.org/}}: \textbf{fannkuch},
 \textbf{nbody}, \textbf{meteor-contest}, \textbf{spectral-norm}.


From cfbolz at codespeak.net  Thu Oct 14 16:10:54 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 16:10:54 +0200 (CEST)
Subject: [pypy-svn] r77939 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014141054.CBD65282BEB@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 16:10:53 2010
New Revision: 77939

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
fix another XXX


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 16:10:53 2010
@@ -945,13 +945,11 @@
 locations (typically the CPU stack). Therefore in the common case,
 nothing is stored into the frame objects, making the problem of too
 much escaping go away. This special handling of frame objects is a common approach in VM implementations
-\cite{miranda_context_1999}. The only novelty in our approach lays in
+\cite{miranda_context_1999,andreas_gal_trace-based_2009}. The only novelty in our approach lays in
 its generality: all the delicate support code for this is generated, as opposed to
 most other JITs, which are just specifically written for one particular
 language.
 
-XXX one of the JS tracing JITs does this as well
-
 % section Taming Frame Objects (end)
 
 \section{Evaluation}


From cfbolz at codespeak.net  Thu Oct 14 16:12:06 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 16:12:06 +0200 (CEST)
Subject: [pypy-svn] r77940 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014141206.44DB3282BEC@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 16:12:04 2010
New Revision: 77940

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
I decided to ignore this comment, because that is dealt with by tracing itself,
not by the optimization that the paper is about


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 16:12:04 2010
@@ -140,10 +140,6 @@
 dispatching. Those are problems that are usually not present or at least less
 severe in statically typed languages.
 
-XXX [fijal] I would say that the other goal of the JIT in case of dynamic
-languages is to compile only a common scenario and leave a guard (instead
-of proving that something will never ever happen)
-
 Boxing of primitive types is necessary because dynamic languages need to be able to handle
 all objects, even integers, floats, booleans etc. in the same way as user-defined
 instances. Thus those primitive types are usually \emph{boxed}, i.e. a small


From arigo at codespeak.net  Thu Oct 14 17:15:19 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 17:15:19 +0200 (CEST)
Subject: [pypy-svn] r77941 - pypy/trunk/pypy/module/_rawffi
Message-ID: <20101014151519.4AED4282BEA@codespeak.net>

Author: arigo
Date: Thu Oct 14 17:15:12 2010
New Revision: 77941

Modified:
   pypy/trunk/pypy/module/_rawffi/array.py
Log:
Add a workaround for the libffi x86_64 bug.


Modified: pypy/trunk/pypy/module/_rawffi/array.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/array.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/array.py	Thu Oct 14 17:15:12 2010
@@ -97,7 +97,14 @@
 
 class W_ArrayInstance(W_DataInstance):
     def __init__(self, space, shape, length, address=r_uint(0)):
-        W_DataInstance.__init__(self, space, shape.size * length, address)
+        # XXX workaround for a bug in libffi on x86_64: make sure that
+        # we always have at least 8 bytes.  For W_ArrayInstances that are
+        # used as the result value of a function call, ffi_call() writes
+        # 8 bytes into it even if the function's result type asks for less.
+        memsize = shape.size * length
+        if memsize < 8:
+            memsize = 8
+        W_DataInstance.__init__(self, space, memsize, address)
         self.length = length
         self.shape = shape
 


From arigo at codespeak.net  Thu Oct 14 17:16:01 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 17:16:01 +0200 (CEST)
Subject: [pypy-svn] r77942 - pypy/branch/32ptr-on-64bit/pypy/module/_rawffi
Message-ID: <20101014151601.F06FF282BEA@codespeak.net>

Author: arigo
Date: Thu Oct 14 17:15:59 2010
New Revision: 77942

Modified:
   pypy/branch/32ptr-on-64bit/pypy/module/_rawffi/array.py
Log:
Merge r77941 from trunk.


Modified: pypy/branch/32ptr-on-64bit/pypy/module/_rawffi/array.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/module/_rawffi/array.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/module/_rawffi/array.py	Thu Oct 14 17:15:59 2010
@@ -97,7 +97,14 @@
 
 class W_ArrayInstance(W_DataInstance):
     def __init__(self, space, shape, length, address=r_uint(0)):
-        W_DataInstance.__init__(self, space, shape.size * length, address)
+        # XXX workaround for a bug in libffi on x86_64: make sure that
+        # we always have at least 8 bytes.  For W_ArrayInstances that are
+        # used as the result value of a function call, ffi_call() writes
+        # 8 bytes into it even if the function's result type asks for less.
+        memsize = shape.size * length
+        if memsize < 8:
+            memsize = 8
+        W_DataInstance.__init__(self, space, memsize, address)
         self.length = length
         self.shape = shape
 


From cfbolz at codespeak.net  Thu Oct 14 17:24:52 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 17:24:52 +0200 (CEST)
Subject: [pypy-svn] r77943 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014152452.8353B282BEA@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 17:24:50 2010
New Revision: 77943

Modified:
   pypy/extradoc/talk/pepm2011/paper.bib
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
mention self


Modified: pypy/extradoc/talk/pepm2011/paper.bib
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.bib	(original)
+++ pypy/extradoc/talk/pepm2011/paper.bib	Thu Oct 14 17:24:50 2010
@@ -296,6 +296,19 @@
 	pages = {144--153}
 },
 
+ at article{chambers_efficient_1989,
+	title = {An efficient implementation of {SELF} a dynamically-typed object-oriented language based on prototypes},
+	volume = {24},
+	url = {http://portal.acm.org/citation.cfm?id=74884},
+	doi = {10.1145/74878.74884},
+	abstract = {We have developed and implemented techniques that double the performance of dynamically-typed object-oriented languages. Our {SELF} implementation runs twice as fast as the fastest Smalltalk implementation, despite {SELF's} lack of classes and explicit variables. To compensate for the absence of classes, our system uses implementation-level maps to transparently group objects cloned from the same prototype, providing data type information and eliminating the apparent space overhead for prototype-based systems. To compensate for dynamic typing, user-defined control structures, and the lack of explicit variables, our system dynamically compiles multiple versions of a source method, each customized according to its receiver's map. Within each version the type of the receiver is fixed, and thus the compiler can statically bind and inline all messages sent to self. Message splitting and type prediction extract and preserve even more static type information, allowing the compiler to inline many other messages. Inlining dramatically improves performance and eliminates the need to hard-wire low-level methods such as +,==, and {ifTrue:.} Despite inlining and other optimizations, our system still supports interactive programming environments. The system traverses internal dependency lists to invalidate all compiled methods affected by a programming change. The debugger reconstructs inlined stack frames from compiler-generated debugging information, making inlining invisible to the {SELF} programmer.},
+	number = {10},
+	journal = {{SIGPLAN} Not.},
+	author = {C. Chambers and D. Ungar and E. Lee},
+	year = {1989},
+	pages = {49--70},
+},
+
 @inproceedings{rigo_representation-based_2004,
 	address = {Verona, Italy},
 	title = {Representation-based just-in-time specialization and the Psyco prototype for Python},

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 17:24:50 2010
@@ -1123,6 +1123,13 @@
 PyPy's JIT can be seen as some successor of Psyco for a general context (one of
 the authors of this paper is the author of Psyco).
 
+The original SELF JIT compiler \cite{chambers_efficient_1989} used an algorithm for
+forward-propagating the types of variables as part of its optimizations. This
+makes it possible to remove all type checks on a variable but the first one.
+The optimization does not deal with removing the full object, if it is
+short-lived, but the type check removals are similar to what our optimization
+achieves.
+
 partial evaluation:
 
 Prolog:
@@ -1134,8 +1141,6 @@
 
 xxx: relation to compile-time garbage collection \cite{mazur_practical_2001};  separation logic; John Hughes: type specialization
 
-XXX cite SELF paper for the type propagation effects
-
 \section{Conclusion}
 \label{sec:conclusion}
 


From antocuni at codespeak.net  Thu Oct 14 17:25:39 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 14 Oct 2010 17:25:39 +0200 (CEST)
Subject: [pypy-svn] r77944 - in pypy/branch/jitffi/pypy/jit/metainterp:
	optimizeopt test
Message-ID: <20101014152539.2C39F282BEC@codespeak.net>

Author: antocuni
Date: Thu Oct 14 17:25:37 2010
New Revision: 77944

Modified:
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
disable the ffi call optimization if we find any operation between the various
prepare_call/push_arg/do_call (the code in rlib/libffi.py is written in such a
way that we should never get any operation in between)



Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	Thu Oct 14 17:25:37 2010
@@ -12,10 +12,11 @@
     restype = None
     descr = None
 
-    def __init__(self, funcval, cpu):
+    def __init__(self, funcval, cpu, prepare_op):
         self.opargs = []
         argtypes, restype = self._get_signature(funcval)
         self.descr = cpu.calldescrof_dynamic(argtypes, restype)
+        self.prepare_op = prepare_op
 
     def _get_signature(self, funcval):
         """
@@ -64,7 +65,8 @@
 class OptFfiCall(Optimization):
 
     def __init__(self):
-        self.func_infos = {}
+        self.funcval = None
+        self.funcinfo = None
 
     def _get_oopspec(self, op):
         effectinfo = op.getdescr().get_extra_info()
@@ -72,6 +74,18 @@
             return effectinfo.oopspecindex
         return EffectInfo.OS_NONE
 
+    def rollback(self):
+        self.emit_operation(self.funcinfo.prepare_op)
+        for op in self.funcinfo.opargs:
+            self.emit_operation(op)
+        self.funcval = None
+        self.funcinfo = None
+
+    def optimize_default(self, op):
+        if self.funcval:
+            self.rollback()
+        self.emit_operation(op)
+
     def optimize_CALL(self, op):
         oopspec = self._get_oopspec(op)
         try:
@@ -92,31 +106,40 @@
 
     def _get_funcval(self, op):
         funcval = self.getvalue(op.getarg(1))
+        if self.funcval:
+            assert self.funcval is funcval # XXX do something nice
         if not funcval.is_constant():
             raise NonConstantFuncVal
         return funcval
 
     def do_prepare_call(self, op):
         funcval = self._get_funcval(op)
-        assert funcval not in self.func_infos # XXX: do something nice etc. etc.
-        self.func_infos[funcval] = FuncInfo(funcval, self.optimizer.cpu)
+        assert self.funcval is None # XXX: do something nice etc. etc.
+        self.funcval = funcval
+        self.funcinfo = FuncInfo(funcval, self.optimizer.cpu, op)
 
     def do_push_arg(self, op):
         # we store the op in funcs because we might want to emit it later,
         # in case we give up with the optimization
+        if self.funcval is None:
+            self.emit_operation(op)
+            return
         funcval = self._get_funcval(op)
-        self.func_infos[funcval].opargs.append(op)
+        self.funcinfo.opargs.append(op)
 
     def do_call(self, op):
+        if self.funcval is None:
+            return op
         funcval = self._get_funcval(op)
-        info = self.func_infos[funcval]
+        info = self.funcinfo
         funcsymval = self.getvalue(op.getarg(2))
         arglist = [funcsymval.force_box()]
         for push_op in info.opargs:
             argval = self.getvalue(push_op.getarg(2))
             arglist.append(argval.force_box())
         newop = ResOperation(rop.CALL_MAY_FORCE, arglist, op.result, descr=info.descr)
-        del self.func_infos[funcval]
+        self.funcval = None
+        self.funcinfo = None
         return newop
 
     def propagate_forward(self, op):
@@ -126,6 +149,6 @@
                 func(self, op)
                 break
         else:
-            self.emit_operation(op)
+            self.optimize_default(op)
 
 optimize_ops = _findall(OptFfiCall, 'optimize_')

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	Thu Oct 14 17:25:37 2010
@@ -85,3 +85,17 @@
         expected = ops
         loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
 
+    def test_rollback_if_op_in_between(self):
+        ops = """
+        [i0, f1]
+        call(0, ConstPtr(func),             descr=libffi_prepare)
+        call(0, ConstPtr(func), i0,         descr=libffi_push_arg)
+        i1 = int_add(i0, 1)
+        call(0, ConstPtr(func), f1,         descr=libffi_push_arg)
+        i3 = call_may_force(0, ConstPtr(func), 12345, descr=libffi_call)
+        guard_not_forced() []
+        guard_no_exception() []
+        jump(i3, f1)
+        """
+        expected = ops
+        loop = self.optimize_loop(ops, 'Not, Not', expected)

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py	Thu Oct 14 17:25:37 2010
@@ -139,6 +139,7 @@
     totwidth = py.io.get_terminal_width()
     width = totwidth / 2 - 1
     print ' Comparing lists '.center(totwidth, '-')
+    print '%s| %s' % ('optimized'.center(width), 'expected'.center(width))
     for op1, op2 in zip(oplist1, oplist2):
         txt1 = str(op1)
         txt2 = str(op2)


From cfbolz at codespeak.net  Thu Oct 14 17:27:07 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 17:27:07 +0200 (CEST)
Subject: [pypy-svn] r77945 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014152707.1720F282BEF@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 17:27:05 2010
New Revision: 77945

Modified:
   pypy/extradoc/talk/pepm2011/paper.bib
Log:
full information for prolog JIT paper


Modified: pypy/extradoc/talk/pepm2011/paper.bib
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.bib	(original)
+++ pypy/extradoc/talk/pepm2011/paper.bib	Thu Oct 14 17:27:05 2010
@@ -38,15 +38,6 @@
 	pages = {152--160}
 },
 
- at inproceedings{carl_friedrich_bolz_towards_2010,
-	address = {Hagenberg, Austria},
-	title = {Towards a Jitting {VM} for Prolog Execution},
-	abstract = {Most Prolog implementations are implemented in low-level languages such as C and are based on a variation of the {WAM} instruction set, which enhances their performance but makes them hard to write. In addition, many of the more dynamic features of Prolog (like assert), despite their popularity, are not well supported. We present a high-level continuation-based Prolog interpreter based on the {PyPy} project. The {PyPy} project makes it possible to easily and efficiently implement dynamic languages. It provides tools that automatically generate a just-in-time compiler for a given interpreter of the target language, by using partial evaluation techniques. The resulting Prolog implementation is surprisingly efficient: it clearly outperforms existing interpreters of Prolog in high-level languages such as Java. Moreover, on some benchmarks, our system outperforms state-of-the-art {WAM-based} Prolog implementations. Our paper aims to show that declarative languages such as Prolog can indeed benefit from having a just-in-time compiler and that {PyPy} can form the basis for implementing programming languages other than Python.},
-	author = {Carl Friedrich Bolz and Michael Leuschel and David Schneider},
-	year = {2010},
-	note = {accepted for publication}
-},
-
 @misc{_property_????,
 	title = {Property cache - {MDC}},
 	url = {https://developer.mozilla.org/en/SpiderMonkey/Internals/Property_cache},
@@ -130,6 +121,20 @@
 	note = {Technical Report {DISI-TH-2010-05}}
 },
 
+ at inproceedings{carl_friedrich_bolz_towards_2010,
+	address = {Hagenberg, Austria},
+	title = {Towards a Jitting {VM} for Prolog execution},
+	isbn = {978-1-4503-0132-9},
+	url = {http://portal.acm.org/citation.cfm?id=1836102},
+	doi = {10.1145/1836089.1836102},
+	abstract = {Most Prolog implementations are implemented in low-level languages such as C and are based on a variation of the {WAM} instruction set, which enhances their performance but makes them hard to write. In addition, many of the more dynamic features of Prolog (like assert), despite their popularity, are not well supported. We present a high-level continuation-based Prolog interpreter based on the {PyPy} project. The {PyPy} project makes it possible to easily and efficiently implement dynamic languages. It provides tools that automatically generate a just-in-time compiler for a given interpreter of the target language, by using partial evaluation techniques. The resulting Prolog implementation is surprisingly efficient: it clearly outperforms existing interpreters of Prolog in high-level languages such as Java. Moreover, on some benchmarks, our system outperforms state-of-the-art {WAM-based} Prolog implementations. Our paper aims to show that declarative languages such as Prolog can indeed benefit from having a just-in-time compiler and that {PyPy} can form the basis for implementing programming languages other than Python.},
+	booktitle = {Proceedings of the 12th international {ACM} {SIGPLAN} symposium on Principles and practice of declarative programming},
+	publisher = {{ACM}},
+	author = {Carl Friedrich Bolz and Michael Leuschel and David Schneider},
+	year = {2010},
+	pages = {99--108}
+},
+
 @inproceedings{chang_tracing_2009,
 	address = {Washington, {DC,} {USA}},
 	title = {Tracing for Web 3.0: Trace Compilation for the Next Generation Web Applications},


From david at codespeak.net  Thu Oct 14 17:37:19 2010
From: david at codespeak.net (david at codespeak.net)
Date: Thu, 14 Oct 2010 17:37:19 +0200 (CEST)
Subject: [pypy-svn] r77946 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . test
Message-ID: <20101014153719.CD52736E076@codespeak.net>

Author: david
Date: Thu Oct 14 17:37:17 2010
New Revision: 77946

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
Log:
Encode register based data processing and generate functions for them.


Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Thu Oct 14 17:37:17 2010
@@ -1,4 +1,5 @@
 import conditions as cond
+import registers as reg
 from pypy.rlib.rmmap import alloc
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.jit.backend.arm.instruction_builder import define_instructions
@@ -26,14 +27,6 @@
                         | (rd & 0xF) << 12
                         | (imm & 0xFFF))
 
-
-    def MOV_rr(self, rd, rm, cond=cond.AL, s=0):
-        self.write32(cond << 28
-                    | 0xD << 21
-                    | (s & 0x1) << 20
-                    | (rd & 0xFF) << 12
-                    | (rm & 0xFF))
-
     def MOV_ri(self, rt, imm=0, cond=cond.AL):
         # XXX Check the actual allowed size for imm
         # XXX S bit
@@ -44,27 +37,8 @@
                     | (rt & 0xF) << 12
                     | (imm & 0xFFF))
 
-    def ASR_ri(self, rd, rm, imm=0, cond=cond.AL, s=0):
-        self.write32(cond << 28
-                    | 0xD << 21
-                    | (s & 0x1) << 20
-                    | (rd & 0xF) << 12
-                    | (imm & 0x1F) << 7
-                    | 0x4 << 4
-                    | (rm & 0xF))
-
-    #XXX encode shifttype correctly
-    def ORR_rr(self, rd, rn, rm, imm=0, cond=cond.AL, s=0, shifttype=0):
-        self.write32(cond << 28
-                    | 0x3 << 23
-                    | (s & 0x1) << 20
-                    | (rn & 0xFF) << 16
-                    | (rd & 0xFF) << 12
-                    | (imm & 0x1F) << 7
-                    | (shifttype & 0x3) << 5
-                    | (rm & 0xFF))
-
     def PUSH(self, regs, cond=cond.AL):
+        assert reg.sp not in regs
         instr = self._encode_reg_list(cond << 28 | 0x92D << 16, regs)
         self.write32(instr)
 

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	Thu Oct 14 17:37:17 2010
@@ -1,7 +1,7 @@
 from pypy.jit.backend.arm import conditions as cond
 from pypy.jit.backend.arm import instructions
 
-def define_load_store_func(target, name, table):
+def define_load_store_func(name, table):
     #  XXX W and P bits are not encoded yet
     n = (0x1 << 26
         | (table['A'] & 0x1) << 25
@@ -30,14 +30,60 @@
                         | (p & 0x1) <<  24
                         | (u & 0x1) << 23
                         | (w & 0x1) << 21
-                        | (rn & 0xFF) << 16
-                        | (rt & 0xFF) << 12
+                        | reg_operation(rt, rn, rm, imm, s, shifttype))
+    return f
+
+def define_data_proc(name, table):
+    n = ((table['op1'] & 0x1F) << 20
+        | (table['op2'] & 0x1F) << 7
+        | (table['op3'] & 0x3) << 5)
+    if name[-2:] == 'ri':
+        def f(self, rd, rm, imm=0, cond=cond.AL, s=0):
+            if table['op2cond'] == '!0':
+                assert imm != 0
+            elif table['op2cond'] == '0':
+                assert imm == 0
+            self.write32(n
+                        | cond << 28
+                        | (s & 0x1) << 20
+                        | (rd & 0xFF) << 12
                         | (imm & 0x1F) << 7
-                        | (shifttype & 0x3) << 5
                         | (rm & 0xFF))
 
-    setattr(target, name, f)
+    elif not table['result']:
+        # ops without result
+        def f(self, rn, rm, imm=0, cond=cond.AL, s=0, shifttype=0):
+            self.write32(n
+                        | cond << 28
+                        | reg_operation(0, rn, rm, imm, s, shifttype))
+    elif not table['base']:
+        # ops without base register
+        def f(self, rd, rm, imm=0, cond=cond.AL, s=0, shifttype=0):
+            self.write32(n
+                        | cond << 28
+                        | reg_operation(rd, 0, rm, imm, s, shifttype))
+    else:
+        def f(self, rd, rn, rm, imm=0, cond=cond.AL, s=0, shifttype=0):
+            self.write32(n
+                        | cond << 28
+                        | reg_operation(rd, rn, rm, imm, s, shifttype))
+    return f
+
+
+def reg_operation(rt, rn, rm, imm, s, shifttype):
+    # XXX encode shiftype correctly
+    return ((s & 0x1 << 20)
+            | (rn & 0xFF) << 16
+            | (rt & 0xFF) << 12
+            | (imm & 0x1F) << 7
+            | (shifttype & 0x3) << 5
+            | (rm & 0xFF))
 
 def define_instructions(target):
     for key, val in instructions.load_store.iteritems():
-        define_load_store_func(target, key, val)
+        f = define_load_store_func(key, val)
+        setattr(target, key, f)
+
+    for key, val in instructions.data_proc.iteritems():
+        f = define_data_proc(key, val)
+        setattr(target, key, f)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	Thu Oct 14 17:37:17 2010
@@ -10,3 +10,25 @@
     'LDRB_ri': {'A':0, 'op1': 0x5, 'B': 0, 'imm': True},
     'LDRB_rr': {'A':1, 'op1': 0x5, 'B': 0, 'imm': False},
 }
+
+data_proc = {
+    'AND_rr': {'op1':0x0, 'op2':0, 'op3':0, 'result':True, 'base':True},
+    'EOR_rr': {'op1':0x2, 'op2':0, 'op3':0, 'result':True, 'base':True},
+    'SUB_rr': {'op1':0x4, 'op2':0, 'op3':0, 'result':True, 'base':True},
+    'RSB_rr': {'op1':0x6, 'op2':0, 'op3':0, 'result':True, 'base':True},
+    'ADD_rr': {'op1':0x8, 'op2':0, 'op3':0, 'result':True, 'base':True},
+    'ADC_rr': {'op1':0xA, 'op2':0, 'op3':0, 'result':True, 'base':True},
+    'SBC_rr': {'op1':0xC, 'op2':0, 'op3':0, 'result':True, 'base':True},
+    'RSC_rr': {'op1':0xE, 'op2':0, 'op3':0, 'result':True, 'base':True},
+    'TST_rr': {'op1':0x11, 'op2':0, 'op3':0, 'result':False, 'base':True},
+    'TEQ_rr': {'op1':0x13, 'op2':0, 'op3':0, 'result':False, 'base':True},
+    'CMP_rr': {'op1':0x15, 'op2':0, 'op3':0, 'result':False, 'base':True},
+    'CMN_rr': {'op1':0x17, 'op2':0, 'op3':0, 'result':False, 'base':True},
+    'ORR_rr': {'op1':0x18, 'op2':0, 'op3':0, 'result':True, 'base':True},
+    'MOV_rr': {'op1':0x1A, 'op2':0, 'op3':0, 'result':True, 'base':False},
+    'LSL_ri': {'op1':0x1A, 'op2':0x0, 'op3':0, 'op2cond':'!0', 'result':False, 'base':True},
+    'LSR_ri': {'op1':0x1A, 'op2':0, 'op3':0x1, 'op2cond':'', 'result':False, 'base':True},
+    'ASR_ri': {'op1':0x1A, 'op2':0, 'op3':0x2, 'op2cond':'', 'result':False, 'base':True},
+    #'RRX_ri': {'op1':0x1A, 'op2':0, 'op3':0x3, 'op2cond':'0', 'result':False, 'base':True},
+    'ROR_ri': {'op1':0x1A, 'op2':0x0, 'op3':0x3, 'op2cond':'!0', 'result':True, 'base':False},
+}

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/gen.py	Thu Oct 14 17:37:17 2010
@@ -38,7 +38,7 @@
         res = self.body % (self.instr)
         self.file.write(res)
         self.file.flush()
-        os.system("%s %s %s -o %s/a.out" % (AS, self.asm_opts, self.name, self.tmpdir))
+        os.system("%s --fatal-warnings %s %s -o %s/a.out" % (AS, self.asm_opts, self.name, self.tmpdir))
 
     def __del__(self):
         self.file.close()

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	Thu Oct 14 17:37:17 2010
@@ -82,8 +82,8 @@
         self.assert_equal('PUSH {r1}')
 
     def test_push_multiple(self):
-        self.cb.PUSH([r.r3, r.r1, r.r6, r.r8, r.sp, r.pc])
-        self.assert_equal('PUSH {r3, r1, r6, r8, sp, pc}')
+        self.cb.PUSH([r.r1, r.r3, r.r6, r.r8, r.pc])
+        self.assert_equal('PUSH {r1, r3, r6, r8, pc}')
 
     def test_push_multiple2(self):
         self.cb.PUSH([r.fp, r.ip, r.lr, r.pc])
@@ -126,7 +126,11 @@
         else:
             f = gen_test_reg_func
         test = f(key, value)
-    setattr(TestInstrCodeBuilderForGeneratedInstr, 'test_%s' % key, test)
+        setattr(TestInstrCodeBuilderForGeneratedInstr, 'test_%s' % key, test)
+
+    for key, value, in instructions.data_proc.iteritems():
+        test = gen_test_data_reg_func(key, value)
+        setattr(TestInstrCodeBuilderForGeneratedInstr, 'test_%s' % key, test)
 
 def gen_test_imm_func(name, table):
     def f(self):
@@ -141,4 +145,25 @@
         func(r.r3, r.r7, r.r12)
         self.assert_equal('%s r3, [r7, r12]' % name[:name.index('_')])
     return f
+
+def gen_test_data_reg_func(name, table):
+    if name[-2:] == 'ri':
+        def f(self):
+            func = getattr(self.cb, name)
+            func(r.r3, r.r7, 12)
+            self.assert_equal('%s r3, r7, #12' % name[:name.index('_')])
+
+    elif table['base'] and table['result']:
+        def f(self):
+            func = getattr(self.cb, name)
+            func(r.r3, r.r7, r.r12)
+            self.assert_equal('%s r3, r7, r12' % name[:name.index('_')])
+    else:
+        def f(self):
+            func = getattr(self.cb, name)
+            func(r.r3, r.r7)
+            self.assert_equal('%s r3, r7' % name[:name.index('_')])
+
+    return f
+
 build_tests()


From antocuni at codespeak.net  Thu Oct 14 17:39:06 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 14 Oct 2010 17:39:06 +0200 (CEST)
Subject: [pypy-svn] r77947 -
	pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt
Message-ID: <20101014153906.49CD436E076@codespeak.net>

Author: antocuni
Date: Thu Oct 14 17:39:04 2010
New Revision: 77947

Modified:
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
Log:
simplify the logic, it's much more readable now


Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	Thu Oct 14 17:39:04 2010
@@ -59,9 +59,6 @@
             return f.inst_argtypes, f.inst_restype
 
 
-class NonConstantFuncVal(Exception):
-    pass
-
 class OptFfiCall(Optimization):
 
     def __init__(self):
@@ -88,18 +85,14 @@
 
     def optimize_CALL(self, op):
         oopspec = self._get_oopspec(op)
-        try:
-            if oopspec == EffectInfo.OS_LIBFFI_PREPARE:
-                self.do_prepare_call(op)
-            elif oopspec == EffectInfo.OS_LIBFFI_PUSH_ARG:
-                self.do_push_arg(op)
-            elif oopspec == EffectInfo.OS_LIBFFI_CALL:
-                op = self.do_call(op)
-                self.emit_operation(op)
-            else:
-                raise NonConstantFuncVal # it's not a libffi call
-        except NonConstantFuncVal:
-            # normal case
+        if oopspec == EffectInfo.OS_LIBFFI_PREPARE:
+            op = self.do_prepare_call(op)
+        elif oopspec == EffectInfo.OS_LIBFFI_PUSH_ARG:
+            op = self.do_push_arg(op)
+        elif oopspec == EffectInfo.OS_LIBFFI_CALL:
+            op = self.do_call(op)
+        #
+        if op:
             self.emit_operation(op)
 
     optimize_CALL_MAY_FORCE = optimize_CALL
@@ -109,21 +102,20 @@
         if self.funcval:
             assert self.funcval is funcval # XXX do something nice
         if not funcval.is_constant():
-            raise NonConstantFuncVal
+            return None
         return funcval
 
     def do_prepare_call(self, op):
         funcval = self._get_funcval(op)
+        if not funcval:
+            return op
         assert self.funcval is None # XXX: do something nice etc. etc.
         self.funcval = funcval
         self.funcinfo = FuncInfo(funcval, self.optimizer.cpu, op)
 
     def do_push_arg(self, op):
-        # we store the op in funcs because we might want to emit it later,
-        # in case we give up with the optimization
         if self.funcval is None:
-            self.emit_operation(op)
-            return
+            return op
         funcval = self._get_funcval(op)
         self.funcinfo.opargs.append(op)
 


From antocuni at codespeak.net  Thu Oct 14 17:43:47 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 14 Oct 2010 17:43:47 +0200 (CEST)
Subject: [pypy-svn] r77948 - in pypy/branch/jitffi/pypy/jit/metainterp:
	optimizeopt test
Message-ID: <20101014154347.95C7636E076@codespeak.net>

Author: antocuni
Date: Thu Oct 14 17:43:46 2010
New Revision: 77948

Modified:
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
Log:
add one more case when to rollback


Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	Thu Oct 14 17:43:46 2010
@@ -100,7 +100,9 @@
     def _get_funcval(self, op):
         funcval = self.getvalue(op.getarg(1))
         if self.funcval:
-            assert self.funcval is funcval # XXX do something nice
+            if self.funcval is not funcval:
+                self.rollback()
+                return None
         if not funcval.is_constant():
             return None
         return funcval

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	Thu Oct 14 17:43:46 2010
@@ -40,6 +40,9 @@
         func = FakeLLObject(_fake_class=Func,
                             argtypes=[types.sint, types.double],
                             restype=types.sint)
+        func2 = FakeLLObject(_fake_class=Func,
+                             argtypes=[types.sint, types.double],
+                             restype=types.sint)
         #
         def calldescr(cpu, FUNC, oopspecindex):
             einfo = EffectInfo([], [], [], oopspecindex=oopspecindex)
@@ -99,3 +102,21 @@
         """
         expected = ops
         loop = self.optimize_loop(ops, 'Not, Not', expected)
+
+    def test_rollback_multiple_calls(self):
+        ops = """
+        [i0, i2, f1]
+        call(0, ConstPtr(func),             descr=libffi_prepare)
+        call(0, ConstPtr(func), i0,         descr=libffi_push_arg)
+        call(0, ConstPtr(func2),             descr=libffi_prepare) # culprit!
+        call(0, ConstPtr(func), f1,         descr=libffi_push_arg)
+        i3 = call_may_force(0, ConstPtr(func), 12345, descr=libffi_call)
+        call(0, ConstPtr(func2), i0,         descr=libffi_push_arg)
+        call(0, ConstPtr(func2), f1,         descr=libffi_push_arg)
+        i4 = call_may_force(0, ConstPtr(func), 12345, descr=libffi_call)
+        guard_not_forced() []
+        guard_no_exception() []
+        jump(i3, i4, f1)
+        """
+        expected = ops
+        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)


From cfbolz at codespeak.net  Thu Oct 14 17:45:10 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 17:45:10 +0200 (CEST)
Subject: [pypy-svn] r77949 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014154510.883B036E076@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 17:45:08 2010
New Revision: 77949

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
flesh out section about tracing


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 17:45:08 2010
@@ -1132,9 +1132,14 @@
 
 partial evaluation:
 
-Prolog:
-partially static datastructures are already built-in to Prolog and was built-into partial
-evaluation from the beginning \cite{lloyd_partial_1991}.
+Partially known data structures are built directly into Prolog (via unbound
+logic variables) and thus the optimization of partially static data structures
+was part of partial evaluation of Prolog programs from the beginning
+\cite{lloyd_partial_1991}. One effect of unfolding in Prolog is that terms that
+are constructed and immediately matched again completely disappear in the
+residual program, which is equivalent to what our optimization does for an
+imperative language.
+
  FP
 
 partially static data structures: kenichi asai's thesis?


From cfbolz at codespeak.net  Thu Oct 14 18:22:17 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 18:22:17 +0200 (CEST)
Subject: [pypy-svn] r77951 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014162217.5F81E282BEC@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 18:22:11 2010
New Revision: 77951

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
fixes by david


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 18:22:11 2010
@@ -132,11 +132,11 @@
 
 The goal of a just-in-time (JIT) compiler for a dynamic language is obviously to
 improve the speed of the language over an implementation of the language that
-uses interpretation. The first goal of a JIT is thus to remove the
+uses interpretation. The first goal of a JIT is therefore to remove the
 interpretation overhead, i.e. the overhead of bytecode (or AST) dispatch and the
 overhead of the interpreter's data structures, such as operand stack etc. The
 second important problem that any JIT for a dynamic language needs to solve is
-how to deal with the overhead of boxing of primitive types and of type
+how to deal with the overhead of boxing primitive types and of type
 dispatching. Those are problems that are usually not present or at least less
 severe in statically typed languages.
 
@@ -167,7 +167,7 @@
 simplicity. They can often be added to an interpreter and a lot of the
 infrastructure of the interpreter can be reused. They give some important
 optimizations like inlining and constant-folding for free. A tracing JIT always
-produces linear pieces of code, which simplifies many optimizations that are usually
+produces linear pieces of code, which simplifies many algorithms that are usually
 hard in a compiler, such as register allocation.
 
 The usage of a tracing JIT can remove the overhead of bytecode dispatch and that
@@ -262,7 +262,7 @@
 The core idea of tracing JITs is to focus the optimization effort of the JIT
 compiler on the hot paths of the core loops of the program and to just use an
 interpreter for the less commonly executed parts. VMs that use a tracing JIT are
-thus mixed-mode execution environments, they contain both an interpreter and a
+mostly mixed-mode execution environments, they contain both an interpreter and a
 JIT compiler. By default the interpreter is used to execute the program, doing
 some light-weight profiling at the same time. This profiling is used to identify
 the hot loops of the program. If a hot loop is found in that way, the
@@ -276,7 +276,7 @@
 
 This trace of operations is then the basis of the generated code. The trace is
 first optimized, and then turned into machine code. Both optimization
-and machine code generation is simple, because the traces are linear. This
+and machine code generation are simple, because the traces are linear. This
 linearity makes many optimizations a lot more tractable, and the inlining that
 happens gives the optimizations automatically more context to work with.
 
@@ -288,8 +288,9 @@
 trace. As an example, if a loop contains an \lstinline{if} statement, the trace
 will contain the execution of one of the paths only, which is the path that was
 taken during the production of the trace. The trace will also contain a guard
-that checks that the condition of the \lstinline{if} statement is true, because if
-it isn't, the rest of the trace is not valid.
+that checks that the condition of the \lstinline{if} statement is the same as
+during tracing, because if
+it isn't, the rest of the trace is not valid. \cfbolz{The "if" shouldn't be bold}
 
 When generating machine code, every guard is be turned into a quick check to
 see whether the assumption still holds. When such a guard is hit during the
@@ -565,7 +566,7 @@
 The main insight to improve the code shown in the last section is that objects
 in category 1 don't survive very long -- they are used only inside the loop and
 nobody else in the program stores a reference to them. The idea for improving
-the code is thus to analyze which objects fall in category 1 and thus do
+the code is to analyze which objects fall in category 1 and therefore do
 not have to be allocated at all.
 
 This is a process that is usually called \emph{escape analysis}. In this paper we will
@@ -835,7 +836,7 @@
 The static heap is a partial function from $V^*$ into the
 set of static objects, which are triples of a type and two elements of $V^*$.
 A variable $v^*$ is in the domain of the static heap $S$ as long as the
-optimizer can fully keep track of the object. The image of $v^*$ is what is
+optimizer can fully keep track of the object. The object $S(v^*)$ is what is
 statically known about the object stored in it, \ie its type and its fields. The
 fields of objects in the static heap are also elements of $V^*$ (or null, for
 short periods of time).
@@ -1048,7 +1049,8 @@
 result. The errors were computed using a confidence interval with a 95\%
 confidence level \cite{georges_statistically_2007}. The results are reported in
 Figure~\ref{fig:times}. In addition to the run times the table also reports the
-speedup that PyPy with optimization turned on achieves.
+speedup that PyPy achieves when the optimization is turned on.
+
 With the optimization turned on, PyPy's Python interpreter outperforms CPython
 in all benchmarks except spambayes (which heavily relies on regular expression
 performance and thus is not helped much by our Python JIT) and meteor-contest.
@@ -1089,7 +1091,7 @@
 \section{Related Work}
 \label{sec:related}
 
-There exists a large number of works on escape analysis, which is an program
+There exists a large number of works on escape analysis, which is a program
 analysis that tries to find an upper bound for the lifetime of objects allocated
 at specific program points
 \cite{goldberg_higher_1990,park_escape_1992,choi_escape_1999,bruno_blanchet_escape_2003}.
@@ -1157,7 +1159,7 @@
 hardest part of partial evaluation: the tracing JIT selects the parts
 of the program that are worthwhile to optimize, and extracts linear
 paths through them, inlining functions as necessary.  What is left to
-optimize is only those linear paths.
+optimize are only those linear paths.
 
 We expect a similar result for other optimizations that usually require
 a complex analysis phase and are thus normally too slow to use at


From afa at codespeak.net  Thu Oct 14 18:35:06 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 18:35:06 +0200 (CEST)
Subject: [pypy-svn] r77952 - pypy/branch/fast-forward/pypy/module/thread/test
Message-ID: <20101014163506.6E982282BEA@codespeak.net>

Author: afa
Date: Thu Oct 14 18:35:04 2010
New Revision: 77952

Modified:
   pypy/branch/fast-forward/pypy/module/thread/test/support.py
   pypy/branch/fast-forward/pypy/module/thread/test/test_fork.py
Log:
A failing test which shows why test_multiprocessing hangs on startup...


Modified: pypy/branch/fast-forward/pypy/module/thread/test/support.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/thread/test/support.py	(original)
+++ pypy/branch/fast-forward/pypy/module/thread/test/support.py	Thu Oct 14 18:35:04 2010
@@ -1,5 +1,5 @@
 import py
-import time, gc
+import time, gc, thread, os
 from pypy.conftest import gettestobjspace, option
 from pypy.interpreter.gateway import ObjSpace, W_Root, interp2app_temp
 from pypy.module.thread import gil
@@ -21,6 +21,13 @@
     print '*** timed out ***'
 waitfor.unwrap_spec = [ObjSpace, W_Root, float]
 
+def timeout_killer(pid, delay):
+    def kill():
+        time.sleep(delay)
+        os.kill(pid, 9)
+        print "process %s killed!" % (pid,)
+    thread.start_new_thread(kill, ())
+timeout_killer.unwrap_spec = [int, float]
 
 class GenericTestThread:
 
@@ -47,3 +54,4 @@
             import time
             return time.sleep
         """)
+        cls.w_timeout_killer = space.wrap(interp2app_temp(timeout_killer))

Modified: pypy/branch/fast-forward/pypy/module/thread/test/test_fork.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/thread/test/test_fork.py	(original)
+++ pypy/branch/fast-forward/pypy/module/thread/test/test_fork.py	Thu Oct 14 18:35:04 2010
@@ -36,3 +36,20 @@
         finally:
             run = False
             self.waitfor(lambda: done)
+
+    def test_forked_can_thread(self):
+        "Checks that a forked interpreter can start a thread"
+        import os, thread, time
+
+        # pre-allocate some locks
+        thread.start_new_thread(lambda: None, ())
+
+        pid = os.fork()
+        if pid == 0:
+            print 'in child'
+            thread.start_new_thread(lambda: None, ())
+            os._exit(0)
+        else:
+            self.timeout_killer(pid, 5)
+            exitcode = os.waitpid(pid, 0)[1]
+            assert exitcode == 0 # if 9, process was killed by timer!


From arigo at codespeak.net  Thu Oct 14 18:35:50 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 18:35:50 +0200 (CEST)
Subject: [pypy-svn] r77953 - pypy/trunk/pypy/module/_rawffi
Message-ID: <20101014163550.6F7E1282BEA@codespeak.net>

Author: arigo
Date: Thu Oct 14 18:35:48 2010
New Revision: 77953

Modified:
   pypy/trunk/pypy/module/_rawffi/array.py
Log:
Fix the comment: remove the XXX and says "it's documented".


Modified: pypy/trunk/pypy/module/_rawffi/array.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/array.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/array.py	Thu Oct 14 18:35:48 2010
@@ -97,10 +97,11 @@
 
 class W_ArrayInstance(W_DataInstance):
     def __init__(self, space, shape, length, address=r_uint(0)):
-        # XXX workaround for a bug in libffi on x86_64: make sure that
+        # Workaround for a strange behavior of libffi: make sure that
         # we always have at least 8 bytes.  For W_ArrayInstances that are
         # used as the result value of a function call, ffi_call() writes
         # 8 bytes into it even if the function's result type asks for less.
+        # This strange behavior is documented.
         memsize = shape.size * length
         if memsize < 8:
             memsize = 8


From arigo at codespeak.net  Thu Oct 14 18:43:40 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 14 Oct 2010 18:43:40 +0200 (CEST)
Subject: [pypy-svn] r77954 - pypy/branch/leak-finder
Message-ID: <20101014164340.5BCFD282BEA@codespeak.net>

Author: arigo
Date: Thu Oct 14 18:43:38 2010
New Revision: 77954

Added:
   pypy/branch/leak-finder/
      - copied from r77953, pypy/trunk/
Log:
Trying to figure out how to detect leaks of raw-flavoured mallocs.



From cfbolz at codespeak.net  Thu Oct 14 18:45:54 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 14 Oct 2010 18:45:54 +0200 (CEST)
Subject: [pypy-svn] r77955 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014164554.B2C8936E076@codespeak.net>

Author: cfbolz
Date: Thu Oct 14 18:45:53 2010
New Revision: 77955

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
remove a number of "thus"s. simplify two paragraphs in the background section.


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 18:45:53 2010
@@ -162,8 +162,11 @@
 
 A recently popular approach to implementing just-in-time compilers for dynamic
 languages is that of a tracing JIT. A tracing JIT works by observing the running
-program and recording its hot spots into linear execution traces, which are then turned into
-machine code. One reason for the popularity of tracing JITs is their relative
+program and recording its hot spots into linear execution traces. Working on
+traces is the central idea of a tracing JIT. Those traces are optimized and
+turned into machine code.
+
+One reason for the popularity of tracing JITs is their relative
 simplicity. They can often be added to an interpreter and a lot of the
 infrastructure of the interpreter can be reused. They give some important
 optimizations like inlining and constant-folding for free. A tracing JIT always
@@ -215,11 +218,10 @@
 \emph{RPython} \cite{davide_ancona_rpython:_2007}. RPython ("restricted Python")
 is a subset of Python chosen in such a way that type inference becomes
 possible. The language interpreter can then be compiled (``translated'') with
-PyPy's tools into a VM on the C level. Because the interpreter is written at a
-relatively high level, the language implementation is kept free of low-level
-details, such as object layout, garbage collection or memory model. Those
-aspects of the final VM are woven into the generated code during the translation
-to C.
+PyPy's tools into a VM on the C level. During translation to C, many low-level
+aspects of the final VM, such as object layout, garbage collection and memory
+model, are woven into the generated code. Therefore the interpreter itself can
+remain at a relatively high level of abstraction.
 
 A number of languages have been implemented with PyPy. The project was initiated
 to get a better Python implementation, which inspired the name of the project
@@ -231,7 +233,7 @@
 
 The feature that makes PyPy more than a compiler with a runtime system is its
 support for automated JIT compiler generation \cite{bolz_tracing_2009}. During
-the translation to C, PyPy's tools can generate a just-in-time compiler for the
+the translation to C, PyPy's tools can generate a tracing just-in-time compiler for the
 language that the interpreter is implementing. This process is mostly
 automatic; it only needs to be guided by the language implementer using a small number of
 source-code hints. Mostly-automatically generating a JIT compiler has many advantages
@@ -240,12 +242,9 @@
 Many optimizations can benefit all languages implemented as an interpreter in RPython.
 
 Moreover, thanks to the internal design of the JIT generator, it is very easy
-to add new \emph{backends} for producing the actual machine code, in addition
-to the original backend for the Intel \emph{x86} architecture.  Examples of
-additional JIT backends are the one for Intel \emph{x86-64} and an
+to add new \emph{backends} for producing the actual machine code.  Examples of
+JIT backends that are implemented are the one for Intel x86 and x86-64 and an
 experimental one for the CLI .NET Virtual Machine \cite{cuni_high_2010}.
-PyPy's JIT generator generates a \emph{tracing JIT
-compiler}, a concept which we now explain in more details.
 
 \subsection{Tracing JIT Compilers}
 \label{sub:JIT_background}
@@ -373,9 +372,9 @@
 
 Using these classes to implement arithmetic shows the basic problem that a
 dynamic language implementation has. All the numbers are instances of either
-\lstinline{BoxedInteger} or \lstinline{BoxedFloat}, thus they consume space on the
+\lstinline{BoxedInteger} or \lstinline{BoxedFloat}, therefore they consume space on the
 heap. Performing many arithmetic operations produces lots of garbage quickly,
-thus putting pressure on the garbage collector. Using double dispatching to
+which puts pressure on the garbage collector. Using double dispatching to
 implement the numeric tower needs two method calls per arithmetic operation,
 which is costly due to the method dispatch.
 
@@ -593,8 +592,8 @@
 the type check the guard does is statically known.
 
 In the example from last section, the following operations in the upper half
-of Fig.~\ref{fig:unopt-trace} would produce two
-static objects, and would thus be completely removed from the optimized trace:
+of Fig.~\ref{fig:unopt-trace} produce two
+static objects, and can be completely removed from the optimized trace:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 $p_{5}$ = new(BoxedInteger)
@@ -625,7 +624,7 @@
 
 First, the \lstinline{guard_class} operations can be removed, because the classes of $p_{5}$ and
 $p_{6}$ are known to be \lstinline{BoxedInteger}. Second, the \lstinline{get} operations can be removed
-and $i_{7}$ and $i_{8}$ are just replaced by $i_{4}$ and -100. Thus the only
+and $i_{7}$ and $i_{8}$ are just replaced by $i_{4}$ and -100. The only
 remaining operation in the optimized trace would be:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
@@ -746,8 +745,8 @@
 to locations and the heap is a partial function from locations to objects. Note
 that a variable can never be null in the environment, otherwise the trace would
 be malformed. The environment could not directly map variables to object,
-because several variables can contain a pointer to the \emph{same} object. Thus
-the "indirection" is needed to express sharing.
+because several variables can contain a pointer to the \emph{same} object. 
+The "indirection" is needed to express sharing.
 
 We use the following notation for updating partial functions:
 $E[v\mapsto l]$ denotes the environment which is just like $E$, but maps $v$ to
@@ -830,7 +829,7 @@
 themselves written with a
 $\ ^*$ for clarity). The reason for introducing new variables in the optimized
 trace is that several variables that appear in the unoptimized trace can turn
-into the same variables in the optimized trace. Thus the environment of the
+into the same variables in the optimized trace. The environment of the
 optimizer serves a function similar to that of the environment in the semantics: sharing.
 
 The static heap is a partial function from $V^*$ into the


From agaynor at codespeak.net  Thu Oct 14 19:30:53 2010
From: agaynor at codespeak.net (agaynor at codespeak.net)
Date: Thu, 14 Oct 2010 19:30:53 +0200 (CEST)
Subject: [pypy-svn] r77956 - pypy/extradoc/talk/pepm2011
Message-ID: <20101014173053.ADC06282BEA@codespeak.net>

Author: agaynor
Date: Thu Oct 14 19:30:51 2010
New Revision: 77956

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
Small grammar fix.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 14 19:30:51 2010
@@ -184,7 +184,7 @@
 
 The core of our trace optimization technique can be
 viewed as partial evaluation: the partial evaluation
-performs a form of escape analysis \cite{bruno_blanchet_escape_2003} on the traces and make some
+performs a form of escape analysis \cite{bruno_blanchet_escape_2003} on the traces and makes some
 objects that are allocated in the trace \emph{static,} which
 means that they do not occur any more in the optimized trace. This technique is
 informally described in Section~\ref{sec:statics}; a more formal description is


From afa at codespeak.net  Thu Oct 14 21:10:43 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 21:10:43 +0200 (CEST)
Subject: [pypy-svn] r77957 - in pypy/branch/fast-forward/pypy/module: posix
	thread
Message-ID: <20101014191043.E6897282BEA@codespeak.net>

Author: afa
Date: Thu Oct 14 21:10:17 2010
New Revision: 77957

Modified:
   pypy/branch/fast-forward/pypy/module/posix/interp_posix.py
   pypy/branch/fast-forward/pypy/module/thread/__init__.py
   pypy/branch/fast-forward/pypy/module/thread/gil.py
   pypy/branch/fast-forward/pypy/module/thread/os_thread.py
Log:
Add infrastructure for hooks around os.fork():
'before', 'parent', and 'child'.
Use it to reinit the thread.bootstrap lock,
and the threadlocals structure as well which seems initialized at the
same time.

This fixes the previous test, and helps test_multiprocessing a lot.


Modified: pypy/branch/fast-forward/pypy/module/posix/interp_posix.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/posix/interp_posix.py	(original)
+++ pypy/branch/fast-forward/pypy/module/posix/interp_posix.py	Thu Oct 14 21:10:17 2010
@@ -473,7 +473,7 @@
         return True
 
 def get(space): 
-    return space.fromcache(State) 
+    return space.fromcache(State)
 
 def _convertenviron(space, w_env):
     space.call_method(w_env, 'clear')
@@ -600,11 +600,48 @@
     return space.wrap(result)
 readlink.unwrap_spec = [ObjSpace, str]
 
+before_fork_hooks = []
+after_fork_child_hooks = []
+after_fork_parent_hooks = []
+
+ at specialize.memo()
+def get_fork_hooks(where):
+    if where == 'before':
+        return before_fork_hooks
+    elif where == 'child':
+        return after_fork_child_hooks
+    elif where == 'parent':
+        return after_fork_parent_hooks
+    else:
+        assert False, "Unknown fork hook"
+
+def add_fork_hook(where, hook):
+    "NOT_RPYTHON"
+    get_fork_hooks(where).append(hook)
+
+ at specialize.arg(0)
+def run_fork_hooks(where, space):
+    for hook in get_fork_hooks(where):
+        hook(space)
+
 def fork(space):
+    run_fork_hooks('before', space)
+
     try:
         pid = os.fork()
-    except OSError, e: 
-        raise wrap_oserror(space, e) 
+    except OSError, e:
+        try:
+            run_fork_hooks('parent', space)
+        except:
+            # Don't clobber the OSError if the fork failed
+            pass
+        raise wrap_oserror(space, e)
+
+    if pid == 0:
+        run_fork_hooks('child', space)
+    else:
+        run_fork_hooks('parent', space)
+
     return space.wrap(pid)
 
 def openpty(space):

Modified: pypy/branch/fast-forward/pypy/module/thread/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/thread/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/thread/__init__.py	Thu Oct 14 21:10:17 2010
@@ -29,3 +29,8 @@
         space.threadlocals = gil.GILThreadLocals()
         space.threadlocals.initialize(space)
         space.threadlocals.setvalue(prev)
+
+        from pypy.module.posix.interp_posix import add_fork_hook
+        from pypy.module.thread.os_thread import reinit_threads
+        add_fork_hook('child', reinit_threads)
+

Modified: pypy/branch/fast-forward/pypy/module/thread/gil.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/thread/gil.py	(original)
+++ pypy/branch/fast-forward/pypy/module/thread/gil.py	Thu Oct 14 21:10:17 2010
@@ -48,6 +48,12 @@
         invoke_around_extcall(before_external_call, after_external_call)
         return result
 
+    def reinit_threads(self, space):
+        if self.ll_GIL:
+            self.ll_GIL = thread.allocate_ll_lock()
+            thread.acquire_NOAUTO(self.ll_GIL, True)
+            self.enter_thread(space)
+
     def yield_thread(self):
         thread.yield_thread()  # explicitly release the gil (used by test_gil)
 

Modified: pypy/branch/fast-forward/pypy/module/thread/os_thread.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/thread/os_thread.py	(original)
+++ pypy/branch/fast-forward/pypy/module/thread/os_thread.py	Thu Oct 14 21:10:17 2010
@@ -67,13 +67,17 @@
     args = None
     w_callable = None
 
+    @staticmethod
     def setup(space):
         if bootstrapper.lock is None:
             try:
                 bootstrapper.lock = thread.allocate_lock()
             except thread.error:
                 raise wrap_thread_error(space, "can't allocate bootstrap lock")
-    setup = staticmethod(setup)
+
+    @staticmethod
+    def reinit(space):
+        bootstrapper.lock = None
 
     def bootstrap():
         # Note that when this runs, we already hold the GIL.  This is ensured
@@ -135,6 +139,17 @@
     space.threadlocals.setup_threads(space)
     bootstrapper.setup(space)
 
+def reinit_threads(space):
+    "Called in the child process after a fork()"
+    space.threadlocals.reinit_threads(space)
+    bootstrapper.reinit(space)
+
+    # Clean the threading module after a fork()
+    w_modules = space.sys.get('modules')
+    w_threading = space.finditem_str(w_modules, 'threading')
+    if w_threading is not None:
+        space.call_method(w_threading, "_after_fork")
+
 
 def start_new_thread(space, w_callable, w_args, w_kwargs=NoneNotWrapped):
     """Start a new thread and return its identifier.  The thread will call the


From hakanardo at codespeak.net  Thu Oct 14 21:49:26 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Thu, 14 Oct 2010 21:49:26 +0200 (CEST)
Subject: [pypy-svn] r77958 - in pypy/branch/jit-unroll-loops/pypy/jit:
	metainterp/optimizeopt metainterp/test tl
Message-ID: <20101014194926.ECF41282BEC@codespeak.net>

Author: hakanardo
Date: Thu Oct 14 21:49:24 2010
New Revision: 77958

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py
Log:
Clear all last_guard_index before inlining, support the reording of operations in force_lazy_setfield and allow new inputargs to be added by operations induced by forcing the boxes of the final jump.

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Thu Oct 14 21:49:24 2010
@@ -39,6 +39,7 @@
            argmap[loop_args[i]] = jump_args[i]
 
         for v in self.optimizer.values.values():
+            v.last_guard_index = -1 # FIXME: Are there any more indexes stored?
             if not v.is_constant() and v.box:
                 v.fromstart = True
 
@@ -72,10 +73,16 @@
                     args.extend(self.getvalue(arg).get_forced_boxes())
                 newop.initarglist(args + inputargs[len(args):])
 
+            #print 'P: ', newop
             current = len(self.optimizer.newoperations)
             self.emit_operation(newop)
 
+            # FIXME: force_lazy_setfield in heap.py may reorder last ops
+            if current > 0:
+                current -= 1
+                
             for op in self.optimizer.newoperations[current:]:
+                #print 'E: ', op,  self.optimizer.newoperations.index(op)
                 if op.is_guard():
                     descr = op.getdescr()
                     assert isinstance(descr, ResumeGuardDescr)
@@ -83,13 +90,18 @@
                 args = op.getarglist()
                 if op.is_guard():
                     args = args + op.getfailargs()
+                #if op.getopnum() == rop.SETFIELD_GC:
+                #    import pdb; pdb.set_trace()
                 for a in args:
                     if not isinstance(a, Const) and a in self.optimizer.values:
                         v = self.getvalue(a)
                         if v.fromstart and a not in inputargs:
                             inputargs.append(a)
-                            if op.getopnum() == rop.JUMP:
-                                op.initarglist(op.getarglist() + [argmap[a]])
+                            jmp = self.optimizer.newoperations[-1]
+                            if jmp.getopnum() == rop.JUMP:
+                                newval = self.getvalue(argmap[a])
+                                newarg = newval.get_forced_boxes()
+                                jmp.initarglist(jmp.getarglist() + newarg)
 
         return inputargs
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Thu Oct 14 21:49:24 2010
@@ -8,7 +8,7 @@
 import pypy.jit.metainterp.optimizeopt.virtualize as virtualize
 from pypy.jit.metainterp.optimizeopt import optimize_loop_1
 from pypy.jit.metainterp.optimizeutil import InvalidLoop
-from pypy.jit.metainterp.history import AbstractDescr, ConstInt, BoxInt
+from pypy.jit.metainterp.history import AbstractDescr, ConstInt, BoxInt, TreeLoop
 from pypy.jit.metainterp.jitprof import EmptyProfiler
 from pypy.jit.metainterp import executor, compile, resume, history
 from pypy.jit.metainterp.resoperation import rop, opname, ResOperation
@@ -214,6 +214,8 @@
         op.setfailargs(boxes)
     def __eq__(self, other):
         return type(self) is type(other)      # xxx obscure
+    def clone_if_mutable(self):
+        return self
 
 def _sortboxes(boxes):
     _kind2count = {history.INT: 1, history.REF: 2, history.FLOAT: 3}
@@ -254,12 +256,15 @@
             loop.token.specnodes = self.unpack_specnodes(spectext)
         #
         self.loop = loop
+        loop.preamble = TreeLoop('preamble')
         metainterp_sd = FakeMetaInterpStaticData(self.cpu)
         if hasattr(self, 'vrefinfo'):
             metainterp_sd.virtualref_info = self.vrefinfo
         optimize_loop_1(metainterp_sd, loop)
         #
         expected = self.parse(optops)
+        print
+        print "Ops: "
         print '\n'.join([str(o) for o in loop.operations])
         self.assert_equal(loop, expected)
 
@@ -3559,14 +3564,10 @@
         jump(p0, i22)
         """
         expected = """
-        [p0, i22]
-        i1 = getfield_gc(p0, descr=valuedescr)
-        i2 = int_gt(i1, i22)
-        guard_false(i2) []
-        i3 = int_add(i1, 1)
+        [p0, i22, i1]
         i331 = force_token()
         setfield_gc(p0, i1, descr=valuedescr)
-        jump(p0, i22)
+        jump(p0, i22, i1)
         """
         self.optimize_loop(ops, 'Not, Not', expected)
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py	Thu Oct 14 21:49:24 2010
@@ -2,7 +2,7 @@
 from pypy.rpython.lltypesystem import lltype
 from pypy.jit.metainterp import warmspot
 from pypy.module.pypyjit.policy import PyPyJitPolicy
-from pypy.rlib.jit import OPTIMIZER_FULL
+from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_NO_PERFECTSPEC
 
 
 def run_child(glob, loc):
@@ -35,5 +35,5 @@
     warmspot.jittify_and_run(interp, graph, [], policy=policy,
                              listops=True, CPUClass=CPUClass,
                              backendopt=True, inline=True,
-                             optimizer=OPTIMIZER_FULL)
+                             optimizer=OPTIMIZER_NO_PERFECTSPEC)
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py	Thu Oct 14 21:49:24 2010
@@ -38,25 +38,45 @@
 ## print t2 - t1
 
 try:
+##     from array import array
+
+##     def coords(w,h):
+##         y = 0
+##         while y < h:
+##             x = 0
+##             while x < w:
+##                 yield x,y
+##                 x += 1
+##             y += 1
+
+##     def f(img):
+##         sa=0
+##         for x, y in coords(4,4):
+##             sa += x * y
+##         return sa
+
+##     #img=array('h',(1,2,3,4))
+##     print f(3)
     from array import array
+    class Circular(array):
+        def __new__(cls):
+            self = array.__new__(cls, 'i', range(16))
+            return self
+        def __getitem__(self, i):
+            assert self.__len__() == 16 
+            return array.__getitem__(self, i & 15)
 
-    def coords(w,h):
-        y = 0
-        while y < h:
-            x = 0
-            while x < w:
-                yield x,y
-                x += 1
-            y += 1
-
-    def f(img):
-        sa=0
-        for x, y in coords(4,4):
-            sa += x * y
+    def main():
+        buf = Circular()
+        i = 10
+        sa = 0
+        while i < 20:
+            sa += buf[i-2] + buf[i-1] + buf[i] + buf[i+1] + buf[i+2]
+            i += 1
         return sa
 
-    #img=array('h',(1,2,3,4))
-    print f(3)
+    print main()
+    
 except Exception, e:
     print "Exception: ", type(e)
     print e


From afa at codespeak.net  Thu Oct 14 21:58:36 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 21:58:36 +0200 (CEST)
Subject: [pypy-svn] r77959 - pypy/branch/fast-forward/pypy/module/imp
Message-ID: <20101014195836.7A118282BEC@codespeak.net>

Author: afa
Date: Thu Oct 14 21:58:34 2010
New Revision: 77959

Modified:
   pypy/branch/fast-forward/pypy/module/imp/__init__.py
   pypy/branch/fast-forward/pypy/module/imp/importing.py
   pypy/branch/fast-forward/pypy/module/imp/interp_imp.py
Log:
Add disabled code that *could* help with the import lock
being held while another thread forks the interpreter.

Let's see if a failing test in cpython tests suite needs it.


Modified: pypy/branch/fast-forward/pypy/module/imp/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/imp/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/imp/__init__.py	Thu Oct 14 21:58:34 2010
@@ -34,3 +34,13 @@
 
     appleveldefs = {
         }
+
+    def __init__(self, space, *args):
+        "NOT_RPYTHON"
+        MixedModule.__init__(self, space, *args)
+        # from pypy.module.posix.interp_posix import add_fork_hook
+        # from pypy.module.imp import interp_imp
+        # add_fork_hook('before', interp_imp.acquire_lock)
+        # add_fork_hook('parent', interp_imp.release_lock)
+        # add_fork_hook('child', interp_imp.reinit_lock)
+

Modified: pypy/branch/fast-forward/pypy/module/imp/importing.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/imp/importing.py	(original)
+++ pypy/branch/fast-forward/pypy/module/imp/importing.py	Thu Oct 14 21:58:34 2010
@@ -580,6 +580,13 @@
             self.lockowner = None
             self.lock.release()
 
+    def reinit_lock(self):
+        # Called after fork() to ensure that newly created child
+        # processes do not share locks with the parent
+        self.lock = None
+        self.lockowner = None
+        self.lockcounter = 0
+
 def getimportlock(space):
     return space.fromcache(ImportRLock)
 

Modified: pypy/branch/fast-forward/pypy/module/imp/interp_imp.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/imp/interp_imp.py	(original)
+++ pypy/branch/fast-forward/pypy/module/imp/interp_imp.py	Thu Oct 14 21:58:34 2010
@@ -152,3 +152,7 @@
 def release_lock(space):
     if space.config.objspace.usemodules.thread:
         importing.getimportlock(space).release_lock()
+
+def reinit_lock(space):
+    if space.config.objspace.usemodules.thread:
+        importing.getimportlock(space).reinit_lock()


From afa at codespeak.net  Thu Oct 14 23:50:01 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 23:50:01 +0200 (CEST)
Subject: [pypy-svn] r77960 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101014215001.79381282BEA@codespeak.net>

Author: afa
Date: Thu Oct 14 23:49:58 2010
New Revision: 77960

Added:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py   (contents, props changed)
Modified:
   pypy/branch/fast-forward/pypy/module/_io/__init__.py
   pypy/branch/fast-forward/pypy/module/_io/interp_io.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
Log:
Start implementing _io.FileIO.
cheat and have io.open() use _pyio.open()...


Modified: pypy/branch/fast-forward/pypy/module/_io/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/__init__.py	Thu Oct 14 23:49:58 2010
@@ -14,7 +14,7 @@
         '_BufferedIOBase': 'interp_io.W_BufferedIOBase',
         '_TextIOBase': 'interp_io.W_TextIOBase',
 
-        'FileIO': 'interp_io.W_FileIO',
+        'FileIO': 'interp_fileio.W_FileIO',
         'BytesIO': 'interp_io.W_BytesIO',
         'StringIO': 'interp_stringio.W_StringIO',
         'BufferedReader': 'interp_io.W_BufferedReader',
@@ -23,7 +23,7 @@
         'BufferedRandom': 'interp_io.W_BufferedRandom',
         'TextIOWrapper': 'interp_io.W_TextIOWrapper',
 
-        'open': 'space.w_None',
+        'open': 'interp_io.open',
         'UnsupportedOperation': 'space.w_None',
         'IncrementalNewlineDecoder': 'space.w_None',
         }

Added: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- (empty file)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Thu Oct 14 23:49:58 2010
@@ -0,0 +1,129 @@
+from pypy.module._io.interp_io import W_RawIOBase
+from pypy.interpreter.typedef import TypeDef
+from pypy.interpreter.gateway import interp2app, unwrap_spec, Arguments
+from pypy.interpreter.baseobjspace import ObjSpace, W_Root
+from pypy.interpreter.error import OperationError, wrap_oserror2
+import os
+
+def _bad_mode(space):
+    raise OperationError(space.w_ValueError, space.wrap(
+        "Must have exactly one of read/write/append mode"))
+
+def decode_mode(spac, mode):
+    flags = 0
+    rwa = False
+    readable = False
+    writable = False
+    append = False
+    plus = False
+
+    for s in mode:
+        if s == 'r':
+            if rwa:
+                _bad_mode(space)
+            rwa = True
+            readable = True
+        elif s == 'w':
+            if rwa:
+                _bad_mode(space)
+            rwa = True
+            writable = True
+            flags |= os.O_CREAT | os.O_TRUNC
+        elif s == 'a':
+            if rwa:
+                _bad_mode(space)
+            rwa = 1
+            writable = True
+            flags |= os.O_CREAT
+            append = True
+        elif s == 'b':
+            pass
+        elif s == '+':
+            if plus:
+                _bad_mode(space)
+            readable = writable = True
+            plus = True
+        else:
+            raise OperationError(space.w_ValueError, space.wrap(
+                "invalid mode: %s" % (mode,)))
+
+    if not rwa:
+        _bad_mode(space)
+
+    if readable and writable:
+        flags |= os.O_RDWR
+    elif readable:
+        flags |= os.O_RDONLY
+    else:
+        flags |= os.O_WRONLY
+
+    if hasattr(os, 'O_BINARY'):
+        flags |= os.O_BINARY
+
+    if hasattr(os, 'O_APPEND') and append:
+        flags |= os.O_APPEND
+
+    return readable, writable, flags
+
+class W_FileIO(W_RawIOBase):
+    def __init__(self, space):
+        W_RawIOBase.__init__(self, space)
+        self.fd = -1
+        self.readable = False
+        self.writable = False
+        self.seekable = -1
+        self.closefd = True
+
+    @unwrap_spec(ObjSpace, W_Root, Arguments)
+    def descr_new(space, w_subtype, __args__):
+        self = space.allocate_instance(W_FileIO, w_subtype)
+        W_FileIO.__init__(self, space)
+        return space.wrap(self)
+
+    @unwrap_spec('self', ObjSpace, W_Root, str, int)
+    def descr_init(self, space, w_name, mode, closefd):
+        if space.isinstance_w(w_name, space.w_float):
+            raise OperationError(space.w_TypeError, space.wrap(
+                "integer argument expected, got float"))
+        try:
+            fd = space.int_w(w_name)
+        except OperationError, e:
+            pass
+        else:
+            if fd < 0:
+                raise OperationError(space.w_ValueError, space.wrap(
+                    "negative file descriptor"))
+
+        self.readable, self.writable, flags = decode_mode(space, mode)
+
+        from pypy.module.posix.interp_posix import dispatch_filename, rposix
+        try:
+            self.fd = dispatch_filename(rposix.open)(
+                space, w_name, flags, 0666)
+        except OSError, e:
+            raise wrap_oserror2(space, e, w_fname)
+        self.closefd = bool(closefd)
+
+    def _check_closed(self, space):
+        if self.fd < 0:
+            raise OperationError(space.w_ValueError, space.wrap(
+                "I/O operation on closed file"))
+
+    @unwrap_spec('self', ObjSpace)
+    def readable_w(self, space):
+        self._check_closed(space)
+        return space.wrap(self.readable)
+
+    @unwrap_spec('self', ObjSpace)
+    def writable_w(self, space):
+        self._check_closed(space)
+        return space.wrap(self.writable)
+
+W_FileIO.typedef = TypeDef(
+    'FileIO', W_RawIOBase.typedef,
+    __new__  = interp2app(W_FileIO.descr_new.im_func),
+    __init__  = interp2app(W_FileIO.descr_init),
+    readable = interp2app(W_FileIO.readable_w),
+    writable = interp2app(W_FileIO.writable_w),
+    )
+

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py	Thu Oct 14 23:49:58 2010
@@ -94,6 +94,22 @@
             raise OperationError(space.w_StopIteration, space.w_None)
         return w_line
 
+    @unwrap_spec('self', ObjSpace)
+    def isatty_w(self, space):
+        return space.w_False
+
+    @unwrap_spec('self', ObjSpace)
+    def readable_w(self, space):
+        return space.w_False
+
+    @unwrap_spec('self', ObjSpace)
+    def writable_w(self, space):
+        return space.w_False
+
+    @unwrap_spec('self', ObjSpace)
+    def seekable_w(self, space):
+        return space.w_False
+
 W_IOBase.typedef = TypeDef(
     '_IOBase',
     __new__ = generic_new_descr(W_IOBase),
@@ -103,6 +119,10 @@
     next = interp2app(W_IOBase.next_w),
     close = interp2app(W_IOBase.close_w),
     flush = interp2app(W_IOBase.flush_w),
+    isatty = interp2app(W_IOBase.isatty_w),
+    readable = interp2app(W_IOBase.readable_w),
+    writable = interp2app(W_IOBase.writable_w),
+    seekable = interp2app(W_IOBase.seekable_w),
     closed = GetSetProperty(W_IOBase.closed_get_w),
     )
 
@@ -128,12 +148,6 @@
     __new__ = generic_new_descr(W_TextIOBase),
     )
 
-class W_FileIO(W_RawIOBase):
-    pass
-W_FileIO.typedef = TypeDef(
-    'FileIO', W_RawIOBase.typedef,
-    )
-
 class W_BytesIO(W_BufferedIOBase):
     pass
 W_BytesIO.typedef = TypeDef(
@@ -169,3 +183,12 @@
 W_TextIOWrapper.typedef = TypeDef(
     'TextIOWrapper', W_TextIOBase.typedef,
     )
+
+ at unwrap_spec(ObjSpace, Arguments)
+def open(space, __args__):
+    # XXX cheat!
+    w_pyio = space.call_method(space.builtin, '__import__',
+                             space.wrap("_pyio"))
+    w_func = space.getattr(w_pyio, space.wrap("open"))
+    return space.call_args(w_func, __args__)
+

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	Thu Oct 14 23:49:58 2010
@@ -36,3 +36,14 @@
                 return ""
 
         assert list(MyFile()) == ["line1", "line2"]
+
+class AppTestOpen:
+    def setup_class(cls):
+        from pypy.tool.udir import udir
+        tmpfile = udir.join('tmpfile').ensure()
+        cls.w_tmpfile = cls.space.wrap(str(tmpfile))
+
+    def test_open(self):
+        import io
+        f = io.open(self.tmpfile, "rb")
+        f.close()


From afa at codespeak.net  Thu Oct 14 23:50:51 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 23:50:51 +0200 (CEST)
Subject: [pypy-svn] r77961 - pypy/branch/fast-forward/lib_pypy
Message-ID: <20101014215051.4C79F282BEA@codespeak.net>

Author: afa
Date: Thu Oct 14 23:50:49 2010
New Revision: 77961

Modified:
   pypy/branch/fast-forward/lib_pypy/_csv.py
Log:
csv.dialect attributes are read-only


Modified: pypy/branch/fast-forward/lib_pypy/_csv.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/_csv.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/_csv.py	Thu Oct 14 23:50:49 2010
@@ -135,8 +135,8 @@
                 converter = attributes[name][1]
                 if converter:
                     value = converter(value)
-                
-            setattr(self, name, value)
+
+            setattr(self, '_' + name, value)
 
         if not self.delimiter:
             raise TypeError("delimiter must be set")
@@ -148,7 +148,16 @@
             raise TypeError("lineterminator must be set")
 
         return self
-        
+
+    delimiter        = property(lambda self: self._delimiter)
+    doublequote      = property(lambda self: self._doublequote)
+    escapechar       = property(lambda self: self._escapechar)
+    lineterminator   = property(lambda self: self._lineterminator)
+    quotechar        = property(lambda self: self._quotechar)
+    quoting          = property(lambda self: self._quoting)
+    skipinitialspace = property(lambda self: self._skipinitialspace)
+    strict           = property(lambda self: self._strict)
+
 
 def _call_dialect(dialect_inst, kwargs):
     return Dialect(dialect_inst, **kwargs)


From afa at codespeak.net  Thu Oct 14 23:58:55 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 14 Oct 2010 23:58:55 +0200 (CEST)
Subject: [pypy-svn] r77962 - in pypy/branch/fast-forward:
	lib-python/modified-2.7.0/test lib_pypy
Message-ID: <20101014215855.B381F282BEA@codespeak.net>

Author: afa
Date: Thu Oct 14 23:58:54 2010
New Revision: 77962

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_csv.py
      - copied, changed from r77956, pypy/branch/fast-forward/lib-python/2.7.0/test/test_csv.py
Modified:
   pypy/branch/fast-forward/lib_pypy/_csv.py
Log:
Fix, and relax the test a little

With CPython, delattr and setattr raise TypeError or AttributeError,
depending on the descriptor being a readonly member or a readonly property...


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_csv.py (from r77956, pypy/branch/fast-forward/lib-python/2.7.0/test/test_csv.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_csv.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_csv.py	Thu Oct 14 23:58:54 2010
@@ -54,8 +54,10 @@
         self.assertEqual(obj.dialect.skipinitialspace, False)
         self.assertEqual(obj.dialect.strict, False)
         # Try deleting or changing attributes (they are read-only)
-        self.assertRaises(TypeError, delattr, obj.dialect, 'delimiter')
-        self.assertRaises(TypeError, setattr, obj.dialect, 'delimiter', ':')
+        self.assertRaises((TypeError, AttributeError), delattr, obj.dialect,
+                          'delimiter')
+        self.assertRaises((TypeError, AttributeError), setattr, obj.dialect,
+                          'delimiter', ':')
         self.assertRaises(AttributeError, delattr, obj.dialect, 'quoting')
         self.assertRaises(AttributeError, setattr, obj.dialect,
                           'quoting', None)

Modified: pypy/branch/fast-forward/lib_pypy/_csv.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/_csv.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/_csv.py	Thu Oct 14 23:58:54 2010
@@ -70,13 +70,14 @@
 
     The Dialect type records CSV parsing and generation options."""
 
-    __slots__ = ["delimiter", "doublequote", "escapechar", "lineterminator",
-                 "quotechar", "quoting", "skipinitialspace", "strict"]
+    __slots__ = ["_delimiter", "_doublequote", "_escapechar",
+                 "_lineterminator", "_quotechar", "_quoting",
+                 "_skipinitialspace", "_strict"]
 
     def __new__(cls, dialect, **kwargs):
 
         for name in kwargs:
-            if name not in Dialect.__slots__:
+            if '_' + name not in Dialect.__slots__:
                 raise TypeError("unexpected keyword argument '%s'" %
                                 (name,))
 
@@ -120,6 +121,7 @@
         # Copy attributes
         notset = object()
         for name in Dialect.__slots__:
+            name = name[1:]
             value = notset
             if name in kwargs:
                 value = kwargs[name]


From afa at codespeak.net  Fri Oct 15 00:12:58 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 00:12:58 +0200 (CEST)
Subject: [pypy-svn] r77963 - pypy/branch/fast-forward/pypy/module/_io
Message-ID: <20101014221258.70FC6282BEA@codespeak.net>

Author: afa
Date: Fri Oct 15 00:12:56 2010
New Revision: 77963

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
Log:
Translation fix


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Fri Oct 15 00:12:56 2010
@@ -101,7 +101,7 @@
             self.fd = dispatch_filename(rposix.open)(
                 space, w_name, flags, 0666)
         except OSError, e:
-            raise wrap_oserror2(space, e, w_fname)
+            raise wrap_oserror2(space, e, w_name)
         self.closefd = bool(closefd)
 
     def _check_closed(self, space):


From leuschel at codespeak.net  Fri Oct 15 07:02:04 2010
From: leuschel at codespeak.net (leuschel at codespeak.net)
Date: Fri, 15 Oct 2010 07:02:04 +0200 (CEST)
Subject: [pypy-svn] r77964 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015050204.34676282BEA@codespeak.net>

Author: leuschel
Date: Fri Oct 15 07:02:01 2010
New Revision: 77964

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
minor edit


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 07:02:01 2010
@@ -450,7 +450,7 @@
 guard_true($i_{17}$)
 jump($p_{15}$, $p_{10}$)
 \end{lstlisting}
-\caption{Unoptimized Trace for the Simple Object Model}
+\caption{An Unoptimized Trace of the Example Interpreter}
 \label{fig:unopt-trace}
 \end{figure}
 
@@ -473,7 +473,7 @@
     \item \lstinline{set} correspond to attribute writes.
     \item \lstinline{guard_class} correspond to method calls and are followed by
     the trace of the called method.
-    \item \lstinline{int_add} and \lstinline{int_get} are integer addition and
+    \item \lstinline{int_add} and \lstinline{int_gt} are integer addition and
     comparison (``greater than''), respectively.
 \end{itemize}
 
@@ -1134,18 +1134,20 @@
 partial evaluation:
 
 Partially known data structures are built directly into Prolog (via unbound
-logic variables) and thus the optimization of partially static data structures
-was part of partial evaluation of Prolog programs from the beginning
+logic variables) and thus the treatment of partially static data structures
+was part of partial evaluation of Prolog programs from the early stages
 \cite{lloyd_partial_1991}. One effect of unfolding in Prolog is that terms that
-are constructed and immediately matched again completely disappear in the
-residual program, which is equivalent to what our optimization does for an
+are constructed and immediately matched again, completely disappear in the
+residual program. This is similar to what our optimization does for an
 imperative language.
 
- FP
-
+xxx:
 partially static data structures: kenichi asai's thesis?
 
-xxx: relation to compile-time garbage collection \cite{mazur_practical_2001};  separation logic; John Hughes: type specialization
+xxx: 
+Other related work is compile-time garbage collection \cite{mazur_practical_2001}, which
+tries to ....;
+  separation logic; John Hughes: type specialization
 
 \section{Conclusion}
 \label{sec:conclusion}


From hakanardo at codespeak.net  Fri Oct 15 07:21:42 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Fri, 15 Oct 2010 07:21:42 +0200 (CEST)
Subject: [pypy-svn] r77965 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101015052142.8E16E282BEA@codespeak.net>

Author: hakanardo
Date: Fri Oct 15 07:21:40 2010
New Revision: 77965

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
Log:
translation fix

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Fri Oct 15 07:21:40 2010
@@ -78,8 +78,7 @@
             self.emit_operation(newop)
 
             # FIXME: force_lazy_setfield in heap.py may reorder last ops
-            if current > 0:
-                current -= 1
+            current = max(current-1, 0)
                 
             for op in self.optimizer.newoperations[current:]:
                 #print 'E: ', op,  self.optimizer.newoperations.index(op)


From afa at codespeak.net  Fri Oct 15 08:22:57 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 08:22:57 +0200 (CEST)
Subject: [pypy-svn] r77966 - pypy/branch/fast-forward/pypy/module/_io
Message-ID: <20101015062257.1B0A9282BEA@codespeak.net>

Author: afa
Date: Fri Oct 15 08:22:55 2010
New Revision: 77966

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
Log:
Typos


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Fri Oct 15 08:22:55 2010
@@ -9,7 +9,7 @@
     raise OperationError(space.w_ValueError, space.wrap(
         "Must have exactly one of read/write/append mode"))
 
-def decode_mode(spac, mode):
+def decode_mode(space, mode):
     flags = 0
     rwa = False
     readable = False
@@ -32,7 +32,7 @@
         elif s == 'a':
             if rwa:
                 _bad_mode(space)
-            rwa = 1
+            rwa = True
             writable = True
             flags |= os.O_CREAT
             append = True


From afa at codespeak.net  Fri Oct 15 08:40:02 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 08:40:02 +0200 (CEST)
Subject: [pypy-svn] r77967 - pypy/branch/fast-forward/pypy/module/_io
Message-ID: <20101015064002.A60F8282BEA@codespeak.net>

Author: afa
Date: Fri Oct 15 08:40:00 2010
New Revision: 77967

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
Log:
Try to fix translation again


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Fri Oct 15 08:40:00 2010
@@ -3,8 +3,12 @@
 from pypy.interpreter.gateway import interp2app, unwrap_spec, Arguments
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.error import OperationError, wrap_oserror2
+from os import O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_TRUNC
 import os
 
+O_BINARY = getattr(os, "O_BINARY", 0)
+O_APPEND = getattr(os, "O_APPEND", 0)
+
 def _bad_mode(space):
     raise OperationError(space.w_ValueError, space.wrap(
         "Must have exactly one of read/write/append mode"))
@@ -28,13 +32,13 @@
                 _bad_mode(space)
             rwa = True
             writable = True
-            flags |= os.O_CREAT | os.O_TRUNC
+            flags |= O_CREAT | O_TRUNC
         elif s == 'a':
             if rwa:
                 _bad_mode(space)
             rwa = True
             writable = True
-            flags |= os.O_CREAT
+            flags |= O_CREAT
             append = True
         elif s == 'b':
             pass
@@ -51,17 +55,16 @@
         _bad_mode(space)
 
     if readable and writable:
-        flags |= os.O_RDWR
+        flags |= O_RDWR
     elif readable:
-        flags |= os.O_RDONLY
+        flags |= O_RDONLY
     else:
-        flags |= os.O_WRONLY
+        flags |= O_WRONLY
 
-    if hasattr(os, 'O_BINARY'):
-        flags |= os.O_BINARY
+    flags |= O_BINARY
 
-    if hasattr(os, 'O_APPEND') and append:
-        flags |= os.O_APPEND
+    if append:
+        flags |= O_APPEND
 
     return readable, writable, flags
 


From afa at codespeak.net  Fri Oct 15 10:45:33 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 10:45:33 +0200 (CEST)
Subject: [pypy-svn] r77968 -
	pypy/branch/fast-forward/pypy/module/_hashlib/test
Message-ID: <20101015084533.EBA71282BEA@codespeak.net>

Author: afa
Date: Fri Oct 15 10:45:31 2010
New Revision: 77968

Modified:
   pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py
Log:
Avoid __name mangling when running tests with -A


Modified: pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py	Fri Oct 15 10:45:31 2010
@@ -38,7 +38,8 @@
             c_hexdigest = hexdigest
 
             # also test the pure Python implementation
-            h = hashlib.__get_builtin_constructor(name)('')
+            py_new = getattr(hashlib, '__get_builtin_constructor')
+            h = py_new(name)('')
             assert h.digest_size == expected_size
             assert h.digestsize == expected_size
             #


From afa at codespeak.net  Fri Oct 15 11:08:43 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 11:08:43 +0200 (CEST)
Subject: [pypy-svn] r77969 - pypy/branch/fast-forward/lib_pypy/_ctypes
Message-ID: <20101015090843.3608B282BEA@codespeak.net>

Author: afa
Date: Fri Oct 15 11:08:41 2010
New Revision: 77969

Modified:
   pypy/branch/fast-forward/lib_pypy/_ctypes/array.py
Log:
Fix ctypes tests


Modified: pypy/branch/fast-forward/lib_pypy/_ctypes/array.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/_ctypes/array.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/_ctypes/array.py	Fri Oct 15 11:08:41 2010
@@ -109,15 +109,16 @@
 def array_get_slice_params(self, index):
     if index.step is not None:
         raise TypeError("3 arg slices not supported (for no reason)")
+    check_bounds = hasattr(self, '_length_')
     if index.start is not None:
         start = index.start
-        if start < 0:
+        if check_bounds and start < 0:
             start = 0
     else:
         start = 0
     if index.stop is not None:
         stop = index.stop
-        if stop > self._length_:
+        if check_bounds and stop > self._length_:
             stop = self._length_
     else:
         stop = self._length_


From antocuni at codespeak.net  Fri Oct 15 11:10:07 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Fri, 15 Oct 2010 11:10:07 +0200 (CEST)
Subject: [pypy-svn] r77970 - in pypy/branch/jitffi/pypy/jit/metainterp:
	optimizeopt test
Message-ID: <20101015091007.804A4282BEA@codespeak.net>

Author: antocuni
Date: Fri Oct 15 11:10:06 2010
New Revision: 77970

Modified:
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
Log:
- simplify even more the logic of the optimization: now I think it's really
  easy to understand, and moreover it optimize nicely one more case, as the
  new test proves

- reformat the tests to be visually nicer



Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	Fri Oct 15 11:10:06 2010
@@ -13,6 +13,7 @@
     descr = None
 
     def __init__(self, funcval, cpu, prepare_op):
+        self.funcval = funcval
         self.opargs = []
         argtypes, restype = self._get_signature(funcval)
         self.descr = cpu.calldescrof_dynamic(argtypes, restype)
@@ -62,26 +63,31 @@
 class OptFfiCall(Optimization):
 
     def __init__(self):
-        self.funcval = None
         self.funcinfo = None
 
-    def _get_oopspec(self, op):
-        effectinfo = op.getdescr().get_extra_info()
-        if effectinfo is not None:
-            return effectinfo.oopspecindex
-        return EffectInfo.OS_NONE
+    def begin_optimization(self, funcval, op):
+        self.rollback_maybe()
+        self.funcinfo = FuncInfo(funcval, self.optimizer.cpu, op)
 
-    def rollback(self):
-        self.emit_operation(self.funcinfo.prepare_op)
-        for op in self.funcinfo.opargs:
-            self.emit_operation(op)
-        self.funcval = None
+    def commit_optimization(self):
         self.funcinfo = None
 
-    def optimize_default(self, op):
-        if self.funcval:
-            self.rollback()
-        self.emit_operation(op)
+    def rollback_maybe(self):
+        if self.funcinfo is None:
+            return # nothing to rollback
+        #
+        # we immediately set funcinfo to None to prevent recursion when
+        # calling emit_op
+        funcinfo = self.funcinfo
+        self.funcinfo = None
+        self.emit_operation(funcinfo.prepare_op)
+        for op in funcinfo.opargs:
+            self.emit_operation(op)
+
+    def emit_operation(self, op):
+        # we cannot emit any operation during the optimization
+        self.rollback_maybe()
+        Optimization.emit_operation(self, op)
 
     def optimize_CALL(self, op):
         oopspec = self._get_oopspec(op)
@@ -97,43 +103,33 @@
 
     optimize_CALL_MAY_FORCE = optimize_CALL
 
-    def _get_funcval(self, op):
-        funcval = self.getvalue(op.getarg(1))
-        if self.funcval:
-            if self.funcval is not funcval:
-                self.rollback()
-                return None
-        if not funcval.is_constant():
-            return None
-        return funcval
-
     def do_prepare_call(self, op):
+        self.rollback_maybe()
         funcval = self._get_funcval(op)
-        if not funcval:
-            return op
-        assert self.funcval is None # XXX: do something nice etc. etc.
-        self.funcval = funcval
-        self.funcinfo = FuncInfo(funcval, self.optimizer.cpu, op)
+        if not funcval.is_constant():
+            return op # cannot optimize
+        self.begin_optimization(funcval, op)
+        return None
 
     def do_push_arg(self, op):
-        if self.funcval is None:
-            return op
         funcval = self._get_funcval(op)
+        if not self.funcinfo or self.funcinfo.funcval is not funcval:
+            return op # cannot optimize
         self.funcinfo.opargs.append(op)
+        return None
 
     def do_call(self, op):
-        if self.funcval is None:
-            return op
         funcval = self._get_funcval(op)
-        info = self.funcinfo
+        if not self.funcinfo or self.funcinfo.funcval is not funcval:
+            return op # cannot optimize
         funcsymval = self.getvalue(op.getarg(2))
         arglist = [funcsymval.force_box()]
-        for push_op in info.opargs:
+        for push_op in self.funcinfo.opargs:
             argval = self.getvalue(push_op.getarg(2))
             arglist.append(argval.force_box())
-        newop = ResOperation(rop.CALL_MAY_FORCE, arglist, op.result, descr=info.descr)
-        self.funcval = None
-        self.funcinfo = None
+        newop = ResOperation(rop.CALL_MAY_FORCE, arglist, op.result,
+                             descr=self.funcinfo.descr)
+        self.commit_optimization()
         return newop
 
     def propagate_forward(self, op):
@@ -143,6 +139,15 @@
                 func(self, op)
                 break
         else:
-            self.optimize_default(op)
+            self.emit_operation(op)
+
+    def _get_oopspec(self, op):
+        effectinfo = op.getdescr().get_extra_info()
+        if effectinfo is not None:
+            return effectinfo.oopspecindex
+        return EffectInfo.OS_NONE
+
+    def _get_funcval(self, op):
+        return self.getvalue(op.getarg(1))
 
 optimize_ops = _findall(OptFfiCall, 'optimize_')

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	Fri Oct 15 11:10:06 2010
@@ -54,12 +54,16 @@
     
     namespace = namespace.__dict__
 
+    # ----------------------------------------------------------------------
+    # this group of tests is the most important, as they represent the "real"
+    # cases you actually get when using rlib.libffi
+    
     def test_ffi_call_opt(self):
         ops = """
         [i0, f1]
-        call(0, ConstPtr(func),             descr=libffi_prepare)
-        call(0, ConstPtr(func), i0,         descr=libffi_push_arg)
-        call(0, ConstPtr(func), f1,         descr=libffi_push_arg)
+        call(0, ConstPtr(func),                       descr=libffi_prepare)
+        call(0, ConstPtr(func), i0,                   descr=libffi_push_arg)
+        call(0, ConstPtr(func), f1,                   descr=libffi_push_arg)
         i3 = call_may_force(0, ConstPtr(func), 12345, descr=libffi_call)
         guard_not_forced() []
         guard_no_exception() []
@@ -77,9 +81,9 @@
     def test_ffi_call_nonconst(self):
         ops = """
         [i0, f1, p2]
-        call(0, p2,             descr=libffi_prepare)
-        call(0, p2, i0,         descr=libffi_push_arg)
-        call(0, p2, f1,         descr=libffi_push_arg)
+        call(0, p2,                       descr=libffi_prepare)
+        call(0, p2, i0,                   descr=libffi_push_arg)
+        call(0, p2, f1,                   descr=libffi_push_arg)
         i3 = call_may_force(0, p2, 12345, descr=libffi_call)
         guard_not_forced() []
         guard_no_exception() []
@@ -88,13 +92,17 @@
         expected = ops
         loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
 
+    # ----------------------------------------------------------------------
+    # in pratice, the situations described in these tests should never happen,
+    # but we still want to ensure correctness
+
     def test_rollback_if_op_in_between(self):
         ops = """
         [i0, f1]
-        call(0, ConstPtr(func),             descr=libffi_prepare)
-        call(0, ConstPtr(func), i0,         descr=libffi_push_arg)
+        call(0, ConstPtr(func),                       descr=libffi_prepare)
+        call(0, ConstPtr(func), i0,                   descr=libffi_push_arg)
         i1 = int_add(i0, 1)
-        call(0, ConstPtr(func), f1,         descr=libffi_push_arg)
+        call(0, ConstPtr(func), f1,                   descr=libffi_push_arg)
         i3 = call_may_force(0, ConstPtr(func), 12345, descr=libffi_call)
         guard_not_forced() []
         guard_no_exception() []
@@ -106,17 +114,84 @@
     def test_rollback_multiple_calls(self):
         ops = """
         [i0, i2, f1]
-        call(0, ConstPtr(func),             descr=libffi_prepare)
-        call(0, ConstPtr(func), i0,         descr=libffi_push_arg)
-        call(0, ConstPtr(func2),             descr=libffi_prepare) # culprit!
-        call(0, ConstPtr(func), f1,         descr=libffi_push_arg)
-        i3 = call_may_force(0, ConstPtr(func), 12345, descr=libffi_call)
-        call(0, ConstPtr(func2), i0,         descr=libffi_push_arg)
-        call(0, ConstPtr(func2), f1,         descr=libffi_push_arg)
-        i4 = call_may_force(0, ConstPtr(func), 12345, descr=libffi_call)
+        call(0, ConstPtr(func),                        descr=libffi_prepare)
+        call(0, ConstPtr(func),  i0,                   descr=libffi_push_arg)
+        #
+        # this is the culprit!
+        call(0, ConstPtr(func2),                       descr=libffi_prepare)
+        #
+        call(0, ConstPtr(func),  f1,                   descr=libffi_push_arg)
+        i3 = call_may_force(0, ConstPtr(func),  12345, descr=libffi_call)
+        guard_not_forced() []
+        guard_no_exception() []
+        call(0, ConstPtr(func2), i0,                   descr=libffi_push_arg)
+        call(0, ConstPtr(func2), f1,                   descr=libffi_push_arg)
+        i4 = call_may_force(0, ConstPtr(func2), 67890, descr=libffi_call)
+        guard_not_forced() []
+        guard_no_exception() []
+        jump(i3, i4, f1)
+        """
+        expected = ops
+        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+
+    def test_rollback_multiple_prepare(self):
+        ops = """
+        [i0, i2, f1]
+        call(0, ConstPtr(func),                        descr=libffi_prepare)
+        #
+        # this is the culprit!
+        call(0, ConstPtr(func2),                       descr=libffi_prepare)
+        #
+        call(0, ConstPtr(func),  i0,                   descr=libffi_push_arg)
+        call(0, ConstPtr(func),  f1,                   descr=libffi_push_arg)
+        i3 = call_may_force(0, ConstPtr(func),  12345, descr=libffi_call)
+        guard_not_forced() []
+        guard_no_exception() []
+        call(0, ConstPtr(func2), i0,                   descr=libffi_push_arg)
+        call(0, ConstPtr(func2), f1,                   descr=libffi_push_arg)
+        i4 = call_may_force(0, ConstPtr(func2), 67890, descr=libffi_call)
         guard_not_forced() []
         guard_no_exception() []
         jump(i3, i4, f1)
         """
         expected = ops
         loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+
+    def test_optimize_nested_call(self):
+        ops = """
+        [i0, i2, f1]
+        call(0, ConstPtr(func),                        descr=libffi_prepare)
+        #
+        # this "nested" call is nicely optimized
+        call(0, ConstPtr(func2),                       descr=libffi_prepare)
+        call(0, ConstPtr(func2), i0,                   descr=libffi_push_arg)
+        call(0, ConstPtr(func2), f1,                   descr=libffi_push_arg)
+        i4 = call_may_force(0, ConstPtr(func2), 67890, descr=libffi_call)
+        guard_not_forced() []
+        guard_no_exception() []
+        #
+        call(0, ConstPtr(func),  i0,                   descr=libffi_push_arg)
+        call(0, ConstPtr(func),  f1,                   descr=libffi_push_arg)
+        i3 = call_may_force(0, ConstPtr(func),  12345, descr=libffi_call)
+        guard_not_forced() []
+        guard_no_exception() []
+        jump(i3, i4, f1)
+        """
+        expected = """
+        [i0, i2, f1]
+        call(0, ConstPtr(func),                        descr=libffi_prepare)
+        #
+        # this "nested" call is nicely optimized
+        i4 = call_may_force(67890, i0, f1, descr=int_float__int)
+        guard_not_forced() []
+        guard_no_exception() []
+        #
+        call(0, ConstPtr(func),  i0,                   descr=libffi_push_arg)
+        call(0, ConstPtr(func),  f1,                   descr=libffi_push_arg)
+        i3 = call_may_force(0, ConstPtr(func),  12345, descr=libffi_call)
+        guard_not_forced() []
+        guard_no_exception() []
+        jump(i3, i4, f1)
+        """
+        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+


From afa at codespeak.net  Fri Oct 15 11:40:36 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 11:40:36 +0200 (CEST)
Subject: [pypy-svn] r77971 - in pypy/branch/fast-forward/lib_pypy: .
	pypy_test
Message-ID: <20101015094036.C9E2B282BEA@codespeak.net>

Author: afa
Date: Fri Oct 15 11:40:35 2010
New Revision: 77971

Modified:
   pypy/branch/fast-forward/lib_pypy/_md5.py
   pypy/branch/fast-forward/lib_pypy/pypy_test/test_md5_extra.py
   pypy/branch/fast-forward/lib_pypy/pypy_test/test_sha_extra.py
Log:
Try to fix tests


Modified: pypy/branch/fast-forward/lib_pypy/_md5.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/_md5.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/_md5.py	Fri Oct 15 11:40:35 2010
@@ -375,11 +375,14 @@
 
 digest_size = 16
 
-def md5(arg=None):
-    """Same as new().
-
-    For backward compatibility reasons, this is an alternative
-    name for the new() function.
+def new(arg=None):
+    """Return a new md5 crypto object.
+    If arg is present, the method call update(arg) is made.
     """
 
-    return new(arg)
+    crypto = MD5Type()
+    if arg:
+        crypto.update(arg)
+
+    return crypto
+

Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_md5_extra.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/pypy_test/test_md5_extra.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_md5_extra.py	Fri Oct 15 11:40:35 2010
@@ -57,7 +57,7 @@
     d1h = m1.hexdigest()
     
     # Use MD5 module in pure Python.
-    m2 = pymd5.md5()
+    m2 = pymd5.new()
     m2.update(message)
     d2 = m2.digest()
     d2h = m2.hexdigest()
@@ -203,7 +203,7 @@
         m1.update(prefix1)
         m1c = m1.copy()
 
-        m2 = pymd5.md5()
+        m2 = pymd5.new()
         m2.update(prefix1)
         m2c = m2.copy()
 
@@ -222,7 +222,6 @@
 
 def test_attributes():
     assert pymd5.digest_size == 16
-    assert pymd5.digestsize == 16
-    assert pymd5.blocksize == 1
-    assert pymd5.md5().digest_size == 16
-    assert pymd5.md5().digestsize == 16
+    assert pymd5.new().digest_size == 16
+    assert pymd5.new().digestsize == 16
+    assert pymd5.new().block_size == 64

Modified: pypy/branch/fast-forward/lib_pypy/pypy_test/test_sha_extra.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/pypy_test/test_sha_extra.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/pypy_test/test_sha_extra.py	Fri Oct 15 11:40:35 2010
@@ -4,11 +4,11 @@
 # Publication 180-1, Secure Hash Standard,  1995 April 17
 # http://www.itl.nist.gov/div897/pubs/fip180-1.htm
 from __future__ import absolute_import
-from .. import _sha 
+from .. import _sha as pysha
 
 class TestSHA: 
     def check(self, data, digest):
-        computed = sha.new(data).hexdigest()
+        computed = pysha.new(data).hexdigest()
         assert computed == digest
 
     def test_case_1(self):
@@ -25,8 +25,8 @@
 
 
 def test_attributes():
-    assert sha.digest_size == 20
-    assert sha.digestsize == 20
-    assert sha.blocksize == 1
-    assert sha.sha().digest_size == 20
-    assert sha.sha().digestsize == 20
+    assert pysha.digest_size == 20
+    assert pysha.digestsize == 20
+    assert pysha.blocksize == 1
+    assert pysha.new().digest_size == 20
+    assert pysha.new().digestsize == 20


From david at codespeak.net  Fri Oct 15 11:51:26 2010
From: david at codespeak.net (david at codespeak.net)
Date: Fri, 15 Oct 2010 11:51:26 +0200 (CEST)
Subject: [pypy-svn] r77972 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . test
Message-ID: <20101015095126.23C4F282BEA@codespeak.net>

Author: david
Date: Fri Oct 15 11:51:24 2010
New Revision: 77972

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
Log:
Encode ARM imm data instructions

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Fri Oct 15 11:51:24 2010
@@ -10,33 +10,6 @@
         self._data = alloc(1024)
         self._pos = 0
 
-    def ADD_ri(self, rt, rn, imm, cond=cond.AL):
-        # XXX S bit
-        self.write32(cond << 28
-                        | 2 << 24
-                        | 8 << 20
-                        | (rn & 0xF) << 16
-                        | (rt & 0xF) << 12
-                        | (imm & 0xFFF))
-
-    def SUB_ri(self, rd, rn, imm=0, cond=cond.AL, s=0):
-        self.write32(cond << 28
-                        | 9 << 22
-                        | (s & 0x1) << 20
-                        | (rn & 0xF) << 16
-                        | (rd & 0xF) << 12
-                        | (imm & 0xFFF))
-
-    def MOV_ri(self, rt, imm=0, cond=cond.AL):
-        # XXX Check the actual allowed size for imm
-        # XXX S bit
-        self.write32(cond << 28
-                    | 0x3 << 24
-                    | 0xA << 20
-                    #| 0x0 << 16
-                    | (rt & 0xF) << 12
-                    | (imm & 0xFFF))
-
     def PUSH(self, regs, cond=cond.AL):
         assert reg.sp not in regs
         instr = self._encode_reg_list(cond << 28 | 0x92D << 16, regs)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	Fri Oct 15 11:51:24 2010
@@ -17,9 +17,7 @@
                     | (p & 0x1) <<  24
                     | (u & 0x1) << 23
                     | (w & 0x1) << 21
-                    | (rn & 0xFF) << 16
-                    | (rt & 0xFF) << 12
-                    | (imm & 0xFFF))
+                    | imm_operation(rt, rn, imm))
     else:
         def f(self, rt, rn, rm, imm=0, cond=cond.AL, s=0, shifttype=0):
             p = 1
@@ -32,6 +30,31 @@
                         | (w & 0x1) << 21
                         | reg_operation(rt, rn, rm, imm, s, shifttype))
     return f
+def define_data_proc_imm(name, table):
+    n = (0x1 << 25
+        | (table['op'] & 0x1F) << 20)
+    if table['result'] and table['base']:
+        def imm_func(self, rd, rn, imm=0, cond=cond.AL, s=0):
+            # XXX check condition on rn
+            self.write32(n
+                | cond << 28
+                | s << 20
+                | imm_operation(rd, rn, imm))
+    elif not table['base']:
+        def imm_func(self, rd, imm=0, cond=cond.AL, s=0):
+            # XXX check condition on rn
+            self.write32(n
+                | cond << 28
+                | s << 20
+                | imm_operation(rd, 0, imm))
+    else:
+        def imm_func(self, rn, imm=0, cond=cond.AL, s=0):
+            # XXX check condition on rn
+            self.write32(n
+                | cond << 28
+                | s << 20
+                | imm_operation(0, rn, imm))
+    return imm_func
 
 def define_data_proc(name, table):
     n = ((table['op1'] & 0x1F) << 20
@@ -69,6 +92,10 @@
                         | reg_operation(rd, rn, rm, imm, s, shifttype))
     return f
 
+def imm_operation(rt, rn, imm):
+    return ((rn & 0xFF) << 16
+    | (rt & 0xFF) << 12
+    | (imm & 0xFFF))
 
 def reg_operation(rt, rn, rm, imm, s, shifttype):
     # XXX encode shiftype correctly
@@ -79,11 +106,16 @@
             | (shifttype & 0x3) << 5
             | (rm & 0xFF))
 
+def define_instruction(builder, key, val, target):
+        f = builder(key, val)
+        setattr(target, key, f)
+
 def define_instructions(target):
     for key, val in instructions.load_store.iteritems():
-        f = define_load_store_func(key, val)
-        setattr(target, key, f)
+        define_instruction(define_load_store_func, key, val, target)
 
     for key, val in instructions.data_proc.iteritems():
-        f = define_data_proc(key, val)
-        setattr(target, key, f)
+        define_instruction(define_data_proc, key, val, target)
+
+    for key, val in instructions.data_proc_imm.iteritems():
+        define_instruction(define_data_proc_imm, key, val, target)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	Fri Oct 15 11:51:24 2010
@@ -32,3 +32,23 @@
     #'RRX_ri': {'op1':0x1A, 'op2':0, 'op3':0x3, 'op2cond':'0', 'result':False, 'base':True},
     'ROR_ri': {'op1':0x1A, 'op2':0x0, 'op3':0x3, 'op2cond':'!0', 'result':True, 'base':False},
 }
+
+data_proc_imm = {
+    'ADD_ri': {'op': 0, 'rncond':'', 'result':True, 'base':True},
+    'EOR_ri': {'op': 0x2, 'rncond':'', 'result':True, 'base':True},
+    'SUB_ri': {'op': 0x4, 'rncond':'!0xF', 'result':True, 'base':True},
+    #'ADR_ri': {'op': 0x4, 'rncond':'0xF', 'result':True, 'base':True},
+    'RSB_ri': {'op': 0x6, 'rncond':'', 'result':True, 'base':True},
+    'ADD_ri': {'op': 0x8, 'rncond':'!0xF', 'result':True, 'base':True},
+    'ADC_ri': {'op': 0xA, 'rncond':'', 'result':True, 'base':True},
+    'SBC_ri': {'op': 0xC, 'rncond':'', 'result':True, 'base':True},
+    'RSC_ri': {'op': 0xE, 'rncond':'', 'result':True, 'base':True},
+    'TST_ri': {'op': 0x11, 'rncond':'', 'result':False, 'base':True},
+    'TEQ_ri': {'op': 0x13, 'rncond':'', 'result':False, 'base':True},
+    'CMP_ri': {'op': 0x15, 'rncond':'', 'result':False, 'base':True},
+    'CMN_ri': {'op': 0x17, 'rncond':'', 'result':False, 'base':True},
+    'ORR_ri': {'op': 0x18, 'rncond':'', 'result':True, 'base':True},
+    'MOV_ri': {'op': 0x1A, 'rncond':'', 'result':True, 'base':False},
+    'BIC_ri': {'op': 0x1C, 'rncond':'', 'result':True, 'base':True},
+    'MVN_ri': {'op': 0x1E, 'rncond':'', 'result':True, 'base':False},
+}

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	Fri Oct 15 11:51:24 2010
@@ -120,17 +120,43 @@
         self.cb = CodeBuilder()
 
 def build_tests():
+    test_name = 'test_generated_%s'
     for key, value in instructions.load_store.iteritems():
         if value['imm']:
             f = gen_test_imm_func
         else:
             f = gen_test_reg_func
-        test = f(key, value)
-        setattr(TestInstrCodeBuilderForGeneratedInstr, 'test_%s' % key, test)
+        build_test(f, key, value, test_name)
 
     for key, value, in instructions.data_proc.iteritems():
-        test = gen_test_data_reg_func(key, value)
-        setattr(TestInstrCodeBuilderForGeneratedInstr, 'test_%s' % key, test)
+        build_test(gen_test_data_reg_func, key, value, test_name)
+
+    for key, value, in instructions.data_proc_imm.iteritems():
+        build_test(gen_test_data_proc_imm_func, key, value, test_name)
+
+# XXX refactor this functions
+
+def build_test(builder, key, value, test_name):
+    test = builder(key, value)
+    setattr(TestInstrCodeBuilderForGeneratedInstr, test_name % key, test)
+
+def gen_test_data_proc_imm_func(name, table):
+    if table['result'] and table['base']:
+        def f(self):
+            func = getattr(self.cb, name)
+            func(r.r3, r.r7, 23)
+            self.assert_equal('%s r3, r7, #23' % name[:name.index('_')])
+    elif not table['base']:
+        def f(self):
+            func = getattr(self.cb, name)
+            func(r.r3, 23)
+            self.assert_equal('%s r3, #23' % name[:name.index('_')])
+    else:
+        def f(self):
+            func = getattr(self.cb, name)
+            func(r.r3, 23)
+            self.assert_equal('%s r3, #23' % name[:name.index('_')])
+    return f
 
 def gen_test_imm_func(name, table):
     def f(self):


From leuschel at codespeak.net  Fri Oct 15 11:54:20 2010
From: leuschel at codespeak.net (leuschel at codespeak.net)
Date: Fri, 15 Oct 2010 11:54:20 +0200 (CEST)
Subject: [pypy-svn] r77973 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015095420.DDF66282BEA@codespeak.net>

Author: leuschel
Date: Fri Oct 15 11:54:18 2010
New Revision: 77973

Modified:
   pypy/extradoc/talk/pepm2011/paper.bib
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
added reference to constructor specialisation


Modified: pypy/extradoc/talk/pepm2011/paper.bib
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.bib	(original)
+++ pypy/extradoc/talk/pepm2011/paper.bib	Fri Oct 15 11:54:18 2010
@@ -342,3 +342,13 @@
 	year = {2008},
 	pages = {123--139}
 }
+ at inproceedings{Mogensen:PEPM93,
+   author =    {Torben Mogensen},
+   title =     {Constructor Specialization},
+   booktitle = {Proceedings of PEPM'93, the ACM Sigplan Symposium on Partial 
+                Evaluation and Semantics-Based Program Manipulation},
+   year =      1993,
+   month =     {June},
+   publisher = {ACM Press},
+   pages =     {22--32},
+}
\ No newline at end of file

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 11:54:18 2010
@@ -1140,6 +1140,8 @@
 are constructed and immediately matched again, completely disappear in the
 residual program. This is similar to what our optimization does for an
 imperative language.
+In functional programming this idea was introduced as
+ constructor specialisation by Mogensen \cite{Mogensen:PEPM93}.)
 
 xxx:
 partially static data structures: kenichi asai's thesis?


From leuschel at codespeak.net  Fri Oct 15 13:23:46 2010
From: leuschel at codespeak.net (leuschel at codespeak.net)
Date: Fri, 15 Oct 2010 13:23:46 +0200 (CEST)
Subject: [pypy-svn] r77974 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015112346.E1C73282BEA@codespeak.net>

Author: leuschel
Date: Fri Oct 15 13:23:43 2010
New Revision: 77974

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
some minor changes + requeusts


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 13:23:43 2010
@@ -876,6 +876,8 @@
 recursively lifting its fields.
 
 
+XXX Apply the rules of the algorithm to parts of the sample trace
+
 % subsection Optimizing Traces (end)
 
 \subsection{Analysis of the Algorithm}
@@ -887,9 +889,10 @@
 single pass over the list of operations.  We can check that altough
 recursively lifting a static object is not a constant-time operation,
 the algorithm only takes a total time linear in the length of the trace.
-Moreover, it gives the ``best'' possible result within its constrains,
+Moreover, it gives the ``best'' possible result within its constraints,
+\marginpar{Why? What are the constraints ? Justify}
 \eg in term of the number of residual operations.  The
-algorithm itself is not particularly complex or innovative; our focus is
+algorithm itself is not particularly complex; our focus is
 rather that \emph{in the context of tracing JITs} it is possible to find a
 simple enough algorithm that still gives the best results.
 
@@ -921,6 +924,9 @@
 
 \cfbolz{probably can be cut in case of space problems}
 
+XXX unclear what the relationship is to the technique; has it been used in the experiments??
+XXX Maybe move to discussion and related work section?
+
 One problem to the successful application of the allocation removal
 techniques described in the previous sections is the presence of
 frame-introspection features in many dynamic languages and the fact
@@ -928,7 +934,7 @@
 allow the programmer to get access to the frames object that the
 interpreter uses to store local variables. This is a useful feature,
 as it makes the implementation of a debugger possible in Python
-without needing much explicit support from the VM level. On the other
+without extra support from the VM. On the other
 hand, it severely hinders the effectiveness of allocation removal,
 because every time an object is stored into a local variable, it is
 stored into the frame-object, which makes it escape.
@@ -1143,13 +1149,14 @@
 In functional programming this idea was introduced as
  constructor specialisation by Mogensen \cite{Mogensen:PEPM93}.)
 
-xxx:
-partially static data structures: kenichi asai's thesis?
+%xxx:
+%partially static data structures: kenichi asai's thesis?
 
 xxx: 
 Other related work is compile-time garbage collection \cite{mazur_practical_2001}, which
 tries to ....;
-  separation logic; John Hughes: type specialization
+  separation logic
+  %; John Hughes: type specialization
 
 \section{Conclusion}
 \label{sec:conclusion}


From afa at codespeak.net  Fri Oct 15 13:47:48 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 13:47:48 +0200 (CEST)
Subject: [pypy-svn] r77975 -
	pypy/branch/fast-forward/lib-python/modified-2.7.0
Message-ID: <20101015114748.D6A81282BEA@codespeak.net>

Author: afa
Date: Fri Oct 15 13:47:47 2010
New Revision: 77975

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/subprocess.py
      - copied, changed from r77967, pypy/branch/fast-forward/lib-python/2.7.0/subprocess.py
Log:
Don't rely on reference counting to close the non-heritable handle.
This avoids a deadlock on Windows, when stdout=PIPE.


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/subprocess.py (from r77967, pypy/branch/fast-forward/lib-python/2.7.0/subprocess.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/subprocess.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/subprocess.py	Fri Oct 15 13:47:47 2010
@@ -792,7 +792,7 @@
             elif stderr == PIPE:
                 errread, errwrite = _subprocess.CreatePipe(None, 0)
             elif stderr == STDOUT:
-                errwrite = c2pwrite
+                errwrite = c2pwrite.handle # pass id to not close it
             elif isinstance(stderr, int):
                 errwrite = msvcrt.get_osfhandle(stderr)
             else:
@@ -807,9 +807,13 @@
 
         def _make_inheritable(self, handle):
             """Return a duplicate of handle, which is inheritable"""
-            return _subprocess.DuplicateHandle(_subprocess.GetCurrentProcess(),
+            dupl = _subprocess.DuplicateHandle(_subprocess.GetCurrentProcess(),
                                 handle, _subprocess.GetCurrentProcess(), 0, 1,
                                 _subprocess.DUPLICATE_SAME_ACCESS)
+            # If the initial handle was obtained with CreatePipe, close it.
+            if not isinstance(handle, int):
+                handle.Close()
+            return dupl
 
 
         def _find_w9xpopen(self):


From cfbolz at codespeak.net  Fri Oct 15 14:06:00 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 15 Oct 2010 14:06:00 +0200 (CEST)
Subject: [pypy-svn] r77976 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015120600.B22B1282BEA@codespeak.net>

Author: cfbolz
Date: Fri Oct 15 14:05:59 2010
New Revision: 77976

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
tone down claims a bit


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 14:05:59 2010
@@ -886,20 +886,17 @@
 While we do not offer a formal proof of it, it should be relatively clear
 that the algorithm presented above is sound: it works by delaying (and
 often completely removing) some operations.  The algorithm runs in a
-single pass over the list of operations.  We can check that altough
+single pass over the list of operations.  We can check that although
 recursively lifting a static object is not a constant-time operation,
 the algorithm only takes a total time linear in the length of the trace.
-Moreover, it gives the ``best'' possible result within its constraints,
-\marginpar{Why? What are the constraints ? Justify}
-\eg in term of the number of residual operations.  The
-algorithm itself is not particularly complex; our focus is
+The algorithm itself is not particularly complex; our focus is
 rather that \emph{in the context of tracing JITs} it is possible to find a
-simple enough algorithm that still gives the best results.
+simple enough algorithm that still gives very good results.
 
 Note in particular that objects in category 1 (\ie the ones that do
 not escape) are completely removed; moreover, objects in category 2
 (\ie escaping) are still partially dealt with: if such an object
-escapes later than its creation point, all the operations inbetween that
+escapes later than its creation point, all the operations in between that
 involve the object are removed.
 
 The optimization is particularly effective for chains of operations.


From cfbolz at codespeak.net  Fri Oct 15 14:21:38 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 15 Oct 2010 14:21:38 +0200 (CEST)
Subject: [pypy-svn] r77977 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015122138.829F1282BEA@codespeak.net>

Author: cfbolz
Date: Fri Oct 15 14:21:37 2010
New Revision: 77977

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
kill the section about frames, as michael thinks its too confusing. Instead
mention frames minimally in the evaluation section.


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 14:21:37 2010
@@ -188,10 +188,7 @@
 objects that are allocated in the trace \emph{static,} which
 means that they do not occur any more in the optimized trace. This technique is
 informally described in Section~\ref{sec:statics}; a more formal description is
-given in Section~\ref{sec:formal}.
-
-In Section~\ref{sec:frames} we describe some supporting techniques that are not
-central to the approach, but are essential to improve the results. The introduced
+given in Section~\ref{sec:formal}. The introduced
 techniques are evaluated in Section~\ref{sec:Evaluation} using PyPy's Python
 interpreter as a case study.
 
@@ -903,10 +900,8 @@
 For example, it is typical for an interpreter to generate sequences of
 writes-followed-by-reads, where one interpreted opcode writes to some
 object's field and the next interpreted opcode reads it back, possibly
-dispatching on the type of the object created just before.  In the case
-of PyPy's Python interpreter, this optimization can even remove the
-allocation of all intermediate frames that occur in the interpreter,
-corresponding to all calls that have been inlined in the trace.
+dispatching on the type of the object created just before.  A typical example
+would be chains of arithmetic operations.
 
 % subsection Analysis of the Algorithm (end)
 
@@ -916,44 +911,28 @@
 
 % section Escape Analysis in a Tracing JIT (end)
 
-\section{Taming Frame Objects}
-\label{sec:frames}
-
-\cfbolz{probably can be cut in case of space problems}
-
-XXX unclear what the relationship is to the technique; has it been used in the experiments??
-XXX Maybe move to discussion and related work section?
-
-One problem to the successful application of the allocation removal
-techniques described in the previous sections is the presence of
-frame-introspection features in many dynamic languages and the fact
-that frames are heap allocated. Languages such as Python and Smalltalk
-allow the programmer to get access to the frames object that the
-interpreter uses to store local variables. This is a useful feature,
-as it makes the implementation of a debugger possible in Python
-without extra support from the VM. On the other
-hand, it severely hinders the effectiveness of allocation removal,
-because every time an object is stored into a local variable, it is
-stored into the frame-object, which makes it escape.
-
-This problem is solved by making it possible to the interpreter author
-to add some hints into the source code to declare instances of one
-class as frame objects needing special treatment. The JIT will then
-fill these objects only lazily when they are actually accessed (\eg
-because a debugger is used) using values from JIT-emitted code runtime
-locations (typically the CPU stack). Therefore in the common case,
-nothing is stored into the frame objects, making the problem of too
-much escaping go away. This special handling of frame objects is a common approach in VM implementations
-\cite{miranda_context_1999,andreas_gal_trace-based_2009}. The only novelty in our approach lays in
-its generality: all the delicate support code for this is generated, as opposed to
-most other JITs, which are just specifically written for one particular
-language.
-
-% section Taming Frame Objects (end)
-
-\section{Evaluation}
+\section{Implementation and Evaluation}
 \label{sec:Evaluation}
 
+The allocation removal techniques described in this paper were implemented in
+the optimizer of PyPy's tracing JIT. The optimization is independent of which
+interpreter a JIT is generated for. There are some practical issues beyond the
+techniques described in this paper. The actual implementation needs to deal with
+more operations than described in Section~\ref{sec:formal}, for example to
+also support static arrays in addition to static objects. The implementation of
+this optimization is about 400 lines of RPython code.
+
+A further complication is that most interpreters written with PyPy use
+heap-allocated frame objects to store local variables. Those severely hinders
+the effectiveness of allocation removal, because every time an object is stored
+into a local variable, it is stored into the frame-object, which makes it
+escape. We implemented a general technique to treat such frames objects in a
+special way to solve this problem. This is a common approach in VM
+implementations \cite{miranda_context_1999,andreas_gal_trace-based_2009}. The
+only novelty in our approach lays in its generality: most other JITs, are just
+specifically written for one particular language instead of being generated from
+an interpreter.
+
 To evaluate the effectiveness of our allocation removal algorithm, we look at
 the effectiveness when used in the tracing JIT of PyPy's Python interpreter. The
 benchmarks we used are small-to-medium Python programs, some synthetic


From cfbolz at codespeak.net  Fri Oct 15 14:28:37 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 15 Oct 2010 14:28:37 +0200 (CEST)
Subject: [pypy-svn] r77978 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015122837.5ADFE282BEA@codespeak.net>

Author: cfbolz
Date: Fri Oct 15 14:28:35 2010
New Revision: 77978

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
 - shorten a bit
 - mention size of python interpreter


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 14:28:35 2010
@@ -923,18 +923,21 @@
 this optimization is about 400 lines of RPython code.
 
 A further complication is that most interpreters written with PyPy use
-heap-allocated frame objects to store local variables. Those severely hinders
+heap-allocated frame objects to store local variables. Those severely hinder
 the effectiveness of allocation removal, because every time an object is stored
 into a local variable, it is stored into the frame-object, which makes it
-escape. We implemented a general technique to treat such frames objects in a
+escape. We implemented a technique to treat such frames objects in a
 special way to solve this problem. This is a common approach in VM
-implementations \cite{miranda_context_1999,andreas_gal_trace-based_2009}. The
-only novelty in our approach lays in its generality: most other JITs, are just
-specifically written for one particular language instead of being generated from
-an interpreter.
+implementations \cite{miranda_context_1999,andreas_gal_trace-based_2009}, the
+novelty of our approach is that we generalized it enough to be usable for
+different interpreter.
+
+To evaluate our allocation removal algorithm, we look at the effectiveness when
+used in the generated tracing JIT of PyPy's Python interpreter. This interpreter
+is a full implementation of Python 2.5 language semantics and is about 30'000
+lines of code.
 
-To evaluate the effectiveness of our allocation removal algorithm, we look at
-the effectiveness when used in the tracing JIT of PyPy's Python interpreter. The
+The
 benchmarks we used are small-to-medium Python programs, some synthetic
 benchmarks, some real applications.\footnote{All the source code of the
 benchmarks can be found at \texttt{http://codespeak.net/svn/pypy/benchmarks/}.
@@ -947,7 +950,7 @@
 
 Furthermore there are the following benchmarks:
 \begin{itemize}
-    \item \textbf{crypto\_pyaes}: AES implementation.
+    \item \textbf{crypto\_pyaes}: An AES implementation.
     \item \textbf{django}: The templating engine of the Django web
     framework\footnote{\texttt{http://www.djangoproject.com/}}.
     \item \textbf{go}: A Monte-Carlo Go


From arigo at codespeak.net  Fri Oct 15 14:37:19 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 14:37:19 +0200 (CEST)
Subject: [pypy-svn] r77979 - in pypy/branch/leak-finder/pypy: . annotation
	jit/backend/llsupport jit/backend/test jit/backend/x86
	jit/backend/x86/test jit/metainterp jit/metainterp/test
	module/cpyext/test rlib/test rpython rpython/lltypesystem
	rpython/lltypesystem/test rpython/memory
	rpython/memory/gctransform rpython/test tool tool/test
	translator/c/test
Message-ID: <20101015123719.3F83E36E0C3@codespeak.net>

Author: arigo
Date: Fri Oct 15 14:37:16 2010
New Revision: 77979

Added:
   pypy/branch/leak-finder/pypy/tool/leakfinder.py
   pypy/branch/leak-finder/pypy/tool/test/test_leakfinder.py
Modified:
   pypy/branch/leak-finder/pypy/annotation/builtin.py
   pypy/branch/leak-finder/pypy/conftest.py
   pypy/branch/leak-finder/pypy/jit/backend/llsupport/gc.py
   pypy/branch/leak-finder/pypy/jit/backend/llsupport/llmodel.py
   pypy/branch/leak-finder/pypy/jit/backend/test/runner_test.py
   pypy/branch/leak-finder/pypy/jit/backend/x86/assembler.py
   pypy/branch/leak-finder/pypy/jit/backend/x86/regalloc.py
   pypy/branch/leak-finder/pypy/jit/backend/x86/runner.py
   pypy/branch/leak-finder/pypy/jit/backend/x86/test/test_assembler.py
   pypy/branch/leak-finder/pypy/jit/metainterp/test/test_basic.py
   pypy/branch/leak-finder/pypy/jit/metainterp/virtualref.py
   pypy/branch/leak-finder/pypy/module/cpyext/test/test_cpyext.py
   pypy/branch/leak-finder/pypy/rlib/test/test_rdynload.py
   pypy/branch/leak-finder/pypy/rlib/test/test_rzlib.py
   pypy/branch/leak-finder/pypy/rpython/llinterp.py
   pypy/branch/leak-finder/pypy/rpython/lltypesystem/llmemory.py
   pypy/branch/leak-finder/pypy/rpython/lltypesystem/lltype.py
   pypy/branch/leak-finder/pypy/rpython/lltypesystem/rclass.py
   pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_ll2ctypes.py
   pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_llmemory.py
   pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_lltype.py
   pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_rffi.py
   pypy/branch/leak-finder/pypy/rpython/memory/gctransform/transform.py
   pypy/branch/leak-finder/pypy/rpython/memory/gcwrapper.py
   pypy/branch/leak-finder/pypy/rpython/memory/support.py
   pypy/branch/leak-finder/pypy/rpython/rbuiltin.py
   pypy/branch/leak-finder/pypy/rpython/test/test_llinterp.py
   pypy/branch/leak-finder/pypy/rpython/test/test_nongc.py
   pypy/branch/leak-finder/pypy/rpython/test/test_rptr.py
   pypy/branch/leak-finder/pypy/translator/c/test/test_lltyped.py
Log:
In-progress: remove the 'self.mallocs' from llinterp, and instead
use systematically the one from lltype.py introduced around r73335.


Modified: pypy/branch/leak-finder/pypy/annotation/builtin.py
==============================================================================
--- pypy/branch/leak-finder/pypy/annotation/builtin.py	(original)
+++ pypy/branch/leak-finder/pypy/annotation/builtin.py	Fri Oct 15 14:37:16 2010
@@ -423,7 +423,7 @@
 from pypy.annotation.model import SomePtr
 from pypy.rpython.lltypesystem import lltype
 
-def malloc(s_T, s_n=None, s_flavor=None, s_zero=None):
+def malloc(s_T, s_n=None, s_flavor=None, s_zero=None, s_track_allocation=None):
     assert (s_n is None or s_n.knowntype == int
             or issubclass(s_n.knowntype, pypy.rlib.rarithmetic.base_int))
     assert s_T.is_constant()
@@ -438,13 +438,15 @@
         r = SomePtr(lltype.typeOf(p))
     else:
         assert s_flavor.is_constant()
+        assert s_track_allocation is None or s_track_allocation.is_constant()
         # not sure how to call malloc() for the example 'p' in the
         # presence of s_extraargs
         r = SomePtr(lltype.Ptr(s_T.const))
     return r
 
-def free(s_p, s_flavor):
+def free(s_p, s_flavor, s_track_allocation=None):
     assert s_flavor.is_constant()
+    assert s_track_allocation is None or s_track_allocation.is_constant()
     # same problem as in malloc(): some flavors are not easy to
     # malloc-by-example
     #T = s_p.ll_ptrtype.TO

Modified: pypy/branch/leak-finder/pypy/conftest.py
==============================================================================
--- pypy/branch/leak-finder/pypy/conftest.py	(original)
+++ pypy/branch/leak-finder/pypy/conftest.py	Fri Oct 15 14:37:16 2010
@@ -7,6 +7,7 @@
 from inspect import isclass, getmro
 from pypy.tool.udir import udir
 from pypy.tool.autopath import pypydir
+from pypy.tool import leakfinder
 
 # pytest settings
 pytest_plugins = "resultlog",
@@ -354,7 +355,14 @@
 
     def runtest(self):
         try:
-            super(IntTestFunction, self).runtest()
+            leakfinder.start_tracking_allocations()
+            try:
+                super(IntTestFunction, self).runtest()
+            finally:
+                if leakfinder.TRACK_ALLOCATIONS:
+                    leaks = leakfinder.stop_tracking_allocations(False)
+                else:
+                    leaks = None   # stop_tracking_allocations() already called
         except OperationError, e:
             check_keyboard_interrupt(e)
             raise
@@ -373,6 +381,8 @@
                 _pygame_imported = True
                 assert option.view, ("should not invoke Pygame "
                                      "if conftest.option.view is False")
+        if leaks:        # check for leaks, but only if the test passed so far
+            raise leakfinder.MallocMismatch(leaks)
 
 class AppTestFunction(PyPyTestFunction):
     def _prunetraceback(self, traceback):

Modified: pypy/branch/leak-finder/pypy/jit/backend/llsupport/gc.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/backend/llsupport/gc.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/backend/llsupport/gc.py	Fri Oct 15 14:37:16 2010
@@ -158,7 +158,7 @@
         # used to avoid too many duplications in the GCREF_LISTs.
         self.hashtable = lltype.malloc(self.HASHTABLE,
                                        self.HASHTABLE_SIZE+1,
-                                       flavor='raw')
+                                       flavor='raw', track_allocation=False)
         dummy = lltype.direct_ptradd(lltype.direct_arrayitems(self.hashtable),
                                      self.HASHTABLE_SIZE)
         dummy = llmemory.cast_ptr_to_adr(dummy)
@@ -252,14 +252,15 @@
 
     def _enlarge_gcmap(self):
         newlength = 250 + self._gcmap_maxlength * 2
-        newgcmap = lltype.malloc(self.GCMAP_ARRAY, newlength, flavor='raw')
+        newgcmap = lltype.malloc(self.GCMAP_ARRAY, newlength, flavor='raw',
+                                 track_allocation=False)
         oldgcmap = self._gcmap
         for i in range(self._gcmap_curlength):
             newgcmap[i] = oldgcmap[i]
         self._gcmap = newgcmap
         self._gcmap_maxlength = newlength
         if oldgcmap:
-            lltype.free(oldgcmap, flavor='raw')
+            lltype.free(oldgcmap, flavor='raw', track_allocation=False)
 
     def get_basic_shape(self, is_64_bit=False):
         # XXX: Should this code even really know about stack frame layout of
@@ -308,7 +309,8 @@
         # them inside bigger arrays) and we never try to share them.
         length = len(shape)
         compressed = lltype.malloc(self.CALLSHAPE_ARRAY, length,
-                                   flavor='raw')
+                                   flavor='raw',
+                                   track_allocation=False)   # memory leak
         for i in range(length):
             compressed[length-1-i] = rffi.cast(rffi.UCHAR, shape[i])
         return llmemory.cast_ptr_to_adr(compressed)

Modified: pypy/branch/leak-finder/pypy/jit/backend/llsupport/llmodel.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/backend/llsupport/llmodel.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/backend/llsupport/llmodel.py	Fri Oct 15 14:37:16 2010
@@ -82,7 +82,8 @@
         # read back by the machine code reading at the address given by
         # pos_exception() and pos_exc_value().
         _exception_emulator = lltype.malloc(rffi.CArray(lltype.Signed), 2,
-                                            zero=True, flavor='raw')
+                                            zero=True, flavor='raw',
+                                            immortal=True)
         self._exception_emulator = _exception_emulator
 
         def _store_exception(lle):

Modified: pypy/branch/leak-finder/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/backend/test/runner_test.py	Fri Oct 15 14:37:16 2010
@@ -1304,6 +1304,7 @@
                                    descr=fd)
             res = self.execute_operation(get_op, [s_box], 'int', descr=fd)
             assert res.getint()  == 32
+        lltype.free(s, flavor='raw')
 
     def test_new_with_vtable(self):
         cpu = self.cpu

Modified: pypy/branch/leak-finder/pypy/jit/backend/x86/assembler.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/backend/x86/assembler.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/backend/x86/assembler.py	Fri Oct 15 14:37:16 2010
@@ -249,7 +249,8 @@
 
     def _build_float_constants(self):
         # 44 bytes: 32 bytes for the data, and up to 12 bytes for alignment
-        addr = lltype.malloc(rffi.CArray(lltype.Char), 44, flavor='raw')
+        addr = lltype.malloc(rffi.CArray(lltype.Char), 44, flavor='raw',
+                             track_allocation=False)
         if not we_are_translated():
             self._keepalive_malloced_float_consts = addr
         float_constants = rffi.cast(lltype.Signed, addr)
@@ -399,7 +400,8 @@
             funcname = "" % len(self.loop_run_counters)
         # invent the counter, so we don't get too confused
         if self._debug:
-            struct = lltype.malloc(DEBUG_COUNTER, flavor='raw')
+            struct = lltype.malloc(DEBUG_COUNTER, flavor='raw',
+                                   track_allocation=False)   # known to leak
             struct.i = 0
             self.loop_run_counters.append((len(self.loop_run_counters), struct))
         return funcname

Modified: pypy/branch/leak-finder/pypy/jit/backend/x86/regalloc.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/backend/x86/regalloc.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/backend/x86/regalloc.py	Fri Oct 15 14:37:16 2010
@@ -70,8 +70,9 @@
 
     def _get_new_array(self):
         n = self.BASE_CONSTANT_SIZE
+        # known to leak
         self.cur_array = lltype.malloc(rffi.CArray(lltype.Float), n,
-                                       flavor='raw')
+                                       flavor='raw', track_allocation=False)
         self.cur_array_free = n
     _get_new_array._dont_inline_ = True
 

Modified: pypy/branch/leak-finder/pypy/jit/backend/x86/runner.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/backend/x86/runner.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/backend/x86/runner.py	Fri Oct 15 14:37:16 2010
@@ -113,7 +113,8 @@
         return CPU386.cast_adr_to_int(adr)
 
     all_null_registers = lltype.malloc(rffi.LONGP.TO, 24,
-                                       flavor='raw', zero=True)
+                                       flavor='raw', zero=True,
+                                       immortal=True)
 
     def force(self, addr_of_force_index):
         TP = rffi.CArrayPtr(lltype.Signed)

Modified: pypy/branch/leak-finder/pypy/jit/backend/x86/test/test_assembler.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/backend/x86/test/test_assembler.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/backend/x86/test/test_assembler.py	Fri Oct 15 14:37:16 2010
@@ -81,7 +81,8 @@
 
     # also test rebuild_faillocs_from_descr(), which should not
     # reproduce the holes at all
-    bytecode = lltype.malloc(rffi.UCHARP.TO, len(mc.content), flavor='raw')
+    bytecode = lltype.malloc(rffi.UCHARP.TO, len(mc.content), flavor='raw',
+                             immortal=True)
     for i in range(len(mc.content)):
         assert 0 <= mc.content[i] <= 255
         bytecode[i] = rffi.cast(rffi.UCHAR, mc.content[i])
@@ -115,7 +116,8 @@
         assert withfloats
         value = random.random() - 0.5
         # make sure it fits into 64 bits
-        tmp = lltype.malloc(rffi.LONGP.TO, 2, flavor='raw')
+        tmp = lltype.malloc(rffi.LONGP.TO, 2, flavor='raw',
+                            track_allocation=False)
         rffi.cast(rffi.DOUBLEP, tmp)[0] = value
         return rffi.cast(rffi.DOUBLEP, tmp)[0], tmp[0], tmp[1]
 
@@ -152,10 +154,12 @@
 
     # prepare the expected target arrays, the descr_bytecode,
     # the 'registers' and the 'stack' arrays according to 'content'
-    xmmregisters = lltype.malloc(rffi.LONGP.TO, 16+ACTUAL_CPU.NUM_REGS+1, flavor='raw')
+    xmmregisters = lltype.malloc(rffi.LONGP.TO, 16+ACTUAL_CPU.NUM_REGS+1,
+                                 flavor='raw', immortal=True)
     registers = rffi.ptradd(xmmregisters, 16)
     stacklen = baseloc + 10
-    stack = lltype.malloc(rffi.LONGP.TO, stacklen, flavor='raw')
+    stack = lltype.malloc(rffi.LONGP.TO, stacklen, flavor='raw',
+                          immortal=True)
     expected_ints = [0] * len(content)
     expected_ptrs = [lltype.nullptr(llmemory.GCREF.TO)] * len(content)
     expected_floats = [0.0] * len(content)
@@ -221,7 +225,7 @@
     descr_bytecode.append(0x00)
     descr_bytecode.append(0xCC)   # end marker
     descr_bytes = lltype.malloc(rffi.UCHARP.TO, len(descr_bytecode),
-                                flavor='raw')
+                                flavor='raw', immortal=True)
     for i in range(len(descr_bytecode)):
         assert 0 <= descr_bytecode[i] <= 255
         descr_bytes[i] = rffi.cast(rffi.UCHAR, descr_bytecode[i])

Modified: pypy/branch/leak-finder/pypy/jit/metainterp/test/test_basic.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/metainterp/test/test_basic.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/metainterp/test/test_basic.py	Fri Oct 15 14:37:16 2010
@@ -1751,7 +1751,7 @@
             c = bool(p1)
             d = not bool(p2)
             return 1000*a + 100*b + 10*c + d
-        prebuilt = [lltype.malloc(TP, flavor='raw')] * 2
+        prebuilt = [lltype.malloc(TP, flavor='raw', immortal=True)] * 2
         expected = f(0, 1)
         assert self.interp_operations(f, [0, 1]) == expected
 

Modified: pypy/branch/leak-finder/pypy/jit/metainterp/virtualref.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/metainterp/virtualref.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/metainterp/virtualref.py	Fri Oct 15 14:37:16 2010
@@ -16,7 +16,8 @@
             ('virtualref_index', lltype.Signed),
             ('forced', rclass.OBJECTPTR))
         self.jit_virtual_ref_vtable = lltype.malloc(rclass.OBJECT_VTABLE,
-                                                    zero=True, flavor='raw')
+                                                    zero=True, flavor='raw',
+                                                    immortal=True)
         self.jit_virtual_ref_vtable.name = rclass.alloc_array_name(
             'jit_virtual_ref')
         # build some constants

Modified: pypy/branch/leak-finder/pypy/module/cpyext/test/test_cpyext.py
==============================================================================
--- pypy/branch/leak-finder/pypy/module/cpyext/test/test_cpyext.py	(original)
+++ pypy/branch/leak-finder/pypy/module/cpyext/test/test_cpyext.py	Fri Oct 15 14:37:16 2010
@@ -126,17 +126,17 @@
         for w_obj in lost_objects_w:
             print >>sys.stderr, "Lost object %r" % (w_obj, )
             leaking = True
-        for llvalue in set(ll2ctypes.ALLOCATED.values()) - self.frozen_ll2callocations:
-            if getattr(llvalue, "_traceback", None): # this means that the allocation should be tracked
-                leaking = True
-                print >>sys.stderr, "Did not deallocate %r (ll2ctypes)" % (llvalue, )
-                print >>sys.stderr, "\t" + "\n\t".join(llvalue._traceback.splitlines())
-        for llvalue in lltype.ALLOCATED.keys():
+##        for llvalue in set(ll2ctypes.ALLOCATED.values()) - self.frozen_ll2callocations:
+##            if llvalue in lltype.ALLOCATED:
+##                leaking = True
+##                print >>sys.stderr, "Did not deallocate %r (ll2ctypes)" % (llvalue, )
+##                print >>sys.stderr, "\t" + "\n\t".join(llvalue._traceback.splitlines())
+        for llvalue, traceback in lltype.ALLOCATED.items():
             leaking = True
             print >>sys.stderr, "Did not deallocate %r (llvalue)" % (llvalue, )
-            print >>sys.stderr, "\t" + "\n\t".join(llvalue._traceback.splitlines())
+            print >>sys.stderr, "\t" + "\n\t".join(traceback.splitlines())
 
-        lltype.stop_tracking_allocations()
+        lltype.stop_tracking_allocations(check=False)
         return leaking
 
 class AppTestCpythonExtensionBase(LeakCheckingTest):

Modified: pypy/branch/leak-finder/pypy/rlib/test/test_rdynload.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rlib/test/test_rdynload.py	(original)
+++ pypy/branch/leak-finder/pypy/rlib/test/test_rdynload.py	Fri Oct 15 14:37:16 2010
@@ -5,11 +5,18 @@
 
 class TestDLOperations:
     def test_dlopen(self):
-        py.test.raises(DLOpenError, "dlopen(rffi.str2charp('xxxxxxxxxxxx'))")
-        assert dlopen(rffi.str2charp(get_libc_name()))
+        s = rffi.str2charp('xxxxxxxxxxxx')
+        py.test.raises(DLOpenError, "dlopen(s)")
+        rffi.free_charp(s)
+        #
+        s = rffi.str2charp(get_libc_name())
+        assert dlopen(s)
+        rffi.free_charp(s)
 
     def test_dlsym(self):
-        lib = dlopen(rffi.str2charp(get_libc_name()))
+        s = rffi.str2charp(get_libc_name())
+        lib = dlopen(s)
+        rffi.free_charp(s)
         handle = rffi.cast(lltype.Ptr(lltype.FuncType([lltype.Signed],
                            lltype.Signed)), dlsym(lib, 'abs'))
         assert 1 == handle(1)

Modified: pypy/branch/leak-finder/pypy/rlib/test/test_rzlib.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rlib/test/test_rzlib.py	(original)
+++ pypy/branch/leak-finder/pypy/rlib/test/test_rzlib.py	Fri Oct 15 14:37:16 2010
@@ -189,6 +189,8 @@
     assert unused3 == len('more_garbage')
     assert data3 == ''
 
+    rzlib.deflateEnd(stream)
+
 
 def test_decompress_max_length():
     """
@@ -205,6 +207,8 @@
     assert finished2 is True
     assert unused2 == 0
 
+    rzlib.deflateEnd(stream)
+
 
 def test_cornercases():
     """
@@ -215,6 +219,7 @@
     bytes += rzlib.compress(stream, "")
     bytes += rzlib.compress(stream, "", rzlib.Z_FINISH)
     assert zlib.decompress(bytes) == ""
+    rzlib.deflateEnd(stream)
 
     stream = rzlib.inflateInit()
     data, finished, unused = rzlib.decompress(stream, "")
@@ -228,3 +233,4 @@
         assert finished is False
         assert unused > 0
         buf = buf[-unused:]
+    rzlib.deflateEnd(stream)

Modified: pypy/branch/leak-finder/pypy/rpython/llinterp.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/llinterp.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/llinterp.py	Fri Oct 15 14:37:16 2010
@@ -48,8 +48,7 @@
 
     current_interpreter = None
 
-    def __init__(self, typer, tracing=True, exc_data_ptr=None,
-                 malloc_check=True):
+    def __init__(self, typer, tracing=True, exc_data_ptr=None):
         self.bindings = {}
         self.typer = typer
         # 'heap' is module or object that provides malloc, etc for lltype ops
@@ -57,9 +56,7 @@
         self.exc_data_ptr = exc_data_ptr
         self.frame_stack = []
         self.tracer = None
-        self.malloc_check = malloc_check
         self.frame_class = LLFrame
-        self.mallocs = {}
         if tracing:
             self.tracer = Tracer()
 
@@ -163,24 +160,6 @@
             return self.exc_data_ptr
         return None
 
-    def remember_malloc(self, ptr, llframe):
-        # err....
-        self.mallocs[ptr._obj] = llframe
-
-    def remember_free(self, ptr):
-        try:
-            del self.mallocs[ptr._obj]
-        except KeyError:
-            self._rehash_mallocs()
-            del self.mallocs[ptr._obj]
-
-    def _rehash_mallocs(self):
-        # rehashing is needed because some objects' hash may change
-        # when being turned to 
-        items = self.mallocs.items()
-        self.mallocs = {}
-        self.mallocs.update(items)
-
     def _store_exception(self, exc):
         raise PleaseOverwriteStoreException("You just invoked ll2ctypes callback without overwriting _store_exception on llinterpreter")
 
@@ -726,13 +705,13 @@
     def op_malloc(self, obj, flags):
         flavor = flags['flavor']
         zero = flags.get('zero', False)
+        track_allocation = flags.get('track_allocation', True)
         if flavor == "stack":
             result = self.heap.malloc(obj, zero=zero, flavor='raw')
             self.alloca_objects.append(result)
             return result
-        ptr = self.heap.malloc(obj, zero=zero, flavor=flavor)
-        if flavor == 'raw' and self.llinterpreter.malloc_check:
-            self.llinterpreter.remember_malloc(ptr, self)
+        ptr = self.heap.malloc(obj, zero=zero, flavor=flavor,
+                               track_allocation=track_allocation)
         return ptr
 
     def op_malloc_varsize(self, obj, flags, size):
@@ -741,8 +720,6 @@
         assert flavor in ('gc', 'raw')
         try:
             ptr = self.heap.malloc(obj, size, zero=zero, flavor=flavor)
-            if flavor == 'raw' and self.llinterpreter.malloc_check:
-                self.llinterpreter.remember_malloc(ptr, self)
             return ptr
         except MemoryError:
             self.make_llexception()
@@ -759,11 +736,10 @@
         zero = flags.get('zero', False)
         return self.heap.malloc_nonmovable(TYPE, size, zero=zero)
 
-    def op_free(self, obj, flavor):
-        assert isinstance(flavor, str)
-        if flavor == 'raw' and self.llinterpreter.malloc_check:
-            self.llinterpreter.remember_free(obj)
-        self.heap.free(obj, flavor=flavor)
+    def op_free(self, obj, flags):
+        assert flags['flavor'] == 'raw'
+        track_allocation = flags.get('track_allocation', True)
+        self.heap.free(obj, flavor='raw', track_allocation=track_allocation)
 
     def op_shrink_array(self, obj, smallersize):
         return self.heap.shrink_array(obj, smallersize)

Modified: pypy/branch/leak-finder/pypy/rpython/lltypesystem/llmemory.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/lltypesystem/llmemory.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/lltypesystem/llmemory.py	Fri Oct 15 14:37:16 2010
@@ -105,11 +105,13 @@
         if (isinstance(self.TYPE, lltype.ContainerType)
             and self.TYPE._gckind == 'gc'):
             assert self.repeat == 1
-            p = lltype.malloc(self.TYPE, flavor='raw', zero=zero)
+            p = lltype.malloc(self.TYPE, flavor='raw', zero=zero,
+                              track_allocation=False)
             return cast_ptr_to_adr(p)
         else:
             T = lltype.FixedSizeArray(self.TYPE, self.repeat)
-            p = lltype.malloc(T, flavor='raw', zero=zero)
+            p = lltype.malloc(T, flavor='raw', zero=zero,
+                              track_allocation=False)
             array_adr = cast_ptr_to_adr(p)
             return array_adr + ArrayItemsOffset(T)
 
@@ -288,7 +290,8 @@
             count = 0
         p = lltype.malloc(parenttype or self.TYPE, count,
                           immortal = self.TYPE._gckind == 'raw',
-                          zero = zero)
+                          zero = zero,
+                          track_allocation = False)
         return cast_ptr_to_adr(p)
 
     def raw_memcopy(self, srcadr, dstadr):

Modified: pypy/branch/leak-finder/pypy/rpython/lltypesystem/lltype.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/lltypesystem/lltype.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/lltypesystem/lltype.py	Fri Oct 15 14:37:16 2010
@@ -1,7 +1,3 @@
-import StringIO
-import traceback
-import sys
-
 import py
 from pypy.rlib.rarithmetic import (r_int, r_uint, intmask, r_singlefloat,
                                    r_ulonglong, r_longlong, base_int,
@@ -10,25 +6,13 @@
 from pypy.tool.uid import Hashable
 from pypy.tool.tls import tlsobject
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 from types import NoneType
 from sys import maxint
 import weakref
 
 TLS = tlsobject()
 
-# Track allocations to detect memory leaks
-# Don't track 'gc' and immortal mallocs
-TRACK_ALLOCATIONS = False
-ALLOCATED = identity_dict()
-
-def start_tracking_allocations():
-    global TRACK_ALLOCATIONS
-    TRACK_ALLOCATIONS = True
-    ALLOCATED.clear()
-
-def stop_tracking_allocations():
-    global TRACK_ALLOCATIONS
-    TRACK_ALLOCATIONS = False
 
 class _uninitialized(object):
     def __init__(self, TYPE):
@@ -1380,41 +1364,21 @@
     __slots__ = ('_TYPE',
                  '_parent_type', '_parent_index', '_keepparent',
                  '_wrparent',
-                 '__weakref__', '_traceback',
-                 '__storage')
+                 '__weakref__',
+                 '_storage')
 
-    def __init__(self, TYPE, track_allocation=None):
+    def __init__(self, TYPE):
         self._wrparent = None
         self._TYPE = TYPE
         self._storage = True    # means "use default storage", as opposed to:
                                 #    None            - container was freed
                                 #     - using ctypes
                                 #                      (see ll2ctypes.py)
-        if track_allocation is not False and TRACK_ALLOCATIONS:
-            self._traceback = self._get_traceback()
-            ALLOCATED[self] = None
-        else:
-            self._traceback = None
-
-    def _get_traceback(self):
-        frame = sys._getframe().f_back.f_back.f_back.f_back
-        sio = StringIO.StringIO()
-        traceback.print_stack(frame, file=sio)
-        return sio.getvalue()
 
     def _free(self):
         self._check()   # no double-frees
         self._storage = None
 
-    def _storage_get(self):
-        return self.__storage
-
-    def _storage_set(self, value):
-        self.__storage = value
-        if value is not True and self in ALLOCATED:
-            del ALLOCATED[self]
-    _storage = property(_storage_get, _storage_set)
-
     def _was_freed(self):
         if self._storage is None:
             return True
@@ -1493,12 +1457,12 @@
 
     __slots__ = ('_hash_cache_', '_compilation_info')
 
-    def __new__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None, track_allocation=None):
+    def __new__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None):
         my_variety = _struct_variety(TYPE._names)
         return object.__new__(my_variety)
 
-    def __init__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None, track_allocation=None):
-        _parentable.__init__(self, TYPE, track_allocation)
+    def __init__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None):
+        _parentable.__init__(self, TYPE)
         if n is not None and TYPE._arrayfld is None:
             raise TypeError("%r is not variable-sized" % (TYPE,))
         if n is None and TYPE._arrayfld is not None:
@@ -1506,8 +1470,7 @@
         first, FIRSTTYPE = TYPE._first_struct()
         for fld, typ in TYPE._flds.items():
             if fld == TYPE._arrayfld:
-                value = _array(typ, n, initialization=initialization, parent=self, parentindex=fld,
-                               track_allocation=track_allocation)
+                value = _array(typ, n, initialization=initialization, parent=self, parentindex=fld)
             else:
                 value = typ._allocate(initialization=initialization, parent=self, parentindex=fld)
             setattr(self, fld, value)
@@ -1568,12 +1531,12 @@
 
     __slots__ = ('items',)
 
-    def __init__(self, TYPE, n, initialization=None, parent=None, parentindex=None, track_allocation=None):
+    def __init__(self, TYPE, n, initialization=None, parent=None, parentindex=None):
         if not isinstance(n, int):
             raise TypeError, "array length must be an int"
         if n < 0:
             raise ValueError, "negative array length"
-        _parentable.__init__(self, TYPE, track_allocation)
+        _parentable.__init__(self, TYPE)
         try:
             myrange = range(n)
         except OverflowError:
@@ -1640,7 +1603,7 @@
     _cache = weakref.WeakKeyDictionary()  # parentarray -> {subarrays}
 
     def __init__(self, TYPE, parent, baseoffset_or_fieldname):
-        _parentable.__init__(self, TYPE, track_allocation=False)
+        _parentable.__init__(self, TYPE)
         self._setparentstructure(parent, baseoffset_or_fieldname)
         # Keep the parent array alive, we share the same allocation.
         # Don't do it if we are inside a GC object, though -- it's someone
@@ -1648,6 +1611,13 @@
         if typeOf(top_container(parent))._gckind == 'raw':
             self._keepparent = parent
 
+    def __str__(self):
+        parent = self._wrparent()
+        if parent is None:
+            return '_subarray at %s in already freed' % (self._parent_index,)
+        return '_subarray at %r in %s' % (self._parent_index,
+                                          parent._TYPE)
+
     def __repr__(self):
         parent = self._wrparent()
         if parent is None:
@@ -1861,8 +1831,9 @@
         return id(self.value)
 
 
-def malloc(T, n=None, flavor='gc', immortal=False, zero=False):
-    assert flavor != 'cpy'
+def malloc(T, n=None, flavor='gc', immortal=False, zero=False,
+           track_allocation=True):
+    assert flavor in ('gc', 'raw')
     if zero or immortal:
         initialization = 'example'
     elif flavor == 'raw':
@@ -1870,9 +1841,9 @@
     else:
         initialization = 'malloc'
     if isinstance(T, Struct):
-        o = _struct(T, n, initialization=initialization, track_allocation=flavor == "raw" and not immortal)
+        o = _struct(T, n, initialization=initialization)
     elif isinstance(T, Array):
-        o = _array(T, n, initialization=initialization, track_allocation=flavor == "raw" and not immortal)
+        o = _array(T, n, initialization=initialization)
     elif isinstance(T, OpaqueType):
         assert n is None
         o = _opaque(T, initialization=initialization)
@@ -1880,15 +1851,19 @@
         raise TypeError, "malloc for Structs and Arrays only"
     if T._gckind != 'gc' and not immortal and flavor.startswith('gc'):
         raise TypeError, "gc flavor malloc of a non-GC non-immortal structure"
+    if flavor == "raw" and not immortal and track_allocation:
+        leakfinder.remember_malloc(o, framedepth=2)
     solid = immortal or not flavor.startswith('gc') # immortal or non-gc case
     return _ptr(Ptr(T), o, solid)
 
-def free(p, flavor):
+def free(p, flavor, track_allocation=True):
     if flavor.startswith('gc'):
         raise TypeError, "gc flavor free"
     T = typeOf(p)
     if not isinstance(T, Ptr) or p._togckind() != 'raw':
         raise TypeError, "free(): only for pointers to non-gc containers"
+    if track_allocation:
+        leakfinder.remember_free(p._obj0)
     p._obj0._free()
 
 def functionptr(TYPE, name, **attrs):

Modified: pypy/branch/leak-finder/pypy/rpython/lltypesystem/rclass.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/lltypesystem/rclass.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/lltypesystem/rclass.py	Fri Oct 15 14:37:16 2010
@@ -420,7 +420,7 @@
         return cast_pointer(self.lowleveltype, result)
 
     def create_instance(self):
-        return malloc(self.object_type, flavor=self.gcflavor)
+        return malloc(self.object_type, flavor=self.gcflavor, immortal=True)
 
     def initialize_prebuilt_data(self, value, classdef, result):
         if self.classdef is not None:

Modified: pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_ll2ctypes.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_ll2ctypes.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_ll2ctypes.py	Fri Oct 15 14:37:16 2010
@@ -765,6 +765,7 @@
         assert abs(float(b[1]) - 1.1) < 1E-6
         assert isinstance(b[2], rffi.r_singlefloat)
         assert abs(float(b[2]) - 2.2) < 1E-6
+        lltype.free(a, flavor='raw')
 
     def test_different_signatures(self):
         if sys.platform=='win32':
@@ -879,6 +880,7 @@
         qsort(rffi.cast(rffi.VOIDP, a), 5, rffi.sizeof(rffi.INT), compare)
         for i in range(5):
             assert a[i] == i + 1
+        lltype.free(a, flavor='raw')
 
     def test_array_type_bug(self):
         A = lltype.Array(lltype.Signed)

Modified: pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_llmemory.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_llmemory.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_llmemory.py	Fri Oct 15 14:37:16 2010
@@ -324,12 +324,14 @@
     p_t = lltype.malloc(T)
     assert p_t.s == lltype.nullptr(S)
     # raw malloc does not
-    p_raw_t = lltype.malloc(T, flavor="raw")
-    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.s")
+    U = lltype.Struct("U", ('x', lltype.Signed))
+    p_raw_t = lltype.malloc(U, flavor="raw")
+    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.x")
+    lltype.free(p_raw_t, flavor="raw")
     # this sort of raw_malloc too
-    p_raw_t = cast_adr_to_ptr(raw_malloc(sizeof(T)), lltype.Ptr(T))
-    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.s")
-    
+    p_raw_t = cast_adr_to_ptr(raw_malloc(sizeof(U)), lltype.Ptr(U))
+    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.x")
+
 
 def test_raw_malloc_signed_bunch():
     adr = raw_malloc(sizeof(lltype.Signed) * 50)
@@ -601,7 +603,8 @@
     a = lltype.malloc(A, flavor='raw')
     src = cast_ptr_to_adr(a) + itemoffsetof(A, 0)
     raw_memclear(src, sizeof(lltype.Signed) * 0)
-    
+    lltype.free(a, flavor="raw")
+
 def test_nonneg():
     S1 = lltype.GcStruct('S1', ('x', lltype.Float))
     A1 = lltype.GcArray(lltype.Float)

Modified: pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_lltype.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_lltype.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_lltype.py	Fri Oct 15 14:37:16 2010
@@ -2,6 +2,7 @@
 from pypy.rpython.lltypesystem.lltype import *
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 
 def isweak(p, T):
     try:
@@ -804,22 +805,20 @@
 
 
 class TestTrackAllocation:
-    def setup_method(self, func):
-        start_tracking_allocations()
-
-    def teardown_method(self, func):
-        assert not lltype.ALLOCATED, "Memory was not correctly freed"
-        stop_tracking_allocations()
+    def test_automatic_tracking(self):
+        # calls to start_tracking_allocations/stop_tracking_allocations
+        # should occur automatically from pypy/conftest.py.  Check that.
+        assert leakfinder.TRACK_ALLOCATIONS
 
     def test_track_allocation(self):
         """A malloc'd buffer fills the ALLOCATED dictionary"""
-        assert lltype.TRACK_ALLOCATIONS
-        assert not lltype.ALLOCATED
+        assert leakfinder.TRACK_ALLOCATIONS
+        assert not leakfinder.ALLOCATED
         buf = malloc(Array(Signed), 1, flavor="raw")
-        assert len(lltype.ALLOCATED) == 1
-        assert lltype.ALLOCATED.keys() == [buf._obj]
+        assert len(leakfinder.ALLOCATED) == 1
+        assert leakfinder.ALLOCATED.keys() == [buf._obj]
         free(buf, flavor="raw")
-        assert not lltype.ALLOCATED
+        assert not leakfinder.ALLOCATED
 
     def test_str_from_buffer(self):
         """gc-managed memory does not need to be freed"""
@@ -828,16 +827,22 @@
         for i in range(size): raw_buf[i] = 'a'
         rstr = rffi.str_from_buffer(raw_buf, gc_buf, size, size)
         rffi.keep_buffer_alive_until_here(raw_buf, gc_buf)
-        assert not lltype.ALLOCATED
+        assert not leakfinder.ALLOCATED
 
     def test_leak_traceback(self):
         """Test info stored for allocated items"""
         buf = malloc(Array(Signed), 1, flavor="raw")
-        traceback = lltype.ALLOCATED.keys()[0]._traceback
+        traceback = leakfinder.ALLOCATED.values()[0]
         lines = traceback.splitlines()
         assert 'malloc(' in lines[-1] and 'flavor="raw")' in lines[-1]
 
-        # XXX The traceback should not be too long
+        # The traceback should not be too long
         print traceback
 
         free(buf, flavor="raw")
+
+    def test_no_tracking(self):
+        p1 = malloc(Array(Signed), 1, flavor='raw', track_allocation=False)
+        p2 = malloc(Array(Signed), 1, flavor='raw', track_allocation=False)
+        free(p2, flavor='raw', track_allocation=False)
+        # p1 is not freed

Modified: pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_rffi.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_rffi.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/lltypesystem/test/test_rffi.py	Fri Oct 15 14:37:16 2010
@@ -9,7 +9,7 @@
 from pypy.rpython.lltypesystem.rstr import STR
 from pypy.rpython.lltypesystem import lltype
 from pypy.tool.udir import udir
-from pypy.rpython.test.test_llinterp import interpret, MallocMismatch
+from pypy.rpython.test.test_llinterp import interpret
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.annotation.annrpython import RPythonAnnotator
 from pypy.rpython.rtyper import RPythonTyper

Modified: pypy/branch/leak-finder/pypy/rpython/memory/gctransform/transform.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/memory/gctransform/transform.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/memory/gctransform/transform.py	Fri Oct 15 14:37:16 2010
@@ -430,7 +430,8 @@
         return self.parenttransformer.gct_malloc_varsize(hop)
     
     def gct_free(self, hop):
-        flavor = hop.spaceop.args[1].value
+        flags = hop.spaceop.args[1].value
+        flavor = flags['flavor']
         assert flavor == 'raw'
         return self.parenttransformer.gct_free(hop)
 
@@ -606,7 +607,8 @@
 
     def gct_free(self, hop):
         op = hop.spaceop
-        flavor = op.args[1].value
+        flags = op.args[1].value
+        flavor = flags['flavor']
         v = op.args[0]
         assert flavor != 'cpy', "cannot free CPython objects directly"
         if flavor == 'raw':

Modified: pypy/branch/leak-finder/pypy/rpython/memory/gcwrapper.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/memory/gcwrapper.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/memory/gcwrapper.py	Fri Oct 15 14:37:16 2010
@@ -42,7 +42,8 @@
     #
     # Interface for the llinterp
     #
-    def malloc(self, TYPE, n=None, flavor='gc', zero=False):
+    def malloc(self, TYPE, n=None, flavor='gc', zero=False,
+               track_allocation=True):
         if flavor == 'gc':
             typeid = self.get_type_id(TYPE)
             addr = self.gc.malloc(typeid, n, zero=zero)
@@ -51,7 +52,8 @@
                 gctypelayout.zero_gc_pointers(result)
             return result
         else:
-            return lltype.malloc(TYPE, n, flavor=flavor, zero=zero)
+            return lltype.malloc(TYPE, n, flavor=flavor, zero=zero,
+                                 track_allocation=track_allocation)
 
     def malloc_nonmovable(self, TYPE, n=None, zero=False):
         typeid = self.get_type_id(TYPE)
@@ -69,9 +71,10 @@
             return self.gc.shrink_array(addr, smallersize)
         return False
 
-    def free(self, TYPE, flavor='gc'):
+    def free(self, TYPE, flavor='gc', track_allocation=True):
         assert flavor != 'gc'
-        return lltype.free(TYPE, flavor=flavor)
+        return lltype.free(TYPE, flavor=flavor,
+                           track_allocation=track_allocation)
 
     def setfield(self, obj, fieldname, fieldvalue):
         STRUCT = lltype.typeOf(obj).TO

Modified: pypy/branch/leak-finder/pypy/rpython/memory/support.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/memory/support.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/memory/support.py	Fri Oct 15 14:37:16 2010
@@ -30,7 +30,8 @@
                 # we zero-initialize the chunks to make the translation
                 # backends happy, but we don't need to do it at run-time.
                 zero = not we_are_translated()
-                return lltype.malloc(CHUNK, flavor="raw", zero=zero)
+                return lltype.malloc(CHUNK, flavor="raw", zero=zero,
+                                     track_allocation=False)
                 
             result = self.free_list
             self.free_list = result.next
@@ -44,7 +45,7 @@
                 # Don't cache the old chunks but free them immediately.
                 # Helps debugging, and avoids that old chunks full of
                 # addresses left behind by a test end up in genc...
-                lltype.free(chunk, flavor="raw")
+                lltype.free(chunk, flavor="raw", track_allocation=False)
 
     unused_chunks = FreeList()
     cache[chunk_size] = unused_chunks, null_chunk

Modified: pypy/branch/leak-finder/pypy/rpython/rbuiltin.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/rbuiltin.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/rbuiltin.py	Fri Oct 15 14:37:16 2010
@@ -345,17 +345,22 @@
 BUILTIN_TYPER[object.__init__] = rtype_object__init__
 # annotation of low-level types
 
-def rtype_malloc(hop, i_flavor=None, i_zero=None):
+def rtype_malloc(hop, i_flavor=None, i_zero=None, i_track_allocation=None):
     assert hop.args_s[0].is_constant()
     vlist = [hop.inputarg(lltype.Void, arg=0)]
     opname = 'malloc'
-    v_flavor, v_zero = parse_kwds(hop, (i_flavor, lltype.Void), (i_zero, None))
+    v_flavor, v_zero, v_track_allocation = parse_kwds(hop,
+        (i_flavor, lltype.Void),
+        (i_zero, None),
+        (i_track_allocation, None))
 
     flags = {'flavor': 'gc'}
     if v_flavor is not None:
         flags['flavor'] = v_flavor.value
     if i_zero is not None:
         flags['zero'] = v_zero.value
+    if i_track_allocation is not None:
+        flags['track_allocation'] = v_track_allocation.value
     vlist.append(hop.inputconst(lltype.Void, flags))
         
     if hop.nb_args == 2:
@@ -366,10 +371,19 @@
     hop.exception_is_here()
     return hop.genop(opname, vlist, resulttype = hop.r_result.lowleveltype)
 
-def rtype_free(hop, i_flavor):
-    assert i_flavor == 1
+def rtype_free(hop, i_flavor, i_track_allocation=None):
+    vlist = [hop.inputarg(hop.args_r[0], arg=0)]
+    v_flavor, v_track_allocation = parse_kwds(hop,
+        (i_flavor, lltype.Void),
+        (i_track_allocation, None))
+    #
+    assert v_flavor is not None and v_flavor.value == 'raw'
+    flags = {'flavor': 'raw'}
+    if i_track_allocation is not None:
+        flags['track_allocation'] = v_track_allocation.value
+    vlist.append(hop.inputconst(lltype.Void, flags))
+    #
     hop.exception_cannot_occur()
-    vlist = hop.inputargs(hop.args_r[0], lltype.Void)
     hop.genop('free', vlist)
 
 def rtype_const_result(hop):
@@ -584,8 +598,9 @@
     vinst, = hop.inputargs(hop.args_r[0])
     flavor = hop.args_r[0].gcflavor
     assert flavor != 'gc'
-    cflavor = hop.inputconst(lltype.Void, flavor)
-    return hop.genop('free', [vinst, cflavor])
+    flags = {'flavor': flavor}
+    cflags = hop.inputconst(lltype.Void, flags)
+    return hop.genop('free', [vinst, cflags])
     
 BUILTIN_TYPER[objectmodel.free_non_gc_object] = rtype_free_non_gc_object
 

Modified: pypy/branch/leak-finder/pypy/rpython/test/test_llinterp.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/test/test_llinterp.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/test/test_llinterp.py	Fri Oct 15 14:37:16 2010
@@ -12,13 +12,11 @@
 from pypy.annotation.model import lltype_to_annotation
 from pypy.rlib.rarithmetic import r_uint, ovfcheck
 from pypy.rpython.ootypesystem import ootype
+from pypy.tool import leakfinder
 from pypy import conftest
 
 # switch on logging of interp to show more info on failing tests
 
-class MallocMismatch(Exception):
-    pass
-
 def setup_module(mod):
     mod.logstate = py.log._getstate()
     py.log.setconsumer("llinterp", py.log.STDOUT)
@@ -72,7 +70,7 @@
 
 def get_interpreter(func, values, view='auto', viewbefore='auto', policy=None,
                     someobjects=False, type_system="lltype", backendopt=False,
-                    config=None, malloc_check=True, **extraconfigopts):
+                    config=None, **extraconfigopts):
     extra_key = [(key, value) for key, value in extraconfigopts.iteritems()]
     extra_key.sort()
     extra_key = tuple(extra_key)
@@ -97,7 +95,7 @@
                                    viewbefore, policy, type_system=type_system,
                                    backendopt=backendopt, config=config,
                                    **extraconfigopts)
-        interp = LLInterpreter(typer, malloc_check=malloc_check)
+        interp = LLInterpreter(typer)
         _tcache[key] = (t, interp, graph)
         # keep the cache small 
         _lastinterpreted.append(key) 
@@ -115,10 +113,17 @@
     interp, graph = get_interpreter(func, values, view, viewbefore, policy,
                                     someobjects, type_system=type_system,
                                     backendopt=backendopt, config=config,
-                                    malloc_check=malloc_check, **kwargs)
-    result = interp.eval_graph(graph, values)
-    if malloc_check and interp.mallocs:
-        raise MallocMismatch(interp.mallocs)
+                                    **kwargs)
+    if not malloc_check:
+        result = interp.eval_graph(graph, values)
+    else:
+        prev = leakfinder.start_tracking_allocations()
+        try:
+            result = interp.eval_graph(graph, values)
+        finally:
+            leaks = leakfinder.stop_tracking_allocations(False, prev)
+        if leaks:
+            raise leakfinder.MallocMismatch(leaks)
     return result
 
 def interpret_raises(exc, func, values, view='auto', viewbefore='auto',
@@ -418,6 +423,7 @@
             assert result
 
 def test_stack_malloc():
+    py.test.skip("stack-flavored mallocs no longer supported")
     class A(object):
         pass
     def f():
@@ -430,6 +436,7 @@
     assert result == 1
 
 def test_invalid_stack_access():
+    py.test.skip("stack-flavored mallocs no longer supported")
     class A(object):
         pass
     globala = A()
@@ -605,7 +612,7 @@
         if x:
             free(t, flavor='raw')
     interpret(f, [1])
-    py.test.raises(MallocMismatch, "interpret(f, [0])")
+    py.test.raises(leakfinder.MallocMismatch, "interpret(f, [0])")
     
     def f():
         t1 = malloc(T, flavor='raw')

Modified: pypy/branch/leak-finder/pypy/rpython/test/test_nongc.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/test/test_nongc.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/test/test_nongc.py	Fri Oct 15 14:37:16 2010
@@ -79,7 +79,7 @@
     py.test.raises(TypeError,rtyper.specialize) # results in an invalid cast
 
 def test_isinstance():
-    class A:
+    class A(object):
         _alloc_flavor_ = "raw"
     class B(A):
         pass
@@ -95,7 +95,24 @@
             o = B()
         else:
             o = C()
-        return 100*isinstance(o, A)+10*isinstance(o, B)+1*isinstance(o ,C)
+        res = 100*isinstance(o, A) + 10*isinstance(o, B) + 1*isinstance(o, C)
+        if i == 0:
+            pass
+        elif i == 1:
+            assert isinstance(o, A)
+            free_non_gc_object(o)
+        elif i == 2:
+            assert isinstance(o, B)
+            free_non_gc_object(o)
+        else:
+            assert isinstance(o, C)
+            free_non_gc_object(o)
+        return res
+
+    assert f(1) == 100
+    assert f(2) == 110
+    assert f(3) == 111
+    assert f(0) == 0
 
     a = RPythonAnnotator()
     #does not raise:
@@ -131,10 +148,14 @@
             d = b
         elif i == 2:
             e = c
-        return (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
+        res =  (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
                 0x0008*(a is e) | 0x0010*(b is c) | 0x0020*(b is d) |
                 0x0040*(b is e) | 0x0080*(c is d) | 0x0100*(c is e) |
                 0x0200*(d is e))
+        free_non_gc_object(a)
+        free_non_gc_object(b)
+        free_non_gc_object(c)
+        return res
     a = RPythonAnnotator()
     #does not raise:
     s = a.build_types(f, [int])
@@ -169,10 +190,13 @@
             d = b
         elif i == 2:
             e = c
-        return (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
+        res =  (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
                 0x0008*(a is e) | 0x0010*(b is c) | 0x0020*(b is d) |
                 0x0040*(b is e) | 0x0080*(c is d) | 0x0100*(c is e) |
                 0x0200*(d is e))
+        free_non_gc_object(a)
+        free_non_gc_object(b)
+        return res
     a = RPythonAnnotator()
     #does not raise:
     s = a.build_types(f, [int])

Modified: pypy/branch/leak-finder/pypy/rpython/test/test_rptr.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/test/test_rptr.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/test/test_rptr.py	Fri Oct 15 14:37:16 2010
@@ -212,10 +212,31 @@
 
     S = Struct('S', ('x', Signed))
     def fn(n):
-        p = malloc(S, flavor='whatever')
+        p = malloc(S, flavor='raw')
         p.x = n
         result = p.x
-        free(p, flavor='whatever')
+        free(p, flavor='raw')
+        return n
+
+    res = interpret(fn, [23])
+    assert res == 23
+
+    S = Struct('S', ('x', Signed))
+    def fn(n):
+        p = malloc(S, flavor='raw', track_allocation=False)
+        p.x = n
+        result = p.x
+        return n
+
+    res = interpret(fn, [23])
+    assert res == 23
+
+    S = Struct('S', ('x', Signed))
+    def fn(n):
+        p = malloc(S, flavor='raw', track_allocation=False)
+        p.x = n
+        result = p.x
+        free(p, flavor='raw', track_allocation=False)
         return n
 
     res = interpret(fn, [23])

Added: pypy/branch/leak-finder/pypy/tool/leakfinder.py
==============================================================================
--- (empty file)
+++ pypy/branch/leak-finder/pypy/tool/leakfinder.py	Fri Oct 15 14:37:16 2010
@@ -0,0 +1,73 @@
+import sys, gc
+import cStringIO
+import traceback
+
+# Track allocations to detect memory leaks.
+# So far, this is used for lltype.malloc(flavor='raw').
+TRACK_ALLOCATIONS = False
+ALLOCATED = {}
+
+class MallocMismatch(Exception):
+    def __str__(self):
+        dict = self.args[0]
+        dict2 = {}
+        for obj, traceback in dict.items():
+            traceback = traceback.splitlines()
+            if len(traceback) > 8:
+                traceback = ['    ...'] + traceback[-6:]
+            traceback = '\n'.join(traceback)
+            dict2.setdefault(traceback, [])
+            dict2[traceback].append(obj)
+        lines = ['{']
+        for traceback, objs in dict2.items():
+            lines.append('')
+            for obj in objs:
+                lines.append('%s:' % (obj,))
+            lines.append(traceback)
+        lines.append('}')
+        return '\n'.join(lines)
+
+def start_tracking_allocations():
+    global TRACK_ALLOCATIONS
+    if TRACK_ALLOCATIONS:
+        result = ALLOCATED.copy()   # nested start
+    else:
+        result = None
+    TRACK_ALLOCATIONS = True
+    ALLOCATED.clear()
+    return result
+
+def stop_tracking_allocations(check, prev=None):
+    global TRACK_ALLOCATIONS
+    assert TRACK_ALLOCATIONS
+    for i in range(5):
+        if not ALLOCATED:
+            break
+        gc.collect()
+    result = ALLOCATED.copy()
+    ALLOCATED.clear()
+    if prev is None:
+        TRACK_ALLOCATIONS = False
+    else:
+        ALLOCATED.update(prev)
+    if check and result:
+        raise MallocMismatch(result)
+    return result
+
+def remember_malloc(obj, framedepth=1):
+    if TRACK_ALLOCATIONS:
+        frame = sys._getframe(framedepth)
+        sio = cStringIO.StringIO()
+        traceback.print_stack(frame, limit=10, file=sio)
+        tb = sio.getvalue()
+        ALLOCATED[obj] = tb
+
+def remember_free(obj):
+    if TRACK_ALLOCATIONS:
+        if obj not in ALLOCATED:
+            # rehashing is needed because some objects' hash may change
+            # e.g. when lltype objects are turned into 
+            items = ALLOCATED.items()
+            ALLOCATED.clear()
+            ALLOCATED.update(items)
+        del ALLOCATED[obj]

Added: pypy/branch/leak-finder/pypy/tool/test/test_leakfinder.py
==============================================================================
--- (empty file)
+++ pypy/branch/leak-finder/pypy/tool/test/test_leakfinder.py	Fri Oct 15 14:37:16 2010
@@ -0,0 +1,70 @@
+import py
+from pypy.tool import leakfinder
+
+def test_start_stop():
+    leakfinder.start_tracking_allocations()
+    assert leakfinder.TRACK_ALLOCATIONS
+    leakfinder.stop_tracking_allocations(True)
+    assert not leakfinder.TRACK_ALLOCATIONS
+
+def test_start_stop_nested():
+    leakfinder.start_tracking_allocations()
+    p2 = leakfinder.start_tracking_allocations()
+    assert leakfinder.TRACK_ALLOCATIONS
+    leakfinder.stop_tracking_allocations(True, prev=p2)
+    assert leakfinder.TRACK_ALLOCATIONS
+    leakfinder.stop_tracking_allocations(True)
+    assert not leakfinder.TRACK_ALLOCATIONS
+
+def test_remember_free():
+    leakfinder.start_tracking_allocations()
+    x = 1234
+    leakfinder.remember_malloc(x)
+    leakfinder.remember_free(x)
+    leakfinder.stop_tracking_allocations(True)
+
+def test_remember_forget():
+    leakfinder.start_tracking_allocations()
+    x = 1234
+    leakfinder.remember_malloc(x)
+    py.test.raises(leakfinder.MallocMismatch,
+                   leakfinder.stop_tracking_allocations, True)
+
+def test_nested_remember_forget_1():
+    leakfinder.start_tracking_allocations()
+    x = 1234
+    leakfinder.remember_malloc(x)
+    p2 = leakfinder.start_tracking_allocations()
+    leakfinder.stop_tracking_allocations(True, prev=p2)
+    py.test.raises(leakfinder.MallocMismatch,
+                   leakfinder.stop_tracking_allocations, True)
+
+def test_nested_remember_forget_2():
+    p2 = leakfinder.start_tracking_allocations()
+    x = 1234
+    leakfinder.remember_malloc(x)
+    py.test.raises(leakfinder.MallocMismatch,
+                   leakfinder.stop_tracking_allocations, True, prev=p2)
+    leakfinder.stop_tracking_allocations(True)
+
+def test_traceback():
+    leakfinder.start_tracking_allocations()
+    x = 1234
+    leakfinder.remember_malloc(x)
+    res = leakfinder.stop_tracking_allocations(check=False)
+    assert res.keys() == [x]
+    print res[x]
+    assert isinstance(res[x], str)
+    assert 'test_traceback' in res[x]
+    assert 'leakfinder.remember_malloc(x)' in res[x]
+
+def test_malloc_mismatch():
+    import sys, traceback, cStringIO
+    sio = cStringIO.StringIO()
+    traceback.print_stack(sys._getframe(), limit=10, file=sio)
+    tb = sio.getvalue()
+    e = leakfinder.MallocMismatch({1234: tb, 2345: tb})
+    print str(e)
+    # grouped entries for 1234 and 2345
+    assert '1234:\n2345:\n' in str(e) or '2345:\n1234:\n' in str(e)
+    assert tb[-80:] in str(e)

Modified: pypy/branch/leak-finder/pypy/translator/c/test/test_lltyped.py
==============================================================================
--- pypy/branch/leak-finder/pypy/translator/c/test/test_lltyped.py	(original)
+++ pypy/branch/leak-finder/pypy/translator/c/test/test_lltyped.py	Fri Oct 15 14:37:16 2010
@@ -401,6 +401,7 @@
             for i in range(n):
                 p = malloc(S, flavor='raw', zero=True)
                 if p.x != 0 or p.y != 0:
+                    free(p, flavor='raw')
                     return -1
                 p.x = i
                 p.y = i
@@ -418,14 +419,16 @@
         def f(n):
             for length in range(n-1, -1, -1):
                 p = malloc(S, length, flavor='raw', zero=True)
-                if p.x != 0:
-                    return -1
-                p.x = n
-                for j in range(length):
-                    if p.y[j] != 0:
-                        return -3
-                    p.y[j] = n^j
-                free(p, flavor='raw')
+                try:
+                    if p.x != 0:
+                        return -1
+                    p.x = n
+                    for j in range(length):
+                        if p.y[j] != 0:
+                            return -3
+                        p.y[j] = n^j
+                finally:
+                    free(p, flavor='raw')
             return 42
 
         fn = self.getcompiled(f, [int])
@@ -655,7 +658,7 @@
     def test_prebuilt_ll2ctypes_array(self):
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         A = rffi.CArray(Char)
-        a = malloc(A, 6, flavor='raw')
+        a = malloc(A, 6, flavor='raw', immortal=True)
         a[0] = 'a'
         a[1] = 'b'
         a[2] = 'c'
@@ -676,7 +679,7 @@
     def test_ll2ctypes_array_from_c(self):
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         A = rffi.CArray(Char)
-        a = malloc(A, 6, flavor='raw')
+        a = malloc(A, 6, flavor='raw', immortal=True)
         a[0] = 'a'
         a[1] = 'b'
         a[2] = 'c'


From arigo at codespeak.net  Fri Oct 15 14:48:26 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 14:48:26 +0200 (CEST)
Subject: [pypy-svn] r77980 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015124826.12C8E36E0C3@codespeak.net>

Author: arigo
Date: Fri Oct 15 14:48:25 2010
New Revision: 77980

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
Semicolonization.


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 14:48:25 2010
@@ -928,7 +928,7 @@
 into a local variable, it is stored into the frame-object, which makes it
 escape. We implemented a technique to treat such frames objects in a
 special way to solve this problem. This is a common approach in VM
-implementations \cite{miranda_context_1999,andreas_gal_trace-based_2009}, the
+implementations \cite{miranda_context_1999,andreas_gal_trace-based_2009}; the
 novelty of our approach is that we generalized it enough to be usable for
 different interpreter.
 


From arigo at codespeak.net  Fri Oct 15 14:51:58 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 14:51:58 +0200 (CEST)
Subject: [pypy-svn] r77981 - in pypy/branch/leak-finder/pypy/rpython:
	lltypesystem memory/gctransform
Message-ID: <20101015125158.CB37036E0C3@codespeak.net>

Author: arigo
Date: Fri Oct 15 14:51:57 2010
New Revision: 77981

Modified:
   pypy/branch/leak-finder/pypy/rpython/lltypesystem/lloperation.py
   pypy/branch/leak-finder/pypy/rpython/memory/gctransform/transform.py
Log:
Add tracking operations, to be implemented by the C backend in debug mode.


Modified: pypy/branch/leak-finder/pypy/rpython/lltypesystem/lloperation.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/lltypesystem/lloperation.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/lltypesystem/lloperation.py	Fri Oct 15 14:51:57 2010
@@ -403,6 +403,8 @@
     'raw_load':             LLOp(sideeffects=False),
     'raw_store':            LLOp(),
     'stack_malloc':         LLOp(), # mmh
+    'track_alloc_start':    LLOp(),
+    'track_alloc_stop':     LLOp(),
     'adr_add':              LLOp(canfold=True),
     'adr_sub':              LLOp(canfold=True),
     'adr_delta':            LLOp(canfold=True),

Modified: pypy/branch/leak-finder/pypy/rpython/memory/gctransform/transform.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/memory/gctransform/transform.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/memory/gctransform/transform.py	Fri Oct 15 14:51:57 2010
@@ -533,6 +533,8 @@
                           resulttype=llmemory.Address)
         if flags.get('zero'):
             hop.genop("raw_memclear", [v_raw, c_size])
+        if flags.get('track_allocation', True):
+            hop.genop("track_alloc_start", [v_raw])
         return v_raw
 
     def gct_fv_stack_malloc(self, hop, flags, TYPE, c_size):
@@ -603,6 +605,8 @@
                                [self.raw_malloc_varsize_ptr, v_length,
                                 c_const_size, c_item_size, c_offset_to_length],
                                resulttype=llmemory.Address)
+        if flags.get('track_allocation', True):
+            hop.genop("track_alloc_start", [v_raw])
         return v_raw
 
     def gct_free(self, hop):
@@ -613,6 +617,8 @@
         assert flavor != 'cpy', "cannot free CPython objects directly"
         if flavor == 'raw':
             v = hop.genop("cast_ptr_to_adr", [v], resulttype=llmemory.Address)
+            if flags.get('track_allocation', True):
+                hop.genop("track_alloc_stop", [v])
             hop.genop('raw_free', [v])
         else:
             assert False, "%s has no support for free with flavor %r" % (self, flavor)           


From arigo at codespeak.net  Fri Oct 15 15:04:36 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 15:04:36 +0200 (CEST)
Subject: [pypy-svn] r77982 - pypy/branch/leak-finder/pypy/rpython
Message-ID: <20101015130436.0C84536C22E@codespeak.net>

Author: arigo
Date: Fri Oct 15 15:04:35 2010
New Revision: 77982

Modified:
   pypy/branch/leak-finder/pypy/rpython/llinterp.py
Log:
Don't do any checking when running the GC-transformed graphs on llinterp.
Probably not worth it, and messy because the GC allocates some structures
like AddressStack.


Modified: pypy/branch/leak-finder/pypy/rpython/llinterp.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/llinterp.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/llinterp.py	Fri Oct 15 15:04:35 2010
@@ -1013,6 +1013,13 @@
     def op_stack_malloc(self, size): # mmh
         raise NotImplementedError("backend only")
 
+    def op_track_alloc_start(self, addr):
+        # we don't do tracking at this level
+        checkadr(addr)
+
+    def op_track_alloc_stop(self, addr):
+        checkadr(addr)
+
     # ____________________________________________________________
     # Overflow-detecting variants
 


From cfbolz at codespeak.net  Fri Oct 15 15:05:30 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 15 Oct 2010 15:05:30 +0200 (CEST)
Subject: [pypy-svn] r77983 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015130530.B537F36C22E@codespeak.net>

Author: cfbolz
Date: Fri Oct 15 15:05:29 2010
New Revision: 77983

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
add lifting example


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 15:05:29 2010
@@ -872,8 +872,21 @@
 twice, the \lstinline{liftfield} operation removes it from the static heap \emph{before}
 recursively lifting its fields.
 
+As an example for lifting, consider the static heap $$\{v^* \mapsto (T_1, w^*,
+v^*), w^* \mapsto (T_2, u^*, u^*)\}$$ which contains two static objects. If $v^*$
+now needs to be lifted, the following residual operations are produced:
+
+\begin{lstlisting}[mathescape,xleftmargin=20pt]
+$v^*$ = new($T_1$)
+$w^*$ = new($T_2$)
+set($w^*$, $L$, $u^*$)
+set($w^*$, $R$, $u^*$)
+set($v^*$, $L$, $w^*$)
+set($v^*$, $R$, $v^*$)
+\end{lstlisting}
 
-XXX Apply the rules of the algorithm to parts of the sample trace
+After the lifting the static heap is the empty set, because both static objects
+were lifted.
 
 % subsection Optimizing Traces (end)
 


From antocuni at codespeak.net  Fri Oct 15 15:12:13 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Fri, 15 Oct 2010 15:12:13 +0200 (CEST)
Subject: [pypy-svn] r77984 - in pypy/branch/jitffi/pypy/jit/metainterp:
	optimizeopt test
Message-ID: <20101015131213.E405A36E0C1@codespeak.net>

Author: antocuni
Date: Fri Oct 15 15:12:12 2010
New Revision: 77984

Modified:
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
Log:
nicely optimize the case in which we have virtualizables around, and thus we
get force_token between the call_may_force



Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	Fri Oct 15 15:12:12 2010
@@ -18,6 +18,7 @@
         argtypes, restype = self._get_signature(funcval)
         self.descr = cpu.calldescrof_dynamic(argtypes, restype)
         self.prepare_op = prepare_op
+        self.force_token_op = None
 
     def _get_signature(self, funcval):
         """
@@ -83,6 +84,8 @@
         self.emit_operation(funcinfo.prepare_op)
         for op in funcinfo.opargs:
             self.emit_operation(op)
+        if funcinfo.force_token_op:
+            self.emit_operation(funcinfo.force_token_op)
 
     def emit_operation(self, op):
         # we cannot emit any operation during the optimization
@@ -91,46 +94,80 @@
 
     def optimize_CALL(self, op):
         oopspec = self._get_oopspec(op)
+        ops = [op]
         if oopspec == EffectInfo.OS_LIBFFI_PREPARE:
-            op = self.do_prepare_call(op)
+            ops = self.do_prepare_call(op)
         elif oopspec == EffectInfo.OS_LIBFFI_PUSH_ARG:
-            op = self.do_push_arg(op)
+            ops = self.do_push_arg(op)
         elif oopspec == EffectInfo.OS_LIBFFI_CALL:
-            op = self.do_call(op)
+            ops = self.do_call(op)
         #
-        if op:
+        for op in ops:
             self.emit_operation(op)
 
     optimize_CALL_MAY_FORCE = optimize_CALL
 
+    def optimize_FORCE_TOKEN(self, op):
+        # The handling of force_token needs a bit of exaplanation.
+        # The original trace which is getting optimized looks like this:
+        #    i1 = force_token()
+        #    setfield_gc(p0, i1, ...)
+        #    call_may_force(...)
+        #
+        # In theory, fficall should take care of both force_token and
+        # setfield_gc.  However, the lazy setfield optimization in heap.py
+        # delays the setfield_gc, with the effect that fficall.py sees them in
+        # this order:
+        #    i1 = force_token()
+        #    call_may_force(...)
+        #    setfield_gc(p0, i1, ...)
+        #
+        # This means that see the setfield_gc only the call_may_force, when
+        # the optimization has already been done, and thus we need to take
+        # special care just of force_token.
+        #
+        # Finally, the method force_lazy_setfield in heap.py reorders the
+        # call_may_force and the setfield_gc, so the final result we get is
+        # again force_token/setfield_gc/call_may_force.
+        #
+        if self.funcinfo is None:
+            self.emit_operation(op)
+        else:
+            self.funcinfo.force_token_op = op
+
     def do_prepare_call(self, op):
         self.rollback_maybe()
         funcval = self._get_funcval(op)
         if not funcval.is_constant():
-            return op # cannot optimize
+            return [op] # cannot optimize
         self.begin_optimization(funcval, op)
-        return None
+        return []
 
     def do_push_arg(self, op):
         funcval = self._get_funcval(op)
         if not self.funcinfo or self.funcinfo.funcval is not funcval:
-            return op # cannot optimize
+            return [op] # cannot optimize
         self.funcinfo.opargs.append(op)
-        return None
+        return []
 
     def do_call(self, op):
         funcval = self._get_funcval(op)
-        if not self.funcinfo or self.funcinfo.funcval is not funcval:
-            return op # cannot optimize
+        funcinfo = self.funcinfo
+        if not funcinfo or funcinfo.funcval is not funcval:
+            return [op] # cannot optimize
         funcsymval = self.getvalue(op.getarg(2))
         arglist = [funcsymval.force_box()]
-        for push_op in self.funcinfo.opargs:
+        for push_op in funcinfo.opargs:
             argval = self.getvalue(push_op.getarg(2))
             arglist.append(argval.force_box())
         newop = ResOperation(rop.CALL_MAY_FORCE, arglist, op.result,
-                             descr=self.funcinfo.descr)
+                             descr=funcinfo.descr)
         self.commit_optimization()
-        return newop
+        ops = []
+        if funcinfo.force_token_op:
+            ops.append(funcinfo.force_token_op)
+        ops.append(newop)
+        return ops
 
     def propagate_forward(self, op):
         opnum = op.getopnum()

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py	Fri Oct 15 15:12:12 2010
@@ -35,6 +35,7 @@
     class namespace:
         cpu = LLtypeMixin.cpu
         FUNC = LLtypeMixin.FUNC
+        vable_token_descr = LLtypeMixin.valuedescr
         int_float__int = MyCallDescr('if', 'i')
         funcptr = FakeLLObject()
         func = FakeLLObject(_fake_class=Func,
@@ -92,6 +93,32 @@
         expected = ops
         loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
 
+    def test_handle_virtualizables(self):
+        # this test needs an explanation to understand what goes on: see the
+        # coment in optimize_FORCE_TOKEN
+        ops = """
+        [i0, f1, p2]
+        call(0, ConstPtr(func),                       descr=libffi_prepare)
+        call(0, ConstPtr(func), i0,                   descr=libffi_push_arg)
+        call(0, ConstPtr(func), f1,                   descr=libffi_push_arg)
+        i4 = force_token()
+        setfield_gc(p2, i4, descr=vable_token_descr)
+        i3 = call_may_force(0, ConstPtr(func), 12345, descr=libffi_call)
+        guard_not_forced() [p2]
+        guard_no_exception() [p2]
+        jump(i3, f1, p2)
+        """
+        expected = """
+        [i0, f1, p2]
+        i4 = force_token()
+        setfield_gc(p2, i4, descr=vable_token_descr)
+        i3 = call_may_force(12345, i0, f1, descr=int_float__int)
+        guard_not_forced() [p2]
+        guard_no_exception() [p2]
+        jump(i3, f1, p2)
+        """
+        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+
     # ----------------------------------------------------------------------
     # in pratice, the situations described in these tests should never happen,
     # but we still want to ensure correctness
@@ -195,3 +222,19 @@
         """
         loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
 
+    def test_rollback_force_token(self):
+        ops = """
+        [i0, f1, p2]
+        call(0, ConstPtr(func),                       descr=libffi_prepare)
+        call(0, ConstPtr(func), i0,                   descr=libffi_push_arg)
+        call(0, ConstPtr(func), f1,                   descr=libffi_push_arg)
+        i4 = force_token()
+        i5 = int_add(i0, 1) # culprit!
+        setfield_gc(p2, i4, descr=vable_token_descr)
+        i3 = call_may_force(0, ConstPtr(func), 12345, descr=libffi_call)
+        guard_not_forced() [p2]
+        guard_no_exception() [p2]
+        jump(i3, f1, p2)
+        """
+        expected = ops
+        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)


From cfbolz at codespeak.net  Fri Oct 15 15:13:04 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 15 Oct 2010 15:13:04 +0200 (CEST)
Subject: [pypy-svn] r77985 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015131304.D1C2336E0C1@codespeak.net>

Author: cfbolz
Date: Fri Oct 15 15:13:03 2010
New Revision: 77985

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
enhance example


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 15:13:03 2010
@@ -888,6 +888,19 @@
 After the lifting the static heap is the empty set, because both static objects
 were lifted.
 
+If from the same heap $w^*$ is lifted instead, the following operations are
+produced:
+
+\begin{lstlisting}[mathescape,xleftmargin=20pt]
+$w^*$ = new($T_2$)
+set($w^*$, $L$, $u^*$)
+set($w^*$, $R$, $u^*$)
+\end{lstlisting}
+
+The static heap afterwards is $\{v^* \mapsto (T_1, w^*, v^*)\}$.
+
+
+
 % subsection Optimizing Traces (end)
 
 \subsection{Analysis of the Algorithm}


From arigo at codespeak.net  Fri Oct 15 15:13:45 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 15:13:45 +0200 (CEST)
Subject: [pypy-svn] r77986 - pypy/branch/32ptr-on-64bit/pypy/translator/c
Message-ID: <20101015131345.9A52536E0C3@codespeak.net>

Author: arigo
Date: Fri Oct 15 15:13:44 2010
New Revision: 77986

Modified:
   pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py
Log:
Change the value of DEBUGFLAGS in non-asmgcc builds to
remove -O1.  This is fine with the extra stack that we
get by the recent change to c/src/stack.h.


Modified: pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py
==============================================================================
--- pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py	(original)
+++ pypy/branch/32ptr-on-64bit/pypy/translator/c/genc.py	Fri Oct 15 15:13:44 2010
@@ -628,7 +628,7 @@
                         python + '$(PYPYDIR)/translator/c/gcc/trackgcroot.py $(GCMAPFILES) > $@')
 
         else:
-            mk.definition('DEBUGFLAGS', '-O1 -g')
+            mk.definition('DEBUGFLAGS', '-g')
         mk.write()
         #self.translator.platform,
         #                           ,


From arigo at codespeak.net  Fri Oct 15 15:16:31 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 15:16:31 +0200 (CEST)
Subject: [pypy-svn] r77987 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015131631.67D6D36E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 15:16:30 2010
New Revision: 77987

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
Rewording.


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 15:16:30 2010
@@ -887,9 +887,8 @@
 
 After the lifting the static heap is the empty set, because both static objects
 were lifted.
-
-If from the same heap $w^*$ is lifted instead, the following operations are
-produced:
+If we had lifted $w^*$ instead of $v^*$, then the following operations would
+have been produced:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 $w^*$ = new($T_2$)
@@ -897,7 +896,8 @@
 set($w^*$, $R$, $u^*$)
 \end{lstlisting}
 
-The static heap afterwards is $\{v^* \mapsto (T_1, w^*, v^*)\}$.
+In this case, the static heap afterwards would be
+$\{v^* \mapsto (T_1, w^*, v^*)\}$.
 
 
 


From arigo at codespeak.net  Fri Oct 15 15:50:14 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 15:50:14 +0200 (CEST)
Subject: [pypy-svn] r77989 - pypy/branch/leak-finder/pypy/translator/c/src
Message-ID: <20101015135014.D2DAF36E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 15:50:13 2010
New Revision: 77989

Added:
   pypy/branch/leak-finder/pypy/translator/c/src/debug_alloc.h
Modified:
   pypy/branch/leak-finder/pypy/translator/c/src/g_include.h
   pypy/branch/leak-finder/pypy/translator/c/src/main.h
Log:
In RPY_ASSERT mode, print a warning at the end of running a translated
program if there are malloc()s without a corresponding free().  If you
get more than a few ones, you have a leak.


Added: pypy/branch/leak-finder/pypy/translator/c/src/debug_alloc.h
==============================================================================
--- (empty file)
+++ pypy/branch/leak-finder/pypy/translator/c/src/debug_alloc.h	Fri Oct 15 15:50:13 2010
@@ -0,0 +1,81 @@
+/**************************************************************/
+ /***  tracking raw mallocs and frees for debugging          ***/
+
+#ifndef RPY_ASSERT
+
+#  define OP_TRACK_ALLOC_START(addr, r)   /* nothing */
+#  define OP_TRACK_ALLOC_STOP(addr, r)    /* nothing */
+
+#else   /* ifdef RPY_ASSERT */
+
+#  define OP_TRACK_ALLOC_START(addr, r)  pypy_debug_alloc_start(addr, \
+                                                                __FUNCTION__)
+#  define OP_TRACK_ALLOC_STOP(addr, r)   pypy_debug_alloc_stop(addr)
+
+void pypy_debug_alloc_start(void*, const char*);
+void pypy_debug_alloc_stop(void*);
+void pypy_debug_alloc_results(void);
+
+/************************************************************/
+
+
+#ifndef PYPY_NOT_MAIN_FILE
+
+struct pypy_debug_alloc_s {
+  struct pypy_debug_alloc_s *next;
+  void *addr;
+  const char *funcname;
+};
+
+static struct pypy_debug_alloc_s *pypy_debug_alloc_list = NULL;
+
+void pypy_debug_alloc_start(void *addr, const char *funcname)
+{
+  struct pypy_debug_alloc_s *p = malloc(sizeof(struct pypy_debug_alloc_s));
+  RPyAssert(p, "out of memory");
+  p->next = pypy_debug_alloc_list;
+  p->addr = addr;
+  p->funcname = funcname;
+  pypy_debug_alloc_list = p;
+}
+
+void pypy_debug_alloc_stop(void *addr)
+{
+  struct pypy_debug_alloc_s **p;
+  for (p = &pypy_debug_alloc_list; *p; p = &((*p)->next))
+    if ((*p)->addr == addr)
+      {
+        struct pypy_debug_alloc_s *dying;
+        dying = *p;
+        *p = dying->next;
+        free(dying);
+        return;
+      }
+  RPyAssert(0, "free() of a never-malloc()ed object");
+}
+
+void pypy_debug_alloc_results(void)
+{
+  long count = 0;
+  struct pypy_debug_alloc_s *p;
+  for (p = pypy_debug_alloc_list; p; p = p->next)
+    count++;
+  if (count > 0)
+    {
+      fprintf(stderr, "debug_alloc.h: %ld mallocs left", count);
+      char *env = getenv("PYPY_ALLOC");
+      if (env && *env)
+        {
+          fprintf(stderr, " (most recent first):\n");
+          for (p = pypy_debug_alloc_list; p; p = p->next)
+            fprintf(stderr, "    %p  %s\n", p->addr, p->funcname);
+        }
+      else
+        fprintf(stderr, " (use PYPY_ALLOC=1 to see the list)\n");
+    }
+}
+
+#endif
+
+
+#endif  /* RPY_ASSERT */

Modified: pypy/branch/leak-finder/pypy/translator/c/src/g_include.h
==============================================================================
--- pypy/branch/leak-finder/pypy/translator/c/src/g_include.h	(original)
+++ pypy/branch/leak-finder/pypy/translator/c/src/g_include.h	Fri Oct 15 15:50:13 2010
@@ -53,6 +53,7 @@
 #  include "src/rtyper.h"
 #  include "src/debug_print.h"
 #  include "src/debug_traceback.h"
+#  include "src/debug_alloc.h"
 #ifndef AVR
 #  include "src/ll_os.h"
 #  include "src/ll_strtod.h"

Modified: pypy/branch/leak-finder/pypy/translator/c/src/main.h
==============================================================================
--- pypy/branch/leak-finder/pypy/translator/c/src/main.h	(original)
+++ pypy/branch/leak-finder/pypy/translator/c/src/main.h	Fri Oct 15 15:50:13 2010
@@ -53,10 +53,16 @@
     }
 
     exitcode = STANDALONE_ENTRY_POINT(list);
+
+#ifdef RPY_ASSERT
+    pypy_debug_alloc_results();
+#endif
+
     if (RPyExceptionOccurred()) {
         /* print the RPython traceback */
         pypy_debug_catch_fatal_exception();
     }
+
     return exitcode;
 
  memory_out:


From antocuni at codespeak.net  Fri Oct 15 16:12:28 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Fri, 15 Oct 2010 16:12:28 +0200 (CEST)
Subject: [pypy-svn] r77990 - pypy/branch/jitffi/pypy/module/pypyjit/test
Message-ID: <20101015141228.090B736E0C1@codespeak.net>

Author: antocuni
Date: Fri Oct 15 16:12:27 2010
New Revision: 77990

Modified:
   pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py
Log:
update the number of expected operations: we now have a force_token, setfield_gc and guard_not_forced more. Since we get two loops, it's 6 more ops in total


Modified: pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py	Fri Oct 15 16:12:27 2010
@@ -1151,7 +1151,7 @@
                 i += 1
             return res
         ''' % locals(),
-                              70, ([], 8.0*2000), threshold=1000)
+                              76, ([], 8.0*2000), threshold=1000)
         pow_addr = int(out.splitlines()[0])
         ops = self.get_by_bytecode('CALL_FUNCTION')
         assert len(ops) == 2 # we get two loops, because of specialization


From antocuni at codespeak.net  Fri Oct 15 16:18:26 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Fri, 15 Oct 2010 16:18:26 +0200 (CEST)
Subject: [pypy-svn] r77991 - pypy/branch/jitffi/pypy/module/pypyjit/test
Message-ID: <20101015141826.5223836E0C1@codespeak.net>

Author: antocuni
Date: Fri Oct 15 16:18:24 2010
New Revision: 77991

Modified:
   pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py
Log:
fix the test, and check that we get the expected operations


Modified: pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py	Fri Oct 15 16:18:24 2010
@@ -1155,11 +1155,14 @@
         pow_addr = int(out.splitlines()[0])
         ops = self.get_by_bytecode('CALL_FUNCTION')
         assert len(ops) == 2 # we get two loops, because of specialization
-        op = ops[0]
-        # XXX: there should be a guard_not_forced
-        assert op[-1].getopname() == 'guard_no_exception'
-        call = op[-2]
-        assert call.getopname() == 'call'
+        call_function = ops[0]
+        last_ops = [op.getopname() for op in call_function[-5:]]
+        assert last_ops == ['force_token',
+                            'setfield_gc',
+                            'call_may_force',
+                            'guard_not_forced',
+                            'guard_no_exception']
+        call = call_function[-3]
         assert call.getarg(0).value == pow_addr
         assert call.getarg(1).value == 2.0
         assert call.getarg(2).value == 3.0


From arigo at codespeak.net  Fri Oct 15 16:25:29 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 16:25:29 +0200 (CEST)
Subject: [pypy-svn] r77992 - pypy/branch/leak-finder/pypy/rpython/memory/gc
Message-ID: <20101015142529.82D6636E076@codespeak.net>

Author: arigo
Date: Fri Oct 15 16:25:27 2010
New Revision: 77992

Modified:
   pypy/branch/leak-finder/pypy/rpython/memory/gc/minimarkpage.py
Log:
Add track_allocation=False to the arenas; reporting them at process shutdown is unhelpful.

Modified: pypy/branch/leak-finder/pypy/rpython/memory/gc/minimarkpage.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/memory/gc/minimarkpage.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/memory/gc/minimarkpage.py	Fri Oct 15 16:25:27 2010
@@ -100,11 +100,14 @@
         # allocation of the given size.
         length = small_request_threshold / WORD + 1
         self.page_for_size = lltype.malloc(rffi.CArray(PAGE_PTR), length,
-                                           flavor='raw', zero=True)
+                                           flavor='raw', zero=True,
+                                           immortal=True)
         self.full_page_for_size = lltype.malloc(rffi.CArray(PAGE_PTR), length,
-                                                flavor='raw', zero=True)
+                                                flavor='raw', zero=True,
+                                                immortal=True)
         self.nblocks_for_size = lltype.malloc(rffi.CArray(lltype.Signed),
-                                              length, flavor='raw')
+                                              length, flavor='raw',
+                                              immortal=True)
         self.hdrsize = llmemory.raw_malloc_usage(llmemory.sizeof(PAGE_HEADER))
         assert page_size > self.hdrsize
         self.nblocks_for_size[0] = 0    # unused
@@ -114,11 +117,13 @@
         self.max_pages_per_arena = arena_size // page_size
         self.arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR),
                                           self.max_pages_per_arena,
-                                          flavor='raw', zero=True)
+                                          flavor='raw', zero=True,
+                                          immortal=True)
         # this is used in mass_free() only
         self.old_arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR),
                                               self.max_pages_per_arena,
-                                              flavor='raw', zero=True)
+                                              flavor='raw', zero=True,
+                                              immortal=True)
         #
         # the arena currently consumed; it must have at least one page
         # available, or be NULL.  The arena object that we point to is
@@ -281,7 +286,7 @@
         npages = (arena_end - firstpage) // self.page_size
         #
         # Allocate an ARENA object and initialize it
-        arena = lltype.malloc(ARENA, flavor='raw')
+        arena = lltype.malloc(ARENA, flavor='raw', track_allocation=False)
         arena.base = arena_base
         arena.nfreepages = 0        # they are all uninitialized pages
         arena.totalpages = npages
@@ -332,7 +337,7 @@
                     #
                     # The whole arena is empty.  Free it.
                     llarena.arena_free(arena.base)
-                    lltype.free(arena, flavor='raw')
+                    lltype.free(arena, flavor='raw', track_allocation=False)
                     #
                 else:
                     # Insert 'arena' in the correct arenas_lists[n]


From cfbolz at codespeak.net  Fri Oct 15 16:47:12 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 15 Oct 2010 16:47:12 +0200 (CEST)
Subject: [pypy-svn] r77993 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015144712.4A62636E0C1@codespeak.net>

Author: cfbolz
Date: Fri Oct 15 16:47:10 2010
New Revision: 77993

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
incorporate first round of comments by stephan


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 16:47:10 2010
@@ -106,7 +106,7 @@
 The performance of many dynamic language implementations suffers from
 high allocation rates and runtime type checks.  This makes dynamic
 languages less applicable to purely algorithmic problems, despite their
-growing popularity.  In this paper, we present a simple optimization
+growing popularity.  In this paper we present a simple compiler optimization
 based on online partial evaluation to remove object allocations and
 runtime type checks in the context of a tracing JIT.  We evaluate the
 optimization using a Python VM and find that it gives good results for
@@ -130,7 +130,7 @@
 
 \section{Introduction}
 
-The goal of a just-in-time (JIT) compiler for a dynamic language is obviously to
+The objective of a just-in-time (JIT) compiler for a dynamic language is to
 improve the speed of the language over an implementation of the language that
 uses interpretation. The first goal of a JIT is therefore to remove the
 interpretation overhead, i.e. the overhead of bytecode (or AST) dispatch and the
@@ -142,38 +142,37 @@
 
 Boxing of primitive types is necessary because dynamic languages need to be able to handle
 all objects, even integers, floats, booleans etc. in the same way as user-defined
-instances. Thus those primitive types are usually \emph{boxed}, i.e. a small
-heap-structure is allocated for them, that contains the actual value. Boxing
+instances. Thus those primitive types are usually \emph{boxed}, \ie a small
+heap-structure is allocated for them that contains the actual value. Boxing
 primitive types can be very costly, because a lot of common operations,
-particularly all arithmetic operations, have to produce a new box, in addition
+particularly all arithmetic operations, have to produce new boxes, in addition
 to the actual computation they do. Because the boxes are allocated on the heap,
-producing a lot of them puts pressure on the garbage collector.
+producing many of them puts pressure on the garbage collector.
 
 Type dispatching is the process of finding the concrete implementation that is
-applicable to the objects at hand when doing a generic operation on them. An
-example would be the addition of two objects: The addition needs to check what
-the concrete objects that should be added are, and choose the implementation
-that is fitting for them. Type dispatching is a very common operation in
+applicable to the objects at hand when performing a generic operation on them. An
+example would be the addition of two objects: For addition the types of the
+concrete objects need to be checked and the suiting implementation chosen.
+Type dispatching is a very common operation in
 modern\footnote{For languages in the LISP family, basic arithmetic operations
 are typically not overloaded; even in Smalltalk, type dispatching is much
 simpler than in Python or JavaScript.}
-dynamic languages because no types are known at compile time, so all operations
-need it.
+dynamic languages because no types are known at compile time. Therefore all
+operations need it.
 
 A recently popular approach to implementing just-in-time compilers for dynamic
 languages is that of a tracing JIT. A tracing JIT works by observing the running
-program and recording its hot spots into linear execution traces. Working on
-traces is the central idea of a tracing JIT. Those traces are optimized and
-turned into machine code.
+program and recording its hot spots into \emph{linear execution traces}. Those
+traces are optimized and turned into machine code.
 
 One reason for the popularity of tracing JITs is their relative
-simplicity. They can often be added to an interpreter and a lot of the
-infrastructure of the interpreter can be reused. They give some important
+simplicity. They can often be added to an existing interpreter, reusing a lot of
+the interpreter's infrastructure. They give some important
 optimizations like inlining and constant-folding for free. A tracing JIT always
-produces linear pieces of code, which simplifies many algorithms that are usually
-hard in a compiler, such as register allocation.
+produces linear pieces of code, which simplifies many of the hard algorithms in
+a compiler, such as register allocation.
 
-The usage of a tracing JIT can remove the overhead of bytecode dispatch and that
+The use of a tracing JIT can remove the overhead of bytecode dispatch and that
 of the interpreter data structures. In this paper we want to present a new
 optimization that can be added to a tracing JIT that further removes some of the
 overhead more closely associated to dynamic languages, such as boxing overhead
@@ -190,14 +189,15 @@
 informally described in Section~\ref{sec:statics}; a more formal description is
 given in Section~\ref{sec:formal}. The introduced
 techniques are evaluated in Section~\ref{sec:Evaluation} using PyPy's Python
-interpreter as a case study.
+interpreter.
 
-The contributions of this paper are:
+The contributions made by this paper are:
 
 \begin{enumerate}
-    \item An efficient and effective algorithm for removing object allocations in a tracing JIT.
+    \item A description of an efficient and effective algorithm for removing
+          object allocations in a tracing JIT.
     \item A characterization of this algorithm as partial evaluation.
-    \item A rigorous evaluation of this algorithm.
+    \item Performance benchmarks for this algorithm.
 \end{enumerate}
 
 
@@ -215,7 +215,7 @@
 \emph{RPython} \cite{davide_ancona_rpython:_2007}. RPython ("restricted Python")
 is a subset of Python chosen in such a way that type inference becomes
 possible. The language interpreter can then be compiled (``translated'') with
-PyPy's tools into a VM on the C level. During translation to C, many low-level
+PyPy's tools into a VM on C level. During translation to C, many low-level
 aspects of the final VM, such as object layout, garbage collection and memory
 model, are woven into the generated code. Therefore the interpreter itself can
 remain at a relatively high level of abstraction.
@@ -234,13 +234,13 @@
 language that the interpreter is implementing. This process is mostly
 automatic; it only needs to be guided by the language implementer using a small number of
 source-code hints. Mostly-automatically generating a JIT compiler has many advantages
-over writing one manually, which is an error-prone and tedious process.
+over writing one manually, an error-prone and tedious process.
 By construction, the generated JIT has the same semantics as the interpreter.
-Many optimizations can benefit all languages implemented as an interpreter in RPython.
+Optimizations can be shared between different languages implemented with PyPy.
 
 Moreover, thanks to the internal design of the JIT generator, it is very easy
 to add new \emph{backends} for producing the actual machine code.  Examples of
-JIT backends that are implemented are the one for Intel x86 and x86-64 and an
+JIT backends that are implemented are those for Intel x86 and x86-64 and an
 experimental one for the CLI .NET Virtual Machine \cite{cuni_high_2010}.
 
 \subsection{Tracing JIT Compilers}
@@ -256,7 +256,7 @@
 and now Python (and other languages) via PyPy.
 
 The core idea of tracing JITs is to focus the optimization effort of the JIT
-compiler on the hot paths of the core loops of the program and to just use an
+compiler on the commonly executed, \ie \emph{hot} paths of the core loops of the program and to just use an
 interpreter for the less commonly executed parts. VMs that use a tracing JIT are
 mostly mixed-mode execution environments, they contain both an interpreter and a
 JIT compiler. By default the interpreter is used to execute the program, doing
@@ -269,24 +269,23 @@
 it always ends with a jump to its own beginning. The trace also contains all
 operations that are performed in functions that were called in the loop, thus a
 tracing JIT automatically performs inlining.
-
-This trace of operations is then the basis of the generated code. The trace is
+This trace of operations subsequently forms the basis of the generated code. The trace is
 first optimized, and then turned into machine code. Both optimization
 and machine code generation are simple, because the traces are linear. This
 linearity makes many optimizations a lot more tractable, and the inlining that
 happens gives the optimizations automatically more context to work with.
 
 Since the trace corresponds to one concrete execution of a loop,
-the code generated from it is only one possible path through it.
-To make sure that the trace is maintaining the correct semantics, it contains a
+the code generated from it is only one possible path through the loop.
+To make sure that the trace maintains the correct semantics, it contains a
 \emph{guard} at all places where the execution could have diverged from the
 path. Those guards check the assumptions under which execution can stay on the
-trace. As an example, if a loop contains an \lstinline{if} statement, the trace
+trace. As an example, if a loop contains an if-statement, the trace
 will contain the execution of one of the paths only, which is the path that was
 taken during the production of the trace. The trace will also contain a guard
-that checks that the condition of the \lstinline{if} statement is the same as
+that checks that the condition of the if-statement is the same as
 during tracing, because if
-it isn't, the rest of the trace is not valid. \cfbolz{The "if" shouldn't be bold}
+it isn't, the rest of the trace would not be valid.
 
 When generating machine code, every guard is be turned into a quick check to
 see whether the assumption still holds. When such a guard is hit during the
@@ -367,11 +366,11 @@
 \label{fig:objmodel}
 \end{figure}
 
-Using these classes to implement arithmetic shows the basic problem that a
-dynamic language implementation has. All the numbers are instances of either
+Using these classes to implement arithmetic shows the basic problem of a
+dynamic language implementation. All the numbers are instances of either
 \lstinline{BoxedInteger} or \lstinline{BoxedFloat}, therefore they consume space on the
 heap. Performing many arithmetic operations produces lots of garbage quickly,
-which puts pressure on the garbage collector. Using double dispatching to
+putthing pressure on the garbage collector. Using double dispatching to
 implement the numeric tower needs two method calls per arithmetic operation,
 which is costly due to the method dispatch.
 
@@ -384,7 +383,7 @@
 calls inside the loop, one for each \lstinline{is_positive} and even two for each
 call to \lstinline{add}. These method calls need to check the type of the involved
 objects repeatedly and redundantly. In addition, a lot of objects are created
-when executing that loop, many of these objects do not survive for very long.
+when executing that loop, many of these objects are short-lived.
 The actual computation that is performed by \lstinline{f} is simply a sequence of
 float or integer additions.
 
@@ -589,7 +588,7 @@
 the type check the guard does is statically known.
 
 In the example from last section, the following operations in the upper half
-of Fig.~\ref{fig:unopt-trace} produce two
+of Figure~\ref{fig:unopt-trace} produce two
 static objects, and can be completely removed from the optimized trace:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
@@ -605,7 +604,7 @@
 one associated with $p_{6}$ would know that it is a \lstinline{BoxedInteger}
 whose \lstinline{intval} field contains the constant -100.
 
-The subsequent operations in Fig.~\ref{fig:unopt-trace},
+The subsequent operations in Figure~\ref{fig:unopt-trace},
  which use $p_{5}$ and $p_{6}$, could then be
 optimized using that knowledge:
 
@@ -628,7 +627,7 @@
 $i_{9}$ = int_add($i_{4}$, -100)
 \end{lstlisting}
 
-The rest of the trace from Fig.~\ref{fig:unopt-trace} is optimized similarly.
+The rest of the trace from Figure~\ref{fig:unopt-trace} is optimized similarly.
 
 So far we have only described what happens when static objects are used in guards and in
 operations that read and write fields. When the static
@@ -640,7 +639,7 @@
 necessary to put operations into the residual code that allocate the
 static object at runtime.
 
-This is what happens at the end of the trace in Fig.~\ref{fig:unopt-trace}, when the \lstinline{jump} operation
+This is what happens at the end of the trace in Figure~\ref{fig:unopt-trace}, when the \lstinline{jump} operation
 is hit. The arguments of the jump are at this point static objects. Before the
 jump is emitted, they are \emph{lifted}. This means that the optimizer produces code
 that allocates a new object of the right type and sets its fields to the field
@@ -897,7 +896,7 @@
 \end{lstlisting}
 
 In this case, the static heap afterwards would be
-$\{v^* \mapsto (T_1, w^*, v^*)\}$.
+$$\{v^* \mapsto (T_1, w^*, v^*)\}$$.
 
 
 


From cfbolz at codespeak.net  Fri Oct 15 17:17:15 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 15 Oct 2010 17:17:15 +0200 (CEST)
Subject: [pypy-svn] r77994 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015151715.6408E36E0C1@codespeak.net>

Author: cfbolz
Date: Fri Oct 15 17:17:13 2010
New Revision: 77994

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
more suggestions by stephan


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 17:17:13 2010
@@ -374,10 +374,8 @@
 implement the numeric tower needs two method calls per arithmetic operation,
 which is costly due to the method dispatch.
 
-To understand the problems more directly, let us consider the simple
-interpreter function \lstinline{f} that uses the object model (see the bottom of
-Figure~\ref{fig:objmodel}).
-
+Let us now consider a simple interpreter function \lstinline{f} that uses the
+object model (see the bottom of Figure~\ref{fig:objmodel}).
 The loop in \lstinline{f} iterates \lstinline{y} times, and computes something in the process.
 Simply running this function is slow, because there are lots of virtual method
 calls inside the loop, one for each \lstinline{is_positive} and even two for each
@@ -451,24 +449,25 @@
 \end{figure}
 
 If the function is executed using the tracing JIT, with \lstinline{y} being a
-\lstinline{BoxedInteger}, the produced trace looks like
-Figure~\ref{fig:unopt-trace} (lines starting with the hash ``\#'' are comments).
+\lstinline{BoxedInteger}, the produced trace looks like the one of
+Figure~\ref{fig:unopt-trace} (lines starting with a hash ``\#'' are comments).
+The trace corresponds to one iteration of the while-loop in \lstinline{f}.
 
-The operations in the trace are shown indented to
-correspond to the stack level of the function that contains the traced
+The operations in the trace are indented
+corresponding to the stack level of the function that contains the traced
 operation. The trace is in single-assignment form, meaning that each variable is
-assigned to exactly once. The arguments $p_0$ and $p_1$ of the loop correspond
+assigned a value exactly once. The arguments $p_0$ and $p_1$ of the loop correspond
 to the live variables \lstinline{y} and \lstinline{res} in the original function.
 
 The operations in the trace correspond to the operations in the RPython program
 in Figure~\ref{fig:objmodel}:
 
 \begin{itemize}
-    \item \lstinline{new} corresponds to object creation.
-    \item \lstinline{get} correspond to attribute reads.
-    \item \lstinline{set} correspond to attribute writes.
-    \item \lstinline{guard_class} correspond to method calls and are followed by
-    the trace of the called method.
+    \item \lstinline{new} creates a new object.
+    \item \lstinline{get} reads an attribute of an object.
+    \item \lstinline{set} writes to an attribute of an object.
+    \item \lstinline{guard_class} precedes an (inlined) method call and is
+    followed by the trace of the called method.
     \item \lstinline{int_add} and \lstinline{int_gt} are integer addition and
     comparison (``greater than''), respectively.
 \end{itemize}
@@ -477,9 +476,9 @@
 operation, to check that the class of the receiver is the same as the one that
 was observed during tracing.\footnote{\lstinline{guard_class} performs a precise
 class check, not checking for subclasses.} These guards make the trace specific
-to the situation where \lstinline{y} is really a \lstinline{BoxedInteger}, it can
-already be said to be specialized for \lstinline{BoxedIntegers}. When the trace is
-turned into machine code and then executed with \lstinline{BoxedFloats}, the
+to the situation where \lstinline{y} is really a \lstinline{BoxedInteger}. When
+the trace is turned into machine code and afterwards executed with
+\lstinline{BoxedFloat}, the
 first \lstinline{guard_class} instruction will fail and execution will continue
 using the interpreter.
 
@@ -488,8 +487,8 @@
 operations. The number of \lstinline{guard_class} operation is particularly
 problematic, not only because of the time it takes to run them. All guards also
 have additional information attached that makes it possible to return to the
-interpreter, should the guard fail. This means that many guard operations also
-lead to a memory problem.
+interpreter, should the guard fail. This means that too many guard operations also
+consume a lot of memory.
 
 In the rest of the paper we will see how this trace can be optimized using
 partial evaluation.
@@ -499,7 +498,7 @@
 
 % section Object Lifetimes in a Tracing JIT (end)
 
-To understand the problems that this paper is trying to solve some more, we
+To understand the problems that this paper is trying to solve in more detail, we
 first need to understand various cases of object lifetimes that can occur in a
 tracing JIT compiler.
 
@@ -520,7 +519,7 @@
 aborted and interpretation resumes.
 
 Some of the operations within this trace are \lstinline{new} operations, which each
-create a new instance of some class. These instances are used for a while, e.g.
+create a new instance of some class. These instances are used for some time, \eg
 by calling methods on them (which are inlined into the trace), reading and
 writing their fields. Some of these instances \emph{escape}, which means that
 they are stored in some globally accessible place or are passed into a
@@ -530,19 +529,19 @@
 created objects. The objects that are created within a trace using \lstinline{new}
 fall into one of several categories:
 
-\begin{itemize}
-    \item Category 1: Objects that live for a while, and are then just not
-    used any more.
+\begin{enumerate}
+    \item Objects that live for some time, and are then just not
+    used any more afterwards.
 
-    \item Category 2: Objects that live for a while and then escape.
+    \item Objects that live for some time and then escape.
 
-    \item Category 3: Objects that live for a while, survive across the jump to
+    \item Objects that live for some time, survive across the jump to
     the beginning of the loop, and are then not used any more.
 
-    \item Category 4: Objects that live for a while, survive across the jump,
+    \item Objects that live for some time, survive across the jump,
     and then escape. To these we also count the objects that live across several
     jumps and then either escape or stop being used.
-\end{itemize}
+\end{enumerate}
 
 The objects that are allocated in the example trace in
 Figure~\ref{fig:unopt-trace} fall into categories 1 and 3. Objects stored in
@@ -551,7 +550,9 @@
 
 The creation of objects in category 1 is removed by the optimization described
 in Sections~\ref{sec:statics} and \ref{sec:formal}. Objects in the other
-categories are partially optimized by this approach as well.
+categories are partially optimized by this approach as well.\footnote{We also started to
+work on optimizing objects in category 3, which will be the subject of a later
+paper.}
 
 \section{Allocation Removal in Traces}
 \label{sec:statics}
@@ -895,8 +896,8 @@
 set($w^*$, $R$, $u^*$)
 \end{lstlisting}
 
-In this case, the static heap afterwards would be
-$$\{v^* \mapsto (T_1, w^*, v^*)\}$$.
+In this case, the static heap afterwards would be:
+$$\{v^* \mapsto (T_1, w^*, v^*)\}$$
 
 
 


From arigo at codespeak.net  Fri Oct 15 17:19:36 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 17:19:36 +0200 (CEST)
Subject: [pypy-svn] r77995 - pypy/branch/leak-finder/pypy/jit/codewriter
Message-ID: <20101015151936.38FBC36E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 17:19:34 2010
New Revision: 77995

Modified:
   pypy/branch/leak-finder/pypy/jit/codewriter/jtransform.py
Log:
Fix.


Modified: pypy/branch/leak-finder/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/codewriter/jtransform.py	Fri Oct 15 17:19:34 2010
@@ -429,7 +429,8 @@
                                   op.result)
 
     def rewrite_op_free(self, op):
-        assert op.args[1].value == 'raw'
+        flags = op.args[1].value
+        assert flags['flavor'] == 'raw'
         ARRAY = op.args[0].concretetype.TO
         return self._do_builtin_call(op, 'raw_free', [op.args[0]],
                                      extra = (ARRAY,), extrakey = ARRAY)


From arigo at codespeak.net  Fri Oct 15 17:21:24 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 17:21:24 +0200 (CEST)
Subject: [pypy-svn] r77996 - pypy/branch/leak-finder/pypy/module/_socket/test
Message-ID: <20101015152124.56B8F36E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 17:21:22 2010
New Revision: 77996

Modified:
   pypy/branch/leak-finder/pypy/module/_socket/test/test_sock_app.py
Log:
Fix the test.


Modified: pypy/branch/leak-finder/pypy/module/_socket/test/test_sock_app.py
==============================================================================
--- pypy/branch/leak-finder/pypy/module/_socket/test/test_sock_app.py	(original)
+++ pypy/branch/leak-finder/pypy/module/_socket/test/test_sock_app.py	Fri Oct 15 17:21:22 2010
@@ -254,6 +254,7 @@
     s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
     fd = s.fileno()
     w_obj = rsocket.make_address(c_addr, addrlen).as_object(fd, space)
+    lltype.free(c_addr_ll, flavor='raw')
     assert space.is_true(space.eq(w_obj, space.newtuple([
         space.wrap('lo'),
         space.wrap(socket.ntohs(8)),


From cfbolz at codespeak.net  Fri Oct 15 17:22:31 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 15 Oct 2010 17:22:31 +0200 (CEST)
Subject: [pypy-svn] r77997 - pypy/extradoc/talk/pepm2011
Message-ID: <20101015152231.3FCC536E0C1@codespeak.net>

Author: cfbolz
Date: Fri Oct 15 17:22:29 2010
New Revision: 77997

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
make the paper sort of presentable


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 15 17:22:29 2010
@@ -1113,7 +1113,6 @@
 garbage-collection pressure but does not optimize away the actual accesses to
 the stack-allocated object. In our case, an object is not needed at all any
 more.
-XXX more papers?
 
 Chang \etal describe a tracing JIT for JavaScript running on top of a JVM
 \cite{mason_chang_efficient_2007}. They mention in passing an approach to
@@ -1142,8 +1141,6 @@
 short-lived, but the type check removals are similar to what our optimization
 achieves.
 
-partial evaluation:
-
 Partially known data structures are built directly into Prolog (via unbound
 logic variables) and thus the treatment of partially static data structures
 was part of partial evaluation of Prolog programs from the early stages
@@ -1152,15 +1149,15 @@
 residual program. This is similar to what our optimization does for an
 imperative language.
 In functional programming this idea was introduced as
- constructor specialisation by Mogensen \cite{Mogensen:PEPM93}.)
+ constructor specialisation by Mogensen \cite{Mogensen:PEPM93}.
 
 %xxx:
 %partially static data structures: kenichi asai's thesis?
 
-xxx: 
-Other related work is compile-time garbage collection \cite{mazur_practical_2001}, which
-tries to ....;
-  separation logic
+%xxx: 
+%Other related work is compile-time garbage collection \cite{mazur_practical_2001}, which
+%tries to ....;
+%  separation logic
   %; John Hughes: type specialization
 
 \section{Conclusion}


From arigo at codespeak.net  Fri Oct 15 17:30:43 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 17:30:43 +0200 (CEST)
Subject: [pypy-svn] r77998 - pypy/branch/leak-finder/pypy/module/thread
Message-ID: <20101015153043.52D1F36E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 17:30:41 2010
New Revision: 77998

Modified:
   pypy/branch/leak-finder/pypy/module/thread/ll_thread.py
Log:
Set track_allocation=False on the malloced locks.
Explain why.


Modified: pypy/branch/leak-finder/pypy/module/thread/ll_thread.py
==============================================================================
--- pypy/branch/leak-finder/pypy/module/thread/ll_thread.py	(original)
+++ pypy/branch/leak-finder/pypy/module/thread/ll_thread.py	Fri Oct 15 17:30:41 2010
@@ -111,7 +111,7 @@
             c_thread_releaselock(self._lock)
 
     def __del__(self):
-        lltype.free(self._lock, flavor='raw')
+        lltype.free(self._lock, flavor='raw', track_allocation=False)
 
 # ____________________________________________________________
 #
@@ -128,10 +128,13 @@
 null_ll_lock = lltype.nullptr(TLOCKP.TO)
 
 def allocate_ll_lock():
-    ll_lock = lltype.malloc(TLOCKP.TO, flavor='raw')
+    # track_allocation=False here; be careful to lltype.free() it.  The
+    # reason it is set to False is that we get it from all app-level
+    # lock objects, as well as from the GIL, which exists at shutdown.
+    ll_lock = lltype.malloc(TLOCKP.TO, flavor='raw', track_allocation=False)
     res = c_thread_lock_init(ll_lock)
     if res == -1:
-        lltype.free(ll_lock, flavor='raw')
+        lltype.free(ll_lock, flavor='raw', track_allocation=False)
         raise error("out of resources")
     return ll_lock
 


From arigo at codespeak.net  Fri Oct 15 17:35:04 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 17:35:04 +0200 (CEST)
Subject: [pypy-svn] r77999 - pypy/branch/leak-finder/pypy/translator/c/test
Message-ID: <20101015153504.16F2836E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 17:35:02 2010
New Revision: 77999

Modified:
   pypy/branch/leak-finder/pypy/translator/c/test/test_genc.py
Log:
Fix test.


Modified: pypy/branch/leak-finder/pypy/translator/c/test/test_genc.py
==============================================================================
--- pypy/branch/leak-finder/pypy/translator/c/test/test_genc.py	(original)
+++ pypy/branch/leak-finder/pypy/translator/c/test/test_genc.py	Fri Oct 15 17:35:02 2010
@@ -426,6 +426,7 @@
     if py.test.config.option.view:
         t.view()
     assert ' BarStruct ' in t.driver.cbuilder.c_source_filename.read()
+    free(foo, flavor="raw")
 
 def test_recursive_llhelper():
     from pypy.rpython.annlowlevel import llhelper
@@ -473,7 +474,7 @@
         return f(s)
     a_f = A(f, "f")
     a_g = A(g, "g")
-    t = lltype.malloc(STRUCT, flavor="raw")
+    t = lltype.malloc(STRUCT, flavor="raw", immortal=True)
     t.bar = llhelper(FTPTR, a_f.make_func())
     fn = compile(chooser, [bool])
     assert fn(True)


From arigo at codespeak.net  Fri Oct 15 17:36:31 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 17:36:31 +0200 (CEST)
Subject: [pypy-svn] r78000 -
	pypy/branch/leak-finder/pypy/rpython/memory/gctransform
Message-ID: <20101015153631.7635E36E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 17:36:29 2010
New Revision: 78000

Modified:
   pypy/branch/leak-finder/pypy/rpython/memory/gctransform/asmgcroot.py
Log:
Fix.


Modified: pypy/branch/leak-finder/pypy/rpython/memory/gctransform/asmgcroot.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/memory/gctransform/asmgcroot.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/memory/gctransform/asmgcroot.py	Fri Oct 15 17:36:29 2010
@@ -75,7 +75,8 @@
             key = (TYPE, num)
             if key not in sradict:
                 CONTAINER = lltype.FixedSizeArray(TYPE, 1)
-                p = lltype.malloc(CONTAINER, flavor='raw', zero=True)
+                p = lltype.malloc(CONTAINER, flavor='raw', zero=True,
+                                  immortal=True)
                 sradict[key] = Constant(p, lltype.Ptr(CONTAINER))
             sra.append(sradict[key])
         #


From afa at codespeak.net  Fri Oct 15 18:00:15 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 18:00:15 +0200 (CEST)
Subject: [pypy-svn] r78001 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101015160015.7FBC836E0C1@codespeak.net>

Author: afa
Date: Fri Oct 15 18:00:10 2010
New Revision: 78001

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
Log:
_io.open() now accepts a file descriptor


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Fri Oct 15 18:00:10 2010
@@ -88,6 +88,8 @@
         if space.isinstance_w(w_name, space.w_float):
             raise OperationError(space.w_TypeError, space.wrap(
                 "integer argument expected, got float"))
+
+        fd = -1
         try:
             fd = space.int_w(w_name)
         except OperationError, e:
@@ -99,12 +101,16 @@
 
         self.readable, self.writable, flags = decode_mode(space, mode)
 
-        from pypy.module.posix.interp_posix import dispatch_filename, rposix
-        try:
-            self.fd = dispatch_filename(rposix.open)(
-                space, w_name, flags, 0666)
-        except OSError, e:
-            raise wrap_oserror2(space, e, w_name)
+        if fd >= 0:
+            self.fd = fd
+        else:
+            from pypy.module.posix.interp_posix import (
+                dispatch_filename, rposix)
+            try:
+                self.fd = dispatch_filename(rposix.open)(
+                    space, w_name, flags, 0666)
+            except OSError, e:
+                raise wrap_oserror2(space, e, w_name)
         self.closefd = bool(closefd)
 
     def _check_closed(self, space):

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	Fri Oct 15 18:00:10 2010
@@ -1,4 +1,5 @@
 from pypy.conftest import gettestobjspace
+import os
 
 class AppTestIoModule:
     def setup_class(cls):
@@ -42,8 +43,18 @@
         from pypy.tool.udir import udir
         tmpfile = udir.join('tmpfile').ensure()
         cls.w_tmpfile = cls.space.wrap(str(tmpfile))
+        cls.w_posix = cls.space.appexec([], """():
+            import %s as m;
+            return m""" % os.name)
 
     def test_open(self):
         import io
         f = io.open(self.tmpfile, "rb")
         f.close()
+
+    def test_open_fd(self):
+        import io
+        os = self.posix
+        fd = os.open(self.tmpfile, os.O_RDONLY, 0666)
+        f = io.open(fd, "rb")
+        f.close()


From arigo at codespeak.net  Fri Oct 15 18:35:46 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 18:35:46 +0200 (CEST)
Subject: [pypy-svn] r78002 - in pypy/branch/leak-finder/pypy: rlib
	rpython/lltypesystem rpython/memory/test
Message-ID: <20101015163546.B0D7D36E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 18:35:43 2010
New Revision: 78002

Modified:
   pypy/branch/leak-finder/pypy/rlib/rgc.py
   pypy/branch/leak-finder/pypy/rpython/lltypesystem/rffi.py
   pypy/branch/leak-finder/pypy/rpython/memory/test/test_gc.py
Log:
Test and fix for *a real leak*!  Yay :-)


Modified: pypy/branch/leak-finder/pypy/rlib/rgc.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rlib/rgc.py	(original)
+++ pypy/branch/leak-finder/pypy/rlib/rgc.py	Fri Oct 15 18:35:43 2010
@@ -170,7 +170,14 @@
         return hop.genop('gc_set_max_heap_size', [v_nbytes],
                          resulttype=lltype.Void)
 
-def can_move(p):    # NB. must not be called with NULL pointers
+def can_move(p):
+    """Check if the GC object 'p' is at an address that can move.
+    Must not be called with None.  With non-moving GCs, it is always False.
+    With some moving GCs like the SemiSpace GC, it is always True.
+    With other moving GCs like the MiniMark GC, it can be True for some
+    time, then False for the same object, when we are sure that it won't
+    move any more.
+    """
     return True
 
 class CanMoveEntry(ExtRegistryEntry):

Modified: pypy/branch/leak-finder/pypy/rpython/lltypesystem/rffi.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/lltypesystem/rffi.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/lltypesystem/rffi.py	Fri Oct 15 18:35:43 2010
@@ -644,7 +644,14 @@
         """
         Either free a non-moving buffer or keep the original storage alive.
         """
-        if rgc.can_move(data):
+        # We cannot rely on rgc.can_move(data) here, because its result
+        # might have changed since get_nonmovingbuffer().  Instead we check
+        # if 'buf' points inside 'data'.  This is only possible if we
+        # followed the 2nd case in get_nonmovingbuffer(); in the first case,
+        # 'buf' points to its own raw-malloced memory.
+        data_start = cast_ptr_to_adr(llstrtype(data)) + \
+            offsetof(STRTYPE, 'chars') + itemoffsetof(STRTYPE.chars, 0)
+        if buf != cast(TYPEP, data_start):
             lltype.free(buf, flavor='raw')
         else:
             keepalive_until_here(data)

Modified: pypy/branch/leak-finder/pypy/rpython/memory/test/test_gc.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/memory/test/test_gc.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/memory/test/test_gc.py	Fri Oct 15 18:35:43 2010
@@ -368,6 +368,14 @@
         res = self.interpret(f, [4, 42])
         assert res == 12
 
+    def test_print_leak(self):
+        def f(n):
+            for i in range(n):
+                print i
+            return 42
+        res = self.interpret(f, [10])
+        assert res == 42
+
     def test_weakref_across_minor_collection(self):
         import weakref
         class A:


From arigo at codespeak.net  Fri Oct 15 18:42:24 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 18:42:24 +0200 (CEST)
Subject: [pypy-svn] r78003 -
	pypy/branch/leak-finder/pypy/rpython/lltypesystem
Message-ID: <20101015164224.C384C36E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 18:42:23 2010
New Revision: 78003

Modified:
   pypy/branch/leak-finder/pypy/rpython/lltypesystem/rffi.py
Log:
Fix.


Modified: pypy/branch/leak-finder/pypy/rpython/lltypesystem/rffi.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rpython/lltypesystem/rffi.py	(original)
+++ pypy/branch/leak-finder/pypy/rpython/lltypesystem/rffi.py	Fri Oct 15 18:42:23 2010
@@ -649,12 +649,13 @@
         # if 'buf' points inside 'data'.  This is only possible if we
         # followed the 2nd case in get_nonmovingbuffer(); in the first case,
         # 'buf' points to its own raw-malloced memory.
-        data_start = cast_ptr_to_adr(llstrtype(data)) + \
+        data = llstrtype(data)
+        data_start = cast_ptr_to_adr(data) + \
             offsetof(STRTYPE, 'chars') + itemoffsetof(STRTYPE.chars, 0)
-        if buf != cast(TYPEP, data_start):
+        followed_2nd_path = (buf == cast(TYPEP, data_start))
+        keepalive_until_here(data)
+        if not followed_2nd_path:
             lltype.free(buf, flavor='raw')
-        else:
-            keepalive_until_here(data)
 
     # int -> (char*, str)
     def alloc_buffer(count):


From arigo at codespeak.net  Fri Oct 15 18:59:47 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 18:59:47 +0200 (CEST)
Subject: [pypy-svn] r78004 - pypy/branch/leak-finder/pypy/module/cpyext/test
Message-ID: <20101015165947.60F5B36E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 18:59:45 2010
New Revision: 78004

Modified:
   pypy/branch/leak-finder/pypy/module/cpyext/test/test_cpyext.py
Log:
Fix this test to depend on the default leakfinder.


Modified: pypy/branch/leak-finder/pypy/module/cpyext/test/test_cpyext.py
==============================================================================
--- pypy/branch/leak-finder/pypy/module/cpyext/test/test_cpyext.py	(original)
+++ pypy/branch/leak-finder/pypy/module/cpyext/test/test_cpyext.py	Fri Oct 15 18:59:45 2010
@@ -16,6 +16,7 @@
 from pypy.module.cpyext.pyobject import Py_DecRef, InvalidPointerException
 from pypy.translator.goal import autopath
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 
 @api.cpython_api([], api.PyObject)
 def PyPy_Crash1(space):
@@ -78,7 +79,6 @@
         self.frozen_refcounts[w_obj] = obj.c_ob_refcnt
     #state.print_refcounts()
     self.frozen_ll2callocations = set(ll2ctypes.ALLOCATED.values())
-    lltype.start_tracking_allocations()
 
 class LeakCheckingTest(object):
     def check_and_print_leaks(self):
@@ -126,17 +126,8 @@
         for w_obj in lost_objects_w:
             print >>sys.stderr, "Lost object %r" % (w_obj, )
             leaking = True
-##        for llvalue in set(ll2ctypes.ALLOCATED.values()) - self.frozen_ll2callocations:
-##            if llvalue in lltype.ALLOCATED:
-##                leaking = True
-##                print >>sys.stderr, "Did not deallocate %r (ll2ctypes)" % (llvalue, )
-##                print >>sys.stderr, "\t" + "\n\t".join(llvalue._traceback.splitlines())
-        for llvalue, traceback in lltype.ALLOCATED.items():
-            leaking = True
-            print >>sys.stderr, "Did not deallocate %r (llvalue)" % (llvalue, )
-            print >>sys.stderr, "\t" + "\n\t".join(traceback.splitlines())
-
-        lltype.stop_tracking_allocations(check=False)
+        # the actual low-level leak checking is done by pypy.tool.leakfinder,
+        # enabled automatically by pypy.conftest.
         return leaking
 
 class AppTestCpythonExtensionBase(LeakCheckingTest):


From arigo at codespeak.net  Fri Oct 15 19:03:29 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 15 Oct 2010 19:03:29 +0200 (CEST)
Subject: [pypy-svn] r78005 - pypy/trunk/pypy/module/cpyext/test
Message-ID: <20101015170329.D9B7836E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 15 19:03:28 2010
New Revision: 78005

Modified:
   pypy/trunk/pypy/module/cpyext/test/test_unicodeobject.py
Log:
64-bit fix.


Modified: pypy/trunk/pypy/module/cpyext/test/test_unicodeobject.py
==============================================================================
--- pypy/trunk/pypy/module/cpyext/test/test_unicodeobject.py	(original)
+++ pypy/trunk/pypy/module/cpyext/test/test_unicodeobject.py	Fri Oct 15 19:03:28 2010
@@ -177,13 +177,14 @@
             encoded_charp = rffi.str2charp(encoded)
             strict_charp = rffi.str2charp("strict")
             if endian is not None:
-                pendian = lltype.malloc(rffi.INTP.TO, 1, flavor='raw')
                 if endian < 0:
-                    pendian[0] = -1
+                    value = -1
                 elif endian > 0:
-                    pendian[0] = 1
+                    value = 1
                 else:
-                    pendian[0] = 0
+                    value = 0
+                pendian = lltype.malloc(rffi.INTP.TO, 1, flavor='raw')
+                pendian[0] = rffi.cast(rffi.INT, value)
             else:
                 pendian = None
 


From afa at codespeak.net  Fri Oct 15 19:26:06 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 19:26:06 +0200 (CEST)
Subject: [pypy-svn] r78006 - pypy/branch/fast-forward/pypy/module/imp
Message-ID: <20101015172606.9244336E0C3@codespeak.net>

Author: afa
Date: Fri Oct 15 19:26:04 2010
New Revision: 78006

Modified:
   pypy/branch/fast-forward/pypy/module/imp/__init__.py
Log:
Grab and release the import lock when forking the interpreter.

I could not find any test for this change, but:
- CPython 2.7 has similar code
- when translated, it helps a lot test_multiprocessing, which did not even start a single test before.


Modified: pypy/branch/fast-forward/pypy/module/imp/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/imp/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/imp/__init__.py	Fri Oct 15 19:26:04 2010
@@ -38,9 +38,9 @@
     def __init__(self, space, *args):
         "NOT_RPYTHON"
         MixedModule.__init__(self, space, *args)
-        # from pypy.module.posix.interp_posix import add_fork_hook
-        # from pypy.module.imp import interp_imp
-        # add_fork_hook('before', interp_imp.acquire_lock)
-        # add_fork_hook('parent', interp_imp.release_lock)
-        # add_fork_hook('child', interp_imp.reinit_lock)
+        from pypy.module.posix.interp_posix import add_fork_hook
+        from pypy.module.imp import interp_imp
+        add_fork_hook('before', interp_imp.acquire_lock)
+        add_fork_hook('parent', interp_imp.release_lock)
+        add_fork_hook('child', interp_imp.reinit_lock)
 


From afa at codespeak.net  Fri Oct 15 19:30:03 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 19:30:03 +0200 (CEST)
Subject: [pypy-svn] r78007 - in pypy/branch/fast-forward:
	lib-python/modified-2.7.0/test pypy/config
Message-ID: <20101015173003.0E60436E0CB@codespeak.net>

Author: afa
Date: Fri Oct 15 19:30:02 2010
New Revision: 78007

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_multiprocessing.py
      - copied, changed from r77992, pypy/branch/fast-forward/lib-python/2.7.0/test/test_multiprocessing.py
Modified:
   pypy/branch/fast-forward/pypy/config/pypyoption.py
Log:
Enable the _mulitprocessing module, and be careful to skip tests that are known to block.


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_multiprocessing.py (from r77992, pypy/branch/fast-forward/lib-python/2.7.0/test/test_multiprocessing.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_multiprocessing.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_multiprocessing.py	Fri Oct 15 19:30:02 2010
@@ -501,6 +501,7 @@
 
         p.join()
 
+    @unittest.skipIf(os.name == 'posix', "PYPY: FIXME")
     def test_qsize(self):
         q = self.Queue()
         try:
@@ -521,6 +522,7 @@
             time.sleep(DELTA)
             q.task_done()
 
+    @unittest.skipIf(os.name == 'posix', "PYPY: FIXME")
     def test_task_done(self):
         queue = self.JoinableQueue()
 
@@ -1074,6 +1076,7 @@
 class _TestPoolWorkerLifetime(BaseTestCase):
 
     ALLOWED_TYPES = ('processes', )
+    @unittest.skipIf(os.name == 'posix', "PYPY: FIXME")
     def test_pool_worker_lifetime(self):
         p = multiprocessing.Pool(3, maxtasksperchild=10)
         self.assertEqual(3, len(p._pool))
@@ -1260,6 +1263,7 @@
         queue = manager.get_queue()
         queue.put('hello world')
 
+    @unittest.skipIf(os.name == 'posix', "PYPY: FIXME")
     def test_rapid_restart(self):
         authkey = os.urandom(32)
         manager = QueueManager(
@@ -1546,6 +1550,7 @@
 
     ALLOWED_TYPES = ('processes',)
 
+    @unittest.skipIf(os.name == 'posix', "PYPY: FIXME")
     def test_heap(self):
         iterations = 5000
         maxblocks = 50

Modified: pypy/branch/fast-forward/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/fast-forward/pypy/config/pypyoption.py	(original)
+++ pypy/branch/fast-forward/pypy/config/pypyoption.py	Fri Oct 15 19:30:02 2010
@@ -30,7 +30,8 @@
       "rctime" , "select", "zipimport", "_lsprof",
      "crypt", "signal", "_rawffi", "termios", "zlib", "bz2",
      "struct", "_hashlib", "_md5", "_sha", "_minimal_curses", "cStringIO",
-     "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array"]
+     "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array",
+     "_multiprocessing"]
 ))
 
 working_oo_modules = default_modules.copy()


From afa at codespeak.net  Fri Oct 15 22:26:52 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 22:26:52 +0200 (CEST)
Subject: [pypy-svn] r78008 - pypy/branch/fast-forward/pypy/module/_io
Message-ID: <20101015202652.B96C036E35D@codespeak.net>

Author: afa
Date: Fri Oct 15 22:26:50 2010
New Revision: 78008

Added:
   pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py   (contents, props changed)
Modified:
   pypy/branch/fast-forward/pypy/module/_io/__init__.py
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/interp_io.py
Log:
Move _io.IOBase into its own file, like CPython's _io/iobase.c


Modified: pypy/branch/fast-forward/pypy/module/_io/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/__init__.py	Fri Oct 15 22:26:50 2010
@@ -9,8 +9,8 @@
     interpleveldefs = {
         'DEFAULT_BUFFER_SIZE': 'space.wrap(interp_io.DEFAULT_BUFFER_SIZE)',
         'BlockingIOError': 'interp_io.W_BlockingIOError',
-        '_IOBase': 'interp_io.W_IOBase',
-        '_RawIOBase': 'interp_io.W_RawIOBase',
+        '_IOBase': 'interp_iobase.W_IOBase',
+        '_RawIOBase': 'interp_iobase.W_RawIOBase',
         '_BufferedIOBase': 'interp_io.W_BufferedIOBase',
         '_TextIOBase': 'interp_io.W_TextIOBase',
 

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Fri Oct 15 22:26:50 2010
@@ -1,4 +1,4 @@
-from pypy.module._io.interp_io import W_RawIOBase
+from pypy.module._io.interp_iobase import W_RawIOBase
 from pypy.interpreter.typedef import TypeDef
 from pypy.interpreter.gateway import interp2app, unwrap_spec, Arguments
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
@@ -84,7 +84,7 @@
         return space.wrap(self)
 
     @unwrap_spec('self', ObjSpace, W_Root, str, int)
-    def descr_init(self, space, w_name, mode, closefd):
+    def descr_init(self, space, w_name, mode='r', closefd=True):
         if space.isinstance_w(w_name, space.w_float):
             raise OperationError(space.w_TypeError, space.wrap(
                 "integer argument expected, got float"))

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py	Fri Oct 15 22:26:50 2010
@@ -1,10 +1,9 @@
-from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root
+from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.typedef import (
-    TypeDef, interp_attrproperty, GetSetProperty, generic_new_descr)
+    TypeDef, interp_attrproperty, generic_new_descr)
 from pypy.interpreter.gateway import interp2app, Arguments, unwrap_spec
-from pypy.interpreter.error import OperationError
 from pypy.module.exceptions.interp_exceptions import W_IOError
-from pypy.tool.sourcetools import func_renamer
+from pypy.module._io.interp_iobase import W_IOBase
 
 DEFAULT_BUFFER_SIZE = 8192
 
@@ -27,112 +26,6 @@
     characters_written = interp_attrproperty('written', W_BlockingIOError),
     )
 
-class W_IOBase(Wrappable):
-    def __init__(self, space):
-        # XXX: IOBase thinks it has to maintain its own internal state in
-        # `__IOBase_closed` and call flush() by itself, but it is redundant
-        # with whatever behaviour a non-trivial derived class will implement.
-        self.__IOBase_closed = False
-
-    def _closed(self, space):
-        # This gets the derived attribute, which is *not* __IOBase_closed
-        # in most cases!
-        w_closed = space.findattr(self, space.wrap('closed'))
-        if w_closed is not None and space.is_true(w_closed):
-            return True
-        return False
-
-    def _CLOSED(self):
-        # Use this macro whenever you want to check the internal `closed`
-        # status of the IOBase object rather than the virtual `closed`
-        # attribute as returned by whatever subclass.
-        return self.__IOBase_closed
-
-    def _check_closed(self, space):
-        if self._closed(space):
-            raise OperationError(
-                space.w_ValueError,
-                space.wrap("I/O operation on closed file"))
-
-    def closed_get_w(space, self):
-        return space.newbool(self.__IOBase_closed)
-
-    @unwrap_spec('self', ObjSpace)
-    def close_w(self, space):
-        if self._CLOSED():
-            return
-        try:
-            space.call_method(self, "flush")
-        finally:
-            self.__IOBase_closed = True
-
-    @unwrap_spec('self', ObjSpace)
-    def flush_w(self, space):
-        if self._CLOSED():
-            raise OperationError(
-                space.w_ValueError,
-                space.wrap("I/O operation on closed file"))
-
-    @unwrap_spec('self', ObjSpace)
-    def enter_w(self, space):
-        self._check_closed(space)
-        return space.wrap(self)
-
-    @unwrap_spec('self', ObjSpace, Arguments)
-    def exit_w(self, space, __args__):
-        space.call_method(self, "close")
-
-    @unwrap_spec('self', ObjSpace)
-    def iter_w(self, space):
-        self._check_closed(space)
-        return space.wrap(self)
-
-    @unwrap_spec('self', ObjSpace)
-    def next_w(self, space):
-        w_line = space.call_method(self, "readline")
-        if space.int_w(space.len(w_line)) == 0:
-            raise OperationError(space.w_StopIteration, space.w_None)
-        return w_line
-
-    @unwrap_spec('self', ObjSpace)
-    def isatty_w(self, space):
-        return space.w_False
-
-    @unwrap_spec('self', ObjSpace)
-    def readable_w(self, space):
-        return space.w_False
-
-    @unwrap_spec('self', ObjSpace)
-    def writable_w(self, space):
-        return space.w_False
-
-    @unwrap_spec('self', ObjSpace)
-    def seekable_w(self, space):
-        return space.w_False
-
-W_IOBase.typedef = TypeDef(
-    '_IOBase',
-    __new__ = generic_new_descr(W_IOBase),
-    __enter__ = interp2app(W_IOBase.enter_w),
-    __exit__ = interp2app(W_IOBase.exit_w),
-    __iter__ = interp2app(W_IOBase.iter_w),
-    next = interp2app(W_IOBase.next_w),
-    close = interp2app(W_IOBase.close_w),
-    flush = interp2app(W_IOBase.flush_w),
-    isatty = interp2app(W_IOBase.isatty_w),
-    readable = interp2app(W_IOBase.readable_w),
-    writable = interp2app(W_IOBase.writable_w),
-    seekable = interp2app(W_IOBase.seekable_w),
-    closed = GetSetProperty(W_IOBase.closed_get_w),
-    )
-
-class W_RawIOBase(W_IOBase):
-    pass
-W_RawIOBase.typedef = TypeDef(
-    '_RawIOBase', W_IOBase.typedef,
-    __new__ = generic_new_descr(W_RawIOBase),
-    )
-
 class W_BufferedIOBase(W_IOBase):
     pass
 

Added: pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
==============================================================================
--- (empty file)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	Fri Oct 15 22:26:50 2010
@@ -0,0 +1,112 @@
+from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root
+from pypy.interpreter.typedef import (
+    TypeDef, GetSetProperty, generic_new_descr)
+from pypy.interpreter.gateway import interp2app, Arguments, unwrap_spec
+from pypy.interpreter.error import OperationError
+
+class W_IOBase(Wrappable):
+    def __init__(self, space):
+        # XXX: IOBase thinks it has to maintain its own internal state in
+        # `__IOBase_closed` and call flush() by itself, but it is redundant
+        # with whatever behaviour a non-trivial derived class will implement.
+        self.__IOBase_closed = False
+
+    def _closed(self, space):
+        # This gets the derived attribute, which is *not* __IOBase_closed
+        # in most cases!
+        w_closed = space.findattr(self, space.wrap('closed'))
+        if w_closed is not None and space.is_true(w_closed):
+            return True
+        return False
+
+    def _CLOSED(self):
+        # Use this macro whenever you want to check the internal `closed`
+        # status of the IOBase object rather than the virtual `closed`
+        # attribute as returned by whatever subclass.
+        return self.__IOBase_closed
+
+    def _check_closed(self, space):
+        if self._closed(space):
+            raise OperationError(
+                space.w_ValueError,
+                space.wrap("I/O operation on closed file"))
+
+    def closed_get_w(space, self):
+        return space.newbool(self.__IOBase_closed)
+
+    @unwrap_spec('self', ObjSpace)
+    def close_w(self, space):
+        if self._CLOSED():
+            return
+        try:
+            space.call_method(self, "flush")
+        finally:
+            self.__IOBase_closed = True
+
+    @unwrap_spec('self', ObjSpace)
+    def flush_w(self, space):
+        if self._CLOSED():
+            raise OperationError(
+                space.w_ValueError,
+                space.wrap("I/O operation on closed file"))
+
+    @unwrap_spec('self', ObjSpace)
+    def enter_w(self, space):
+        self._check_closed(space)
+        return space.wrap(self)
+
+    @unwrap_spec('self', ObjSpace, Arguments)
+    def exit_w(self, space, __args__):
+        space.call_method(self, "close")
+
+    @unwrap_spec('self', ObjSpace)
+    def iter_w(self, space):
+        self._check_closed(space)
+        return space.wrap(self)
+
+    @unwrap_spec('self', ObjSpace)
+    def next_w(self, space):
+        w_line = space.call_method(self, "readline")
+        if space.int_w(space.len(w_line)) == 0:
+            raise OperationError(space.w_StopIteration, space.w_None)
+        return w_line
+
+    @unwrap_spec('self', ObjSpace)
+    def isatty_w(self, space):
+        return space.w_False
+
+    @unwrap_spec('self', ObjSpace)
+    def readable_w(self, space):
+        return space.w_False
+
+    @unwrap_spec('self', ObjSpace)
+    def writable_w(self, space):
+        return space.w_False
+
+    @unwrap_spec('self', ObjSpace)
+    def seekable_w(self, space):
+        return space.w_False
+
+W_IOBase.typedef = TypeDef(
+    '_IOBase',
+    __new__ = generic_new_descr(W_IOBase),
+    __enter__ = interp2app(W_IOBase.enter_w),
+    __exit__ = interp2app(W_IOBase.exit_w),
+    __iter__ = interp2app(W_IOBase.iter_w),
+    next = interp2app(W_IOBase.next_w),
+    close = interp2app(W_IOBase.close_w),
+    flush = interp2app(W_IOBase.flush_w),
+    isatty = interp2app(W_IOBase.isatty_w),
+    readable = interp2app(W_IOBase.readable_w),
+    writable = interp2app(W_IOBase.writable_w),
+    seekable = interp2app(W_IOBase.seekable_w),
+    closed = GetSetProperty(W_IOBase.closed_get_w),
+    )
+
+class W_RawIOBase(W_IOBase):
+    pass
+W_RawIOBase.typedef = TypeDef(
+    '_RawIOBase', W_IOBase.typedef,
+    __new__ = generic_new_descr(W_RawIOBase),
+    )
+


From afa at codespeak.net  Fri Oct 15 22:47:55 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 22:47:55 +0200 (CEST)
Subject: [pypy-svn] r78009 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101015204755.ADBCC36E363@codespeak.net>

Author: afa
Date: Fri Oct 15 22:47:54 2010
New Revision: 78009

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
Log:
More progress in the _io module,
one more instruction passed by some functions of the test suite.


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Fri Oct 15 22:47:54 2010
@@ -3,6 +3,7 @@
 from pypy.interpreter.gateway import interp2app, unwrap_spec, Arguments
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.error import OperationError, wrap_oserror2
+from pypy.rlib.rarithmetic import r_longlong
 from os import O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_TRUNC
 import os
 
@@ -118,6 +119,12 @@
             raise OperationError(space.w_ValueError, space.wrap(
                 "I/O operation on closed file"))
 
+    @unwrap_spec('self', ObjSpace, r_longlong, int)
+    def seek_w(self, space, pos, whence):
+        self._check_closed(space)
+        pos = os.lseek(self.fd, pos, whence)
+        return space.wrap(pos)
+
     @unwrap_spec('self', ObjSpace)
     def readable_w(self, space):
         self._check_closed(space)
@@ -128,11 +135,27 @@
         self._check_closed(space)
         return space.wrap(self.writable)
 
+    @unwrap_spec('self', ObjSpace)
+    def seekable_w(self, space):
+        self._check_closed(space)
+        if self.seekable < 0:
+            try:
+                pos = os.lseek(self.fd, 0, os.SEEK_CUR)
+            except OSError:
+                self.seekable = 0
+            else:
+                self.seekable = 1
+        return space.newbool(self.seekable)
+
 W_FileIO.typedef = TypeDef(
     'FileIO', W_RawIOBase.typedef,
     __new__  = interp2app(W_FileIO.descr_new.im_func),
     __init__  = interp2app(W_FileIO.descr_init),
+
+    seek = interp2app(W_FileIO.seek_w),
+
     readable = interp2app(W_FileIO.readable_w),
     writable = interp2app(W_FileIO.writable_w),
+    seekable = interp2app(W_FileIO.seekable_w),
     )
 

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	Fri Oct 15 22:47:54 2010
@@ -87,6 +87,13 @@
     def seekable_w(self, space):
         return space.w_False
 
+    @unwrap_spec('self', ObjSpace)
+    def check_seekable_w(self, space):
+        if not space.is_true(space.call_method(self, 'seekable')):
+            raise OperationError(
+                space.w_IOError,
+                space.wrap("file or stream is not seekable"))
+
 W_IOBase.typedef = TypeDef(
     '_IOBase',
     __new__ = generic_new_descr(W_IOBase),
@@ -100,6 +107,7 @@
     readable = interp2app(W_IOBase.readable_w),
     writable = interp2app(W_IOBase.writable_w),
     seekable = interp2app(W_IOBase.seekable_w),
+    _checkSeekable = interp2app(W_IOBase.check_seekable_w),
     closed = GetSetProperty(W_IOBase.closed_get_w),
     )
 

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	Fri Oct 15 22:47:54 2010
@@ -52,6 +52,12 @@
         f = io.open(self.tmpfile, "rb")
         f.close()
 
+    def test_open_writable(self):
+        import io
+        f = io.open(self.tmpfile, "w+b")
+        f.seek(0)
+        f.close()
+
     def test_open_fd(self):
         import io
         os = self.posix


From afa at codespeak.net  Fri Oct 15 23:22:42 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 15 Oct 2010 23:22:42 +0200 (CEST)
Subject: [pypy-svn] r78010 - pypy/branch/fast-forward/pypy/module/_io
Message-ID: <20101015212242.DA96136E393@codespeak.net>

Author: afa
Date: Fri Oct 15 23:22:41 2010
New Revision: 78010

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
Log:
Fix translation


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Fri Oct 15 23:22:41 2010
@@ -145,7 +145,7 @@
                 self.seekable = 0
             else:
                 self.seekable = 1
-        return space.newbool(self.seekable)
+        return space.newbool(self.seekable == 1)
 
 W_FileIO.typedef = TypeDef(
     'FileIO', W_RawIOBase.typedef,


From afa at codespeak.net  Sat Oct 16 00:34:51 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sat, 16 Oct 2010 00:34:51 +0200 (CEST)
Subject: [pypy-svn] r78011 - in pypy/branch/fast-forward/pypy:
	module/__builtin__ module/__builtin__/test objspace/std
	objspace/std/test
Message-ID: <20101015223451.AA78736E3B5@codespeak.net>

Author: afa
Date: Sat Oct 16 00:34:43 2010
New Revision: 78011

Modified:
   pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py
   pypy/branch/fast-forward/pypy/module/__builtin__/test/test_classobj.py
   pypy/branch/fast-forward/pypy/objspace/std/inttype.py
   pypy/branch/fast-forward/pypy/objspace/std/longtype.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_intobject.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_longobject.py
Log:
Make int() and long() fall back to __trunc__().

This is the same as CPython r60566, issue2002.
I hope I did it right.


Modified: pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py	(original)
+++ pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py	Sat Oct 16 00:34:43 2010
@@ -595,6 +595,27 @@
                 space.wrap("__hash__ must return int or long"))
         return w_ret
 
+    def descr_int(self, space):
+        w_func = self.getattr(space, space.wrap('__int__'), False)
+        if w_func is not None:
+            return space.call_function(w_func)
+
+        w_truncated = space.trunc(self)
+        # int() needs to return an int
+        try:
+            return space.int(w_truncated)
+        except OperationError:
+            # Raise a different error
+            raise OperationError(
+                space.w_TypeError,
+                space.wrap("__trunc__ returned non-Integral"))
+
+    def descr_long(self, space):
+        w_func = self.getattr(space, space.wrap('__long__'), False)
+        if w_func is not None:
+            return space.call_function(w_func)
+        return self.descr_int(space)
+
     def descr_index(self, space):
         w_func = self.getattr(space, space.wrap('__index__'), False)
         if w_func is not None:
@@ -685,7 +706,7 @@
 rawdict = {}
 
 # unary operations
-for op in "neg pos abs invert int long float oct hex enter reversed".split():
+for op in "neg pos abs invert trunc float oct hex enter reversed".split():
     specialname = "__%s__" % (op, )
     # fool the gateway logic by giving it a real unbound method
     meth = new.instancemethod(
@@ -762,6 +783,10 @@
                          unwrap_spec=['self', ObjSpace, W_Root]),
     __hash__ = interp2app(W_InstanceObject.descr_hash,
                           unwrap_spec=['self', ObjSpace]),
+    __int__ = interp2app(W_InstanceObject.descr_int,
+                           unwrap_spec=['self', ObjSpace]),
+    __long__ = interp2app(W_InstanceObject.descr_long,
+                           unwrap_spec=['self', ObjSpace]),
     __index__ = interp2app(W_InstanceObject.descr_index,
                            unwrap_spec=['self', ObjSpace]),
     __contains__ = interp2app(W_InstanceObject.descr_contains,

Modified: pypy/branch/fast-forward/pypy/module/__builtin__/test/test_classobj.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/__builtin__/test/test_classobj.py	(original)
+++ pypy/branch/fast-forward/pypy/module/__builtin__/test/test_classobj.py	Sat Oct 16 00:34:43 2010
@@ -898,10 +898,18 @@
                       "long": long}
         for opname, opfunc in op_by_name.items():
             assert opfunc(b) == 42
-            assert b.called == ("__" + opname + "__", ())
+            called = b.called
+            assert called == ("__" + opname + "__", ())
         assert oct(a) == '__oct__()'
         assert hex(a) == '__hex__()'
         #
+        class JustTrunc:
+            def __trunc__(self):
+                return 42
+        assert int(JustTrunc()) == 42
+        assert long(JustTrunc()) == 42
+        #
+        #
         class C:
             def __getattr__(self, name):
                 return lambda: 5.5

Modified: pypy/branch/fast-forward/pypy/objspace/std/inttype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/inttype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/inttype.py	Sat Oct 16 00:34:43 2010
@@ -101,8 +101,13 @@
             except ParseStringOverflowError, e:
                  w_longval = retry_to_w_long(space, e.parser)
         else:
-            # otherwise, use the __int__() method
-            w_obj = space.int(w_value)
+            # otherwise, use the __int__() then __trunc__() methods
+            try:
+                w_obj = space.int(w_value)
+            except OperationError, e:
+                if not e.match(space,space.w_TypeError):
+                    raise
+                w_obj = space.trunc(w_value)
             # 'int(x)' should return whatever x.__int__() returned
             if space.is_w(w_inttype, space.w_int):
                 return w_obj

Modified: pypy/branch/fast-forward/pypy/objspace/std/longtype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/longtype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/longtype.py	Sat Oct 16 00:34:43 2010
@@ -36,8 +36,13 @@
                 raise OperationError(space.w_ValueError,
                                      space.wrap(e.msg))
         else:
-            # otherwise, use the __long__() method
-            w_obj = space.long(w_value)
+            # otherwise, use the __long__(), then the __trunc__ methods
+            try:
+                w_obj = space.long(w_value)
+            except OperationError, e:
+                if not e.match(space,space.w_TypeError):
+                    raise
+                w_obj = space.trunc(w_value)
             # 'long(x)' should return whatever x.__long__() returned
             if space.is_w(w_longtype, space.w_long):
                 return w_obj

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_intobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_intobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_intobject.py	Sat Oct 16 00:34:43 2010
@@ -418,6 +418,12 @@
             pass 
         raises((AttributeError,TypeError), long, b())
 
+    def test_just_trunc(self):
+        class myint(object):
+            def __trunc__(self):
+                return 42
+        assert int(myint()) == 42
+
     def test_override___int__(self):
         class myint(int):
             def __int__(self):

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_longobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_longobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_longobject.py	Sat Oct 16 00:34:43 2010
@@ -246,6 +246,12 @@
         raises(OverflowError, operator.truediv, huge, 3)
         raises(OverflowError, operator.truediv, huge, 3L)
 
+    def test_just_trunc(self):
+        class myint(object):
+            def __trunc__(self):
+                return 42
+        assert long(myint()) == 42
+
     def test_override___long__(self):
         class mylong(long):
             def __long__(self):


From trundle at codespeak.net  Sat Oct 16 00:43:24 2010
From: trundle at codespeak.net (trundle at codespeak.net)
Date: Sat, 16 Oct 2010 00:43:24 +0200 (CEST)
Subject: [pypy-svn] r78012 - pypy/branch/fast-forward/lib-python
Message-ID: <20101015224324.C130036E3BC@codespeak.net>

Author: trundle
Date: Sat Oct 16 00:43:19 2010
New Revision: 78012

Modified:
   pypy/branch/fast-forward/lib-python/TODO
Log:
Add note about catching string exceptions to TODO


Modified: pypy/branch/fast-forward/lib-python/TODO
==============================================================================
--- pypy/branch/fast-forward/lib-python/TODO	(original)
+++ pypy/branch/fast-forward/lib-python/TODO	Sat Oct 16 00:43:19 2010
@@ -35,7 +35,9 @@
   test_pickle()
 
 - "exceptions must be old-style classes or derived from BaseException, not str"
-  in the 'raise' statement and generator.throw()
+  in the 'raise' statement and generator.throw(), catching a string
+  should emit a DeprecationWarning ("catching of string exceptions is
+  deprecated")
 
 - missing builtin: memoryview
 


From arigo at codespeak.net  Sat Oct 16 16:38:29 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 16:38:29 +0200 (CEST)
Subject: [pypy-svn] r78013 - in pypy/branch/leak-finder/pypy/rlib/rsre: .
	test
Message-ID: <20101016143829.5D61B36E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 16:38:26 2010
New Revision: 78013

Added:
   pypy/branch/leak-finder/pypy/rlib/rsre/test/conftest.py
   pypy/branch/leak-finder/pypy/rlib/rsre/test/test_zjit.py
Modified:
   pypy/branch/leak-finder/pypy/rlib/rsre/rsre_core.py
Log:
In-progress.  Start with some tests.


Modified: pypy/branch/leak-finder/pypy/rlib/rsre/rsre_core.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rlib/rsre/rsre_core.py	(original)
+++ pypy/branch/leak-finder/pypy/rlib/rsre/rsre_core.py	Sat Oct 16 16:38:26 2010
@@ -4,6 +4,7 @@
 from pypy.rlib.rsre import rsre_char
 from pypy.tool.sourcetools import func_with_new_name
 from pypy.rlib.objectmodel import we_are_translated
+from pypy.rlib import jit
 
 
 OPCODE_FAILURE            = 0
@@ -56,15 +57,18 @@
     _seen_specname[specname] = True
     # Install a copy of the function under the name '_spec_funcname' in each
     # concrete subclass
+    specialized_methods = []
     for prefix, concreteclass in [('str', StrMatchContext),
                                   ('uni', UnicodeMatchContext)]:
         newfunc = func_with_new_name(func, prefix + specname)
         assert not hasattr(concreteclass, specname)
         setattr(concreteclass, specname, newfunc)
+        specialized_methods.append(newfunc)
     # Return a dispatcher function, specialized on the exact type of 'ctx'
     def dispatch(ctx, *args):
         return getattr(ctx, specname)(*args)
     dispatch._annspecialcase_ = 'specialize:argtype(0)'
+    dispatch._specialized_methods_ = specialized_methods
     return dispatch
 
 # ____________________________________________________________
@@ -75,6 +79,7 @@
 
 class AbstractMatchContext(object):
     """Abstract base class"""
+    _immutable_fields_ = ['pattern[*]', 'flags']
     match_start = 0
     match_end = 0
     match_marks = None
@@ -238,6 +243,7 @@
         self.start_ptr = ptr
         self.start_marks = marks
 
+    @jit.unroll_safe
     def find_first_result(self, ctx):
         ppos = self.ppos
         while ctx.pat(ppos):
@@ -250,6 +256,8 @@
     find_next_result = find_first_result
 
 class RepeatOneMatchResult(MatchResult):
+    jitdriver = jit.JitDriver(greens=['nextppos', 'pattern'],
+                              reds=['ptr', 'self', 'ctx'])
 
     def __init__(self, nextppos, minptr, ptr, marks):
         self.nextppos = nextppos
@@ -259,8 +267,18 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
+        nextppos = self.nextppos
         while ptr >= self.minptr:
-            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
+            #
+            pattern = ctx.pattern
+            self.jitdriver.can_enter_jit(self=self, ptr=ptr, ctx=ctx,
+                                         nextppos=nextppos, pattern=pattern)
+            self.jitdriver.jit_merge_point(self=self, ptr=ptr, ctx=ctx,
+                                           nextppos=nextppos, pattern=pattern)
+            if jit.we_are_jitted():
+                ctx.pattern = pattern
+            #
+            result = sre_match(ctx, nextppos, ptr, self.start_marks)
             ptr -= 1
             if result is not None:
                 self.subresult = result
@@ -270,6 +288,8 @@
 
 
 class MinRepeatOneMatchResult(MatchResult):
+    jitdriver = jit.JitDriver(greens=['nextppos', 'ppos3', 'pattern'],
+                              reds=['ptr', 'self', 'ctx'])
 
     def __init__(self, nextppos, ppos3, maxptr, ptr, marks):
         self.nextppos = nextppos
@@ -280,27 +300,39 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
+        nextppos = self.nextppos
+        ppos3 = self.ppos3
         while ptr <= self.maxptr:
-            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
+            #
+            pattern = ctx.pattern
+            self.jitdriver.can_enter_jit(self=self, ptr=ptr, ctx=ctx,
+                                         nextppos=nextppos, pattern=pattern,
+                                         ppos3=ppos3)
+            self.jitdriver.jit_merge_point(self=self, ptr=ptr, ctx=ctx,
+                                           nextppos=nextppos, pattern=pattern,
+                                           ppos3=ppos3)
+            if jit.we_are_jitted():
+                ctx.pattern = pattern
+            #
+            result = sre_match(ctx, nextppos, ptr, self.start_marks)
             if result is not None:
                 self.subresult = result
                 self.start_ptr = ptr
                 return self
-            if not self.next_char_ok(ctx, ptr):
+            if not self.next_char_ok(ctx, ptr, ppos3):
                 break
             ptr += 1
 
     def find_next_result(self, ctx):
         ptr = self.start_ptr
-        if not self.next_char_ok(ctx, ptr):
+        if not self.next_char_ok(ctx, ptr, self.ppos3):
             return
         self.start_ptr = ptr + 1
         return self.find_first_result(ctx)
 
-    def next_char_ok(self, ctx, ptr):
+    def next_char_ok(self, ctx, ptr, ppos):
         if ptr == ctx.end:
             return False
-        ppos = self.ppos3
         op = ctx.pat(ppos)
         for op1, (checkerfn, _) in unroll_char_checker:
             if op1 == op:
@@ -429,6 +461,7 @@
 # ____________________________________________________________
 
 @specializectx
+ at jit.unroll_safe
 def sre_match(ctx, ppos, ptr, marks):
     """Returns either None or a MatchResult object.  Usually we only need
     the first result, but there is the case of REPEAT...UNTIL where we

Added: pypy/branch/leak-finder/pypy/rlib/rsre/test/conftest.py
==============================================================================
--- (empty file)
+++ pypy/branch/leak-finder/pypy/rlib/rsre/test/conftest.py	Sat Oct 16 16:38:26 2010
@@ -0,0 +1,5 @@
+# import the option --viewloops from the JIT
+
+def pytest_addoption(parser):
+    from pypy.jit.conftest import pytest_addoption
+    pytest_addoption(parser)

Added: pypy/branch/leak-finder/pypy/rlib/rsre/test/test_zjit.py
==============================================================================
--- (empty file)
+++ pypy/branch/leak-finder/pypy/rlib/rsre/test/test_zjit.py	Sat Oct 16 16:38:26 2010
@@ -0,0 +1,69 @@
+from pypy.jit.metainterp.test import test_basic
+from pypy.rlib.nonconst import NonConstant
+from pypy.rlib.debug import make_sure_not_modified
+from pypy.rlib.rsre.test.test_match import get_code
+from pypy.rlib.rsre import rsre_core
+from pypy.rpython.lltypesystem import lltype
+from pypy.rpython.annlowlevel import llstr, hlstr
+
+def entrypoint1(r, string):
+    r = array2list(r)
+    string = hlstr(string)
+    make_sure_not_modified(r)
+    match = rsre_core.match(r, string)
+    if match is None:
+        return -1
+    else:
+        return match.match_end
+
+def list2array(lst):
+    a = lltype.malloc(lltype.GcArray(lltype.Signed), len(lst))
+    for i, x in enumerate(lst):
+        a[i] = x
+    return a
+
+def array2list(a):
+    return [a[i] for i in range(len(a))]
+
+
+def test_jit_unroll_safe():
+    # test that the decorators are applied in the right order
+    assert not hasattr(rsre_core.sre_match, '_jit_unroll_safe_')
+    for m in rsre_core.sre_match._specialized_methods_:
+        assert m._jit_unroll_safe_
+
+
+class TestJitRSre(test_basic.LLJitMixin):
+
+    def meta_interp_match(self, pattern, string):
+        r = get_code(pattern)
+        return self.meta_interp(entrypoint1, [list2array(r), llstr(string)],
+                                listcomp=True, backendopt=True)
+
+    def test_simple_match_1(self):
+        res = self.meta_interp_match(r"ab*bbbbbbbc", "abbbbbbbbbcdef")
+        assert res == 11
+
+    def test_simple_match_2(self):
+        res = self.meta_interp_match(r".*abc", "xxabcyyyyyyyyyyyyy")
+        assert res == 5
+
+    def test_match_minrepeat_1(self):
+        res = self.meta_interp_match(r".*?abc", "xxxxxxxxxxxxxxabc")
+        assert res == 17
+
+    #def test_match_maxuntil_1(self):
+    #    res = self.meta_interp_match(r"(ab)*c", "ababababababababc")
+    #    assert res == 17
+
+    def test_branch_1(self):
+        res = self.meta_interp_match(r".*?(ab|x)c", "xxxxxxxxxxxxxxabc")
+        assert res == 17
+
+    def test_match_minrepeat_2(self):
+        s = ("xxxxxxxxxxabbbbbbbbbb" +
+             "xxxxxxxxxxabbbbbbbbbb" +
+             "xxxxxxxxxxabbbbbbbbbb" +
+             "xxxxxxxxxxabbbbbbbbbbc")
+        res = self.meta_interp_match(r".*?ab+?c", s)
+        assert res == len(s)


From arigo at codespeak.net  Sat Oct 16 16:39:34 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 16:39:34 +0200 (CEST)
Subject: [pypy-svn] r78014 - pypy/branch/leak-finder/pypy/jit/metainterp
Message-ID: <20101016143934.0E7EE36E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 16:39:33 2010
New Revision: 78014

Modified:
   pypy/branch/leak-finder/pypy/jit/metainterp/warmspot.py
Log:
Bug fix: backend optimizations may remove some operations and invalidate
the 'index' of an existing operation inside its block.


Modified: pypy/branch/leak-finder/pypy/jit/metainterp/warmspot.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/metainterp/warmspot.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/metainterp/warmspot.py	Sat Oct 16 16:39:33 2010
@@ -211,9 +211,9 @@
                 "there are multiple jit_merge_points with the same jitdriver"
 
     def split_graph_and_record_jitdriver(self, graph, block, pos):
-        jd = JitDriverStaticData()
-        jd._jit_merge_point_pos = (graph, block, pos)
         op = block.operations[pos]
+        jd = JitDriverStaticData()
+        jd._jit_merge_point_pos = (graph, op)
         args = op.args[2:]
         s_binding = self.translator.annotator.binding
         jd._portal_args_s = [s_binding(v) for v in args]
@@ -457,8 +457,7 @@
             self.make_args_specification(jd)
 
     def make_args_specification(self, jd):
-        graph, block, index = jd._jit_merge_point_pos
-        op = block.operations[index]
+        graph, op = jd._jit_merge_point_pos
         greens_v, reds_v = support.decode_hp_hint_args(op)
         ALLARGS = [v.concretetype for v in (greens_v + reds_v)]
         jd._green_args_spec = [v.concretetype for v in greens_v]
@@ -709,8 +708,14 @@
         # ____________________________________________________________
         # Now mutate origportalgraph to end with a call to portal_runner_ptr
         #
-        _, origblock, origindex = jd._jit_merge_point_pos
-        op = origblock.operations[origindex]
+        _, op = jd._jit_merge_point_pos
+        for origblock in origportalgraph.iterblocks():
+            if op in origblock.operations:
+                break
+        else:
+            assert False, "lost the operation %r in the graph %r" % (
+                op, origportalgraph)
+        origindex = origblock.operations.index(op)
         assert op.opname == 'jit_marker'
         assert op.args[0].value == 'jit_merge_point'
         greens_v, reds_v = support.decode_hp_hint_args(op)


From arigo at codespeak.net  Sat Oct 16 16:40:51 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 16:40:51 +0200 (CEST)
Subject: [pypy-svn] r78015 - in pypy/branch/leak-finder: dotviewer
	pypy/jit/metainterp
Message-ID: <20101016144051.E216936E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 16:40:50 2010
New Revision: 78015

Modified:
   pypy/branch/leak-finder/dotviewer/drawgraph.py
   pypy/branch/leak-finder/pypy/jit/metainterp/graphpage.py
   pypy/branch/leak-finder/pypy/jit/metainterp/resoperation.py
Log:
Improve the readability of the JIT flow graphs by changing the color of
the prefix to the operations (the name of where it comes from).



Modified: pypy/branch/leak-finder/dotviewer/drawgraph.py
==============================================================================
--- pypy/branch/leak-finder/dotviewer/drawgraph.py	(original)
+++ pypy/branch/leak-finder/dotviewer/drawgraph.py	Sat Oct 16 16:40:50 2010
@@ -423,20 +423,43 @@
         else:
             for line in lines:
                 raw_line = line.replace('\\l','').replace('\r','') or ' '
-                img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
-                w, h = img.get_size()
-                if w>wmax: wmax = w
-                if raw_line.strip():
-                    if line.endswith('\\l'):
-                        def cmd(img=img, y=hmax):
-                            img.draw(xleft, ytop+y)
-                    elif line.endswith('\r'):
-                        def cmd(img=img, y=hmax, w=w):
-                            img.draw(xright-w, ytop+y)
-                    else:
-                        def cmd(img=img, y=hmax, w=w):
-                            img.draw(xcenter-w//2, ytop+y)
+                if '\f' in raw_line:   # grayed out parts of the line
+                    imgs = []
+                    graytext = True
+                    h = 16
+                    w_total = 0
+                    for linepart in raw_line.split('\f'):
+                        graytext = not graytext
+                        if not linepart.strip():
+                            continue
+                        if graytext:
+                            fgcolor = (128, 160, 160)
+                        else:
+                            fgcolor = (0, 0, 0)
+                        img = TextSnippet(self, linepart, fgcolor, bgcolor)
+                        imgs.append((w_total, img))
+                        w, h = img.get_size()
+                        w_total += w
+                    if w_total > wmax: wmax = w_total
+                    def cmd(imgs=imgs, y=hmax):
+                        for x, img in imgs:
+                            img.draw(xleft+x, ytop+y)
                     commands.append(cmd)
+                else:
+                    img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
+                    w, h = img.get_size()
+                    if w>wmax: wmax = w
+                    if raw_line.strip():
+                        if line.endswith('\\l'):
+                            def cmd(img=img, y=hmax):
+                                img.draw(xleft, ytop+y)
+                        elif line.endswith('\r'):
+                            def cmd(img=img, y=hmax, w=w):
+                                img.draw(xright-w, ytop+y)
+                        else:
+                            def cmd(img=img, y=hmax, w=w):
+                                img.draw(xcenter-w//2, ytop+y)
+                        commands.append(cmd)
                 hmax += h
                 #hmax += 8
 

Modified: pypy/branch/leak-finder/pypy/jit/metainterp/graphpage.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/metainterp/graphpage.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/metainterp/graphpage.py	Sat Oct 16 16:40:50 2010
@@ -153,7 +153,7 @@
         opindex = opstartindex
         while True:
             op = operations[opindex]
-            lines.append(repr(op))
+            lines.append(op.repr(graytext=True))
             if is_interesting_guard(op):
                 tgt = op.getdescr()._debug_suboperations[0]
                 tgt_g, tgt_i = self.all_operations[tgt]

Modified: pypy/branch/leak-finder/pypy/jit/metainterp/resoperation.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/metainterp/resoperation.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/metainterp/resoperation.py	Sat Oct 16 16:40:50 2010
@@ -93,7 +93,7 @@
     def __repr__(self):
         return self.repr()
 
-    def repr(self):
+    def repr(self, graytext=False):
         # RPython-friendly version
         if self.result is not None:
             sres = '%s = ' % (self.result,)
@@ -101,6 +101,8 @@
             sres = ''
         if self.name:
             prefix = "%s:%s   " % (self.name, self.pc)
+            if graytext:
+                prefix = "\f%s\f" % prefix
         else:
             prefix = ""
         args = self.getarglist()


From arigo at codespeak.net  Sat Oct 16 16:44:49 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 16:44:49 +0200 (CEST)
Subject: [pypy-svn] r78016 - pypy/branch/leak-finder/pypy/jit/codewriter/test
Message-ID: <20101016144449.A53AF36E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 16:44:48 2010
New Revision: 78016

Modified:
   pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py
Log:
An extra test.


Modified: pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py	Sat Oct 16 16:44:48 2010
@@ -687,6 +687,38 @@
     assert block.operations[1].result is None
     assert block.exits[0].args == [v1]
 
+def test_jit_merge_point_1():
+    class FakeJitDriverSD:
+        index = 42
+        class jitdriver:
+            greens = ['green1', 'green2']
+            reds = ['red1', 'red2']
+    jd = FakeJitDriverSD()
+    v1 = varoftype(lltype.Signed)
+    v2 = varoftype(lltype.Signed)
+    v3 = varoftype(lltype.Signed)
+    v4 = varoftype(lltype.Signed)
+    v5 = varoftype(lltype.Void)
+    op = SpaceOperation('jit_marker',
+                        [Constant('jit_merge_point', lltype.Void),
+                         Constant(jd.jitdriver, lltype.Void),
+                         v1, v2, v3, v4], v5)
+    tr = Transformer()
+    tr.portal_jd = jd
+    oplist = tr.rewrite_operation(op)
+    assert len(oplist) == 6
+    assert oplist[0].opname == '-live-'
+    assert oplist[1].opname == 'int_guard_value'
+    assert oplist[1].args   == [v1]
+    assert oplist[2].opname == '-live-'
+    assert oplist[3].opname == 'int_guard_value'
+    assert oplist[3].args   == [v2]
+    assert oplist[4].opname == 'jit_merge_point'
+    assert oplist[4].args[0].value == 42
+    assert list(oplist[4].args[1]) == [v1, v2]
+    assert list(oplist[4].args[4]) == [v3, v4]
+    assert oplist[5].opname == '-live-'
+
 def test_int_abs():
     v1 = varoftype(lltype.Signed)
     v2 = varoftype(lltype.Signed)


From arigo at codespeak.net  Sat Oct 16 17:01:32 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 17:01:32 +0200 (CEST)
Subject: [pypy-svn] r78017 - pypy/branch/leak-finder/pypy/jit/codewriter/test
Message-ID: <20101016150132.2D65836E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 17:01:30 2010
New Revision: 78017

Modified:
   pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py
Log:
Add voids.


Modified: pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py	Sat Oct 16 17:01:30 2010
@@ -691,18 +691,20 @@
     class FakeJitDriverSD:
         index = 42
         class jitdriver:
-            greens = ['green1', 'green2']
-            reds = ['red1', 'red2']
+            greens = ['green1', 'green2', 'voidgreen3']
+            reds = ['red1', 'red2', 'voidred3']
     jd = FakeJitDriverSD()
     v1 = varoftype(lltype.Signed)
     v2 = varoftype(lltype.Signed)
+    vvoid1 = varoftype(lltype.Void)
     v3 = varoftype(lltype.Signed)
     v4 = varoftype(lltype.Signed)
+    vvoid2 = varoftype(lltype.Void)
     v5 = varoftype(lltype.Void)
     op = SpaceOperation('jit_marker',
                         [Constant('jit_merge_point', lltype.Void),
                          Constant(jd.jitdriver, lltype.Void),
-                         v1, v2, v3, v4], v5)
+                         v1, v2, vvoid1, v3, v4, vvoid2], v5)
     tr = Transformer()
     tr.portal_jd = jd
     oplist = tr.rewrite_operation(op)


From arigo at codespeak.net  Sat Oct 16 17:46:02 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 17:46:02 +0200 (CEST)
Subject: [pypy-svn] r78018 - pypy/branch/rsre-jit
Message-ID: <20101016154602.86E2E36E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 17:46:00 2010
New Revision: 78018

Added:
   pypy/branch/rsre-jit/
      - copied from r78017, pypy/trunk/
Log:
A branch in which to play again with JITting the _sre module.



From arigo at codespeak.net  Sat Oct 16 17:47:50 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 17:47:50 +0200 (CEST)
Subject: [pypy-svn] r78019 - in pypy/branch/leak-finder: dotviewer
	pypy/jit/codewriter/test pypy/jit/metainterp pypy/rlib/rsre
	pypy/rlib/rsre/test
Message-ID: <20101016154750.789C736E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 17:47:48 2010
New Revision: 78019

Removed:
   pypy/branch/leak-finder/pypy/rlib/rsre/test/conftest.py
   pypy/branch/leak-finder/pypy/rlib/rsre/test/test_zjit.py
Modified:
   pypy/branch/leak-finder/dotviewer/drawgraph.py
   pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py
   pypy/branch/leak-finder/pypy/jit/metainterp/graphpage.py
   pypy/branch/leak-finder/pypy/jit/metainterp/resoperation.py
   pypy/branch/leak-finder/pypy/jit/metainterp/warmspot.py
   pypy/branch/leak-finder/pypy/rlib/rsre/rsre_core.py
Log:
Oups!  Wrong branch.  Thanks agaynor.


Modified: pypy/branch/leak-finder/dotviewer/drawgraph.py
==============================================================================
--- pypy/branch/leak-finder/dotviewer/drawgraph.py	(original)
+++ pypy/branch/leak-finder/dotviewer/drawgraph.py	Sat Oct 16 17:47:48 2010
@@ -423,43 +423,20 @@
         else:
             for line in lines:
                 raw_line = line.replace('\\l','').replace('\r','') or ' '
-                if '\f' in raw_line:   # grayed out parts of the line
-                    imgs = []
-                    graytext = True
-                    h = 16
-                    w_total = 0
-                    for linepart in raw_line.split('\f'):
-                        graytext = not graytext
-                        if not linepart.strip():
-                            continue
-                        if graytext:
-                            fgcolor = (128, 160, 160)
-                        else:
-                            fgcolor = (0, 0, 0)
-                        img = TextSnippet(self, linepart, fgcolor, bgcolor)
-                        imgs.append((w_total, img))
-                        w, h = img.get_size()
-                        w_total += w
-                    if w_total > wmax: wmax = w_total
-                    def cmd(imgs=imgs, y=hmax):
-                        for x, img in imgs:
-                            img.draw(xleft+x, ytop+y)
+                img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
+                w, h = img.get_size()
+                if w>wmax: wmax = w
+                if raw_line.strip():
+                    if line.endswith('\\l'):
+                        def cmd(img=img, y=hmax):
+                            img.draw(xleft, ytop+y)
+                    elif line.endswith('\r'):
+                        def cmd(img=img, y=hmax, w=w):
+                            img.draw(xright-w, ytop+y)
+                    else:
+                        def cmd(img=img, y=hmax, w=w):
+                            img.draw(xcenter-w//2, ytop+y)
                     commands.append(cmd)
-                else:
-                    img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
-                    w, h = img.get_size()
-                    if w>wmax: wmax = w
-                    if raw_line.strip():
-                        if line.endswith('\\l'):
-                            def cmd(img=img, y=hmax):
-                                img.draw(xleft, ytop+y)
-                        elif line.endswith('\r'):
-                            def cmd(img=img, y=hmax, w=w):
-                                img.draw(xright-w, ytop+y)
-                        else:
-                            def cmd(img=img, y=hmax, w=w):
-                                img.draw(xcenter-w//2, ytop+y)
-                        commands.append(cmd)
                 hmax += h
                 #hmax += 8
 

Modified: pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/codewriter/test/test_jtransform.py	Sat Oct 16 17:47:48 2010
@@ -687,40 +687,6 @@
     assert block.operations[1].result is None
     assert block.exits[0].args == [v1]
 
-def test_jit_merge_point_1():
-    class FakeJitDriverSD:
-        index = 42
-        class jitdriver:
-            greens = ['green1', 'green2', 'voidgreen3']
-            reds = ['red1', 'red2', 'voidred3']
-    jd = FakeJitDriverSD()
-    v1 = varoftype(lltype.Signed)
-    v2 = varoftype(lltype.Signed)
-    vvoid1 = varoftype(lltype.Void)
-    v3 = varoftype(lltype.Signed)
-    v4 = varoftype(lltype.Signed)
-    vvoid2 = varoftype(lltype.Void)
-    v5 = varoftype(lltype.Void)
-    op = SpaceOperation('jit_marker',
-                        [Constant('jit_merge_point', lltype.Void),
-                         Constant(jd.jitdriver, lltype.Void),
-                         v1, v2, vvoid1, v3, v4, vvoid2], v5)
-    tr = Transformer()
-    tr.portal_jd = jd
-    oplist = tr.rewrite_operation(op)
-    assert len(oplist) == 6
-    assert oplist[0].opname == '-live-'
-    assert oplist[1].opname == 'int_guard_value'
-    assert oplist[1].args   == [v1]
-    assert oplist[2].opname == '-live-'
-    assert oplist[3].opname == 'int_guard_value'
-    assert oplist[3].args   == [v2]
-    assert oplist[4].opname == 'jit_merge_point'
-    assert oplist[4].args[0].value == 42
-    assert list(oplist[4].args[1]) == [v1, v2]
-    assert list(oplist[4].args[4]) == [v3, v4]
-    assert oplist[5].opname == '-live-'
-
 def test_int_abs():
     v1 = varoftype(lltype.Signed)
     v2 = varoftype(lltype.Signed)

Modified: pypy/branch/leak-finder/pypy/jit/metainterp/graphpage.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/metainterp/graphpage.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/metainterp/graphpage.py	Sat Oct 16 17:47:48 2010
@@ -153,7 +153,7 @@
         opindex = opstartindex
         while True:
             op = operations[opindex]
-            lines.append(op.repr(graytext=True))
+            lines.append(repr(op))
             if is_interesting_guard(op):
                 tgt = op.getdescr()._debug_suboperations[0]
                 tgt_g, tgt_i = self.all_operations[tgt]

Modified: pypy/branch/leak-finder/pypy/jit/metainterp/resoperation.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/metainterp/resoperation.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/metainterp/resoperation.py	Sat Oct 16 17:47:48 2010
@@ -93,7 +93,7 @@
     def __repr__(self):
         return self.repr()
 
-    def repr(self, graytext=False):
+    def repr(self):
         # RPython-friendly version
         if self.result is not None:
             sres = '%s = ' % (self.result,)
@@ -101,8 +101,6 @@
             sres = ''
         if self.name:
             prefix = "%s:%s   " % (self.name, self.pc)
-            if graytext:
-                prefix = "\f%s\f" % prefix
         else:
             prefix = ""
         args = self.getarglist()

Modified: pypy/branch/leak-finder/pypy/jit/metainterp/warmspot.py
==============================================================================
--- pypy/branch/leak-finder/pypy/jit/metainterp/warmspot.py	(original)
+++ pypy/branch/leak-finder/pypy/jit/metainterp/warmspot.py	Sat Oct 16 17:47:48 2010
@@ -211,9 +211,9 @@
                 "there are multiple jit_merge_points with the same jitdriver"
 
     def split_graph_and_record_jitdriver(self, graph, block, pos):
-        op = block.operations[pos]
         jd = JitDriverStaticData()
-        jd._jit_merge_point_pos = (graph, op)
+        jd._jit_merge_point_pos = (graph, block, pos)
+        op = block.operations[pos]
         args = op.args[2:]
         s_binding = self.translator.annotator.binding
         jd._portal_args_s = [s_binding(v) for v in args]
@@ -457,7 +457,8 @@
             self.make_args_specification(jd)
 
     def make_args_specification(self, jd):
-        graph, op = jd._jit_merge_point_pos
+        graph, block, index = jd._jit_merge_point_pos
+        op = block.operations[index]
         greens_v, reds_v = support.decode_hp_hint_args(op)
         ALLARGS = [v.concretetype for v in (greens_v + reds_v)]
         jd._green_args_spec = [v.concretetype for v in greens_v]
@@ -708,14 +709,8 @@
         # ____________________________________________________________
         # Now mutate origportalgraph to end with a call to portal_runner_ptr
         #
-        _, op = jd._jit_merge_point_pos
-        for origblock in origportalgraph.iterblocks():
-            if op in origblock.operations:
-                break
-        else:
-            assert False, "lost the operation %r in the graph %r" % (
-                op, origportalgraph)
-        origindex = origblock.operations.index(op)
+        _, origblock, origindex = jd._jit_merge_point_pos
+        op = origblock.operations[origindex]
         assert op.opname == 'jit_marker'
         assert op.args[0].value == 'jit_merge_point'
         greens_v, reds_v = support.decode_hp_hint_args(op)

Modified: pypy/branch/leak-finder/pypy/rlib/rsre/rsre_core.py
==============================================================================
--- pypy/branch/leak-finder/pypy/rlib/rsre/rsre_core.py	(original)
+++ pypy/branch/leak-finder/pypy/rlib/rsre/rsre_core.py	Sat Oct 16 17:47:48 2010
@@ -4,7 +4,6 @@
 from pypy.rlib.rsre import rsre_char
 from pypy.tool.sourcetools import func_with_new_name
 from pypy.rlib.objectmodel import we_are_translated
-from pypy.rlib import jit
 
 
 OPCODE_FAILURE            = 0
@@ -57,18 +56,15 @@
     _seen_specname[specname] = True
     # Install a copy of the function under the name '_spec_funcname' in each
     # concrete subclass
-    specialized_methods = []
     for prefix, concreteclass in [('str', StrMatchContext),
                                   ('uni', UnicodeMatchContext)]:
         newfunc = func_with_new_name(func, prefix + specname)
         assert not hasattr(concreteclass, specname)
         setattr(concreteclass, specname, newfunc)
-        specialized_methods.append(newfunc)
     # Return a dispatcher function, specialized on the exact type of 'ctx'
     def dispatch(ctx, *args):
         return getattr(ctx, specname)(*args)
     dispatch._annspecialcase_ = 'specialize:argtype(0)'
-    dispatch._specialized_methods_ = specialized_methods
     return dispatch
 
 # ____________________________________________________________
@@ -79,7 +75,6 @@
 
 class AbstractMatchContext(object):
     """Abstract base class"""
-    _immutable_fields_ = ['pattern[*]', 'flags']
     match_start = 0
     match_end = 0
     match_marks = None
@@ -243,7 +238,6 @@
         self.start_ptr = ptr
         self.start_marks = marks
 
-    @jit.unroll_safe
     def find_first_result(self, ctx):
         ppos = self.ppos
         while ctx.pat(ppos):
@@ -256,8 +250,6 @@
     find_next_result = find_first_result
 
 class RepeatOneMatchResult(MatchResult):
-    jitdriver = jit.JitDriver(greens=['nextppos', 'pattern'],
-                              reds=['ptr', 'self', 'ctx'])
 
     def __init__(self, nextppos, minptr, ptr, marks):
         self.nextppos = nextppos
@@ -267,18 +259,8 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
-        nextppos = self.nextppos
         while ptr >= self.minptr:
-            #
-            pattern = ctx.pattern
-            self.jitdriver.can_enter_jit(self=self, ptr=ptr, ctx=ctx,
-                                         nextppos=nextppos, pattern=pattern)
-            self.jitdriver.jit_merge_point(self=self, ptr=ptr, ctx=ctx,
-                                           nextppos=nextppos, pattern=pattern)
-            if jit.we_are_jitted():
-                ctx.pattern = pattern
-            #
-            result = sre_match(ctx, nextppos, ptr, self.start_marks)
+            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
             ptr -= 1
             if result is not None:
                 self.subresult = result
@@ -288,8 +270,6 @@
 
 
 class MinRepeatOneMatchResult(MatchResult):
-    jitdriver = jit.JitDriver(greens=['nextppos', 'ppos3', 'pattern'],
-                              reds=['ptr', 'self', 'ctx'])
 
     def __init__(self, nextppos, ppos3, maxptr, ptr, marks):
         self.nextppos = nextppos
@@ -300,39 +280,27 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
-        nextppos = self.nextppos
-        ppos3 = self.ppos3
         while ptr <= self.maxptr:
-            #
-            pattern = ctx.pattern
-            self.jitdriver.can_enter_jit(self=self, ptr=ptr, ctx=ctx,
-                                         nextppos=nextppos, pattern=pattern,
-                                         ppos3=ppos3)
-            self.jitdriver.jit_merge_point(self=self, ptr=ptr, ctx=ctx,
-                                           nextppos=nextppos, pattern=pattern,
-                                           ppos3=ppos3)
-            if jit.we_are_jitted():
-                ctx.pattern = pattern
-            #
-            result = sre_match(ctx, nextppos, ptr, self.start_marks)
+            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
             if result is not None:
                 self.subresult = result
                 self.start_ptr = ptr
                 return self
-            if not self.next_char_ok(ctx, ptr, ppos3):
+            if not self.next_char_ok(ctx, ptr):
                 break
             ptr += 1
 
     def find_next_result(self, ctx):
         ptr = self.start_ptr
-        if not self.next_char_ok(ctx, ptr, self.ppos3):
+        if not self.next_char_ok(ctx, ptr):
             return
         self.start_ptr = ptr + 1
         return self.find_first_result(ctx)
 
-    def next_char_ok(self, ctx, ptr, ppos):
+    def next_char_ok(self, ctx, ptr):
         if ptr == ctx.end:
             return False
+        ppos = self.ppos3
         op = ctx.pat(ppos)
         for op1, (checkerfn, _) in unroll_char_checker:
             if op1 == op:
@@ -461,7 +429,6 @@
 # ____________________________________________________________
 
 @specializectx
- at jit.unroll_safe
 def sre_match(ctx, ppos, ptr, marks):
     """Returns either None or a MatchResult object.  Usually we only need
     the first result, but there is the case of REPEAT...UNTIL where we


From arigo at codespeak.net  Sat Oct 16 17:49:59 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 17:49:59 +0200 (CEST)
Subject: [pypy-svn] r78020 - in pypy/branch/rsre-jit/pypy/rlib/rsre: . test
Message-ID: <20101016154959.C595136E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 17:49:58 2010
New Revision: 78020

Added:
   pypy/branch/rsre-jit/pypy/rlib/rsre/test/conftest.py
      - copied unchanged from r78013, pypy/branch/leak-finder/pypy/rlib/rsre/test/conftest.py
   pypy/branch/rsre-jit/pypy/rlib/rsre/test/test_zjit.py
      - copied unchanged from r78013, pypy/branch/leak-finder/pypy/rlib/rsre/test/test_zjit.py
Modified:
   pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_core.py
Log:
In-progress.  Start with some tests.


Modified: pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_core.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_core.py	(original)
+++ pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_core.py	Sat Oct 16 17:49:58 2010
@@ -4,6 +4,7 @@
 from pypy.rlib.rsre import rsre_char
 from pypy.tool.sourcetools import func_with_new_name
 from pypy.rlib.objectmodel import we_are_translated
+from pypy.rlib import jit
 
 
 OPCODE_FAILURE            = 0
@@ -56,15 +57,18 @@
     _seen_specname[specname] = True
     # Install a copy of the function under the name '_spec_funcname' in each
     # concrete subclass
+    specialized_methods = []
     for prefix, concreteclass in [('str', StrMatchContext),
                                   ('uni', UnicodeMatchContext)]:
         newfunc = func_with_new_name(func, prefix + specname)
         assert not hasattr(concreteclass, specname)
         setattr(concreteclass, specname, newfunc)
+        specialized_methods.append(newfunc)
     # Return a dispatcher function, specialized on the exact type of 'ctx'
     def dispatch(ctx, *args):
         return getattr(ctx, specname)(*args)
     dispatch._annspecialcase_ = 'specialize:argtype(0)'
+    dispatch._specialized_methods_ = specialized_methods
     return dispatch
 
 # ____________________________________________________________
@@ -75,6 +79,7 @@
 
 class AbstractMatchContext(object):
     """Abstract base class"""
+    _immutable_fields_ = ['pattern[*]', 'flags']
     match_start = 0
     match_end = 0
     match_marks = None
@@ -238,6 +243,7 @@
         self.start_ptr = ptr
         self.start_marks = marks
 
+    @jit.unroll_safe
     def find_first_result(self, ctx):
         ppos = self.ppos
         while ctx.pat(ppos):
@@ -250,6 +256,8 @@
     find_next_result = find_first_result
 
 class RepeatOneMatchResult(MatchResult):
+    jitdriver = jit.JitDriver(greens=['nextppos', 'pattern'],
+                              reds=['ptr', 'self', 'ctx'])
 
     def __init__(self, nextppos, minptr, ptr, marks):
         self.nextppos = nextppos
@@ -259,8 +267,18 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
+        nextppos = self.nextppos
         while ptr >= self.minptr:
-            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
+            #
+            pattern = ctx.pattern
+            self.jitdriver.can_enter_jit(self=self, ptr=ptr, ctx=ctx,
+                                         nextppos=nextppos, pattern=pattern)
+            self.jitdriver.jit_merge_point(self=self, ptr=ptr, ctx=ctx,
+                                           nextppos=nextppos, pattern=pattern)
+            if jit.we_are_jitted():
+                ctx.pattern = pattern
+            #
+            result = sre_match(ctx, nextppos, ptr, self.start_marks)
             ptr -= 1
             if result is not None:
                 self.subresult = result
@@ -270,6 +288,8 @@
 
 
 class MinRepeatOneMatchResult(MatchResult):
+    jitdriver = jit.JitDriver(greens=['nextppos', 'ppos3', 'pattern'],
+                              reds=['ptr', 'self', 'ctx'])
 
     def __init__(self, nextppos, ppos3, maxptr, ptr, marks):
         self.nextppos = nextppos
@@ -280,27 +300,39 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
+        nextppos = self.nextppos
+        ppos3 = self.ppos3
         while ptr <= self.maxptr:
-            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
+            #
+            pattern = ctx.pattern
+            self.jitdriver.can_enter_jit(self=self, ptr=ptr, ctx=ctx,
+                                         nextppos=nextppos, pattern=pattern,
+                                         ppos3=ppos3)
+            self.jitdriver.jit_merge_point(self=self, ptr=ptr, ctx=ctx,
+                                           nextppos=nextppos, pattern=pattern,
+                                           ppos3=ppos3)
+            if jit.we_are_jitted():
+                ctx.pattern = pattern
+            #
+            result = sre_match(ctx, nextppos, ptr, self.start_marks)
             if result is not None:
                 self.subresult = result
                 self.start_ptr = ptr
                 return self
-            if not self.next_char_ok(ctx, ptr):
+            if not self.next_char_ok(ctx, ptr, ppos3):
                 break
             ptr += 1
 
     def find_next_result(self, ctx):
         ptr = self.start_ptr
-        if not self.next_char_ok(ctx, ptr):
+        if not self.next_char_ok(ctx, ptr, self.ppos3):
             return
         self.start_ptr = ptr + 1
         return self.find_first_result(ctx)
 
-    def next_char_ok(self, ctx, ptr):
+    def next_char_ok(self, ctx, ptr, ppos):
         if ptr == ctx.end:
             return False
-        ppos = self.ppos3
         op = ctx.pat(ppos)
         for op1, (checkerfn, _) in unroll_char_checker:
             if op1 == op:
@@ -429,6 +461,7 @@
 # ____________________________________________________________
 
 @specializectx
+ at jit.unroll_safe
 def sre_match(ctx, ppos, ptr, marks):
     """Returns either None or a MatchResult object.  Usually we only need
     the first result, but there is the case of REPEAT...UNTIL where we


From arigo at codespeak.net  Sat Oct 16 17:50:30 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 17:50:30 +0200 (CEST)
Subject: [pypy-svn] r78021 - pypy/branch/rsre-jit/pypy/jit/metainterp
Message-ID: <20101016155030.14B6A36E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 17:50:28 2010
New Revision: 78021

Modified:
   pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py
Log:
Bug fix: backend optimizations may remove some operations and invalidate
the 'index' of an existing operation inside its block.


Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py	Sat Oct 16 17:50:28 2010
@@ -211,9 +211,9 @@
                 "there are multiple jit_merge_points with the same jitdriver"
 
     def split_graph_and_record_jitdriver(self, graph, block, pos):
-        jd = JitDriverStaticData()
-        jd._jit_merge_point_pos = (graph, block, pos)
         op = block.operations[pos]
+        jd = JitDriverStaticData()
+        jd._jit_merge_point_pos = (graph, op)
         args = op.args[2:]
         s_binding = self.translator.annotator.binding
         jd._portal_args_s = [s_binding(v) for v in args]
@@ -457,8 +457,7 @@
             self.make_args_specification(jd)
 
     def make_args_specification(self, jd):
-        graph, block, index = jd._jit_merge_point_pos
-        op = block.operations[index]
+        graph, op = jd._jit_merge_point_pos
         greens_v, reds_v = support.decode_hp_hint_args(op)
         ALLARGS = [v.concretetype for v in (greens_v + reds_v)]
         jd._green_args_spec = [v.concretetype for v in greens_v]
@@ -709,8 +708,14 @@
         # ____________________________________________________________
         # Now mutate origportalgraph to end with a call to portal_runner_ptr
         #
-        _, origblock, origindex = jd._jit_merge_point_pos
-        op = origblock.operations[origindex]
+        _, op = jd._jit_merge_point_pos
+        for origblock in origportalgraph.iterblocks():
+            if op in origblock.operations:
+                break
+        else:
+            assert False, "lost the operation %r in the graph %r" % (
+                op, origportalgraph)
+        origindex = origblock.operations.index(op)
         assert op.opname == 'jit_marker'
         assert op.args[0].value == 'jit_merge_point'
         greens_v, reds_v = support.decode_hp_hint_args(op)


From arigo at codespeak.net  Sat Oct 16 17:50:59 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 17:50:59 +0200 (CEST)
Subject: [pypy-svn] r78022 - in pypy/branch/rsre-jit: dotviewer
	pypy/jit/metainterp
Message-ID: <20101016155059.CF96B36E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 17:50:58 2010
New Revision: 78022

Modified:
   pypy/branch/rsre-jit/dotviewer/drawgraph.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/graphpage.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/resoperation.py
Log:
Improve the readability of the JIT flow graphs by changing the color of
the prefix to the operations (the name of where it comes from).


Modified: pypy/branch/rsre-jit/dotviewer/drawgraph.py
==============================================================================
--- pypy/branch/rsre-jit/dotviewer/drawgraph.py	(original)
+++ pypy/branch/rsre-jit/dotviewer/drawgraph.py	Sat Oct 16 17:50:58 2010
@@ -423,20 +423,43 @@
         else:
             for line in lines:
                 raw_line = line.replace('\\l','').replace('\r','') or ' '
-                img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
-                w, h = img.get_size()
-                if w>wmax: wmax = w
-                if raw_line.strip():
-                    if line.endswith('\\l'):
-                        def cmd(img=img, y=hmax):
-                            img.draw(xleft, ytop+y)
-                    elif line.endswith('\r'):
-                        def cmd(img=img, y=hmax, w=w):
-                            img.draw(xright-w, ytop+y)
-                    else:
-                        def cmd(img=img, y=hmax, w=w):
-                            img.draw(xcenter-w//2, ytop+y)
+                if '\f' in raw_line:   # grayed out parts of the line
+                    imgs = []
+                    graytext = True
+                    h = 16
+                    w_total = 0
+                    for linepart in raw_line.split('\f'):
+                        graytext = not graytext
+                        if not linepart.strip():
+                            continue
+                        if graytext:
+                            fgcolor = (128, 160, 160)
+                        else:
+                            fgcolor = (0, 0, 0)
+                        img = TextSnippet(self, linepart, fgcolor, bgcolor)
+                        imgs.append((w_total, img))
+                        w, h = img.get_size()
+                        w_total += w
+                    if w_total > wmax: wmax = w_total
+                    def cmd(imgs=imgs, y=hmax):
+                        for x, img in imgs:
+                            img.draw(xleft+x, ytop+y)
                     commands.append(cmd)
+                else:
+                    img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
+                    w, h = img.get_size()
+                    if w>wmax: wmax = w
+                    if raw_line.strip():
+                        if line.endswith('\\l'):
+                            def cmd(img=img, y=hmax):
+                                img.draw(xleft, ytop+y)
+                        elif line.endswith('\r'):
+                            def cmd(img=img, y=hmax, w=w):
+                                img.draw(xright-w, ytop+y)
+                        else:
+                            def cmd(img=img, y=hmax, w=w):
+                                img.draw(xcenter-w//2, ytop+y)
+                        commands.append(cmd)
                 hmax += h
                 #hmax += 8
 

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/graphpage.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/graphpage.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/graphpage.py	Sat Oct 16 17:50:58 2010
@@ -153,7 +153,7 @@
         opindex = opstartindex
         while True:
             op = operations[opindex]
-            lines.append(repr(op))
+            lines.append(op.repr(graytext=True))
             if is_interesting_guard(op):
                 tgt = op.getdescr()._debug_suboperations[0]
                 tgt_g, tgt_i = self.all_operations[tgt]

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/resoperation.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/resoperation.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/resoperation.py	Sat Oct 16 17:50:58 2010
@@ -93,7 +93,7 @@
     def __repr__(self):
         return self.repr()
 
-    def repr(self):
+    def repr(self, graytext=False):
         # RPython-friendly version
         if self.result is not None:
             sres = '%s = ' % (self.result,)
@@ -101,6 +101,8 @@
             sres = ''
         if self.name:
             prefix = "%s:%s   " % (self.name, self.pc)
+            if graytext:
+                prefix = "\f%s\f" % prefix
         else:
             prefix = ""
         args = self.getarglist()


From arigo at codespeak.net  Sat Oct 16 17:51:23 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 17:51:23 +0200 (CEST)
Subject: [pypy-svn] r78023 - pypy/branch/rsre-jit/pypy/jit/codewriter/test
Message-ID: <20101016155123.0C37A36E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 17:51:21 2010
New Revision: 78023

Modified:
   pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_jtransform.py
Log:
Add an extra test, including voids.


Modified: pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_jtransform.py	Sat Oct 16 17:51:21 2010
@@ -687,6 +687,40 @@
     assert block.operations[1].result is None
     assert block.exits[0].args == [v1]
 
+def test_jit_merge_point_1():
+    class FakeJitDriverSD:
+        index = 42
+        class jitdriver:
+            greens = ['green1', 'green2', 'voidgreen3']
+            reds = ['red1', 'red2', 'voidred3']
+    jd = FakeJitDriverSD()
+    v1 = varoftype(lltype.Signed)
+    v2 = varoftype(lltype.Signed)
+    vvoid1 = varoftype(lltype.Void)
+    v3 = varoftype(lltype.Signed)
+    v4 = varoftype(lltype.Signed)
+    vvoid2 = varoftype(lltype.Void)
+    v5 = varoftype(lltype.Void)
+    op = SpaceOperation('jit_marker',
+                        [Constant('jit_merge_point', lltype.Void),
+                         Constant(jd.jitdriver, lltype.Void),
+                         v1, v2, vvoid1, v3, v4, vvoid2], v5)
+    tr = Transformer()
+    tr.portal_jd = jd
+    oplist = tr.rewrite_operation(op)
+    assert len(oplist) == 6
+    assert oplist[0].opname == '-live-'
+    assert oplist[1].opname == 'int_guard_value'
+    assert oplist[1].args   == [v1]
+    assert oplist[2].opname == '-live-'
+    assert oplist[3].opname == 'int_guard_value'
+    assert oplist[3].args   == [v2]
+    assert oplist[4].opname == 'jit_merge_point'
+    assert oplist[4].args[0].value == 42
+    assert list(oplist[4].args[1]) == [v1, v2]
+    assert list(oplist[4].args[4]) == [v3, v4]
+    assert oplist[5].opname == '-live-'
+
 def test_int_abs():
     v1 = varoftype(lltype.Signed)
     v2 = varoftype(lltype.Signed)


From arigo at codespeak.net  Sat Oct 16 18:17:01 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 18:17:01 +0200 (CEST)
Subject: [pypy-svn] r78024 - in pypy/branch/rsre-jit/pypy/rlib: . test
Message-ID: <20101016161701.4DDC036E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 18:16:59 2010
New Revision: 78024

Modified:
   pypy/branch/rsre-jit/pypy/rlib/jit.py
   pypy/branch/rsre-jit/pypy/rlib/test/test_jit.py
Log:
Test showing the final goal: letting 'x.foo' be written
in the JitDriver, in the 'greens' list.  Currently expands
as a extra argument in the jit_marker operation.


Modified: pypy/branch/rsre-jit/pypy/rlib/jit.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/rlib/jit.py	(original)
+++ pypy/branch/rsre-jit/pypy/rlib/jit.py	Sat Oct 16 18:16:59 2010
@@ -358,7 +358,8 @@
         driver = self.instance.im_self
         keys = kwds_s.keys()
         keys.sort()
-        expected = ['s_' + name for name in driver.greens + driver.reds]
+        expected = ['s_' + name for name in driver.greens + driver.reds
+                                if '.' not in name]
         expected.sort()
         if keys != expected:
             raise JitHintError("%s expects the following keyword "
@@ -403,7 +404,13 @@
         uniquekey = 'jitdriver.%s' % func.func_name
         args_s = args_s[:]
         for name in variables:
-            s_arg = kwds_s['s_' + name]
+            if '.' not in name:
+                s_arg = kwds_s['s_' + name]
+            else:
+                objname, fieldname = name.split('.')
+                s_instance = kwds_s['s_' + objname]
+                s_arg = s_instance.classdef.about_attribute(fieldname)
+                assert s_arg is not None
             args_s.append(s_arg)
         bk.emulate_pbc_call(uniquekey, s_func, args_s)
 
@@ -416,9 +423,33 @@
         greens_v = []
         reds_v = []
         for name in driver.greens:
-            i = kwds_i['i_' + name]
-            r_green = hop.args_r[i]
-            v_green = hop.inputarg(r_green, arg=i)
+            if '.' not in name:
+                i = kwds_i['i_' + name]
+                r_green = hop.args_r[i]
+                v_green = hop.inputarg(r_green, arg=i)
+            else:
+                if hop.rtyper.type_system.name == 'ootypesystem':
+                    py.test.skip("lltype only")
+                objname, fieldname = name.split('.')   # see test_green_field
+                assert objname in driver.reds
+                i = kwds_i['i_' + objname]
+                r_red = hop.args_r[i]
+                while True:
+                    try:
+                        mangled_name, r_field = r_red._get_field(fieldname)
+                        break
+                    except KeyError:
+                        pass
+                    assert r_red.rbase is not None, (
+                        "field %r not found in %r" % (name,
+                                                      r_red.lowleveltype.TO))
+                    r_red = r_red.rbase
+                assert r_red.lowleveltype.TO._immutable_field(mangled_name), (
+                    "field %r must be declared as immutable" % name)
+                v_red = hop.inputarg(r_red, arg=i)
+                c_llname = hop.inputconst(lltype.Void, mangled_name)
+                v_green = hop.genop('getfield', [v_red, c_llname],
+                                    resulttype = r_field)
             greens_v.append(v_green)
         for name in driver.reds:
             i = kwds_i['i_' + name]

Modified: pypy/branch/rsre-jit/pypy/rlib/test/test_jit.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/rlib/test/test_jit.py	(original)
+++ pypy/branch/rsre-jit/pypy/rlib/test/test_jit.py	Sat Oct 16 18:16:59 2010
@@ -1,4 +1,5 @@
 import py
+from pypy.conftest import option
 from pypy.rlib.jit import hint, we_are_jitted, JitDriver, purefunction_promote
 from pypy.rlib.jit import JitHintError
 from pypy.translator.translator import TranslationContext, graphof
@@ -104,6 +105,26 @@
             return n
         py.test.raises(JitHintError, self.gengraph, fn, [int])
 
+    def test_green_field(self):
+        def get_printable_location(xfoo):
+            return str(ord(xfoo))   # xfoo must be annotated as a character
+        myjitdriver = JitDriver(greens=['x.foo'], reds=['n', 'x'],
+                                get_printable_location=get_printable_location)
+        class A(object):
+            _immutable_fields_ = ['foo']
+        def fn(n):
+            x = A()
+            x.foo = chr(n)
+            while n > 0:
+                myjitdriver.can_enter_jit(x=x, n=n)
+                myjitdriver.jit_merge_point(x=x, n=n)
+                n -= 1
+            return n
+        t = self.gengraph(fn, [int])[0]
+        if option.view:
+            t.view()
+        # assert did not raise
+
 
 class TestJITLLtype(BaseTestJIT, LLRtypeMixin):
     pass


From arigo at codespeak.net  Sat Oct 16 18:24:41 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 18:24:41 +0200 (CEST)
Subject: [pypy-svn] r78025 - in pypy/branch/rsre-jit/pypy:
	jit/metainterp/test rlib
Message-ID: <20101016162441.A295B36E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 18:24:40 2010
New Revision: 78025

Modified:
   pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_immutable.py
   pypy/branch/rsre-jit/pypy/rlib/jit.py
Log:
The first test about it passes.


Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_immutable.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_immutable.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_immutable.py	Sat Oct 16 18:24:40 2010
@@ -1,4 +1,5 @@
 from pypy.jit.metainterp.test.test_basic import LLJitMixin, OOJitMixin
+from pypy.rlib.jit import JitDriver
 
 
 class ImmutableFieldsTests:
@@ -85,6 +86,29 @@
                             int_add=3)
 
 
+    def test_green_field(self):
+        myjitdriver = JitDriver(greens=['ctx.x'], reds=['ctx'])
+        class Ctx(object):
+            _immutable_fields_ = ['x']
+            def __init__(self, x, y):
+                self.x = x
+                self.y = y
+        def f(x, y):
+            ctx = Ctx(x, y)
+            while 1:
+                myjitdriver.can_enter_jit(ctx=ctx)
+                myjitdriver.jit_merge_point(ctx=ctx)
+                ctx.y -= 1
+                if ctx.y < 0:
+                    return ctx.y
+        def g(y):
+            return f(5, y) + f(6, y)
+        #
+        res = self.meta_interp(g, [7])
+        assert res == -2
+        self.check_loop_count(2)
+
+
 class TestLLtypeImmutableFieldsTests(ImmutableFieldsTests, LLJitMixin):
     pass
 

Modified: pypy/branch/rsre-jit/pypy/rlib/jit.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/rlib/jit.py	(original)
+++ pypy/branch/rsre-jit/pypy/rlib/jit.py	Sat Oct 16 18:24:40 2010
@@ -433,6 +433,7 @@
                 objname, fieldname = name.split('.')   # see test_green_field
                 assert objname in driver.reds
                 i = kwds_i['i_' + objname]
+                s_red = hop.args_s[i]
                 r_red = hop.args_r[i]
                 while True:
                     try:
@@ -450,6 +451,9 @@
                 c_llname = hop.inputconst(lltype.Void, mangled_name)
                 v_green = hop.genop('getfield', [v_red, c_llname],
                                     resulttype = r_field)
+                s_green = s_red.classdef.about_attribute(fieldname)
+                assert s_green is not None
+                hop.rtyper.annotator.setbinding(v_green, s_green)
             greens_v.append(v_green)
         for name in driver.reds:
             i = kwds_i['i_' + name]


From arigo at codespeak.net  Sat Oct 16 20:05:30 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 16 Oct 2010 20:05:30 +0200 (CEST)
Subject: [pypy-svn] r78026 - in pypy/branch/rsre-jit/pypy: jit/codewriter
	jit/metainterp jit/metainterp/test rlib
Message-ID: <20101016180530.2D95136E393@codespeak.net>

Author: arigo
Date: Sat Oct 16 20:05:28 2010
New Revision: 78026

Added:
   pypy/branch/rsre-jit/pypy/jit/metainterp/greenfield.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_greenfield.py
Modified:
   pypy/branch/rsre-jit/pypy/jit/codewriter/call.py
   pypy/branch/rsre-jit/pypy/jit/codewriter/jtransform.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/blackhole.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/compile.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/jitdriver.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/pyjitpl.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/resume.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_immutable.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py
   pypy/branch/rsre-jit/pypy/rlib/jit.py
Log:
Implement green fields.  Probably buggy in various ways: I used a single test
to touch all these files...  I will write more tests now.


Modified: pypy/branch/rsre-jit/pypy/jit/codewriter/call.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/codewriter/call.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/codewriter/call.py	Sat Oct 16 20:05:28 2010
@@ -277,3 +277,11 @@
             return seen.pop()
         else:
             return None
+
+    def could_be_green_field(self, GTYPE, fieldname):
+        GTYPE_fieldname = (GTYPE, fieldname)
+        for jd in self.jitdrivers_sd:
+            if jd.greenfield_info is not None:
+                if GTYPE_fieldname in jd.greenfield_info.green_fields:
+                    return True
+        return False

Modified: pypy/branch/rsre-jit/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/codewriter/jtransform.py	Sat Oct 16 20:05:28 2010
@@ -521,7 +521,12 @@
         # check for deepfrozen structures that force constant-folding
         immut = v_inst.concretetype.TO._immutable_field(c_fieldname.value)
         if immut:
-            pure = '_pure'
+            if (self.callcontrol is not None and
+                self.callcontrol.could_be_green_field(v_inst.concretetype.TO,
+                                                      c_fieldname.value)):
+                pure = '_greenfield'
+            else:
+                pure = '_pure'
             if immut == "[*]":
                 self.immutable_arrays[op.result] = True
         else:

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/blackhole.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/blackhole.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/blackhole.py	Sat Oct 16 20:05:28 2010
@@ -1073,6 +1073,10 @@
     bhimpl_getfield_vable_r = bhimpl_getfield_gc_r
     bhimpl_getfield_vable_f = bhimpl_getfield_gc_f
 
+    bhimpl_getfield_gc_i_greenfield = bhimpl_getfield_gc_i
+    bhimpl_getfield_gc_r_greenfield = bhimpl_getfield_gc_r
+    bhimpl_getfield_gc_f_greenfield = bhimpl_getfield_gc_f
+
     @arguments("cpu", "i", "d", returns="i")
     def bhimpl_getfield_raw_i(cpu, struct, fielddescr):
         return cpu.bh_getfield_raw_i(struct, fielddescr)

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/compile.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/compile.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/compile.py	Sat Oct 16 20:05:28 2010
@@ -370,7 +370,8 @@
         from pypy.jit.metainterp.resume import force_from_resumedata
         metainterp_sd = self.metainterp_sd
         vinfo = self.jitdriver_sd.virtualizable_info
-        all_virtuals = force_from_resumedata(metainterp_sd, self, vinfo)
+        ginfo = self.jitdriver_sd.greenfield_info
+        all_virtuals = force_from_resumedata(metainterp_sd, self, vinfo, ginfo)
         # The virtualizable data was stored on the real virtualizable above.
         # Handle all_virtuals: keep them for later blackholing from the
         # future failure of the GUARD_NOT_FORCED

Added: pypy/branch/rsre-jit/pypy/jit/metainterp/greenfield.py
==============================================================================
--- (empty file)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/greenfield.py	Sat Oct 16 20:05:28 2010
@@ -0,0 +1,27 @@
+from pypy.jit.metainterp.typesystem import deref
+
+
+class GreenFieldInfo(object):
+
+    def __init__(self, cpu, jd):
+        self.cpu = cpu
+        self.jitdriver_sd = jd
+        # XXX for now, only supports a single instance,
+        # but several fields of it can be green
+        seen = set()
+        for name in jd.jitdriver.greens:
+            if '.' in name:
+                objname, fieldname = name.split('.')
+                seen.add(objname)
+        assert len(seen) == 1, (
+            "Current limitation: you can only give one instance with green "
+            "fields.  Found %r" % seen.keys())
+        self.red_index = jd.jitdriver.reds.index(objname)
+        #
+        # a list of (GTYPE, fieldname)
+        self.green_fields = jd.jitdriver.ll_greenfields.values()
+        self.green_field_descrs = [cpu.fielddescrof(GTYPE, fieldname)
+                                   for GTYPE, fieldname in self.green_fields]
+
+    def _freeze_(self):
+        return True

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/jitdriver.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/jitdriver.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/jitdriver.py	Sat Oct 16 20:05:28 2010
@@ -13,6 +13,7 @@
     #    self.num_red_args      ... pypy.jit.metainterp.warmspot
     #    self.result_type       ... pypy.jit.metainterp.warmspot
     #    self.virtualizable_info... pypy.jit.metainterp.warmspot
+    #    self.greenfield_info   ... pypy.jit.metainterp.warmspot
     #    self.warmstate         ... pypy.jit.metainterp.warmspot
     #    self.handle_jitexc_from_bh pypy.jit.metainterp.warmspot
     #    self.portal_finishtoken... pypy.jit.metainterp.pyjitpl

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/pyjitpl.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/pyjitpl.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/pyjitpl.py	Sat Oct 16 20:05:28 2010
@@ -498,6 +498,22 @@
     opimpl_getfield_gc_r_pure = _opimpl_getfield_gc_pure_any
     opimpl_getfield_gc_f_pure = _opimpl_getfield_gc_pure_any
 
+    @arguments("orgpc", "box", "descr")
+    def _opimpl_getfield_gc_greenfield_any(self, pc, box, fielddescr):
+        ginfo = self.metainterp.jitdriver_sd.greenfield_info
+        if (ginfo is not None and fielddescr in ginfo.green_field_descrs
+            and not self._nonstandard_virtualizable(pc, box)):
+            # fetch the result, but consider it as a Const box and don't
+            # record any operation
+            resbox = executor.execute(self.metainterp.cpu, self.metainterp,
+                                      rop.GETFIELD_GC_PURE, fielddescr, box)
+            return resbox.constbox()
+        # fall-back
+        return self.execute_with_descr(rop.GETFIELD_GC_PURE, fielddescr, box)
+    opimpl_getfield_gc_i_greenfield = _opimpl_getfield_gc_greenfield_any
+    opimpl_getfield_gc_r_greenfield = _opimpl_getfield_gc_greenfield_any
+    opimpl_getfield_gc_f_greenfield = _opimpl_getfield_gc_greenfield_any
+
     @arguments("box", "descr", "box")
     def _opimpl_setfield_gc_any(self, box, fielddescr, valuebox):
         self.execute_with_descr(rop.SETFIELD_GC, fielddescr, box, valuebox)
@@ -529,7 +545,8 @@
     def _nonstandard_virtualizable(self, pc, box):
         # returns True if 'box' is actually not the "standard" virtualizable
         # that is stored in metainterp.virtualizable_boxes[-1]
-        if self.metainterp.jitdriver_sd.virtualizable_info is None:
+        if (self.metainterp.jitdriver_sd.virtualizable_info is None and
+            self.metainterp.jitdriver_sd.greenfield_info is None):
             return True      # can occur in case of multiple JITs
         standard_box = self.metainterp.virtualizable_boxes[-1]
         if standard_box is box:
@@ -998,13 +1015,15 @@
         guard_op = metainterp.history.record(opnum, moreargs, None,
                                              descr=resumedescr)
         virtualizable_boxes = None
-        if metainterp.jitdriver_sd.virtualizable_info is not None:
+        if (metainterp.jitdriver_sd.virtualizable_info is not None or
+            metainterp.jitdriver_sd.greenfield_info is not None):
             virtualizable_boxes = metainterp.virtualizable_boxes
         saved_pc = self.pc
         if resumepc >= 0:
             self.pc = resumepc
         resume.capture_resumedata(metainterp.framestack, virtualizable_boxes,
-                                  metainterp.virtualref_boxes, resumedescr)
+                                  metainterp.virtualref_boxes,
+                                  resumedescr)
         self.pc = saved_pc
         self.metainterp.staticdata.profiler.count_ops(opnum, GUARDS)
         # count
@@ -1646,6 +1665,7 @@
                                               duplicates)
             live_arg_boxes += self.virtualizable_boxes
             live_arg_boxes.pop()
+        #
         assert len(self.virtualref_boxes) == 0, "missing virtual_ref_finish()?"
         # Called whenever we reach the 'loop_header' hint.
         # First, attempt to make a bridge:
@@ -1832,6 +1852,7 @@
         f.setup_call(original_boxes)
         assert self.in_recursion == 0
         self.virtualref_boxes = []
+        self.initialize_withgreenfields(original_boxes)
         self.initialize_virtualizable(original_boxes)
 
     def initialize_state_from_guard_failure(self, resumedescr):
@@ -1856,6 +1877,14 @@
             self.virtualizable_boxes.append(virtualizable_box)
             self.initialize_virtualizable_enter()
 
+    def initialize_withgreenfields(self, original_boxes):
+        ginfo = self.jitdriver_sd.greenfield_info
+        if ginfo is not None:
+            assert self.jitdriver_sd.virtualizable_info is None
+            index = (self.jitdriver_sd.num_green_args +
+                     ginfo.red_index)
+            self.virtualizable_boxes = [original_boxes[index]]
+
     def initialize_virtualizable_enter(self):
         vinfo = self.jitdriver_sd.virtualizable_info
         virtualizable_box = self.virtualizable_boxes[-1]
@@ -1949,8 +1978,10 @@
 
     def rebuild_state_after_failure(self, resumedescr):
         vinfo = self.jitdriver_sd.virtualizable_info
+        ginfo = self.jitdriver_sd.greenfield_info
         self.framestack = []
-        boxlists = resume.rebuild_from_resumedata(self, resumedescr, vinfo)
+        boxlists = resume.rebuild_from_resumedata(self, resumedescr, vinfo,
+                                                  ginfo)
         inputargs_and_holes, virtualizable_boxes, virtualref_boxes = boxlists
         #
         # virtual refs: make the vrefs point to the freshly allocated virtuals
@@ -1975,6 +2006,12 @@
             assert not virtualizable.vable_token
             # fill the virtualizable with the local boxes
             self.synchronize_virtualizable()
+        #
+        elif self.jitdriver_sd.greenfield_info:
+            self.virtualizable_boxes = virtualizable_boxes
+        else:
+            assert not virtualizable_boxes
+        #
         return inputargs_and_holes
 
     def check_synchronized_virtualizable(self):
@@ -2048,7 +2085,8 @@
         for i in range(len(boxes)):
             if boxes[i] is oldbox:
                 boxes[i] = newbox
-        if self.jitdriver_sd.virtualizable_info is not None:
+        if (self.jitdriver_sd.virtualizable_info is not None or
+            self.jitdriver_sd.greenfield_info is not None):
             boxes = self.virtualizable_boxes
             for i in range(len(boxes)):
                 if boxes[i] is oldbox:

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/resume.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/resume.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/resume.py	Sat Oct 16 20:05:28 2010
@@ -689,9 +689,11 @@
 
 # ---------- when resuming for pyjitpl.py, make boxes ----------
 
-def rebuild_from_resumedata(metainterp, storage, virtualizable_info):
+def rebuild_from_resumedata(metainterp, storage, virtualizable_info,
+                            greenfield_info):
     resumereader = ResumeDataBoxReader(storage, metainterp)
-    boxes = resumereader.consume_vref_and_vable_boxes(virtualizable_info)
+    boxes = resumereader.consume_vref_and_vable_boxes(virtualizable_info,
+                                                      greenfield_info)
     virtualizable_boxes, virtualref_boxes = boxes
     frameinfo = storage.rd_frame_info_list
     while True:
@@ -901,8 +903,9 @@
     resumereader = ResumeDataDirectReader(blackholeinterpbuilder.cpu, storage,
                                           all_virtuals)
     vinfo = jitdriver_sd.virtualizable_info
+    ginfo = jitdriver_sd.greenfield_info
     vrefinfo = blackholeinterpbuilder.metainterp_sd.virtualref_info
-    resumereader.consume_vref_and_vable(vrefinfo, vinfo)
+    resumereader.consume_vref_and_vable(vrefinfo, vinfo, ginfo)
     #
     # First get a chain of blackhole interpreters whose length is given
     # by the depth of rd_frame_info_list.  The first one we get must be
@@ -932,11 +935,11 @@
     resumereader.done()
     return firstbh
 
-def force_from_resumedata(metainterp_sd, storage, vinfo=None):
+def force_from_resumedata(metainterp_sd, storage, vinfo, ginfo):
     resumereader = ResumeDataDirectReader(metainterp_sd.cpu, storage)
     resumereader.handling_async_forcing()
     vrefinfo = metainterp_sd.virtualref_info
-    resumereader.consume_vref_and_vable(vrefinfo, vinfo)
+    resumereader.consume_vref_and_vable(vrefinfo, vinfo, ginfo)
     return resumereader.force_all_virtuals()
 
 class ResumeDataDirectReader(AbstractResumeDataReader):
@@ -1011,11 +1014,12 @@
         return specialize_value(TYPE, x)
     load_value_of_type._annspecialcase_ = 'specialize:arg(1)'
 
-    def consume_vref_and_vable(self, vrefinfo, vinfo):
+    def consume_vref_and_vable(self, vrefinfo, vinfo, ginfo):
         nums = self.cur_numb.nums
         self.cur_numb = self.cur_numb.prev
         if self.resume_after_guard_not_forced != 2:
             end_vref = self.consume_vable_info(vinfo, nums)
+            if ginfo is not None: end_vref -= 1
             self.consume_virtualref_info(vrefinfo, nums, end_vref)
 
     def allocate_with_vtable(self, known_class):

Added: pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_greenfield.py
==============================================================================
--- (empty file)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_greenfield.py	Sat Oct 16 20:05:28 2010
@@ -0,0 +1,35 @@
+from pypy.jit.metainterp.test.test_basic import LLJitMixin, OOJitMixin
+from pypy.rlib.jit import JitDriver
+
+
+class GreenFieldsTests:
+
+    def test_green_field_1(self):
+        myjitdriver = JitDriver(greens=['ctx.x'], reds=['ctx'])
+        class Ctx(object):
+            _immutable_fields_ = ['x']
+            def __init__(self, x, y):
+                self.x = x
+                self.y = y
+        def f(x, y):
+            ctx = Ctx(x, y)
+            while 1:
+                myjitdriver.can_enter_jit(ctx=ctx)
+                myjitdriver.jit_merge_point(ctx=ctx)
+                ctx.y -= 1
+                if ctx.y < 0:
+                    return ctx.y
+        def g(y):
+            return f(5, y) + f(6, y)
+        #
+        res = self.meta_interp(g, [7])
+        assert res == -2
+        self.check_loop_count(2)
+        self.check_loops(guard_value=0)
+
+
+class TestLLtypeGreenFieldsTests(GreenFieldsTests, LLJitMixin):
+    pass
+
+class TestOOtypeGreenFieldsTests(GreenFieldsTests, OOJitMixin):
+   pass

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_immutable.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_immutable.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_immutable.py	Sat Oct 16 20:05:28 2010
@@ -1,5 +1,4 @@
 from pypy.jit.metainterp.test.test_basic import LLJitMixin, OOJitMixin
-from pypy.rlib.jit import JitDriver
 
 
 class ImmutableFieldsTests:
@@ -86,29 +85,6 @@
                             int_add=3)
 
 
-    def test_green_field(self):
-        myjitdriver = JitDriver(greens=['ctx.x'], reds=['ctx'])
-        class Ctx(object):
-            _immutable_fields_ = ['x']
-            def __init__(self, x, y):
-                self.x = x
-                self.y = y
-        def f(x, y):
-            ctx = Ctx(x, y)
-            while 1:
-                myjitdriver.can_enter_jit(ctx=ctx)
-                myjitdriver.jit_merge_point(ctx=ctx)
-                ctx.y -= 1
-                if ctx.y < 0:
-                    return ctx.y
-        def g(y):
-            return f(5, y) + f(6, y)
-        #
-        res = self.meta_interp(g, [7])
-        assert res == -2
-        self.check_loop_count(2)
-
-
 class TestLLtypeImmutableFieldsTests(ImmutableFieldsTests, LLJitMixin):
     pass
 

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py	Sat Oct 16 20:05:28 2010
@@ -286,10 +286,20 @@
     def make_virtualizable_infos(self):
         vinfos = {}
         for jd in self.jitdrivers_sd:
+            #
+            jd.greenfield_info = None
+            for name in jd.jitdriver.greens:
+                if '.' in name:
+                    from pypy.jit.metainterp.greenfield import GreenFieldInfo
+                    jd.greenfield_info = GreenFieldInfo(self.cpu, jd)
+                    break
+            #
             if not jd.jitdriver.virtualizables:
                 jd.virtualizable_info = None
                 jd.index_of_virtualizable = -1
                 continue
+            else:
+                assert jd.greenfield_info is None, "XXX not supported yet"
             #
             jitdriver = jd.jitdriver
             assert len(jitdriver.virtualizables) == 1    # for now

Modified: pypy/branch/rsre-jit/pypy/rlib/jit.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/rlib/jit.py	(original)
+++ pypy/branch/rsre-jit/pypy/rlib/jit.py	Sat Oct 16 20:05:28 2010
@@ -445,8 +445,13 @@
                         "field %r not found in %r" % (name,
                                                       r_red.lowleveltype.TO))
                     r_red = r_red.rbase
-                assert r_red.lowleveltype.TO._immutable_field(mangled_name), (
+                GTYPE = r_red.lowleveltype.TO
+                assert GTYPE._immutable_field(mangled_name), (
                     "field %r must be declared as immutable" % name)
+                if not hasattr(driver, 'll_greenfields'):
+                    driver.ll_greenfields = {}
+                driver.ll_greenfields[name] = GTYPE, mangled_name
+                #
                 v_red = hop.inputarg(r_red, arg=i)
                 c_llname = hop.inputconst(lltype.Void, mangled_name)
                 v_green = hop.genop('getfield', [v_red, c_llname],


From trundle at codespeak.net  Sun Oct 17 01:29:32 2010
From: trundle at codespeak.net (trundle at codespeak.net)
Date: Sun, 17 Oct 2010 01:29:32 +0200 (CEST)
Subject: [pypy-svn] r78027 - in pypy/branch/fast-forward: lib-python
	pypy/interpreter pypy/interpreter/test
Message-ID: <20101016232932.83C1836E3F0@codespeak.net>

Author: trundle
Date: Sun Oct 17 01:29:28 2010
New Revision: 78027

Modified:
   pypy/branch/fast-forward/lib-python/TODO
   pypy/branch/fast-forward/pypy/interpreter/error.py
   pypy/branch/fast-forward/pypy/interpreter/test/test_generator.py
Log:
Raising a string as exception is not allowed anymore.


Modified: pypy/branch/fast-forward/lib-python/TODO
==============================================================================
--- pypy/branch/fast-forward/lib-python/TODO	(original)
+++ pypy/branch/fast-forward/lib-python/TODO	Sun Oct 17 01:29:28 2010
@@ -34,10 +34,8 @@
 - Ast objects should be picklable, see in pypy/module/_ast/test/test_ast.py:
   test_pickle()
 
-- "exceptions must be old-style classes or derived from BaseException, not str"
-  in the 'raise' statement and generator.throw(), catching a string
-  should emit a DeprecationWarning ("catching of string exceptions is
-  deprecated")
+- catching a string should emit a DeprecationWarning ("catching of
+  string exceptions is deprecated")
 
 - missing builtin: memoryview
 

Modified: pypy/branch/fast-forward/pypy/interpreter/error.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/error.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/error.py	Sun Oct 17 01:29:28 2010
@@ -202,19 +202,14 @@
                         w_value = space.call_function(w_type, w_value)
                     w_type = space.exception_getclass(w_value)
 
-        elif space.full_exceptions and space.is_w(space.type(w_type),
-                                                  space.w_str):
-            space.warn("raising a string exception is deprecated", 
-                       space.w_DeprecationWarning)
-
         else:
             # the only case left here is (inst, None), from a 'raise inst'.
             w_inst = w_type
             w_instclass = space.exception_getclass(w_inst)
             if not space.exception_is_valid_class_w(w_instclass):
                 instclassname = w_instclass.getname(space, '?')
-                msg = ("exceptions must be classes, or instances, "
-                       "or strings (deprecated), not %s")
+                msg = ("exceptions must be old-style classes or derived "
+                       "from BaseException, not %s")
                 raise operationerrfmt(space.w_TypeError, msg, instclassname)
 
             if not space.is_w(w_value, space.w_None):

Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_generator.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/test/test_generator.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/test/test_generator.py	Sun Oct 17 01:29:28 2010
@@ -89,9 +89,10 @@
                 pass
         g = f()
         g.next()
-        # String exceptions are allowed (with DeprecationWarning)
-        assert g.throw("Error") == 3
-        raises(StopIteration, g.throw, "Error")
+        # String exceptions are not allowed anymore
+        raises(TypeError, g.throw, "Error")
+        assert g.throw(Exception) == 3
+        raises(StopIteration, g.throw, Exception)
 
     def test_throw6(self):
         def f():


From fijall at gmail.com  Sun Oct 17 10:29:58 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 17 Oct 2010 10:29:58 +0200
Subject: [pypy-svn] r77933 - pypy/branch/jitffi/pypy/jit/metainterp/test
In-Reply-To: <20101014133223.A97D9282BEA@codespeak.net>
References: <20101014133223.A97D9282BEA@codespeak.net>
Message-ID: 

The exception still can't happen though.

On Thu, Oct 14, 2010 at 3:32 PM,   wrote:
> Author: antocuni
> Date: Thu Oct 14 15:32:22 2010
> New Revision: 77933
>
> Modified:
> ? pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
> Log:
> update this test too
>
>
> Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
> ==============================================================================
> --- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py (original)
> +++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py Thu Oct 14 15:32:22 2010
> @@ -77,7 +77,9 @@
> ? ? ? ? call(0, p2, ? ? ? ? ? ? descr=libffi_prepare)
> ? ? ? ? call(0, p2, i0, ? ? ? ? descr=libffi_push_arg)
> ? ? ? ? call(0, p2, f1, ? ? ? ? descr=libffi_push_arg)
> - ? ? ? ?i3 = call(0, p2, 12345, descr=libffi_call)
> + ? ? ? ?i3 = call_may_force(0, p2, 12345, descr=libffi_call)
> + ? ? ? ?guard_not_forced() []
> + ? ? ? ?guard_no_exception() []
> ? ? ? ? jump(i3, f1, p2)
> ? ? ? ? """
> ? ? ? ? expected = ops
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
>


From afa at codespeak.net  Mon Oct 18 10:00:22 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 18 Oct 2010 10:00:22 +0200 (CEST)
Subject: [pypy-svn] r78031 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101018080022.0359A282B90@codespeak.net>

Author: afa
Date: Mon Oct 18 10:00:19 2010
New Revision: 78031

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
Log:
Add FileIO.name, FileIO.mode


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Mon Oct 18 10:00:19 2010
@@ -1,5 +1,6 @@
 from pypy.module._io.interp_iobase import W_RawIOBase
-from pypy.interpreter.typedef import TypeDef
+from pypy.interpreter.typedef import (
+    TypeDef, interp_attrproperty_w, GetSetProperty)
 from pypy.interpreter.gateway import interp2app, unwrap_spec, Arguments
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.error import OperationError, wrap_oserror2
@@ -77,6 +78,7 @@
         self.writable = False
         self.seekable = -1
         self.closefd = True
+        self.w_name = None
 
     @unwrap_spec(ObjSpace, W_Root, Arguments)
     def descr_new(space, w_subtype, __args__):
@@ -113,6 +115,19 @@
             except OSError, e:
                 raise wrap_oserror2(space, e, w_name)
         self.closefd = bool(closefd)
+        self.w_name = w_name
+
+    def _mode(self):
+        if self.readable:
+            if self.writable:
+                return 'rb+'
+            else:
+                return 'rb'
+        else:
+            return 'wb'
+
+    def descr_get_mode(space, self):
+        return space.wrap(self._mode())
 
     def _check_closed(self, space):
         if self.fd < 0:
@@ -157,5 +172,7 @@
     readable = interp2app(W_FileIO.readable_w),
     writable = interp2app(W_FileIO.writable_w),
     seekable = interp2app(W_FileIO.seekable_w),
+    name = interp_attrproperty_w('w_name', cls=W_FileIO),
+    mode = GetSetProperty(W_FileIO.descr_get_mode),
     )
 

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	Mon Oct 18 10:00:19 2010
@@ -50,6 +50,8 @@
     def test_open(self):
         import io
         f = io.open(self.tmpfile, "rb")
+        assert f.name.endswith('tmpfile')
+        assert f.mode == 'rb'
         f.close()
 
     def test_open_writable(self):


From afa at codespeak.net  Mon Oct 18 10:04:15 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 18 Oct 2010 10:04:15 +0200 (CEST)
Subject: [pypy-svn] r78032 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101018080415.68403282B90@codespeak.net>

Author: afa
Date: Mon Oct 18 10:04:13 2010
New Revision: 78032

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
Log:
add FileIO.fileno()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Mon Oct 18 10:04:13 2010
@@ -162,6 +162,11 @@
                 self.seekable = 1
         return space.newbool(self.seekable == 1)
 
+    @unwrap_spec('self', ObjSpace)
+    def fileno_w(self, space):
+        self._check_closed(space)
+        return space.wrap(self.fd)
+
 W_FileIO.typedef = TypeDef(
     'FileIO', W_RawIOBase.typedef,
     __new__  = interp2app(W_FileIO.descr_new.im_func),
@@ -172,6 +177,7 @@
     readable = interp2app(W_FileIO.readable_w),
     writable = interp2app(W_FileIO.writable_w),
     seekable = interp2app(W_FileIO.seekable_w),
+    fileno = interp2app(W_FileIO.fileno_w),
     name = interp_attrproperty_w('w_name', cls=W_FileIO),
     mode = GetSetProperty(W_FileIO.descr_get_mode),
     )

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	Mon Oct 18 10:04:13 2010
@@ -65,4 +65,5 @@
         os = self.posix
         fd = os.open(self.tmpfile, os.O_RDONLY, 0666)
         f = io.open(fd, "rb")
+        assert f.fileno() == fd
         f.close()


From antocuni at codespeak.net  Mon Oct 18 11:22:26 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Mon, 18 Oct 2010 11:22:26 +0200 (CEST)
Subject: [pypy-svn] r78033 - in pypy/branch/jitffi: .
	lib-python/modified-2.5.2/ctypes lib_pypy/_ctypes pypy
	pypy/config pypy/doc/config pypy/interpreter
	pypy/interpreter/pyparser pypy/jit/backend/test
	pypy/jit/backend/x86 pypy/jit/backend/x86/test
	pypy/jit/codewriter pypy/jit/codewriter/test
	pypy/jit/metainterp pypy/jit/metainterp/optimizeopt
	pypy/jit/metainterp/test pypy/jit/tool
	pypy/module/__builtin__ pypy/module/__builtin__/test
	pypy/module/_rawffi pypy/module/_weakref
	pypy/module/array/benchmark pypy/module/array/test
	pypy/module/cpyext pypy/module/cpyext/test pypy/module/gc
	pypy/module/pypyjit/test pypy/module/signal pypy/objspace/std
	pypy/objspace/std/test pypy/rlib pypy/rlib/test pypy/rpython
	pypy/rpython/lltypesystem pypy/rpython/module
	pypy/rpython/test pypy/translator/c/gcc
	pypy/translator/c/gcc/test/elf pypy/translator/c/src
Message-ID: <20101018092226.0B2EA282B90@codespeak.net>

Author: antocuni
Date: Mon Oct 18 11:22:20 2010
New Revision: 78033

Added:
   pypy/branch/jitffi/pypy/doc/config/objspace.std.withmapdict.txt
      - copied unchanged from r78032, pypy/trunk/pypy/doc/config/objspace.std.withmapdict.txt
   pypy/branch/jitffi/pypy/doc/config/objspace.std.withstrbuf.txt
      - copied unchanged from r78032, pypy/trunk/pypy/doc/config/objspace.std.withstrbuf.txt
   pypy/branch/jitffi/pypy/objspace/std/mapdict.py
      - copied unchanged from r78032, pypy/trunk/pypy/objspace/std/mapdict.py
   pypy/branch/jitffi/pypy/objspace/std/strbufobject.py
      - copied unchanged from r78032, pypy/trunk/pypy/objspace/std/strbufobject.py
   pypy/branch/jitffi/pypy/objspace/std/test/test_mapdict.py
      - copied unchanged from r78032, pypy/trunk/pypy/objspace/std/test/test_mapdict.py
   pypy/branch/jitffi/pypy/objspace/std/test/test_strbufobject.py
      - copied unchanged from r78032, pypy/trunk/pypy/objspace/std/test/test_strbufobject.py
   pypy/branch/jitffi/pypy/rlib/rerased.py
      - copied unchanged from r78032, pypy/trunk/pypy/rlib/rerased.py
   pypy/branch/jitffi/pypy/rlib/test/test_rerased.py
      - copied unchanged from r78032, pypy/trunk/pypy/rlib/test/test_rerased.py
   pypy/branch/jitffi/pypy/translator/c/gcc/test/elf/track10.s
      - copied unchanged from r78032, pypy/trunk/pypy/translator/c/gcc/test/elf/track10.s
   pypy/branch/jitffi/pypy/translator/c/gcc/test/elf/track11.s
      - copied unchanged from r78032, pypy/trunk/pypy/translator/c/gcc/test/elf/track11.s
Modified:
   pypy/branch/jitffi/   (props changed)
   pypy/branch/jitffi/lib-python/modified-2.5.2/ctypes/__init__.py
   pypy/branch/jitffi/lib_pypy/_ctypes/function.py
   pypy/branch/jitffi/pypy/   (props changed)
   pypy/branch/jitffi/pypy/config/pypyoption.py
   pypy/branch/jitffi/pypy/interpreter/baseobjspace.py
   pypy/branch/jitffi/pypy/interpreter/pycode.py
   pypy/branch/jitffi/pypy/interpreter/pyopcode.py
   pypy/branch/jitffi/pypy/interpreter/pyparser/genpytokenize.py
   pypy/branch/jitffi/pypy/interpreter/pyparser/pytokenize.py
   pypy/branch/jitffi/pypy/interpreter/typedef.py
   pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py
   pypy/branch/jitffi/pypy/jit/backend/test/test_ll_random.py
   pypy/branch/jitffi/pypy/jit/backend/x86/regalloc.py
   pypy/branch/jitffi/pypy/jit/backend/x86/test/test_runner.py
   pypy/branch/jitffi/pypy/jit/backend/x86/test/test_string.py
   pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py
   pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py
   pypy/branch/jitffi/pypy/jit/codewriter/test/test_jtransform.py
   pypy/branch/jitffi/pypy/jit/metainterp/history.py
   pypy/branch/jitffi/pypy/jit/metainterp/optimize_nopspec.py
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/intbounds.py
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/optimizer.py   (props changed)
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/rewrite.py
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/string.py
   pypy/branch/jitffi/pypy/jit/metainterp/resume.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefindnode.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_resume.py
   pypy/branch/jitffi/pypy/jit/metainterp/test/test_string.py
   pypy/branch/jitffi/pypy/jit/tool/traceviewer.py
   pypy/branch/jitffi/pypy/module/__builtin__/interp_classobj.py
   pypy/branch/jitffi/pypy/module/__builtin__/test/test_classobj.py
   pypy/branch/jitffi/pypy/module/_rawffi/array.py
   pypy/branch/jitffi/pypy/module/_weakref/interp__weakref.py
   pypy/branch/jitffi/pypy/module/array/benchmark/Makefile   (props changed)
   pypy/branch/jitffi/pypy/module/array/benchmark/intimg.c   (props changed)
   pypy/branch/jitffi/pypy/module/array/benchmark/intimgtst.c   (props changed)
   pypy/branch/jitffi/pypy/module/array/benchmark/intimgtst.py   (props changed)
   pypy/branch/jitffi/pypy/module/array/benchmark/loop.c   (props changed)
   pypy/branch/jitffi/pypy/module/array/benchmark/sum.c   (props changed)
   pypy/branch/jitffi/pypy/module/array/benchmark/sumtst.c   (props changed)
   pypy/branch/jitffi/pypy/module/array/benchmark/sumtst.py   (props changed)
   pypy/branch/jitffi/pypy/module/array/test/test_array_old.py   (props changed)
   pypy/branch/jitffi/pypy/module/cpyext/classobject.py
   pypy/branch/jitffi/pypy/module/cpyext/test/test_unicodeobject.py
   pypy/branch/jitffi/pypy/module/gc/referents.py
   pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py
   pypy/branch/jitffi/pypy/module/signal/interp_signal.py
   pypy/branch/jitffi/pypy/objspace/std/celldict.py
   pypy/branch/jitffi/pypy/objspace/std/dictmultiobject.py
   pypy/branch/jitffi/pypy/objspace/std/model.py
   pypy/branch/jitffi/pypy/objspace/std/objspace.py
   pypy/branch/jitffi/pypy/objspace/std/sharingdict.py
   pypy/branch/jitffi/pypy/objspace/std/stringobject.py
   pypy/branch/jitffi/pypy/objspace/std/stringtype.py
   pypy/branch/jitffi/pypy/objspace/std/test/test_dictmultiobject.py
   pypy/branch/jitffi/pypy/objspace/std/test/test_shadowtracking.py
   pypy/branch/jitffi/pypy/objspace/std/typeobject.py
   pypy/branch/jitffi/pypy/rlib/_rsocket_rffi.py
   pypy/branch/jitffi/pypy/rlib/clibffi.py
   pypy/branch/jitffi/pypy/rlib/rmmap.py
   pypy/branch/jitffi/pypy/rlib/rstring.py
   pypy/branch/jitffi/pypy/rlib/test/test_rsocket.py
   pypy/branch/jitffi/pypy/rlib/test/test_rstring.py
   pypy/branch/jitffi/pypy/rlib/test/test_runicode.py
   pypy/branch/jitffi/pypy/rpython/lltypesystem/ll2ctypes.py
   pypy/branch/jitffi/pypy/rpython/lltypesystem/rbuilder.py
   pypy/branch/jitffi/pypy/rpython/lltypesystem/rclass.py
   pypy/branch/jitffi/pypy/rpython/module/ll_time.py
   pypy/branch/jitffi/pypy/rpython/rbuilder.py
   pypy/branch/jitffi/pypy/rpython/test/test_rbuilder.py
   pypy/branch/jitffi/pypy/rpython/test/test_rclass.py
   pypy/branch/jitffi/pypy/translator/c/gcc/test/elf/track5.s
   pypy/branch/jitffi/pypy/translator/c/gcc/trackgcroot.py
   pypy/branch/jitffi/pypy/translator/c/src/signals.h
Log:
merge from trunk: svn merge svn+ssh://codespeak.net/svn/pypy/trunk -r77720:78032

Modified: pypy/branch/jitffi/lib-python/modified-2.5.2/ctypes/__init__.py
==============================================================================
--- pypy/branch/jitffi/lib-python/modified-2.5.2/ctypes/__init__.py	(original)
+++ pypy/branch/jitffi/lib-python/modified-2.5.2/ctypes/__init__.py	Mon Oct 18 11:22:20 2010
@@ -345,9 +345,11 @@
         _restype_ = c_int # default, can be overridden in instances
 
     def __init__(self, name, mode=DEFAULT_MODE, handle=None):
+        import _ffi
         self._name = name
         if handle is None:
-            self._handle = _dlopen(self._name, mode)
+            #self._handle = _dlopen(self._name, mode)
+            self._handle = _ffi.CDLL(name)
         else:
             self._handle = handle
 

Modified: pypy/branch/jitffi/lib_pypy/_ctypes/function.py
==============================================================================
--- pypy/branch/jitffi/lib_pypy/_ctypes/function.py	(original)
+++ pypy/branch/jitffi/lib_pypy/_ctypes/function.py	Mon Oct 18 11:22:20 2010
@@ -110,7 +110,8 @@
                 self.dll = ctypes.CDLL(self.dll)
             # we need to check dll anyway
             ptr = self._getfuncptr([], ctypes.c_int)
-            self._buffer = ptr.byptr()
+            #self._buffer = ptr.byptr()
+            self._buffer = None
 
         elif (sys.platform == 'win32' and
               len(args) >= 2 and isinstance(args[0], (int, long))):
@@ -163,12 +164,25 @@
             
         if argtypes is None:
             argtypes = self._guess_argtypes(args)
-        argtypes, argsandobjs = self._wrap_args(argtypes, args)
+
+        # XXX
+        #argtypes, argsandobjs = self._wrap_args(argtypes, args)
         
         restype = self._restype_
         funcptr = self._getfuncptr(argtypes, restype, thisarg)
-        resbuffer = funcptr(*[arg._buffer for _, arg in argsandobjs])
-        return self._build_result(restype, resbuffer, argtypes, argsandobjs)
+        return funcptr(*args)
+        #resbuffer = funcptr(*[arg._buffer for _, arg in argsandobjs])
+        #return self._build_result(restype, resbuffer, argtypes, argsandobjs)
+
+    def _shape_to_ffi_type(self, shape):
+        from _ffi import types
+        if shape == 'l':
+            return types.slong
+        elif shape == 'd':
+            return types.double
+        else:
+            print 'unknown shape %s' % shape
+            import pdb;pdb.set_trace()
 
     def _getfuncptr(self, argtypes, restype, thisarg=None):
         if self._ptr is not None and argtypes is self._argtypes_:
@@ -194,7 +208,11 @@
         
         cdll = self.dll._handle
         try:
-            return cdll.ptr(self.name, argshapes, resshape, self._flags_)
+            #return cdll.ptr(self.name, argshapes, resshape, self._flags_)
+            ffi_argtypes = [self._shape_to_ffi_type(shape) for shape in argshapes]
+            ffi_restype = self._shape_to_ffi_type(resshape)
+            self._ptr = cdll.getfunc(self.name, ffi_argtypes, ffi_restype)
+            return self._ptr
         except AttributeError:
             if self._flags_ & _rawffi.FUNCFLAG_CDECL:
                 raise

Modified: pypy/branch/jitffi/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/jitffi/pypy/config/pypyoption.py	(original)
+++ pypy/branch/jitffi/pypy/config/pypyoption.py	Mon Oct 18 11:22:20 2010
@@ -198,6 +198,9 @@
         BoolOption("withstrslice", "use strings optimized for slicing",
                    default=False),
 
+        BoolOption("withstrbuf", "use strings optimized for addition (ver 2)",
+                   default=False),
+
         BoolOption("withprebuiltchar",
                    "use prebuilt single-character string objects",
                    default=False),
@@ -210,7 +213,8 @@
         BoolOption("withrope", "use ropes as the string implementation",
                    default=False,
                    requires=[("objspace.std.withstrslice", False),
-                             ("objspace.std.withstrjoin", False)],
+                             ("objspace.std.withstrjoin", False),
+                             ("objspace.std.withstrbuf", False)],
                    suggests=[("objspace.std.withprebuiltchar", True),
                              ("objspace.std.sharesmallstr", True)]),
 
@@ -238,6 +242,16 @@
                    default=False,
                    requires=[("objspace.std.withshadowtracking", False)]),
 
+        BoolOption("withmapdict",
+                   "make instances really small but slow without the JIT",
+                   default=False,
+                   requires=[("objspace.std.withshadowtracking", False),
+                             ("objspace.std.withinlineddict", False),
+                             ("objspace.std.withsharingdict", False),
+                             ("objspace.std.getattributeshortcut", True),
+                             ("objspace.std.withtypeversion", True),
+                       ]),
+
         BoolOption("withrangelist",
                    "enable special range list implementation that does not "
                    "actually create the full list until the resulting "
@@ -343,7 +357,7 @@
         config.objspace.std.suggest(withprebuiltint=True)
         config.objspace.std.suggest(withrangelist=True)
         config.objspace.std.suggest(withprebuiltchar=True)
-        config.objspace.std.suggest(withinlineddict=True)
+        config.objspace.std.suggest(withmapdict=True)
         config.objspace.std.suggest(withstrslice=True)
         config.objspace.std.suggest(withstrjoin=True)
         # xxx other options? ropes maybe?
@@ -359,6 +373,7 @@
     # extra optimizations with the JIT
     if level == 'jit':
         config.objspace.std.suggest(withcelldict=True)
+        #config.objspace.std.suggest(withmapdict=True)
 
 
 def enable_allworkingmodules(config):

Modified: pypy/branch/jitffi/pypy/interpreter/baseobjspace.py
==============================================================================
--- pypy/branch/jitffi/pypy/interpreter/baseobjspace.py	(original)
+++ pypy/branch/jitffi/pypy/interpreter/baseobjspace.py	Mon Oct 18 11:22:20 2010
@@ -168,6 +168,20 @@
     def _call_builtin_destructor(self):
         pass     # method overridden in typedef.py
 
+    # hooks that the mapdict implementations needs:
+    def _get_mapdict_map(self):
+        return None
+    def _set_mapdict_map(self, map):
+        raise NotImplementedError
+    def _mapdict_read_storage(self, index):
+        raise NotImplementedError
+    def _mapdict_write_storage(self, index, value):
+        raise NotImplementedError
+    def _mapdict_storage_length(self):
+        raise NotImplementedError
+    def _set_mapdict_storage_and_map(self, storage, map):
+        raise NotImplementedError
+
 
 class Wrappable(W_Root):
     """A subclass of Wrappable is an internal, interpreter-level class

Modified: pypy/branch/jitffi/pypy/interpreter/pycode.py
==============================================================================
--- pypy/branch/jitffi/pypy/interpreter/pycode.py	(original)
+++ pypy/branch/jitffi/pypy/interpreter/pycode.py	Mon Oct 18 11:22:20 2010
@@ -117,6 +117,10 @@
 
         self._compute_flatcall()
 
+        if self.space.config.objspace.std.withmapdict:
+            from pypy.objspace.std.mapdict import init_mapdict_cache
+            init_mapdict_cache(self)
+
     def _freeze_(self):
         if (self.magic == cpython_magic and
             '__pypy__' not in sys.builtin_module_names):

Modified: pypy/branch/jitffi/pypy/interpreter/pyopcode.py
==============================================================================
--- pypy/branch/jitffi/pypy/interpreter/pyopcode.py	(original)
+++ pypy/branch/jitffi/pypy/interpreter/pyopcode.py	Mon Oct 18 11:22:20 2010
@@ -710,9 +710,14 @@
 
     def LOAD_ATTR(self, nameindex, next_instr):
         "obj.attributename"
-        w_attributename = self.getname_w(nameindex)
         w_obj = self.popvalue()
-        w_value = self.space.getattr(w_obj, w_attributename)
+        if (self.space.config.objspace.std.withmapdict
+            and not jit.we_are_jitted()):
+            from pypy.objspace.std.mapdict import LOAD_ATTR_caching
+            w_value = LOAD_ATTR_caching(self.getcode(), w_obj, nameindex)
+        else:
+            w_attributename = self.getname_w(nameindex)
+            w_value = self.space.getattr(w_obj, w_attributename)
         self.pushvalue(w_value)
     LOAD_ATTR._always_inline_ = True
 

Modified: pypy/branch/jitffi/pypy/interpreter/pyparser/genpytokenize.py
==============================================================================
--- pypy/branch/jitffi/pypy/interpreter/pyparser/genpytokenize.py	(original)
+++ pypy/branch/jitffi/pypy/interpreter/pyparser/genpytokenize.py	Mon Oct 18 11:22:20 2010
@@ -176,6 +176,7 @@
     pseudoToken = chain(states,
                         makeWhitespace(),
                         group(states,
+                              newArcPair(states, EMPTY),
                               pseudoExtras, number, funny, contStr, name))
     dfaStates, dfaAccepts = nfaToDfa(states, *pseudoToken)
     return DFA(dfaStates, dfaAccepts)

Modified: pypy/branch/jitffi/pypy/interpreter/pyparser/pytokenize.py
==============================================================================
--- pypy/branch/jitffi/pypy/interpreter/pyparser/pytokenize.py	(original)
+++ pypy/branch/jitffi/pypy/interpreter/pyparser/pytokenize.py	Mon Oct 18 11:22:20 2010
@@ -23,7 +23,7 @@
 # ______________________________________________________________________
 # Automatically generated DFA's
 
-accepts = [False, True, True, True, True, True, True, True,
+accepts = [True, True, True, True, True, True, True, True,
            True, True, False, True, True, True, False, False,
            False, False, True, True, True, False, True,
            False, True, False, True, False, False, True,

Modified: pypy/branch/jitffi/pypy/interpreter/typedef.py
==============================================================================
--- pypy/branch/jitffi/pypy/interpreter/typedef.py	(original)
+++ pypy/branch/jitffi/pypy/interpreter/typedef.py	Mon Oct 18 11:22:20 2010
@@ -133,6 +133,13 @@
     typedef = cls.typedef
     if wants_dict and typedef.hasdict:
         wants_dict = False
+    if config.objspace.std.withmapdict and not typedef.hasdict:
+        # mapdict only works if the type does not already have a dict
+        if wants_del:
+            parentcls = get_unique_interplevel_subclass(config, cls, True, True,
+                                                        False, True)
+            return _usersubclswithfeature(config, parentcls, "del")
+        return _usersubclswithfeature(config, cls, "user", "dict", "weakref", "slots")
     # Forest of if's - see the comment above.
     if wants_del:
         if wants_dict:
@@ -186,10 +193,20 @@
 
     def add(Proto):
         for key, value in Proto.__dict__.items():
-            if not key.startswith('__') or key == '__del__':
+            if (not key.startswith('__') and not key.startswith('_mixin_') 
+                    or key == '__del__'):
+                if hasattr(value, "func_name"):
+                    value = func_with_new_name(value, value.func_name)
                 body[key] = value
 
+    if (config.objspace.std.withmapdict and "dict" in features):
+        from pypy.objspace.std.mapdict import BaseMapdictObject, ObjectMixin
+        add(BaseMapdictObject)
+        add(ObjectMixin)
+        features = ()
+
     if "user" in features:     # generic feature needed by all subcls
+
         class Proto(object):
             user_overridden_class = True
 
@@ -255,6 +272,9 @@
             wantdict = False
 
     if wantdict:
+        base_user_setup = supercls.user_setup.im_func
+        if "user_setup" in body:
+            base_user_setup = body["user_setup"]
         class Proto(object):
             def getdict(self):
                 return self.w__dict__
@@ -263,11 +283,9 @@
                 self.w__dict__ = check_new_dictionary(space, w_dict)
             
             def user_setup(self, space, w_subtype):
-                self.space = space
-                self.w__class__ = w_subtype
                 self.w__dict__ = space.newdict(
                     instance=True, classofinstance=w_subtype)
-                self.user_setup_slots(w_subtype.nslots)
+                base_user_setup(self, space, w_subtype)
 
             def setclass(self, space, w_subtype):
                 # only used by descr_set___class__

Modified: pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/branch/jitffi/pypy/jit/backend/test/runner_test.py	Mon Oct 18 11:22:20 2010
@@ -859,6 +859,23 @@
                                                         length_box], 'void')
                                 assert self.look_string(r_box) == "!??cdef?!"
 
+    def test_copyunicodecontent(self):
+        s_box = self.alloc_unicode(u"abcdef")
+        for s_box in [s_box, s_box.constbox()]:
+            for srcstart_box in [BoxInt(2), ConstInt(2)]:
+                for dststart_box in [BoxInt(3), ConstInt(3)]:
+                    for length_box in [BoxInt(4), ConstInt(4)]:
+                        for r_box_is_const in [False, True]:
+                            r_box = self.alloc_unicode(u"!???????!")
+                            if r_box_is_const:
+                                r_box = r_box.constbox()
+                                self.execute_operation(rop.COPYUNICODECONTENT,
+                                                       [s_box, r_box,
+                                                        srcstart_box,
+                                                        dststart_box,
+                                                        length_box], 'void')
+                                assert self.look_unicode(r_box) == u"!??cdef?!"
+
     def test_do_unicode_basic(self):
         u = self.cpu.bh_newunicode(5)
         self.cpu.bh_unicodesetitem(u, 4, 123)
@@ -1253,6 +1270,10 @@
         u_box = BoxPtr(lltype.cast_opaque_ptr(llmemory.GCREF, u))
         return u_box
 
+    def look_unicode(self, unicode_box):
+        u = unicode_box.getref(lltype.Ptr(rstr.UNICODE))
+        return u''.join(u.chars)
+
 
     def test_casts(self):
         py.test.skip("xxx fix or kill")

Modified: pypy/branch/jitffi/pypy/jit/backend/test/test_ll_random.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/backend/test/test_ll_random.py	(original)
+++ pypy/branch/jitffi/pypy/jit/backend/test/test_ll_random.py	Mon Oct 18 11:22:20 2010
@@ -599,7 +599,7 @@
     OPERATIONS.append(StrLenOperation(rop.STRLEN))
     OPERATIONS.append(UnicodeLenOperation(rop.UNICODELEN))
     OPERATIONS.append(CopyStrContentOperation(rop.COPYSTRCONTENT))
-    #OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT))
+    OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT))
 
 for i in range(2):
     OPERATIONS.append(GuardClassOperation(rop.GUARD_CLASS))

Modified: pypy/branch/jitffi/pypy/jit/backend/x86/regalloc.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/backend/x86/regalloc.py	(original)
+++ pypy/branch/jitffi/pypy/jit/backend/x86/regalloc.py	Mon Oct 18 11:22:20 2010
@@ -778,15 +778,11 @@
             loc = self.loc(op.getarg(0))
             return self._call(op, [loc])
         # boehm GC (XXX kill the following code at some point)
-        ofs_items, itemsize, ofs = symbolic.get_array_token(rstr.UNICODE, self.translate_support_code)
-        if itemsize == 4:
-            return self._malloc_varsize(ofs_items, ofs, 2, op.getarg(0),
-                                        op.result)
-        elif itemsize == 2:
-            return self._malloc_varsize(ofs_items, ofs, 1, op.getarg(0),
-                                        op.result)
-        else:
-            assert False, itemsize
+        ofs_items, _, ofs = symbolic.get_array_token(rstr.UNICODE,
+                                                   self.translate_support_code)
+        scale = self._get_unicode_item_scale()
+        return self._malloc_varsize(ofs_items, ofs, scale, op.getarg(0),
+                                    op.result)
 
     def _malloc_varsize(self, ofs_items, ofs_length, scale, v, res_v):
         # XXX kill this function at some point
@@ -959,6 +955,12 @@
     consider_unicodegetitem = consider_strgetitem
 
     def consider_copystrcontent(self, op):
+        self._consider_copystrcontent(op, is_unicode=False)
+
+    def consider_copyunicodecontent(self, op):
+        self._consider_copystrcontent(op, is_unicode=True)
+
+    def _consider_copystrcontent(self, op, is_unicode):
         # compute the source address
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(args[0], args)
@@ -970,7 +972,8 @@
         srcaddr_box = TempBox()
         forbidden_vars = [args[1], args[3], args[4], srcaddr_box]
         srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box, forbidden_vars)
-        self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc)
+        self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc,
+                                        is_unicode=is_unicode)
         # compute the destination address
         base_loc = self.rm.make_sure_var_in_reg(args[1], forbidden_vars)
         ofs_loc = self.rm.make_sure_var_in_reg(args[3], forbidden_vars)
@@ -980,25 +983,57 @@
         forbidden_vars = [args[4], srcaddr_box]
         dstaddr_box = TempBox()
         dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box, forbidden_vars)
-        self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc)
+        self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc,
+                                        is_unicode=is_unicode)
+        # compute the length in bytes
+        length_box = args[4]
+        length_loc = self.loc(length_box)
+        if is_unicode:
+            self.rm.possibly_free_var(length_box)
+            forbidden_vars = [srcaddr_box, dstaddr_box]
+            bytes_box = TempBox()
+            bytes_loc = self.rm.force_allocate_reg(bytes_box, forbidden_vars)
+            scale = self._get_unicode_item_scale()
+            if not (isinstance(length_loc, ImmedLoc) or
+                    isinstance(length_loc, RegLoc)):
+                self.assembler.mov(length_loc, bytes_loc)
+                length_loc = bytes_loc
+            self.assembler.load_effective_addr(length_loc, 0, scale, bytes_loc)
+            length_box = bytes_box
+            length_loc = bytes_loc
         # call memcpy()
-        length_loc = self.loc(args[4])
         self.rm.before_call()
         self.xrm.before_call()
         self.assembler._emit_call(imm(self.assembler.memcpy_addr),
                                   [dstaddr_loc, srcaddr_loc, length_loc])
-        self.rm.possibly_free_var(args[4])
+        self.rm.possibly_free_var(length_box)
         self.rm.possibly_free_var(dstaddr_box)
         self.rm.possibly_free_var(srcaddr_box)
 
-    def _gen_address_inside_string(self, baseloc, ofsloc, resloc):
+    def _gen_address_inside_string(self, baseloc, ofsloc, resloc, is_unicode):
         cpu = self.assembler.cpu
-        ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR,
+        if is_unicode:
+            ofs_items, _, _ = symbolic.get_array_token(rstr.UNICODE,
+                                                  self.translate_support_code)
+            scale = self._get_unicode_item_scale()
+        else:
+            ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR,
                                                   self.translate_support_code)
-        assert itemsize == 1
-        self.assembler.load_effective_addr(ofsloc, ofs_items, 0,
+            assert itemsize == 1
+            scale = 0
+        self.assembler.load_effective_addr(ofsloc, ofs_items, scale,
                                            resloc, baseloc)
 
+    def _get_unicode_item_scale(self):
+        _, itemsize, _ = symbolic.get_array_token(rstr.UNICODE,
+                                                  self.translate_support_code)
+        if itemsize == 4:
+            return 2
+        elif itemsize == 2:
+            return 1
+        else:
+            raise AssertionError("bad unicode item size")
+
     def consider_jump(self, op):
         assembler = self.assembler
         assert self.jump_target_descr is None

Modified: pypy/branch/jitffi/pypy/jit/backend/x86/test/test_runner.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/backend/x86/test/test_runner.py	(original)
+++ pypy/branch/jitffi/pypy/jit/backend/x86/test/test_runner.py	Mon Oct 18 11:22:20 2010
@@ -506,8 +506,8 @@
         self.cpu.execute_token(ops.token)
         # check debugging info
         name, struct = self.cpu.assembler.loop_run_counters[0]
-        assert name == 'xyz'
+        assert name == 0       # 'xyz'
         assert struct.i == 10
         self.cpu.finish_once()
         lines = py.path.local(self.logfile + ".count").readlines()
-        assert lines[0] == '10      xyz\n'
+        assert lines[0] == '0:10\n'  # '10      xyz\n'

Modified: pypy/branch/jitffi/pypy/jit/backend/x86/test/test_string.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/backend/x86/test/test_string.py	(original)
+++ pypy/branch/jitffi/pypy/jit/backend/x86/test/test_string.py	Mon Oct 18 11:22:20 2010
@@ -2,8 +2,12 @@
 from pypy.jit.metainterp.test import test_string
 from pypy.jit.backend.x86.test.test_basic import Jit386Mixin
 
-class TestString(Jit386Mixin, test_string.StringTests):
+class TestString(Jit386Mixin, test_string.TestLLtype):
     # for the individual tests see
     # ====> ../../../metainterp/test/test_string.py
-    CALL = 'call'
-    CALL_PURE = 'call_pure'
+    pass
+
+class TestUnicode(Jit386Mixin, test_string.TestLLtypeUnicode):
+    # for the individual tests see
+    # ====> ../../../metainterp/test/test_string.py
+    pass

Modified: pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py	(original)
+++ pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py	Mon Oct 18 11:22:20 2010
@@ -18,23 +18,34 @@
     # the 'oopspecindex' field is one of the following values:
     OS_NONE                     = 0    # normal case, no oopspec
     OS_ARRAYCOPY                = 1    # "list.ll_arraycopy"
-    OS_STR_CONCAT               = 2    # "stroruni.concat"
-    OS_UNI_CONCAT               = 3    # "stroruni.concat"
-    OS_STR_SLICE                = 4    # "stroruni.slice"
-    OS_UNI_SLICE                = 5    # "stroruni.slice"
-    OS_STR_EQUAL                = 6    # "stroruni.equal"
-    OS_UNI_EQUAL                = 7    # "stroruni.equal"
-    OS_STREQ_SLICE_CHECKNULL    = 8    # s2!=NULL and s1[x:x+length]==s2
-    OS_STREQ_SLICE_NONNULL      = 9    # s1[x:x+length]==s2   (assert s2!=NULL)
-    OS_STREQ_SLICE_CHAR         = 10   # s1[x:x+length]==char
-    OS_STREQ_NONNULL            = 11   # s1 == s2    (assert s1!=NULL,s2!=NULL)
-    OS_STREQ_NONNULL_CHAR       = 12   # s1 == char  (assert s1!=NULL)
-    OS_STREQ_CHECKNULL_CHAR     = 13   # s1!=NULL and s1==char
-    OS_STREQ_LENGTHOK           = 14   # s1 == s2    (assert len(s1)==len(s2))
+    OS_STR2UNICODE              = 2    # "str.str2unicode"
     OS_LIBFFI_PREPARE           = 15
     OS_LIBFFI_PUSH_ARG          = 16
     OS_LIBFFI_CALL              = 17
 
+    OS_STR_CONCAT               = 22   # "stroruni.concat"
+    OS_STR_SLICE                = 23   # "stroruni.slice"
+    OS_STR_EQUAL                = 24   # "stroruni.equal"
+    OS_STREQ_SLICE_CHECKNULL    = 25   # s2!=NULL and s1[x:x+length]==s2
+    OS_STREQ_SLICE_NONNULL      = 26   # s1[x:x+length]==s2   (assert s2!=NULL)
+    OS_STREQ_SLICE_CHAR         = 27   # s1[x:x+length]==char
+    OS_STREQ_NONNULL            = 28   # s1 == s2    (assert s1!=NULL,s2!=NULL)
+    OS_STREQ_NONNULL_CHAR       = 29   # s1 == char  (assert s1!=NULL)
+    OS_STREQ_CHECKNULL_CHAR     = 30   # s1!=NULL and s1==char
+    OS_STREQ_LENGTHOK           = 31   # s1 == s2    (assert len(s1)==len(s2))
+
+    OS_UNI_CONCAT               = 42   #
+    OS_UNI_SLICE                = 43   #
+    OS_UNI_EQUAL                = 44   #
+    OS_UNIEQ_SLICE_CHECKNULL    = 45   #
+    OS_UNIEQ_SLICE_NONNULL      = 46   #
+    OS_UNIEQ_SLICE_CHAR         = 47   #
+    OS_UNIEQ_NONNULL            = 48   #   the same for unicode
+    OS_UNIEQ_NONNULL_CHAR       = 49   #   (must be the same amount as for
+    OS_UNIEQ_CHECKNULL_CHAR     = 50   #   STR, in the same order)
+    OS_UNIEQ_LENGTHOK           = 51   #
+    _OS_offset_uni              = OS_UNI_CONCAT - OS_STR_CONCAT
+
     def __new__(cls, readonly_descrs_fields,
                 write_descrs_fields, write_descrs_arrays,
                 extraeffect=EF_CAN_RAISE,

Modified: pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/branch/jitffi/pypy/jit/codewriter/jtransform.py	Mon Oct 18 11:22:20 2010
@@ -316,6 +316,8 @@
             prepare = self._handle_list_call
         elif oopspec_name.startswith('stroruni.'):
             prepare = self._handle_stroruni_call
+        elif oopspec_name == 'str.str2unicode':
+            prepare = self._handle_str2unicode_call
         elif oopspec_name.startswith('virtual_ref'):
             prepare = self._handle_virtual_ref_call
         elif oopspec_name.startswith('libffi_'):
@@ -1059,28 +1061,32 @@
                             [c_func] + [varoftype(T) for T in argtypes],
                             varoftype(resulttype))
         calldescr = self.callcontrol.getcalldescr(op, oopspecindex)
-        func = heaptracker.adr2int(
-            llmemory.cast_ptr_to_adr(c_func.value))
+        if isinstance(c_func.value, str):    # in tests only
+            func = c_func.value
+        else:
+            func = heaptracker.adr2int(
+                llmemory.cast_ptr_to_adr(c_func.value))
         _callinfo_for_oopspec[oopspecindex] = calldescr, func
 
     def _handle_stroruni_call(self, op, oopspec_name, args):
-        if args[0].concretetype.TO == rstr.STR:
+        SoU = args[0].concretetype     # Ptr(STR) or Ptr(UNICODE)
+        if SoU.TO == rstr.STR:
             dict = {"stroruni.concat": EffectInfo.OS_STR_CONCAT,
                     "stroruni.slice":  EffectInfo.OS_STR_SLICE,
                     "stroruni.equal":  EffectInfo.OS_STR_EQUAL,
                     }
-        elif args[0].concretetype.TO == rstr.UNICODE:
+            CHR = lltype.Char
+        elif SoU.TO == rstr.UNICODE:
             dict = {"stroruni.concat": EffectInfo.OS_UNI_CONCAT,
                     "stroruni.slice":  EffectInfo.OS_UNI_SLICE,
                     "stroruni.equal":  EffectInfo.OS_UNI_EQUAL,
                     }
+            CHR = lltype.UniChar
         else:
             assert 0, "args[0].concretetype must be STR or UNICODE"
         #
         if oopspec_name == "stroruni.equal":
-            SoU = args[0].concretetype     # Ptr(STR) or Ptr(UNICODE)
             for otherindex, othername, argtypes, resulttype in [
-
                 (EffectInfo.OS_STREQ_SLICE_CHECKNULL,
                      "str.eq_slice_checknull",
                      [SoU, lltype.Signed, lltype.Signed, SoU],
@@ -1091,7 +1097,7 @@
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_SLICE_CHAR,
                      "str.eq_slice_char",
-                     [SoU, lltype.Signed, lltype.Signed, lltype.Char],
+                     [SoU, lltype.Signed, lltype.Signed, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_NONNULL,
                      "str.eq_nonnull",
@@ -1099,22 +1105,27 @@
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_NONNULL_CHAR,
                      "str.eq_nonnull_char",
-                     [SoU, lltype.Char],
+                     [SoU, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_CHECKNULL_CHAR,
                      "str.eq_checknull_char",
-                     [SoU, lltype.Char],
+                     [SoU, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_LENGTHOK,
                      "str.eq_lengthok",
                      [SoU, SoU],
                      lltype.Signed),
                 ]:
+                if args[0].concretetype.TO == rstr.UNICODE:
+                    otherindex += EffectInfo._OS_offset_uni
                 self._register_extra_helper(otherindex, othername,
                                             argtypes, resulttype)
         #
         return self._handle_oopspec_call(op, args, dict[oopspec_name])
 
+    def _handle_str2unicode_call(self, op, oopspec_name, args):
+        return self._handle_oopspec_call(op, args, EffectInfo.OS_STR2UNICODE)
+
     # ----------
     # VirtualRefs.
 

Modified: pypy/branch/jitffi/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/branch/jitffi/pypy/jit/codewriter/test/test_jtransform.py	Mon Oct 18 11:22:20 2010
@@ -77,7 +77,32 @@
 class FakeBuiltinCallControl:
     def guess_call_kind(self, op):
         return 'builtin'
-    def getcalldescr(self, op, oopspecindex):
+    def getcalldescr(self, op, oopspecindex=None):
+        assert oopspecindex is not None    # in this test
+        EI = effectinfo.EffectInfo
+        if oopspecindex != EI.OS_ARRAYCOPY:
+            PSTR = lltype.Ptr(rstr.STR)
+            PUNICODE = lltype.Ptr(rstr.UNICODE)
+            INT = lltype.Signed
+            UNICHAR = lltype.UniChar
+            argtypes = {
+             EI.OS_STR2UNICODE:([PSTR], PUNICODE),
+             EI.OS_STR_CONCAT: ([PSTR, PSTR], PSTR),
+             EI.OS_STR_SLICE:  ([PSTR, INT, INT], PSTR),
+             EI.OS_UNI_CONCAT: ([PUNICODE, PUNICODE], PUNICODE),
+             EI.OS_UNI_SLICE:  ([PUNICODE, INT, INT], PUNICODE),
+             EI.OS_UNI_EQUAL:  ([PUNICODE, PUNICODE], lltype.Bool),
+             EI.OS_UNIEQ_SLICE_CHECKNULL:([PUNICODE, INT, INT, PUNICODE], INT),
+             EI.OS_UNIEQ_SLICE_NONNULL:  ([PUNICODE, INT, INT, PUNICODE], INT),
+             EI.OS_UNIEQ_SLICE_CHAR:     ([PUNICODE, INT, INT, UNICHAR], INT),
+             EI.OS_UNIEQ_NONNULL:        ([PUNICODE, PUNICODE], INT),
+             EI.OS_UNIEQ_NONNULL_CHAR:   ([PUNICODE, UNICHAR], INT),
+             EI.OS_UNIEQ_CHECKNULL_CHAR: ([PUNICODE, UNICHAR], INT),
+             EI.OS_UNIEQ_LENGTHOK:       ([PUNICODE, PUNICODE], INT),
+            }
+            argtypes = argtypes[oopspecindex]
+            assert argtypes[0] == [v.concretetype for v in op.args[1:]]
+            assert argtypes[1] == op.result.concretetype
         return 'calldescr-%d' % oopspecindex
     def calldescr_canraise(self, calldescr):
         return False
@@ -766,6 +791,46 @@
     assert op1.args[3] == ListOfKind('ref', [v1])
     assert op1.result == v4
 
+def test_str2unicode():
+    # test that the oopspec is present and correctly transformed
+    PSTR = lltype.Ptr(rstr.STR)
+    PUNICODE = lltype.Ptr(rstr.UNICODE)
+    FUNC = lltype.FuncType([PSTR], PUNICODE)
+    func = lltype.functionptr(FUNC, 'll_str2unicode',
+                            _callable=rstr.LLHelpers.ll_str2unicode)
+    v1 = varoftype(PSTR)
+    v2 = varoftype(PUNICODE)
+    op = SpaceOperation('direct_call', [const(func), v1], v2)
+    tr = Transformer(FakeCPU(), FakeBuiltinCallControl())
+    op1 = tr.rewrite_operation(op)
+    assert op1.opname == 'residual_call_r_r'
+    assert op1.args[0].value == func
+    assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_STR2UNICODE
+    assert op1.args[2] == ListOfKind('ref', [v1])
+    assert op1.result == v2
+
+def test_unicode_eq_checknull_char():
+    # test that the oopspec is present and correctly transformed
+    PUNICODE = lltype.Ptr(rstr.UNICODE)
+    FUNC = lltype.FuncType([PUNICODE, PUNICODE], lltype.Bool)
+    func = lltype.functionptr(FUNC, 'll_streq',
+                              _callable=rstr.LLHelpers.ll_streq)
+    v1 = varoftype(PUNICODE)
+    v2 = varoftype(PUNICODE)
+    v3 = varoftype(lltype.Bool)
+    op = SpaceOperation('direct_call', [const(func), v1, v2], v3)
+    tr = Transformer(FakeCPU(), FakeBuiltinCallControl())
+    op1 = tr.rewrite_operation(op)
+    assert op1.opname == 'residual_call_r_i'
+    assert op1.args[0].value == func
+    assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_UNI_EQUAL
+    assert op1.args[2] == ListOfKind('ref', [v1, v2])
+    assert op1.result == v3
+    # test that the OS_UNIEQ_* functions are registered
+    cifo = effectinfo._callinfo_for_oopspec
+    assert effectinfo.EffectInfo.OS_UNIEQ_SLICE_NONNULL in cifo
+    assert effectinfo.EffectInfo.OS_UNIEQ_CHECKNULL_CHAR in cifo
+
 def test_list_ll_arraycopy():
     from pypy.rlib.rgc import ll_arraycopy
     LIST = lltype.GcArray(lltype.Signed)

Modified: pypy/branch/jitffi/pypy/jit/metainterp/history.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/history.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/history.py	Mon Oct 18 11:22:20 2010
@@ -698,6 +698,21 @@
     return result
 _const_ptr_for_string = {}
 
+def get_const_ptr_for_unicode(s):
+    from pypy.rpython.annlowlevel import llunicode
+    if not we_are_translated():
+        try:
+            return _const_ptr_for_unicode[s]
+        except KeyError:
+            pass
+    if isinstance(s, str):
+        s = unicode(s)
+    result = ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, llunicode(s)))
+    if not we_are_translated():
+        _const_ptr_for_unicode[s] = result
+    return result
+_const_ptr_for_unicode = {}
+
 # ____________________________________________________________
 
 # The TreeLoop class contains a loop or a generalized loop, i.e. a tree

Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimize_nopspec.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimize_nopspec.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimize_nopspec.py	Mon Oct 18 11:22:20 2010
@@ -14,6 +14,9 @@
 def _optimize_loop(metainterp_sd, old_loop_tokens, loop):
     cpu = metainterp_sd.cpu
     metainterp_sd.logger_noopt.log_loop(loop.inputargs, loop.operations)
+    # XXX the following lines are probably still needed, to discard invalid
+    # loops. bit silly to run a full perfect specialization and throw the
+    # result away.
     finder = PerfectSpecializationFinder(cpu)
     finder.find_nodes_loop(loop, False)
     if old_loop_tokens:
@@ -31,6 +34,7 @@
 def _optimize_bridge(metainterp_sd, old_loop_tokens, bridge):
     cpu = metainterp_sd.cpu    
     metainterp_sd.logger_noopt.log_loop(bridge.inputargs, bridge.operations)
+    # XXX same comment as above applies
     finder = BridgeSpecializationFinder(cpu)
     finder.find_nodes_bridge(bridge)
     if old_loop_tokens:

Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/intbounds.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/intbounds.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/intbounds.py	Mon Oct 18 11:22:20 2010
@@ -191,6 +191,7 @@
         v1.intbound.make_ge(IntLowerBound(0))
 
     optimize_STRLEN = optimize_ARRAYLEN_GC
+    optimize_UNICODELEN = optimize_ARRAYLEN_GC
 
     def make_int_lt(self, box1, box2):
         v1 = self.getvalue(box1)

Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/rewrite.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/rewrite.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/rewrite.py	Mon Oct 18 11:22:20 2010
@@ -3,6 +3,7 @@
 from pypy.jit.metainterp.history import ConstInt
 from pypy.jit.metainterp.optimizeutil import _findall
 from pypy.jit.metainterp.resoperation import rop, ResOperation
+from pypy.jit.codewriter.effectinfo import EffectInfo
 
 class OptRewrite(Optimization):
     """Rewrite operations into equivalent, cheaper operations.
@@ -326,8 +327,37 @@
 ##            return
 ##        self.emit_operation(op)
 
-optimize_ops = _findall(OptRewrite, 'optimize_')
-        
+    def optimize_CALL(self, op):
+        # dispatch based on 'oopspecindex' to a method that handles
+        # specifically the given oopspec call.  For non-oopspec calls,
+        # oopspecindex is just zero.
+        effectinfo = op.getdescr().get_extra_info()
+        if effectinfo is not None:
+            oopspecindex = effectinfo.oopspecindex
+            if oopspecindex == EffectInfo.OS_ARRAYCOPY:
+                if self._optimize_CALL_ARRAYCOPY(op):
+                    return
+        self.emit_operation(op)
 
+    def _optimize_CALL_ARRAYCOPY(self, op):
+        source_value = self.getvalue(op.getarg(1))
+        dest_value = self.getvalue(op.getarg(2))
+        source_start_box = self.get_constant_box(op.getarg(3))
+        dest_start_box = self.get_constant_box(op.getarg(4))
+        length = self.get_constant_box(op.getarg(5))
+        if (source_value.is_virtual() and source_start_box and dest_start_box
+            and length and dest_value.is_virtual()):
+            # XXX optimize the case where dest value is not virtual,
+            #     but we still can avoid a mess
+            source_start = source_start_box.getint()
+            dest_start = dest_start_box.getint()
+            for index in range(length.getint()):
+                val = source_value.getitem(index + source_start)
+                dest_value.setitem(index + dest_start, val)
+            return True
+        if length and length.getint() == 0:
+            return True # 0-length arraycopy
+        return False
+
+optimize_ops = _findall(OptRewrite, 'optimize_')
 
-        

Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/string.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/string.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/string.py	Mon Oct 18 11:22:20 2010
@@ -3,6 +3,7 @@
 from pypy.jit.metainterp.history import Box, BoxInt, BoxPtr
 from pypy.jit.metainterp.history import Const, ConstInt, ConstPtr
 from pypy.jit.metainterp.history import get_const_ptr_for_string
+from pypy.jit.metainterp.history import get_const_ptr_for_unicode
 from pypy.jit.metainterp.resoperation import rop, ResOperation
 from pypy.jit.metainterp.optimizeopt import optimizer, virtualize
 from pypy.jit.metainterp.optimizeopt.optimizer import CONST_0, CONST_1
@@ -11,56 +12,103 @@
 from pypy.jit.codewriter.effectinfo import EffectInfo, callinfo_for_oopspec
 from pypy.jit.codewriter import heaptracker
 from pypy.rlib.unroll import unrolling_iterable
+from pypy.rlib.objectmodel import specialize
+
+
+class StrOrUnicode(object):
+    def __init__(self, LLTYPE, hlstr, emptystr, chr,
+                 NEWSTR, STRLEN, STRGETITEM, STRSETITEM, COPYSTRCONTENT,
+                 OS_offset):
+        self.LLTYPE = LLTYPE
+        self.hlstr = hlstr
+        self.emptystr = emptystr
+        self.chr = chr
+        self.NEWSTR = NEWSTR
+        self.STRLEN = STRLEN
+        self.STRGETITEM = STRGETITEM
+        self.STRSETITEM = STRSETITEM
+        self.COPYSTRCONTENT = COPYSTRCONTENT
+        self.OS_offset = OS_offset
+
+    def _freeze_(self):
+        return True
+
+mode_string = StrOrUnicode(rstr.STR, annlowlevel.hlstr, '', chr,
+                           rop.NEWSTR, rop.STRLEN, rop.STRGETITEM,
+                           rop.STRSETITEM, rop.COPYSTRCONTENT, 0)
+mode_unicode = StrOrUnicode(rstr.UNICODE, annlowlevel.hlunicode, u'', unichr,
+                            rop.NEWUNICODE, rop.UNICODELEN, rop.UNICODEGETITEM,
+                            rop.UNICODESETITEM, rop.COPYUNICODECONTENT,
+                            EffectInfo._OS_offset_uni)
+
+# ____________________________________________________________
 
 
 class __extend__(optimizer.OptValue):
     """New methods added to the base class OptValue for this file."""
 
-    def getstrlen(self, newoperations):
-        s = self.get_constant_string()
-        if s is not None:
-            return ConstInt(len(s))
+    def getstrlen(self, newoperations, mode):
+        if mode is mode_string:
+            s = self.get_constant_string_spec(mode_string)
+            if s is not None:
+                return ConstInt(len(s))
         else:
-            if newoperations is None:
-                return None
-            self.ensure_nonnull()
-            box = self.force_box()
-            lengthbox = BoxInt()
-            newoperations.append(ResOperation(rop.STRLEN, [box], lengthbox))
-            return lengthbox
+            s = self.get_constant_string_spec(mode_unicode)
+            if s is not None:
+                return ConstInt(len(s))
+        if newoperations is None:
+            return None
+        self.ensure_nonnull()
+        box = self.force_box()
+        lengthbox = BoxInt()
+        newoperations.append(ResOperation(mode.STRLEN, [box], lengthbox))
+        return lengthbox
 
-    def get_constant_string(self):
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
         if self.is_constant():
-            s = self.box.getref(lltype.Ptr(rstr.STR))
-            return annlowlevel.hlstr(s)
+            s = self.box.getref(lltype.Ptr(mode.LLTYPE))
+            return mode.hlstr(s)
         else:
             return None
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
         # Copies the pointer-to-string 'self' into the target string
         # given by 'targetbox', at the specified offset.  Returns the offset
         # at the end of the copy.
-        lengthbox = self.getstrlen(newoperations)
+        lengthbox = self.getstrlen(newoperations, mode)
         srcbox = self.force_box()
         return copy_str_content(newoperations, srcbox, targetbox,
-                                CONST_0, offsetbox, lengthbox)
+                                CONST_0, offsetbox, lengthbox, mode)
 
 
 class VAbstractStringValue(virtualize.AbstractVirtualValue):
-    _attrs_ = ()
+    _attrs_ = ('mode',)
+
+    def __init__(self, optimizer, keybox, source_op, mode):
+        virtualize.AbstractVirtualValue.__init__(self, optimizer, keybox,
+                                                 source_op)
+        self.mode = mode
 
     def _really_force(self):
-        s = self.get_constant_string()
-        if s is not None:
-            c_s = get_const_ptr_for_string(s)
-            self.make_constant(c_s)
-            return
+        if self.mode is mode_string:
+            s = self.get_constant_string_spec(mode_string)
+            if s is not None:
+                c_s = get_const_ptr_for_string(s)
+                self.make_constant(c_s)
+                return
+        else:
+            s = self.get_constant_string_spec(mode_unicode)
+            if s is not None:
+                c_s = get_const_ptr_for_unicode(s)
+                self.make_constant(c_s)
+                return
         assert self.source_op is not None
         self.box = box = self.source_op.result
         newoperations = self.optimizer.newoperations
-        lengthbox = self.getstrlen(newoperations)
-        newoperations.append(ResOperation(rop.NEWSTR, [lengthbox], box))
-        self.string_copy_parts(newoperations, box, CONST_0)
+        lengthbox = self.getstrlen(newoperations, self.mode)
+        newoperations.append(ResOperation(self.mode.NEWSTR, [lengthbox], box))
+        self.string_copy_parts(newoperations, box, CONST_0, self.mode)
 
 
 class VStringPlainValue(VAbstractStringValue):
@@ -74,7 +122,7 @@
         assert 0 <= start <= stop <= len(longerlist)
         self._chars = longerlist[start:stop]
 
-    def getstrlen(self, _):
+    def getstrlen(self, _, mode):
         if self._lengthbox is None:
             self._lengthbox = ConstInt(len(self._chars))
         return self._lengthbox
@@ -86,18 +134,21 @@
         assert isinstance(charvalue, optimizer.OptValue)
         self._chars[index] = charvalue
 
-    def get_constant_string(self):
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
         for c in self._chars:
             if c is optimizer.CVAL_UNINITIALIZED_ZERO or not c.is_constant():
                 return None
-        return ''.join([chr(c.box.getint()) for c in self._chars])
+        return mode.emptystr.join([mode.chr(c.box.getint())
+                                   for c in self._chars])
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
         for i in range(len(self._chars)):
             charbox = self._chars[i].force_box()
-            newoperations.append(ResOperation(rop.STRSETITEM, [targetbox,
-                                                               offsetbox,
-                                                               charbox], None))
+            newoperations.append(ResOperation(mode.STRSETITEM, [targetbox,
+                                                                offsetbox,
+                                                                charbox],
+                                              None))
             offsetbox = _int_add(newoperations, offsetbox, CONST_1)
         return offsetbox
 
@@ -109,7 +160,7 @@
                 value.get_args_for_fail(modifier)
 
     def _make_virtual(self, modifier):
-        return modifier.make_vstrplain()
+        return modifier.make_vstrplain(self.mode is mode_unicode)
 
 
 class VStringConcatValue(VAbstractStringValue):
@@ -120,23 +171,24 @@
         self.right = right
         self.lengthbox = lengthbox
 
-    def getstrlen(self, _):
+    def getstrlen(self, _, mode):
         return self.lengthbox
 
-    def get_constant_string(self):
-        s1 = self.left.get_constant_string()
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
+        s1 = self.left.get_constant_string_spec(mode)
         if s1 is None:
             return None
-        s2 = self.right.get_constant_string()
+        s2 = self.right.get_constant_string_spec(mode)
         if s2 is None:
             return None
         return s1 + s2
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
         offsetbox = self.left.string_copy_parts(newoperations, targetbox,
-                                                offsetbox)
+                                                offsetbox, mode)
         offsetbox = self.right.string_copy_parts(newoperations, targetbox,
-                                                 offsetbox)
+                                                 offsetbox, mode)
         return offsetbox
 
     def get_args_for_fail(self, modifier):
@@ -150,7 +202,7 @@
             self.right.get_args_for_fail(modifier)
 
     def _make_virtual(self, modifier):
-        return modifier.make_vstrconcat()
+        return modifier.make_vstrconcat(self.mode is mode_unicode)
 
 
 class VStringSliceValue(VAbstractStringValue):
@@ -162,12 +214,13 @@
         self.vstart = vstart
         self.vlength = vlength
 
-    def getstrlen(self, _):
+    def getstrlen(self, _, mode):
         return self.vlength.force_box()
 
-    def get_constant_string(self):
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
         if self.vstart.is_constant() and self.vlength.is_constant():
-            s1 = self.vstr.get_constant_string()
+            s1 = self.vstr.get_constant_string_spec(mode)
             if s1 is None:
                 return None
             start = self.vstart.box.getint()
@@ -177,12 +230,12 @@
             return s1[start : start + length]
         return None
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
-        lengthbox = self.getstrlen(newoperations)
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
+        lengthbox = self.getstrlen(newoperations, mode)
         return copy_str_content(newoperations,
                                 self.vstr.force_box(), targetbox,
                                 self.vstart.force_box(), offsetbox,
-                                lengthbox)
+                                lengthbox, mode)
 
     def get_args_for_fail(self, modifier):
         if self.box is None and not modifier.already_seen_virtual(self.keybox):
@@ -195,11 +248,11 @@
             self.vlength.get_args_for_fail(modifier)
 
     def _make_virtual(self, modifier):
-        return modifier.make_vstrslice()
+        return modifier.make_vstrslice(self.mode is mode_unicode)
 
 
 def copy_str_content(newoperations, srcbox, targetbox,
-                     srcoffsetbox, offsetbox, lengthbox):
+                     srcoffsetbox, offsetbox, lengthbox, mode):
     if isinstance(srcbox, ConstPtr) and isinstance(srcoffsetbox, Const):
         M = 5
     else:
@@ -208,17 +261,18 @@
         # up to M characters are done "inline", i.e. with STRGETITEM/STRSETITEM
         # instead of just a COPYSTRCONTENT.
         for i in range(lengthbox.value):
-            charbox = _strgetitem(newoperations, srcbox, srcoffsetbox)
+            charbox = _strgetitem(newoperations, srcbox, srcoffsetbox, mode)
             srcoffsetbox = _int_add(newoperations, srcoffsetbox, CONST_1)
-            newoperations.append(ResOperation(rop.STRSETITEM, [targetbox,
-                                                               offsetbox,
-                                                               charbox], None))
+            newoperations.append(ResOperation(mode.STRSETITEM, [targetbox,
+                                                                offsetbox,
+                                                                charbox],
+                                              None))
             offsetbox = _int_add(newoperations, offsetbox, CONST_1)
     else:
         nextoffsetbox = _int_add(newoperations, offsetbox, lengthbox)
-        op = ResOperation(rop.COPYSTRCONTENT, [srcbox, targetbox,
-                                               srcoffsetbox, offsetbox,
-                                               lengthbox], None)
+        op = ResOperation(mode.COPYSTRCONTENT, [srcbox, targetbox,
+                                                srcoffsetbox, offsetbox,
+                                                lengthbox], None)
         newoperations.append(op)
         offsetbox = nextoffsetbox
     return offsetbox
@@ -245,12 +299,16 @@
     newoperations.append(ResOperation(rop.INT_SUB, [box1, box2], resbox))
     return resbox
 
-def _strgetitem(newoperations, strbox, indexbox):
+def _strgetitem(newoperations, strbox, indexbox, mode):
     if isinstance(strbox, ConstPtr) and isinstance(indexbox, ConstInt):
-        s = strbox.getref(lltype.Ptr(rstr.STR))
-        return ConstInt(ord(s.chars[indexbox.getint()]))
+        if mode is mode_string:
+            s = strbox.getref(lltype.Ptr(rstr.STR))
+            return ConstInt(ord(s.chars[indexbox.getint()]))
+        else:
+            s = strbox.getref(lltype.Ptr(rstr.UNICODE))
+            return ConstInt(ord(s.chars[indexbox.getint()]))
     resbox = BoxInt()
-    newoperations.append(ResOperation(rop.STRGETITEM, [strbox, indexbox],
+    newoperations.append(ResOperation(mode.STRGETITEM, [strbox, indexbox],
                                       resbox))
     return resbox
 
@@ -258,62 +316,34 @@
 class OptString(optimizer.Optimization):
     "Handling of strings and unicodes."
 
-    def make_vstring_plain(self, box, source_op=None):
-        vvalue = VStringPlainValue(self.optimizer, box, source_op)
+    def make_vstring_plain(self, box, source_op, mode):
+        vvalue = VStringPlainValue(self.optimizer, box, source_op, mode)
         self.make_equal_to(box, vvalue)
         return vvalue
 
-    def make_vstring_concat(self, box, source_op=None):
-        vvalue = VStringConcatValue(self.optimizer, box, source_op)
+    def make_vstring_concat(self, box, source_op, mode):
+        vvalue = VStringConcatValue(self.optimizer, box, source_op, mode)
         self.make_equal_to(box, vvalue)
         return vvalue
 
-    def make_vstring_slice(self, box, source_op=None):
-        vvalue = VStringSliceValue(self.optimizer, box, source_op)
+    def make_vstring_slice(self, box, source_op, mode):
+        vvalue = VStringSliceValue(self.optimizer, box, source_op, mode)
         self.make_equal_to(box, vvalue)
         return vvalue
 
-    def optimize_CALL(self, op):
-        # dispatch based on 'oopspecindex' to a method that handles
-        # specifically the given oopspec call.  For non-oopspec calls,
-        # oopspecindex is just zero.
-        effectinfo = op.getdescr().get_extra_info()
-        if effectinfo is not None:
-            oopspecindex = effectinfo.oopspecindex
-            for value, meth in opt_call_oopspec_ops:
-                if oopspecindex == value:
-                    if meth(self, op):
-                        return
-        self.emit_operation(op)
-
-    def opt_call_oopspec_ARRAYCOPY(self, op):
-        source_value = self.getvalue(op.getarg(1))
-        dest_value = self.getvalue(op.getarg(2))
-        source_start_box = self.get_constant_box(op.getarg(3))
-        dest_start_box = self.get_constant_box(op.getarg(4))
-        length = self.get_constant_box(op.getarg(5))
-        if (source_value.is_virtual() and source_start_box and dest_start_box
-            and length and dest_value.is_virtual()):
-            # XXX optimize the case where dest value is not virtual,
-            #     but we still can avoid a mess
-            source_start = source_start_box.getint()
-            dest_start = dest_start_box.getint()
-            for index in range(length.getint()):
-                val = source_value.getitem(index + source_start)
-                dest_value.setitem(index + dest_start, val)
-            return True
-        if length and length.getint() == 0:
-            return True # 0-length arraycopy
-        return False
-
     def optimize_NEWSTR(self, op):
+        self._optimize_NEWSTR(op, mode_string)
+    def optimize_NEWUNICODE(self, op):
+        self._optimize_NEWSTR(op, mode_unicode)
+
+    def _optimize_NEWSTR(self, op, mode):
         length_box = self.get_constant_box(op.getarg(0))
         if length_box:
             # if the original 'op' did not have a ConstInt as argument,
             # build a new one with the ConstInt argument
             if not isinstance(op.getarg(0), ConstInt):
-                op = ResOperation(rop.NEWSTR, [length_box], op.result)
-            vvalue = self.make_vstring_plain(op.result, op)
+                op = ResOperation(mode.NEWSTR, [length_box], op.result)
+            vvalue = self.make_vstring_plain(op.result, op, mode)
             vvalue.setup(length_box.getint())
         else:
             self.getvalue(op.result).ensure_nonnull()
@@ -329,13 +359,20 @@
         value.ensure_nonnull()
         self.emit_operation(op)
 
+    optimize_UNICODESETITEM = optimize_STRSETITEM
+
     def optimize_STRGETITEM(self, op):
+        self._optimize_STRGETITEM(op, mode_string)
+    def optimize_UNICODEGETITEM(self, op):
+        self._optimize_STRGETITEM(op, mode_unicode)
+
+    def _optimize_STRGETITEM(self, op, mode):
         value = self.getvalue(op.getarg(0))
         vindex = self.getvalue(op.getarg(1))
-        vresult = self.strgetitem(value, vindex)
+        vresult = self.strgetitem(value, vindex, mode)
         self.make_equal_to(op.result, vresult)
 
-    def strgetitem(self, value, vindex):
+    def strgetitem(self, value, vindex, mode):
         value.ensure_nonnull()
         #
         if value.is_virtual() and isinstance(value, VStringSliceValue):
@@ -350,28 +387,71 @@
                 return value.getitem(vindex.box.getint())
         #
         resbox = _strgetitem(self.optimizer.newoperations,
-                             value.force_box(),vindex.force_box())
+                             value.force_box(),vindex.force_box(), mode)
         return self.getvalue(resbox)
 
     def optimize_STRLEN(self, op):
+        self._optimize_STRLEN(op, mode_string)
+    def optimize_UNICODELEN(self, op):
+        self._optimize_STRLEN(op, mode_unicode)
+
+    def _optimize_STRLEN(self, op, mode):
         value = self.getvalue(op.getarg(0))
-        lengthbox = value.getstrlen(self.optimizer.newoperations)
+        lengthbox = value.getstrlen(self.optimizer.newoperations, mode)
         self.make_equal_to(op.result, self.getvalue(lengthbox))
 
-    def opt_call_oopspec_STR_CONCAT(self, op):
+    def optimize_CALL(self, op):
+        # dispatch based on 'oopspecindex' to a method that handles
+        # specifically the given oopspec call.  For non-oopspec calls,
+        # oopspecindex is just zero.
+        effectinfo = op.getdescr().get_extra_info()
+        if effectinfo is not None:
+            oopspecindex = effectinfo.oopspecindex
+            for value, meth in opt_call_oopspec_ops:
+                if oopspecindex == value:      # a match with the OS_STR_xxx
+                    if meth(self, op, mode_string):
+                        return
+                    break
+                if oopspecindex == value + EffectInfo._OS_offset_uni:
+                    # a match with the OS_UNI_xxx
+                    if meth(self, op, mode_unicode):
+                        return
+                    break
+            if oopspecindex == EffectInfo.OS_STR2UNICODE:
+                if self.opt_call_str_STR2UNICODE(op):
+                    return
+        self.emit_operation(op)
+
+    def opt_call_str_STR2UNICODE(self, op):
+        # Constant-fold unicode("constant string").
+        # More generally, supporting non-constant but virtual cases is
+        # not obvious, because of the exception UnicodeDecodeError that
+        # can be raised by ll_str2unicode()
+        varg = self.getvalue(op.getarg(1))
+        s = varg.get_constant_string_spec(mode_string)
+        if s is None:
+            return False
+        try:
+            u = unicode(s)
+        except UnicodeDecodeError:
+            return False
+        self.make_constant(op.result, get_const_ptr_for_unicode(u))
+        return True
+
+    def opt_call_stroruni_STR_CONCAT(self, op, mode):
         vleft = self.getvalue(op.getarg(1))
         vright = self.getvalue(op.getarg(2))
         vleft.ensure_nonnull()
         vright.ensure_nonnull()
         newoperations = self.optimizer.newoperations
-        len1box = vleft.getstrlen(newoperations)
-        len2box = vright.getstrlen(newoperations)
+        len1box = vleft.getstrlen(newoperations, mode)
+        len2box = vright.getstrlen(newoperations, mode)
         lengthbox = _int_add(newoperations, len1box, len2box)
-        value = self.make_vstring_concat(op.result, op)
+        value = self.make_vstring_concat(op.result, op, mode)
         value.setup(vleft, vright, lengthbox)
         return True
 
-    def opt_call_oopspec_STR_SLICE(self, op):
+    def opt_call_stroruni_STR_SLICE(self, op, mode):
         newoperations = self.optimizer.newoperations
         vstr = self.getvalue(op.getarg(1))
         vstart = self.getvalue(op.getarg(2))
@@ -380,7 +460,7 @@
         if (isinstance(vstr, VStringPlainValue) and vstart.is_constant()
             and vstop.is_constant()):
             # slicing with constant bounds of a VStringPlainValue
-            value = self.make_vstring_plain(op.result, op)
+            value = self.make_vstring_plain(op.result, op, mode)
             value.setup_slice(vstr._chars, vstart.box.getint(),
                                            vstop.box.getint())
             return True
@@ -398,16 +478,16 @@
                                 vstart.force_box())
             vstart = self.getvalue(startbox)
         #
-        value = self.make_vstring_slice(op.result, op)
+        value = self.make_vstring_slice(op.result, op, mode)
         value.setup(vstr, vstart, self.getvalue(lengthbox))
         return True
 
-    def opt_call_oopspec_STR_EQUAL(self, op):
+    def opt_call_stroruni_STR_EQUAL(self, op, mode):
         v1 = self.getvalue(op.getarg(1))
         v2 = self.getvalue(op.getarg(2))
         #
-        l1box = v1.getstrlen(None)
-        l2box = v2.getstrlen(None)
+        l1box = v1.getstrlen(None, mode)
+        l2box = v2.getstrlen(None, mode)
         if (l1box is not None and l2box is not None and
             isinstance(l1box, ConstInt) and
             isinstance(l2box, ConstInt) and
@@ -416,13 +496,13 @@
             self.make_constant(op.result, CONST_0)
             return True
         #
-        if self.handle_str_equal_level1(v1, v2, op.result):
+        if self.handle_str_equal_level1(v1, v2, op.result, mode):
             return True
-        if self.handle_str_equal_level1(v2, v1, op.result):
+        if self.handle_str_equal_level1(v2, v1, op.result, mode):
             return True
-        if self.handle_str_equal_level2(v1, v2, op.result):
+        if self.handle_str_equal_level2(v1, v2, op.result, mode):
             return True
-        if self.handle_str_equal_level2(v2, v1, op.result):
+        if self.handle_str_equal_level2(v2, v1, op.result, mode):
             return True
         #
         if v1.is_nonnull() and v2.is_nonnull():
@@ -434,37 +514,37 @@
             else:
                 do = EffectInfo.OS_STREQ_NONNULL
             self.generate_modified_call(do, [v1.force_box(),
-                                             v2.force_box()], op.result)
+                                             v2.force_box()], op.result, mode)
             return True
         return False
 
-    def handle_str_equal_level1(self, v1, v2, resultbox):
-        l2box = v2.getstrlen(None)
+    def handle_str_equal_level1(self, v1, v2, resultbox, mode):
+        l2box = v2.getstrlen(None, mode)
         if isinstance(l2box, ConstInt):
             if l2box.value == 0:
-                lengthbox = v1.getstrlen(self.optimizer.newoperations)
+                lengthbox = v1.getstrlen(self.optimizer.newoperations, mode)
                 seo = self.optimizer.send_extra_operation
                 seo(ResOperation(rop.INT_EQ, [lengthbox, CONST_0], resultbox))
                 return True
             if l2box.value == 1:
-                l1box = v1.getstrlen(None)
+                l1box = v1.getstrlen(None, mode)
                 if isinstance(l1box, ConstInt) and l1box.value == 1:
                     # comparing two single chars
-                    vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO)
-                    vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO)
+                    vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO, mode)
+                    vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO, mode)
                     seo = self.optimizer.send_extra_operation
                     seo(ResOperation(rop.INT_EQ, [vchar1.force_box(),
                                                   vchar2.force_box()],
                                      resultbox))
                     return True
                 if isinstance(v1, VStringSliceValue):
-                    vchar = self.strgetitem(v2, optimizer.CVAL_ZERO)
+                    vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode)
                     do = EffectInfo.OS_STREQ_SLICE_CHAR
                     self.generate_modified_call(do, [v1.vstr.force_box(),
                                                      v1.vstart.force_box(),
                                                      v1.vlength.force_box(),
                                                      vchar.force_box()],
-                                                resultbox)
+                                                resultbox, mode)
                     return True
         #
         if v2.is_null():
@@ -482,17 +562,18 @@
         #
         return False
 
-    def handle_str_equal_level2(self, v1, v2, resultbox):
-        l2box = v2.getstrlen(None)
+    def handle_str_equal_level2(self, v1, v2, resultbox, mode):
+        l2box = v2.getstrlen(None, mode)
         if isinstance(l2box, ConstInt):
             if l2box.value == 1:
-                vchar = self.strgetitem(v2, optimizer.CVAL_ZERO)
+                vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode)
                 if v1.is_nonnull():
                     do = EffectInfo.OS_STREQ_NONNULL_CHAR
                 else:
                     do = EffectInfo.OS_STREQ_CHECKNULL_CHAR
                 self.generate_modified_call(do, [v1.force_box(),
-                                                 vchar.force_box()], resultbox)
+                                                 vchar.force_box()], resultbox,
+                                            mode)
                 return True
         #
         if v1.is_virtual() and isinstance(v1, VStringSliceValue):
@@ -503,11 +584,12 @@
             self.generate_modified_call(do, [v1.vstr.force_box(),
                                              v1.vstart.force_box(),
                                              v1.vlength.force_box(),
-                                             v2.force_box()], resultbox)
+                                             v2.force_box()], resultbox, mode)
             return True
         return False
 
-    def generate_modified_call(self, oopspecindex, args, result):
+    def generate_modified_call(self, oopspecindex, args, result, mode):
+        oopspecindex += mode.OS_offset
         calldescr, func = callinfo_for_oopspec(oopspecindex)
         op = ResOperation(rop.CALL, [ConstInt(func)] + args, result,
                           descr=calldescr)
@@ -525,7 +607,7 @@
 optimize_ops = _findall(OptString, 'optimize_')
 
 def _findall_call_oopspec():
-    prefix = 'opt_call_oopspec_'
+    prefix = 'opt_call_stroruni_'
     result = []
     for name in dir(OptString):
         if name.startswith(prefix):

Modified: pypy/branch/jitffi/pypy/jit/metainterp/resume.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/resume.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/resume.py	Mon Oct 18 11:22:20 2010
@@ -255,13 +255,19 @@
     def make_varray(self, arraydescr):
         return VArrayInfo(arraydescr)
 
-    def make_vstrplain(self):
+    def make_vstrplain(self, is_unicode=False):
+        if is_unicode:
+            return VUniPlainInfo()
         return VStrPlainInfo()
 
-    def make_vstrconcat(self):
+    def make_vstrconcat(self, is_unicode=False):
+        if is_unicode:
+            return VUniConcatInfo()
         return VStrConcatInfo()
 
-    def make_vstrslice(self):
+    def make_vstrslice(self, is_unicode=False):
+        if is_unicode:
+            return VUniSliceInfo()
         return VStrSliceInfo()
 
     def register_virtual_fields(self, virtualbox, fieldboxes):
@@ -550,6 +556,60 @@
         for i in self.fieldnums:
             debug_print("\t\t", str(untag(i)))
 
+
+class VUniPlainInfo(AbstractVirtualInfo):
+    """Stands for the unicode string made out of the characters of all
+    fieldnums."""
+
+    @specialize.argtype(1)
+    def allocate(self, decoder, index):
+        length = len(self.fieldnums)
+        string = decoder.allocate_unicode(length)
+        decoder.virtuals_cache[index] = string
+        for i in range(length):
+            decoder.unicode_setitem(string, i, self.fieldnums[i])
+        return string
+
+    def debug_prints(self):
+        debug_print("\tvuniplaininfo length", len(self.fieldnums))
+
+
+class VUniConcatInfo(AbstractVirtualInfo):
+    """Stands for the unicode string made out of the concatenation of two
+    other unicode strings."""
+
+    @specialize.argtype(1)
+    def allocate(self, decoder, index):
+        # xxx for blackhole resuming, this will build all intermediate
+        # strings and throw them away immediately, which is a bit sub-
+        # efficient.  Not sure we care.
+        left, right = self.fieldnums
+        string = decoder.concat_unicodes(left, right)
+        decoder.virtuals_cache[index] = string
+        return string
+
+    def debug_prints(self):
+        debug_print("\tvuniconcatinfo")
+        for i in self.fieldnums:
+            debug_print("\t\t", str(untag(i)))
+
+
+class VUniSliceInfo(AbstractVirtualInfo):
+    """Stands for the unicode string made out of slicing another
+    unicode string."""
+
+    @specialize.argtype(1)
+    def allocate(self, decoder, index):
+        largerstr, start, length = self.fieldnums
+        string = decoder.slice_unicode(largerstr, start, length)
+        decoder.virtuals_cache[index] = string
+        return string
+
+    def debug_prints(self):
+        debug_print("\tvunisliceinfo")
+        for i in self.fieldnums:
+            debug_print("\t\t", str(untag(i)))
+
 # ____________________________________________________________
 
 class AbstractResumeDataReader(object):
@@ -725,6 +785,32 @@
         return self.metainterp.execute_and_record_varargs(
             rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr)
 
+    def allocate_unicode(self, length):
+        return self.metainterp.execute_and_record(rop.NEWUNICODE,
+                                                  None, ConstInt(length))
+
+    def unicode_setitem(self, strbox, index, charnum):
+        charbox = self.decode_box(charnum, INT)
+        self.metainterp.execute_and_record(rop.UNICODESETITEM, None,
+                                           strbox, ConstInt(index), charbox)
+
+    def concat_unicodes(self, str1num, str2num):
+        calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_CONCAT)
+        str1box = self.decode_box(str1num, REF)
+        str2box = self.decode_box(str2num, REF)
+        return self.metainterp.execute_and_record_varargs(
+            rop.CALL, [ConstInt(func), str1box, str2box], calldescr)
+
+    def slice_unicode(self, strnum, startnum, lengthnum):
+        calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_SLICE)
+        strbox = self.decode_box(strnum, REF)
+        startbox = self.decode_box(startnum, INT)
+        lengthbox = self.decode_box(lengthnum, INT)
+        stopbox = self.metainterp.execute_and_record(rop.INT_ADD, None,
+                                                     startbox, lengthbox)
+        return self.metainterp.execute_and_record_varargs(
+            rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr)
+
     def setfield(self, descr, structbox, fieldnum):
         if descr.is_pointer_field():
             kind = REF
@@ -967,6 +1053,31 @@
         result = funcptr(str, start, start + length)
         return lltype.cast_opaque_ptr(llmemory.GCREF, result)
 
+    def allocate_unicode(self, length):
+        return self.cpu.bh_newunicode(length)
+
+    def unicode_setitem(self, str, index, charnum):
+        char = self.decode_int(charnum)
+        self.cpu.bh_unicodesetitem(str, index, char)
+
+    def concat_unicodes(self, str1num, str2num):
+        str1 = self.decode_ref(str1num)
+        str2 = self.decode_ref(str2num)
+        str1 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str1)
+        str2 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str2)
+        funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_CONCAT)
+        result = funcptr(str1, str2)
+        return lltype.cast_opaque_ptr(llmemory.GCREF, result)
+
+    def slice_unicode(self, strnum, startnum, lengthnum):
+        str = self.decode_ref(strnum)
+        start = self.decode_int(startnum)
+        length = self.decode_int(lengthnum)
+        str = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str)
+        funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_SLICE)
+        result = funcptr(str, start, start + length)
+        return lltype.cast_opaque_ptr(llmemory.GCREF, result)
+
     def setfield(self, descr, struct, fieldnum):
         if descr.is_pointer_field():
             newvalue = self.decode_ref(fieldnum)

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/oparser.py	Mon Oct 18 11:22:20 2010
@@ -5,7 +5,7 @@
 
 from pypy.jit.metainterp.history import TreeLoop, BoxInt, ConstInt,\
      ConstObj, ConstPtr, Box, BasicFailDescr, BoxFloat, ConstFloat,\
-     LoopToken, get_const_ptr_for_string
+     LoopToken, get_const_ptr_for_string, get_const_ptr_for_unicode
 from pypy.jit.metainterp.resoperation import rop, ResOperation, ResOpWithDescr, N_aryOp
 from pypy.jit.metainterp.typesystem import llhelper
 from pypy.jit.codewriter.heaptracker import adr2int
@@ -158,10 +158,15 @@
         except ValueError:
             if self.is_float(arg):
                 return ConstFloat(float(arg))
-            if arg.startswith('"') or arg.startswith("'"):
+            if (arg.startswith('"') or arg.startswith("'") or
+                arg.startswith('s"')):
                 # XXX ootype
-                info = arg.strip("'\"")
+                info = arg[1:].strip("'\"")
                 return get_const_ptr_for_string(info)
+            if arg.startswith('u"'):
+                # XXX ootype
+                info = arg[1:].strip("'\"")
+                return get_const_ptr_for_unicode(info)
             if arg.startswith('ConstClass('):
                 name = arg[len('ConstClass('):-1]
                 return self.get_const(name, 'class')

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefindnode.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefindnode.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefindnode.py	Mon Oct 18 11:22:20 2010
@@ -117,33 +117,32 @@
                             EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE))
     arraycopydescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
                  EffectInfo([], [], [], oopspecindex=EffectInfo.OS_ARRAYCOPY))
-    strconcatdescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_CONCAT))
-    slicedescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_SLICE))
-    strequaldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_EQUAL))
-    streq_slice_checknull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_CHECKNULL))
-    streq_slice_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_NONNULL))
-    streq_slice_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_CHAR))
-    streq_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_NONNULL))
-    streq_nonnull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_NONNULL_CHAR))
-    streq_checknull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_CHECKNULL_CHAR))
-    streq_lengthok_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_LENGTHOK))
+
+    for _name, _os in [
+        ('strconcatdescr',               'OS_STR_CONCAT'),
+        ('strslicedescr',                'OS_STR_SLICE'),
+        ('strequaldescr',                'OS_STR_EQUAL'),
+        ('streq_slice_checknull_descr',  'OS_STREQ_SLICE_CHECKNULL'),
+        ('streq_slice_nonnull_descr',    'OS_STREQ_SLICE_NONNULL'),
+        ('streq_slice_char_descr',       'OS_STREQ_SLICE_CHAR'),
+        ('streq_nonnull_descr',          'OS_STREQ_NONNULL'),
+        ('streq_nonnull_char_descr',     'OS_STREQ_NONNULL_CHAR'),
+        ('streq_checknull_char_descr',   'OS_STREQ_CHECKNULL_CHAR'),
+        ('streq_lengthok_descr',         'OS_STREQ_LENGTHOK'),
+        ]:
+        _oopspecindex = getattr(EffectInfo, _os)
+        locals()[_name] = \
+            cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=_oopspecindex))
+        #
+        _oopspecindex = getattr(EffectInfo, _os.replace('STR', 'UNI'))
+        locals()[_name.replace('str', 'unicode')] = \
+            cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=_oopspecindex))
+
+    s2u_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR2UNICODE))
+    #
 
     class LoopToken(AbstractDescr):
         pass

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizeopt.py	Mon Oct 18 11:22:20 2010
@@ -3498,7 +3498,7 @@
         i0 = strlen(p0)
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not', expected)
 
     def test_addsub_const(self):
         ops = """
@@ -3904,6 +3904,15 @@
         """
         self.optimize_loop(ops, 'Not, Not', expected)
 
+    # ----------
+    def optimize_strunicode_loop(self, ops, spectext, optops):
+        # check with the arguments passed in
+        self.optimize_loop(ops, spectext, optops)
+        # check with replacing 'str' with 'unicode' everywhere
+        self.optimize_loop(ops.replace('str','unicode').replace('s"', 'u"'),
+                           spectext,
+                           optops.replace('str','unicode').replace('s"', 'u"'))
+
     def test_newstr_1(self):
         ops = """
         [i0]
@@ -3916,7 +3925,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not', expected)
 
     def test_newstr_2(self):
         ops = """
@@ -3932,7 +3941,7 @@
         [i0, i1]
         jump(i1, i0)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_concat_1(self):
         ops = """
@@ -3953,7 +3962,7 @@
         copystrcontent(p2, p3, 0, i4, i5)
         jump(p2, p3)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_concat_vstr2_str(self):
         ops = """
@@ -3976,7 +3985,7 @@
         copystrcontent(p2, p3, 0, 2, i4)
         jump(i1, i0, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_concat_str_vstr2(self):
         ops = """
@@ -4000,7 +4009,7 @@
         i6 = int_add(i5, 1)      # will be killed by the backend
         jump(i1, i0, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_concat_str_str_str(self):
         ops = """
@@ -4027,12 +4036,12 @@
         copystrcontent(p3, p5, 0, i12b, i3b)
         jump(p2, p3, p5)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_concat_str_cstr1(self):
         ops = """
         [p2]
-        p3 = call(0, p2, "x", descr=strconcatdescr)
+        p3 = call(0, p2, s"x", descr=strconcatdescr)
         jump(p3)
         """
         expected = """
@@ -4046,28 +4055,28 @@
         i5 = int_add(i4, 1)      # will be killed by the backend
         jump(p3)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not', expected)
 
     def test_str_concat_consts(self):
         ops = """
         []
-        p1 = same_as("ab")
-        p2 = same_as("cde")
+        p1 = same_as(s"ab")
+        p2 = same_as(s"cde")
         p3 = call(0, p1, p2, descr=strconcatdescr)
         escape(p3)
         jump()
         """
         expected = """
         []
-        escape("abcde")
+        escape(s"abcde")
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_strunicode_loop(ops, '', expected)
 
     def test_str_slice_1(self):
         ops = """
         [p1, i1, i2]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
         jump(p2, i1, i2)
         """
         expected = """
@@ -4077,12 +4086,12 @@
         copystrcontent(p1, p2, i1, 0, i3)
         jump(p2, i1, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_slice_2(self):
         ops = """
         [p1, i2]
-        p2 = call(0, p1, 0, i2, descr=slicedescr)
+        p2 = call(0, p1, 0, i2, descr=strslicedescr)
         jump(p2, i2)
         """
         expected = """
@@ -4091,13 +4100,13 @@
         copystrcontent(p1, p2, 0, 0, i2)
         jump(p2, i2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_slice_3(self):
         ops = """
         [p1, i1, i2, i3, i4]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
-        p3 = call(0, p2, i3, i4, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
+        p3 = call(0, p2, i3, i4, descr=strslicedescr)
         jump(p3, i1, i2, i3, i4)
         """
         expected = """
@@ -4109,12 +4118,12 @@
         copystrcontent(p1, p3, i6, 0, i5)
         jump(p3, i1, i2, i3, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not, Not', expected)
 
     def test_str_slice_getitem1(self):
         ops = """
         [p1, i1, i2, i3]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
         i4 = strgetitem(p2, i3)
         escape(i4)
         jump(p1, i1, i2, i3)
@@ -4127,7 +4136,7 @@
         escape(i4)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected)
 
     def test_str_slice_plain(self):
         ops = """
@@ -4135,7 +4144,7 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i3)
         strsetitem(p1, 1, i4)
-        p2 = call(0, p1, 1, 2, descr=slicedescr)
+        p2 = call(0, p1, 1, 2, descr=strslicedescr)
         i5 = strgetitem(p2, 0)
         escape(i5)
         jump(i3, i4)
@@ -4145,12 +4154,12 @@
         escape(i4)
         jump(i3, i4)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_slice_concat(self):
         ops = """
         [p1, i1, i2, p2]
-        p3 = call(0, p1, i1, i2, descr=slicedescr)
+        p3 = call(0, p1, i1, i2, descr=strslicedescr)
         p4 = call(0, p3, p2, descr=strconcatdescr)
         jump(p4, i1, i2, p2)
         """
@@ -4166,10 +4175,10 @@
         copystrcontent(p2, p4, 0, i3, i4b)
         jump(p4, i1, i2, p2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected)
 
     # ----------
-    def optimize_loop_extradescrs(self, ops, spectext, optops):
+    def optimize_strunicode_loop_extradescrs(self, ops, spectext, optops):
         from pypy.jit.metainterp.optimizeopt import string
         def my_callinfo_for_oopspec(oopspecindex):
             calldescrtype = type(LLtypeMixin.strequaldescr)
@@ -4184,7 +4193,7 @@
         saved = string.callinfo_for_oopspec
         try:
             string.callinfo_for_oopspec = my_callinfo_for_oopspec
-            self.optimize_loop(ops, spectext, optops)
+            self.optimize_strunicode_loop(ops, spectext, optops)
         finally:
             string.callinfo_for_oopspec = saved
 
@@ -4195,7 +4204,7 @@
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', ops)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', ops)
 
     def test_str_equal_noop2(self):
         ops = """
@@ -4220,12 +4229,13 @@
         escape(i0)
         jump(p1, p2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice1(self):
         ops = """
         [p1, i1, i2, p3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p4, p3, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4237,12 +4247,13 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice2(self):
         ops = """
         [p1, i1, i2, p3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p3, p4, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4254,13 +4265,14 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice3(self):
         ops = """
         [p1, i1, i2, p3]
         guard_nonnull(p3) []
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p3, p4, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4273,13 +4285,14 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice4(self):
         ops = """
         [p1, i1, i2]
-        p3 = call(0, p1, i1, i2, descr=slicedescr)
-        i0 = call(0, p3, "x", descr=strequaldescr)
+        p3 = call(0, p1, i1, i2, descr=strslicedescr)
+        i0 = call(0, p3, s"x", descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2)
         """
@@ -4290,12 +4303,13 @@
         escape(i0)
         jump(p1, i1, i2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice5(self):
         ops = """
         [p1, i1, i2, i3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         p5 = newstr(1)
         strsetitem(p5, 0, i3)
         i0 = call(0, p5, p4, descr=strequaldescr)
@@ -4309,7 +4323,8 @@
         escape(i0)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_none1(self):
         ops = """
@@ -4324,7 +4339,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_none2(self):
         ops = """
@@ -4339,30 +4354,30 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull1(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "hello world", descr=strequaldescr)
+        i0 = call(0, p1, s"hello world", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
         expected = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "hello world", descr=streq_nonnull_descr)
+        i0 = call(0, p1, s"hello world", descr=streq_nonnull_descr)
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull2(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "", descr=strequaldescr)
+        i0 = call(0, p1, s"", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4374,13 +4389,13 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull3(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "x", descr=strequaldescr)
+        i0 = call(0, p1, s"x", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4391,13 +4406,13 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull4(self):
         ops = """
         [p1, p2]
         p4 = call(0, p1, p2, descr=strconcatdescr)
-        i0 = call(0, "hello world", p4, descr=strequaldescr)
+        i0 = call(0, s"hello world", p4, descr=strequaldescr)
         escape(i0)
         jump(p1, p2)
         """
@@ -4412,17 +4427,17 @@
         i5 = strlen(p2)
         i6 = int_add(i4, i5)      # will be killed by the backend
         copystrcontent(p2, p4, 0, i4, i5)
-        i0 = call(0, "hello world", p4, descr=streq_nonnull_descr)
+        i0 = call(0, s"hello world", p4, descr=streq_nonnull_descr)
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected)
 
     def test_str_equal_chars0(self):
         ops = """
         [i1]
         p1 = newstr(0)
-        i0 = call(0, p1, "", descr=strequaldescr)
+        i0 = call(0, p1, s"", descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4431,14 +4446,14 @@
         escape(1)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_chars1(self):
         ops = """
         [i1]
         p1 = newstr(1)
         strsetitem(p1, 0, i1)
-        i0 = call(0, p1, "x", descr=strequaldescr)
+        i0 = call(0, p1, s"x", descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4448,7 +4463,7 @@
         escape(i0)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_chars2(self):
         ops = """
@@ -4456,7 +4471,7 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i1)
         strsetitem(p1, 1, i2)
-        i0 = call(0, p1, "xy", descr=strequaldescr)
+        i0 = call(0, p1, s"xy", descr=strequaldescr)
         escape(i0)
         jump(i1, i2)
         """
@@ -4465,16 +4480,16 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i1)
         strsetitem(p1, 1, i2)
-        i0 = call(0, p1, "xy", descr=streq_lengthok_descr)
+        i0 = call(0, p1, s"xy", descr=streq_lengthok_descr)
         escape(i0)
         jump(i1, i2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected)
 
     def test_str_equal_chars3(self):
         ops = """
         [p1]
-        i0 = call(0, "x", p1, descr=strequaldescr)
+        i0 = call(0, s"x", p1, descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4484,14 +4499,14 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_lengthmismatch1(self):
         ops = """
         [i1]
         p1 = newstr(1)
         strsetitem(p1, 0, i1)
-        i0 = call(0, "xy", p1, descr=strequaldescr)
+        i0 = call(0, s"xy", p1, descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4500,12 +4515,33 @@
         escape(0)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
-
-    # XXX unicode operations
-    # XXX str2unicode
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
+    def test_str2unicode_constant(self):
+        ops = """
+        []
+        p0 = call(0, "xy", descr=s2u_descr)      # string -> unicode
+        escape(p0)
+        jump()
+        """
+        expected = """
+        []
+        escape(u"xy")
+        jump()
+        """
+        self.optimize_strunicode_loop_extradescrs(ops, '', expected)
 
+    def test_str2unicode_nonconstant(self):
+        ops = """
+        [p0]
+        p1 = call(0, p0, descr=s2u_descr)      # string -> unicode
+        escape(p1)
+        jump(p1)
+        """
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', ops)
+        # more generally, supporting non-constant but virtual cases is
+        # not obvious, because of the exception UnicodeDecodeError that
+        # can be raised by ll_str2unicode()
 
 
 ##class TestOOtype(OptimizeOptTest, OOtypeMixin):

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_resume.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_resume.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_resume.py	Mon Oct 18 11:22:20 2010
@@ -240,6 +240,17 @@
         return FakeBuiltObject(strconcat=[left, right])
     def slice_string(self, str, start, length):
         return FakeBuiltObject(strslice=[str, start, length])
+    def allocate_unicode(self, length):
+        return FakeBuiltObject(unistring=[None]*length)
+    def unicode_setitem(self, unistring, i, fieldnum):
+        value, tag = untag(fieldnum)
+        assert tag == TAGINT
+        assert 0 <= i < len(unistring.unistring)
+        unistring.unistring[i] = value
+    def concat_unicodes(self, left, right):
+        return FakeBuiltObject(uniconcat=[left, right])
+    def slice_unicode(self, str, start, length):
+        return FakeBuiltObject(unislice=[str, start, length])
 
 class FakeBuiltObject(object):
     def __init__(self, **kwds):
@@ -304,6 +315,30 @@
     assert reader.force_all_virtuals() == [
         FakeBuiltObject(strslice=info.fieldnums)]
 
+def test_vuniplaininfo():
+    info = VUniPlainInfo()
+    info.fieldnums = [tag(60, TAGINT)]
+    reader = FakeResumeDataReader()
+    reader._prepare_virtuals([info])
+    assert reader.force_all_virtuals() == [
+        FakeBuiltObject(unistring=[60])]
+
+def test_vuniconcatinfo():
+    info = VUniConcatInfo()
+    info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX)]
+    reader = FakeResumeDataReader()
+    reader._prepare_virtuals([info])
+    assert reader.force_all_virtuals() == [
+        FakeBuiltObject(uniconcat=info.fieldnums)]
+
+def test_vunisliceinfo():
+    info = VUniSliceInfo()
+    info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX), tag(30, TAGBOX)]
+    reader = FakeResumeDataReader()
+    reader._prepare_virtuals([info])
+    assert reader.force_all_virtuals() == [
+        FakeBuiltObject(unislice=info.fieldnums)]
+
 # ____________________________________________________________
 
 

Modified: pypy/branch/jitffi/pypy/jit/metainterp/test/test_string.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/test/test_string.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/test/test_string.py	Mon Oct 18 11:22:20 2010
@@ -6,14 +6,17 @@
 
 
 class StringTests:
+    _str, _chr = str, chr
+
     def test_eq_residual(self):
+        _str = self._str
         jitdriver = JitDriver(greens = [], reds = ['n', 'i', 's'])
-        global_s = "hello"
+        global_s = _str("hello")
         def f(n, b, s):
             if b:
-                s += "ello"
+                s += _str("ello")
             else:
-                s += "allo"
+                s += _str("allo")
             i = 0
             while n > 0:
                 jitdriver.can_enter_jit(s=s, n=n, i=i)
@@ -21,18 +24,19 @@
                 n -= 1 + (s == global_s)
                 i += 1
             return i
-        res = self.meta_interp(f, [10, True, 'h'], listops=True)
+        res = self.meta_interp(f, [10, True, _str('h')], listops=True)
         assert res == 5
         self.check_loops(**{self.CALL: 1, self.CALL_PURE: 0})
 
     def test_eq_folded(self):
+        _str = self._str
         jitdriver = JitDriver(greens = ['s'], reds = ['n', 'i'])
-        global_s = "hello"
+        global_s = _str("hello")
         def f(n, b, s):
             if b:
-                s += "ello"
+                s += _str("ello")
             else:
-                s += "allo"
+                s += _str("allo")
             i = 0
             while n > 0:
                 jitdriver.can_enter_jit(s=s, n=n, i=i)
@@ -40,31 +44,18 @@
                 n -= 1 + (s == global_s)
                 i += 1
             return i
-        res = self.meta_interp(f, [10, True, 'h'], listops=True)
+        res = self.meta_interp(f, [10, True, _str('h')], listops=True)
         assert res == 5
         self.check_loops(**{self.CALL: 0, self.CALL_PURE: 0})
 
     def test_newstr(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['n', 'm'])
         def f(n, m):
             while True:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                bytecode = 'adlfkj' + chr(n)
-                res = bytecode[n]
-                m -= 1
-                if m < 0:
-                    return ord(res)
-        res = self.meta_interp(f, [6, 10])
-        assert res == 6
-
-    def test_newunicode(self):
-        jitdriver = JitDriver(greens = [], reds = ['n', 'm'])
-        def f(n, m):
-            while True:
-                jitdriver.can_enter_jit(m=m, n=n)
-                jitdriver.jit_merge_point(m=m, n=n)
-                bytecode = u'adlfkj' + unichr(n)
+                bytecode = _str('adlfkj') + _chr(n)
                 res = bytecode[n]
                 m -= 1
                 if m < 0:
@@ -73,95 +64,96 @@
         assert res == 6
 
     def test_char2string_pure(self):
-        for dochr in [chr, ]: #unichr]:
-            jitdriver = JitDriver(greens = [], reds = ['n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            def f(n):
-                while n > 0:
-                    jitdriver.can_enter_jit(n=n)
-                    jitdriver.jit_merge_point(n=n)
-                    s = dochr(n)
-                    if not we_are_jitted():
-                        s += s     # forces to be a string
-                    if n > 100:
-                        escape(s)
-                    n -= 1
-                return 42
-            self.meta_interp(f, [6])
-            self.check_loops(newstr=0, strsetitem=0, strlen=0,
-                             newunicode=0, unicodesetitem=0, unicodelen=0)
+        _str, _chr = self._str, self._chr
+        jitdriver = JitDriver(greens = [], reds = ['n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        def f(n):
+            while n > 0:
+                jitdriver.can_enter_jit(n=n)
+                jitdriver.jit_merge_point(n=n)
+                s = _chr(n)
+                if not we_are_jitted():
+                    s += s     # forces to be a string
+                if n > 100:
+                    escape(s)
+                n -= 1
+            return 42
+        self.meta_interp(f, [6])
+        self.check_loops(newstr=0, strsetitem=0, strlen=0,
+                         newunicode=0, unicodesetitem=0, unicodelen=0)
 
     def test_char2string_escape(self):
-        for dochr in [chr, ]: #unichr]:
-            jitdriver = JitDriver(greens = [], reds = ['n', 'total'])
-            @dont_look_inside
-            def escape(x):
-                return ord(x[0])
-            def f(n):
-                total = 0
-                while n > 0:
-                    jitdriver.can_enter_jit(n=n, total=total)
-                    jitdriver.jit_merge_point(n=n, total=total)
-                    s = dochr(n)
-                    if not we_are_jitted():
-                        s += s    # forces to be a string
-                    total += escape(s)
-                    n -= 1
-                return total
-            res = self.meta_interp(f, [6])
-            assert res == 21
+        _str, _chr = self._str, self._chr
+        jitdriver = JitDriver(greens = [], reds = ['n', 'total'])
+        @dont_look_inside
+        def escape(x):
+            return ord(x[0])
+        def f(n):
+            total = 0
+            while n > 0:
+                jitdriver.can_enter_jit(n=n, total=total)
+                jitdriver.jit_merge_point(n=n, total=total)
+                s = _chr(n)
+                if not we_are_jitted():
+                    s += s    # forces to be a string
+                total += escape(s)
+                n -= 1
+            return total
+        res = self.meta_interp(f, [6])
+        assert res == 21
 
     def test_char2string2char(self):
-        for dochr in [chr, ]: #unichr]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'total'])
-            def f(m):
-                total = 0
-                while m > 0:
-                    jitdriver.can_enter_jit(m=m, total=total)
-                    jitdriver.jit_merge_point(m=m, total=total)
-                    string = dochr(m)
-                    if m > 100:
-                        string += string    # forces to be a string
-                    # read back the character
-                    c = string[0]
-                    total += ord(c)
-                    m -= 1
-                return total
-            res = self.meta_interp(f, [6])
-            assert res == 21
-            self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0,
-                             newunicode=0, unicodegetitem=0, unicodesetitem=0,
-                             unicodelen=0)
+        _str, _chr = self._str, self._chr
+        jitdriver = JitDriver(greens = [], reds = ['m', 'total'])
+        def f(m):
+            total = 0
+            while m > 0:
+                jitdriver.can_enter_jit(m=m, total=total)
+                jitdriver.jit_merge_point(m=m, total=total)
+                string = _chr(m)
+                if m > 100:
+                    string += string    # forces to be a string
+                # read back the character
+                c = string[0]
+                total += ord(c)
+                m -= 1
+            return total
+        res = self.meta_interp(f, [6])
+        assert res == 21
+        self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0,
+                         newunicode=0, unicodegetitem=0, unicodesetitem=0,
+                         unicodelen=0)
 
     def test_strconcat_pure(self):
-        for somestr in ["abc", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            mylist = [somestr+str(i) for i in range(10)]
-            def f(n, m):
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = mylist[n] + mylist[m]
-                    if m > 100:
-                        escape(s)
-                    m -= 1
-                return 42
-            self.meta_interp(f, [6, 7])
-            self.check_loops(newstr=0, strsetitem=0,
-                             newunicode=0, unicodesetitem=0,
-                             call=0, call_pure=0)
+        _str = self._str
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        mylist = [_str("abc") + _str(i) for i in range(10)]
+        def f(n, m):
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = mylist[n] + mylist[m]
+                if m > 100:
+                    escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 7])
+        self.check_loops(newstr=0, strsetitem=0,
+                         newunicode=0, unicodesetitem=0,
+                         call=0, call_pure=0)
 
     def test_strconcat_escape_str_str(self):
+        _str = self._str
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
@@ -171,46 +163,64 @@
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=0, copystrcontent=2,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=0, copystrcontent=2,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=0,
+                             copyunicodecontent=2,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_str_char(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = mylist[n] + chr(m)
+                s = mylist[n] + _chr(m)
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=1,
+                             copyunicodecontent=1,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_char_str(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = chr(n) + mylist[m]
+                s = _chr(n) + mylist[m]
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=1,
+                             copyunicodecontent=1,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_char_char(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
@@ -219,91 +229,132 @@
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = chr(n) + chr(m)
+                s = _chr(n) + _chr(m)
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=2, copystrcontent=0,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=2, copystrcontent=0,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=2,
+                             copyunicodecontent=0,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_str_char_str(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = mylist[n] + chr(n) + mylist[m]
+                s = mylist[n] + _chr(n) + mylist[m]
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=1, copystrcontent=2,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=1, copystrcontent=2,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=1,
+                             copyunicodecontent=2,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_guard_fail(self):
-        for somestr in ["abc", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            mylist = [somestr+str(i) for i in range(12)]
-            def f(n, m):
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = mylist[n] + mylist[m]
-                    if m & 1:
-                        escape(s)
-                    m -= 1
-                return 42
-            self.meta_interp(f, [6, 10])
+        _str = self._str
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        mylist = [_str("abc") + _str(i) for i in range(12)]
+        def f(n, m):
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = mylist[n] + mylist[m]
+                if m & 1:
+                    escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 10])
 
     def test_strslice(self):
-        for somestr in ["abc", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            def f(n, m):
-                assert n >= 0
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = "foobarbazetc"[m:n]
-                    if m <= 5:
-                        escape(s)
-                    m -= 1
-                return 42
-            self.meta_interp(f, [10, 10])
+        _str = self._str
+        longstring = _str("foobarbazetc")
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        def f(n, m):
+            assert n >= 0
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = longstring[m:n]
+                if m <= 5:
+                    escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [10, 10])
 
     def test_streq_char(self):
-        for somestr in ["?abcdefg", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            def f(n, m):
-                assert n >= 0
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = somestr[:m]
-                    escape(s == "?")
-                    m -= 1
-                return 42
-            self.meta_interp(f, [6, 7])
-            self.check_loops(newstr=0, newunicode=0)
-
-
-class TestOOtype(StringTests, OOJitMixin):
-    CALL = "oosend"
-    CALL_PURE = "oosend_pure"
+        _str = self._str
+        longstring = _str("?abcdefg")
+        somechar = _str("?")
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        def f(n, m):
+            assert n >= 0
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = longstring[:m]
+                escape(s == somechar)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 7])
+        self.check_loops(newstr=0, newunicode=0)
+
+
+#class TestOOtype(StringTests, OOJitMixin):
+#    CALL = "oosend"
+#    CALL_PURE = "oosend_pure"
 
 class TestLLtype(StringTests, LLJitMixin):
     CALL = "call"
     CALL_PURE = "call_pure"
+
+class TestLLtypeUnicode(TestLLtype):
+    _str, _chr = unicode, unichr
+
+    def test_str2unicode(self):
+        _str = self._str
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        class Foo:
+            pass
+        @dont_look_inside
+        def escape(x):
+            assert x == _str("6y")
+        def f(n, m):
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                foo = Foo()
+                foo.y = chr(m)
+                foo.y = "y"
+                s = _str(str(n)) + _str(foo.y)
+                escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 7])
+        self.check_loops(call=3,    # str(), _str(), escape()
+                         newunicode=1, unicodegetitem=0,
+                         unicodesetitem=1, copyunicodecontent=1)

Modified: pypy/branch/jitffi/pypy/jit/tool/traceviewer.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/tool/traceviewer.py	(original)
+++ pypy/branch/jitffi/pypy/jit/tool/traceviewer.py	Mon Oct 18 11:22:20 2010
@@ -253,10 +253,10 @@
 def main(loopfile, use_threshold, view=True):
     countname = py.path.local(loopfile + '.count')
     if countname.check():
-        counts = [re.split('( 20 and use_threshold:
             counts.threshold = l[-20]

Modified: pypy/branch/jitffi/pypy/module/__builtin__/interp_classobj.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/__builtin__/interp_classobj.py	(original)
+++ pypy/branch/jitffi/pypy/module/__builtin__/interp_classobj.py	Mon Oct 18 11:22:20 2010
@@ -2,9 +2,11 @@
 from pypy.interpreter.error import OperationError, operationerrfmt
 from pypy.interpreter.gateway import ObjSpace, W_Root, NoneNotWrapped, applevel
 from pypy.interpreter.gateway import interp2app, ObjSpace
-from pypy.interpreter.typedef import TypeDef, make_weakref_descr
+from pypy.interpreter.typedef import TypeDef
 from pypy.interpreter.argument import Arguments
 from pypy.interpreter.baseobjspace import Wrappable
+from pypy.interpreter.typedef import GetSetProperty, descr_get_dict
+from pypy.interpreter.typedef import descr_set_dict
 from pypy.rlib.rarithmetic import r_uint, intmask
 from pypy.rlib.objectmodel import compute_identity_hash
 from pypy.rlib.debug import make_sure_not_resized
@@ -57,6 +59,14 @@
         self.bases_w = bases
         self.w_dict = w_dict
  
+    def instantiate(self, space):
+        cache = space.fromcache(Cache)
+        if self.lookup(space, '__del__') is not None:
+            w_inst = cache.cls_with_del(space, self)
+        else:
+            w_inst = cache.cls_without_del(space, self)
+        return w_inst
+
     def getdict(self):
         return self.w_dict
 
@@ -100,15 +110,15 @@
         return False
 
     @jit.unroll_safe
-    def lookup(self, space, w_attr):
+    def lookup(self, space, attr):
         # returns w_value or interplevel None
-        w_result = space.finditem(self.w_dict, w_attr)
+        w_result = space.finditem_str(self.w_dict, attr)
         if w_result is not None:
             return w_result
         for base in self.bases_w:
             # XXX fix annotation of bases_w to be a list of W_ClassObjects
             assert isinstance(base, W_ClassObject)
-            w_result = base.lookup(space, w_attr)
+            w_result = base.lookup(space, attr)
             if w_result is not None:
                 return w_result
         return None
@@ -122,7 +132,7 @@
                 return space.wrap(self.name)
             elif name == "__bases__":
                 return space.newtuple(self.bases_w)
-        w_value = self.lookup(space, w_attr)
+        w_value = self.lookup(space, name)
         if w_value is None:
             raise operationerrfmt(
                 space.w_AttributeError,
@@ -147,7 +157,7 @@
                 self.setbases(space, w_value)
                 return
             elif name == "__del__":
-                if self.lookup(space, w_attr) is None:
+                if self.lookup(space, name) is None:
                     msg = ("a __del__ method added to an existing class "
                            "will not be called")
                     space.warn(msg, space.w_RuntimeWarning)
@@ -195,13 +205,20 @@
         # NOT_RPYTHON
         return '' % self.name
 
+class Cache:
+    def __init__(self, space):
+        from pypy.interpreter.typedef import _usersubclswithfeature
+        # evil
+        self.cls_without_del = _usersubclswithfeature(
+                space.config, W_InstanceObject, "dict", "weakref")
+        self.cls_with_del = _usersubclswithfeature(
+                space.config, self.cls_without_del, "del")
+
+
 def class_descr_call(space, w_self, __args__):
     self = space.interp_w(W_ClassObject, w_self)
-    if self.lookup(space, space.wrap('__del__')) is not None:
-        w_inst = W_InstanceObjectWithDel(space, self)
-    else:
-        w_inst = W_InstanceObject(space, self)
-    w_init = w_inst.getattr_from_class(space, space.wrap('__init__'))
+    w_inst = self.instantiate(space)
+    w_init = w_inst.getattr_from_class(space, '__init__')
     if w_init is not None:
         w_result = space.call_args(w_init, __args__)
         if not space.is_w(w_result, space.w_None):
@@ -234,7 +251,7 @@
 
 def make_unary_instance_method(name):
     def unaryop(self, space):
-        w_meth = self.getattr(space, space.wrap(name), True)
+        w_meth = self.getattr(space, name, True)
         return space.call_function(w_meth)
     unaryop.func_name = name
     return unaryop
@@ -242,7 +259,7 @@
 def make_binary_returning_notimplemented_instance_method(name):
     def binaryop(self, space, w_other):
         try:
-            w_meth = self.getattr(space, space.wrap(name), False)
+            w_meth = self.getattr(space, name, False)
         except OperationError, e:
             if e.match(space, space.w_AttributeError):
                 return space.w_NotImplemented
@@ -267,7 +284,7 @@
             w_a = self
             w_b = w_other
         if w_a is self:
-            w_meth = self.getattr(space, space.wrap(specialname), False)
+            w_meth = self.getattr(space, specialname, False)
             if w_meth is None:
                 return space.w_NotImplemented
             return space.call_function(w_meth, w_b)
@@ -278,7 +295,7 @@
     def rbinaryop(self, space, w_other):
         w_a, w_b = _coerce_helper(space, self, w_other)
         if w_a is None or w_a is self:
-            w_meth = self.getattr(space, space.wrap(rspecialname), False)
+            w_meth = self.getattr(space, rspecialname, False)
             if w_meth is None:
                 return space.w_NotImplemented
             return space.call_function(w_meth, w_other)
@@ -302,46 +319,34 @@
         raise OperationError(
             space.w_TypeError,
             space.wrap("instance() first arg must be class"))
-    if space.is_w(w_dict, space.w_None):
-        w_dict = None
-    elif not space.is_true(space.isinstance(w_dict, space.w_dict)):
-        raise OperationError(
-            space.w_TypeError,
-            space.wrap("instance() second arg must be dictionary or None"))
-    return W_InstanceObject(space, w_class, w_dict)
+    w_result = w_class.instantiate(space)
+    if not space.is_w(w_dict, space.w_None):
+        w_result.setdict(space, w_dict)
+    return w_result
 
 class W_InstanceObject(Wrappable):
-    def __init__(self, space, w_class, w_dict=None):
-        if w_dict is None:
-            w_dict = space.newdict(instance=True)
+    def __init__(self, space, w_class):
+        # note that user_setup is overridden by the typedef.py machinery
+        self.user_setup(space, space.gettypeobject(self.typedef))
         assert isinstance(w_class, W_ClassObject)
         self.w_class = w_class
-        self.w_dict = w_dict
-        self.space = space
-
-    def getdict(self):
-        return self.w_dict
 
-    def setdict(self, space, w_dict):
-        if (w_dict is None or
-            not space.is_true(space.isinstance(w_dict, space.w_dict))):
-            raise OperationError(
-                space.w_TypeError,
-                space.wrap("__dict__ must be a dictionary object"))
-        self.w_dict = w_dict
+    def user_setup(self, space, w_subtype):
+        self.space = space
 
-    def setclass(self, space, w_class):
+    def set_oldstyle_class(self, space, w_class):
         if w_class is None or not isinstance(w_class, W_ClassObject):
             raise OperationError(
                 space.w_TypeError,
                 space.wrap("__class__ must be set to a class"))
         self.w_class = w_class
 
-    def getattr_from_class(self, space, w_name):
+    def getattr_from_class(self, space, name):
         # Look up w_name in the class dict, and call its __get__.
         # This method ignores the instance dict and the __getattr__.
         # Returns None if not found.
-        w_value = self.w_class.lookup(space, w_name)
+        assert isinstance(name, str)
+        w_value = self.w_class.lookup(space, name)
         if w_value is None:
             return None
         w_descr_get = space.lookup(w_value, '__get__')
@@ -349,19 +354,20 @@
             return w_value
         return space.call_function(w_descr_get, w_value, self, self.w_class)
 
-    def getattr(self, space, w_name, exc=True):
+    def getattr(self, space, name, exc=True):
         # Normal getattr rules: look up w_name in the instance dict,
         # in the class dict, and then via a call to __getatttr__.
-        w_result = space.finditem(self.w_dict, w_name)
+        assert isinstance(name, str)
+        w_result = self.getdictvalue(space, name)
         if w_result is not None:
             return w_result
-        w_result = self.getattr_from_class(space, w_name)
+        w_result = self.getattr_from_class(space, name)
         if w_result is not None:
             return w_result
-        w_meth = self.getattr_from_class(space, space.wrap('__getattr__'))
+        w_meth = self.getattr_from_class(space, '__getattr__')
         if w_meth is not None:
             try:
-                return space.call_function(w_meth, w_name)
+                return space.call_function(w_meth, space.wrap(name))
             except OperationError, e:
                 if not exc and e.match(space, space.w_AttributeError):
                     return None     # eat the AttributeError
@@ -371,7 +377,7 @@
             raise operationerrfmt(
                 space.w_AttributeError,
                 "%s instance has no attribute '%s'",
-                self.w_class.name, space.str_w(w_name))
+                self.w_class.name, name)
         else:
             return None
 
@@ -379,44 +385,46 @@
         name = space.str_w(w_attr)
         if len(name) >= 8 and name[0] == '_':
             if name == "__dict__":
-                return self.w_dict
+                return self.getdict()
             elif name == "__class__":
                 return self.w_class
-        return self.getattr(space, w_attr)
+        return self.getattr(space, name)
 
     def descr_setattr(self, space, w_name, w_value):
         name = unwrap_attr(space, w_name)
-        w_meth = self.getattr_from_class(space, space.wrap('__setattr__'))
+        w_meth = self.getattr_from_class(space, '__setattr__')
         if name and name[0] == "_":
             if name == '__dict__':
                 self.setdict(space, w_value)
                 return
             if name == '__class__':
-                self.setclass(space, w_value)
+                self.set_oldstyle_class(space, w_value)
                 return
             if name == '__del__' and w_meth is None:
-                if (not isinstance(self, W_InstanceObjectWithDel)
-                    and space.finditem(self.w_dict, w_name) is None):
+                cache = space.fromcache(Cache)
+                if (not isinstance(self, cache.cls_with_del)
+                    and self.getdictvalue(space, '__del__') is None):
                     msg = ("a __del__ method added to an instance "
                            "with no __del__ in the class will not be called")
                     space.warn(msg, space.w_RuntimeWarning)
         if w_meth is not None:
             space.call_function(w_meth, w_name, w_value)
         else:
-            self.setdictvalue(space, name, w_value)
+            # bit obscure: appease normalization
+            self.setdictvalue(space, name, w_value, True)
 
     def descr_delattr(self, space, w_name):
         name = unwrap_attr(space, w_name)
         if name and name[0] == "_":
             if name == '__dict__':
                 # use setdict to raise the error
-                self.setdict(space, None)
+                self.setdict(space, space.w_None)
                 return
             elif name == '__class__':
-                # use setclass to raise the error
-                self.setclass(space, None)
+                # use set_oldstyle_class to raise the error
+                self.set_oldstyle_class(space, None)
                 return
-        w_meth = self.getattr_from_class(space, space.wrap('__delattr__'))
+        w_meth = self.getattr_from_class(space, '__delattr__')
         if w_meth is not None:
             space.call_function(w_meth, w_name)
         else:
@@ -427,7 +435,7 @@
                     self.w_class.name, name)
 
     def descr_repr(self, space):
-        w_meth = self.getattr(space, space.wrap('__repr__'), False)
+        w_meth = self.getattr(space, '__repr__', False)
         if w_meth is None:
             w_class = self.w_class
             mod = w_class.get_module_string(space)
@@ -435,19 +443,19 @@
         return space.call_function(w_meth)
 
     def descr_str(self, space):
-        w_meth = self.getattr(space, space.wrap('__str__'), False)
+        w_meth = self.getattr(space, '__str__', False)
         if w_meth is None:
             return self.descr_repr(space)
         return space.call_function(w_meth)
 
     def descr_unicode(self, space):
-        w_meth = self.getattr(space, space.wrap('__unicode__'), False)
+        w_meth = self.getattr(space, '__unicode__', False)
         if w_meth is None:
             return self.descr_str(space)
         return space.call_function(w_meth)
 
     def descr_len(self, space):
-        w_meth = self.getattr(space, space.wrap('__len__'))
+        w_meth = self.getattr(space, '__len__')
         w_result = space.call_function(w_meth)
         if space.is_true(space.isinstance(w_result, space.w_int)):
             if space.is_true(space.lt(w_result, space.wrap(0))):
@@ -460,22 +468,22 @@
             space.wrap("__len__() should return an int"))
 
     def descr_getitem(self, space, w_key):
-        w_meth = self.getattr(space, space.wrap('__getitem__'))
+        w_meth = self.getattr(space, '__getitem__')
         return space.call_function(w_meth, w_key)
 
     def descr_setitem(self, space, w_key, w_value):
-        w_meth = self.getattr(space, space.wrap('__setitem__'))
+        w_meth = self.getattr(space, '__setitem__')
         space.call_function(w_meth, w_key, w_value)
 
     def descr_delitem(self, space, w_key):
-        w_meth = self.getattr(space, space.wrap('__delitem__'))
+        w_meth = self.getattr(space, '__delitem__')
         space.call_function(w_meth, w_key)
 
     def descr_iter(self, space):
-        w_meth = self.getattr(space, space.wrap('__iter__'), False)
+        w_meth = self.getattr(space, '__iter__', False)
         if w_meth is not None:
             return space.call_function(w_meth)
-        w_meth = self.getattr(space, space.wrap('__getitem__'), False)
+        w_meth = self.getattr(space, '__getitem__', False)
         if w_meth is None:
             raise OperationError(
                 space.w_TypeError,
@@ -485,14 +493,14 @@
     # don't see the point
 
     def descr_getslice(self, space, w_i, w_j):
-        w_meth = self.getattr(space, space.wrap('__getslice__'), False)
+        w_meth = self.getattr(space, '__getslice__', False)
         if w_meth is not None:
             return space.call_function(w_meth, w_i, w_j)
         else:
             return space.getitem(self, space.newslice(w_i, w_j, space.w_None))
 
     def descr_setslice(self, space, w_i, w_j, w_sequence):
-        w_meth = self.getattr(space, space.wrap('__setslice__'), False)
+        w_meth = self.getattr(space, '__setslice__', False)
         if w_meth is not None:
             space.call_function(w_meth, w_i, w_j, w_sequence)
         else:
@@ -500,20 +508,20 @@
                           w_sequence)
 
     def descr_delslice(self, space, w_i, w_j):
-        w_meth = self.getattr(space, space.wrap('__delslice__'), False)
+        w_meth = self.getattr(space, '__delslice__', False)
         if w_meth is not None:
             space.call_function(w_meth, w_i, w_j)
         else:
             return space.delitem(self, space.newslice(w_i, w_j, space.w_None))
 
     def descr_call(self, space, __args__):
-        w_meth = self.getattr(space, space.wrap('__call__'))
+        w_meth = self.getattr(space, '__call__')
         return space.call_args(w_meth, __args__)
 
     def descr_nonzero(self, space):
-        w_func = self.getattr(space, space.wrap('__nonzero__'), False)
+        w_func = self.getattr(space, '__nonzero__', False)
         if w_func is None:
-            w_func = self.getattr(space, space.wrap('__len__'), False)
+            w_func = self.getattr(space, '__len__', False)
             if w_func is None:
                 return space.w_True
         w_result = space.call_function(w_func)
@@ -537,7 +545,7 @@
                 not isinstance(w_b, W_InstanceObject)):
                 return space.cmp(w_a, w_b)
         if isinstance(w_a, W_InstanceObject):
-            w_func = w_a.getattr(space, space.wrap('__cmp__'), False)
+            w_func = w_a.getattr(space, '__cmp__', False)
             if w_func is not None:
                 w_res = space.call_function(w_func, w_b)
                 if space.is_w(w_res, space.w_NotImplemented):
@@ -556,7 +564,7 @@
                     return space.wrap(-1)
                 return space.wrap(0)
         if isinstance(w_b, W_InstanceObject):
-            w_func = w_b.getattr(space, space.wrap('__cmp__'), False)
+            w_func = w_b.getattr(space, '__cmp__', False)
             if w_func is not None:
                 w_res = space.call_function(w_func, w_a)
                 if space.is_w(w_res, space.w_NotImplemented):
@@ -577,10 +585,10 @@
         return space.w_NotImplemented
 
     def descr_hash(self, space):
-        w_func = self.getattr(space, space.wrap('__hash__'), False)
+        w_func = self.getattr(space, '__hash__', False)
         if w_func is None:
-            w_eq =  self.getattr(space, space.wrap('__eq__'), False)
-            w_cmp =  self.getattr(space, space.wrap('__cmp__'), False)
+            w_eq =  self.getattr(space, '__eq__', False)
+            w_cmp =  self.getattr(space, '__cmp__', False)
             if w_eq is not None or w_cmp is not None:
                 raise OperationError(space.w_TypeError,
                                      space.wrap("unhashable instance"))
@@ -595,7 +603,7 @@
         return w_ret
 
     def descr_index(self, space):
-        w_func = self.getattr(space, space.wrap('__index__'), False)
+        w_func = self.getattr(space, '__index__', False)
         if w_func is not None:
             return space.call_function(w_func)
         raise OperationError(
@@ -603,7 +611,7 @@
             space.wrap("object cannot be interpreted as an index"))
 
     def descr_contains(self, space, w_obj):
-        w_func = self.getattr(space, space.wrap('__contains__'), False)
+        w_func = self.getattr(space, '__contains__', False)
         if w_func is not None:
             return space.wrap(space.is_true(space.call_function(w_func, w_obj)))
         # now do it ourselves
@@ -626,7 +634,7 @@
                 w_a = self
                 w_b = w_other
             if w_a is self:
-                w_func = self.getattr(space, space.wrap('__pow__'), False)
+                w_func = self.getattr(space, '__pow__', False)
                 if w_func is not None:
                     return space.call_function(w_func, w_other)
                 return space.w_NotImplemented
@@ -634,7 +642,7 @@
                 return space.pow(w_a, w_b, space.w_None)
         else:
             # CPython also doesn't try coercion in this case
-            w_func = self.getattr(space, space.wrap('__pow__'), False)
+            w_func = self.getattr(space, '__pow__', False)
             if w_func is not None:
                 return space.call_function(w_func, w_other, w_modulo)
             return space.w_NotImplemented
@@ -646,7 +654,7 @@
                 w_a = self
                 w_b = w_other
             if w_a is self:
-                w_func = self.getattr(space, space.wrap('__rpow__'), False)
+                w_func = self.getattr(space, '__rpow__', False)
                 if w_func is not None:
                     return space.call_function(w_func, w_other)
                 return space.w_NotImplemented
@@ -654,13 +662,13 @@
                 return space.pow(w_b, w_a, space.w_None)
         else:
             # CPython also doesn't try coercion in this case
-            w_func = self.getattr(space, space.wrap('__rpow__'), False)
+            w_func = self.getattr(space, '__rpow__', False)
             if w_func is not None:
                 return space.call_function(w_func, w_other, w_modulo)
             return space.w_NotImplemented
 
     def descr_next(self, space):
-        w_func = self.getattr(space, space.wrap('next'), False)
+        w_func = self.getattr(space, 'next', False)
         if w_func is None:
             raise OperationError(space.w_TypeError,
                                  space.wrap("instance has no next() method"))
@@ -669,10 +677,9 @@
     def descr_del(self, space):
         # Note that this is called from executioncontext.UserDelAction
         # via the space.userdel() method.
-        w_name = space.wrap('__del__')
-        w_func = space.finditem(self.w_dict, w_name)
+        w_func = self.getdictvalue(space, '__del__')
         if w_func is None:
-            w_func = self.getattr_from_class(space, w_name)
+            w_func = self.getattr_from_class(space, '__del__')
         if w_func is not None:
             space.call_function(w_func)
 
@@ -717,6 +724,14 @@
         rmeth,
         unwrap_spec=["self", ObjSpace, W_Root])
 
+
+def descr_del_dict(space, w_inst):
+    # use setdict to raise the error
+    w_inst.setdict(space, space.w_None)
+
+dict_descr = GetSetProperty(descr_get_dict, descr_set_dict, descr_del_dict)
+dict_descr.name = '__dict__'
+
 W_InstanceObject.typedef = TypeDef("instance",
     __new__ = interp2app(descr_instance_new),
     __getattribute__ = interp2app(W_InstanceObject.descr_getattribute,
@@ -766,12 +781,9 @@
                          unwrap_spec=['self', ObjSpace, W_Root, W_Root]),
     next = interp2app(W_InstanceObject.descr_next,
                       unwrap_spec=['self', ObjSpace]),
-    __weakref__ = make_weakref_descr(W_InstanceObject),
     __del__ = interp2app(W_InstanceObject.descr_del,
                          unwrap_spec=['self', ObjSpace]),
+    __dict__ = dict_descr,
     **rawdict
 )
-
-class W_InstanceObjectWithDel(W_InstanceObject):
-    def __del__(self):
-        self._enqueue_for_destruction(self.space)
+W_InstanceObject.typedef.acceptable_as_base_class = False

Modified: pypy/branch/jitffi/pypy/module/__builtin__/test/test_classobj.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/__builtin__/test/test_classobj.py	(original)
+++ pypy/branch/jitffi/pypy/module/__builtin__/test/test_classobj.py	Mon Oct 18 11:22:20 2010
@@ -928,6 +928,31 @@
         assert x is b
         assert y == 5
 
+    def test_cant_subclass_instance(self):
+        class A:
+            pass
+        try:
+            class B(type(A())):
+                pass
+        except TypeError:
+            pass
+        else:
+            assert 0, "should have raised"
+
+    def test_dict_descriptor(self):
+        import sys
+        if not hasattr(sys, 'pypy_objspaceclass'):
+            skip("on CPython old-style instances don't have a __dict__ descriptor")
+        class A:
+            pass
+        a = A()
+        a.x = 1
+        descr = type(a).__dict__['__dict__']
+        assert descr.__get__(a) == {'x': 1}
+        descr.__set__(a, {'x': 2})
+        assert a.x == 2
+        raises(TypeError, descr.__delete__, a)
+
 
 class AppTestOldStyleSharing(AppTestOldstyle):
     def setup_class(cls):
@@ -966,3 +991,22 @@
             a = 1
             b = 2
         assert self.is_strdict(A)
+
+class AppTestOldStyleMapDict(AppTestOldstyle):
+    def setup_class(cls):
+        cls.space = gettestobjspace(**{"objspace.std.withmapdict": True})
+        if option.runappdirect:
+            py.test.skip("can only be run on py.py")
+        def has_mapdict(space, w_inst):
+            return space.wrap(w_inst._get_mapdict_map() is not None)
+        cls.w_has_mapdict = cls.space.wrap(gateway.interp2app(has_mapdict))
+
+
+    def test_has_mapdict(self):
+        class A:
+            def __init__(self):
+                self.x = 42
+        a = A()
+        assert a.x == 42
+        assert self.has_mapdict(a)
+

Modified: pypy/branch/jitffi/pypy/module/_rawffi/array.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/_rawffi/array.py	(original)
+++ pypy/branch/jitffi/pypy/module/_rawffi/array.py	Mon Oct 18 11:22:20 2010
@@ -97,7 +97,15 @@
 
 class W_ArrayInstance(W_DataInstance):
     def __init__(self, space, shape, length, address=r_uint(0)):
-        W_DataInstance.__init__(self, space, shape.size * length, address)
+        # Workaround for a strange behavior of libffi: make sure that
+        # we always have at least 8 bytes.  For W_ArrayInstances that are
+        # used as the result value of a function call, ffi_call() writes
+        # 8 bytes into it even if the function's result type asks for less.
+        # This strange behavior is documented.
+        memsize = shape.size * length
+        if memsize < 8:
+            memsize = 8
+        W_DataInstance.__init__(self, space, memsize, address)
         self.length = length
         self.shape = shape
 

Modified: pypy/branch/jitffi/pypy/module/_weakref/interp__weakref.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/_weakref/interp__weakref.py	(original)
+++ pypy/branch/jitffi/pypy/module/_weakref/interp__weakref.py	Mon Oct 18 11:22:20 2010
@@ -7,7 +7,7 @@
 import weakref
 
 
-class WeakrefLifeline(object):
+class WeakrefLifeline(W_Root):
     def __init__(self, space):
         self.space = space       # this is here for W_Root.clear_all_weakrefs()
         self.refs_weak = []

Modified: pypy/branch/jitffi/pypy/module/cpyext/classobject.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/cpyext/classobject.py	(original)
+++ pypy/branch/jitffi/pypy/module/cpyext/classobject.py	Mon Oct 18 11:22:20 2010
@@ -15,16 +15,20 @@
     class is the class of new object.  The dict parameter will be used as the
     object's __dict__; if NULL, a new dictionary will be created for the
     instance."""
-    if not PyClass_Check(space, w_class):
+    if not isinstance(w_class, W_ClassObject):
         return PyErr_BadInternalCall(space)
-    return W_InstanceObject(space, w_class, w_dict)
+    w_result = w_class.instantiate(space)
+    if w_dict is not None:
+        w_result.setdict(space, w_dict)
+    return w_result
 
 @cpython_api([PyObject, PyObject], PyObject, error=CANNOT_FAIL)
 def _PyInstance_Lookup(space, w_instance, w_name):
+    name = space.str_w(w_name)
     assert isinstance(w_instance, W_InstanceObject)
-    w_result = space.finditem(w_instance.w_dict, w_name)
+    w_result = w_instance.getdictvalue(space, name)
     if w_result is not None:
         return w_result
-    return w_instance.w_class.lookup(space, w_name)
+    return w_instance.w_class.lookup(space, name)
 
 

Modified: pypy/branch/jitffi/pypy/module/cpyext/test/test_unicodeobject.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/cpyext/test/test_unicodeobject.py	(original)
+++ pypy/branch/jitffi/pypy/module/cpyext/test/test_unicodeobject.py	Mon Oct 18 11:22:20 2010
@@ -177,13 +177,14 @@
             encoded_charp = rffi.str2charp(encoded)
             strict_charp = rffi.str2charp("strict")
             if endian is not None:
-                pendian = lltype.malloc(rffi.INTP.TO, 1, flavor='raw')
                 if endian < 0:
-                    pendian[0] = -1
+                    value = -1
                 elif endian > 0:
-                    pendian[0] = 1
+                    value = 1
                 else:
-                    pendian[0] = 0
+                    value = 0
+                pendian = lltype.malloc(rffi.INTP.TO, 1, flavor='raw')
+                pendian[0] = rffi.cast(rffi.INT, value)
             else:
                 pendian = None
 

Modified: pypy/branch/jitffi/pypy/module/gc/referents.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/gc/referents.py	(original)
+++ pypy/branch/jitffi/pypy/module/gc/referents.py	Mon Oct 18 11:22:20 2010
@@ -15,8 +15,16 @@
 
 def try_cast_gcref_to_w_root(gcref):
     w_obj = rgc.try_cast_gcref_to_instance(W_Root, gcref)
-    if not we_are_translated() and not hasattr(w_obj, 'typedef'):
-        w_obj = None
+    # Ignore the instances of W_Root that are not really valid as Python
+    # objects.  There is e.g. WeakrefLifeline in module/_weakref that
+    # inherits from W_Root for internal reasons.  Such instances don't
+    # have a typedef at all (or have a null typedef after translation).
+    if not we_are_translated():
+        if not hasattr(w_obj, 'typedef'):
+            return None
+    else:
+        if w_obj is None or not w_obj.typedef:
+            return None
     return w_obj
 
 def wrap(space, gcref):

Modified: pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/branch/jitffi/pypy/module/pypyjit/test/test_pypy_c.py	Mon Oct 18 11:22:20 2010
@@ -276,7 +276,7 @@
         assert len(ops) == 2
         assert not ops[0].get_opnames("call")
         assert not ops[0].get_opnames("new")
-        assert len(ops[0].get_opnames("guard")) <= 7
+        assert len(ops[0].get_opnames("guard")) <= 3     # we get 2 withmapdict
         assert not ops[1] # second LOOKUP_METHOD folded away
 
         ops = self.get_by_bytecode("CALL_METHOD")
@@ -287,7 +287,7 @@
             else:
                 assert not bytecode.get_opnames("call")
             assert not bytecode.get_opnames("new")
-            assert len(bytecode.get_opnames("guard")) <= 9
+            assert len(bytecode.get_opnames("guard")) <= 6
         assert len(ops[1]) < len(ops[0])
 
         ops = self.get_by_bytecode("LOAD_ATTR")
@@ -321,8 +321,8 @@
         assert len(ops) == 2
         assert not ops[0].get_opnames("call")
         assert not ops[0].get_opnames("new")
-        assert len(ops[0].get_opnames("guard")) <= 7
-        assert len(ops[0].get_opnames("getfield")) < 6
+        assert len(ops[0].get_opnames("guard")) <= 3    # we get 2 withmapdict
+        assert len(ops[0].get_opnames("getfield")) <= 5 # we get <5 withmapdict
         assert not ops[1] # second LOOKUP_METHOD folded away
 
     def test_default_and_kw(self):
@@ -386,7 +386,7 @@
                     a.x = 2
                     i = i + a.x
                 return i
-        ''', 67,
+        ''', 69,
                    ([20], 20),
                    ([31], 32))
 
@@ -394,7 +394,7 @@
                 self.get_by_bytecode("CALL_FUNCTION"))
         assert not callA.get_opnames("call")
         assert not callA.get_opnames("new")
-        assert len(callA.get_opnames("guard")) <= 8
+        assert len(callA.get_opnames("guard")) <= 2
         assert not callisinstance1.get_opnames("call")
         assert not callisinstance1.get_opnames("new")
         assert len(callisinstance1.get_opnames("guard")) <= 2
@@ -746,6 +746,8 @@
                     '''%(op1, float(a)/4.0, float(b)/4.0, op2), 109, ([], res))
 
     def test_boolrewrite_ptr(self):
+        # XXX this test is way too imprecise in what it is actually testing
+        # it should count the number of guards instead
         compares = ('a == b', 'b == a', 'a != b', 'b != a', 'a == c', 'c != b')
         for e1 in compares:
             for e2 in compares:
@@ -769,7 +771,7 @@
                 print
                 print 'Test:', e1, e2, n, res
                 self.run_source('''
-                class tst:
+                class tst(object):
                     pass
                 def main():
                     a = tst()
@@ -851,6 +853,8 @@
             ''', 65, ([], 122880))
 
     def test_array_intimg(self):
+        # XXX this test is way too imprecise in what it is actually testing
+        # it should count the number of guards instead
         for tc, maxops in zip('ilILd', (67, 67, 69, 69, 61)):
             res = 73574560
             if tc in 'IL':

Modified: pypy/branch/jitffi/pypy/module/signal/interp_signal.py
==============================================================================
--- pypy/branch/jitffi/pypy/module/signal/interp_signal.py	(original)
+++ pypy/branch/jitffi/pypy/module/signal/interp_signal.py	Mon Oct 18 11:22:20 2010
@@ -1,7 +1,6 @@
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import W_Root, ObjSpace
 from pypy.interpreter.executioncontext import AsyncAction, AbstractActionFlag
-from pypy.rlib.rarithmetic import LONG_BIT, intmask
 import signal as cpy_signal
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.translator.tool.cbuild import ExternalCompilationInfo
@@ -64,8 +63,8 @@
 class CheckSignalAction(AsyncAction):
     """An action that is automatically invoked when a signal is received."""
 
-    # The C-level signal handler sets the highest bit of pypysig_occurred:
-    bitmask = intmask(1 << (LONG_BIT-1))
+    # The C-level signal handler sets the bit 30 of pypysig_occurred:
+    bitmask = 1 << 30
 
     def __init__(self, space):
         AsyncAction.__init__(self, space)

Modified: pypy/branch/jitffi/pypy/objspace/std/celldict.py
==============================================================================
--- pypy/branch/jitffi/pypy/objspace/std/celldict.py	(original)
+++ pypy/branch/jitffi/pypy/objspace/std/celldict.py	Mon Oct 18 11:22:20 2010
@@ -45,7 +45,7 @@
         if space.is_w(space.type(w_key), space.w_str):
             self.impl_setitem_str(self.space.str_w(w_key), w_value)
         else:
-            self._as_rdict().setitem(w_key, w_value)
+            self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
     def impl_setitem_str(self, name, w_value, shadows_type=True):
         self.getcell(name, True).w_value = w_value
@@ -66,7 +66,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
         
     def impl_length(self):
         # inefficient, but do we care?
@@ -85,7 +85,7 @@
         elif _is_sane_hash(space, w_lookup_type):
             return None
         else:
-            return self._as_rdict().getitem(w_lookup)
+            return self._as_rdict().impl_fallback_getitem(w_lookup)
 
     def impl_getitem_str(self, lookup):
         res = self.getcell(lookup, False)

Modified: pypy/branch/jitffi/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/jitffi/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/jitffi/pypy/objspace/std/dictmultiobject.py	Mon Oct 18 11:22:20 2010
@@ -102,17 +102,17 @@
         else:
             return None
 
-    # _________________________________________________________________ 
+    # _________________________________________________________________
     # implementation methods
     def impl_getitem(self, w_key):
         #return w_value or None
         raise NotImplementedError("abstract base class")
 
-    def impl_getitem_str(self, w_key):
+    def impl_getitem_str(self, key):
         #return w_value or None
         raise NotImplementedError("abstract base class")
 
-    def impl_setitem_str(self,  key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value, shadows_type=True):
         raise NotImplementedError("abstract base class")
 
     def impl_setitem(self,  w_key, w_value):
@@ -120,7 +120,7 @@
 
     def impl_delitem(self, w_key):
         raise NotImplementedError("abstract base class")
- 
+
     def impl_length(self):
         raise NotImplementedError("abstract base class")
 
@@ -310,7 +310,7 @@
         if space.is_w(space.type(w_key), space.w_str):
             self.impl_setitem_str(self.space.str_w(w_key), w_value)
         else:
-            self._as_rdict().setitem(w_key, w_value)
+            self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
     def impl_setitem_str(self, key, w_value, shadows_type=True):
         self.content[key] = w_value
@@ -324,7 +324,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
         
     def impl_length(self):
         return len(self.content)
@@ -344,7 +344,7 @@
         elif _is_sane_hash(space, w_lookup_type):
             return None
         else:
-            return self._as_rdict().getitem(w_key)
+            return self._as_rdict().impl_fallback_getitem(w_key)
 
     def impl_iter(self):
         return StrIteratorImplementation(self.space, self)
@@ -414,7 +414,7 @@
             StrDictImplementation.impl_setitem_str(
                 self, self.space.str_w(w_key), w_value, False)
         else:
-            self._as_rdict().setitem(w_key, w_value)
+            self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
     def impl_shadows_anything(self):
         return (self._shadows_anything or 
@@ -446,7 +446,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
 
     def impl_get_builtin_indexed(self, i):
         return self.shadowed[i]

Modified: pypy/branch/jitffi/pypy/objspace/std/model.py
==============================================================================
--- pypy/branch/jitffi/pypy/objspace/std/model.py	(original)
+++ pypy/branch/jitffi/pypy/objspace/std/model.py	Mon Oct 18 11:22:20 2010
@@ -18,6 +18,7 @@
     "withsmallint"   : ["smallintobject.W_SmallIntObject"],
     "withstrslice"   : ["strsliceobject.W_StringSliceObject"],
     "withstrjoin"    : ["strjoinobject.W_StringJoinObject"],
+    "withstrbuf"     : ["strbufobject.W_StringBufferObject"],
     "withrope"       : ["ropeobject.W_RopeObject",
                         "ropeobject.W_RopeIterObject"],
     "withropeunicode": ["ropeunicodeobject.W_RopeUnicodeObject",
@@ -75,6 +76,7 @@
         from pypy.objspace.std import ropeunicodeobject
         from pypy.objspace.std import strsliceobject
         from pypy.objspace.std import strjoinobject
+        from pypy.objspace.std import strbufobject
         from pypy.objspace.std import typeobject
         from pypy.objspace.std import sliceobject
         from pypy.objspace.std import longobject
@@ -222,6 +224,13 @@
                 (unicodeobject.W_UnicodeObject,
                                        strjoinobject.delegate_join2unicode)
                 ]
+        elif config.objspace.std.withstrbuf:
+            self.typeorder[strbufobject.W_StringBufferObject] += [
+                (stringobject.W_StringObject,
+                                       strbufobject.delegate_buf2str),
+                (unicodeobject.W_UnicodeObject,
+                                       strbufobject.delegate_buf2unicode)
+                ]
         if config.objspace.std.withrangelist:
             self.typeorder[rangeobject.W_RangeListObject] += [
                 (listobject.W_ListObject,

Modified: pypy/branch/jitffi/pypy/objspace/std/objspace.py
==============================================================================
--- pypy/branch/jitffi/pypy/objspace/std/objspace.py	(original)
+++ pypy/branch/jitffi/pypy/objspace/std/objspace.py	Mon Oct 18 11:22:20 2010
@@ -23,6 +23,7 @@
 from pypy.objspace.std.listobject import W_ListObject
 from pypy.objspace.std.longobject import W_LongObject
 from pypy.objspace.std.noneobject import W_NoneObject
+from pypy.objspace.std.objectobject import W_ObjectObject
 from pypy.objspace.std.ropeobject import W_RopeObject
 from pypy.objspace.std.iterobject import W_SeqIterObject
 from pypy.objspace.std.setobject import W_SetObject, W_FrozensetObject
@@ -317,9 +318,14 @@
             w_subtype = w_type.check_user_subclass(w_subtype)
             if cls.typedef.applevel_subclasses_base is not None:
                 cls = cls.typedef.applevel_subclasses_base
-            subcls = get_unique_interplevel_subclass(
-                    self.config, cls, w_subtype.hasdict, w_subtype.nslots != 0,
-                    w_subtype.needsdel, w_subtype.weakrefable)
+            if (self.config.objspace.std.withmapdict and cls is W_ObjectObject
+                    and not w_subtype.needsdel):
+                from pypy.objspace.std.mapdict import get_subclass_of_correct_size
+                subcls = get_subclass_of_correct_size(self, cls, w_subtype)
+            else:
+                subcls = get_unique_interplevel_subclass(
+                        self.config, cls, w_subtype.hasdict, w_subtype.nslots != 0,
+                        w_subtype.needsdel, w_subtype.weakrefable)
             instance = instantiate(subcls)
             assert isinstance(instance, cls)
             instance.user_setup(self, w_subtype)

Modified: pypy/branch/jitffi/pypy/objspace/std/sharingdict.py
==============================================================================
--- pypy/branch/jitffi/pypy/objspace/std/sharingdict.py	(original)
+++ pypy/branch/jitffi/pypy/objspace/std/sharingdict.py	Mon Oct 18 11:22:20 2010
@@ -71,7 +71,7 @@
         elif _is_sane_hash(space, w_lookup_type):
             return None
         else:
-            return self._as_rdict().getitem(w_lookup)
+            return self._as_rdict().impl_fallback_getitem(w_lookup)
 
     def impl_getitem_str(self, lookup):
         i = self.structure.lookup_position(lookup)
@@ -84,7 +84,7 @@
         if space.is_w(space.type(w_key), space.w_str):
             self.impl_setitem_str(self.space.str_w(w_key), w_value)
         else:
-            self._as_rdict().setitem(w_key, w_value)
+            self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
     @unroll_safe
     def impl_setitem_str(self, key, w_value, shadows_type=True):
@@ -132,7 +132,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
         
     def impl_length(self):
         return self.structure.length

Modified: pypy/branch/jitffi/pypy/objspace/std/stringobject.py
==============================================================================
--- pypy/branch/jitffi/pypy/objspace/std/stringobject.py	(original)
+++ pypy/branch/jitffi/pypy/objspace/std/stringobject.py	Mon Oct 18 11:22:20 2010
@@ -14,7 +14,7 @@
 from pypy.rlib.rstring import StringBuilder, string_repeat
 from pypy.interpreter.buffer import StringBuffer
 
-from pypy.objspace.std.stringtype import sliced, joined, wrapstr, wrapchar, \
+from pypy.objspace.std.stringtype import sliced, wrapstr, wrapchar, \
      stringendswith, stringstartswith, joined2
 
 from pypy.objspace.std.formatting import mod_format

Modified: pypy/branch/jitffi/pypy/objspace/std/stringtype.py
==============================================================================
--- pypy/branch/jitffi/pypy/objspace/std/stringtype.py	(original)
+++ pypy/branch/jitffi/pypy/objspace/std/stringtype.py	Mon Oct 18 11:22:20 2010
@@ -55,19 +55,14 @@
             return W_StringSliceObject(s, start, stop)
     return wrapstr(space, s[start:stop])
 
-def joined(space, strlist):
-    assert not space.config.objspace.std.withrope
-    if space.config.objspace.std.withstrjoin:
-        from pypy.objspace.std.strjoinobject import W_StringJoinObject
-        return W_StringJoinObject(strlist)
-    else:
-        return wrapstr(space, "".join(strlist))
-
 def joined2(space, str1, str2):
     assert not space.config.objspace.std.withrope
     if space.config.objspace.std.withstrjoin:
         from pypy.objspace.std.strjoinobject import W_StringJoinObject
         return W_StringJoinObject([str1, str2])
+    elif space.config.objspace.std.withstrbuf:
+        from pypy.objspace.std.strbufobject import joined2
+        return joined2(str1, str2)
     else:
         return wrapstr(space, str1 + str2)
 

Modified: pypy/branch/jitffi/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/branch/jitffi/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/branch/jitffi/pypy/objspace/std/test/test_dictmultiobject.py	Mon Oct 18 11:22:20 2010
@@ -602,6 +602,15 @@
                 classofinstance=classofinstance,
                 from_strdict_shared=from_strdict_shared)
 
+    def finditem_str(self, w_dict, s):
+        return w_dict.getitem_str(s) # assume it's a multidict
+
+    def setitem_str(self, w_dict, s, w_value):
+        return w_dict.setitem_str(s, w_value) # assume it's a multidict
+
+    def delitem(self, w_dict, w_s):
+        return w_dict.delitem(w_s) # assume it's a multidict
+
     def allocate_instance(self, cls, type):
         return object.__new__(cls)
 
@@ -611,7 +620,7 @@
     w_StopIteration = StopIteration
     w_None = None
     StringObjectCls = FakeString
-    w_dict = None
+    w_dict = W_DictMultiObject
     iter = iter
     fixedview = list
     listview  = list
@@ -687,6 +696,14 @@
         assert self.impl.length() == 0
         self.check_not_devolved()
 
+    def test_clear(self):
+        self.fill_impl()
+        assert self.impl.length() == 2
+        self.impl.clear()
+        assert self.impl.length() == 0
+        self.check_not_devolved()
+
+
     def test_keys(self):
         self.fill_impl()
         keys = self.impl.keys()

Modified: pypy/branch/jitffi/pypy/objspace/std/test/test_shadowtracking.py
==============================================================================
--- pypy/branch/jitffi/pypy/objspace/std/test/test_shadowtracking.py	(original)
+++ pypy/branch/jitffi/pypy/objspace/std/test/test_shadowtracking.py	Mon Oct 18 11:22:20 2010
@@ -3,7 +3,8 @@
 
 class TestShadowTracking(object):
     def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withshadowtracking": True})
+        cls.space = gettestobjspace(**{"objspace.std.withshadowtracking": True,
+                                       "objspace.std.withmapdict": False})
 
     def test_simple_shadowing(self):
         space = self.space

Modified: pypy/branch/jitffi/pypy/objspace/std/typeobject.py
==============================================================================
--- pypy/branch/jitffi/pypy/objspace/std/typeobject.py	(original)
+++ pypy/branch/jitffi/pypy/objspace/std/typeobject.py	Mon Oct 18 11:22:20 2010
@@ -75,7 +75,9 @@
                           'weakrefable',
                           'hasdict',
                           'nslots',
-                          'instancetypedef']
+                          'instancetypedef',
+                          'terminator',
+                          ]
 
     # for config.objspace.std.getattributeshortcut
     # (False is a conservative default, fixed during real usage)
@@ -116,6 +118,12 @@
                 # dict_w of any of the types in the mro changes, or if the mro
                 # itself changes
                 w_self._version_tag = VersionTag()
+        if space.config.objspace.std.withmapdict:
+            from pypy.objspace.std.mapdict import DictTerminator, NoDictTerminator
+            if w_self.hasdict:
+                w_self.terminator = DictTerminator(space, w_self)
+            else:
+                w_self.terminator = NoDictTerminator(space, w_self)
 
     def mutated(w_self):
         space = w_self.space

Modified: pypy/branch/jitffi/pypy/rlib/_rsocket_rffi.py
==============================================================================
--- pypy/branch/jitffi/pypy/rlib/_rsocket_rffi.py	(original)
+++ pypy/branch/jitffi/pypy/rlib/_rsocket_rffi.py	Mon Oct 18 11:22:20 2010
@@ -324,10 +324,11 @@
                                ('sll_hatype', rffi.INT),
                                ('sll_addr', rffi.CFixedArray(rffi.CHAR, 8)),
                                ('sll_halen', rffi.INT)],
-                              )
+                              ifdef='AF_PACKET')
 
     CConfig.ifreq = platform.Struct('struct ifreq', [('ifr_ifindex', rffi.INT),
-                                 ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))])
+                                 ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))],
+                                    ifdef='AF_PACKET')
 
 if _WIN32:
     CConfig.WSAEVENT = platform.SimpleType('WSAEVENT', rffi.VOIDP)
@@ -532,8 +533,9 @@
     socketpair_t = rffi.CArray(socketfd_type)
     socketpair = external('socketpair', [rffi.INT, rffi.INT, rffi.INT,
                           lltype.Ptr(socketpair_t)], rffi.INT)
-    ioctl = external('ioctl', [socketfd_type, rffi.INT, lltype.Ptr(ifreq)],
-                     rffi.INT)
+    if ifreq is not None:
+        ioctl = external('ioctl', [socketfd_type, rffi.INT, lltype.Ptr(ifreq)],
+                         rffi.INT)
 
 if _WIN32:
     ioctlsocket = external('ioctlsocket',

Modified: pypy/branch/jitffi/pypy/rlib/clibffi.py
==============================================================================
--- pypy/branch/jitffi/pypy/rlib/clibffi.py	(original)
+++ pypy/branch/jitffi/pypy/rlib/clibffi.py	Mon Oct 18 11:22:20 2010
@@ -17,7 +17,6 @@
 import sys
 import ctypes.util
 
-
 # maaaybe isinstance here would be better. Think
 _MSVC = platform.name == "msvc"
 _MINGW = platform.name == "mingw32"

Modified: pypy/branch/jitffi/pypy/rlib/rmmap.py
==============================================================================
--- pypy/branch/jitffi/pypy/rlib/rmmap.py	(original)
+++ pypy/branch/jitffi/pypy/rlib/rmmap.py	Mon Oct 18 11:22:20 2010
@@ -50,7 +50,7 @@
     constant_names = ['MAP_SHARED', 'MAP_PRIVATE',
                       'PROT_READ', 'PROT_WRITE',
                       'MS_SYNC']
-    opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS',
+    opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS', 'MAP_NORESERVE',
                           'PROT_EXEC',
                           'MAP_DENYWRITE', 'MAP_EXECUTABLE']
     for name in constant_names:

Modified: pypy/branch/jitffi/pypy/rlib/rstring.py
==============================================================================
--- pypy/branch/jitffi/pypy/rlib/rstring.py	(original)
+++ pypy/branch/jitffi/pypy/rlib/rstring.py	Mon Oct 18 11:22:20 2010
@@ -54,6 +54,7 @@
         self.l = []
 
     def append(self, s):
+        assert isinstance(s, self._type)
         self.l.append(s)
 
     def append_slice(self, s, start, end):
@@ -63,11 +64,16 @@
     def append_multiple_char(self, c, times):
         self.l.append(c * times)
 
+    def getlength(self):
+        return len(self.build())
+
 class StringBuilder(AbstractStringBuilder):
+    _type = str
     def build(self):
         return "".join(self.l)
 
 class UnicodeBuilder(AbstractStringBuilder):
+    _type = unicode
     def build(self):
         return u''.join(self.l)
 
@@ -121,9 +127,12 @@
         assert s_times.nonneg
         return s_None
 
+    def method_getlength(self):
+        return SomeInteger(nonneg=True)
+
     def method_build(self):
         return SomeString()
-    
+
     def rtyper_makerepr(self, rtyper):
         return rtyper.type_system.rbuilder.stringbuilder_repr
 
@@ -146,6 +155,9 @@
         assert s_times.nonneg
         return s_None
 
+    def method_getlength(self):
+        return SomeInteger(nonneg=True)
+
     def method_build(self):
         return SomeUnicodeString()
     

Modified: pypy/branch/jitffi/pypy/rlib/test/test_rsocket.py
==============================================================================
--- pypy/branch/jitffi/pypy/rlib/test/test_rsocket.py	(original)
+++ pypy/branch/jitffi/pypy/rlib/test/test_rsocket.py	Mon Oct 18 11:22:20 2010
@@ -437,3 +437,31 @@
             foo = self.serv.accept()
         py.test.raises(SocketError, raise_error)
 
+def _test_cond_include(cond):
+    # Test that _rsocket_rffi is importable even on platforms where
+    # AF_PACKET or AF_NETLINK is not defined.
+    import re
+    from pypy.rlib import _rsocket_rffi
+    srcfile = _rsocket_rffi.__file__
+    if srcfile.lower().endswith('c') or srcfile.lower().endswith('o'):
+        srcfile = srcfile[:-1]      # .pyc => .py
+    assert srcfile.lower().endswith('.py')
+    sourcelines = open(srcfile, 'rb').read().splitlines()
+    found = False
+    for i, line in enumerate(sourcelines):
+        line2 = re.sub(r"(\s*COND_HEADER\s*=)",
+                      r"\1'#undef %s\\n'+" % cond,
+                      line)
+        if line2 != line:
+            found = True
+            sourcelines[i] = line2
+    assert found
+    d = {}
+    sourcelines.append('')
+    exec '\n'.join(sourcelines) in d
+
+def test_no_AF_PACKET():
+    _test_cond_include('AF_PACKET')
+
+def test_no_AF_NETLINK():
+    _test_cond_include('AF_NETLINK')

Modified: pypy/branch/jitffi/pypy/rlib/test/test_rstring.py
==============================================================================
--- pypy/branch/jitffi/pypy/rlib/test/test_rstring.py	(original)
+++ pypy/branch/jitffi/pypy/rlib/test/test_rstring.py	Mon Oct 18 11:22:20 2010
@@ -29,6 +29,7 @@
     s = StringBuilder()
     s.append("a")
     s.append("abc")
+    assert s.getlength() == len('aabc')
     s.append("a")
     s.append_slice("abc", 1, 2)
     s.append_multiple_char('d', 4)
@@ -39,6 +40,7 @@
     s.append(u'a')
     s.append(u'abc')
     s.append_slice(u'abcdef', 1, 2)
+    assert s.getlength() == len('aabcb')
     s.append_multiple_char('d', 4)
     assert s.build() == 'aabcbdddd'
     assert isinstance(s.build(), unicode)

Modified: pypy/branch/jitffi/pypy/rlib/test/test_runicode.py
==============================================================================
--- pypy/branch/jitffi/pypy/rlib/test/test_runicode.py	(original)
+++ pypy/branch/jitffi/pypy/rlib/test/test_runicode.py	Mon Oct 18 11:22:20 2010
@@ -76,7 +76,7 @@
                 assert start == startingpos
                 assert stop == endingpos
                 return u"42424242", stop
-            return "", endingpos
+            return u"", endingpos
         decoder = self.getdecoder(encoding)
         if addstuff:
             s += "some rest in ascii"

Modified: pypy/branch/jitffi/pypy/rpython/lltypesystem/ll2ctypes.py
==============================================================================
--- pypy/branch/jitffi/pypy/rpython/lltypesystem/ll2ctypes.py	(original)
+++ pypy/branch/jitffi/pypy/rpython/lltypesystem/ll2ctypes.py	Mon Oct 18 11:22:20 2010
@@ -69,7 +69,7 @@
                 PIECESIZE = 0x08000000
         PIECES = 10
         m = rmmap.mmap(-1, PIECES * PIECESIZE,
-                       rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS,
+                       rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS|rmmap.MAP_NORESERVE,
                        rmmap.PROT_READ|rmmap.PROT_WRITE)
         m.close = lambda : None    # leak instead of giving a spurious
                                    # error at CPython's shutdown

Modified: pypy/branch/jitffi/pypy/rpython/lltypesystem/rbuilder.py
==============================================================================
--- pypy/branch/jitffi/pypy/rpython/lltypesystem/rbuilder.py	(original)
+++ pypy/branch/jitffi/pypy/rpython/lltypesystem/rbuilder.py	Mon Oct 18 11:22:20 2010
@@ -100,6 +100,10 @@
         ll_builder.used = used
 
     @staticmethod
+    def ll_getlength(ll_builder):
+        return ll_builder.used
+
+    @staticmethod
     def ll_build(ll_builder):
         final_size = ll_builder.used
         assert final_size >= 0

Modified: pypy/branch/jitffi/pypy/rpython/lltypesystem/rclass.py
==============================================================================
--- pypy/branch/jitffi/pypy/rpython/lltypesystem/rclass.py	(original)
+++ pypy/branch/jitffi/pypy/rpython/lltypesystem/rclass.py	Mon Oct 18 11:22:20 2010
@@ -329,16 +329,33 @@
             fields['__class__'] = 'typeptr', get_type_repr(self.rtyper)
         else:
             # instance attributes
-            if llfields is None:
-                llfields = []
             attrs = self.classdef.attrs.items()
             attrs.sort()
+            myllfields = []
             for name, attrdef in attrs:
                 if not attrdef.readonly:
                     r = self.rtyper.getrepr(attrdef.s_value)
                     mangled_name = 'inst_' + name
                     fields[name] = mangled_name, r
-                    llfields.append((mangled_name, r.lowleveltype))
+                    myllfields.append((mangled_name, r.lowleveltype))
+
+            # Sort the instance attributes by decreasing "likely size",
+            # as reported by rffi.sizeof(), to minimize padding holes in C.
+            # Fields of the same size are sorted by name (by attrs.sort()
+            # above) just to minimize randomness.
+            def keysize((_, T)):
+                if T is lltype.Void:
+                    return None
+                from pypy.rpython.lltypesystem.rffi import sizeof
+                try:
+                    return -sizeof(T)
+                except StandardError:
+                    return None
+            myllfields.sort(key = keysize)
+            if llfields is None:
+                llfields = myllfields
+            else:
+                llfields = llfields + myllfields
 
             self.rbase = getinstancerepr(self.rtyper, self.classdef.basedef,
                                          self.gcflavor)

Modified: pypy/branch/jitffi/pypy/rpython/module/ll_time.py
==============================================================================
--- pypy/branch/jitffi/pypy/rpython/module/ll_time.py	(original)
+++ pypy/branch/jitffi/pypy/rpython/module/ll_time.py	Mon Oct 18 11:22:20 2010
@@ -108,7 +108,7 @@
 
                 errcode = -1
                 if self.GETTIMEOFDAY_NO_TZ:
-                    errcode = g_gettimeofday(t)
+                    errcode = c_gettimeofday(t)
                 else:
                     errcode = c_gettimeofday(t, void)
 

Modified: pypy/branch/jitffi/pypy/rpython/rbuilder.py
==============================================================================
--- pypy/branch/jitffi/pypy/rpython/rbuilder.py	(original)
+++ pypy/branch/jitffi/pypy/rpython/rbuilder.py	Mon Oct 18 11:22:20 2010
@@ -36,8 +36,12 @@
         hop.exception_cannot_occur()
         return hop.gendirectcall(self.ll_append_multiple_char, *vlist)
 
+    def rtype_method_getlength(self, hop):
+        vlist = hop.inputargs(self)
+        hop.exception_cannot_occur()
+        return hop.gendirectcall(self.ll_getlength, *vlist)
+
     def rtype_method_build(self, hop):
         vlist = hop.inputargs(self)
         hop.exception_cannot_occur()
         return hop.gendirectcall(self.ll_build, *vlist)
-

Modified: pypy/branch/jitffi/pypy/rpython/test/test_rbuilder.py
==============================================================================
--- pypy/branch/jitffi/pypy/rpython/test/test_rbuilder.py	(original)
+++ pypy/branch/jitffi/pypy/rpython/test/test_rbuilder.py	Mon Oct 18 11:22:20 2010
@@ -1,4 +1,4 @@
-
+import py
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.rpython.lltypesystem.rbuilder import *
 from pypy.rpython.annlowlevel import llstr, hlstr
@@ -55,8 +55,29 @@
         assert res == 'aabcabcdefbuuuu'
         assert isinstance(res, unicode)
 
+    def test_string_getlength(self):
+        def func():
+            s = StringBuilder()
+            s.append("a")
+            s.append("abc")
+            return s.getlength()
+        res = self.interpret(func, [])
+        assert res == 4
+
+    def test_unicode_getlength(self):
+        def func():
+            s = UnicodeBuilder()
+            s.append(u"a")
+            s.append(u"abc")
+            return s.getlength()
+        res = self.interpret(func, [])
+        assert res == 4
+
 class TestLLtype(BaseTestStringBuilder, LLRtypeMixin):
     pass
 
 class TestOOtype(BaseTestStringBuilder, OORtypeMixin):
-    pass
+    def test_string_getlength(self):
+        py.test.skip("getlength(): not implemented on ootype")
+    def test_unicode_getlength(self):
+        py.test.skip("getlength(): not implemented on ootype")

Modified: pypy/branch/jitffi/pypy/rpython/test/test_rclass.py
==============================================================================
--- pypy/branch/jitffi/pypy/rpython/test/test_rclass.py	(original)
+++ pypy/branch/jitffi/pypy/rpython/test/test_rclass.py	Mon Oct 18 11:22:20 2010
@@ -3,7 +3,7 @@
 from pypy.translator.translator import TranslationContext, graphof
 from pypy.rpython.lltypesystem.lltype import *
 from pypy.rpython.ootypesystem import ootype
-from pypy.rlib.rarithmetic import intmask
+from pypy.rlib.rarithmetic import intmask, r_longlong
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.objspace.flow.model import summary
 
@@ -1012,6 +1012,40 @@
         res = self.interpret(f, [5])
         assert res == 0
 
+    def test_order_of_fields(self):
+        class A(object):
+            pass
+        def f(n):
+            a = A()
+            a.as_int = n
+            a.as_char = chr(n)
+            a.as_unichar = unichr(n)
+            a.as_double = n + 0.5
+            a.as_bool = bool(n)
+            a.as_void = None
+            a.as_longlong = r_longlong(n)
+            a.as_reference = A()
+            return a
+
+        res = self.interpret(f, [5])
+        names = list(typeOf(res).TO._names)
+        i = names.index('inst_as_int')
+        c = names.index('inst_as_char')
+        u = names.index('inst_as_unichar')
+        d = names.index('inst_as_double')
+        b = names.index('inst_as_bool')
+        v = names.index('inst_as_void')
+        l = names.index('inst_as_longlong')
+        r = names.index('inst_as_reference')
+        assert v == 1      # void fields are first
+        assert sorted([c, b]) == [7, 8]
+        if sys.maxint == 2147483647:
+            assert sorted([u, i, r]) == [4, 5, 6]        # 32-bit types
+            assert sorted([d, l]) == [2, 3]              # 64-bit types
+        else:
+            assert sorted([u]) == [6]                    # 32-bit types
+            assert sorted([i, r, d, l]) == [2, 3, 4, 5]  # 64-bit types
+
 
 class TestOOtype(BaseTestRclass, OORtypeMixin):
 

Modified: pypy/branch/jitffi/pypy/translator/c/gcc/test/elf/track5.s
==============================================================================
--- pypy/branch/jitffi/pypy/translator/c/gcc/test/elf/track5.s	(original)
+++ pypy/branch/jitffi/pypy/translator/c/gcc/test/elf/track5.s	Mon Oct 18 11:22:20 2010
@@ -44,7 +44,7 @@
 	addl	%eax, %ebx
 	jmp	.L1221
 .L1227:
-	call	RPyAbort
+	;;call	RPyAbort
 	cmpl	12(%esi), %ebx
 	jb	.L1229
 	addl	$20, %esp

Modified: pypy/branch/jitffi/pypy/translator/c/gcc/trackgcroot.py
==============================================================================
--- pypy/branch/jitffi/pypy/translator/c/gcc/trackgcroot.py	(original)
+++ pypy/branch/jitffi/pypy/translator/c/gcc/trackgcroot.py	Mon Oct 18 11:22:20 2010
@@ -46,6 +46,7 @@
         self.findlabels()
         self.parse_instructions()
         try:
+            self.trim_unreachable_instructions()
             self.find_noncollecting_calls()
             if not self.list_collecting_call_insns():
                 return []
@@ -122,6 +123,26 @@
                 assert label not in self.labels, "duplicate label: %s" % label
                 self.labels[label] = Label(label, lineno)
 
+    def trim_unreachable_instructions(self):
+        reached = set([self.insns[0]])
+        prevlen = 0
+        while len(reached) > prevlen:
+            prevlen = len(reached)
+            for insn in self.insns:
+                if insn not in reached:
+                    for previnsn in insn.previous_insns:
+                        if previnsn in reached:
+                            # this instruction is reachable too
+                            reached.add(insn)
+                            break
+        # now kill all unreachable instructions
+        i = 0
+        while i < len(self.insns):
+            if self.insns[i] in reached:
+                i += 1
+            else:
+                del self.insns[i]
+
     def find_noncollecting_calls(self):
         cannot_collect = {}
         for line in self.lines:
@@ -752,7 +773,7 @@
                     target, = sources
 
         if target in self.FUNCTIONS_NOT_RETURNING:
-            return [InsnStop(target), InsnCannotFollowEsp()]
+            return [InsnStop(target)]
         if self.format == 'mingw32' and target == '__alloca':
             # in functions with large stack requirements, windows
             # needs a call to _alloca(), to turn reserved pages

Modified: pypy/branch/jitffi/pypy/translator/c/src/signals.h
==============================================================================
--- pypy/branch/jitffi/pypy/translator/c/src/signals.h	(original)
+++ pypy/branch/jitffi/pypy/translator/c/src/signals.h	Mon Oct 18 11:22:20 2010
@@ -6,20 +6,6 @@
 
 #include 
 
-#ifndef LONG_MAX
-#if SIZEOF_LONG == 4
-#define LONG_MAX 0X7FFFFFFFL
-#elif SIZEOF_LONG == 8
-#define LONG_MAX 0X7FFFFFFFFFFFFFFFL
-#else
-#error "could not set LONG_MAX in pyport.h"
-#endif
-#endif
-
-#ifndef LONG_MIN
-#define LONG_MIN (-LONG_MAX-1)
-#endif
-
 #include 
 
 #ifdef MS_WINDOWS
@@ -28,10 +14,6 @@
 
 #include 
 
-#ifndef SIG_ERR
-#define SIG_ERR ((PyOS_sighandler_t)(-1))
-#endif
-
 #if defined(PYOS_OS2) && !defined(PYCC_GCC)
 #define NSIG 12
 #include 
@@ -65,11 +47,11 @@
 /* utility to poll for signals that arrived */
 int pypysig_poll(void);   /* => signum or -1 */
 
-/* When a signal is received, the high bit of pypysig_occurred is set.
-   After all signals are processed by pypysig_poll(), the high bit is
+/* When a signal is received, the bit 30 of pypysig_occurred is set.
+   After all signals are processed by pypysig_poll(), the bit 30 is
    cleared again.  The variable is exposed and RPython code is free to
    use the other bits in any way. */
-#define PENDING_SIGNAL_BIT   (LONG_MIN)   /* high bit */
+#define PENDING_SIGNAL_BIT   (1 << 30)
 /* This is a struct for the JIT. See interp_signal.py. */
 struct pypysig_long_struct {
     long value;


From antocuni at codespeak.net  Mon Oct 18 11:27:55 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Mon, 18 Oct 2010 11:27:55 +0200 (CEST)
Subject: [pypy-svn] r78034 - in pypy/branch/jitffi/pypy/jit/codewriter: .
	test
Message-ID: <20101018092755.C1037282B90@codespeak.net>

Author: antocuni
Date: Mon Oct 18 11:27:54 2010
New Revision: 78034

Modified:
   pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py
   pypy/branch/jitffi/pypy/jit/codewriter/test/test_effectinfo.py
Log:
add a test to check that we don't have any duplicate in EffectInfo.OS_*. Also, move libffi stuff at the bottom


Modified: pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py	(original)
+++ pypy/branch/jitffi/pypy/jit/codewriter/effectinfo.py	Mon Oct 18 11:27:54 2010
@@ -19,10 +19,7 @@
     OS_NONE                     = 0    # normal case, no oopspec
     OS_ARRAYCOPY                = 1    # "list.ll_arraycopy"
     OS_STR2UNICODE              = 2    # "str.str2unicode"
-    OS_LIBFFI_PREPARE           = 15
-    OS_LIBFFI_PUSH_ARG          = 16
-    OS_LIBFFI_CALL              = 17
-
+    #
     OS_STR_CONCAT               = 22   # "stroruni.concat"
     OS_STR_SLICE                = 23   # "stroruni.slice"
     OS_STR_EQUAL                = 24   # "stroruni.equal"
@@ -33,7 +30,7 @@
     OS_STREQ_NONNULL_CHAR       = 29   # s1 == char  (assert s1!=NULL)
     OS_STREQ_CHECKNULL_CHAR     = 30   # s1!=NULL and s1==char
     OS_STREQ_LENGTHOK           = 31   # s1 == s2    (assert len(s1)==len(s2))
-
+    #
     OS_UNI_CONCAT               = 42   #
     OS_UNI_SLICE                = 43   #
     OS_UNI_EQUAL                = 44   #
@@ -45,6 +42,10 @@
     OS_UNIEQ_CHECKNULL_CHAR     = 50   #   STR, in the same order)
     OS_UNIEQ_LENGTHOK           = 51   #
     _OS_offset_uni              = OS_UNI_CONCAT - OS_STR_CONCAT
+    #
+    OS_LIBFFI_PREPARE           = 60
+    OS_LIBFFI_PUSH_ARG          = 61
+    OS_LIBFFI_CALL              = 62
 
     def __new__(cls, readonly_descrs_fields,
                 write_descrs_fields, write_descrs_arrays,

Modified: pypy/branch/jitffi/pypy/jit/codewriter/test/test_effectinfo.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/codewriter/test/test_effectinfo.py	(original)
+++ pypy/branch/jitffi/pypy/jit/codewriter/test/test_effectinfo.py	Mon Oct 18 11:27:54 2010
@@ -1,7 +1,8 @@
 from pypy.rpython.lltypesystem.rclass import OBJECT
 from pypy.rpython.lltypesystem import lltype
 from pypy.rpython.ootypesystem import ootype
-from pypy.jit.codewriter.effectinfo import effectinfo_from_writeanalyze
+from pypy.jit.codewriter.effectinfo import effectinfo_from_writeanalyze,\
+    EffectInfo
 
 class FakeCPU:
     def fielddescrof(self, T, fieldname):
@@ -9,6 +10,14 @@
     def arraydescrof(self, A):
         return ('arraydescr', A)
 
+def test_no_oopspec_duplicate():
+    # check that all the various EffectInfo.OS_* have unique values
+    oopspecs = set()
+    for name, value in EffectInfo.__dict__.iteritems():
+        if name.startswith('OS_'):
+            assert value not in oopspecs
+            oopspecs.add(value)
+
 def test_include_read_field():
     S = lltype.GcStruct("S", ("a", lltype.Signed))
     effects = frozenset([("readstruct", lltype.Ptr(S), "a")])


From antocuni at codespeak.net  Mon Oct 18 11:32:29 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Mon, 18 Oct 2010 11:32:29 +0200 (CEST)
Subject: [pypy-svn] r78035 -
	pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt
Message-ID: <20101018093229.55E75282B90@codespeak.net>

Author: antocuni
Date: Mon Oct 18 11:32:27 2010
New Revision: 78035

Modified:
   pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
Log:
try to make the annotator happier (should not be necessary, but tests misteriously fail with buildbot :-( )


Modified: pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	(original)
+++ pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py	Mon Oct 18 11:32:27 2010
@@ -11,6 +11,8 @@
     argtypes = None
     restype = None
     descr = None
+    prepare_op = None
+    force_token_op = None
 
     def __init__(self, funcval, cpu, prepare_op):
         self.funcval = funcval
@@ -18,7 +20,6 @@
         argtypes, restype = self._get_signature(funcval)
         self.descr = cpu.calldescrof_dynamic(argtypes, restype)
         self.prepare_op = prepare_op
-        self.force_token_op = None
 
     def _get_signature(self, funcval):
         """


From antocuni at codespeak.net  Mon Oct 18 13:18:15 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Mon, 18 Oct 2010 13:18:15 +0200 (CEST)
Subject: [pypy-svn] r78036 - pypy/branch/jitffi/pypy/doc/config
Message-ID: <20101018111815.EA178282B90@codespeak.net>

Author: antocuni
Date: Mon Oct 18 13:18:13 2010
New Revision: 78036

Added:
   pypy/branch/jitffi/pypy/doc/config/objspace.usemodules._ffi.txt   (contents, props changed)
Log:
add doc for _ffi


Added: pypy/branch/jitffi/pypy/doc/config/objspace.usemodules._ffi.txt
==============================================================================
--- (empty file)
+++ pypy/branch/jitffi/pypy/doc/config/objspace.usemodules._ffi.txt	Mon Oct 18 13:18:13 2010
@@ -0,0 +1 @@
+Applevel interface to libffi.  It is more high level than _rawffi, and most importantly it is JIT friendly


From afa at codespeak.net  Mon Oct 18 14:54:19 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 18 Oct 2010 14:54:19 +0200 (CEST)
Subject: [pypy-svn] r78038 - in pypy/branch/fast-forward/pypy/objspace/std:
	. test
Message-ID: <20101018125419.BE6CF282B90@codespeak.net>

Author: afa
Date: Mon Oct 18 14:54:17 2010
New Revision: 78038

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py
   pypy/branch/fast-forward/pypy/objspace/std/typeobject.py
Log:
Better test for the "safe subclass" case:
allow subclasses with a TypeDef, as long as they don't redefine the __new__ slot.
(like CPython which compares the tp_new slots)


Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py	Mon Oct 18 14:54:17 2010
@@ -28,6 +28,39 @@
             raises(TypeError, type, 'sub', (stufftype,), {})
         """)
 
+    def test_safe_subclass(self):
+        space = self.space
+        class W_Base(W_Object):
+            pass
+        def descr__new__(space, w_subtype):
+            return space.allocate_instance(W_Base, w_subtype)
+        W_Base.typedef = StdTypeDef("base",
+                                     __new__ = interp2app(descr__new__))
+        w_base = space.gettypeobject(W_Base.typedef)
+
+        # Cannot be created with base.__new__(derived)
+        class W_Derived(W_Base):
+            pass
+        def descr__new__(space, w_subtype):
+            return space.allocate_instance(W_Derived, w_subtype)
+        W_Derived.typedef = StdTypeDef("derived", W_Base.typedef,
+                                     __new__ = interp2app(descr__new__))
+        w_derived = space.gettypeobject(W_Derived.typedef)
+
+        # Can be created with base.__new__(derived2)
+        class W_Derived2(W_Base):
+            pass
+        W_Derived2.typedef = StdTypeDef("derived2", W_Base.typedef,
+                                        )
+        w_derived2 = space.gettypeobject(W_Derived2.typedef)
+
+        space.appexec([w_base, w_derived, w_derived2],
+                      """(base, derived, derived2):
+            raises(TypeError, base.__new__, derived)
+            x = base.__new__(derived2)
+            assert isinstance(x, derived2)
+            """)
+
     def test_del_warning(self):
         warnings = []
         def my_warn(msg, warningscls):

Modified: pypy/branch/fast-forward/pypy/objspace/std/typeobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/typeobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/typeobject.py	Mon Oct 18 14:54:17 2010
@@ -326,7 +326,8 @@
             raise operationerrfmt(space.w_TypeError,
                 "%s.__new__(%s): %s is not a subtype of %s",
                 w_self.name, w_subtype.name, w_subtype.name, w_self.name)
-        if w_self.instancetypedef is not w_subtype.instancetypedef:
+        if (w_self.instancetypedef is not w_subtype.instancetypedef and
+            w_self.lookup('__new__') is not w_subtype.lookup('__new__')):
             raise operationerrfmt(space.w_TypeError,
                 "%s.__new__(%s) is not safe, use %s.__new__()",
                 w_self.name, w_subtype.name, w_subtype.name)


From antocuni at codespeak.net  Mon Oct 18 15:26:46 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Mon, 18 Oct 2010 15:26:46 +0200 (CEST)
Subject: [pypy-svn] r78039 - pypy/branch/jitffi/pypy/jit/backend/x86/test
Message-ID: <20101018132646.0E629282B90@codespeak.net>

Author: antocuni
Date: Mon Oct 18 15:26:45 2010
New Revision: 78039

Modified:
   pypy/branch/jitffi/pypy/jit/backend/x86/test/test_zrpy_gc.py
Log:
a workaround for these failing tests, and a huge comment explaining it


Modified: pypy/branch/jitffi/pypy/jit/backend/x86/test/test_zrpy_gc.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/backend/x86/test/test_zrpy_gc.py	(original)
+++ pypy/branch/jitffi/pypy/jit/backend/x86/test/test_zrpy_gc.py	Mon Oct 18 15:26:45 2010
@@ -191,6 +191,32 @@
     def run_orig(self, name, n, x):
         self.main_allfuncs(name, n, x)
 
+    def define_libffi_workaround(cls):
+        # This is a workaround for a bug in database.py.  It seems that the
+        # problem is triggered by optimizeopt/fficall.py, and in particular by
+        # the ``cast_base_ptr_to_instance(Func, llfunc)``: in these tests,
+        # that line is the only place where libffi.Func is referenced.
+        #
+        # The problem occurs because the gctransformer tries to annotate a
+        # low-level helper to call the __del__ of libffi.Func when it's too
+        # late.
+        #
+        # This workaround works by forcing the annotator (and all the rest of
+        # the toolchain) to see libffi.Func in a "proper" context, not just as
+        # the target of cast_base_ptr_to_instance.  Note that the function
+        # below is *never* called by any actual test, it's just annotated.
+        #
+        from pypy.rlib.libffi import get_libc_name, CDLL, types, ArgChain
+        libc_name = get_libc_name()
+        def f(n, x, *args):
+            libc = CDLL(libc_name)
+            ptr = libc.getpointer('labs', [types.slong], types.slong)
+            chain = ArgChain()
+            chain.arg(n)
+            n = ptr.call(chain, lltype.Signed)
+            return (n, x) + args
+        return None, f, None
+
     def define_compile_framework_1(cls):
         # a moving GC.  Supports malloc_varsize_nonmovable.  Simple test, works
         # without write_barriers and root stack enumeration.


From antocuni at codespeak.net  Mon Oct 18 15:42:04 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Mon, 18 Oct 2010 15:42:04 +0200 (CEST)
Subject: [pypy-svn] r78040 - pypy/branch/jitffi/pypy/jit/backend/x86/test
Message-ID: <20101018134204.BBFC2282B90@codespeak.net>

Author: antocuni
Date: Mon Oct 18 15:42:03 2010
New Revision: 78040

Modified:
   pypy/branch/jitffi/pypy/jit/backend/x86/test/test_zrpy_gc.py
Log:
add an XXX to the comment, to make armin happy :-)


Modified: pypy/branch/jitffi/pypy/jit/backend/x86/test/test_zrpy_gc.py
==============================================================================
--- pypy/branch/jitffi/pypy/jit/backend/x86/test/test_zrpy_gc.py	(original)
+++ pypy/branch/jitffi/pypy/jit/backend/x86/test/test_zrpy_gc.py	Mon Oct 18 15:42:03 2010
@@ -192,10 +192,11 @@
         self.main_allfuncs(name, n, x)
 
     def define_libffi_workaround(cls):
-        # This is a workaround for a bug in database.py.  It seems that the
-        # problem is triggered by optimizeopt/fficall.py, and in particular by
-        # the ``cast_base_ptr_to_instance(Func, llfunc)``: in these tests,
-        # that line is the only place where libffi.Func is referenced.
+        # XXX: this is a workaround for a bug in database.py.  It seems that
+        # the problem is triggered by optimizeopt/fficall.py, and in
+        # particular by the ``cast_base_ptr_to_instance(Func, llfunc)``: in
+        # these tests, that line is the only place where libffi.Func is
+        # referenced.
         #
         # The problem occurs because the gctransformer tries to annotate a
         # low-level helper to call the __del__ of libffi.Func when it's too


From arigo at codespeak.net  Mon Oct 18 15:43:19 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 15:43:19 +0200 (CEST)
Subject: [pypy-svn] r78041 - pypy/branch/rsre-jit/pypy/jit/codewriter
Message-ID: <20101018134319.CACEF36E3D6@codespeak.net>

Author: arigo
Date: Mon Oct 18 15:43:18 2010
New Revision: 78041

Modified:
   pypy/branch/rsre-jit/pypy/jit/codewriter/codewriter.py
Log:
Write the jitcode into files, even if it is also printed.


Modified: pypy/branch/rsre-jit/pypy/jit/codewriter/codewriter.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/codewriter/codewriter.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/codewriter/codewriter.py	Mon Oct 18 15:43:18 2010
@@ -95,18 +95,18 @@
             print '%s:' % (ssarepr.name,)
             print format_assembler(ssarepr)
         else:
-            dir = udir.ensure("jitcodes", dir=1)
-            if portal_jitdriver:
-                name = "%02d_portal_runner" % (portal_jitdriver.index,)
-            elif ssarepr.name and ssarepr.name != '?':
-                name = ssarepr.name
-            else:
-                name = 'unnamed' % id(ssarepr)
-            i = 1
-            extra = ''
-            while name+extra in self._seen_files:
-                i += 1
-                extra = '.%d' % i
-            self._seen_files.add(name+extra)
-            dir.join(name+extra).write(format_assembler(ssarepr))
             log.dot()
+        dir = udir.ensure("jitcodes", dir=1)
+        if portal_jitdriver:
+            name = "%02d_portal_runner" % (portal_jitdriver.index,)
+        elif ssarepr.name and ssarepr.name != '?':
+            name = ssarepr.name
+        else:
+            name = 'unnamed' % id(ssarepr)
+        i = 1
+        extra = ''
+        while name+extra in self._seen_files:
+            i += 1
+            extra = '.%d' % i
+        self._seen_files.add(name+extra)
+        dir.join(name+extra).write(format_assembler(ssarepr))


From arigo at codespeak.net  Mon Oct 18 15:43:33 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 15:43:33 +0200 (CEST)
Subject: [pypy-svn] r78042 - pypy/branch/rsre-jit/pypy/jit/codewriter
Message-ID: <20101018134333.0AD90282B90@codespeak.net>

Author: arigo
Date: Mon Oct 18 15:43:31 2010
New Revision: 78042

Modified:
   pypy/branch/rsre-jit/pypy/jit/codewriter/support.py
Log:
Fix comment.


Modified: pypy/branch/rsre-jit/pypy/jit/codewriter/support.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/codewriter/support.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/codewriter/support.py	Mon Oct 18 15:43:31 2010
@@ -60,7 +60,7 @@
     return rtyper.annotator.translator.graphs[0]
 
 def split_before_jit_merge_point(graph, portalblock, portalopindex):
-    """Find the block with 'jit_merge_point' and split just before,
+    """Split the block just before the 'jit_merge_point',
     making sure the input args are in the canonical order.
     """
     # split the block just before the jit_merge_point()


From arigo at codespeak.net  Mon Oct 18 15:49:50 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 15:49:50 +0200 (CEST)
Subject: [pypy-svn] r78043 - in pypy/branch/rsre-jit/pypy: jit/codewriter
	jit/codewriter/test jit/metainterp jit/metainterp/optimizeopt
	jit/metainterp/test rlib rlib/rsre rlib/rsre/test
Message-ID: <20101018134950.EA7C036E3D6@codespeak.net>

Author: arigo
Date: Mon Oct 18 15:49:48 2010
New Revision: 78043

Added:
   pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_jit.py
Modified:
   pypy/branch/rsre-jit/pypy/jit/codewriter/jtransform.py
   pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_codewriter.py
   pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_jtransform.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/blackhole.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/executor.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/jitdriver.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/optimizeopt/string.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/optimizeopt/virtualize.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/pyjitpl.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/resoperation.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/resume.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_basic.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_greenfield.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_virtualref.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_warmspot.py
   pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py
   pypy/branch/rsre-jit/pypy/rlib/jit.py
   pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_char.py
   pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_core.py
   pypy/branch/rsre-jit/pypy/rlib/rsre/test/test_zjit.py
Log:
Main change: allows us to call jitdriver.jit_merge_point()
without ever calling jitdriver.can_enter_jit().  In that case
the jit_merge_point() plays both roles.  The difference with
putting explicitly a can_enter_jit() just before is that such
a can_enter_jit() is not seen unless we are closing a loop;
in particular, it does not work if we have no loop at all.

Tests and fixes for greenfields.

Implement rlib.jit.jit_debug(), which just causes a JIT_DEBUG
resoperation to show up in the trace of the JIT.

Implement rlib.jit.assert_green(), which fails if the JIT considers
that the argument is not a Const.

When not translated, add the 'FORCE' prefix to the name of resoperations
produced by forcing virtualizables.


Modified: pypy/branch/rsre-jit/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/codewriter/jtransform.py	Mon Oct 18 15:49:48 2010
@@ -320,6 +320,8 @@
             prepare = self._handle_str2unicode_call
         elif oopspec_name.startswith('virtual_ref'):
             prepare = self._handle_virtual_ref_call
+        elif oopspec_name.startswith('jit.'):
+            prepare = self._handle_jit_call
         else:
             prepare = self.prepare_builtin_call
         try:
@@ -859,6 +861,15 @@
                     (self.graph,))
         return []
 
+    def _handle_jit_call(self, op, oopspec_name, args):
+        if oopspec_name == 'jit.debug':
+            return SpaceOperation('jit_debug', args, None)
+        elif oopspec_name == 'jit.assert_green':
+            kind = getkind(args[0].concretetype)
+            return SpaceOperation('%s_assert_green' % kind, args, None)
+        else:
+            raise AssertionError("missing support for %r" % oopspec_name)
+
     # ----------
     # Lists.
 

Modified: pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_codewriter.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_codewriter.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_codewriter.py	Mon Oct 18 15:49:48 2010
@@ -45,6 +45,7 @@
         self.portal_graph = portal_graph
         self.portal_runner_ptr = "???"
         self.virtualizable_info = None
+        self.greenfield_info = None
 
 
 def test_loop():

Modified: pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/codewriter/test/test_jtransform.py	Mon Oct 18 15:49:48 2010
@@ -721,6 +721,45 @@
     assert list(oplist[4].args[4]) == [v3, v4]
     assert oplist[5].opname == '-live-'
 
+def test_getfield_gc():
+    S = lltype.GcStruct('S', ('x', lltype.Char))
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
+def test_getfield_gc_pure():
+    S = lltype.GcStruct('S', ('x', lltype.Char),
+                        hints={'immutable': True})
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i_pure'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
+def test_getfield_gc_greenfield():
+    class FakeCC:
+        def get_vinfo(self, v):
+            return None
+        def could_be_green_field(self, S1, name1):
+            assert S1 is S
+            assert name1 == 'x'
+            return True
+    S = lltype.GcStruct('S', ('x', lltype.Char),
+                        hints={'immutable': True})
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU(), FakeCC()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i_greenfield'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
 def test_int_abs():
     v1 = varoftype(lltype.Signed)
     v2 = varoftype(lltype.Signed)

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/blackhole.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/blackhole.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/blackhole.py	Mon Oct 18 15:49:48 2010
@@ -760,6 +760,20 @@
     def bhimpl_debug_fatalerror(msg):
         llop.debug_fatalerror(lltype.Void, msg)
 
+    @arguments("r", "i", "i", "i", "i")
+    def bhimpl_jit_debug(string, arg1=0, arg2=0, arg3=0, arg4=0):
+        pass
+
+    @arguments("i")
+    def bhimpl_int_assert_green(x):
+        pass
+    @arguments("r")
+    def bhimpl_ref_assert_green(x):
+        pass
+    @arguments("f")
+    def bhimpl_float_assert_green(x):
+        pass
+
     # ----------
     # the main hints and recursive calls
 

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/executor.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/executor.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/executor.py	Mon Oct 18 15:49:48 2010
@@ -304,6 +304,7 @@
                          rop.CALL_ASSEMBLER,
                          rop.COND_CALL_GC_WB,
                          rop.DEBUG_MERGE_POINT,
+                         rop.JIT_DEBUG,
                          rop.SETARRAYITEM_RAW,
                          ):      # list of opcodes never executed by pyjitpl
                 continue

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/jitdriver.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/jitdriver.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/jitdriver.py	Mon Oct 18 15:49:48 2010
@@ -16,6 +16,7 @@
     #    self.greenfield_info   ... pypy.jit.metainterp.warmspot
     #    self.warmstate         ... pypy.jit.metainterp.warmspot
     #    self.handle_jitexc_from_bh pypy.jit.metainterp.warmspot
+    #    self.no_loop_header    ... pypy.jit.metainterp.warmspot
     #    self.portal_finishtoken... pypy.jit.metainterp.pyjitpl
     #    self.index             ... pypy.jit.codewriter.call
     #    self.mainjitcode       ... pypy.jit.codewriter.call

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/optimizeopt/string.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/optimizeopt/string.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/optimizeopt/string.py	Mon Oct 18 15:49:48 2010
@@ -12,7 +12,7 @@
 from pypy.jit.codewriter.effectinfo import EffectInfo, callinfo_for_oopspec
 from pypy.jit.codewriter import heaptracker
 from pypy.rlib.unroll import unrolling_iterable
-from pypy.rlib.objectmodel import specialize
+from pypy.rlib.objectmodel import specialize, we_are_translated
 
 
 class StrOrUnicode(object):
@@ -107,7 +107,10 @@
         self.box = box = self.source_op.result
         newoperations = self.optimizer.newoperations
         lengthbox = self.getstrlen(newoperations, self.mode)
-        newoperations.append(ResOperation(self.mode.NEWSTR, [lengthbox], box))
+        op = ResOperation(self.mode.NEWSTR, [lengthbox], box)
+        if not we_are_translated():
+            op.name = 'FORCE'
+        newoperations.append(op)
         self.string_copy_parts(newoperations, box, CONST_0, self.mode)
 
 

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/optimizeopt/virtualize.py	Mon Oct 18 15:49:48 2010
@@ -74,6 +74,8 @@
         assert self.source_op is not None
         # ^^^ This case should not occur any more (see test_bug_3).
         #
+        if not we_are_translated():
+            self.source_op.name = 'FORCE ' + self.source_op.name
         newoperations = self.optimizer.newoperations
         newoperations.append(self.source_op)
         self.box = box = self.source_op.result
@@ -165,6 +167,8 @@
 
     def _really_force(self):
         assert self.source_op is not None
+        if not we_are_translated():
+            self.source_op.name = 'FORCE ' + self.source_op.name
         newoperations = self.optimizer.newoperations
         newoperations.append(self.source_op)
         self.box = box = self.source_op.result

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/pyjitpl.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/pyjitpl.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/pyjitpl.py	Mon Oct 18 15:49:48 2010
@@ -1,4 +1,4 @@
-import py, os
+import py, os, sys
 from pypy.rpython.lltypesystem import lltype, llmemory, rclass
 from pypy.rlib.objectmodel import we_are_translated
 from pypy.rlib.unroll import unrolling_iterable
@@ -816,12 +816,16 @@
 
     @arguments("orgpc", "int", "boxes3", "boxes3")
     def opimpl_jit_merge_point(self, orgpc, jdindex, greenboxes, redboxes):
+        any_operation = len(self.metainterp.history.operations) > 0
         jitdriver_sd = self.metainterp.staticdata.jitdrivers_sd[jdindex]
         self.verify_green_args(jitdriver_sd, greenboxes)
         # xxx we may disable the following line in some context later
         self.debug_merge_point(jitdriver_sd, greenboxes)
         if self.metainterp.seen_loop_header_for_jdindex < 0:
-            return
+            if not jitdriver_sd.no_loop_header or not any_operation:
+                return
+            # automatically add a loop_header if there is none
+            self.metainterp.seen_loop_header_for_jdindex = jdindex
         #
         assert self.metainterp.seen_loop_header_for_jdindex == jdindex, (
             "found a loop_header for a JitDriver that does not match "
@@ -910,6 +914,40 @@
         msg = box.getref(lltype.Ptr(rstr.STR))
         lloperation.llop.debug_fatalerror(msg)
 
+    @arguments("box", "box", "box", "box", "box")
+    def opimpl_jit_debug(self, stringbox, arg1box, arg2box, arg3box, arg4box):
+        from pypy.rpython.lltypesystem import rstr
+        from pypy.rpython.annlowlevel import hlstr
+        msg = stringbox.getref(lltype.Ptr(rstr.STR))
+        debug_print('jit_debug:', hlstr(msg),
+                    arg1box.getint(), arg2box.getint(),
+                    arg3box.getint(), arg4box.getint())
+        args = [stringbox, arg1box, arg2box, arg3box, arg4box]
+        i = 4
+        while i > 0 and args[i].getint() == -sys.maxint-1:
+            i -= 1
+        assert i >= 0
+        op = self.metainterp.history.record(rop.JIT_DEBUG, args[:i+1], None)
+        self.metainterp.attach_debug_info(op)
+
+    @arguments("box")
+    def _opimpl_assert_green(self, box):
+        if not isinstance(box, Const):
+            msg = "assert_green failed at %s:%d" % (
+                self.jitcode.name,
+                self.pc)
+            if we_are_translated():
+                from pypy.rpython.annlowlevel import llstr
+                from pypy.rpython.lltypesystem import lloperation
+                lloperation.llop.debug_fatalerror(lltype.Void, llstr(msg))
+            else:
+                from pypy.rlib.jit import AssertGreenFailed
+                raise AssertGreenFailed(msg)
+
+    opimpl_int_assert_green   = _opimpl_assert_green
+    opimpl_ref_assert_green   = _opimpl_assert_green
+    opimpl_float_assert_green = _opimpl_assert_green
+
     @arguments("box")
     def opimpl_virtual_ref(self, box):
         # Details on the content of metainterp.virtualref_boxes:
@@ -1022,8 +1060,7 @@
         if resumepc >= 0:
             self.pc = resumepc
         resume.capture_resumedata(metainterp.framestack, virtualizable_boxes,
-                                  metainterp.virtualref_boxes,
-                                  resumedescr)
+                                  metainterp.virtualref_boxes, resumedescr)
         self.pc = saved_pc
         self.metainterp.staticdata.profiler.count_ops(opnum, GUARDS)
         # count

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/resoperation.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/resoperation.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/resoperation.py	Mon Oct 18 15:49:48 2010
@@ -459,6 +459,7 @@
     #'RUNTIMENEW/1',     # ootype operation    
     'COND_CALL_GC_WB/2d', # [objptr, newvalue]   (for the write barrier)
     'DEBUG_MERGE_POINT/1',      # debugging only
+    'JIT_DEBUG/*',              # debugging only
     'VIRTUAL_REF_FINISH/2',   # removed before it's passed to the backend
     'COPYSTRCONTENT/5',       # src, dst, srcstart, dststart, length
     'COPYUNICODECONTENT/5',

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/resume.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/resume.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/resume.py	Mon Oct 18 15:49:48 2010
@@ -738,15 +738,18 @@
         assert (end & 1) == 0
         return [self.decode_ref(nums[i]) for i in range(end)]
 
-    def consume_vref_and_vable_boxes(self, vinfo):
+    def consume_vref_and_vable_boxes(self, vinfo, ginfo):
         nums = self.cur_numb.nums
         self.cur_numb = self.cur_numb.prev
-        if vinfo is None:
-            virtualizable_boxes = None
-            end = len(nums)
-        else:
+        if vinfo is not None:
             virtualizable_boxes = self.consume_virtualizable_boxes(vinfo, nums)
             end = len(nums) - len(virtualizable_boxes)
+        elif ginfo is not None:
+            virtualizable_boxes = [self.decode_ref(nums[-1])]
+            end = len(nums) - 1
+        else:
+            virtualizable_boxes = None
+            end = len(nums)
         virtualref_boxes = self.consume_virtualref_boxes(nums, end)
         return virtualizable_boxes, virtualref_boxes
 

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_basic.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_basic.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_basic.py	Mon Oct 18 15:49:48 2010
@@ -2,6 +2,7 @@
 import sys
 from pypy.rlib.jit import JitDriver, we_are_jitted, hint, dont_look_inside
 from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_SIMPLE, loop_invariant
+from pypy.rlib.jit import jit_debug, assert_green, AssertGreenFailed
 from pypy.jit.metainterp.warmspot import ll_meta_interp, get_stats
 from pypy.jit.backend.llgraph import runner
 from pypy.jit.metainterp import pyjitpl, history
@@ -44,6 +45,7 @@
         num_green_args = 0
         portal_graph = graphs[0]
         virtualizable_info = None
+        greenfield_info = None
         result_type = result_kind
         portal_runner_ptr = "???"
 
@@ -1644,6 +1646,33 @@
         res = self.interp_operations(f, [10, 3.5])
         assert res == 3.5
 
+    def test_jit_debug(self):
+        myjitdriver = JitDriver(greens = [], reds = ['x'])
+        class A:
+            pass
+        def f(x):
+            while x > 0:
+                myjitdriver.can_enter_jit(x=x)
+                myjitdriver.jit_merge_point(x=x)
+                jit_debug("hi there:", x)
+                jit_debug("foobar")
+                x -= 1
+            return x
+        res = self.meta_interp(f, [8])
+        assert res == 0
+        self.check_loops(jit_debug=2)
+
+    def test_assert_green(self):
+        def f(x, promote):
+            if promote:
+                x = hint(x, promote=True)
+            assert_green(x)
+            return x
+        res = self.interp_operations(f, [8, 1])
+        assert res == 8
+        py.test.raises(AssertGreenFailed, self.interp_operations, f, [8, 0])
+
+
 class TestOOtype(BasicTests, OOJitMixin):
 
     def test_oohash(self):

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_greenfield.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_greenfield.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_greenfield.py	Mon Oct 18 15:49:48 2010
@@ -27,6 +27,31 @@
         self.check_loop_count(2)
         self.check_loops(guard_value=0)
 
+    def test_green_field_2(self):
+        myjitdriver = JitDriver(greens=['ctx.x'], reds=['ctx'])
+        class Ctx(object):
+            _immutable_fields_ = ['x']
+            def __init__(self, x, y):
+                self.x = x
+                self.y = y
+        def f(x, y):
+            ctx = Ctx(x, y)
+            while 1:
+                myjitdriver.can_enter_jit(ctx=ctx)
+                myjitdriver.jit_merge_point(ctx=ctx)
+                ctx.y -= 1
+                if ctx.y < 0:
+                    pass     # to just make two paths
+                if ctx.y < -10:
+                    return ctx.y
+        def g(y):
+            return f(5, y) + f(6, y)
+        #
+        res = self.meta_interp(g, [7])
+        assert res == -22
+        self.check_loop_count(4)
+        self.check_loops(guard_value=0)
+
 
 class TestLLtypeGreenFieldsTests(GreenFieldsTests, LLJitMixin):
     pass

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_virtualref.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_virtualref.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_virtualref.py	Mon Oct 18 15:49:48 2010
@@ -93,7 +93,7 @@
         lst = []
         vrefinfo.continue_tracing = lambda vref, virtual: \
                                         lst.append((vref, virtual))
-        resumereader.consume_vref_and_vable(vrefinfo, None)
+        resumereader.consume_vref_and_vable(vrefinfo, None, None)
         del vrefinfo.continue_tracing
         assert len(lst) == 1
         lltype.cast_opaque_ptr(lltype.Ptr(JIT_VIRTUAL_REF),

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_warmspot.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_warmspot.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_warmspot.py	Mon Oct 18 15:49:48 2010
@@ -296,6 +296,69 @@
         assert res == 1
         self.check_loops(int_add=1)   # I get 13 without the loop_header()
 
+    def test_omit_can_enter_jit(self):
+        # Simple test comparing the effects of always giving a can_enter_jit(),
+        # or not giving any.  Mostly equivalent, except that if given, it is
+        # ignored the first time, and so it ends up taking one extra loop to
+        # start JITting.
+        mydriver = JitDriver(greens=[], reds=['m'])
+        #
+        for i2 in range(10):
+            def f2(m):
+                while m > 0:
+                    mydriver.jit_merge_point(m=m)
+                    m -= 1
+            self.meta_interp(f2, [i2])
+            try:
+                self.check_tree_loop_count(1)
+                break
+            except AssertionError:
+                print "f2: no loop generated for i2==%d" % i2
+        else:
+            raise     # re-raise the AssertionError: check_loop_count never 1
+        #
+        for i1 in range(10):
+            def f1(m):
+                while m > 0:
+                    mydriver.can_enter_jit(m=m)
+                    mydriver.jit_merge_point(m=m)
+                    m -= 1
+            self.meta_interp(f1, [i1])
+            try:
+                self.check_tree_loop_count(1)
+                break
+            except AssertionError:
+                print "f1: no loop generated for i1==%d" % i1
+        else:
+            raise     # re-raise the AssertionError: check_loop_count never 1
+        #
+        assert i1 - 1 == i2
+
+    def test_no_loop_at_all(self):
+        mydriver = JitDriver(greens=[], reds=['m'])
+        def f2(m):
+            mydriver.jit_merge_point(m=m)
+            return m - 1
+        def f1(m):
+            while m > 0:
+                m = f2(m)
+        self.meta_interp(f1, [8])
+        # it should generate one "loop" only, which ends in a FINISH
+        # corresponding to the return from f2.
+        self.check_tree_loop_count(1)
+        self.check_loop_count(0)
+
+    def test_simple_loop(self):
+        mydriver = JitDriver(greens=[], reds=['m'])
+        def f1(m):
+            while m > 0:
+                mydriver.jit_merge_point(m=m)
+                m = m - 1
+        self.meta_interp(f1, [8])
+        self.check_loop_count(1)
+        self.check_loops({'int_sub': 1, 'int_gt': 1, 'guard_true': 1,
+                          'jump': 1})
+
 
 class TestLLWarmspot(WarmspotTests, LLJitMixin):
     CPUClass = runner.LLtypeCPU

Modified: pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/metainterp/warmspot.py	Mon Oct 18 15:49:48 2010
@@ -115,10 +115,10 @@
     return results
 
 def find_can_enter_jit(graphs):
-    results = _find_jit_marker(graphs, 'can_enter_jit')
-    if not results:
-        raise Exception("no can_enter_jit found!")
-    return results
+    return _find_jit_marker(graphs, 'can_enter_jit')
+
+def find_loop_headers(graphs):
+    return _find_jit_marker(graphs, 'loop_header')
 
 def find_jit_merge_points(graphs):
     results = _find_jit_marker(graphs, 'jit_merge_point')
@@ -483,26 +483,37 @@
             [lltype.Signed, llmemory.GCREF], RESTYPE)
 
     def rewrite_can_enter_jits(self):
-        can_enter_jits = find_can_enter_jit(self.translator.graphs)
         sublists = {}
         for jd in self.jitdrivers_sd:
-            sublists[jd.jitdriver] = []
+            sublists[jd.jitdriver] = jd, []
+            jd.no_loop_header = True
+        #
+        loop_headers = find_loop_headers(self.translator.graphs)
+        for graph, block, index in loop_headers:
+            op = block.operations[index]
+            jitdriver = op.args[1].value
+            assert jitdriver in sublists, \
+                   "loop_header with no matching jit_merge_point"
+            jd, sublist = sublists[jitdriver]
+            jd.no_loop_header = False
+        #
+        can_enter_jits = find_can_enter_jit(self.translator.graphs)
         for graph, block, index in can_enter_jits:
             op = block.operations[index]
             jitdriver = op.args[1].value
             assert jitdriver in sublists, \
                    "can_enter_jit with no matching jit_merge_point"
+            jd, sublist = sublists[jitdriver]
             origportalgraph = jd._jit_merge_point_pos[0]
             if graph is not origportalgraph:
-                sublists[jitdriver].append((graph, block, index))
+                sublist.append((graph, block, index))
+                jd.no_loop_header = False
             else:
                 pass   # a 'can_enter_jit' before the 'jit-merge_point', but
                        # originally in the same function: we ignore it here
                        # see e.g. test_jitdriver.test_simple
         for jd in self.jitdrivers_sd:
-            sublist = sublists[jd.jitdriver]
-            assert len(sublist) > 0, \
-                   "found no can_enter_jit for %r" % (jd.jitdriver,)
+            _, sublist = sublists[jd.jitdriver]
             self.rewrite_can_enter_jit(jd, sublist)
 
     def rewrite_can_enter_jit(self, jd, can_enter_jits):
@@ -510,6 +521,19 @@
         FUNCPTR = jd._PTR_JIT_ENTER_FUNCTYPE
         jit_enter_fnptr = self.helper_func(FUNCPTR, jd._maybe_enter_jit_fn)
 
+        if len(can_enter_jits) == 0:
+            # see test_warmspot.test_no_loop_at_all
+            operations = jd.portal_graph.startblock.operations
+            op1 = operations[0]
+            assert (op1.opname == 'jit_marker' and
+                    op1.args[0].value == 'jit_merge_point')
+            op0 = SpaceOperation(
+                'jit_marker',
+                [Constant('can_enter_jit', lltype.Void)] + op1.args[1:],
+                None)
+            operations.insert(0, op0)
+            can_enter_jits = [(jd.portal_graph, jd.portal_graph.startblock, 0)]
+
         for graph, block, index in can_enter_jits:
             if graph is jd._jit_merge_point_pos[0]:
                 continue

Modified: pypy/branch/rsre-jit/pypy/rlib/jit.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/rlib/jit.py	(original)
+++ pypy/branch/rsre-jit/pypy/rlib/jit.py	Mon Oct 18 15:49:48 2010
@@ -139,6 +139,24 @@
         return hop.inputconst(lltype.Signed, _we_are_jitted)
 
 
+def jit_debug(string, arg1=-sys.maxint-1, arg2=-sys.maxint-1,
+                      arg3=-sys.maxint-1, arg4=-sys.maxint-1):
+    """When JITted, cause an extra operation DEBUG_MERGE_POINT to appear in
+    the graphs.  Should not be left after debugging."""
+    keepalive_until_here(string) # otherwise the whole function call is removed
+jit_debug.oopspec = 'jit.debug(string, arg1, arg2, arg3, arg4)'
+
+def assert_green(value):
+    """Very strong assert: checks that 'value' is a green
+    (a JIT compile-time constant)."""
+    keepalive_until_here(value)
+assert_green._annspecialcase_ = 'specialize:argtype(0)'
+assert_green.oopspec = 'jit.assert_green(value)'
+
+class AssertGreenFailed(Exception):
+    pass
+
+
 ##def force_virtualizable(virtualizable):
 ##    pass
 
@@ -266,7 +284,8 @@
             self.virtualizables = virtualizables
         for v in self.virtualizables:
             assert v in self.reds
-        self._alllivevars = dict.fromkeys(self.greens + self.reds)
+        self._alllivevars = dict.fromkeys(
+            [name for name in self.greens + self.reds if '.' not in name])
         self._make_extregistryentries()
         self.get_jitcell_at = get_jitcell_at
         self.set_jitcell_at = set_jitcell_at

Modified: pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_char.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_char.py	(original)
+++ pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_char.py	Mon Oct 18 15:49:48 2010
@@ -4,6 +4,7 @@
 import sys
 from pypy.rlib.rlocale import tolower, isalnum
 from pypy.rlib.unroll import unrolling_iterable
+from pypy.rlib import jit
 
 # Note: the unicode parts of this module require you to call
 # rsre_char.set_unicode_db() first, to select one of the modules
@@ -43,6 +44,7 @@
 # XXX can we import those safely from sre_constants?
 SRE_INFO_PREFIX = 1
 SRE_INFO_LITERAL = 2
+SRE_INFO_CHARSET = 4
 SRE_FLAG_LOCALE = 4 # honour system locale
 SRE_FLAG_UNICODE = 32 # use unicode locale
 OPCODE_INFO = 17
@@ -64,33 +66,27 @@
 
 #### Category helpers
 
-ascii_char_info = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 6, 2,
-2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0,
-0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 25, 25, 25, 25, 25, 25, 25, 25,
-25, 25, 0, 0, 0, 0, 0, 0, 0, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
-24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 0, 0,
-0, 0, 16, 0, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
-24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 0, 0, 0, 0, 0 ]
-
+is_a_word = [(chr(i).isalnum() or chr(i) == '_') for i in range(256)]
 linebreak = ord("\n")
 underline = ord("_")
 
 def is_digit(code):
-    return code < 128 and (ascii_char_info[code] & 1 != 0)
+    return code <= 57 and code >= 48
 
 def is_uni_digit(code):
     assert unicodedb is not None
     return unicodedb.isdigit(code)
 
 def is_space(code):
-    return code < 128 and (ascii_char_info[code] & 2 != 0)
+    return code == 32 or (code <= 13 and code >= 9)
 
 def is_uni_space(code):
     assert unicodedb is not None
     return unicodedb.isspace(code)
 
 def is_word(code):
-    return code < 128 and (ascii_char_info[code] & 16 != 0)
+    assert code >= 0
+    return code < 256 and is_a_word[code]
 
 def is_uni_word(code):
     assert unicodedb is not None
@@ -142,6 +138,7 @@
 SET_OK = -1
 SET_NOT_OK = -2
 
+ at jit.unroll_safe
 def check_charset(pattern, ppos, char_code):
     """Checks whether a character matches set of arbitrary length.
     The set starts at pattern[ppos]."""

Modified: pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_core.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_core.py	(original)
+++ pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_core.py	Mon Oct 18 15:49:48 2010
@@ -5,6 +5,7 @@
 from pypy.tool.sourcetools import func_with_new_name
 from pypy.rlib.objectmodel import we_are_translated
 from pypy.rlib import jit
+from pypy.rlib.rsre.rsre_jit import install_jitdriver, install_jitdriver_spec
 
 
 OPCODE_FAILURE            = 0
@@ -69,7 +70,7 @@
         return getattr(ctx, specname)(*args)
     dispatch._annspecialcase_ = 'specialize:argtype(0)'
     dispatch._specialized_methods_ = specialized_methods
-    return dispatch
+    return func_with_new_name(dispatch, specname)
 
 # ____________________________________________________________
 
@@ -79,7 +80,7 @@
 
 class AbstractMatchContext(object):
     """Abstract base class"""
-    _immutable_fields_ = ['pattern[*]', 'flags']
+    _immutable_fields_ = ['pattern[*]', 'flags', 'end']
     match_start = 0
     match_end = 0
     match_marks = None
@@ -245,7 +246,7 @@
 
     @jit.unroll_safe
     def find_first_result(self, ctx):
-        ppos = self.ppos
+        ppos = jit.hint(self.ppos, promote=True)
         while ctx.pat(ppos):
             result = sre_match(ctx, ppos + 1, self.start_ptr, self.start_marks)
             ppos += ctx.pat(ppos)
@@ -256,8 +257,10 @@
     find_next_result = find_first_result
 
 class RepeatOneMatchResult(MatchResult):
-    jitdriver = jit.JitDriver(greens=['nextppos', 'pattern'],
-                              reds=['ptr', 'self', 'ctx'])
+    install_jitdriver('RepeatOne',
+                      greens=['nextppos', 'ctx.pattern'],
+                      reds=['ptr', 'self', 'ctx'],
+                      debugprint=(1, 0))   # indices in 'greens'
 
     def __init__(self, nextppos, minptr, ptr, marks):
         self.nextppos = nextppos
@@ -269,15 +272,8 @@
         ptr = self.start_ptr
         nextppos = self.nextppos
         while ptr >= self.minptr:
-            #
-            pattern = ctx.pattern
-            self.jitdriver.can_enter_jit(self=self, ptr=ptr, ctx=ctx,
-                                         nextppos=nextppos, pattern=pattern)
-            self.jitdriver.jit_merge_point(self=self, ptr=ptr, ctx=ctx,
-                                           nextppos=nextppos, pattern=pattern)
-            if jit.we_are_jitted():
-                ctx.pattern = pattern
-            #
+            ctx.jitdriver_RepeatOne.jit_merge_point(
+                self=self, ptr=ptr, ctx=ctx, nextppos=nextppos)
             result = sre_match(ctx, nextppos, ptr, self.start_marks)
             ptr -= 1
             if result is not None:
@@ -288,8 +284,10 @@
 
 
 class MinRepeatOneMatchResult(MatchResult):
-    jitdriver = jit.JitDriver(greens=['nextppos', 'ppos3', 'pattern'],
-                              reds=['ptr', 'self', 'ctx'])
+    install_jitdriver('MinRepeatOne',
+                      greens=['nextppos', 'ppos3', 'ctx.pattern'],
+                      reds=['ptr', 'self', 'ctx'],
+                      debugprint=(2, 0))   # indices in 'greens'
 
     def __init__(self, nextppos, ppos3, maxptr, ptr, marks):
         self.nextppos = nextppos
@@ -303,17 +301,8 @@
         nextppos = self.nextppos
         ppos3 = self.ppos3
         while ptr <= self.maxptr:
-            #
-            pattern = ctx.pattern
-            self.jitdriver.can_enter_jit(self=self, ptr=ptr, ctx=ctx,
-                                         nextppos=nextppos, pattern=pattern,
-                                         ppos3=ppos3)
-            self.jitdriver.jit_merge_point(self=self, ptr=ptr, ctx=ctx,
-                                           nextppos=nextppos, pattern=pattern,
-                                           ppos3=ppos3)
-            if jit.we_are_jitted():
-                ctx.pattern = pattern
-            #
+            ctx.jitdriver_MinRepeatOne.jit_merge_point(
+                self=self, ptr=ptr, ctx=ctx, nextppos=nextppos, ppos3=ppos3)
             result = sre_match(ctx, nextppos, ptr, self.start_marks)
             if result is not None:
                 self.subresult = result
@@ -334,7 +323,7 @@
         if ptr == ctx.end:
             return False
         op = ctx.pat(ppos)
-        for op1, (checkerfn, _) in unroll_char_checker:
+        for op1, checkerfn in unroll_char_checker:
             if op1 == op:
                 return checkerfn(ctx, ptr, ppos)
         raise Error("next_char_ok[%d]" % op)
@@ -357,41 +346,34 @@
         self.next = next     # chained list
 
 class MaxUntilMatchResult(AbstractUntilMatchResult):
+    install_jitdriver('MaxUntil',
+                      greens=['ppos', 'tailppos', 'match_more', 'ctx.pattern'],
+                      reds=['ptr', 'marks', 'self', 'ctx'],
+                      debugprint=(3, 0, 2))
 
     def find_first_result(self, ctx):
-        enum = sre_match(ctx, self.ppos + 3, self.cur_ptr, self.cur_marks)
-        return self.search_next(ctx, enum, resume=False)
+        return self.search_next(ctx, match_more=True)
 
     def find_next_result(self, ctx):
-        return self.search_next(ctx, None, resume=True)
+        return self.search_next(ctx, match_more=False)
 
-    def search_next(self, ctx, enum, resume):
+    def search_next(self, ctx, match_more):
         ppos = self.ppos
-        min = ctx.pat(ppos+1)
-        max = ctx.pat(ppos+2)
+        tailppos = self.tailppos
         ptr = self.cur_ptr
         marks = self.cur_marks
         while True:
-            while True:
-                if (enum is not None and
-                    (ptr != ctx.match_end or self.num_pending < min)):
-                    #               ^^^^^^^^^^ zero-width match protection
-                    # matched one more 'item'.  record it and continue.
-                    self.pending = Pending(ptr, marks, enum, self.pending)
-                    self.num_pending += 1
-                    ptr = ctx.match_end
-                    marks = ctx.match_marks
-                    break
-                # 'item' no longer matches.
-                if not resume and self.num_pending >= min:
-                    # try to match 'tail' if we have enough 'item'
-                    result = sre_match(ctx, self.tailppos, ptr, marks)
-                    if result is not None:
-                        self.subresult = result
-                        self.cur_ptr = ptr
-                        self.cur_marks = marks
-                        return self
-                resume = False
+            ctx.jitdriver_MaxUntil.jit_merge_point(
+                ppos=ppos, tailppos=tailppos, match_more=match_more,
+                ptr=ptr, marks=marks, self=self, ctx=ctx)
+            if match_more:
+                max = ctx.pat(ppos+2)
+                if max == 65535 or self.num_pending < max:
+                    # try to match one more 'item'
+                    enum = sre_match(ctx, ppos + 3, ptr, marks)
+                else:
+                    enum = None    # 'max' reached, no more matches
+            else:
                 p = self.pending
                 if p is None:
                     return
@@ -401,11 +383,27 @@
                 marks = p.marks
                 enum = p.enum.move_to_next_result(ctx)
             #
-            if max == 65535 or self.num_pending < max:
-                # try to match one more 'item'
-                enum = sre_match(ctx, ppos + 3, ptr, marks)
+            min = ctx.pat(ppos+1)
+            if (enum is not None and
+                (ptr != ctx.match_end or self.num_pending < min)):
+                #               ^^^^^^^^^^ zero-width match protection
+                # matched one more 'item'.  record it and continue.
+                self.pending = Pending(ptr, marks, enum, self.pending)
+                self.num_pending += 1
+                ptr = ctx.match_end
+                marks = ctx.match_marks
+                match_more = True
             else:
-                enum = None    # 'max' reached, no more matches
+                # 'item' no longer matches.
+                if self.num_pending >= min:
+                    # try to match 'tail' if we have enough 'item'
+                    result = sre_match(ctx, tailppos, ptr, marks)
+                    if result is not None:
+                        self.subresult = result
+                        self.cur_ptr = ptr
+                        self.cur_marks = marks
+                        return self
+                match_more = False
 
 class MinUntilMatchResult(AbstractUntilMatchResult):
 
@@ -416,6 +414,7 @@
         return self.search_next(ctx, resume=True)
 
     def search_next(self, ctx, resume):
+        # XXX missing jit support here
         ppos = self.ppos
         min = ctx.pat(ppos+1)
         max = ctx.pat(ppos+2)
@@ -471,6 +470,12 @@
         op = ctx.pat(ppos)
         ppos += 1
 
+        #jit.jit_debug("sre_match", op, ppos, ptr)
+        #
+        # When using the JIT, calls to sre_match() must always have a constant
+        # (green) argument for 'ppos'.  If not, the following assert fails.
+        jit.assert_green(op)
+
         if op == OPCODE_FAILURE:
             return
 
@@ -745,13 +750,23 @@
 @specializectx
 def find_repetition_end(ctx, ppos, ptr, maxcount):
     end = ctx.end
-    # adjust end
-    if maxcount != 65535:
+    if maxcount <= 1:
+        if maxcount == 1 and ptr < end:
+            # Relatively common case: maxcount == 1.  If we are not at the
+            # end of the string, it's done by a single direct check.
+            op = ctx.pat(ppos)
+            for op1, checkerfn in unroll_char_checker:
+                if op1 == op:
+                    if checkerfn(ctx, ptr, ppos):
+                        return ptr + 1
+        return ptr
+    elif maxcount != 65535:
+        # adjust end
         end1 = ptr + maxcount
         if end1 <= end:
             end = end1
     op = ctx.pat(ppos)
-    for op1, (_, fre) in unroll_char_checker:
+    for op1, fre in unroll_fre_checker:
         if op1 == op:
             return fre(ctx, ptr, end, ppos)
     raise Error("rsre.find_repetition_end[%d]" % op)
@@ -784,23 +799,60 @@
     if checkerfn == match_ANY_ALL:
         def fre(ctx, ptr, end, ppos):
             return end
+    elif checkerfn == match_IN:
+        install_jitdriver_spec('MatchIn', 
+                               greens=['ppos', 'ctx.pattern'],
+                               reds=['ptr', 'end', 'ctx'],
+                               debugprint=(1, 0))
+        @specializectx
+        def fre(ctx, ptr, end, ppos):
+            while True:
+                ctx.jitdriver_MatchIn.jit_merge_point(ctx=ctx, ptr=ptr,
+                                                      end=end, ppos=ppos)
+                if ptr < end and checkerfn(ctx, ptr, ppos):
+                    ptr += 1
+                else:
+                    return ptr
+    elif checkerfn == match_IN_IGNORE:
+        install_jitdriver_spec('MatchInIgnore', 
+                               greens=['ppos', 'ctx.pattern'],
+                               reds=['ptr', 'end', 'ctx'],
+                               debugprint=(1, 0))
+        @specializectx
+        def fre(ctx, ptr, end, ppos):
+            while True:
+                ctx.jitdriver_MatchInIgnore.jit_merge_point(ctx=ctx, ptr=ptr,
+                                                            end=end, ppos=ppos)
+                if ptr < end and checkerfn(ctx, ptr, ppos):
+                    ptr += 1
+                else:
+                    return ptr
     else:
+        # in the other cases, the fre() function is not JITted at all
+        # and is present as a residual call.
+        @specializectx
         def fre(ctx, ptr, end, ppos):
             while ptr < end and checkerfn(ctx, ptr, ppos):
                 ptr += 1
             return ptr
-    return checkerfn, fre
+    fre = func_with_new_name(fre, 'fre_' + checkerfn.__name__)
+    return fre
+
+unroll_char_checker = [
+    (OPCODE_ANY,                match_ANY),
+    (OPCODE_ANY_ALL,            match_ANY_ALL),
+    (OPCODE_IN,                 match_IN),
+    (OPCODE_IN_IGNORE,          match_IN_IGNORE),
+    (OPCODE_LITERAL,            match_LITERAL),
+    (OPCODE_LITERAL_IGNORE,     match_LITERAL_IGNORE),
+    (OPCODE_NOT_LITERAL,        match_NOT_LITERAL),
+    (OPCODE_NOT_LITERAL_IGNORE, match_NOT_LITERAL_IGNORE),
+    ]
+unroll_fre_checker = [(_op, _make_fre(_fn))
+                      for (_op, _fn) in unroll_char_checker]
 
-unroll_char_checker = unrolling_iterable([
-    (OPCODE_ANY,                _make_fre(match_ANY)),
-    (OPCODE_ANY_ALL,            _make_fre(match_ANY_ALL)),
-    (OPCODE_IN,                 _make_fre(match_IN)),
-    (OPCODE_IN_IGNORE,          _make_fre(match_IN_IGNORE)),
-    (OPCODE_LITERAL,            _make_fre(match_LITERAL)),
-    (OPCODE_LITERAL_IGNORE,     _make_fre(match_LITERAL_IGNORE)),
-    (OPCODE_NOT_LITERAL,        _make_fre(match_NOT_LITERAL)),
-    (OPCODE_NOT_LITERAL_IGNORE, _make_fre(match_NOT_LITERAL_IGNORE)),
-    ])
+unroll_char_checker = unrolling_iterable(unroll_char_checker)
+unroll_fre_checker  = unrolling_iterable(unroll_fre_checker)
 
 ##### At dispatch
 
@@ -906,74 +958,139 @@
     else:
         return None
 
+install_jitdriver('Match',
+                  greens=['ctx.pattern'], reds=['ctx'],
+                  debugprint=(0,))
+
 def match_context(ctx):
     ctx.original_pos = ctx.match_start
     if ctx.end < ctx.match_start:
         return False
+    ctx.jitdriver_Match.jit_merge_point(ctx=ctx)
     return sre_match(ctx, 0, ctx.match_start, None) is not None
 
 def search_context(ctx):
     ctx.original_pos = ctx.match_start
     if ctx.end < ctx.match_start:
         return False
-    if ctx.pat(0) == OPCODE_INFO:
-        if ctx.pat(2) & rsre_char.SRE_INFO_PREFIX and ctx.pat(5) > 1:
-            return fast_search(ctx)
-    return regular_search(ctx)
+    base = 0
+    charset = False
+    if ctx.pat(base) == OPCODE_INFO:
+        flags = ctx.pat(2)
+        if flags & rsre_char.SRE_INFO_PREFIX:
+            if ctx.pat(5) > 1:
+                return fast_search(ctx)
+        else:
+            charset = (flags & rsre_char.SRE_INFO_CHARSET)
+        base += 1 + ctx.pat(1)
+    if ctx.pat(base) == OPCODE_LITERAL:
+        return literal_search(ctx, base)
+    if charset:
+        return charset_search(ctx, base)
+    return regular_search(ctx, base)
+
+install_jitdriver('RegularSearch',
+                  greens=['base', 'ctx.pattern'],
+                  reds=['start', 'ctx'],
+                  debugprint=(1, 0))
 
-def regular_search(ctx):
+def regular_search(ctx, base):
     start = ctx.match_start
     while start <= ctx.end:
-        if sre_match(ctx, 0, start, None) is not None:
+        ctx.jitdriver_RegularSearch.jit_merge_point(ctx=ctx, start=start,
+                                                    base=base)
+        if sre_match(ctx, base, start, None) is not None:
             ctx.match_start = start
             return True
         start += 1
     return False
 
+install_jitdriver_spec("LiteralSearch",
+                       greens=['base', 'character', 'ctx.pattern'],
+                       reds=['start', 'ctx'],
+                       debugprint=(2, 0, 1))
+ at specializectx
+def literal_search(ctx, base):
+    # pattern starts with a literal character.  this is used
+    # for short prefixes, and if fast search is disabled
+    character = ctx.pat(base + 1)
+    base += 2
+    start = ctx.match_start
+    while start < ctx.end:
+        ctx.jitdriver_LiteralSearch.jit_merge_point(ctx=ctx, start=start,
+                                          base=base, character=character)
+        if ctx.str(start) == character:
+            if sre_match(ctx, base, start + 1, None) is not None:
+                ctx.match_start = start
+                return True
+        start += 1
+    return False
+
+install_jitdriver_spec("CharsetSearch",
+                       greens=['base', 'ctx.pattern'],
+                       reds=['start', 'ctx'],
+                       debugprint=(1, 0))
+ at specializectx
+def charset_search(ctx, base):
+    # pattern starts with a character from a known set
+    start = ctx.match_start
+    while start < ctx.end:
+        ctx.jitdriver_CharsetSearch.jit_merge_point(ctx=ctx, start=start,
+                                                    base=base)
+        if rsre_char.check_charset(ctx.pattern, 5, ctx.str(start)):
+            if sre_match(ctx, base, start, None) is not None:
+                ctx.match_start = start
+                return True
+        start += 1
+    return False
+
+install_jitdriver_spec('FastSearch',
+                       greens=['i', 'prefix_len', 'ctx.pattern'],
+                       reds=['string_position', 'ctx'],
+                       debugprint=(2, 0))
 @specializectx
 def fast_search(ctx):
     # skips forward in a string as fast as possible using information from
     # an optimization info block
     #  <1=skip> <2=flags> <3=min> <4=...>
     #        <5=length> <6=skip> <7=prefix data> 
-    flags = ctx.pat(2)
+    string_position = ctx.match_start
+    if string_position >= ctx.end:
+        return False
     prefix_len = ctx.pat(5)
     assert prefix_len >= 0
-    prefix_skip = ctx.pat(6)
-    assert prefix_skip >= 0
-    overlap_offset = 7 + prefix_len - 1
-    assert overlap_offset >= 0
-    pattern_offset = ctx.pat(1) + 1
-    ppos_start = pattern_offset + 2 * prefix_skip
-    assert ppos_start >= 0
     i = 0
-    string_position = ctx.match_start
-    end = ctx.end
-    while string_position < end:
-        while True:
-            char_ord = ctx.str(string_position)
-            if char_ord != ctx.pat(7 + i):
-                if i == 0:
-                    break
-                else:
-                    i = ctx.pat(overlap_offset + i)
-            else:
-                i += 1
-                if i == prefix_len:
-                    # found a potential match
-                    start = string_position + 1 - prefix_len
-                    assert start >= 0
-                    ptr = start + prefix_skip
-                    if flags & rsre_char.SRE_INFO_LITERAL:
-                        # matched all of pure literal pattern
-                        ctx.match_start = start
-                        ctx.match_end = ptr
-                        ctx.match_marks = None
-                        return True
-                    if sre_match(ctx, ppos_start, ptr, None) is not None:
-                        ctx.match_start = start
-                        return True
-                    i = ctx.pat(overlap_offset + i)
-                break
+    while True:
+        ctx.jitdriver_FastSearch.jit_merge_point(ctx=ctx,
+                string_position=string_position, i=i, prefix_len=prefix_len)
+        char_ord = ctx.str(string_position)
+        if char_ord != ctx.pat(7 + i):
+            if i > 0:
+                overlap_offset = prefix_len + (7 - 1)
+                i = ctx.pat(overlap_offset + i)
+                continue
+        else:
+            i += 1
+            if i == prefix_len:
+                # found a potential match
+                start = string_position + 1 - prefix_len
+                assert start >= 0
+                prefix_skip = ctx.pat(6)
+                ptr = start + prefix_skip
+                #flags = ctx.pat(2)
+                #if flags & rsre_char.SRE_INFO_LITERAL:
+                #    # matched all of pure literal pattern
+                #    ctx.match_start = start
+                #    ctx.match_end = ptr
+                #    ctx.match_marks = None
+                #    return True
+                pattern_offset = ctx.pat(1) + 1
+                ppos_start = pattern_offset + 2 * prefix_skip
+                if sre_match(ctx, ppos_start, ptr, None) is not None:
+                    ctx.match_start = start
+                    return True
+                overlap_offset = prefix_len + (7 - 1)
+                i = ctx.pat(overlap_offset + i)
         string_position += 1
-    return False
+        if string_position >= ctx.end:
+            return False

Added: pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_jit.py
==============================================================================
--- (empty file)
+++ pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_jit.py	Mon Oct 18 15:49:48 2010
@@ -0,0 +1,40 @@
+from pypy.rlib.jit import JitDriver
+
+
+class RSreJitDriver(JitDriver):
+
+    def __init__(self, name, debugprint, **kwds):
+        JitDriver.__init__(self, **kwds)
+        #
+        def get_printable_location(*args):
+            # we print based on indices in 'args'.  We first print
+            # 'ctx.pattern' from the arg number debugprint[0].
+            pattern = args[debugprint[0]]
+            s = str(pattern)
+            if len(s) > 120:
+                s = s[:110] + '...'
+            if len(debugprint) > 1:
+                # then we print numbers from the args number
+                # debugprint[1] and possibly debugprint[2]
+                info = ' at %d' % (args[debugprint[1]],)
+                if len(debugprint) > 2:
+                    info = '%s/%d' % (info, args[debugprint[2]])
+            else:
+                info = ''
+            return '%s%s %s' % (name, info, s)
+        #
+        self.get_printable_location = get_printable_location
+
+
+def install_jitdriver(name, **kwds):
+    from pypy.rlib.rsre.rsre_core import AbstractMatchContext
+    jitdriver = RSreJitDriver(name, **kwds)
+    setattr(AbstractMatchContext, 'jitdriver_' + name, jitdriver)
+
+def install_jitdriver_spec(name, **kwds):
+    from pypy.rlib.rsre.rsre_core import StrMatchContext
+    from pypy.rlib.rsre.rsre_core import UnicodeMatchContext
+    for prefix, concreteclass in [('Str', StrMatchContext),
+                                  ('Uni', UnicodeMatchContext)]:
+        jitdriver = RSreJitDriver(prefix + name, **kwds)
+        setattr(concreteclass, 'jitdriver_' + name, jitdriver)

Modified: pypy/branch/rsre-jit/pypy/rlib/rsre/test/test_zjit.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/rlib/rsre/test/test_zjit.py	(original)
+++ pypy/branch/rsre-jit/pypy/rlib/rsre/test/test_zjit.py	Mon Oct 18 15:49:48 2010
@@ -6,16 +6,30 @@
 from pypy.rpython.lltypesystem import lltype
 from pypy.rpython.annlowlevel import llstr, hlstr
 
-def entrypoint1(r, string):
+def entrypoint1(r, string, repeat):
     r = array2list(r)
     string = hlstr(string)
     make_sure_not_modified(r)
-    match = rsre_core.match(r, string)
+    match = None
+    for i in range(repeat):
+        match = rsre_core.match(r, string)
     if match is None:
         return -1
     else:
         return match.match_end
 
+def entrypoint2(r, string, repeat):
+    r = array2list(r)
+    string = hlstr(string)
+    make_sure_not_modified(r)
+    match = None
+    for i in range(repeat):
+        match = rsre_core.search(r, string)
+    if match is None:
+        return -1
+    else:
+        return match.match_start
+
 def list2array(lst):
     a = lltype.malloc(lltype.GcArray(lltype.Signed), len(lst))
     for i, x in enumerate(lst):
@@ -35,9 +49,16 @@
 
 class TestJitRSre(test_basic.LLJitMixin):
 
-    def meta_interp_match(self, pattern, string):
+    def meta_interp_match(self, pattern, string, repeat=1):
         r = get_code(pattern)
-        return self.meta_interp(entrypoint1, [list2array(r), llstr(string)],
+        return self.meta_interp(entrypoint1, [list2array(r), llstr(string),
+                                              repeat],
+                                listcomp=True, backendopt=True)
+
+    def meta_interp_search(self, pattern, string, repeat=1):
+        r = get_code(pattern)
+        return self.meta_interp(entrypoint2, [list2array(r), llstr(string),
+                                              repeat],
                                 listcomp=True, backendopt=True)
 
     def test_simple_match_1(self):
@@ -48,6 +69,11 @@
         res = self.meta_interp_match(r".*abc", "xxabcyyyyyyyyyyyyy")
         assert res == 5
 
+    def test_simple_match_repeated(self):
+        res = self.meta_interp_match(r"abcdef", "abcdef", repeat=10)
+        assert res == 6
+        self.check_tree_loop_count(1)
+
     def test_match_minrepeat_1(self):
         res = self.meta_interp_match(r".*?abc", "xxxxxxxxxxxxxxabc")
         assert res == 17
@@ -67,3 +93,28 @@
              "xxxxxxxxxxabbbbbbbbbbc")
         res = self.meta_interp_match(r".*?ab+?c", s)
         assert res == len(s)
+
+
+    def test_fast_search(self):
+        res = self.meta_interp_search(r"", "eua")
+        assert res == 15
+        self.check_loops(guard_value=0)
+
+    def test_regular_search(self):
+        res = self.meta_interp_search(r"<\w+>", "eiofweoxdiwhdohua")
+        assert res == 15
+
+    def test_regular_search_upcase(self):
+        res = self.meta_interp_search(r"<\w+>", "EIOFWEOXDIWHDOHUA")
+        assert res == 15
+
+    def test_max_until_1(self):
+        res = self.meta_interp_match(r"(ab)*abababababc",
+                                     "ababababababababababc")
+        assert res == 21
+
+    def test_example_1(self):
+        res = self.meta_interp_search(
+            r"Active\s+20\d\d-\d\d-\d\d\s+[[]\d+[]]([^[]+)",
+            "Active"*20 + "Active 2010-04-07 [42] Foobar baz boz blah[43]")
+        assert res == 6*20


From arigo at codespeak.net  Mon Oct 18 16:05:51 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 16:05:51 +0200 (CEST)
Subject: [pypy-svn] r78044 - pypy/branch/fast-forward/lib_pypy
Message-ID: <20101018140551.CB81A282B90@codespeak.net>

Author: arigo
Date: Mon Oct 18 16:05:49 2010
New Revision: 78044

Added:
   pypy/branch/fast-forward/lib_pypy/future_builtins.py
Log:
Implement the module future_builtins at app-level.


Added: pypy/branch/fast-forward/lib_pypy/future_builtins.py
==============================================================================
--- (empty file)
+++ pypy/branch/fast-forward/lib_pypy/future_builtins.py	Mon Oct 18 16:05:49 2010
@@ -0,0 +1,33 @@
+"""This module provides functions that will be builtins in Python 3.0,
+but that conflict with builtins that already exist in Python 2.x.
+
+Functions:
+
+hex(arg) -- Returns the hexadecimal representation of an integer
+oct(arg) -- Returns the octal representation of an integer
+ascii(arg) -- Same as repr(arg)
+map, filter, zip -- Same as itertools.imap, ifilter, izip
+
+The typical usage of this module is to replace existing builtins in a
+module's namespace:
+
+from future_builtins import hex, oct
+"""
+
+__all__ = ['hex', 'oct', 'ascii', 'map', 'filter', 'zip']
+
+from itertools import imap as map, ifilter as filter, izip as zip
+
+ascii = repr
+_builtin_hex = hex
+_builtin_oct = oct
+
+def hex(arg):
+    return _builtin_hex(arg).rstrip('L')
+
+def oct(arg):
+    result = _builtin_oct(arg).rstrip('L')
+    if result == '0':
+        return '0o0'
+    i = result.index('0') + 1
+    return result[:i] + 'o' + result[i:]


From arigo at codespeak.net  Mon Oct 18 16:20:35 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 16:20:35 +0200 (CEST)
Subject: [pypy-svn] r78045 -
	pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests
Message-ID: <20101018142035.DD8B736E3D6@codespeak.net>

Author: arigo
Date: Mon Oct 18 16:20:33 2010
New Revision: 78045

Modified:
   pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c
   pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py
Log:
A test for a property of ctypes: default restype.


Modified: pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c
==============================================================================
--- pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c	(original)
+++ pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c	Mon Oct 18 16:20:33 2010
@@ -533,3 +533,7 @@
 	return inp;
 }
 
+int my_unused_function(void)
+{
+    return 42;
+}

Modified: pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py
==============================================================================
--- pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py	(original)
+++ pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py	Mon Oct 18 16:20:33 2010
@@ -148,3 +148,7 @@
         # but it segfaults for some reason.
         if sys.platform == 'win32':
             assert f() == 0x12345678
+
+    def test_restype(self):
+        foo = lib.my_unused_function
+        assert foo.restype is c_int     # by default


From cfbolz at codespeak.net  Mon Oct 18 16:37:53 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Mon, 18 Oct 2010 16:37:53 +0200 (CEST)
Subject: [pypy-svn] r78046 - pypy/extradoc/talk/pepm2011
Message-ID: <20101018143753.A8351282B90@codespeak.net>

Author: cfbolz
Date: Mon Oct 18 16:37:52 2010
New Revision: 78046

Modified:
   pypy/extradoc/talk/pepm2011/math.lyx
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
comments by stephan in section 5


Modified: pypy/extradoc/talk/pepm2011/math.lyx
==============================================================================
--- pypy/extradoc/talk/pepm2011/math.lyx	(original)
+++ pypy/extradoc/talk/pepm2011/math.lyx	Mon Oct 18 16:37:52 2010
@@ -408,7 +408,7 @@
 \begin_inset Text
 
 \begin_layout Plain Layout
-\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\, u^{*}\,\mathrm{fresh}}{u=\mathtt{get}(v,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle u^{*}=\mathtt{get}(E(v),F)\right\rangle ,E\left[u\mapsto u^{*}\right],S}}$
+\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S),\,\,\, u^{*}\,\mathrm{fresh}}{u=\mathtt{get}(v,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle u^{*}=\mathtt{get}(E(v),F)\right\rangle ,E\left[u\mapsto u^{*}\right],S}}$
 \end_inset
 
 
@@ -504,7 +504,7 @@
 \begin_inset Text
 
 \begin_layout Plain Layout
-\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathtt{guard\_class}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$
+\begin_inset Formula ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathtt{guard\_class}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$
 \end_inset
 
 

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Mon Oct 18 16:37:52 2010
@@ -685,6 +685,7 @@
 \emph{get} & ${\displaystyle \frac{\,}{u=\mathtt{get}(v,F),E,H\overset{\mathrm{run}}{\Longrightarrow}E\left[u\mapsto H\left(E\left(v\right)\right)_{F}\right],H}}$ & ~~~ &  & ${\displaystyle \frac{\mathrm{type}(H(E(v))\neq T}{\mathtt{guard\_class}(v,T),E,H\overset{\mathrm{run}}{\Longrightarrow}\bot,\bot}}$\tabularnewline[3em]
 \emph{set} & ${\displaystyle \frac{\,}{\mathtt{set}\left(v,F,u\right),E,H\overset{\mathrm{run}}{\Longrightarrow}E,H\left[E\left(v\right)\mapsto\left(H\left(E\left(v\right)\right)!_{F}E(u)\right)\right]}}$ & ~~~ &  & \tabularnewline[4em]
 \end{tabular}
+\end{center}
 
 \begin{minipage}[b]{7 cm}
 \emph{Object Domains:}
@@ -708,25 +709,25 @@
  \end{array}
 $$
 \end{minipage}
-\end{center}
 \caption{The Operational Semantics of Simplified Traces}
 \label{fig:semantics}
 \end{figure*}
 
 In this section we want to give a formal description of the semantics of the
 traces and of the optimizer and liken the optimization to partial evaluation.
-We concentrate on the operations for manipulating dynamically allocated objects,
+We focus on the operations for manipulating dynamically allocated objects,
 as those are the only ones that are actually optimized. Without loss of
 generality we also consider only objects with two fields in this section.
 
-Traces are lists of operations. The operations considered here are \lstinline{new} (to make
-a new object), \lstinline{get} (to read a field out of an object), \lstinline{set} (to write a field
-into an object) and \lstinline{guard_class} (to check the type of an object). The values of all
-variables are locations (i.e.~pointers). Locations are mapped to objects, which
+Traces are lists of operations. The operations considered here are
+\lstinline{new}, \lstinline{get}, \lstinline{set} and \lstinline{guard_class}.
+The values of all
+variables are locations (\ie pointers). Locations are mapped to objects, which
 are represented by triples of a type $T$, and two locations that represent the
 fields of the object. When a new object is created, the fields are initialized
 to null, but we require that they are initialized to a real
-location before being read, otherwise the trace is malformed.
+location before being read, otherwise the trace is malformed (this condition is
+guaranteed by how the traces are generated in PyPy).
 
 We use some abbreviations when dealing with object triples. To read the type of
 an object, $\mathrm{type}((T,l_1,l_2))=T$ is used. Reading a field $F$ from an
@@ -737,13 +738,12 @@
 
 Figure~\ref{fig:semantics} shows the operational semantics for traces. The
 interpreter formalized there executes one operation at a time. Its state is
-represented by an environment $E$ and a heap $H$, which are potentially changed by the
+represented by an environment $E$ and a heap $H$, which may be changed by the
 execution of an operation. The environment is a partial function from variables
 to locations and the heap is a partial function from locations to objects. Note
-that a variable can never be null in the environment, otherwise the trace would
-be malformed. The environment could not directly map variables to object,
-because several variables can contain a pointer to the \emph{same} object. 
-The "indirection" is needed to express sharing.
+that a variable can never be null in the environment, otherwise the trace would have
+been malformed. The environment could not directly map variables to objects,
+because several variables can point to the \emph{same} object, because of aliasing. 
 
 We use the following notation for updating partial functions:
 $E[v\mapsto l]$ denotes the environment which is just like $E$, but maps $v$ to
@@ -776,15 +776,16 @@
 \begin{tabular}{lc}
 \emph{new} & ${\displaystyle \frac{v^{*}\,\mathrm{fresh}}{v=\mathtt{new}(T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[v\mapsto v^{*}\right],S\left[v^{*}\mapsto\left(T,\mathrm{null,null}\right)\right]}}$\tabularnewline[3em]
 \emph{get} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{u=\mathtt{get}(v,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E\left[u\mapsto S(E(v))_{F}\right],S}}$\tabularnewline[3em]
- & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\, u^{*}\,\mathrm{fresh}}{u=\mathtt{get}(v,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle u^{*}=\mathtt{get}(E(v),F)\right\rangle ,E\left[u\mapsto u^{*}\right],S}}$\tabularnewline[3em]
+ & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S),\,\,\, u^{*}\,\mathrm{fresh}}{u=\mathtt{get}(v,F),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle u^{*}=\mathtt{get}(E(v),F)\right\rangle ,E\left[u\mapsto u^{*}\right],S}}$\tabularnewline[3em]
 \emph{set} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S)}{\mathtt{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S\left[E\left(v\right)\mapsto\left(S(E(v))!_{F}E(u)\right)\right]}}$\tabularnewline[3em]
  & ${\displaystyle \frac{E(v)\notin\mathrm{dom}\left(S\right),\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathtt{set}\left(v,F,u\right),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathtt{set}\left(E(v),F,E(u)\right)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em]
 \emph{guard} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$\tabularnewline[3em]
- & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \mathtt{guard\_class}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em]
+ & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathtt{guard\_class}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em]
 \emph{lifting} & ${\displaystyle \frac{v^{*}\notin\mathrm{dom}(S)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle \,\right\rangle ,S}}$\tabularnewline[3em]
  & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathtt{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em]
  & ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathtt{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathtt{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline[3em]
 \end{tabular}
+\end{center}
 
 \begin{minipage}[b]{7 cm}
 \emph{Object Domains:}
@@ -808,17 +809,16 @@
  \end{array}
 $$
 \end{minipage}
-\end{center}
 \caption{Optimization Rules}
 \label{fig:optimization}
 \end{figure*}
 
-To optimize the simple traces from the last section, we use online partial
-evaluation. The partial evaluator optimizes one operation of the trace at a
+To optimize the simple traces of the last section, we use online partial
+evaluation. The partial evaluator optimizes one operation of a trace at a
 time. Every operation in the unoptimized trace is replaced by a list of
 operations in the optimized trace. This list is empty if the operation
-can be optimized away (which hopefully happens often). The optimization rules
-can be seen in Figure~\ref{fig:optimization}.
+can be optimized away. The optimization rules can be seen in
+Figure~\ref{fig:optimization}.
 
 The state of the optimizer is stored in an environment $E$ and a \emph{static
 heap} $S$. The environment is a partial function from variables in the
@@ -827,13 +827,14 @@
 $\ ^*$ for clarity). The reason for introducing new variables in the optimized
 trace is that several variables that appear in the unoptimized trace can turn
 into the same variables in the optimized trace. The environment of the
-optimizer serves a function similar to that of the environment in the semantics: sharing.
+optimizer serves a function similar to that of the environment in the
+semantics: to express sharing.
 
 The static heap is a partial function from $V^*$ into the
 set of static objects, which are triples of a type and two elements of $V^*$.
 A variable $v^*$ is in the domain of the static heap $S$ as long as the
-optimizer can fully keep track of the object. The object $S(v^*)$ is what is
-statically known about the object stored in it, \ie its type and its fields. The
+optimizer does not need to become dynamic XXX. The object $S(v^*)$ describes
+what is statically known about the object, \ie its type and its fields. The
 fields of objects in the static heap are also elements of $V^*$ (or null, for
 short periods of time).
 
@@ -841,7 +842,7 @@
 assumes that the resulting object can stay static. The optimization for all
 further operations is split into two cases. One case is for when the
 involved variables are in the static heap, which means that the operation can be
-performed at optimization time and removed from the trace. These rules mirror
+performed at optimization time and can be removed from the trace. These rules mirror
 the execution semantics closely. The other case is for when not enough is known about
 the variables, and the operation has to be residualized.
 
@@ -850,8 +851,9 @@
 operation needs to be residualized.
 
 If the first argument $v$ to a \lstinline{set} operation is mapped to something in the
-static heap, then the \lstinline{set} can performed at optimization time and the static heap
-updated. Otherwise the \lstinline{set} operation needs to be residualized. This needs to be
+static heap, then the \lstinline{set} can be performed at optimization time
+(which updates the static heap). Otherwise the \lstinline{set} operation needs
+to be residualized. This needs to be
 done carefully, because the new value for the field, from the variable $u$,
 could itself be static, in which case it needs to be lifted first.
 
@@ -861,10 +863,10 @@
 in the static heap, the \lstinline{guard_class} is residualized. This also needs to
 lift the variable on which the \lstinline{guard_class} is performed.
 
-Lifting takes a variable that is potentially in the static heap and makes sure
-that it is turned into a dynamic variable. This means that operations are
-emitted that construct an object with the shape described in the
-static heap, and the variable is removed from the static heap.
+Lifting takes a variable and turns it into a dynamic variable. If the variable
+is already dynamic, nothing needs to be done. If it is in the static heap,
+operations are emitted that construct an object with the shape described
+there, and the variable is removed from the static heap.
 
 Lifting a static object needs to recursively lift its fields. Some care needs to
 be taken when lifting a static object, because the structures described by the
@@ -874,7 +876,7 @@
 
 As an example for lifting, consider the static heap $$\{v^* \mapsto (T_1, w^*,
 v^*), w^* \mapsto (T_2, u^*, u^*)\}$$ which contains two static objects. If $v^*$
-now needs to be lifted, the following residual operations are produced:
+needs to be lifted, the following residual operations are produced:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 $v^*$ = new($T_1$)
@@ -914,20 +916,20 @@
 the algorithm only takes a total time linear in the length of the trace.
 The algorithm itself is not particularly complex; our focus is
 rather that \emph{in the context of tracing JITs} it is possible to find a
-simple enough algorithm that still gives very good results.
+simple enough algorithm that performs well.
 
-Note in particular that objects in category 1 (\ie the ones that do
+Note in particular that objects in category 1 (\ie those that do
 not escape) are completely removed; moreover, objects in category 2
-(\ie escaping) are still partially dealt with: if such an object
-escapes later than its creation point, all the operations in between that
-involve the object are removed.
+(\ie escaping) are still partially optimized: all the operations in between the
+creation of the object and the point where it escapes that involve the object
+are removed.
 
 The optimization is particularly effective for chains of operations.
 For example, it is typical for an interpreter to generate sequences of
 writes-followed-by-reads, where one interpreted opcode writes to some
 object's field and the next interpreted opcode reads it back, possibly
 dispatching on the type of the object created just before.  A typical example
-would be chains of arithmetic operations.
+would be a chain of arithmetic operations.
 
 % subsection Analysis of the Algorithm (end)
 
@@ -944,7 +946,7 @@
 the optimizer of PyPy's tracing JIT. The optimization is independent of which
 interpreter a JIT is generated for. There are some practical issues beyond the
 techniques described in this paper. The actual implementation needs to deal with
-more operations than described in Section~\ref{sec:formal}, for example to
+more operations than described in Section~\ref{sec:formal}, \eg to
 also support static arrays in addition to static objects. The implementation of
 this optimization is about 400 lines of RPython code.
 
@@ -956,7 +958,7 @@
 special way to solve this problem. This is a common approach in VM
 implementations \cite{miranda_context_1999,andreas_gal_trace-based_2009}; the
 novelty of our approach is that we generalized it enough to be usable for
-different interpreter.
+different interpreters.
 
 To evaluate our allocation removal algorithm, we look at the effectiveness when
 used in the generated tracing JIT of PyPy's Python interpreter. This interpreter
@@ -1016,15 +1018,17 @@
 As the first step, we counted the occurring operations in all generated traces
 before and after the optimization phase for all benchmarks. The resulting
 numbers can be
-seen in Figure~\ref{fig:numops}. The optimization removes as many as 90\% and as
-little as 4\% percent of allocation operations in the traces of the benchmarks.
+seen in Figure~\ref{fig:numops}. The optimization removes between 4\% and 90\%
+and of allocation operations in the traces of the benchmarks.
 All benchmarks taken together, the optimization removes 70\% percent of
 allocation operations. The numbers look similar for reading and writing of
 attributes. There are even more \lstinline{guard} operations that are removed,
 however there is an additional optimization that removes guards, so not all the
-removed guards are an effect of the optimization described here.
+removed guards are an effect of the optimization described here (for technical
+reasons, it would be very hard to separate the two effects).
 
 \begin{figure*}
+{\small
 \begin{center}
 \begin{tabular}{|l||r|rr|rr|rr|rr|}
 \hline
@@ -1049,6 +1053,7 @@
 \hline
 \end{tabular}
 \end{center}
+}
 \caption{Number of Operations and Percentage Removed By Optimization}
 \label{fig:numops}
 \end{figure*}
@@ -1072,6 +1077,7 @@
 further optimizations.
 
 \begin{figure*}
+{\small
 \begin{center}
 \begin{tabular}{|l||r|r||r|r||r|r||r|r|}
 \hline
@@ -1094,6 +1100,7 @@
 \hline
 \end{tabular}
 \end{center}
+}
 \caption{Benchmark Times in Milliseconds, Together With Factor Over PyPy With Optimizations}
 \label{fig:times}
 \end{figure*}
@@ -1160,12 +1167,12 @@
 %  separation logic
   %; John Hughes: type specialization
 
-\section{Conclusion}
+\section{Conclusion and Future Work}
 \label{sec:conclusion}
 
 In this paper, we used an approach based on online partial evaluation
 to optimize away allocations and type guards in the traces of a
-tracing JIT.  In this context a simple approach to partial evaluation
+tracing JIT.  In this context a simple approach based on partial evaluation
 gives good results.  This is due to the fact that the tracing JIT
 itself is responsible for all control issues, which are usually the
 hardest part of partial evaluation: the tracing JIT selects the parts
@@ -1185,7 +1192,8 @@
 \section*{Acknowledgements}
 
 The authors would like to thank Stefan Hallerstede, David Schneider and Thomas
-Stiehl for fruitful discussions during the writing of the paper.
+Stiehl for fruitful discussions and detailed feedback during the writing of the
+paper.
 
 \bibliographystyle{abbrv}
 \bibliography{paper}


From antocuni at codespeak.net  Mon Oct 18 16:45:19 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Mon, 18 Oct 2010 16:45:19 +0200 (CEST)
Subject: [pypy-svn] r78047 - in pypy/trunk/pypy: . annotation
	annotation/test config doc/config jit/backend/llgraph
	jit/backend/llsupport jit/backend/llsupport/test
	jit/backend/test jit/backend/x86/test jit/codewriter
	jit/codewriter/test jit/metainterp jit/metainterp/optimizeopt
	jit/metainterp/test jit/tl module/_ffi module/_rawffi
	module/_rawffi/test module/pypyjit module/pypyjit/test rlib
	rlib/test rpython rpython/lltypesystem rpython/test
	translator/c/test
Message-ID: <20101018144519.D2048282B90@codespeak.net>

Author: antocuni
Date: Mon Oct 18 16:45:16 2010
New Revision: 78047

Added:
   pypy/trunk/pypy/doc/config/objspace.usemodules._ffi.txt
      - copied unchanged from r78044, pypy/branch/jitffi/pypy/doc/config/objspace.usemodules._ffi.txt
   pypy/trunk/pypy/jit/backend/llsupport/ffisupport.py
      - copied unchanged from r78044, pypy/branch/jitffi/pypy/jit/backend/llsupport/ffisupport.py
   pypy/trunk/pypy/jit/backend/llsupport/test/test_ffisupport.py
      - copied unchanged from r78044, pypy/branch/jitffi/pypy/jit/backend/llsupport/test/test_ffisupport.py
   pypy/trunk/pypy/jit/metainterp/optimizeopt/fficall.py
      - copied unchanged from r78044, pypy/branch/jitffi/pypy/jit/metainterp/optimizeopt/fficall.py
   pypy/trunk/pypy/jit/metainterp/test/test_fficall.py
      - copied unchanged from r78044, pypy/branch/jitffi/pypy/jit/metainterp/test/test_fficall.py
   pypy/trunk/pypy/jit/metainterp/test/test_optimizefficall.py
      - copied unchanged from r78044, pypy/branch/jitffi/pypy/jit/metainterp/test/test_optimizefficall.py
   pypy/trunk/pypy/module/_ffi/
      - copied from r78044, pypy/branch/jitffi/pypy/module/_ffi/
   pypy/trunk/pypy/rlib/clibffi.py
      - copied unchanged from r78044, pypy/branch/jitffi/pypy/rlib/clibffi.py
   pypy/trunk/pypy/rlib/libffi.py
      - copied unchanged from r78044, pypy/branch/jitffi/pypy/rlib/libffi.py
   pypy/trunk/pypy/rlib/test/test_clibffi.py
      - copied unchanged from r78044, pypy/branch/jitffi/pypy/rlib/test/test_clibffi.py
   pypy/trunk/pypy/rlib/test/test_libffi.py
      - copied unchanged from r78044, pypy/branch/jitffi/pypy/rlib/test/test_libffi.py
Modified:
   pypy/trunk/pypy/   (props changed)
   pypy/trunk/pypy/annotation/model.py
   pypy/trunk/pypy/annotation/test/test_model.py
   pypy/trunk/pypy/config/pypyoption.py
   pypy/trunk/pypy/jit/backend/llgraph/llimpl.py
   pypy/trunk/pypy/jit/backend/llgraph/runner.py
   pypy/trunk/pypy/jit/backend/llsupport/descr.py
   pypy/trunk/pypy/jit/backend/llsupport/llmodel.py
   pypy/trunk/pypy/jit/backend/test/runner_test.py
   pypy/trunk/pypy/jit/backend/x86/test/test_zrpy_gc.py
   pypy/trunk/pypy/jit/backend/x86/test/test_ztranslation.py
   pypy/trunk/pypy/jit/codewriter/effectinfo.py
   pypy/trunk/pypy/jit/codewriter/jtransform.py
   pypy/trunk/pypy/jit/codewriter/support.py
   pypy/trunk/pypy/jit/codewriter/test/test_effectinfo.py
   pypy/trunk/pypy/jit/metainterp/executor.py
   pypy/trunk/pypy/jit/metainterp/optimize_nopspec.py   (props changed)
   pypy/trunk/pypy/jit/metainterp/optimizeopt/__init__.py
   pypy/trunk/pypy/jit/metainterp/optimizeopt/virtualize.py
   pypy/trunk/pypy/jit/metainterp/resoperation.py
   pypy/trunk/pypy/jit/metainterp/test/test_loop_nopspec.py   (props changed)
   pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py
   pypy/trunk/pypy/jit/metainterp/warmstate.py
   pypy/trunk/pypy/jit/tl/jittest.py   (props changed)
   pypy/trunk/pypy/jit/tl/pypyjit.py
   pypy/trunk/pypy/jit/tl/pypyjit_child.py
   pypy/trunk/pypy/module/_rawffi/__init__.py
   pypy/trunk/pypy/module/_rawffi/callback.py
   pypy/trunk/pypy/module/_rawffi/interp_rawffi.py
   pypy/trunk/pypy/module/_rawffi/structure.py
   pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py
   pypy/trunk/pypy/module/pypyjit/policy.py
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
   pypy/trunk/pypy/rlib/jit.py
   pypy/trunk/pypy/rlib/test/test_jit.py
   pypy/trunk/pypy/rlib/test/test_rdynload.py
   pypy/trunk/pypy/rpython/lltypesystem/lltype.py
   pypy/trunk/pypy/rpython/rpbc.py
   pypy/trunk/pypy/rpython/test/test_rclass.py
   pypy/trunk/pypy/translator/c/test/test_newgc.py
Log:
merge the jitffi branch: it provides a new jit-friendly rlib.libffi module,
and an app-level _ffi module which exposes it.



Modified: pypy/trunk/pypy/annotation/model.py
==============================================================================
--- pypy/trunk/pypy/annotation/model.py	(original)
+++ pypy/trunk/pypy/annotation/model.py	Mon Oct 18 16:45:16 2010
@@ -574,11 +574,11 @@
         
 NUMBER = object()
 annotation_to_ll_map = [
+    (SomeSingleFloat(), lltype.SingleFloat),
     (s_None, lltype.Void),   # also matches SomeImpossibleValue()
     (s_Bool, lltype.Bool),
     (SomeInteger(knowntype=r_ulonglong), NUMBER),    
     (SomeFloat(), lltype.Float),
-    (SomeSingleFloat(), lltype.SingleFloat),
     (SomeChar(), lltype.Char),
     (SomeUnicodeCodePoint(), lltype.UniChar),
     (SomeAddress(), llmemory.Address),

Modified: pypy/trunk/pypy/annotation/test/test_model.py
==============================================================================
--- pypy/trunk/pypy/annotation/test/test_model.py	(original)
+++ pypy/trunk/pypy/annotation/test/test_model.py	Mon Oct 18 16:45:16 2010
@@ -128,7 +128,7 @@
     assert isinstance(s_p, SomeOOInstance) and s_p.ootype == C
 
 def test_annotation_to_lltype():
-    from pypy.rlib.rarithmetic import r_uint
+    from pypy.rlib.rarithmetic import r_uint, r_singlefloat
     s_i = SomeInteger()
     s_pos = SomeInteger(nonneg=True)
     s_1 = SomeInteger(nonneg=True); s_1.const = 1
@@ -151,6 +151,9 @@
     C = ootype.Instance('C', ROOT, {})
     ref = SomeOOInstance(C)
     assert annotation_to_lltype(ref) == C
+    s_singlefloat = SomeSingleFloat()
+    s_singlefloat.const = r_singlefloat(0.0)
+    assert annotation_to_lltype(s_singlefloat) == lltype.SingleFloat
     
 def test_ll_union():
     PS1 = lltype.Ptr(lltype.GcStruct('s'))

Modified: pypy/trunk/pypy/config/pypyoption.py
==============================================================================
--- pypy/trunk/pypy/config/pypyoption.py	(original)
+++ pypy/trunk/pypy/config/pypyoption.py	Mon Oct 18 16:45:16 2010
@@ -73,9 +73,9 @@
     }
 
 module_import_dependencies = {
-    # no _rawffi if importing pypy.rlib.libffi raises ImportError
+    # no _rawffi if importing pypy.rlib.clibffi raises ImportError
     # or CompilationError
-    "_rawffi"   : ["pypy.rlib.libffi"],
+    "_rawffi"   : ["pypy.rlib.clibffi"],
 
     "zlib"      : ["pypy.rlib.rzlib"],
     "bz2"       : ["pypy.module.bz2.interp_bz2"],

Modified: pypy/trunk/pypy/jit/backend/llgraph/llimpl.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llgraph/llimpl.py	(original)
+++ pypy/trunk/pypy/jit/backend/llgraph/llimpl.py	Mon Oct 18 16:45:16 2010
@@ -10,7 +10,7 @@
                                          BoxInt, BoxPtr, BoxObj, BoxFloat,
                                          REF, INT, FLOAT)
 from pypy.jit.codewriter import heaptracker
-from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr
+from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr, rffi
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython.module.support import LLSupport, OOSupport
 from pypy.rpython.llinterp import LLException
@@ -305,12 +305,12 @@
     loop = _from_opaque(loop)
     loop.operations.append(Operation(opnum))
 
-def compile_add_descr(loop, ofs, type):
+def compile_add_descr(loop, ofs, type, arg_types):
     from pypy.jit.backend.llgraph.runner import Descr
     loop = _from_opaque(loop)
     op = loop.operations[-1]
     assert isinstance(type, str) and len(type) == 1
-    op.descr = Descr(ofs, type)
+    op.descr = Descr(ofs, type, arg_types=arg_types)
 
 def compile_add_loop_token(loop, descr):
     if we_are_translated():
@@ -801,7 +801,7 @@
             else:
                 raise TypeError(x)
         try:
-            return _do_call_common(func, args_in_order)
+            return _do_call_common(func, args_in_order, calldescr)
         except LLException, lle:
             _last_exception = lle
             d = {'v': None,
@@ -1018,6 +1018,9 @@
     if isinstance(TYPE, lltype.Ptr):
         if isinstance(x, (int, long, llmemory.AddressAsInt)):
             x = llmemory.cast_int_to_adr(x)
+        if TYPE is rffi.VOIDP:
+            # assume that we want a "C-style" cast, without typechecking the value
+            return rffi.cast(TYPE, x)
         return llmemory.cast_adr_to_ptr(x, TYPE)
     elif TYPE == llmemory.Address:
         if isinstance(x, (int, long, llmemory.AddressAsInt)):
@@ -1411,10 +1414,26 @@
 def do_call_pushfloat(x):
     _call_args_f.append(x)
 
-def _do_call_common(f, args_in_order=None):
+kind2TYPE = {
+    'i': lltype.Signed,
+    'f': lltype.Float,
+    'v': lltype.Void,
+    }
+
+def _do_call_common(f, args_in_order=None, calldescr=None):
     ptr = llmemory.cast_int_to_adr(f).ptr
-    FUNC = lltype.typeOf(ptr).TO
-    ARGS = FUNC.ARGS
+    PTR = lltype.typeOf(ptr)
+    if PTR == rffi.VOIDP:
+        # it's a pointer to a C function, so we don't have a precise
+        # signature: create one from the descr
+        ARGS = map(kind2TYPE.get, calldescr.arg_types)
+        RESULT = kind2TYPE[calldescr.typeinfo]
+        FUNC = lltype.FuncType(ARGS, RESULT)
+        func_to_call = rffi.cast(lltype.Ptr(FUNC), ptr)
+    else:
+        FUNC = PTR.TO
+        ARGS = FUNC.ARGS
+        func_to_call = ptr._obj._callable
     args = cast_call_args(ARGS, _call_args_i, _call_args_r, _call_args_f,
                           args_in_order)
     del _call_args_i[:]
@@ -1426,7 +1445,7 @@
         result = llinterp.eval_graph(ptr._obj.graph, args)
         # ^^^ may raise, in which case we get an LLException
     else:
-        result = ptr._obj._callable(*args)
+        result = func_to_call(*args)
     return result
 
 def do_call_void(f):

Modified: pypy/trunk/pypy/jit/backend/llgraph/runner.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llgraph/runner.py	(original)
+++ pypy/trunk/pypy/jit/backend/llgraph/runner.py	Mon Oct 18 16:45:16 2010
@@ -154,7 +154,7 @@
             llimpl.compile_add(c, op.getopnum())
             descr = op.getdescr()
             if isinstance(descr, Descr):
-                llimpl.compile_add_descr(c, descr.ofs, descr.typeinfo)
+                llimpl.compile_add_descr(c, descr.ofs, descr.typeinfo, descr.arg_types)
             if isinstance(descr, history.LoopToken) and op.getopnum() != rop.JUMP:
                 llimpl.compile_add_loop_token(c, descr)
             if self.is_oo and isinstance(descr, (OODescr, MethDescr)):
@@ -297,6 +297,18 @@
         return self.getdescr(0, token[0], extrainfo=extrainfo,
                              arg_types=''.join(arg_types))
 
+    def calldescrof_dynamic(self, ffi_args, ffi_result, extrainfo=None):
+        from pypy.jit.backend.llsupport.ffisupport import get_ffi_type_kind
+        arg_types = []
+        for arg in ffi_args:
+            kind = get_ffi_type_kind(arg)
+            if kind != history.VOID:
+                arg_types.append(kind)
+        reskind = get_ffi_type_kind(ffi_result)
+        return self.getdescr(0, reskind, extrainfo=extrainfo,
+                             arg_types=''.join(arg_types))
+
+
     def grab_exc_value(self):
         return llimpl.grab_exc_value()
 

Modified: pypy/trunk/pypy/jit/backend/llsupport/descr.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llsupport/descr.py	(original)
+++ pypy/trunk/pypy/jit/backend/llsupport/descr.py	Mon Oct 18 16:45:16 2010
@@ -307,6 +307,21 @@
     _return_type = history.INT
     call_stub = staticmethod(lambda func, args_i, args_r, args_f: 0)
 
+class DynamicIntCallDescr(BaseIntCallDescr):
+    """
+    calldescr that works for every integer type, by explicitly passing it the
+    size of the result. Used only by get_call_descr_dynamic
+    """
+    _clsname = 'DynamicIntCallDescr'
+
+    def __init__(self, arg_classes, result_size, extrainfo=None):
+        BaseIntCallDescr.__init__(self, arg_classes, extrainfo)
+        self._result_size = result_size
+
+    def get_result_size(self, translate_support_code):
+        return self._result_size
+
+
 class NonGcPtrCallDescr(BaseIntCallDescr):
     _clsname = 'NonGcPtrCallDescr'
     def get_result_size(self, translate_support_code):

Modified: pypy/trunk/pypy/jit/backend/llsupport/llmodel.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llsupport/llmodel.py	(original)
+++ pypy/trunk/pypy/jit/backend/llsupport/llmodel.py	Mon Oct 18 16:45:16 2010
@@ -17,6 +17,7 @@
 from pypy.jit.backend.llsupport.descr import get_call_descr
 from pypy.jit.backend.llsupport.descr import BaseIntCallDescr, GcPtrCallDescr
 from pypy.jit.backend.llsupport.descr import FloatCallDescr, VoidCallDescr
+from pypy.jit.backend.llsupport.ffisupport import get_call_descr_dynamic
 from pypy.rpython.annlowlevel import cast_instance_to_base_ptr
 
 
@@ -231,6 +232,9 @@
     def calldescrof(self, FUNC, ARGS, RESULT, extrainfo=None):
         return get_call_descr(self.gc_ll_descr, ARGS, RESULT, extrainfo)
 
+    def calldescrof_dynamic(self, ffi_args, ffi_result, extrainfo=None):
+        return get_call_descr_dynamic(ffi_args, ffi_result, extrainfo)
+
     def get_overflow_error(self):
         ovf_vtable = self.cast_adr_to_int(self._ovf_error_vtable)
         ovf_inst = lltype.cast_opaque_ptr(llmemory.GCREF,

Modified: pypy/trunk/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/trunk/pypy/jit/backend/test/runner_test.py	Mon Oct 18 16:45:16 2010
@@ -421,6 +421,7 @@
             assert x == 3.5 - 42
 
     def test_call(self):
+        from pypy.rlib.libffi import types
 
         def func_int(a, b):
             return a + b
@@ -428,23 +429,31 @@
             return chr(ord(c) + ord(c1))
 
         functions = [
-            (func_int, lltype.Signed, 655360),
-            (func_int, rffi.SHORT, 1213),
-            (func_char, lltype.Char, 12)
+            (func_int, lltype.Signed, types.sint, 655360),
+            (func_int, rffi.SHORT, types.sint16, 1213),
+            (func_char, lltype.Char, types.uchar, 12)
             ]
 
-        for func, TP, num in functions:
+        for func, TP, ffi_type, num in functions:
             cpu = self.cpu
             #
             FPTR = self.Ptr(self.FuncType([TP, TP], TP))
             func_ptr = llhelper(FPTR, func)
             FUNC = deref(FPTR)
-            calldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
             funcbox = self.get_funcbox(cpu, func_ptr)
+            # first, try it with the "normal" calldescr
+            calldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
             res = self.execute_operation(rop.CALL,
                                          [funcbox, BoxInt(num), BoxInt(num)],
                                          'int', descr=calldescr)
             assert res.value == 2 * num
+            # then, try it with the dynamic calldescr
+            dyn_calldescr = cpu.calldescrof_dynamic([ffi_type, ffi_type], ffi_type)
+            res = self.execute_operation(rop.CALL,
+                                         [funcbox, BoxInt(num), BoxInt(num)],
+                                         'int', descr=dyn_calldescr)
+            assert res.value == 2 * num
+            
 
         if cpu.supports_floats:
             def func(f0, f1, f2, f3, f4, f5, f6, i0, i1, f7, f8, f9):
@@ -507,6 +516,23 @@
                                          'int', descr=calldescr)
             assert res.value == func_ints(*args)
 
+    def test_call_to_c_function(self):
+        from pypy.rlib.libffi import CDLL, types, ArgChain
+        libc = CDLL('libc.so.6')
+        c_tolower = libc.getpointer('tolower', [types.uchar], types.sint)
+        argchain = ArgChain().arg(ord('A'))
+        assert c_tolower.call(argchain, rffi.INT) == ord('a')
+
+        func_adr = llmemory.cast_ptr_to_adr(c_tolower.funcsym)
+        funcbox = ConstInt(heaptracker.adr2int(func_adr))
+        calldescr = self.cpu.calldescrof_dynamic([types.uchar], types.sint)
+        res = self.execute_operation(rop.CALL,
+                                     [funcbox, BoxInt(ord('A'))],
+                                     'int',
+                                     descr=calldescr)
+        assert res.value == ord('a')
+
+
     def test_field_basic(self):
         t_box, T_box = self.alloc_instance(self.T)
         fielddescr = self.cpu.fielddescrof(self.S, 'value')

Modified: pypy/trunk/pypy/jit/backend/x86/test/test_zrpy_gc.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/test/test_zrpy_gc.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/test/test_zrpy_gc.py	Mon Oct 18 16:45:16 2010
@@ -191,6 +191,33 @@
     def run_orig(self, name, n, x):
         self.main_allfuncs(name, n, x)
 
+    def define_libffi_workaround(cls):
+        # XXX: this is a workaround for a bug in database.py.  It seems that
+        # the problem is triggered by optimizeopt/fficall.py, and in
+        # particular by the ``cast_base_ptr_to_instance(Func, llfunc)``: in
+        # these tests, that line is the only place where libffi.Func is
+        # referenced.
+        #
+        # The problem occurs because the gctransformer tries to annotate a
+        # low-level helper to call the __del__ of libffi.Func when it's too
+        # late.
+        #
+        # This workaround works by forcing the annotator (and all the rest of
+        # the toolchain) to see libffi.Func in a "proper" context, not just as
+        # the target of cast_base_ptr_to_instance.  Note that the function
+        # below is *never* called by any actual test, it's just annotated.
+        #
+        from pypy.rlib.libffi import get_libc_name, CDLL, types, ArgChain
+        libc_name = get_libc_name()
+        def f(n, x, *args):
+            libc = CDLL(libc_name)
+            ptr = libc.getpointer('labs', [types.slong], types.slong)
+            chain = ArgChain()
+            chain.arg(n)
+            n = ptr.call(chain, lltype.Signed)
+            return (n, x) + args
+        return None, f, None
+
     def define_compile_framework_1(cls):
         # a moving GC.  Supports malloc_varsize_nonmovable.  Simple test, works
         # without write_barriers and root stack enumeration.

Modified: pypy/trunk/pypy/jit/backend/x86/test/test_ztranslation.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/test/test_ztranslation.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/test/test_ztranslation.py	Mon Oct 18 16:45:16 2010
@@ -1,4 +1,4 @@
-import py, os
+import py, os, sys
 from pypy.tool.udir import udir
 from pypy.rlib.jit import JitDriver, OPTIMIZER_FULL, unroll_parameters
 from pypy.rlib.jit import PARAMETERS, dont_look_inside
@@ -63,8 +63,32 @@
                 if k - abs(j):  raise ValueError
                 if k - abs(-j): raise ValueError
             return total * 10
+        #
+        from pypy.rpython.lltypesystem import lltype, rffi
+        from pypy.rlib.libffi import types, CDLL, ArgChain
+        from pypy.rlib.test.test_libffi import get_libm_name
+        libm_name = get_libm_name(sys.platform)
+        jitdriver2 = JitDriver(greens=[], reds = ['i', 'func', 'res', 'x'])
+        def libffi_stuff(i, j):
+            lib = CDLL(libm_name)
+            func = lib.getpointer('fabs', [types.double], types.double)
+            res = 0.0
+            x = float(j)
+            while i > 0:
+                jitdriver2.jit_merge_point(i=i, res=res, func=func, x=x)
+                jitdriver2.can_enter_jit(i=i, res=res, func=func, x=x)
+                func = hint(func, promote=True)
+                argchain = ArgChain()
+                argchain.arg(x)
+                res = func.call(argchain, rffi.DOUBLE)
+                i -= 1
+            return res
+        #
+        def main(i, j):
+            return f(i, j) + libffi_stuff(i, j)
+        expected = f(40, -49)
         res = self.meta_interp(f, [40, -49])
-        assert res == f(40, -49)
+        assert res == expected
 
     def test_direct_assembler_call_translates(self):
         class Thing(object):

Modified: pypy/trunk/pypy/jit/codewriter/effectinfo.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/effectinfo.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/effectinfo.py	Mon Oct 18 16:45:16 2010
@@ -19,7 +19,7 @@
     OS_NONE                     = 0    # normal case, no oopspec
     OS_ARRAYCOPY                = 1    # "list.ll_arraycopy"
     OS_STR2UNICODE              = 2    # "str.str2unicode"
-
+    #
     OS_STR_CONCAT               = 22   # "stroruni.concat"
     OS_STR_SLICE                = 23   # "stroruni.slice"
     OS_STR_EQUAL                = 24   # "stroruni.equal"
@@ -30,7 +30,7 @@
     OS_STREQ_NONNULL_CHAR       = 29   # s1 == char  (assert s1!=NULL)
     OS_STREQ_CHECKNULL_CHAR     = 30   # s1!=NULL and s1==char
     OS_STREQ_LENGTHOK           = 31   # s1 == s2    (assert len(s1)==len(s2))
-
+    #
     OS_UNI_CONCAT               = 42   #
     OS_UNI_SLICE                = 43   #
     OS_UNI_EQUAL                = 44   #
@@ -42,6 +42,10 @@
     OS_UNIEQ_CHECKNULL_CHAR     = 50   #   STR, in the same order)
     OS_UNIEQ_LENGTHOK           = 51   #
     _OS_offset_uni              = OS_UNI_CONCAT - OS_STR_CONCAT
+    #
+    OS_LIBFFI_PREPARE           = 60
+    OS_LIBFFI_PUSH_ARG          = 61
+    OS_LIBFFI_CALL              = 62
 
     def __new__(cls, readonly_descrs_fields,
                 write_descrs_fields, write_descrs_arrays,

Modified: pypy/trunk/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/jtransform.py	Mon Oct 18 16:45:16 2010
@@ -320,6 +320,8 @@
             prepare = self._handle_str2unicode_call
         elif oopspec_name.startswith('virtual_ref'):
             prepare = self._handle_virtual_ref_call
+        elif oopspec_name.startswith('libffi_'):
+            prepare = self._handle_libffi_call
         else:
             prepare = self.prepare_builtin_call
         try:
@@ -1030,8 +1032,10 @@
     # ----------
     # Strings and Unicodes.
 
-    def _handle_oopspec_call(self, op, args, oopspecindex):
+    def _handle_oopspec_call(self, op, args, oopspecindex, extraeffect=None):
         calldescr = self.callcontrol.getcalldescr(op, oopspecindex)
+        if extraeffect:
+            calldescr.get_extra_info().extraeffect = extraeffect
         if isinstance(op.args[0].value, str):
             pass  # for tests only
         else:
@@ -1132,6 +1136,23 @@
                                           vrefinfo.JIT_VIRTUAL_REF)
         return SpaceOperation(oopspec_name, list(args), op.result)
 
+    # -----------
+    # rlib.libffi
+
+    def _handle_libffi_call(self, op, oopspec_name, args):
+        if oopspec_name == 'libffi_prepare_call':
+            oopspecindex = EffectInfo.OS_LIBFFI_PREPARE
+            extraeffect = EffectInfo.EF_CANNOT_RAISE
+        elif oopspec_name.startswith('libffi_push_'):
+            oopspecindex = EffectInfo.OS_LIBFFI_PUSH_ARG
+            extraeffect = EffectInfo.EF_CANNOT_RAISE
+        elif oopspec_name.startswith('libffi_call_'):
+            oopspecindex = EffectInfo.OS_LIBFFI_CALL
+            extraeffect = EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE
+        else:
+            assert False, 'unsupported oopspec: %s' % oopspec_name
+        return self._handle_oopspec_call(op, args, oopspecindex, extraeffect)
+
     def rewrite_op_jit_force_virtual(self, op):
         return self._do_builtin_call(op)
 

Modified: pypy/trunk/pypy/jit/codewriter/support.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/support.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/support.py	Mon Oct 18 16:45:16 2010
@@ -1,5 +1,5 @@
 import sys
-from pypy.rpython.lltypesystem import lltype, rclass
+from pypy.rpython.lltypesystem import lltype, rclass, rffi
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython import rlist
 from pypy.rpython.lltypesystem import rstr as ll_rstr, rdict as ll_rdict
@@ -8,6 +8,7 @@
 from pypy.rpython.ootypesystem import rdict as oo_rdict
 from pypy.rpython.llinterp import LLInterpreter
 from pypy.rpython.extregistry import ExtRegistryEntry
+from pypy.rpython.annlowlevel import cast_base_ptr_to_instance
 from pypy.translator.simplify import get_funcobj
 from pypy.translator.unsimplify import split_block
 from pypy.objspace.flow.model import Constant
@@ -217,6 +218,33 @@
     else:
         return x
 
+
+# libffi support
+# --------------
+
+def func(llfunc):
+    from pypy.rlib.libffi import Func
+    return cast_base_ptr_to_instance(Func, llfunc)
+
+def _ll_1_libffi_prepare_call(llfunc):
+    return func(llfunc)._prepare()
+
+def _ll_4_libffi_push_int(llfunc, value, ll_args, i):
+    return func(llfunc)._push_int(value, ll_args, i)
+
+def _ll_4_libffi_push_float(llfunc, value, ll_args, i):
+    return func(llfunc)._push_float(value, ll_args, i)
+
+def _ll_3_libffi_call_int(llfunc, funcsym, ll_args):
+    return func(llfunc)._do_call(funcsym, ll_args, rffi.LONG)
+
+def _ll_3_libffi_call_float(llfunc, funcsym, ll_args):
+    return func(llfunc)._do_call(funcsym, ll_args, rffi.DOUBLE)
+
+def _ll_3_libffi_call_void(llfunc, funcsym, ll_args):
+    return func(llfunc)._do_call(funcsym, ll_args, lltype.Void)
+
+
 # in the following calls to builtins, the JIT is allowed to look inside:
 inline_calls_to = [
     ('int_floordiv_ovf_zer', [lltype.Signed, lltype.Signed], lltype.Signed),

Modified: pypy/trunk/pypy/jit/codewriter/test/test_effectinfo.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/test/test_effectinfo.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/test/test_effectinfo.py	Mon Oct 18 16:45:16 2010
@@ -1,7 +1,8 @@
 from pypy.rpython.lltypesystem.rclass import OBJECT
 from pypy.rpython.lltypesystem import lltype
 from pypy.rpython.ootypesystem import ootype
-from pypy.jit.codewriter.effectinfo import effectinfo_from_writeanalyze
+from pypy.jit.codewriter.effectinfo import effectinfo_from_writeanalyze,\
+    EffectInfo
 
 class FakeCPU:
     def fielddescrof(self, T, fieldname):
@@ -9,6 +10,14 @@
     def arraydescrof(self, A):
         return ('arraydescr', A)
 
+def test_no_oopspec_duplicate():
+    # check that all the various EffectInfo.OS_* have unique values
+    oopspecs = set()
+    for name, value in EffectInfo.__dict__.iteritems():
+        if name.startswith('OS_'):
+            assert value not in oopspecs
+            oopspecs.add(value)
+
 def test_include_read_field():
     S = lltype.GcStruct("S", ("a", lltype.Signed))
     effects = frozenset([("readstruct", lltype.Ptr(S), "a")])

Modified: pypy/trunk/pypy/jit/metainterp/executor.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/executor.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/executor.py	Mon Oct 18 16:45:16 2010
@@ -80,6 +80,9 @@
 do_call_loopinvariant = do_call
 do_call_may_force = do_call
 
+def do_call_c(cpu, metainterp, argboxes, descr):
+    raise NotImplementedError("Should never be called directly")
+
 def do_getarrayitem_gc(cpu, _, arraybox, indexbox, arraydescr):
     array = arraybox.getref_base()
     index = indexbox.getint()

Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/__init__.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/optimizeopt/__init__.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/optimizeopt/__init__.py	Mon Oct 18 16:45:16 2010
@@ -3,6 +3,7 @@
 from pypy.jit.metainterp.optimizeopt.intbounds import OptIntBounds
 from pypy.jit.metainterp.optimizeopt.virtualize import OptVirtualize
 from pypy.jit.metainterp.optimizeopt.heap import OptHeap
+from pypy.jit.metainterp.optimizeopt.fficall import OptFfiCall
 from pypy.jit.metainterp.optimizeopt.string import OptString
 
 def optimize_loop_1(metainterp_sd, loop, virtuals=True):
@@ -16,6 +17,7 @@
                      OptVirtualize(),
                      OptString(),
                      OptHeap(),
+                     OptFfiCall(),
                     ]
     optimizer = Optimizer(metainterp_sd, loop, optimizations, virtuals)
     optimizer.propagate_all_forward()

Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/optimizeopt/virtualize.py	Mon Oct 18 16:45:16 2010
@@ -134,6 +134,11 @@
         fielddescrs = self._get_field_descr_list()
         return modifier.make_virtual(self.known_class, fielddescrs)
 
+    def __repr__(self):
+        cls_name = self.known_class.value.adr.ptr._obj._TYPE._name
+        field_names = [field.name for field in self._fields]
+        return "" % (cls_name, field_names)
+
 class VStructValue(AbstractVirtualStructValue):
 
     def __init__(self, optimizer, structdescr, keybox, source_op=None):

Modified: pypy/trunk/pypy/jit/metainterp/resoperation.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/resoperation.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/resoperation.py	Mon Oct 18 16:45:16 2010
@@ -463,7 +463,7 @@
 
     '_CANRAISE_FIRST', # ----- start of can_raise operations -----
     'CALL/*d',
-    'CALL_ASSEMBLER/*d',
+    'CALL_ASSEMBLER/*d',  # call already compiled assembler
     'CALL_MAY_FORCE/*d',
     'CALL_LOOPINVARIANT/*d',
     #'OOSEND',                     # ootype operation

Modified: pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py	Mon Oct 18 16:45:16 2010
@@ -132,14 +132,21 @@
 # ____________________________________________________________
 
 def equaloplists(oplist1, oplist2, strict_fail_args=True, remap={}):
-    print '-'*20, 'Comparing lists', '-'*20
+    # try to use the full width of the terminal to display the list
+    # unfortunately, does not work with the default capture method of py.test
+    # (which is fd), you you need to use either -s or --capture=sys, else you
+    # get the standard 80 columns width
+    totwidth = py.io.get_terminal_width()
+    width = totwidth / 2 - 1
+    print ' Comparing lists '.center(totwidth, '-')
+    print '%s| %s' % ('optimized'.center(width), 'expected'.center(width))
     for op1, op2 in zip(oplist1, oplist2):
         txt1 = str(op1)
         txt2 = str(op2)
         while txt1 or txt2:
-            print '%-39s| %s' % (txt1[:39], txt2[:39])
-            txt1 = txt1[39:]
-            txt2 = txt2[39:]
+            print '%s| %s' % (txt1[:width].ljust(width), txt2[:width])
+            txt1 = txt1[width:]
+            txt2 = txt2[width:]
         assert op1.getopnum() == op2.getopnum()
         assert op1.numargs() == op2.numargs()
         for i in range(op1.numargs()):
@@ -262,6 +269,10 @@
         expected = self.parse(optops)
         print '\n'.join([str(o) for o in loop.operations])
         self.assert_equal(loop, expected)
+        return loop
+
+
+class OptimizeOptTest(BaseTestOptimizeOpt):
 
     def test_simple(self):
         ops = """
@@ -2643,7 +2654,7 @@
             ''', rop.GUARD_TRUE)
 
 
-class TestLLtype(BaseTestOptimizeOpt, LLtypeMixin):
+class TestLLtype(OptimizeOptTest, LLtypeMixin):
 
     def test_residual_call_does_not_invalidate_caches(self):
         ops = """
@@ -4533,7 +4544,7 @@
         # can be raised by ll_str2unicode()
 
 
-##class TestOOtype(BaseTestOptimizeOpt, OOtypeMixin):
+##class TestOOtype(OptimizeOptTest, OOtypeMixin):
 
 ##    def test_instanceof(self):
 ##        ops = """

Modified: pypy/trunk/pypy/jit/metainterp/warmstate.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/warmstate.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/warmstate.py	Mon Oct 18 16:45:16 2010
@@ -1,5 +1,5 @@
 import sys
-from pypy.rpython.lltypesystem import lltype, llmemory, rstr
+from pypy.rpython.lltypesystem import lltype, llmemory, rstr, rffi
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython.annlowlevel import hlstr, llstr, cast_base_ptr_to_instance
 from pypy.rpython.annlowlevel import cast_object_to_ptr
@@ -24,7 +24,11 @@
     """
     INPUT = lltype.typeOf(x)
     if INPUT is lltype.Signed:
-        return lltype.cast_primitive(TYPE, x)    # XXX missing: Ptr(non-gc)
+        if isinstance(TYPE, lltype.Ptr) and TYPE.TO._gckind == 'raw':
+            # non-gc pointer
+            return rffi.cast(TYPE, x)
+        else:
+            return lltype.cast_primitive(TYPE, x)
     elif INPUT is lltype.Float:
         assert TYPE is lltype.Float
         return x

Modified: pypy/trunk/pypy/jit/tl/pypyjit.py
==============================================================================
--- pypy/trunk/pypy/jit/tl/pypyjit.py	(original)
+++ pypy/trunk/pypy/jit/tl/pypyjit.py	Mon Oct 18 16:45:16 2010
@@ -40,6 +40,9 @@
 config.objspace.usemodules.array = True
 config.objspace.usemodules._weakref = False
 config.objspace.usemodules._sre = False
+#
+config.objspace.usemodules._ffi = True
+#
 set_pypy_opt_level(config, level='jit')
 config.objspace.std.withinlineddict = True
 

Modified: pypy/trunk/pypy/jit/tl/pypyjit_child.py
==============================================================================
--- pypy/trunk/pypy/jit/tl/pypyjit_child.py	(original)
+++ pypy/trunk/pypy/jit/tl/pypyjit_child.py	Mon Oct 18 16:45:16 2010
@@ -16,7 +16,7 @@
     interp.heap.malloc_nonmovable = returns_null     # XXX
 
     from pypy.jit.backend.llgraph.runner import LLtypeCPU
-    LLtypeCPU.supports_floats = False    # for now
+    #LLtypeCPU.supports_floats = False    # for now
     apply_jit(interp, graph, LLtypeCPU)
 
 

Modified: pypy/trunk/pypy/module/_rawffi/__init__.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/__init__.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/__init__.py	Mon Oct 18 16:45:16 2010
@@ -1,5 +1,5 @@
 
-""" Low-level interface to libffi
+""" Low-level interface to clibffi
 """
 
 from pypy.interpreter.mixedmodule import MixedModule
@@ -39,11 +39,11 @@
         if hasattr(interp_rawffi, 'check_HRESULT'):
             Module.interpleveldefs['check_HRESULT'] = 'interp_rawffi.check_HRESULT'
 
-        from pypy.rlib import libffi
+        from pypy.rlib import clibffi
         for name in ['FUNCFLAG_STDCALL', 'FUNCFLAG_CDECL', 'FUNCFLAG_PYTHONAPI',
                      ]:
-            if hasattr(libffi, name):
-                Module.interpleveldefs[name] = "space.wrap(%r)" % getattr(libffi, name)
+            if hasattr(clibffi, name):
+                Module.interpleveldefs[name] = "space.wrap(%r)" % getattr(clibffi, name)
                 
         super(Module, cls).buildloaders()
     buildloaders = classmethod(buildloaders)

Modified: pypy/trunk/pypy/module/_rawffi/callback.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/callback.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/callback.py	Mon Oct 18 16:45:16 2010
@@ -8,8 +8,8 @@
 from pypy.module._rawffi.array import get_elem, push_elem
 from pypy.module._rawffi.interp_rawffi import W_DataInstance, letter2tp, \
      wrap_value, unwrap_value, unwrap_truncate_int
-from pypy.rlib.libffi import USERDATA_P, CallbackFuncPtr, FUNCFLAG_CDECL
-from pypy.rlib.libffi import ffi_type_void
+from pypy.rlib.clibffi import USERDATA_P, CallbackFuncPtr, FUNCFLAG_CDECL
+from pypy.rlib.clibffi import ffi_type_void
 from pypy.module._rawffi.tracker import tracker
 from pypy.interpreter.error import OperationError
 from pypy.interpreter import gateway

Modified: pypy/trunk/pypy/module/_rawffi/interp_rawffi.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/interp_rawffi.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/interp_rawffi.py	Mon Oct 18 16:45:16 2010
@@ -5,7 +5,7 @@
 from pypy.interpreter.gateway import interp2app, NoneNotWrapped
 from pypy.interpreter.typedef import TypeDef, GetSetProperty
 
-from pypy.rlib.libffi import *
+from pypy.rlib.clibffi import *
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.rlib.unroll import unrolling_iterable
 

Modified: pypy/trunk/pypy/module/_rawffi/structure.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/structure.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/structure.py	Mon Oct 18 16:45:16 2010
@@ -15,7 +15,7 @@
 from pypy.module._rawffi.interp_rawffi import wrap_value, unwrap_value
 from pypy.module._rawffi.interp_rawffi import unpack_shape_with_length
 from pypy.module._rawffi.interp_rawffi import size_alignment
-from pypy.rlib import libffi
+from pypy.rlib import clibffi
 from pypy.rlib.rarithmetic import intmask, r_uint
 
 def unpack_fields(space, w_fields):
@@ -104,14 +104,14 @@
     descr_fieldoffset.unwrap_spec = ['self', ObjSpace, str]
 
     # get the corresponding ffi_type
-    ffi_struct = lltype.nullptr(libffi.FFI_STRUCT_P.TO)
+    ffi_struct = lltype.nullptr(clibffi.FFI_STRUCT_P.TO)
 
     def get_basic_ffi_type(self):
         if not self.ffi_struct:
             # Repeated fields are delicate.  Consider for example
             #     struct { int a[5]; }
             # or  struct { struct {int x;} a[5]; }
-            # Seeing no corresponding doc in libffi, let's just repeat
+            # Seeing no corresponding doc in clibffi, let's just repeat
             # the field 5 times...
             fieldtypes = []
             for name, tp in self.fields:
@@ -122,7 +122,7 @@
                 while count + basic_size <= total_size:
                     fieldtypes.append(basic_ffi_type)
                     count += basic_size
-            self.ffi_struct = libffi.make_struct_ffitype_e(self.size,
+            self.ffi_struct = clibffi.make_struct_ffitype_e(self.size,
                                                            self.alignment,
                                                            fieldtypes)
         return self.ffi_struct.ffistruct

Modified: pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py	Mon Oct 18 16:45:16 2010
@@ -191,7 +191,7 @@
     prepare_c_example = staticmethod(prepare_c_example)
     
     def setup_class(cls):
-        from pypy.rlib.libffi import get_libc_name
+        from pypy.rlib.clibffi import get_libc_name
         space = gettestobjspace(usemodules=('_rawffi', 'struct'))
         cls.space = space
         cls.w_lib_name = space.wrap(cls.prepare_c_example())

Modified: pypy/trunk/pypy/module/pypyjit/policy.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/policy.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/policy.py	Mon Oct 18 16:45:16 2010
@@ -11,7 +11,7 @@
         if '.' in modname:
             modname, _ = modname.split('.', 1)
         if modname in ['pypyjit', 'signal', 'micronumpy', 'math', 'exceptions',
-                       'imp', 'sys', 'array']:
+                       'imp', 'sys', 'array', '_ffi']:
             return True
         return False
 

Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Mon Oct 18 16:45:16 2010
@@ -79,8 +79,11 @@
 
 
 class PyPyCJITTests(object):
-    def run_source(self, source, expected_max_ops, *testcases):
+    def run_source(self, source, expected_max_ops, *testcases, **kwds):
         assert isinstance(expected_max_ops, int)
+        threshold = kwds.pop('threshold', 3)
+        if kwds:
+            raise TypeError, 'Unsupported keyword arguments: %s' % kwds.keys()
         source = py.code.Source(source)
         filepath = self.tmpdir.join('case%d.py' % self.counter)
         logfilepath = filepath.new(ext='.log')
@@ -92,7 +95,7 @@
             import sys
             try: # make the file runnable by CPython
                 import pypyjit
-                pypyjit.set_param(threshold=3)
+                pypyjit.set_param(threshold=%d)
             except ImportError:
                 pass
 
@@ -102,7 +105,7 @@
                 print >> sys.stderr, 'got:', repr(result)
                 assert result == expected
                 assert type(result) is type(expected)
-        """)
+        """ % threshold)
         for testcase in testcases * 2:
             print >> f, "check(%r, %r)" % testcase
         print >> f, "print 'OK :-)'"
@@ -123,6 +126,7 @@
         if self.total_ops > expected_max_ops:
             assert 0, "too many operations: got %d, expected maximum %d" % (
                 self.total_ops, expected_max_ops)
+        return result
 
     def parse_loops(self, opslogfile):
         from pypy.jit.metainterp.test.oparser import parse
@@ -1134,6 +1138,39 @@
             return sa
         ''', 88, ([], 1997001))
 
+    def test__ffi_call(self):
+        from pypy.rlib.test.test_libffi import get_libm_name
+        libm_name = get_libm_name(sys.platform)
+        out = self.run_source('''
+        def main():
+            from _ffi import CDLL, types
+            libm = CDLL('%(libm_name)s')
+            pow = libm.getfunc('pow', [types.double, types.double],
+                               types.double)
+            print pow.getaddr()
+            i = 0
+            res = 0
+            while i < 2000:
+                res += pow(2, 3)
+                i += 1
+            return res
+        ''' % locals(),
+                              76, ([], 8.0*2000), threshold=1000)
+        pow_addr = int(out.splitlines()[0])
+        ops = self.get_by_bytecode('CALL_FUNCTION')
+        assert len(ops) == 2 # we get two loops, because of specialization
+        call_function = ops[0]
+        last_ops = [op.getopname() for op in call_function[-5:]]
+        assert last_ops == ['force_token',
+                            'setfield_gc',
+                            'call_may_force',
+                            'guard_not_forced',
+                            'guard_no_exception']
+        call = call_function[-3]
+        assert call.getarg(0).value == pow_addr
+        assert call.getarg(1).value == 2.0
+        assert call.getarg(2).value == 3.0
+
     # test_circular
 
 class AppTestJIT(PyPyCJITTests):

Modified: pypy/trunk/pypy/rlib/jit.py
==============================================================================
--- pypy/trunk/pypy/rlib/jit.py	(original)
+++ pypy/trunk/pypy/rlib/jit.py	Mon Oct 18 16:45:16 2010
@@ -77,6 +77,12 @@
         return result
     return decorator
 
+def oopspec(spec):
+    def decorator(func):
+        func.oopspec = spec
+        return func
+    return decorator
+
 class Entry(ExtRegistryEntry):
     _about_ = hint
 

Modified: pypy/trunk/pypy/rlib/test/test_jit.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_jit.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_jit.py	Mon Oct 18 16:45:16 2010
@@ -1,10 +1,16 @@
 import py
 from pypy.rlib.jit import hint, we_are_jitted, JitDriver, purefunction_promote
-from pypy.rlib.jit import JitHintError
+from pypy.rlib.jit import JitHintError, oopspec
 from pypy.translator.translator import TranslationContext, graphof
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.rpython.lltypesystem import lltype
 
+def test_oopspec():
+    @oopspec('foobar')
+    def fn():
+        pass
+    assert fn.oopspec == 'foobar'
+    
 class BaseTestJIT(BaseRtypingTest):
     def test_hint(self):
         def f():

Modified: pypy/trunk/pypy/rlib/test/test_rdynload.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_rdynload.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_rdynload.py	Mon Oct 18 16:45:16 2010
@@ -1,5 +1,5 @@
 from pypy.rlib.rdynload import *
-from pypy.rlib.libffi import get_libc_name
+from pypy.rlib.clibffi import get_libc_name
 from pypy.rpython.lltypesystem import rffi, lltype
 import py
 

Modified: pypy/trunk/pypy/rpython/lltypesystem/lltype.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/lltype.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/lltype.py	Mon Oct 18 16:45:16 2010
@@ -794,6 +794,8 @@
             return llmemory.cast_adr_to_ptr(value, TGT)
     elif TGT == llmemory.Address and isinstance(ORIG, Ptr):
         return llmemory.cast_ptr_to_adr(value)
+    elif TGT == Signed and isinstance(ORIG, Ptr) and ORIG.TO._gckind == 'raw':
+        return llmemory.cast_adr_to_int(llmemory.cast_ptr_to_adr(value), 'symbolic')
     raise TypeError("don't know how to cast from %r to %r" % (ORIG, TGT))
 
 

Modified: pypy/trunk/pypy/rpython/rpbc.py
==============================================================================
--- pypy/trunk/pypy/rpython/rpbc.py	(original)
+++ pypy/trunk/pypy/rpython/rpbc.py	Mon Oct 18 16:45:16 2010
@@ -256,6 +256,8 @@
     def convert_const(self, value):
         if isinstance(value, types.MethodType) and value.im_self is None:
             value = value.im_func   # unbound method -> bare function
+        elif isinstance(value, staticmethod):
+            value = value.__get__(42) # hackish, get the function wrapped by staticmethod
         if self.lowleveltype is Void:
             return None
         if value is None:

Modified: pypy/trunk/pypy/rpython/test/test_rclass.py
==============================================================================
--- pypy/trunk/pypy/rpython/test/test_rclass.py	(original)
+++ pypy/trunk/pypy/rpython/test/test_rclass.py	Mon Oct 18 16:45:16 2010
@@ -319,6 +319,17 @@
         res = self.interpret(f, [])
         assert res == 42
 
+    def test_staticmethod2(self):
+        class A(object):
+            f = staticmethod(lambda x, y: x*y)
+        class B(A):
+            f = staticmethod(lambda x, y: x+y)
+        def f():
+            b = B()
+            return b.f(6, 7)
+        res = self.interpret(f, [])
+        assert res == 13
+
     def test_is(self):
         class A: pass
         class B(A): pass

Modified: pypy/trunk/pypy/translator/c/test/test_newgc.py
==============================================================================
--- pypy/trunk/pypy/translator/c/test/test_newgc.py	(original)
+++ pypy/trunk/pypy/translator/c/test/test_newgc.py	Mon Oct 18 16:45:16 2010
@@ -624,13 +624,13 @@
         os.unlink(self.filename)
 
     def define_callback_with_collect(cls):
-        from pypy.rlib.libffi import ffi_type_pointer, cast_type_to_ffitype,\
+        from pypy.rlib.clibffi import ffi_type_pointer, cast_type_to_ffitype,\
              CDLL, ffi_type_void, CallbackFuncPtr, ffi_type_sint
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         import gc
         ffi_size_t = cast_type_to_ffitype(rffi.SIZE_T)
 
-        from pypy.rlib.libffi import get_libc_name
+        from pypy.rlib.clibffi import get_libc_name
 
         def callback(ll_args, ll_res, stuff):
             gc.collect()


From arigo at codespeak.net  Mon Oct 18 17:18:14 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 17:18:14 +0200 (CEST)
Subject: [pypy-svn] r78048 - pypy/branch/rsre-jit/pypy/jit/backend/x86
Message-ID: <20101018151814.68676282B90@codespeak.net>

Author: arigo
Date: Mon Oct 18 17:18:11 2010
New Revision: 78048

Modified:
   pypy/branch/rsre-jit/pypy/jit/backend/x86/regalloc.py
Log:
"Implement" jit_debug in the x86 backend.


Modified: pypy/branch/rsre-jit/pypy/jit/backend/x86/regalloc.py
==============================================================================
--- pypy/branch/rsre-jit/pypy/jit/backend/x86/regalloc.py	(original)
+++ pypy/branch/rsre-jit/pypy/jit/backend/x86/regalloc.py	Mon Oct 18 17:18:11 2010
@@ -1068,6 +1068,9 @@
     def consider_debug_merge_point(self, op):
         pass
 
+    def consider_jit_debug(self, op):
+        pass
+
     def get_mark_gc_roots(self, gcrootmap):
         shape = gcrootmap.get_basic_shape(IS_X86_64)
         for v, val in self.fm.frame_bindings.items():


From arigo at codespeak.net  Mon Oct 18 17:24:53 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 17:24:53 +0200 (CEST)
Subject: [pypy-svn] r78049 - in pypy/trunk: dotviewer pypy/jit/backend/x86
	pypy/jit/codewriter pypy/jit/codewriter/test
	pypy/jit/metainterp pypy/jit/metainterp/optimizeopt
	pypy/jit/metainterp/test pypy/rlib pypy/rlib/rsre
	pypy/rlib/rsre/test pypy/rlib/test
Message-ID: <20101018152453.897C7282B9D@codespeak.net>

Author: arigo
Date: Mon Oct 18 17:24:50 2010
New Revision: 78049

Added:
   pypy/trunk/pypy/jit/metainterp/greenfield.py
      - copied unchanged from r78048, pypy/branch/rsre-jit/pypy/jit/metainterp/greenfield.py
   pypy/trunk/pypy/jit/metainterp/test/test_greenfield.py
      - copied unchanged from r78048, pypy/branch/rsre-jit/pypy/jit/metainterp/test/test_greenfield.py
   pypy/trunk/pypy/rlib/rsre/rsre_jit.py
      - copied unchanged from r78048, pypy/branch/rsre-jit/pypy/rlib/rsre/rsre_jit.py
   pypy/trunk/pypy/rlib/rsre/test/conftest.py
      - copied unchanged from r78048, pypy/branch/rsre-jit/pypy/rlib/rsre/test/conftest.py
   pypy/trunk/pypy/rlib/rsre/test/test_zjit.py
      - copied unchanged from r78048, pypy/branch/rsre-jit/pypy/rlib/rsre/test/test_zjit.py
Modified:
   pypy/trunk/dotviewer/drawgraph.py
   pypy/trunk/pypy/jit/backend/x86/regalloc.py
   pypy/trunk/pypy/jit/codewriter/call.py
   pypy/trunk/pypy/jit/codewriter/codewriter.py
   pypy/trunk/pypy/jit/codewriter/jtransform.py
   pypy/trunk/pypy/jit/codewriter/support.py
   pypy/trunk/pypy/jit/codewriter/test/test_codewriter.py
   pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py
   pypy/trunk/pypy/jit/metainterp/blackhole.py
   pypy/trunk/pypy/jit/metainterp/compile.py
   pypy/trunk/pypy/jit/metainterp/executor.py
   pypy/trunk/pypy/jit/metainterp/graphpage.py
   pypy/trunk/pypy/jit/metainterp/jitdriver.py
   pypy/trunk/pypy/jit/metainterp/optimizeopt/string.py
   pypy/trunk/pypy/jit/metainterp/optimizeopt/virtualize.py
   pypy/trunk/pypy/jit/metainterp/pyjitpl.py
   pypy/trunk/pypy/jit/metainterp/resoperation.py
   pypy/trunk/pypy/jit/metainterp/resume.py
   pypy/trunk/pypy/jit/metainterp/test/test_basic.py
   pypy/trunk/pypy/jit/metainterp/test/test_virtualref.py
   pypy/trunk/pypy/jit/metainterp/test/test_warmspot.py
   pypy/trunk/pypy/jit/metainterp/warmspot.py
   pypy/trunk/pypy/rlib/jit.py
   pypy/trunk/pypy/rlib/rsre/rsre_char.py
   pypy/trunk/pypy/rlib/rsre/rsre_core.py
   pypy/trunk/pypy/rlib/test/test_jit.py
Log:
Merge branch/rsre-jit: put the JIT hints in the rsre module.
Requires small extensions in the JIT:

 - "green fields": the ability to write 'foo.bar' in the list of greens
   in the JitDriver, where 'foo' is written in the list of reds.

 - the ability to only write jitdriver.jit_merge_point(), without any
   jitdriver.can_enter_jit().  In that case the jit_merge_point() plays both
   roles.  The difference with putting explicitly a can_enter_jit() just before
   is that such a can_enter_jit() is not seen unless we are closing a loop; in
   particular, it does not work if we have no loop at all.



Modified: pypy/trunk/dotviewer/drawgraph.py
==============================================================================
--- pypy/trunk/dotviewer/drawgraph.py	(original)
+++ pypy/trunk/dotviewer/drawgraph.py	Mon Oct 18 17:24:50 2010
@@ -423,20 +423,43 @@
         else:
             for line in lines:
                 raw_line = line.replace('\\l','').replace('\r','') or ' '
-                img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
-                w, h = img.get_size()
-                if w>wmax: wmax = w
-                if raw_line.strip():
-                    if line.endswith('\\l'):
-                        def cmd(img=img, y=hmax):
-                            img.draw(xleft, ytop+y)
-                    elif line.endswith('\r'):
-                        def cmd(img=img, y=hmax, w=w):
-                            img.draw(xright-w, ytop+y)
-                    else:
-                        def cmd(img=img, y=hmax, w=w):
-                            img.draw(xcenter-w//2, ytop+y)
+                if '\f' in raw_line:   # grayed out parts of the line
+                    imgs = []
+                    graytext = True
+                    h = 16
+                    w_total = 0
+                    for linepart in raw_line.split('\f'):
+                        graytext = not graytext
+                        if not linepart.strip():
+                            continue
+                        if graytext:
+                            fgcolor = (128, 160, 160)
+                        else:
+                            fgcolor = (0, 0, 0)
+                        img = TextSnippet(self, linepart, fgcolor, bgcolor)
+                        imgs.append((w_total, img))
+                        w, h = img.get_size()
+                        w_total += w
+                    if w_total > wmax: wmax = w_total
+                    def cmd(imgs=imgs, y=hmax):
+                        for x, img in imgs:
+                            img.draw(xleft+x, ytop+y)
                     commands.append(cmd)
+                else:
+                    img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
+                    w, h = img.get_size()
+                    if w>wmax: wmax = w
+                    if raw_line.strip():
+                        if line.endswith('\\l'):
+                            def cmd(img=img, y=hmax):
+                                img.draw(xleft, ytop+y)
+                        elif line.endswith('\r'):
+                            def cmd(img=img, y=hmax, w=w):
+                                img.draw(xright-w, ytop+y)
+                        else:
+                            def cmd(img=img, y=hmax, w=w):
+                                img.draw(xcenter-w//2, ytop+y)
+                        commands.append(cmd)
                 hmax += h
                 #hmax += 8
 

Modified: pypy/trunk/pypy/jit/backend/x86/regalloc.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/regalloc.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/regalloc.py	Mon Oct 18 17:24:50 2010
@@ -1068,6 +1068,9 @@
     def consider_debug_merge_point(self, op):
         pass
 
+    def consider_jit_debug(self, op):
+        pass
+
     def get_mark_gc_roots(self, gcrootmap):
         shape = gcrootmap.get_basic_shape(IS_X86_64)
         for v, val in self.fm.frame_bindings.items():

Modified: pypy/trunk/pypy/jit/codewriter/call.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/call.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/call.py	Mon Oct 18 17:24:50 2010
@@ -277,3 +277,11 @@
             return seen.pop()
         else:
             return None
+
+    def could_be_green_field(self, GTYPE, fieldname):
+        GTYPE_fieldname = (GTYPE, fieldname)
+        for jd in self.jitdrivers_sd:
+            if jd.greenfield_info is not None:
+                if GTYPE_fieldname in jd.greenfield_info.green_fields:
+                    return True
+        return False

Modified: pypy/trunk/pypy/jit/codewriter/codewriter.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/codewriter.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/codewriter.py	Mon Oct 18 17:24:50 2010
@@ -95,18 +95,18 @@
             print '%s:' % (ssarepr.name,)
             print format_assembler(ssarepr)
         else:
-            dir = udir.ensure("jitcodes", dir=1)
-            if portal_jitdriver:
-                name = "%02d_portal_runner" % (portal_jitdriver.index,)
-            elif ssarepr.name and ssarepr.name != '?':
-                name = ssarepr.name
-            else:
-                name = 'unnamed' % id(ssarepr)
-            i = 1
-            extra = ''
-            while name+extra in self._seen_files:
-                i += 1
-                extra = '.%d' % i
-            self._seen_files.add(name+extra)
-            dir.join(name+extra).write(format_assembler(ssarepr))
             log.dot()
+        dir = udir.ensure("jitcodes", dir=1)
+        if portal_jitdriver:
+            name = "%02d_portal_runner" % (portal_jitdriver.index,)
+        elif ssarepr.name and ssarepr.name != '?':
+            name = ssarepr.name
+        else:
+            name = 'unnamed' % id(ssarepr)
+        i = 1
+        extra = ''
+        while name+extra in self._seen_files:
+            i += 1
+            extra = '.%d' % i
+        self._seen_files.add(name+extra)
+        dir.join(name+extra).write(format_assembler(ssarepr))

Modified: pypy/trunk/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/jtransform.py	Mon Oct 18 17:24:50 2010
@@ -320,6 +320,8 @@
             prepare = self._handle_str2unicode_call
         elif oopspec_name.startswith('virtual_ref'):
             prepare = self._handle_virtual_ref_call
+        elif oopspec_name.startswith('jit.'):
+            prepare = self._handle_jit_call
         elif oopspec_name.startswith('libffi_'):
             prepare = self._handle_libffi_call
         else:
@@ -523,7 +525,12 @@
         # check for deepfrozen structures that force constant-folding
         immut = v_inst.concretetype.TO._immutable_field(c_fieldname.value)
         if immut:
-            pure = '_pure'
+            if (self.callcontrol is not None and
+                self.callcontrol.could_be_green_field(v_inst.concretetype.TO,
+                                                      c_fieldname.value)):
+                pure = '_greenfield'
+            else:
+                pure = '_pure'
             if immut == "[*]":
                 self.immutable_arrays[op.result] = True
         else:
@@ -856,6 +863,15 @@
                     (self.graph,))
         return []
 
+    def _handle_jit_call(self, op, oopspec_name, args):
+        if oopspec_name == 'jit.debug':
+            return SpaceOperation('jit_debug', args, None)
+        elif oopspec_name == 'jit.assert_green':
+            kind = getkind(args[0].concretetype)
+            return SpaceOperation('%s_assert_green' % kind, args, None)
+        else:
+            raise AssertionError("missing support for %r" % oopspec_name)
+
     # ----------
     # Lists.
 

Modified: pypy/trunk/pypy/jit/codewriter/support.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/support.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/support.py	Mon Oct 18 17:24:50 2010
@@ -61,7 +61,7 @@
     return rtyper.annotator.translator.graphs[0]
 
 def split_before_jit_merge_point(graph, portalblock, portalopindex):
-    """Find the block with 'jit_merge_point' and split just before,
+    """Split the block just before the 'jit_merge_point',
     making sure the input args are in the canonical order.
     """
     # split the block just before the jit_merge_point()

Modified: pypy/trunk/pypy/jit/codewriter/test/test_codewriter.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/test/test_codewriter.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/test/test_codewriter.py	Mon Oct 18 17:24:50 2010
@@ -45,6 +45,7 @@
         self.portal_graph = portal_graph
         self.portal_runner_ptr = "???"
         self.virtualizable_info = None
+        self.greenfield_info = None
 
 
 def test_loop():

Modified: pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py	Mon Oct 18 17:24:50 2010
@@ -687,6 +687,79 @@
     assert block.operations[1].result is None
     assert block.exits[0].args == [v1]
 
+def test_jit_merge_point_1():
+    class FakeJitDriverSD:
+        index = 42
+        class jitdriver:
+            greens = ['green1', 'green2', 'voidgreen3']
+            reds = ['red1', 'red2', 'voidred3']
+    jd = FakeJitDriverSD()
+    v1 = varoftype(lltype.Signed)
+    v2 = varoftype(lltype.Signed)
+    vvoid1 = varoftype(lltype.Void)
+    v3 = varoftype(lltype.Signed)
+    v4 = varoftype(lltype.Signed)
+    vvoid2 = varoftype(lltype.Void)
+    v5 = varoftype(lltype.Void)
+    op = SpaceOperation('jit_marker',
+                        [Constant('jit_merge_point', lltype.Void),
+                         Constant(jd.jitdriver, lltype.Void),
+                         v1, v2, vvoid1, v3, v4, vvoid2], v5)
+    tr = Transformer()
+    tr.portal_jd = jd
+    oplist = tr.rewrite_operation(op)
+    assert len(oplist) == 6
+    assert oplist[0].opname == '-live-'
+    assert oplist[1].opname == 'int_guard_value'
+    assert oplist[1].args   == [v1]
+    assert oplist[2].opname == '-live-'
+    assert oplist[3].opname == 'int_guard_value'
+    assert oplist[3].args   == [v2]
+    assert oplist[4].opname == 'jit_merge_point'
+    assert oplist[4].args[0].value == 42
+    assert list(oplist[4].args[1]) == [v1, v2]
+    assert list(oplist[4].args[4]) == [v3, v4]
+    assert oplist[5].opname == '-live-'
+
+def test_getfield_gc():
+    S = lltype.GcStruct('S', ('x', lltype.Char))
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
+def test_getfield_gc_pure():
+    S = lltype.GcStruct('S', ('x', lltype.Char),
+                        hints={'immutable': True})
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i_pure'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
+def test_getfield_gc_greenfield():
+    class FakeCC:
+        def get_vinfo(self, v):
+            return None
+        def could_be_green_field(self, S1, name1):
+            assert S1 is S
+            assert name1 == 'x'
+            return True
+    S = lltype.GcStruct('S', ('x', lltype.Char),
+                        hints={'immutable': True})
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU(), FakeCC()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i_greenfield'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
 def test_int_abs():
     v1 = varoftype(lltype.Signed)
     v2 = varoftype(lltype.Signed)

Modified: pypy/trunk/pypy/jit/metainterp/blackhole.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/blackhole.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/blackhole.py	Mon Oct 18 17:24:50 2010
@@ -760,6 +760,20 @@
     def bhimpl_debug_fatalerror(msg):
         llop.debug_fatalerror(lltype.Void, msg)
 
+    @arguments("r", "i", "i", "i", "i")
+    def bhimpl_jit_debug(string, arg1=0, arg2=0, arg3=0, arg4=0):
+        pass
+
+    @arguments("i")
+    def bhimpl_int_assert_green(x):
+        pass
+    @arguments("r")
+    def bhimpl_ref_assert_green(x):
+        pass
+    @arguments("f")
+    def bhimpl_float_assert_green(x):
+        pass
+
     # ----------
     # the main hints and recursive calls
 
@@ -1073,6 +1087,10 @@
     bhimpl_getfield_vable_r = bhimpl_getfield_gc_r
     bhimpl_getfield_vable_f = bhimpl_getfield_gc_f
 
+    bhimpl_getfield_gc_i_greenfield = bhimpl_getfield_gc_i
+    bhimpl_getfield_gc_r_greenfield = bhimpl_getfield_gc_r
+    bhimpl_getfield_gc_f_greenfield = bhimpl_getfield_gc_f
+
     @arguments("cpu", "i", "d", returns="i")
     def bhimpl_getfield_raw_i(cpu, struct, fielddescr):
         return cpu.bh_getfield_raw_i(struct, fielddescr)

Modified: pypy/trunk/pypy/jit/metainterp/compile.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/compile.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/compile.py	Mon Oct 18 17:24:50 2010
@@ -370,7 +370,8 @@
         from pypy.jit.metainterp.resume import force_from_resumedata
         metainterp_sd = self.metainterp_sd
         vinfo = self.jitdriver_sd.virtualizable_info
-        all_virtuals = force_from_resumedata(metainterp_sd, self, vinfo)
+        ginfo = self.jitdriver_sd.greenfield_info
+        all_virtuals = force_from_resumedata(metainterp_sd, self, vinfo, ginfo)
         # The virtualizable data was stored on the real virtualizable above.
         # Handle all_virtuals: keep them for later blackholing from the
         # future failure of the GUARD_NOT_FORCED

Modified: pypy/trunk/pypy/jit/metainterp/executor.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/executor.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/executor.py	Mon Oct 18 17:24:50 2010
@@ -307,6 +307,7 @@
                          rop.CALL_ASSEMBLER,
                          rop.COND_CALL_GC_WB,
                          rop.DEBUG_MERGE_POINT,
+                         rop.JIT_DEBUG,
                          rop.SETARRAYITEM_RAW,
                          ):      # list of opcodes never executed by pyjitpl
                 continue

Modified: pypy/trunk/pypy/jit/metainterp/graphpage.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/graphpage.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/graphpage.py	Mon Oct 18 17:24:50 2010
@@ -153,7 +153,7 @@
         opindex = opstartindex
         while True:
             op = operations[opindex]
-            lines.append(repr(op))
+            lines.append(op.repr(graytext=True))
             if is_interesting_guard(op):
                 tgt = op.getdescr()._debug_suboperations[0]
                 tgt_g, tgt_i = self.all_operations[tgt]

Modified: pypy/trunk/pypy/jit/metainterp/jitdriver.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/jitdriver.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/jitdriver.py	Mon Oct 18 17:24:50 2010
@@ -13,8 +13,10 @@
     #    self.num_red_args      ... pypy.jit.metainterp.warmspot
     #    self.result_type       ... pypy.jit.metainterp.warmspot
     #    self.virtualizable_info... pypy.jit.metainterp.warmspot
+    #    self.greenfield_info   ... pypy.jit.metainterp.warmspot
     #    self.warmstate         ... pypy.jit.metainterp.warmspot
     #    self.handle_jitexc_from_bh pypy.jit.metainterp.warmspot
+    #    self.no_loop_header    ... pypy.jit.metainterp.warmspot
     #    self.portal_finishtoken... pypy.jit.metainterp.pyjitpl
     #    self.index             ... pypy.jit.codewriter.call
     #    self.mainjitcode       ... pypy.jit.codewriter.call

Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/string.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/optimizeopt/string.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/optimizeopt/string.py	Mon Oct 18 17:24:50 2010
@@ -12,7 +12,7 @@
 from pypy.jit.codewriter.effectinfo import EffectInfo, callinfo_for_oopspec
 from pypy.jit.codewriter import heaptracker
 from pypy.rlib.unroll import unrolling_iterable
-from pypy.rlib.objectmodel import specialize
+from pypy.rlib.objectmodel import specialize, we_are_translated
 
 
 class StrOrUnicode(object):
@@ -107,7 +107,10 @@
         self.box = box = self.source_op.result
         newoperations = self.optimizer.newoperations
         lengthbox = self.getstrlen(newoperations, self.mode)
-        newoperations.append(ResOperation(self.mode.NEWSTR, [lengthbox], box))
+        op = ResOperation(self.mode.NEWSTR, [lengthbox], box)
+        if not we_are_translated():
+            op.name = 'FORCE'
+        newoperations.append(op)
         self.string_copy_parts(newoperations, box, CONST_0, self.mode)
 
 

Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/optimizeopt/virtualize.py	Mon Oct 18 17:24:50 2010
@@ -74,6 +74,8 @@
         assert self.source_op is not None
         # ^^^ This case should not occur any more (see test_bug_3).
         #
+        if not we_are_translated():
+            self.source_op.name = 'FORCE ' + self.source_op.name
         newoperations = self.optimizer.newoperations
         newoperations.append(self.source_op)
         self.box = box = self.source_op.result
@@ -170,6 +172,8 @@
 
     def _really_force(self):
         assert self.source_op is not None
+        if not we_are_translated():
+            self.source_op.name = 'FORCE ' + self.source_op.name
         newoperations = self.optimizer.newoperations
         newoperations.append(self.source_op)
         self.box = box = self.source_op.result

Modified: pypy/trunk/pypy/jit/metainterp/pyjitpl.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/pyjitpl.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/pyjitpl.py	Mon Oct 18 17:24:50 2010
@@ -1,4 +1,4 @@
-import py, os
+import py, os, sys
 from pypy.rpython.lltypesystem import lltype, llmemory, rclass
 from pypy.rlib.objectmodel import we_are_translated
 from pypy.rlib.unroll import unrolling_iterable
@@ -498,6 +498,22 @@
     opimpl_getfield_gc_r_pure = _opimpl_getfield_gc_pure_any
     opimpl_getfield_gc_f_pure = _opimpl_getfield_gc_pure_any
 
+    @arguments("orgpc", "box", "descr")
+    def _opimpl_getfield_gc_greenfield_any(self, pc, box, fielddescr):
+        ginfo = self.metainterp.jitdriver_sd.greenfield_info
+        if (ginfo is not None and fielddescr in ginfo.green_field_descrs
+            and not self._nonstandard_virtualizable(pc, box)):
+            # fetch the result, but consider it as a Const box and don't
+            # record any operation
+            resbox = executor.execute(self.metainterp.cpu, self.metainterp,
+                                      rop.GETFIELD_GC_PURE, fielddescr, box)
+            return resbox.constbox()
+        # fall-back
+        return self.execute_with_descr(rop.GETFIELD_GC_PURE, fielddescr, box)
+    opimpl_getfield_gc_i_greenfield = _opimpl_getfield_gc_greenfield_any
+    opimpl_getfield_gc_r_greenfield = _opimpl_getfield_gc_greenfield_any
+    opimpl_getfield_gc_f_greenfield = _opimpl_getfield_gc_greenfield_any
+
     @arguments("box", "descr", "box")
     def _opimpl_setfield_gc_any(self, box, fielddescr, valuebox):
         self.execute_with_descr(rop.SETFIELD_GC, fielddescr, box, valuebox)
@@ -529,7 +545,8 @@
     def _nonstandard_virtualizable(self, pc, box):
         # returns True if 'box' is actually not the "standard" virtualizable
         # that is stored in metainterp.virtualizable_boxes[-1]
-        if self.metainterp.jitdriver_sd.virtualizable_info is None:
+        if (self.metainterp.jitdriver_sd.virtualizable_info is None and
+            self.metainterp.jitdriver_sd.greenfield_info is None):
             return True      # can occur in case of multiple JITs
         standard_box = self.metainterp.virtualizable_boxes[-1]
         if standard_box is box:
@@ -799,12 +816,16 @@
 
     @arguments("orgpc", "int", "boxes3", "boxes3")
     def opimpl_jit_merge_point(self, orgpc, jdindex, greenboxes, redboxes):
+        any_operation = len(self.metainterp.history.operations) > 0
         jitdriver_sd = self.metainterp.staticdata.jitdrivers_sd[jdindex]
         self.verify_green_args(jitdriver_sd, greenboxes)
         # xxx we may disable the following line in some context later
         self.debug_merge_point(jitdriver_sd, greenboxes)
         if self.metainterp.seen_loop_header_for_jdindex < 0:
-            return
+            if not jitdriver_sd.no_loop_header or not any_operation:
+                return
+            # automatically add a loop_header if there is none
+            self.metainterp.seen_loop_header_for_jdindex = jdindex
         #
         assert self.metainterp.seen_loop_header_for_jdindex == jdindex, (
             "found a loop_header for a JitDriver that does not match "
@@ -893,6 +914,40 @@
         msg = box.getref(lltype.Ptr(rstr.STR))
         lloperation.llop.debug_fatalerror(msg)
 
+    @arguments("box", "box", "box", "box", "box")
+    def opimpl_jit_debug(self, stringbox, arg1box, arg2box, arg3box, arg4box):
+        from pypy.rpython.lltypesystem import rstr
+        from pypy.rpython.annlowlevel import hlstr
+        msg = stringbox.getref(lltype.Ptr(rstr.STR))
+        debug_print('jit_debug:', hlstr(msg),
+                    arg1box.getint(), arg2box.getint(),
+                    arg3box.getint(), arg4box.getint())
+        args = [stringbox, arg1box, arg2box, arg3box, arg4box]
+        i = 4
+        while i > 0 and args[i].getint() == -sys.maxint-1:
+            i -= 1
+        assert i >= 0
+        op = self.metainterp.history.record(rop.JIT_DEBUG, args[:i+1], None)
+        self.metainterp.attach_debug_info(op)
+
+    @arguments("box")
+    def _opimpl_assert_green(self, box):
+        if not isinstance(box, Const):
+            msg = "assert_green failed at %s:%d" % (
+                self.jitcode.name,
+                self.pc)
+            if we_are_translated():
+                from pypy.rpython.annlowlevel import llstr
+                from pypy.rpython.lltypesystem import lloperation
+                lloperation.llop.debug_fatalerror(lltype.Void, llstr(msg))
+            else:
+                from pypy.rlib.jit import AssertGreenFailed
+                raise AssertGreenFailed(msg)
+
+    opimpl_int_assert_green   = _opimpl_assert_green
+    opimpl_ref_assert_green   = _opimpl_assert_green
+    opimpl_float_assert_green = _opimpl_assert_green
+
     @arguments("box")
     def opimpl_virtual_ref(self, box):
         # Details on the content of metainterp.virtualref_boxes:
@@ -998,7 +1053,8 @@
         guard_op = metainterp.history.record(opnum, moreargs, None,
                                              descr=resumedescr)
         virtualizable_boxes = None
-        if metainterp.jitdriver_sd.virtualizable_info is not None:
+        if (metainterp.jitdriver_sd.virtualizable_info is not None or
+            metainterp.jitdriver_sd.greenfield_info is not None):
             virtualizable_boxes = metainterp.virtualizable_boxes
         saved_pc = self.pc
         if resumepc >= 0:
@@ -1646,6 +1702,7 @@
                                               duplicates)
             live_arg_boxes += self.virtualizable_boxes
             live_arg_boxes.pop()
+        #
         assert len(self.virtualref_boxes) == 0, "missing virtual_ref_finish()?"
         # Called whenever we reach the 'loop_header' hint.
         # First, attempt to make a bridge:
@@ -1832,6 +1889,7 @@
         f.setup_call(original_boxes)
         assert self.in_recursion == 0
         self.virtualref_boxes = []
+        self.initialize_withgreenfields(original_boxes)
         self.initialize_virtualizable(original_boxes)
 
     def initialize_state_from_guard_failure(self, resumedescr):
@@ -1856,6 +1914,14 @@
             self.virtualizable_boxes.append(virtualizable_box)
             self.initialize_virtualizable_enter()
 
+    def initialize_withgreenfields(self, original_boxes):
+        ginfo = self.jitdriver_sd.greenfield_info
+        if ginfo is not None:
+            assert self.jitdriver_sd.virtualizable_info is None
+            index = (self.jitdriver_sd.num_green_args +
+                     ginfo.red_index)
+            self.virtualizable_boxes = [original_boxes[index]]
+
     def initialize_virtualizable_enter(self):
         vinfo = self.jitdriver_sd.virtualizable_info
         virtualizable_box = self.virtualizable_boxes[-1]
@@ -1949,8 +2015,10 @@
 
     def rebuild_state_after_failure(self, resumedescr):
         vinfo = self.jitdriver_sd.virtualizable_info
+        ginfo = self.jitdriver_sd.greenfield_info
         self.framestack = []
-        boxlists = resume.rebuild_from_resumedata(self, resumedescr, vinfo)
+        boxlists = resume.rebuild_from_resumedata(self, resumedescr, vinfo,
+                                                  ginfo)
         inputargs_and_holes, virtualizable_boxes, virtualref_boxes = boxlists
         #
         # virtual refs: make the vrefs point to the freshly allocated virtuals
@@ -1975,6 +2043,12 @@
             assert not virtualizable.vable_token
             # fill the virtualizable with the local boxes
             self.synchronize_virtualizable()
+        #
+        elif self.jitdriver_sd.greenfield_info:
+            self.virtualizable_boxes = virtualizable_boxes
+        else:
+            assert not virtualizable_boxes
+        #
         return inputargs_and_holes
 
     def check_synchronized_virtualizable(self):
@@ -2048,7 +2122,8 @@
         for i in range(len(boxes)):
             if boxes[i] is oldbox:
                 boxes[i] = newbox
-        if self.jitdriver_sd.virtualizable_info is not None:
+        if (self.jitdriver_sd.virtualizable_info is not None or
+            self.jitdriver_sd.greenfield_info is not None):
             boxes = self.virtualizable_boxes
             for i in range(len(boxes)):
                 if boxes[i] is oldbox:

Modified: pypy/trunk/pypy/jit/metainterp/resoperation.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/resoperation.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/resoperation.py	Mon Oct 18 17:24:50 2010
@@ -93,7 +93,7 @@
     def __repr__(self):
         return self.repr()
 
-    def repr(self):
+    def repr(self, graytext=False):
         # RPython-friendly version
         if self.result is not None:
             sres = '%s = ' % (self.result,)
@@ -101,6 +101,8 @@
             sres = ''
         if self.name:
             prefix = "%s:%s   " % (self.name, self.pc)
+            if graytext:
+                prefix = "\f%s\f" % prefix
         else:
             prefix = ""
         args = self.getarglist()
@@ -457,6 +459,7 @@
     #'RUNTIMENEW/1',     # ootype operation    
     'COND_CALL_GC_WB/2d', # [objptr, newvalue]   (for the write barrier)
     'DEBUG_MERGE_POINT/1',      # debugging only
+    'JIT_DEBUG/*',              # debugging only
     'VIRTUAL_REF_FINISH/2',   # removed before it's passed to the backend
     'COPYSTRCONTENT/5',       # src, dst, srcstart, dststart, length
     'COPYUNICODECONTENT/5',

Modified: pypy/trunk/pypy/jit/metainterp/resume.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/resume.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/resume.py	Mon Oct 18 17:24:50 2010
@@ -689,9 +689,11 @@
 
 # ---------- when resuming for pyjitpl.py, make boxes ----------
 
-def rebuild_from_resumedata(metainterp, storage, virtualizable_info):
+def rebuild_from_resumedata(metainterp, storage, virtualizable_info,
+                            greenfield_info):
     resumereader = ResumeDataBoxReader(storage, metainterp)
-    boxes = resumereader.consume_vref_and_vable_boxes(virtualizable_info)
+    boxes = resumereader.consume_vref_and_vable_boxes(virtualizable_info,
+                                                      greenfield_info)
     virtualizable_boxes, virtualref_boxes = boxes
     frameinfo = storage.rd_frame_info_list
     while True:
@@ -736,15 +738,18 @@
         assert (end & 1) == 0
         return [self.decode_ref(nums[i]) for i in range(end)]
 
-    def consume_vref_and_vable_boxes(self, vinfo):
+    def consume_vref_and_vable_boxes(self, vinfo, ginfo):
         nums = self.cur_numb.nums
         self.cur_numb = self.cur_numb.prev
-        if vinfo is None:
-            virtualizable_boxes = None
-            end = len(nums)
-        else:
+        if vinfo is not None:
             virtualizable_boxes = self.consume_virtualizable_boxes(vinfo, nums)
             end = len(nums) - len(virtualizable_boxes)
+        elif ginfo is not None:
+            virtualizable_boxes = [self.decode_ref(nums[-1])]
+            end = len(nums) - 1
+        else:
+            virtualizable_boxes = None
+            end = len(nums)
         virtualref_boxes = self.consume_virtualref_boxes(nums, end)
         return virtualizable_boxes, virtualref_boxes
 
@@ -901,8 +906,9 @@
     resumereader = ResumeDataDirectReader(blackholeinterpbuilder.cpu, storage,
                                           all_virtuals)
     vinfo = jitdriver_sd.virtualizable_info
+    ginfo = jitdriver_sd.greenfield_info
     vrefinfo = blackholeinterpbuilder.metainterp_sd.virtualref_info
-    resumereader.consume_vref_and_vable(vrefinfo, vinfo)
+    resumereader.consume_vref_and_vable(vrefinfo, vinfo, ginfo)
     #
     # First get a chain of blackhole interpreters whose length is given
     # by the depth of rd_frame_info_list.  The first one we get must be
@@ -932,11 +938,11 @@
     resumereader.done()
     return firstbh
 
-def force_from_resumedata(metainterp_sd, storage, vinfo=None):
+def force_from_resumedata(metainterp_sd, storage, vinfo, ginfo):
     resumereader = ResumeDataDirectReader(metainterp_sd.cpu, storage)
     resumereader.handling_async_forcing()
     vrefinfo = metainterp_sd.virtualref_info
-    resumereader.consume_vref_and_vable(vrefinfo, vinfo)
+    resumereader.consume_vref_and_vable(vrefinfo, vinfo, ginfo)
     return resumereader.force_all_virtuals()
 
 class ResumeDataDirectReader(AbstractResumeDataReader):
@@ -1011,11 +1017,12 @@
         return specialize_value(TYPE, x)
     load_value_of_type._annspecialcase_ = 'specialize:arg(1)'
 
-    def consume_vref_and_vable(self, vrefinfo, vinfo):
+    def consume_vref_and_vable(self, vrefinfo, vinfo, ginfo):
         nums = self.cur_numb.nums
         self.cur_numb = self.cur_numb.prev
         if self.resume_after_guard_not_forced != 2:
             end_vref = self.consume_vable_info(vinfo, nums)
+            if ginfo is not None: end_vref -= 1
             self.consume_virtualref_info(vrefinfo, nums, end_vref)
 
     def allocate_with_vtable(self, known_class):

Modified: pypy/trunk/pypy/jit/metainterp/test/test_basic.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_basic.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_basic.py	Mon Oct 18 17:24:50 2010
@@ -2,6 +2,7 @@
 import sys
 from pypy.rlib.jit import JitDriver, we_are_jitted, hint, dont_look_inside
 from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_SIMPLE, loop_invariant
+from pypy.rlib.jit import jit_debug, assert_green, AssertGreenFailed
 from pypy.jit.metainterp.warmspot import ll_meta_interp, get_stats
 from pypy.jit.backend.llgraph import runner
 from pypy.jit.metainterp import pyjitpl, history
@@ -44,6 +45,7 @@
         num_green_args = 0
         portal_graph = graphs[0]
         virtualizable_info = None
+        greenfield_info = None
         result_type = result_kind
         portal_runner_ptr = "???"
 
@@ -1644,6 +1646,33 @@
         res = self.interp_operations(f, [10, 3.5])
         assert res == 3.5
 
+    def test_jit_debug(self):
+        myjitdriver = JitDriver(greens = [], reds = ['x'])
+        class A:
+            pass
+        def f(x):
+            while x > 0:
+                myjitdriver.can_enter_jit(x=x)
+                myjitdriver.jit_merge_point(x=x)
+                jit_debug("hi there:", x)
+                jit_debug("foobar")
+                x -= 1
+            return x
+        res = self.meta_interp(f, [8])
+        assert res == 0
+        self.check_loops(jit_debug=2)
+
+    def test_assert_green(self):
+        def f(x, promote):
+            if promote:
+                x = hint(x, promote=True)
+            assert_green(x)
+            return x
+        res = self.interp_operations(f, [8, 1])
+        assert res == 8
+        py.test.raises(AssertGreenFailed, self.interp_operations, f, [8, 0])
+
+
 class TestOOtype(BasicTests, OOJitMixin):
 
     def test_oohash(self):

Modified: pypy/trunk/pypy/jit/metainterp/test/test_virtualref.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_virtualref.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_virtualref.py	Mon Oct 18 17:24:50 2010
@@ -93,7 +93,7 @@
         lst = []
         vrefinfo.continue_tracing = lambda vref, virtual: \
                                         lst.append((vref, virtual))
-        resumereader.consume_vref_and_vable(vrefinfo, None)
+        resumereader.consume_vref_and_vable(vrefinfo, None, None)
         del vrefinfo.continue_tracing
         assert len(lst) == 1
         lltype.cast_opaque_ptr(lltype.Ptr(JIT_VIRTUAL_REF),

Modified: pypy/trunk/pypy/jit/metainterp/test/test_warmspot.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_warmspot.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_warmspot.py	Mon Oct 18 17:24:50 2010
@@ -296,6 +296,69 @@
         assert res == 1
         self.check_loops(int_add=1)   # I get 13 without the loop_header()
 
+    def test_omit_can_enter_jit(self):
+        # Simple test comparing the effects of always giving a can_enter_jit(),
+        # or not giving any.  Mostly equivalent, except that if given, it is
+        # ignored the first time, and so it ends up taking one extra loop to
+        # start JITting.
+        mydriver = JitDriver(greens=[], reds=['m'])
+        #
+        for i2 in range(10):
+            def f2(m):
+                while m > 0:
+                    mydriver.jit_merge_point(m=m)
+                    m -= 1
+            self.meta_interp(f2, [i2])
+            try:
+                self.check_tree_loop_count(1)
+                break
+            except AssertionError:
+                print "f2: no loop generated for i2==%d" % i2
+        else:
+            raise     # re-raise the AssertionError: check_loop_count never 1
+        #
+        for i1 in range(10):
+            def f1(m):
+                while m > 0:
+                    mydriver.can_enter_jit(m=m)
+                    mydriver.jit_merge_point(m=m)
+                    m -= 1
+            self.meta_interp(f1, [i1])
+            try:
+                self.check_tree_loop_count(1)
+                break
+            except AssertionError:
+                print "f1: no loop generated for i1==%d" % i1
+        else:
+            raise     # re-raise the AssertionError: check_loop_count never 1
+        #
+        assert i1 - 1 == i2
+
+    def test_no_loop_at_all(self):
+        mydriver = JitDriver(greens=[], reds=['m'])
+        def f2(m):
+            mydriver.jit_merge_point(m=m)
+            return m - 1
+        def f1(m):
+            while m > 0:
+                m = f2(m)
+        self.meta_interp(f1, [8])
+        # it should generate one "loop" only, which ends in a FINISH
+        # corresponding to the return from f2.
+        self.check_tree_loop_count(1)
+        self.check_loop_count(0)
+
+    def test_simple_loop(self):
+        mydriver = JitDriver(greens=[], reds=['m'])
+        def f1(m):
+            while m > 0:
+                mydriver.jit_merge_point(m=m)
+                m = m - 1
+        self.meta_interp(f1, [8])
+        self.check_loop_count(1)
+        self.check_loops({'int_sub': 1, 'int_gt': 1, 'guard_true': 1,
+                          'jump': 1})
+
 
 class TestLLWarmspot(WarmspotTests, LLJitMixin):
     CPUClass = runner.LLtypeCPU

Modified: pypy/trunk/pypy/jit/metainterp/warmspot.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/warmspot.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/warmspot.py	Mon Oct 18 17:24:50 2010
@@ -115,10 +115,10 @@
     return results
 
 def find_can_enter_jit(graphs):
-    results = _find_jit_marker(graphs, 'can_enter_jit')
-    if not results:
-        raise Exception("no can_enter_jit found!")
-    return results
+    return _find_jit_marker(graphs, 'can_enter_jit')
+
+def find_loop_headers(graphs):
+    return _find_jit_marker(graphs, 'loop_header')
 
 def find_jit_merge_points(graphs):
     results = _find_jit_marker(graphs, 'jit_merge_point')
@@ -211,9 +211,9 @@
                 "there are multiple jit_merge_points with the same jitdriver"
 
     def split_graph_and_record_jitdriver(self, graph, block, pos):
-        jd = JitDriverStaticData()
-        jd._jit_merge_point_pos = (graph, block, pos)
         op = block.operations[pos]
+        jd = JitDriverStaticData()
+        jd._jit_merge_point_pos = (graph, op)
         args = op.args[2:]
         s_binding = self.translator.annotator.binding
         jd._portal_args_s = [s_binding(v) for v in args]
@@ -286,10 +286,20 @@
     def make_virtualizable_infos(self):
         vinfos = {}
         for jd in self.jitdrivers_sd:
+            #
+            jd.greenfield_info = None
+            for name in jd.jitdriver.greens:
+                if '.' in name:
+                    from pypy.jit.metainterp.greenfield import GreenFieldInfo
+                    jd.greenfield_info = GreenFieldInfo(self.cpu, jd)
+                    break
+            #
             if not jd.jitdriver.virtualizables:
                 jd.virtualizable_info = None
                 jd.index_of_virtualizable = -1
                 continue
+            else:
+                assert jd.greenfield_info is None, "XXX not supported yet"
             #
             jitdriver = jd.jitdriver
             assert len(jitdriver.virtualizables) == 1    # for now
@@ -457,8 +467,7 @@
             self.make_args_specification(jd)
 
     def make_args_specification(self, jd):
-        graph, block, index = jd._jit_merge_point_pos
-        op = block.operations[index]
+        graph, op = jd._jit_merge_point_pos
         greens_v, reds_v = support.decode_hp_hint_args(op)
         ALLARGS = [v.concretetype for v in (greens_v + reds_v)]
         jd._green_args_spec = [v.concretetype for v in greens_v]
@@ -474,26 +483,37 @@
             [lltype.Signed, llmemory.GCREF], RESTYPE)
 
     def rewrite_can_enter_jits(self):
-        can_enter_jits = find_can_enter_jit(self.translator.graphs)
         sublists = {}
         for jd in self.jitdrivers_sd:
-            sublists[jd.jitdriver] = []
+            sublists[jd.jitdriver] = jd, []
+            jd.no_loop_header = True
+        #
+        loop_headers = find_loop_headers(self.translator.graphs)
+        for graph, block, index in loop_headers:
+            op = block.operations[index]
+            jitdriver = op.args[1].value
+            assert jitdriver in sublists, \
+                   "loop_header with no matching jit_merge_point"
+            jd, sublist = sublists[jitdriver]
+            jd.no_loop_header = False
+        #
+        can_enter_jits = find_can_enter_jit(self.translator.graphs)
         for graph, block, index in can_enter_jits:
             op = block.operations[index]
             jitdriver = op.args[1].value
             assert jitdriver in sublists, \
                    "can_enter_jit with no matching jit_merge_point"
+            jd, sublist = sublists[jitdriver]
             origportalgraph = jd._jit_merge_point_pos[0]
             if graph is not origportalgraph:
-                sublists[jitdriver].append((graph, block, index))
+                sublist.append((graph, block, index))
+                jd.no_loop_header = False
             else:
                 pass   # a 'can_enter_jit' before the 'jit-merge_point', but
                        # originally in the same function: we ignore it here
                        # see e.g. test_jitdriver.test_simple
         for jd in self.jitdrivers_sd:
-            sublist = sublists[jd.jitdriver]
-            assert len(sublist) > 0, \
-                   "found no can_enter_jit for %r" % (jd.jitdriver,)
+            _, sublist = sublists[jd.jitdriver]
             self.rewrite_can_enter_jit(jd, sublist)
 
     def rewrite_can_enter_jit(self, jd, can_enter_jits):
@@ -501,6 +521,19 @@
         FUNCPTR = jd._PTR_JIT_ENTER_FUNCTYPE
         jit_enter_fnptr = self.helper_func(FUNCPTR, jd._maybe_enter_jit_fn)
 
+        if len(can_enter_jits) == 0:
+            # see test_warmspot.test_no_loop_at_all
+            operations = jd.portal_graph.startblock.operations
+            op1 = operations[0]
+            assert (op1.opname == 'jit_marker' and
+                    op1.args[0].value == 'jit_merge_point')
+            op0 = SpaceOperation(
+                'jit_marker',
+                [Constant('can_enter_jit', lltype.Void)] + op1.args[1:],
+                None)
+            operations.insert(0, op0)
+            can_enter_jits = [(jd.portal_graph, jd.portal_graph.startblock, 0)]
+
         for graph, block, index in can_enter_jits:
             if graph is jd._jit_merge_point_pos[0]:
                 continue
@@ -709,8 +742,14 @@
         # ____________________________________________________________
         # Now mutate origportalgraph to end with a call to portal_runner_ptr
         #
-        _, origblock, origindex = jd._jit_merge_point_pos
-        op = origblock.operations[origindex]
+        _, op = jd._jit_merge_point_pos
+        for origblock in origportalgraph.iterblocks():
+            if op in origblock.operations:
+                break
+        else:
+            assert False, "lost the operation %r in the graph %r" % (
+                op, origportalgraph)
+        origindex = origblock.operations.index(op)
         assert op.opname == 'jit_marker'
         assert op.args[0].value == 'jit_merge_point'
         greens_v, reds_v = support.decode_hp_hint_args(op)

Modified: pypy/trunk/pypy/rlib/jit.py
==============================================================================
--- pypy/trunk/pypy/rlib/jit.py	(original)
+++ pypy/trunk/pypy/rlib/jit.py	Mon Oct 18 17:24:50 2010
@@ -145,6 +145,24 @@
         return hop.inputconst(lltype.Signed, _we_are_jitted)
 
 
+def jit_debug(string, arg1=-sys.maxint-1, arg2=-sys.maxint-1,
+                      arg3=-sys.maxint-1, arg4=-sys.maxint-1):
+    """When JITted, cause an extra operation DEBUG_MERGE_POINT to appear in
+    the graphs.  Should not be left after debugging."""
+    keepalive_until_here(string) # otherwise the whole function call is removed
+jit_debug.oopspec = 'jit.debug(string, arg1, arg2, arg3, arg4)'
+
+def assert_green(value):
+    """Very strong assert: checks that 'value' is a green
+    (a JIT compile-time constant)."""
+    keepalive_until_here(value)
+assert_green._annspecialcase_ = 'specialize:argtype(0)'
+assert_green.oopspec = 'jit.assert_green(value)'
+
+class AssertGreenFailed(Exception):
+    pass
+
+
 ##def force_virtualizable(virtualizable):
 ##    pass
 
@@ -272,7 +290,8 @@
             self.virtualizables = virtualizables
         for v in self.virtualizables:
             assert v in self.reds
-        self._alllivevars = dict.fromkeys(self.greens + self.reds)
+        self._alllivevars = dict.fromkeys(
+            [name for name in self.greens + self.reds if '.' not in name])
         self._make_extregistryentries()
         self.get_jitcell_at = get_jitcell_at
         self.set_jitcell_at = set_jitcell_at
@@ -364,7 +383,8 @@
         driver = self.instance.im_self
         keys = kwds_s.keys()
         keys.sort()
-        expected = ['s_' + name for name in driver.greens + driver.reds]
+        expected = ['s_' + name for name in driver.greens + driver.reds
+                                if '.' not in name]
         expected.sort()
         if keys != expected:
             raise JitHintError("%s expects the following keyword "
@@ -409,7 +429,13 @@
         uniquekey = 'jitdriver.%s' % func.func_name
         args_s = args_s[:]
         for name in variables:
-            s_arg = kwds_s['s_' + name]
+            if '.' not in name:
+                s_arg = kwds_s['s_' + name]
+            else:
+                objname, fieldname = name.split('.')
+                s_instance = kwds_s['s_' + objname]
+                s_arg = s_instance.classdef.about_attribute(fieldname)
+                assert s_arg is not None
             args_s.append(s_arg)
         bk.emulate_pbc_call(uniquekey, s_func, args_s)
 
@@ -422,9 +448,42 @@
         greens_v = []
         reds_v = []
         for name in driver.greens:
-            i = kwds_i['i_' + name]
-            r_green = hop.args_r[i]
-            v_green = hop.inputarg(r_green, arg=i)
+            if '.' not in name:
+                i = kwds_i['i_' + name]
+                r_green = hop.args_r[i]
+                v_green = hop.inputarg(r_green, arg=i)
+            else:
+                if hop.rtyper.type_system.name == 'ootypesystem':
+                    py.test.skip("lltype only")
+                objname, fieldname = name.split('.')   # see test_green_field
+                assert objname in driver.reds
+                i = kwds_i['i_' + objname]
+                s_red = hop.args_s[i]
+                r_red = hop.args_r[i]
+                while True:
+                    try:
+                        mangled_name, r_field = r_red._get_field(fieldname)
+                        break
+                    except KeyError:
+                        pass
+                    assert r_red.rbase is not None, (
+                        "field %r not found in %r" % (name,
+                                                      r_red.lowleveltype.TO))
+                    r_red = r_red.rbase
+                GTYPE = r_red.lowleveltype.TO
+                assert GTYPE._immutable_field(mangled_name), (
+                    "field %r must be declared as immutable" % name)
+                if not hasattr(driver, 'll_greenfields'):
+                    driver.ll_greenfields = {}
+                driver.ll_greenfields[name] = GTYPE, mangled_name
+                #
+                v_red = hop.inputarg(r_red, arg=i)
+                c_llname = hop.inputconst(lltype.Void, mangled_name)
+                v_green = hop.genop('getfield', [v_red, c_llname],
+                                    resulttype = r_field)
+                s_green = s_red.classdef.about_attribute(fieldname)
+                assert s_green is not None
+                hop.rtyper.annotator.setbinding(v_green, s_green)
             greens_v.append(v_green)
         for name in driver.reds:
             i = kwds_i['i_' + name]

Modified: pypy/trunk/pypy/rlib/rsre/rsre_char.py
==============================================================================
--- pypy/trunk/pypy/rlib/rsre/rsre_char.py	(original)
+++ pypy/trunk/pypy/rlib/rsre/rsre_char.py	Mon Oct 18 17:24:50 2010
@@ -4,6 +4,7 @@
 import sys
 from pypy.rlib.rlocale import tolower, isalnum
 from pypy.rlib.unroll import unrolling_iterable
+from pypy.rlib import jit
 
 # Note: the unicode parts of this module require you to call
 # rsre_char.set_unicode_db() first, to select one of the modules
@@ -43,6 +44,7 @@
 # XXX can we import those safely from sre_constants?
 SRE_INFO_PREFIX = 1
 SRE_INFO_LITERAL = 2
+SRE_INFO_CHARSET = 4
 SRE_FLAG_LOCALE = 4 # honour system locale
 SRE_FLAG_UNICODE = 32 # use unicode locale
 OPCODE_INFO = 17
@@ -64,33 +66,27 @@
 
 #### Category helpers
 
-ascii_char_info = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 6, 2,
-2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0,
-0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 25, 25, 25, 25, 25, 25, 25, 25,
-25, 25, 0, 0, 0, 0, 0, 0, 0, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
-24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 0, 0,
-0, 0, 16, 0, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
-24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 0, 0, 0, 0, 0 ]
-
+is_a_word = [(chr(i).isalnum() or chr(i) == '_') for i in range(256)]
 linebreak = ord("\n")
 underline = ord("_")
 
 def is_digit(code):
-    return code < 128 and (ascii_char_info[code] & 1 != 0)
+    return code <= 57 and code >= 48
 
 def is_uni_digit(code):
     assert unicodedb is not None
     return unicodedb.isdigit(code)
 
 def is_space(code):
-    return code < 128 and (ascii_char_info[code] & 2 != 0)
+    return code == 32 or (code <= 13 and code >= 9)
 
 def is_uni_space(code):
     assert unicodedb is not None
     return unicodedb.isspace(code)
 
 def is_word(code):
-    return code < 128 and (ascii_char_info[code] & 16 != 0)
+    assert code >= 0
+    return code < 256 and is_a_word[code]
 
 def is_uni_word(code):
     assert unicodedb is not None
@@ -142,6 +138,7 @@
 SET_OK = -1
 SET_NOT_OK = -2
 
+ at jit.unroll_safe
 def check_charset(pattern, ppos, char_code):
     """Checks whether a character matches set of arbitrary length.
     The set starts at pattern[ppos]."""

Modified: pypy/trunk/pypy/rlib/rsre/rsre_core.py
==============================================================================
--- pypy/trunk/pypy/rlib/rsre/rsre_core.py	(original)
+++ pypy/trunk/pypy/rlib/rsre/rsre_core.py	Mon Oct 18 17:24:50 2010
@@ -4,6 +4,8 @@
 from pypy.rlib.rsre import rsre_char
 from pypy.tool.sourcetools import func_with_new_name
 from pypy.rlib.objectmodel import we_are_translated
+from pypy.rlib import jit
+from pypy.rlib.rsre.rsre_jit import install_jitdriver, install_jitdriver_spec
 
 
 OPCODE_FAILURE            = 0
@@ -56,16 +58,19 @@
     _seen_specname[specname] = True
     # Install a copy of the function under the name '_spec_funcname' in each
     # concrete subclass
+    specialized_methods = []
     for prefix, concreteclass in [('str', StrMatchContext),
                                   ('uni', UnicodeMatchContext)]:
         newfunc = func_with_new_name(func, prefix + specname)
         assert not hasattr(concreteclass, specname)
         setattr(concreteclass, specname, newfunc)
+        specialized_methods.append(newfunc)
     # Return a dispatcher function, specialized on the exact type of 'ctx'
     def dispatch(ctx, *args):
         return getattr(ctx, specname)(*args)
     dispatch._annspecialcase_ = 'specialize:argtype(0)'
-    return dispatch
+    dispatch._specialized_methods_ = specialized_methods
+    return func_with_new_name(dispatch, specname)
 
 # ____________________________________________________________
 
@@ -75,6 +80,7 @@
 
 class AbstractMatchContext(object):
     """Abstract base class"""
+    _immutable_fields_ = ['pattern[*]', 'flags', 'end']
     match_start = 0
     match_end = 0
     match_marks = None
@@ -238,8 +244,9 @@
         self.start_ptr = ptr
         self.start_marks = marks
 
+    @jit.unroll_safe
     def find_first_result(self, ctx):
-        ppos = self.ppos
+        ppos = jit.hint(self.ppos, promote=True)
         while ctx.pat(ppos):
             result = sre_match(ctx, ppos + 1, self.start_ptr, self.start_marks)
             ppos += ctx.pat(ppos)
@@ -250,6 +257,10 @@
     find_next_result = find_first_result
 
 class RepeatOneMatchResult(MatchResult):
+    install_jitdriver('RepeatOne',
+                      greens=['nextppos', 'ctx.pattern'],
+                      reds=['ptr', 'self', 'ctx'],
+                      debugprint=(1, 0))   # indices in 'greens'
 
     def __init__(self, nextppos, minptr, ptr, marks):
         self.nextppos = nextppos
@@ -259,8 +270,11 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
+        nextppos = self.nextppos
         while ptr >= self.minptr:
-            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
+            ctx.jitdriver_RepeatOne.jit_merge_point(
+                self=self, ptr=ptr, ctx=ctx, nextppos=nextppos)
+            result = sre_match(ctx, nextppos, ptr, self.start_marks)
             ptr -= 1
             if result is not None:
                 self.subresult = result
@@ -270,6 +284,10 @@
 
 
 class MinRepeatOneMatchResult(MatchResult):
+    install_jitdriver('MinRepeatOne',
+                      greens=['nextppos', 'ppos3', 'ctx.pattern'],
+                      reds=['ptr', 'self', 'ctx'],
+                      debugprint=(2, 0))   # indices in 'greens'
 
     def __init__(self, nextppos, ppos3, maxptr, ptr, marks):
         self.nextppos = nextppos
@@ -280,29 +298,32 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
+        nextppos = self.nextppos
+        ppos3 = self.ppos3
         while ptr <= self.maxptr:
-            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
+            ctx.jitdriver_MinRepeatOne.jit_merge_point(
+                self=self, ptr=ptr, ctx=ctx, nextppos=nextppos, ppos3=ppos3)
+            result = sre_match(ctx, nextppos, ptr, self.start_marks)
             if result is not None:
                 self.subresult = result
                 self.start_ptr = ptr
                 return self
-            if not self.next_char_ok(ctx, ptr):
+            if not self.next_char_ok(ctx, ptr, ppos3):
                 break
             ptr += 1
 
     def find_next_result(self, ctx):
         ptr = self.start_ptr
-        if not self.next_char_ok(ctx, ptr):
+        if not self.next_char_ok(ctx, ptr, self.ppos3):
             return
         self.start_ptr = ptr + 1
         return self.find_first_result(ctx)
 
-    def next_char_ok(self, ctx, ptr):
+    def next_char_ok(self, ctx, ptr, ppos):
         if ptr == ctx.end:
             return False
-        ppos = self.ppos3
         op = ctx.pat(ppos)
-        for op1, (checkerfn, _) in unroll_char_checker:
+        for op1, checkerfn in unroll_char_checker:
             if op1 == op:
                 return checkerfn(ctx, ptr, ppos)
         raise Error("next_char_ok[%d]" % op)
@@ -325,41 +346,34 @@
         self.next = next     # chained list
 
 class MaxUntilMatchResult(AbstractUntilMatchResult):
+    install_jitdriver('MaxUntil',
+                      greens=['ppos', 'tailppos', 'match_more', 'ctx.pattern'],
+                      reds=['ptr', 'marks', 'self', 'ctx'],
+                      debugprint=(3, 0, 2))
 
     def find_first_result(self, ctx):
-        enum = sre_match(ctx, self.ppos + 3, self.cur_ptr, self.cur_marks)
-        return self.search_next(ctx, enum, resume=False)
+        return self.search_next(ctx, match_more=True)
 
     def find_next_result(self, ctx):
-        return self.search_next(ctx, None, resume=True)
+        return self.search_next(ctx, match_more=False)
 
-    def search_next(self, ctx, enum, resume):
+    def search_next(self, ctx, match_more):
         ppos = self.ppos
-        min = ctx.pat(ppos+1)
-        max = ctx.pat(ppos+2)
+        tailppos = self.tailppos
         ptr = self.cur_ptr
         marks = self.cur_marks
         while True:
-            while True:
-                if (enum is not None and
-                    (ptr != ctx.match_end or self.num_pending < min)):
-                    #               ^^^^^^^^^^ zero-width match protection
-                    # matched one more 'item'.  record it and continue.
-                    self.pending = Pending(ptr, marks, enum, self.pending)
-                    self.num_pending += 1
-                    ptr = ctx.match_end
-                    marks = ctx.match_marks
-                    break
-                # 'item' no longer matches.
-                if not resume and self.num_pending >= min:
-                    # try to match 'tail' if we have enough 'item'
-                    result = sre_match(ctx, self.tailppos, ptr, marks)
-                    if result is not None:
-                        self.subresult = result
-                        self.cur_ptr = ptr
-                        self.cur_marks = marks
-                        return self
-                resume = False
+            ctx.jitdriver_MaxUntil.jit_merge_point(
+                ppos=ppos, tailppos=tailppos, match_more=match_more,
+                ptr=ptr, marks=marks, self=self, ctx=ctx)
+            if match_more:
+                max = ctx.pat(ppos+2)
+                if max == 65535 or self.num_pending < max:
+                    # try to match one more 'item'
+                    enum = sre_match(ctx, ppos + 3, ptr, marks)
+                else:
+                    enum = None    # 'max' reached, no more matches
+            else:
                 p = self.pending
                 if p is None:
                     return
@@ -369,11 +383,27 @@
                 marks = p.marks
                 enum = p.enum.move_to_next_result(ctx)
             #
-            if max == 65535 or self.num_pending < max:
-                # try to match one more 'item'
-                enum = sre_match(ctx, ppos + 3, ptr, marks)
+            min = ctx.pat(ppos+1)
+            if (enum is not None and
+                (ptr != ctx.match_end or self.num_pending < min)):
+                #               ^^^^^^^^^^ zero-width match protection
+                # matched one more 'item'.  record it and continue.
+                self.pending = Pending(ptr, marks, enum, self.pending)
+                self.num_pending += 1
+                ptr = ctx.match_end
+                marks = ctx.match_marks
+                match_more = True
             else:
-                enum = None    # 'max' reached, no more matches
+                # 'item' no longer matches.
+                if self.num_pending >= min:
+                    # try to match 'tail' if we have enough 'item'
+                    result = sre_match(ctx, tailppos, ptr, marks)
+                    if result is not None:
+                        self.subresult = result
+                        self.cur_ptr = ptr
+                        self.cur_marks = marks
+                        return self
+                match_more = False
 
 class MinUntilMatchResult(AbstractUntilMatchResult):
 
@@ -384,6 +414,7 @@
         return self.search_next(ctx, resume=True)
 
     def search_next(self, ctx, resume):
+        # XXX missing jit support here
         ppos = self.ppos
         min = ctx.pat(ppos+1)
         max = ctx.pat(ppos+2)
@@ -429,6 +460,7 @@
 # ____________________________________________________________
 
 @specializectx
+ at jit.unroll_safe
 def sre_match(ctx, ppos, ptr, marks):
     """Returns either None or a MatchResult object.  Usually we only need
     the first result, but there is the case of REPEAT...UNTIL where we
@@ -438,6 +470,12 @@
         op = ctx.pat(ppos)
         ppos += 1
 
+        #jit.jit_debug("sre_match", op, ppos, ptr)
+        #
+        # When using the JIT, calls to sre_match() must always have a constant
+        # (green) argument for 'ppos'.  If not, the following assert fails.
+        jit.assert_green(op)
+
         if op == OPCODE_FAILURE:
             return
 
@@ -712,13 +750,23 @@
 @specializectx
 def find_repetition_end(ctx, ppos, ptr, maxcount):
     end = ctx.end
-    # adjust end
-    if maxcount != 65535:
+    if maxcount <= 1:
+        if maxcount == 1 and ptr < end:
+            # Relatively common case: maxcount == 1.  If we are not at the
+            # end of the string, it's done by a single direct check.
+            op = ctx.pat(ppos)
+            for op1, checkerfn in unroll_char_checker:
+                if op1 == op:
+                    if checkerfn(ctx, ptr, ppos):
+                        return ptr + 1
+        return ptr
+    elif maxcount != 65535:
+        # adjust end
         end1 = ptr + maxcount
         if end1 <= end:
             end = end1
     op = ctx.pat(ppos)
-    for op1, (_, fre) in unroll_char_checker:
+    for op1, fre in unroll_fre_checker:
         if op1 == op:
             return fre(ctx, ptr, end, ppos)
     raise Error("rsre.find_repetition_end[%d]" % op)
@@ -751,23 +799,60 @@
     if checkerfn == match_ANY_ALL:
         def fre(ctx, ptr, end, ppos):
             return end
+    elif checkerfn == match_IN:
+        install_jitdriver_spec('MatchIn', 
+                               greens=['ppos', 'ctx.pattern'],
+                               reds=['ptr', 'end', 'ctx'],
+                               debugprint=(1, 0))
+        @specializectx
+        def fre(ctx, ptr, end, ppos):
+            while True:
+                ctx.jitdriver_MatchIn.jit_merge_point(ctx=ctx, ptr=ptr,
+                                                      end=end, ppos=ppos)
+                if ptr < end and checkerfn(ctx, ptr, ppos):
+                    ptr += 1
+                else:
+                    return ptr
+    elif checkerfn == match_IN_IGNORE:
+        install_jitdriver_spec('MatchInIgnore', 
+                               greens=['ppos', 'ctx.pattern'],
+                               reds=['ptr', 'end', 'ctx'],
+                               debugprint=(1, 0))
+        @specializectx
+        def fre(ctx, ptr, end, ppos):
+            while True:
+                ctx.jitdriver_MatchInIgnore.jit_merge_point(ctx=ctx, ptr=ptr,
+                                                            end=end, ppos=ppos)
+                if ptr < end and checkerfn(ctx, ptr, ppos):
+                    ptr += 1
+                else:
+                    return ptr
     else:
+        # in the other cases, the fre() function is not JITted at all
+        # and is present as a residual call.
+        @specializectx
         def fre(ctx, ptr, end, ppos):
             while ptr < end and checkerfn(ctx, ptr, ppos):
                 ptr += 1
             return ptr
-    return checkerfn, fre
+    fre = func_with_new_name(fre, 'fre_' + checkerfn.__name__)
+    return fre
+
+unroll_char_checker = [
+    (OPCODE_ANY,                match_ANY),
+    (OPCODE_ANY_ALL,            match_ANY_ALL),
+    (OPCODE_IN,                 match_IN),
+    (OPCODE_IN_IGNORE,          match_IN_IGNORE),
+    (OPCODE_LITERAL,            match_LITERAL),
+    (OPCODE_LITERAL_IGNORE,     match_LITERAL_IGNORE),
+    (OPCODE_NOT_LITERAL,        match_NOT_LITERAL),
+    (OPCODE_NOT_LITERAL_IGNORE, match_NOT_LITERAL_IGNORE),
+    ]
+unroll_fre_checker = [(_op, _make_fre(_fn))
+                      for (_op, _fn) in unroll_char_checker]
 
-unroll_char_checker = unrolling_iterable([
-    (OPCODE_ANY,                _make_fre(match_ANY)),
-    (OPCODE_ANY_ALL,            _make_fre(match_ANY_ALL)),
-    (OPCODE_IN,                 _make_fre(match_IN)),
-    (OPCODE_IN_IGNORE,          _make_fre(match_IN_IGNORE)),
-    (OPCODE_LITERAL,            _make_fre(match_LITERAL)),
-    (OPCODE_LITERAL_IGNORE,     _make_fre(match_LITERAL_IGNORE)),
-    (OPCODE_NOT_LITERAL,        _make_fre(match_NOT_LITERAL)),
-    (OPCODE_NOT_LITERAL_IGNORE, _make_fre(match_NOT_LITERAL_IGNORE)),
-    ])
+unroll_char_checker = unrolling_iterable(unroll_char_checker)
+unroll_fre_checker  = unrolling_iterable(unroll_fre_checker)
 
 ##### At dispatch
 
@@ -873,74 +958,139 @@
     else:
         return None
 
+install_jitdriver('Match',
+                  greens=['ctx.pattern'], reds=['ctx'],
+                  debugprint=(0,))
+
 def match_context(ctx):
     ctx.original_pos = ctx.match_start
     if ctx.end < ctx.match_start:
         return False
+    ctx.jitdriver_Match.jit_merge_point(ctx=ctx)
     return sre_match(ctx, 0, ctx.match_start, None) is not None
 
 def search_context(ctx):
     ctx.original_pos = ctx.match_start
     if ctx.end < ctx.match_start:
         return False
-    if ctx.pat(0) == OPCODE_INFO:
-        if ctx.pat(2) & rsre_char.SRE_INFO_PREFIX and ctx.pat(5) > 1:
-            return fast_search(ctx)
-    return regular_search(ctx)
+    base = 0
+    charset = False
+    if ctx.pat(base) == OPCODE_INFO:
+        flags = ctx.pat(2)
+        if flags & rsre_char.SRE_INFO_PREFIX:
+            if ctx.pat(5) > 1:
+                return fast_search(ctx)
+        else:
+            charset = (flags & rsre_char.SRE_INFO_CHARSET)
+        base += 1 + ctx.pat(1)
+    if ctx.pat(base) == OPCODE_LITERAL:
+        return literal_search(ctx, base)
+    if charset:
+        return charset_search(ctx, base)
+    return regular_search(ctx, base)
+
+install_jitdriver('RegularSearch',
+                  greens=['base', 'ctx.pattern'],
+                  reds=['start', 'ctx'],
+                  debugprint=(1, 0))
 
-def regular_search(ctx):
+def regular_search(ctx, base):
     start = ctx.match_start
     while start <= ctx.end:
-        if sre_match(ctx, 0, start, None) is not None:
+        ctx.jitdriver_RegularSearch.jit_merge_point(ctx=ctx, start=start,
+                                                    base=base)
+        if sre_match(ctx, base, start, None) is not None:
             ctx.match_start = start
             return True
         start += 1
     return False
 
+install_jitdriver_spec("LiteralSearch",
+                       greens=['base', 'character', 'ctx.pattern'],
+                       reds=['start', 'ctx'],
+                       debugprint=(2, 0, 1))
+ at specializectx
+def literal_search(ctx, base):
+    # pattern starts with a literal character.  this is used
+    # for short prefixes, and if fast search is disabled
+    character = ctx.pat(base + 1)
+    base += 2
+    start = ctx.match_start
+    while start < ctx.end:
+        ctx.jitdriver_LiteralSearch.jit_merge_point(ctx=ctx, start=start,
+                                          base=base, character=character)
+        if ctx.str(start) == character:
+            if sre_match(ctx, base, start + 1, None) is not None:
+                ctx.match_start = start
+                return True
+        start += 1
+    return False
+
+install_jitdriver_spec("CharsetSearch",
+                       greens=['base', 'ctx.pattern'],
+                       reds=['start', 'ctx'],
+                       debugprint=(1, 0))
+ at specializectx
+def charset_search(ctx, base):
+    # pattern starts with a character from a known set
+    start = ctx.match_start
+    while start < ctx.end:
+        ctx.jitdriver_CharsetSearch.jit_merge_point(ctx=ctx, start=start,
+                                                    base=base)
+        if rsre_char.check_charset(ctx.pattern, 5, ctx.str(start)):
+            if sre_match(ctx, base, start, None) is not None:
+                ctx.match_start = start
+                return True
+        start += 1
+    return False
+
+install_jitdriver_spec('FastSearch',
+                       greens=['i', 'prefix_len', 'ctx.pattern'],
+                       reds=['string_position', 'ctx'],
+                       debugprint=(2, 0))
 @specializectx
 def fast_search(ctx):
     # skips forward in a string as fast as possible using information from
     # an optimization info block
     #  <1=skip> <2=flags> <3=min> <4=...>
     #        <5=length> <6=skip> <7=prefix data> 
-    flags = ctx.pat(2)
+    string_position = ctx.match_start
+    if string_position >= ctx.end:
+        return False
     prefix_len = ctx.pat(5)
     assert prefix_len >= 0
-    prefix_skip = ctx.pat(6)
-    assert prefix_skip >= 0
-    overlap_offset = 7 + prefix_len - 1
-    assert overlap_offset >= 0
-    pattern_offset = ctx.pat(1) + 1
-    ppos_start = pattern_offset + 2 * prefix_skip
-    assert ppos_start >= 0
     i = 0
-    string_position = ctx.match_start
-    end = ctx.end
-    while string_position < end:
-        while True:
-            char_ord = ctx.str(string_position)
-            if char_ord != ctx.pat(7 + i):
-                if i == 0:
-                    break
-                else:
-                    i = ctx.pat(overlap_offset + i)
-            else:
-                i += 1
-                if i == prefix_len:
-                    # found a potential match
-                    start = string_position + 1 - prefix_len
-                    assert start >= 0
-                    ptr = start + prefix_skip
-                    if flags & rsre_char.SRE_INFO_LITERAL:
-                        # matched all of pure literal pattern
-                        ctx.match_start = start
-                        ctx.match_end = ptr
-                        ctx.match_marks = None
-                        return True
-                    if sre_match(ctx, ppos_start, ptr, None) is not None:
-                        ctx.match_start = start
-                        return True
-                    i = ctx.pat(overlap_offset + i)
-                break
+    while True:
+        ctx.jitdriver_FastSearch.jit_merge_point(ctx=ctx,
+                string_position=string_position, i=i, prefix_len=prefix_len)
+        char_ord = ctx.str(string_position)
+        if char_ord != ctx.pat(7 + i):
+            if i > 0:
+                overlap_offset = prefix_len + (7 - 1)
+                i = ctx.pat(overlap_offset + i)
+                continue
+        else:
+            i += 1
+            if i == prefix_len:
+                # found a potential match
+                start = string_position + 1 - prefix_len
+                assert start >= 0
+                prefix_skip = ctx.pat(6)
+                ptr = start + prefix_skip
+                #flags = ctx.pat(2)
+                #if flags & rsre_char.SRE_INFO_LITERAL:
+                #    # matched all of pure literal pattern
+                #    ctx.match_start = start
+                #    ctx.match_end = ptr
+                #    ctx.match_marks = None
+                #    return True
+                pattern_offset = ctx.pat(1) + 1
+                ppos_start = pattern_offset + 2 * prefix_skip
+                if sre_match(ctx, ppos_start, ptr, None) is not None:
+                    ctx.match_start = start
+                    return True
+                overlap_offset = prefix_len + (7 - 1)
+                i = ctx.pat(overlap_offset + i)
         string_position += 1
-    return False
+        if string_position >= ctx.end:
+            return False

Modified: pypy/trunk/pypy/rlib/test/test_jit.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_jit.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_jit.py	Mon Oct 18 17:24:50 2010
@@ -1,4 +1,5 @@
 import py
+from pypy.conftest import option
 from pypy.rlib.jit import hint, we_are_jitted, JitDriver, purefunction_promote
 from pypy.rlib.jit import JitHintError, oopspec
 from pypy.translator.translator import TranslationContext, graphof
@@ -110,6 +111,26 @@
             return n
         py.test.raises(JitHintError, self.gengraph, fn, [int])
 
+    def test_green_field(self):
+        def get_printable_location(xfoo):
+            return str(ord(xfoo))   # xfoo must be annotated as a character
+        myjitdriver = JitDriver(greens=['x.foo'], reds=['n', 'x'],
+                                get_printable_location=get_printable_location)
+        class A(object):
+            _immutable_fields_ = ['foo']
+        def fn(n):
+            x = A()
+            x.foo = chr(n)
+            while n > 0:
+                myjitdriver.can_enter_jit(x=x, n=n)
+                myjitdriver.jit_merge_point(x=x, n=n)
+                n -= 1
+            return n
+        t = self.gengraph(fn, [int])[0]
+        if option.view:
+            t.view()
+        # assert did not raise
+
 
 class TestJITLLtype(BaseTestJIT, LLRtypeMixin):
     pass


From arigo at codespeak.net  Mon Oct 18 17:25:39 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 17:25:39 +0200 (CEST)
Subject: [pypy-svn] r78050 - pypy/branch/rsre-jit
Message-ID: <20101018152539.9E304282B9D@codespeak.net>

Author: arigo
Date: Mon Oct 18 17:25:38 2010
New Revision: 78050

Removed:
   pypy/branch/rsre-jit/
Log:
Remove merged branch.



From cfbolz at codespeak.net  Mon Oct 18 17:32:19 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Mon, 18 Oct 2010 17:32:19 +0200 (CEST)
Subject: [pypy-svn] r78051 - pypy/extradoc/talk/pepm2011
Message-ID: <20101018153219.70A44282B9D@codespeak.net>

Author: cfbolz
Date: Mon Oct 18 17:32:17 2010
New Revision: 78051

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
small things, more fundamental changes to come


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Mon Oct 18 17:32:17 2010
@@ -557,20 +557,19 @@
 \section{Allocation Removal in Traces}
 \label{sec:statics}
 
-\subsection{Static Objects}
-
 The main insight to improve the code shown in the last section is that objects
-in category 1 don't survive very long -- they are used only inside the loop and
-nobody else in the program stores a reference to them. The idea for improving
+in category 1 do not survive very long -- they are used only inside the loop and
+there is no other outside reference to them. The idea for improving
 the code is to analyze which objects fall in category 1 and therefore do
 not have to be allocated at all.
 
-This is a process that is usually called \emph{escape analysis}. In this paper we will
-perform escape analysis by using partial evaluation. The partial evalution is a
+This is a process that is usually called \emph{escape analysis}
+\cite{goldberg_higher_1990}. In this paper we will
+perform escape analysis by using partial evaluation. The use of partial evaluation is a
 bit peculiar in that it receives no static input arguments for the trace,
-but it is only used to optimize operations within a trace.
+but it is only used to optimize operations within the trace.
 
-The partial evaluation works by traversing the trace from beginning to end.
+To optimize the trace, it is traversed from beginning to end.
 Whenever a \lstinline{new} operation is seen, the operation is removed and a \emph{static
 object}\footnote{Here ``static'' is meant in the sense of partial
 evaluation, \ie known at partial evaluation time, not in the sense of static


From cfbolz at codespeak.net  Mon Oct 18 17:39:03 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Mon, 18 Oct 2010 17:39:03 +0200 (CEST)
Subject: [pypy-svn] r78052 - pypy/extradoc/talk/pepm2011
Message-ID: <20101018153903.EE018282B9D@codespeak.net>

Author: cfbolz
Date: Mon Oct 18 17:39:02 2010
New Revision: 78052

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
stop using the graphic for the traces


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Mon Oct 18 17:39:02 2010
@@ -664,7 +664,29 @@
 and \lstinline{guard_class} operations).
 
 \begin{figure}
-\includegraphics{figures/step1.pdf}
+\begin{lstlisting}[mathescape]
+# arguments to the trace: $p_{0}$, $p_{1}$
+guard_class($p_1$, BoxedInteger)
+$i_2$ = getfield($p_1$, intval)
+guard_class($p_0$, BoxedInteger)
+$i_3$ = getfield($p_0$, intval)
+$i_4$ = int_add($i_2$, $i_3$)
+$i_9$ = int_add($i_4$, -100)
+
+guard_class($p_0$, BoxedInteger)
+$i_{12}$ = getfield($p_0$, intval)
+$i_{14}$ = int_add($i_{12}$, -1)
+
+$i_{17}$ = int_gt($i_{14}$, 0)
+guard_true($i_{17}$)
+
+$p_{15}$ = new(BoxedInteger)
+setfield($p_{15}$, intval, $i_{14}$)
+$p_{10}$ = new(BoxedInteger)
+setfield($p_{10}$, intval, $i_9$)
+
+jump($p_{15}$, $p_{10}$)
+\end{lstlisting}
 \caption{Resulting Trace After Allocation Removal}
 \label{fig:step1}
 \end{figure}


From arigo at codespeak.net  Mon Oct 18 17:39:11 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 17:39:11 +0200 (CEST)
Subject: [pypy-svn] r78053 - pypy/trunk/pypy/jit/backend/x86/test
Message-ID: <20101018153911.2A5F8282BAD@codespeak.net>

Author: arigo
Date: Mon Oct 18 17:39:09 2010
New Revision: 78053

Modified:
   pypy/trunk/pypy/jit/backend/x86/test/test_runner.py
Log:
Write a (failing) test about calling C functions that return
a shorter integer than a word.


Modified: pypy/trunk/pypy/jit/backend/x86/test/test_runner.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/test/test_runner.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/test/test_runner.py	Mon Oct 18 17:39:09 2010
@@ -12,6 +12,7 @@
 from pypy.jit.backend.test.runner_test import LLtypeBackendTest
 from pypy.jit.metainterp.test.oparser import parse
 from pypy.tool.udir import udir
+from pypy.rlib.rarithmetic import intmask
 import ctypes
 import sys
 import os
@@ -396,6 +397,38 @@
         res = self.execute_operation(rop.CALL, [funcbox, ConstFloat(1.5), ConstFloat(2.5)], 'float', descr=calldescr)
         assert res.value == 4.0
 
+    def test_result_of_call(self):
+        # Test that calling a function that returns a CHAR, SHORT or INT,
+        # signed or unsigned, properly gets zero-extended or sign-extended.
+        from pypy.translator.tool.cbuild import ExternalCompilationInfo
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            # Tested with a function that intentionally does not cast the
+            # result to RESTYPE, but makes sure that we return the whole
+            # value in eax or rax.
+            eci = ExternalCompilationInfo(separate_module_sources=["""
+                long fn_test_result_of_call(long x)
+                {
+                    return x + 1;
+                }
+            """])
+            f = rffi.llexternal('fn_test_result_of_call', [lltype.Signed],
+                                RESTYPE, compilation_info=eci, _nowrapper=True)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value + 1))
+            assert f(value) == expected
+            #
+            FUNC = self.FuncType([lltype.Signed], RESTYPE)
+            FPTR = self.Ptr(FUNC)
+            calldescr = self.cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
+            funcbox = self.get_funcbox(self.cpu, f)
+            res = self.execute_operation(rop.CALL, [funcbox, BoxInt(value)],
+                                         'int', descr=calldescr)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
 
 class TestX86OverflowMC(TestX86):
 


From antocuni at codespeak.net  Mon Oct 18 17:40:22 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Mon, 18 Oct 2010 17:40:22 +0200 (CEST)
Subject: [pypy-svn] r78054 - pypy/branch/jitffi
Message-ID: <20101018154022.037F1282B90@codespeak.net>

Author: antocuni
Date: Mon Oct 18 17:40:21 2010
New Revision: 78054

Removed:
   pypy/branch/jitffi/
Log:
remove merged branch



From cfbolz at codespeak.net  Mon Oct 18 18:03:57 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Mon, 18 Oct 2010 18:03:57 +0200 (CEST)
Subject: [pypy-svn] r78055 - pypy/extradoc/talk/pepm2011
Message-ID: <20101018160357.0943F282B90@codespeak.net>

Author: cfbolz
Date: Mon Oct 18 18:03:56 2010
New Revision: 78055

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
gaaaah


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Mon Oct 18 18:03:56 2010
@@ -667,23 +667,23 @@
 \begin{lstlisting}[mathescape]
 # arguments to the trace: $p_{0}$, $p_{1}$
 guard_class($p_1$, BoxedInteger)
-$i_2$ = getfield($p_1$, intval)
+$i_2$ = get($p_1$, intval)
 guard_class($p_0$, BoxedInteger)
-$i_3$ = getfield($p_0$, intval)
+$i_3$ = get($p_0$, intval)
 $i_4$ = int_add($i_2$, $i_3$)
 $i_9$ = int_add($i_4$, -100)
 
 guard_class($p_0$, BoxedInteger)
-$i_{12}$ = getfield($p_0$, intval)
+$i_{12}$ = get($p_0$, intval)
 $i_{14}$ = int_add($i_{12}$, -1)
 
 $i_{17}$ = int_gt($i_{14}$, 0)
 guard_true($i_{17}$)
 
 $p_{15}$ = new(BoxedInteger)
-setfield($p_{15}$, intval, $i_{14}$)
+set($p_{15}$, intval, $i_{14}$)
 $p_{10}$ = new(BoxedInteger)
-setfield($p_{10}$, intval, $i_9$)
+set($p_{10}$, intval, $i_9$)
 
 jump($p_{15}$, $p_{10}$)
 \end{lstlisting}


From arigo at codespeak.net  Mon Oct 18 18:16:05 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 18:16:05 +0200 (CEST)
Subject: [pypy-svn] r78056 - pypy/trunk/pypy/jit/backend/x86/test
Message-ID: <20101018161605.30A14282B90@codespeak.net>

Author: arigo
Date: Mon Oct 18 18:16:03 2010
New Revision: 78056

Modified:
   pypy/trunk/pypy/jit/backend/x86/test/test_runner.py
Log:
Revert the test.  Will be moved to its own branch.


Modified: pypy/trunk/pypy/jit/backend/x86/test/test_runner.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/test/test_runner.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/test/test_runner.py	Mon Oct 18 18:16:03 2010
@@ -12,7 +12,6 @@
 from pypy.jit.backend.test.runner_test import LLtypeBackendTest
 from pypy.jit.metainterp.test.oparser import parse
 from pypy.tool.udir import udir
-from pypy.rlib.rarithmetic import intmask
 import ctypes
 import sys
 import os
@@ -397,38 +396,6 @@
         res = self.execute_operation(rop.CALL, [funcbox, ConstFloat(1.5), ConstFloat(2.5)], 'float', descr=calldescr)
         assert res.value == 4.0
 
-    def test_result_of_call(self):
-        # Test that calling a function that returns a CHAR, SHORT or INT,
-        # signed or unsigned, properly gets zero-extended or sign-extended.
-        from pypy.translator.tool.cbuild import ExternalCompilationInfo
-        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
-                        rffi.SHORT, rffi.USHORT,
-                        rffi.INT, rffi.UINT,
-                        rffi.LONG, rffi.ULONG]:
-            # Tested with a function that intentionally does not cast the
-            # result to RESTYPE, but makes sure that we return the whole
-            # value in eax or rax.
-            eci = ExternalCompilationInfo(separate_module_sources=["""
-                long fn_test_result_of_call(long x)
-                {
-                    return x + 1;
-                }
-            """])
-            f = rffi.llexternal('fn_test_result_of_call', [lltype.Signed],
-                                RESTYPE, compilation_info=eci, _nowrapper=True)
-            value = intmask(0xFFEEDDCCBBAA9988)
-            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value + 1))
-            assert f(value) == expected
-            #
-            FUNC = self.FuncType([lltype.Signed], RESTYPE)
-            FPTR = self.Ptr(FUNC)
-            calldescr = self.cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
-            funcbox = self.get_funcbox(self.cpu, f)
-            res = self.execute_operation(rop.CALL, [funcbox, BoxInt(value)],
-                                         'int', descr=calldescr)
-            assert res.value == expected, (
-                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
-
 
 class TestX86OverflowMC(TestX86):
 


From arigo at codespeak.net  Mon Oct 18 18:16:56 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 18:16:56 +0200 (CEST)
Subject: [pypy-svn] r78057 - pypy/branch/jit-sign-descr
Message-ID: <20101018161656.CCE67282B90@codespeak.net>

Author: arigo
Date: Mon Oct 18 18:16:55 2010
New Revision: 78057

Added:
   pypy/branch/jit-sign-descr/
      - copied from r78056, pypy/trunk/
Log:
We are missing a flag in all int-supporting descrs to know if the type
is signed or unsigned.



From cfbolz at codespeak.net  Mon Oct 18 18:20:33 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Mon, 18 Oct 2010 18:20:33 +0200 (CEST)
Subject: [pypy-svn] r78058 - pypy/extradoc/talk/pepm2011
Message-ID: <20101018162033.4F8EC282B90@codespeak.net>

Author: cfbolz
Date: Mon Oct 18 18:20:31 2010
New Revision: 78058

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
more nonsense


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Mon Oct 18 18:20:31 2010
@@ -610,8 +610,8 @@
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 guard_class($p_{5}$, BoxedInteger)
-$i_{7}$ = get($p_{5}$, intval)
 # inside BoxedInteger.add
+$i_{7}$ = get($p_{5}$, intval)
 guard_class($p_{6}$, BoxedInteger)
 # inside BoxedInteger.add__int
 $i_{8}$ = get($p_{6}$, intval)


From arigo at codespeak.net  Mon Oct 18 18:23:09 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 18:23:09 +0200 (CEST)
Subject: [pypy-svn] r78059 - pypy/branch/jit-sign-descr/pypy/jit/backend/test
Message-ID: <20101018162309.80EA0282B90@codespeak.net>

Author: arigo
Date: Mon Oct 18 18:23:08 2010
New Revision: 78059

Modified:
   pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py
Log:
Re-check-in this test, this time as a general backend test.


Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py	Mon Oct 18 18:23:08 2010
@@ -15,6 +15,7 @@
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.llinterp import LLException
 from pypy.jit.codewriter import heaptracker
+from pypy.rlib.rarithmetic import intmask
 
 
 class Runner(object):
@@ -2027,6 +2028,38 @@
         assert self.cpu.get_latest_value_float(0) == 13.5
         assert called
 
+    def test_result_of_call(self):
+        # Test that calling a function that returns a CHAR, SHORT or INT,
+        # signed or unsigned, properly gets zero-extended or sign-extended.
+        from pypy.translator.tool.cbuild import ExternalCompilationInfo
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            # Tested with a function that intentionally does not cast the
+            # result to RESTYPE, but makes sure that we return the whole
+            # value in eax or rax.
+            eci = ExternalCompilationInfo(separate_module_sources=["""
+                long fn_test_result_of_call(long x)
+                {
+                    return x + 1;
+                }
+            """])
+            f = rffi.llexternal('fn_test_result_of_call', [lltype.Signed],
+                                RESTYPE, compilation_info=eci, _nowrapper=True)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value + 1))
+            assert intmask(f(value)) == expected
+            #
+            FUNC = self.FuncType([lltype.Signed], RESTYPE)
+            FPTR = self.Ptr(FUNC)
+            calldescr = self.cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
+            funcbox = self.get_funcbox(self.cpu, f)
+            res = self.execute_operation(rop.CALL, [funcbox, BoxInt(value)],
+                                         'int', descr=calldescr)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
 
 class OOtypeBackendTest(BaseBackendTest):
 


From arigo at codespeak.net  Mon Oct 18 18:41:34 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 18:41:34 +0200 (CEST)
Subject: [pypy-svn] r78060 - pypy/branch/jit-sign-descr/pypy/jit/backend/test
Message-ID: <20101018164134.647B6282B9D@codespeak.net>

Author: arigo
Date: Mon Oct 18 18:41:30 2010
New Revision: 78060

Modified:
   pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py
Log:
More tests, trying to cover all descrs, both in "directly executed by
bh_xxx" mode and in "rop.XXX" operation mode.



Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py	Mon Oct 18 18:41:30 2010
@@ -2028,7 +2028,165 @@
         assert self.cpu.get_latest_value_float(0) == 13.5
         assert called
 
-    def test_result_of_call(self):
+    def test_short_result_of_getfield_direct(self):
+        # Test that a getfield that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            S = lltype.GcStruct('S', ('x', RESTYPE))
+            descrfld_x = cpu.fielddescrof(S, 'x')
+            s = lltype.malloc(S)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            s.x = rffi.cast(RESTYPE, value)
+            x = cpu.bh_getfield_gc_i(lltype.cast_opaque_ptr(llmemory.GCREF, s),
+                                     descrfld_x)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_getfield_compiled(self):
+        # Test that a getfield that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            S = lltype.GcStruct('S', ('x', RESTYPE))
+            descrfld_x = cpu.fielddescrof(S, 'x')
+            s = lltype.malloc(S)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            s.x = rffi.cast(RESTYPE, value)
+            s_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, s)
+            res = self.execute_operation(rop.GETFIELD_GC, [BoxPtr(s_gcref)],
+                                         'int', descr=descrfld_x)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_getarrayitem_direct(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = lltype.GcArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            x = cpu.bh_getarrayitem_gc_i(
+                descrarray, lltype.cast_opaque_ptr(llmemory.GCREF, a), 3)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_getarrayitem_compiled(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = lltype.GcArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, a)
+            res = self.execute_operation(rop.GETARRAYITEM_GC,
+                                         [BoxPtr(a_gcref), BoxInt(3)],
+                                         'int', descr=descrarray)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
+    def test_short_result_of_getarrayitem_raw_direct(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = rffi.CArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5, flavor='raw')
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_rawint = heaptracker.adr2int(llmemory.cast_ptr_to_adr(a))
+            x = cpu.bh_getarrayitem_raw_i(descrarray, a_rawint, 3)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+            lltype.free(a, flavor='raw')
+
+    def test_short_result_of_getarrayitem_raw_compiled(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = rffi.CArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5, flavor='raw')
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_rawint = heaptracker.adr2int(llmemory.cast_ptr_to_adr(a))
+            res = self.execute_operation(rop.GETARRAYITEM_RAW,
+                                         [BoxInt(a_rawint), BoxInt(3)],
+                                         'int', descr=descrarray)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+            lltype.free(a, flavor='raw')
+
+    def test_short_result_of_call_direct(self):
+        # Test that calling a function that returns a CHAR, SHORT or INT,
+        # signed or unsigned, properly gets zero-extended or sign-extended.
+        from pypy.translator.tool.cbuild import ExternalCompilationInfo
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            # Tested with a function that intentionally does not cast the
+            # result to RESTYPE, but makes sure that we return the whole
+            # value in eax or rax.
+            eci = ExternalCompilationInfo(separate_module_sources=["""
+                long fn_test_result_of_call(long x)
+                {
+                    return x + 1;
+                }
+            """])
+            f = rffi.llexternal('fn_test_result_of_call', [lltype.Signed],
+                                RESTYPE, compilation_info=eci, _nowrapper=True)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value + 1))
+            assert intmask(f(value)) == expected
+            #
+            FUNC = self.FuncType([lltype.Signed], RESTYPE)
+            FPTR = self.Ptr(FUNC)
+            calldescr = self.cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
+            x = self.cpu.bh_call_i(self.get_funcbox(self.cpu, f).value,
+                                   calldescr, [value], None, None)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_call_compiled(self):
         # Test that calling a function that returns a CHAR, SHORT or INT,
         # signed or unsigned, properly gets zero-extended or sign-extended.
         from pypy.translator.tool.cbuild import ExternalCompilationInfo


From cfbolz at codespeak.net  Mon Oct 18 19:07:31 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Mon, 18 Oct 2010 19:07:31 +0200 (CEST)
Subject: [pypy-svn] r78061 - pypy/extradoc/talk/pepm2011
Message-ID: <20101018170731.B90B1282B9D@codespeak.net>

Author: cfbolz
Date: Mon Oct 18 19:07:26 2010
New Revision: 78061

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
more small tweaks, still missing a more aggressive rewrite


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Mon Oct 18 19:07:26 2010
@@ -302,6 +302,7 @@
 code.
 
 \subsection{Running Example}
+\label{sub:example}
 
 For the purpose of this paper, we are going to use a tiny interpreter for a dynamic language with
  a very simple object
@@ -387,6 +388,11 @@
 
 
 \begin{figure}
+\lstset{
+    numbers = right,
+    numberstyle = \tiny,
+    numbersep = 0pt
+}
 \begin{lstlisting}[mathescape]
 # arguments to the trace: $p_{0}$, $p_{1}$
 # inside f: res.add(y)
@@ -558,10 +564,10 @@
 \label{sec:statics}
 
 The main insight to improve the code shown in the last section is that objects
-in category 1 do not survive very long -- they are used only inside the loop and
-there is no other outside reference to them. The idea for improving
-the code is to analyze which objects fall in category 1 and therefore do
-not have to be allocated at all.
+in category 1 do not survive very long -- they are used only inside the loop
+and there is no other outside reference to them. Therefore the optimizer
+identifies objects in category 1 and removes the allocation of these objects,
+and all operations manipulating them.
 
 This is a process that is usually called \emph{escape analysis}
 \cite{goldberg_higher_1990}. In this paper we will
@@ -587,7 +593,7 @@
 as well, because the shape description stores the type and thus the outcome of
 the type check the guard does is statically known.
 
-In the example from last section, the following operations in the upper half
+In the example from Section~\ref{sub:example}, the following operations in the upper half
 of Figure~\ref{fig:unopt-trace} produce two
 static objects, and can be completely removed from the optimized trace:
 
@@ -737,8 +743,9 @@
 In this section we want to give a formal description of the semantics of the
 traces and of the optimizer and liken the optimization to partial evaluation.
 We focus on the operations for manipulating dynamically allocated objects,
-as those are the only ones that are actually optimized. Without loss of
-generality we also consider only objects with two fields in this section.
+as those are the only ones that are actually optimized. We also consider only
+objects with two fields in this section, generalizing to arbitrary many fields
+is straightforward.
 
 Traces are lists of operations. The operations considered here are
 \lstinline{new}, \lstinline{get}, \lstinline{set} and \lstinline{guard_class}.
@@ -853,8 +860,8 @@
 
 The static heap is a partial function from $V^*$ into the
 set of static objects, which are triples of a type and two elements of $V^*$.
-A variable $v^*$ is in the domain of the static heap $S$ as long as the
-optimizer does not need to become dynamic XXX. The object $S(v^*)$ describes
+The object referenced by a variable $v^*$ is static, if $v^*$ is in the domain
+of the static heap $S$. The object $S(v^*)$ describes
 what is statically known about the object, \ie its type and its fields. The
 fields of objects in the static heap are also elements of $V^*$ (or null, for
 short periods of time).


From arigo at codespeak.net  Mon Oct 18 19:08:42 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 19:08:42 +0200 (CEST)
Subject: [pypy-svn] r78062 -
	pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/test
Message-ID: <20101018170842.2BF70282B9D@codespeak.net>

Author: arigo
Date: Mon Oct 18 19:08:41 2010
New Revision: 78062

Modified:
   pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/test/test_descr.py
Log:
Write tests for the descrs.  Not implemented so far.


Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/test/test_descr.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/test/test_descr.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/test/test_descr.py	Mon Oct 18 19:08:41 2010
@@ -83,6 +83,18 @@
         assert     descr_f.is_float_field()
 
 
+def test_get_field_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        S = lltype.GcStruct('S', ('x', RESTYPE))
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            descr_x = get_field_descr(c2, S, 'x')
+            assert descr_x.is_field_signed() == signed
+
+
 def test_get_array_descr():
     U = lltype.Struct('U')
     T = lltype.GcStruct('T')
@@ -164,6 +176,25 @@
     assert descr.get_base_size(False) == 0
     assert descr.get_ofs_length(False) == -1
 
+
+def test_get_array_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        A = lltype.GcArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            arraydescr = get_array_descr(c2, A)
+            assert arraydescr.is_item_signed() == signed
+        #
+        RA = rffi.CArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            arraydescr = get_array_descr(c2, RA)
+            assert arraydescr.is_item_signed() == signed
+
+
 def test_get_call_descr_not_translated():
     c0 = GcCache(False)
     descr1 = get_call_descr(c0, [lltype.Char, lltype.Signed], lltype.Char)
@@ -219,6 +250,17 @@
     extrainfo = descr3.get_extra_info()
     assert extrainfo is None
 
+def test_get_call_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        A = lltype.GcArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            descr1 = get_call_descr(c2, [], RESTYPE)
+            assert descr1.is_result_signed() == signed
+
 
 def test_repr_of_descr():
     c0 = GcCache(False)


From afa at codespeak.net  Mon Oct 18 19:09:38 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 18 Oct 2010 19:09:38 +0200 (CEST)
Subject: [pypy-svn] r78063 - in pypy/branch/fast-forward/pypy/objspace/std:
	. test
Message-ID: <20101018170938.A6261282B9D@codespeak.net>

Author: afa
Date: Mon Oct 18 19:09:37 2010
New Revision: 78063

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py
   pypy/branch/fast-forward/pypy/objspace/std/typeobject.py
Log:
Revert r78038, will do it differently.


Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py	Mon Oct 18 19:09:37 2010
@@ -28,39 +28,6 @@
             raises(TypeError, type, 'sub', (stufftype,), {})
         """)
 
-    def test_safe_subclass(self):
-        space = self.space
-        class W_Base(W_Object):
-            pass
-        def descr__new__(space, w_subtype):
-            return space.allocate_instance(W_Base, w_subtype)
-        W_Base.typedef = StdTypeDef("base",
-                                     __new__ = interp2app(descr__new__))
-        w_base = space.gettypeobject(W_Base.typedef)
-
-        # Cannot be created with base.__new__(derived)
-        class W_Derived(W_Base):
-            pass
-        def descr__new__(space, w_subtype):
-            return space.allocate_instance(W_Derived, w_subtype)
-        W_Derived.typedef = StdTypeDef("derived", W_Base.typedef,
-                                     __new__ = interp2app(descr__new__))
-        w_derived = space.gettypeobject(W_Derived.typedef)
-
-        # Can be created with base.__new__(derived2)
-        class W_Derived2(W_Base):
-            pass
-        W_Derived2.typedef = StdTypeDef("derived2", W_Base.typedef,
-                                        )
-        w_derived2 = space.gettypeobject(W_Derived2.typedef)
-
-        space.appexec([w_base, w_derived, w_derived2],
-                      """(base, derived, derived2):
-            raises(TypeError, base.__new__, derived)
-            x = base.__new__(derived2)
-            assert isinstance(x, derived2)
-            """)
-
     def test_del_warning(self):
         warnings = []
         def my_warn(msg, warningscls):

Modified: pypy/branch/fast-forward/pypy/objspace/std/typeobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/typeobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/typeobject.py	Mon Oct 18 19:09:37 2010
@@ -326,8 +326,7 @@
             raise operationerrfmt(space.w_TypeError,
                 "%s.__new__(%s): %s is not a subtype of %s",
                 w_self.name, w_subtype.name, w_subtype.name, w_self.name)
-        if (w_self.instancetypedef is not w_subtype.instancetypedef and
-            w_self.lookup('__new__') is not w_subtype.lookup('__new__')):
+        if w_self.instancetypedef is not w_subtype.instancetypedef:
             raise operationerrfmt(space.w_TypeError,
                 "%s.__new__(%s) is not safe, use %s.__new__()",
                 w_self.name, w_subtype.name, w_subtype.name)


From antocuni at codespeak.net  Mon Oct 18 19:13:57 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Mon, 18 Oct 2010 19:13:57 +0200 (CEST)
Subject: [pypy-svn] r78064 - pypy/extradoc/talk/pepm2011
Message-ID: <20101018171357.E3B42282B9D@codespeak.net>

Author: antocuni
Date: Mon Oct 18 19:13:55 2010
New Revision: 78064

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
customize line numbers in the code listing


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Mon Oct 18 19:13:55 2010
@@ -670,16 +670,16 @@
 and \lstinline{guard_class} operations).
 
 \begin{figure}
-\begin{lstlisting}[mathescape]
-# arguments to the trace: $p_{0}$, $p_{1}$
+\begin{lstlisting}[mathescape,numbers=left,escapechar=|]
+# arguments to the trace: $p_{0}$, $p_{1}$ |\setcounter{lstnumber}{2}|
 guard_class($p_1$, BoxedInteger)
 $i_2$ = get($p_1$, intval)
 guard_class($p_0$, BoxedInteger)
 $i_3$ = get($p_0$, intval)
 $i_4$ = int_add($i_2$, $i_3$)
 $i_9$ = int_add($i_4$, -100)
-
-guard_class($p_0$, BoxedInteger)
+|\setcounter{lstnumber}{50}|
+guard_class($p_0$, BoxedInteger) 
 $i_{12}$ = get($p_0$, intval)
 $i_{14}$ = int_add($i_{12}$, -1)
 


From afa at codespeak.net  Mon Oct 18 19:19:57 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 18 Oct 2010 19:19:57 +0200 (CEST)
Subject: [pypy-svn] r78065 - in
	pypy/branch/fast-forward/pypy/module/exceptions: . test
Message-ID: <20101018171957.579FE282B9D@codespeak.net>

Author: afa
Date: Mon Oct 18 19:19:55 2010
New Revision: 78065

Modified:
   pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py
   pypy/branch/fast-forward/pypy/module/exceptions/test/test_exc.py
Log:
This seems enough to fix multiple inheritance of Exception classes:
you don't need a __new__ if the class has the same layout as its base.


Modified: pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py	(original)
+++ pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py	Mon Oct 18 19:19:55 2010
@@ -254,7 +254,6 @@
         base.typedef,
         __doc__ = W_Exc.__doc__,
         __module__ = 'exceptions',
-        __new__ = _new(W_Exc, realbase),
         **kwargs
     )
     W_Exc.typedef.applevel_subclasses_base = realbase

Modified: pypy/branch/fast-forward/pypy/module/exceptions/test/test_exc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/exceptions/test/test_exc.py	(original)
+++ pypy/branch/fast-forward/pypy/module/exceptions/test/test_exc.py	Mon Oct 18 19:19:55 2010
@@ -68,6 +68,7 @@
         assert isinstance(Exception(), BaseException)
         assert repr(Exception(3, "x")) == "Exception(3, 'x')"
         assert str(IOError("foo", "bar")) == "[Errno foo] bar"
+        assert isinstance(IOError("foo", "bar"), IOError)
 
     def test_custom_class(self):
         from exceptions import Exception, BaseException, LookupError
@@ -185,7 +186,7 @@
         raises(TypeError, UnicodeEncodeError, u"x", u"y", 1, 5, "bah")
 
     def test_multiple_inheritance(self):
-        from exceptions import LookupError, ValueError, Exception
+        from exceptions import LookupError, ValueError, Exception, IOError
         class A(LookupError, ValueError):
             pass
         assert issubclass(A, A)
@@ -209,6 +210,17 @@
         else:
             fail("bah")
 
+        class C(ValueError, IOError):
+            pass
+        c = C()
+        assert isinstance(ValueError(), ValueError)
+        assert isinstance(c, C)
+        assert isinstance(c, Exception)
+        assert isinstance(c, ValueError)
+        assert isinstance(c, IOError)
+        assert isinstance(c, EnvironmentError)
+        assert not isinstance(c, KeyError)
+
     def test_doc_and_module(self):
         import exceptions
         for name, e in exceptions.__dict__.items():


From arigo at codespeak.net  Mon Oct 18 19:22:38 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 19:22:38 +0200 (CEST)
Subject: [pypy-svn] r78066 -
	pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport
Message-ID: <20101018172238.AA58C282B9D@codespeak.net>

Author: arigo
Date: Mon Oct 18 19:22:37 2010
New Revision: 78066

Modified:
   pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/descr.py
Log:
Implement the sign flag in the descrs.


Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/descr.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/descr.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/descr.py	Mon Oct 18 19:22:37 2010
@@ -82,6 +82,7 @@
 
     _is_pointer_field = False   # unless overridden by GcPtrFieldDescr
     _is_float_field = False     # unless overridden by FloatFieldDescr
+    _is_field_signed = False    # unless overridden by XxxFieldDescr
 
     def is_pointer_field(self):
         return self._is_pointer_field
@@ -89,6 +90,9 @@
     def is_float_field(self):
         return self._is_float_field
 
+    def is_field_signed(self):
+        return self._is_field_signed
+
     def repr_of_descr(self):
         return '<%s %s %s>' % (self._clsname, self.name, self.offset)
 
@@ -105,7 +109,7 @@
 def getFieldDescrClass(TYPE):
     return getDescrClass(TYPE, BaseFieldDescr, GcPtrFieldDescr,
                          NonGcPtrFieldDescr, 'Field', 'get_field_size',
-                         '_is_float_field')
+                         '_is_float_field', '_is_field_signed')
 
 def get_field_descr(gccache, STRUCT, fieldname):
     cache = gccache._cache_field
@@ -144,6 +148,7 @@
 
     _is_array_of_pointers = False      # unless overridden by GcPtrArrayDescr
     _is_array_of_floats   = False      # unless overridden by FloatArrayDescr
+    _is_item_signed       = False      # unless overridden by XxxArrayDescr
 
     def is_array_of_pointers(self):
         return self._is_array_of_pointers
@@ -151,6 +156,9 @@
     def is_array_of_floats(self):
         return self._is_array_of_floats
 
+    def is_item_signed(self):
+        return self._is_item_signed
+
     def repr_of_descr(self):
         return '<%s>' % self._clsname
 
@@ -186,12 +194,12 @@
 def getArrayDescrClass(ARRAY):
     return getDescrClass(ARRAY.OF, BaseArrayDescr, GcPtrArrayDescr,
                          NonGcPtrArrayDescr, 'Array', 'get_item_size',
-                         '_is_array_of_floats')
+                         '_is_array_of_floats', '_is_item_signed')
 
 def getArrayNoLengthDescrClass(ARRAY):
     return getDescrClass(ARRAY.OF, BaseArrayNoLengthDescr, GcPtrArrayNoLengthDescr,
                          NonGcPtrArrayNoLengthDescr, 'ArrayNoLength', 'get_item_size',
-                         '_is_array_of_floats')
+                         '_is_array_of_floats', '_is_item_signed')
 
 def get_array_descr(gccache, ARRAY):
     cache = gccache._cache_array
@@ -242,6 +250,9 @@
     def get_result_size(self, translate_support_code):
         raise NotImplementedError
 
+    def is_result_signed(self):
+        return False    # unless overridden
+
     def create_call_stub(self, rtyper, RESULT):
         def process(c):
             arg = 'args_%s[%d]' % (c, seen[c])
@@ -307,6 +318,10 @@
     _return_type = history.INT
     call_stub = staticmethod(lambda func, args_i, args_r, args_f: 0)
 
+    _is_result_signed = False      # can be overridden in XxxCallDescr
+    def is_result_signed(self):
+        return self._is_result_signed
+
 class DynamicIntCallDescr(BaseIntCallDescr):
     """
     calldescr that works for every integer type, by explicitly passing it the
@@ -314,12 +329,17 @@
     """
     _clsname = 'DynamicIntCallDescr'
 
-    def __init__(self, arg_classes, result_size, extrainfo=None):
+    def __init__(self, arg_classes, result_size, result_sign, extrainfo=None):
         BaseIntCallDescr.__init__(self, arg_classes, extrainfo)
-        self._result_size = result_size
+        assert isinstance(result_sign, bool)
+        self._result_size = chr(result_size)
+        self._result_sign = result_sign
 
     def get_result_size(self, translate_support_code):
-        return self._result_size
+        return ord(self._result_size)
+
+    def is_result_signed(self):
+        return self._result_sign
 
 
 class NonGcPtrCallDescr(BaseIntCallDescr):
@@ -356,7 +376,8 @@
         return FloatCallDescr
     return getDescrClass(RESULT, BaseIntCallDescr, GcPtrCallDescr,
                          NonGcPtrCallDescr, 'Call', 'get_result_size',
-                         Ellipsis)  # <= floatattrname should not be used here
+                         Ellipsis,  # <= floatattrname should not be used here
+                         '_is_result_signed')
 
 def get_call_descr(gccache, ARGS, RESULT, extrainfo=None):
     arg_classes = []
@@ -383,7 +404,8 @@
 # ____________________________________________________________
 
 def getDescrClass(TYPE, BaseDescr, GcPtrDescr, NonGcPtrDescr,
-                  nameprefix, methodname, floatattrname, _cache={}):
+                  nameprefix, methodname, floatattrname, signedattrname,
+                  _cache={}):
     if isinstance(TYPE, lltype.Ptr):
         if TYPE.TO._gckind == 'gc':
             return GcPtrDescr
@@ -403,6 +425,8 @@
         #
         if TYPE is lltype.Float:
             setattr(Descr, floatattrname, True)
+        elif rffi.cast(TYPE, -1) == -1:
+            setattr(Descr, signedattrname, True)
         #
         _cache[nameprefix, TYPE] = Descr
         return Descr


From arigo at codespeak.net  Mon Oct 18 19:38:53 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 19:38:53 +0200 (CEST)
Subject: [pypy-svn] r78067 - in pypy/branch/jit-sign-descr/pypy:
	jit/backend/llsupport rlib
Message-ID: <20101018173853.21DDB282B90@codespeak.net>

Author: arigo
Date: Mon Oct 18 19:38:52 2010
New Revision: 78067

Modified:
   pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/ffisupport.py
   pypy/branch/jit-sign-descr/pypy/rlib/libffi.py
Log:
Pass 'signed-or-not' information from libffi to ffisupport.


Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/ffisupport.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/ffisupport.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/ffisupport.py	Mon Oct 18 19:38:52 2010
@@ -14,7 +14,8 @@
     arg_classes = ''.join(argkinds)
     if reskind == history.INT:
         size = intmask(ffi_result.c_size)
-        return DynamicIntCallDescr(arg_classes, size, extrainfo)
+        signed = is_ffi_type_signed(ffi_result)
+        return DynamicIntCallDescr(arg_classes, size, signed, extrainfo)
     elif reskind == history.REF:
         return  NonGcPtrCallDescr(arg_classes, extrainfo)
     elif reskind == history.FLOAT:
@@ -26,10 +27,15 @@
 def get_ffi_type_kind(ffi_type):
     from pypy.rlib.libffi import types
     kind = types.getkind(ffi_type)
-    if kind == 'i':
+    if kind == 'i' or kind == 'u':
         return history.INT
     elif kind == 'f':
         return history.FLOAT
     elif kind == 'v':
         return history.VOID
-    assert False, "Unsuported kind '%s'" % kind
+    assert False, "Unsupported kind '%s'" % kind
+
+def is_ffi_type_signed(ffi_type):
+    from pypy.rlib.libffi import types
+    kind = types.getkind(ffi_type)
+    return kind != 'u'

Modified: pypy/branch/jit-sign-descr/pypy/rlib/libffi.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/rlib/libffi.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/rlib/libffi.py	Mon Oct 18 19:38:52 2010
@@ -35,6 +35,9 @@
     @staticmethod
     @jit.purefunction
     def getkind(ffi_type):
+        """Returns 'v' for void, 'f' for float, 'i' for signed integer,
+        and 'u' for unsigned integer.
+        """
         if ffi_type is types.void:
             return 'v'
         elif ffi_type is types.double:
@@ -42,31 +45,31 @@
         elif ffi_type is types.pointer:
             return 'i'
         elif ffi_type is types.uchar:
-            return 'i'
+            return 'u'
         elif ffi_type is types.uint8:
-            return 'i'
+            return 'u'
         elif ffi_type is types.schar:
             return 'i'
         elif ffi_type is types.sint8:
             return 'i'
         elif ffi_type is types.uint16:
-            return 'i'
+            return 'u'
         elif ffi_type is types.ushort:
-            return 'i'
+            return 'u'
         elif ffi_type is types.sint16:
             return 'i'
         elif ffi_type is types.sshort:
             return 'i'
         elif ffi_type is types.uint:
-            return 'i'
+            return 'u'
         elif ffi_type is types.uint32:
-            return 'i'
+            return 'u'
         elif ffi_type is types.sint:
             return 'i'
         elif ffi_type is types.sint32:
             return 'i'
         ## elif ffi_type is types.uint64:
-        ##     return 'i'
+        ##     return 'u'
         ## elif ffi_type is types.sint64:
         ##     return 'i'
         raise KeyError


From arigo at codespeak.net  Mon Oct 18 19:39:42 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 18 Oct 2010 19:39:42 +0200 (CEST)
Subject: [pypy-svn] r78068 -
	pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/test
Message-ID: <20101018173942.7002C282B9D@codespeak.net>

Author: arigo
Date: Mon Oct 18 19:39:41 2010
New Revision: 78068

Modified:
   pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/test/test_ffisupport.py
Log:
Add a missing test for the previous checkin.


Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/test/test_ffisupport.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/test/test_ffisupport.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/test/test_ffisupport.py	Mon Oct 18 19:39:41 2010
@@ -12,6 +12,12 @@
     descr = get_call_descr_dynamic([], types.sint8)
     assert isinstance(descr, DynamicIntCallDescr)
     assert descr.get_result_size(False) == 1
+    assert descr.is_result_signed() == True
+
+    descr = get_call_descr_dynamic([], types.uint8)
+    assert isinstance(descr, DynamicIntCallDescr)
+    assert descr.get_result_size(False) == 1
+    assert descr.is_result_signed() == False
 
     descr = get_call_descr_dynamic([], types.float)
     assert descr is None # single floats are not supported so far


From cfbolz at codespeak.net  Mon Oct 18 19:47:53 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Mon, 18 Oct 2010 19:47:53 +0200 (CEST)
Subject: [pypy-svn] r78070 - pypy/extradoc/talk/pepm2011
Message-ID: <20101018174753.34D09282BD4@codespeak.net>

Author: cfbolz
Date: Mon Oct 18 19:47:50 2010
New Revision: 78070

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
add line numbers, try to gently rewrite things a bit


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Mon Oct 18 19:47:50 2010
@@ -575,26 +575,29 @@
 bit peculiar in that it receives no static input arguments for the trace,
 but it is only used to optimize operations within the trace.
 
-To optimize the trace, it is traversed from beginning to end.
-Whenever a \lstinline{new} operation is seen, the operation is removed and a \emph{static
-object}\footnote{Here ``static'' is meant in the sense of partial
-evaluation, \ie known at partial evaluation time, not in the sense of static
-allocation or static method.} is constructed and associated with the variable
-that would have stored
-the result of \lstinline{new}. The static object describes the shape of the
-original object, \eg where the values that would be stored in the fields of the
-allocated object come from, as well as the type of the object. Whenever the
-optimizer sees a \lstinline{set} that writes into such an object, that shape
-description is updated and the operation can be removed, which means that the
-operation was done at partial evaluation time. When the optimizer encounters a
-\lstinline{get} from such an object, the result is read from the shape
-description, and the operation is also removed. Equivalently, a
-\lstinline{guard_class} on a variable that has a shape description can be removed
-as well, because the shape description stores the type and thus the outcome of
-the type check the guard does is statically known.
+To optimize the trace, it is traversed from beginning to end. Every
+optimization in the input trace is either removed, or new operations are
+produced. Whenever a \lstinline{new} operation is seen, the operation it is
+removed optimistically and a \emph{static object}\footnote{Here ``static'' is
+meant in the sense of partial evaluation, \ie known at partial evaluation time,
+not in the sense of static allocation or static method.} is constructed and
+associated with the result variable. The static object describes the shape of
+the original object, \ie where the values that would be stored in the fields of
+the allocated object come from, as well as the type of the object.
+
+When a \lstinline{set} that writes into a static object is optimized, the
+corresponding shape description is updated and the operation is removed. This
+means that the operation was done at partial evaluation time. When the
+optimizer encounters a \lstinline{get} from a static object, the result is read
+from the shape description, and the operation is also removed. Equivalently, a
+\lstinline{guard_class} on a variable that has a shape description can be
+removed as well, because the shape description stores the type and thus the
+outcome of the type check the guard does is statically known. Operations that
+have dynamic (\ie all other) objects as arguments are just left untouched by
+the optimizer.
 
-In the example from Section~\ref{sub:example}, the following operations in the upper half
-of Figure~\ref{fig:unopt-trace} produce two
+In the example from Section~\ref{sub:example}, the following operations
+of Figure~\ref{fig:unopt-trace} (lines 10-17) produce two
 static objects, and can be completely removed from the optimized trace:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
@@ -604,15 +607,13 @@
 set($p_{6}$, intval, -100)
 \end{lstlisting}
 
-
-The static object associated with $p_{5}$ would know that it is a
+The static object associated with $p_{5}$ would store the knowledge that it is a
 \lstinline{BoxedInteger} whose \lstinline{intval} field contains $i_{4}$; the
-one associated with $p_{6}$ would know that it is a \lstinline{BoxedInteger}
+one associated with $p_{6}$ would store that it is a \lstinline{BoxedInteger}
 whose \lstinline{intval} field contains the constant -100.
 
-The subsequent operations in Figure~\ref{fig:unopt-trace},
- which use $p_{5}$ and $p_{6}$, could then be
-optimized using that knowledge:
+The subsequent operations (line 19-29) in Figure~\ref{fig:unopt-trace}, which
+use $p_{5}$ and $p_{6}$, can then be optimized using that knowledge:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 guard_class($p_{5}$, BoxedInteger)
@@ -635,18 +636,20 @@
 
 The rest of the trace from Figure~\ref{fig:unopt-trace} is optimized similarly.
 
+
 So far we have only described what happens when static objects are used in guards and in
 operations that read and write fields. When the static
 object is used in any other operation, it cannot remain static. For example, when
 a static object is stored in a globally accessible place, the object has to
-be allocated, as it might live longer than one iteration of the loop
-and because the partial evaluator looses track of it. This means that the static
+be allocated, as it might live longer than one iteration of the loop and as
+arbitrary \lstinline{set} operations could change it due to aliasing. This
+means that the static
 object needs to be turned into a dynamic one, \ie lifted. This makes it
 necessary to put operations into the residual code that allocate the
 static object at runtime.
 
 This is what happens at the end of the trace in Figure~\ref{fig:unopt-trace}, when the \lstinline{jump} operation
-is hit. The arguments of the jump are at this point static objects. Before the
+is optimized. The arguments of the jump are at this point static objects. Before the
 jump is emitted, they are \emph{lifted}. This means that the optimizer produces code
 that allocates a new object of the right type and sets its fields to the field
 values that the static object has (if the static object points to other static
@@ -670,26 +673,26 @@
 and \lstinline{guard_class} operations).
 
 \begin{figure}
-\begin{lstlisting}[mathescape,numbers=left,escapechar=|]
+\begin{lstlisting}[mathescape,numbers=right,escapechar=|,numberstyle = \tiny,numbersep=0pt, numberblanklines=false]
 # arguments to the trace: $p_{0}$, $p_{1}$ |\setcounter{lstnumber}{2}|
-guard_class($p_1$, BoxedInteger)
+guard_class($p_1$, BoxedInteger)           |\setcounter{lstnumber}{4}|
 $i_2$ = get($p_1$, intval)
-guard_class($p_0$, BoxedInteger)
+guard_class($p_0$, BoxedInteger)           |\setcounter{lstnumber}{7}|
 $i_3$ = get($p_0$, intval)
-$i_4$ = int_add($i_2$, $i_3$)
-$i_9$ = int_add($i_4$, -100)
-|\setcounter{lstnumber}{50}|
-guard_class($p_0$, BoxedInteger) 
-$i_{12}$ = get($p_0$, intval)
-$i_{14}$ = int_add($i_{12}$, -1)
-
-$i_{17}$ = int_gt($i_{14}$, 0)
-guard_true($i_{17}$)
+$i_4$ = int_add($i_2$, $i_3$)              |\setcounter{lstnumber}{25}|
+$i_9$ = int_add($i_4$, -100)               |\setcounter{lstnumber}{35}|
 
-$p_{15}$ = new(BoxedInteger)
-set($p_{15}$, intval, $i_{14}$)
-$p_{10}$ = new(BoxedInteger)
-set($p_{10}$, intval, $i_9$)
+guard_class($p_0$, BoxedInteger)           |\setcounter{lstnumber}{38}|
+$i_{12}$ = get($p_0$, intval)              |\setcounter{lstnumber}{42}|
+$i_{14}$ = int_add($i_{12}$, -1)           |\setcounter{lstnumber}{50}|
+
+$i_{17}$ = int_gt($i_{14}$, 0)             |\setcounter{lstnumber}{53}|
+guard_true($i_{17}$)                       |\setcounter{lstnumber}{42}|
+
+$p_{15}$ = new(BoxedInteger)               |\setcounter{lstnumber}{45}|
+set($p_{15}$, intval, $i_{14}$)            |\setcounter{lstnumber}{26}|
+$p_{10}$ = new(BoxedInteger)               |\setcounter{lstnumber}{28}|
+set($p_{10}$, intval, $i_9$)               |\setcounter{lstnumber}{53}|
 
 jump($p_{15}$, $p_{10}$)
 \end{lstlisting}
@@ -699,7 +702,9 @@
 
 The final optimized trace of the example can be seen in Figure~\ref{fig:step1}.
 The optimized trace contains only two allocations, instead of the original five,
-and only three \lstinline{guard_class} operations, from the original seven.
+and only three \lstinline{guard_class} operations, from the original seven. The
+line numbers are the lines where the operations occurred in the original trace
+in Figure~\ref{fig:unopt-trace}.
 
 \section{Formal Description of the Algorithm}
 \label{sec:formal}


From afa at codespeak.net  Mon Oct 18 21:30:02 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 18 Oct 2010 21:30:02 +0200 (CEST)
Subject: [pypy-svn] r78071 - pypy/branch/fast-forward/pypy/interpreter/test
Message-ID: <20101018193002.991F436C374@codespeak.net>

Author: afa
Date: Mon Oct 18 21:29:58 2010
New Revision: 78071

Modified:
   pypy/branch/fast-forward/pypy/interpreter/test/test_raise.py
Log:
Fix tests about string exceptions


Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_raise.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/test/test_raise.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/test/test_raise.py	Mon Oct 18 21:29:58 2010
@@ -4,14 +4,8 @@
     def test_arg_as_string(self):
         def f():
             raise "test"
+        raises(TypeError, f)
 
-        import warnings
-        warnings.simplefilter('error', DeprecationWarning)
-        try:
-            raises(DeprecationWarning, f)
-        finally:
-            warnings.simplefilter('default', DeprecationWarning)
-        
     def test_control_flow(self):
         try:
             raise Exception
@@ -48,17 +42,6 @@
             assert e.args[0] == 1
             assert e.args[1] == 2
 
-    def test_stringexc(self):
-        a = "hello world"
-        try:
-            raise a
-        except a, e:
-            assert e == None
-        try:
-            raise a, "message"
-        except a, e:
-            assert e == "message"
-
     def test_builtin_exc(self):
         try:
             [][0]


From afa at codespeak.net  Mon Oct 18 21:35:02 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 18 Oct 2010 21:35:02 +0200 (CEST)
Subject: [pypy-svn] r78072 - pypy/branch/fast-forward/lib-python
Message-ID: <20101018193502.C3E26282B90@codespeak.net>

Author: afa
Date: Mon Oct 18 21:34:57 2010
New Revision: 78072

Modified:
   pypy/branch/fast-forward/lib-python/TODO
Log:
update TODO
(and thanks to Armin for the future_builtins module!)


Modified: pypy/branch/fast-forward/lib-python/TODO
==============================================================================
--- pypy/branch/fast-forward/lib-python/TODO	(original)
+++ pypy/branch/fast-forward/lib-python/TODO	Mon Oct 18 21:34:57 2010
@@ -26,8 +26,6 @@
 
 - Fix fcntl.fcntl(fd, fcntl.F_NOTIFY, fcntl.DN_MULTISHOT) on 32bit platform.
 
-- Missing module future_builtins
-
 Medium tasks
 ------------
 
@@ -47,7 +45,9 @@
 
 - Finish _multiprocessing
 
-- Update the _ssl module (entry point is now _ssl.sslwrap)
+- many features are missing from the _ssl module
+
+- Enable -3 option to run test_py3kwarn.
 
 More difficult issues
 ---------------------


From dan at codespeak.net  Tue Oct 19 00:26:12 2010
From: dan at codespeak.net (dan at codespeak.net)
Date: Tue, 19 Oct 2010 00:26:12 +0200 (CEST)
Subject: [pypy-svn] r78073 -
	pypy/branch/micronumpy-resync/pypy/module/micronumpy
Message-ID: <20101018222612.0ED24282B90@codespeak.net>

Author: dan
Date: Tue Oct 19 00:26:11 2010
New Revision: 78073

Removed:
   pypy/branch/micronumpy-resync/pypy/module/micronumpy/
Log:
Taking the easy way out and just replacing this.



From dan at codespeak.net  Tue Oct 19 07:19:28 2010
From: dan at codespeak.net (dan at codespeak.net)
Date: Tue, 19 Oct 2010 07:19:28 +0200 (CEST)
Subject: [pypy-svn] r78075 - in
	pypy/branch/micronumpy-resync/pypy/module/micronumpy: . test
Message-ID: <20101019051928.73DA6282B90@codespeak.net>

Author: dan
Date: Tue Oct 19 07:19:25 2010
New Revision: 78075

Modified:
   pypy/branch/micronumpy-resync/pypy/module/micronumpy/array.py
   pypy/branch/micronumpy-resync/pypy/module/micronumpy/test/test_numpy.py
Log:
Added support code for array broadcasting, along with a small (unfinished) test.

Modified: pypy/branch/micronumpy-resync/pypy/module/micronumpy/array.py
==============================================================================
--- pypy/branch/micronumpy-resync/pypy/module/micronumpy/array.py	(original)
+++ pypy/branch/micronumpy-resync/pypy/module/micronumpy/array.py	Tue Oct 19 07:19:25 2010
@@ -32,6 +32,47 @@
     else:
         return 0
 
+def broadcast_shapes(a_shape, a_strides, b_shape, b_strides):
+    a_dim = len(a_shape)
+    b_dim = len(b_shape)
+
+    smaller_dim = a_dim if a_dim < b_dim else b_dim
+
+    if a_dim > b_dim:
+        result = a_shape
+        larger_dim = a_dim
+        smaller_dim = b_dim
+        shorter_strides = b_strides
+    else:
+        result = b_shape
+        larger_dim = b_dim
+        smaller_dim = a_dim
+        shorter_strides = a_strides
+
+    i_a = a_dim - 1
+    i_b = b_dim - 1
+    for i in range(smaller_dim):
+        assert i_a >= 0
+        a = a_shape[i_a]
+
+        assert i_b >= 0
+        b = b_shape[i_b]
+
+        if a == b or a == 1 or b == 1:
+            i_a -= 1
+            i_b -= 1
+            result[len(result) - 1 - i] = a if a > b else b
+        else:
+            raise ValueError("frames are not aligned") # FIXME: applevel?
+    
+    if a_dim < b_dim:
+        i_b += 1
+        a_strides = [0] * i_b + a_strides
+    else:
+        i_a += 1
+        b_strides = [0] * i_a + b_strides
+    return result, a_strides, b_strides
+
 def normalize_slice_starts(slice_starts, shape):
     for i in range(len(slice_starts)):
         if slice_starts[i] < 0:

Modified: pypy/branch/micronumpy-resync/pypy/module/micronumpy/test/test_numpy.py
==============================================================================
--- pypy/branch/micronumpy-resync/pypy/module/micronumpy/test/test_numpy.py	(original)
+++ pypy/branch/micronumpy-resync/pypy/module/micronumpy/test/test_numpy.py	Tue Oct 19 07:19:25 2010
@@ -425,6 +425,46 @@
         for w_xs, typecode in data:
             assert typecode == infer_from_iterable(space, w_xs).typecode
 
+class TestArraySupport(object):
+    def test_broadcast_shapes(self, space):
+        from pypy.module.micronumpy.array import broadcast_shapes
+        from pypy.module.micronumpy.array import stride_row as stride
+
+        def test(shape_a, shape_b, expected_result, expected_strides_a=None, expected_strides_b=None):
+            strides_a = [stride(shape_a, i) for i, x in enumerate(shape_a)]
+            strides_a_save = strides_a[:]
+
+            strides_b = [stride(shape_b, i) for i, x in enumerate(shape_b)]
+            strides_b_save = strides_b[:]
+
+            result_shape, result_strides_a, result_strides_b = broadcast_shapes(shape_a, strides_a, shape_b, strides_b)
+            assert result_shape == expected_result
+
+            if expected_strides_a:
+                assert result_strides_a == expected_strides_a
+            else:
+                assert result_strides_a == strides_a_save
+
+            if expected_strides_b:
+                assert result_strides_b == expected_strides_b
+            else:
+                assert result_strides_b == strides_b_save
+
+        shape_a = [256, 256, 3]
+        shape_b = [3]
+
+        test([256, 256, 3], [3],
+             expected_result=[256, 256, 3],
+             expected_strides_b=[0, 0, 1])
+
+        test([3], [256, 256, 3],
+             expected_result=[256, 256, 3],
+             expected_strides_a=[0, 0, 1])
+
+        test([8, 1, 6, 1], [7, 1, 5],
+             expected_result=[8, 7, 6, 5],
+             expected_strides_b=[0, 5, 5, 1])
+
 class TestMicroArray(object):
     @py.test.mark.xfail # XXX: return types changed
     def test_index2strides(self, space):


From afa at codespeak.net  Tue Oct 19 07:57:09 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 19 Oct 2010 07:57:09 +0200 (CEST)
Subject: [pypy-svn] r78076 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101019055709.B78F9282B90@codespeak.net>

Author: afa
Date: Tue Oct 19 07:57:07 2010
New Revision: 78076

Added:
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
Log:
Progress in _io module


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Tue Oct 19 07:57:07 2010
@@ -3,10 +3,10 @@
     TypeDef, interp_attrproperty_w, GetSetProperty)
 from pypy.interpreter.gateway import interp2app, unwrap_spec, Arguments
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
-from pypy.interpreter.error import OperationError, wrap_oserror2
+from pypy.interpreter.error import OperationError, wrap_oserror, wrap_oserror2
 from pypy.rlib.rarithmetic import r_longlong
 from os import O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_TRUNC
-import os
+import os, stat, errno
 
 O_BINARY = getattr(os, "O_BINARY", 0)
 O_APPEND = getattr(os, "O_APPEND", 0)
@@ -68,7 +68,16 @@
     if append:
         flags |= O_APPEND
 
-    return readable, writable, flags
+    return readable, writable, append, flags
+
+def convert_size(space, w_size):
+    if space.is_w(w_size, space.w_None):
+        return -1
+    else:
+        return space.int_w(w_size)
+
+def verify_fd(fd):
+    return
 
 class W_FileIO(W_RawIOBase):
     def __init__(self, space):
@@ -102,21 +111,44 @@
                 raise OperationError(space.w_ValueError, space.wrap(
                     "negative file descriptor"))
 
-        self.readable, self.writable, flags = decode_mode(space, mode)
+        self.readable, self.writable, append, flags = decode_mode(space, mode)
 
         if fd >= 0:
+            verify_fd(fd)
+            try:
+                os.fstat(fd)
+            except OSError, e:
+                if e.errno == errno.EBADF:
+                    raise wrap_oserror(space, e)
+                # else: pass
             self.fd = fd
+            self.closefd = bool(closefd)
         else:
+            if not closefd:
+                raise OperationError(space.w_ValueError, space.wrap(
+                    "Cannot use closefd=False with file name"))
+            self.closefd = True
+
             from pypy.module.posix.interp_posix import (
                 dispatch_filename, rposix)
             try:
                 self.fd = dispatch_filename(rposix.open)(
                     space, w_name, flags, 0666)
             except OSError, e:
-                raise wrap_oserror2(space, e, w_name)
-        self.closefd = bool(closefd)
+                raise wrap_oserror2(space, e, w_name,
+                                    exception_name='w_OSError')
+
+            self._dircheck(space, w_name)
         self.w_name = w_name
 
+        if append:
+            # For consistent behaviour, we explicitly seek to the end of file
+            # (otherwise, it might be done only on the first write()).
+            try:
+                os.lseek(self.fd, 0, os.SEEK_END)
+            except OSError, e:
+                raise wrap_oserror(space, e)
+
     def _mode(self):
         if self.readable:
             if self.writable:
@@ -134,8 +166,44 @@
             raise OperationError(space.w_ValueError, space.wrap(
                 "I/O operation on closed file"))
 
+    def _close(self, space):
+        if self.fd < 0:
+            return
+        fd = self.fd
+        self.fd = -1
+
+        try:
+            verify_fd(fd)
+            os.close(fd)
+        except OSError, e:
+            raise wrap_oserror(space, e,
+                               exception_name='w_IOError')
+
+    @unwrap_spec('self', ObjSpace)
+    def close_w(self, space):
+        if not self.closefd:
+            self.fd = -1
+            return
+        self._close(space)
+        W_RawIOBase.close_w(self, space)
+
+    def _dircheck(self, space, w_filename):
+        # On Unix, fopen will succeed for directories.
+        # In Python, there should be no file objects referring to
+        # directories, so we need a check.
+        if self.fd < 0:
+            return
+        try:
+            st = os.fstat(self.fd)
+        except OSError:
+            return
+        if stat.S_ISDIR(st.st_mode):
+            self._close()
+            raise wrap_oserror2(space, OSError(EISDIR), w_filename,
+                                exception_name='w_IOError')
+
     @unwrap_spec('self', ObjSpace, r_longlong, int)
-    def seek_w(self, space, pos, whence):
+    def seek_w(self, space, pos, whence=0):
         self._check_closed(space)
         pos = os.lseek(self.fd, pos, whence)
         return space.wrap(pos)
@@ -162,22 +230,55 @@
                 self.seekable = 1
         return space.newbool(self.seekable == 1)
 
+    # ______________________________________________
+
     @unwrap_spec('self', ObjSpace)
     def fileno_w(self, space):
         self._check_closed(space)
         return space.wrap(self.fd)
 
+    @unwrap_spec('self', ObjSpace)
+    def isatty_w(self, space):
+        self._check_closed(space)
+        try:
+            res = os.isatty(self.fd)
+        except OSError, e:
+            raise wrap_oserror(space, e)
+        return space.wrap(res)
+
+    # ______________________________________________
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def read_w(self, space, w_size=None):
+        self._check_closed(space)
+        # XXX self._check_readable(space)
+        size = convert_size(space, w_size)
+
+        if size < 0:
+            return self.readall_w(self, space)
+
+        try:
+            s = os.read(self.fd, size)
+        except OSError, e:
+            raise wrap_oserror(space, e,
+                               exception_name='w_OSError')
+
+        return space.wrap(s)
+
 W_FileIO.typedef = TypeDef(
     'FileIO', W_RawIOBase.typedef,
     __new__  = interp2app(W_FileIO.descr_new.im_func),
     __init__  = interp2app(W_FileIO.descr_init),
 
     seek = interp2app(W_FileIO.seek_w),
+    read = interp2app(W_FileIO.read_w),
+    close = interp2app(W_FileIO.close_w),
 
     readable = interp2app(W_FileIO.readable_w),
     writable = interp2app(W_FileIO.writable_w),
     seekable = interp2app(W_FileIO.seekable_w),
     fileno = interp2app(W_FileIO.fileno_w),
+    isatty = interp2app(W_FileIO.isatty_w),
     name = interp_attrproperty_w('w_name', cls=W_FileIO),
     mode = GetSetProperty(W_FileIO.descr_get_mode),
     )

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	Tue Oct 19 07:57:07 2010
@@ -3,6 +3,13 @@
     TypeDef, GetSetProperty, generic_new_descr)
 from pypy.interpreter.gateway import interp2app, Arguments, unwrap_spec
 from pypy.interpreter.error import OperationError
+from pypy.rlib.rstring import StringBuilder
+
+def convert_size(space, w_size):
+    if space.is_w(w_size, space.w_None):
+        return -1
+    else:
+        return space.int_w(w_size)
 
 class W_IOBase(Wrappable):
     def __init__(self, space):
@@ -94,6 +101,69 @@
                 space.w_IOError,
                 space.wrap("file or stream is not seekable"))
 
+    # ______________________________________________________________
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def readline_w(self, space, w_limit=None):
+        # For backwards compatibility, a (slowish) readline().
+        limit = convert_size(space, w_limit)
+
+        old_size = -1
+
+        has_peek = space.findattr(self, space.wrap("peek"))
+
+        builder = StringBuilder()
+
+        while limit < 0 or len(buffer) < limit:
+            nreadahead = 1
+
+            if has_peek:
+                w_readahead = space.call_method(self, "peek", space.wrap(1))
+                if not space.isinstance_w(w_readahead, space.w_str):
+                    raise operationerrorfmt(
+                        space.w_IOError,
+                        "peek() should have returned a bytes object, "
+                        "not '%s'", space.type(w_readahead).getname(space, '?'))
+                length = space.int_w(space.len(w_readahead))
+                if length > 0:
+                    n = 0
+                    buf = space.str_w(w_readahead)
+                    if limit >= 0:
+                        while True:
+                            if n >= length or n >= limit:
+                                break
+                            n += 1
+                            if buf[n-1] == '\n':
+                                break
+                    else:
+                        while True:
+                            if n >= length:
+                                break
+                            n += 1
+                            if buf[n-1] == '\n':
+                                break
+                    nreadahead = n
+
+            w_read = space.call_method(self, "read", space.wrap(nreadahead))
+            print "AFA", nreadahead, w_read
+            if not space.isinstance_w(w_read, space.w_str):
+                raise operationerrorfmt(
+                    space.w_IOError,
+                    "peek() should have returned a bytes object, "
+                    "not '%s'", space.type(w_read).getname(space, '?'))
+            read = space.str_w(w_read)
+            if not read:
+                break
+
+            print "AFA APPEND", repr(read)
+            builder.append(read)
+
+            if read[-1] == '\n':
+                break
+
+        print "AFA RETURN", repr(builder.build())
+        return space.wrap(builder.build())
+
 W_IOBase.typedef = TypeDef(
     '_IOBase',
     __new__ = generic_new_descr(W_IOBase),
@@ -109,6 +179,8 @@
     seekable = interp2app(W_IOBase.seekable_w),
     _checkSeekable = interp2app(W_IOBase.check_seekable_w),
     closed = GetSetProperty(W_IOBase.closed_get_w),
+
+    readline = interp2app(W_IOBase.readline_w),
     )
 
 class W_RawIOBase(W_IOBase):

Added: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- (empty file)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Tue Oct 19 07:57:07 2010
@@ -0,0 +1,43 @@
+from pypy.conftest import gettestobjspace
+from pypy.tool.udir import udir
+import os
+
+class AppTestFileIO:
+    def setup_class(cls):
+        cls.space = gettestobjspace(usemodules=['_io'])
+        tmpfile = udir.join('tmpfile')
+        tmpfile.write("a\nb\nc", mode='wb')
+        cls.w_tmpfile = cls.space.wrap(str(tmpfile))
+        cls.w_tmpdir = cls.space.wrap(str(udir))
+        cls.w_posix = cls.space.appexec([], """():
+            import %s as m;
+            return m""" % os.name)
+
+    def test_constructor(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'a')
+        assert f.name.endswith('tmpfile')
+        assert f.mode == 'wb'
+        f.close()
+
+    def test_open_fd(self):
+        import _io
+        os = self.posix
+        fd = os.open(self.tmpfile, os.O_RDONLY, 0666)
+        f = _io.FileIO(fd, "rb")
+        assert f.fileno() == fd
+        f.close()
+
+    def test_open_directory(self):
+        import _io
+        raises(OSError, _io.FileIO, self.tmpdir, "rb")
+
+    def test_readline(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'rb')
+        f.seek(0)
+        assert f.readline() == 'a\n'
+        assert f.readline() == 'b\n'
+        assert f.readline() == 'c'
+        assert f.readline() == ''
+        f.close()

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	Tue Oct 19 07:57:07 2010
@@ -1,5 +1,5 @@
 from pypy.conftest import gettestobjspace
-import os
+from pypy.tool.udir import udir
 
 class AppTestIoModule:
     def setup_class(cls):
@@ -40,12 +40,8 @@
 
 class AppTestOpen:
     def setup_class(cls):
-        from pypy.tool.udir import udir
         tmpfile = udir.join('tmpfile').ensure()
         cls.w_tmpfile = cls.space.wrap(str(tmpfile))
-        cls.w_posix = cls.space.appexec([], """():
-            import %s as m;
-            return m""" % os.name)
 
     def test_open(self):
         import io
@@ -57,13 +53,5 @@
     def test_open_writable(self):
         import io
         f = io.open(self.tmpfile, "w+b")
-        f.seek(0)
         f.close()
 
-    def test_open_fd(self):
-        import io
-        os = self.posix
-        fd = os.open(self.tmpfile, os.O_RDONLY, 0666)
-        f = io.open(fd, "rb")
-        assert f.fileno() == fd
-        f.close()


From cfbolz at codespeak.net  Tue Oct 19 11:40:44 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Tue, 19 Oct 2010 11:40:44 +0200 (CEST)
Subject: [pypy-svn] r78078 - pypy/extradoc/talk/pepm2011
Message-ID: <20101019094044.1C591282B90@codespeak.net>

Author: cfbolz
Date: Tue Oct 19 11:40:43 2010
New Revision: 78078

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
fixes


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Tue Oct 19 11:40:43 2010
@@ -576,7 +576,7 @@
 but it is only used to optimize operations within the trace.
 
 To optimize the trace, it is traversed from beginning to end. Every
-optimization in the input trace is either removed, or new operations are
+operation in the input trace is either removed, or new operations are
 produced. Whenever a \lstinline{new} operation is seen, the operation it is
 removed optimistically and a \emph{static object}\footnote{Here ``static'' is
 meant in the sense of partial evaluation, \ie known at partial evaluation time,
@@ -593,7 +593,7 @@
 \lstinline{guard_class} on a variable that has a shape description can be
 removed as well, because the shape description stores the type and thus the
 outcome of the type check the guard does is statically known. Operations that
-have dynamic (\ie all other) objects as arguments are just left untouched by
+have dynamic (\ie non-static) objects as arguments are just left untouched by
 the optimizer.
 
 In the example from Section~\ref{sub:example}, the following operations


From antocuni at codespeak.net  Tue Oct 19 11:45:51 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Tue, 19 Oct 2010 11:45:51 +0200 (CEST)
Subject: [pypy-svn] r78079 - pypy/trunk/pypy/module/pypyjit/test
Message-ID: <20101019094551.A49E0282B90@codespeak.net>

Author: antocuni
Date: Tue Oct 19 11:45:50 2010
New Revision: 78079

Modified:
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
Log:
skip the test if the _ffi module is not there


Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Tue Oct 19 11:45:50 2010
@@ -119,6 +119,8 @@
         result = child_stdout.read()
         child_stdout.close()
         assert result
+        if result.strip().startswith('SKIP:'):
+            py.test.skip(result.strip())
         assert result.splitlines()[-1].strip() == 'OK :-)'
         self.parse_loops(logfilepath)
         self.print_loops()
@@ -1143,7 +1145,12 @@
         libm_name = get_libm_name(sys.platform)
         out = self.run_source('''
         def main():
-            from _ffi import CDLL, types
+            try:
+                from _ffi import CDLL, types
+            except ImportError:
+                sys.stdout.write('SKIP: cannot import _ffi')
+                return 0
+
             libm = CDLL('%(libm_name)s')
             pow = libm.getfunc('pow', [types.double, types.double],
                                types.double)


From antocuni at codespeak.net  Tue Oct 19 13:39:50 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Tue, 19 Oct 2010 13:39:50 +0200 (CEST)
Subject: [pypy-svn] r78080 - pypy/trunk/pypy/module/pypyjit/test
Message-ID: <20101019113950.A3751282B90@codespeak.net>

Author: antocuni
Date: Tue Oct 19 13:39:49 2010
New Revision: 78080

Modified:
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
Log:
kill this test. It's just a special case of the previous one, and moreover it
does not really test anything, as it doesn't inspect the actual operations generated for the interesting bytecodes



Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Tue Oct 19 13:39:49 2010
@@ -789,24 +789,6 @@
                     return sa
                 '''%(e1, e2), n, ([], res))
 
-    def test_boolrewrite_ptr_single(self):
-        self.run_source('''
-            class tst:
-                pass
-            def main():
-                a = tst()
-                b = tst()
-                c = tst()
-                sa = 0
-                for i in range(1000):
-                    if a == b: sa += 1
-                    else: sa += 2
-                    if a != b: sa += 10000
-                    else: sa += 20000
-                    if i > 750: a = b
-                return sa
-            ''', 215, ([], 12481752))
-
     def test_array_sum(self):
         for tc, maxops in zip('bhilBHILfd', (38,) * 6 + (40, 40, 41, 38)):
             res = 19352859


From afa at codespeak.net  Tue Oct 19 14:46:32 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 19 Oct 2010 14:46:32 +0200 (CEST)
Subject: [pypy-svn] r78081 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101019124632.9D23F282B90@codespeak.net>

Author: afa
Date: Tue Oct 19 14:46:29 2010
New Revision: 78081

Modified:
   pypy/branch/fast-forward/pypy/module/_io/__init__.py
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
Log:
Implement FileIO.readall(), io.UnsupportedOperation
+ translation fixes


Modified: pypy/branch/fast-forward/pypy/module/_io/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/__init__.py	Tue Oct 19 14:46:29 2010
@@ -24,6 +24,14 @@
         'TextIOWrapper': 'interp_io.W_TextIOWrapper',
 
         'open': 'interp_io.open',
-        'UnsupportedOperation': 'space.w_None',
         'IncrementalNewlineDecoder': 'space.w_None',
         }
+
+    def init(self, space):
+        w_UnsupportedOperation = space.call_function(
+            space.w_type,
+            space.wrap('UnsupportedOperation'),
+            space.newtuple([space.w_ValueError, space.w_IOError]),
+            space.newdict())
+        space.setattr(self, space.wrap('UnsupportedOperation'),
+                      w_UnsupportedOperation)

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Tue Oct 19 14:46:29 2010
@@ -5,6 +5,7 @@
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.error import OperationError, wrap_oserror, wrap_oserror2
 from pypy.rlib.rarithmetic import r_longlong
+from pypy.rlib.rstring import StringBuilder
 from os import O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_TRUNC
 import os, stat, errno
 
@@ -76,6 +77,33 @@
     else:
         return space.int_w(w_size)
 
+SMALLCHUNK = 8 * 1024
+BIGCHUNK = 512 * 1024
+
+def new_buffersize(fd, currentsize):
+    try:
+        st = os.fstat(fd)
+        end = st.st_size
+        pos = os.lseek(fd, 0, 1)
+    except OSError:
+        pass
+    else:
+        # Files claiming a size smaller than SMALLCHUNK may
+        # actually be streaming pseudo-files. In this case, we
+        # apply the more aggressive algorithm below.
+        if end >= SMALLCHUNK and end >= pos:
+            # Add 1 so if the file were to grow we'd notice.
+            return currentsize + end - pos + 1
+
+    if currentsize > SMALLCHUNK:
+        # Keep doubling until we reach BIGCHUNK;
+        # then keep adding BIGCHUNK.
+        if currentsize <= BIGCHUNK:
+            return currentsize + currentsize
+        else:
+            return currentsize + BIGCHUNK
+    return currentsize + SMALLCHUNK
+
 def verify_fd(fd):
     return
 
@@ -136,7 +164,7 @@
                     space, w_name, flags, 0666)
             except OSError, e:
                 raise wrap_oserror2(space, e, w_name,
-                                    exception_name='w_OSError')
+                                    exception_name='w_IOError')
 
             self._dircheck(space, w_name)
         self.w_name = w_name
@@ -198,9 +226,9 @@
         except OSError:
             return
         if stat.S_ISDIR(st.st_mode):
-            self._close()
-            raise wrap_oserror2(space, OSError(EISDIR), w_filename,
-                                exception_name='w_IOError')
+            self._close(space)
+            raise wrap_oserror2(space, OSError(errno.EISDIR, "fstat"),
+                                w_filename, exception_name='w_IOError')
 
     @unwrap_spec('self', ObjSpace, r_longlong, int)
     def seek_w(self, space, pos, whence=0):
@@ -255,16 +283,43 @@
         size = convert_size(space, w_size)
 
         if size < 0:
-            return self.readall_w(self, space)
+            return self.readall_w(space)
 
         try:
             s = os.read(self.fd, size)
         except OSError, e:
             raise wrap_oserror(space, e,
-                               exception_name='w_OSError')
+                               exception_name='w_IOError')
 
         return space.wrap(s)
 
+    @unwrap_spec('self', ObjSpace)
+    def readall_w(self, space):
+        self._check_closed(space)
+        total = 0
+
+        builder = StringBuilder()
+        while True:
+            newsize = int(new_buffersize(self.fd, total))
+
+            try:
+                chunk = os.read(self.fd, newsize - total)
+            except OSError, e:
+                if e.errno == errno.EAGAIN:
+                    if total > 0:
+                        # return what we've got so far
+                        break
+                    return space.w_None
+                raise wrap_oserror(space, e,
+                                   exception_name='w_IOError')
+
+            if not chunk:
+                break
+            builder.append(chunk)
+            total += len(chunk)
+        return space.wrap(builder.build())
+
+
 W_FileIO.typedef = TypeDef(
     'FileIO', W_RawIOBase.typedef,
     __new__  = interp2app(W_FileIO.descr_new.im_func),
@@ -272,6 +327,7 @@
 
     seek = interp2app(W_FileIO.seek_w),
     read = interp2app(W_FileIO.read_w),
+    readall = interp2app(W_FileIO.readall_w),
     close = interp2app(W_FileIO.close_w),
 
     readable = interp2app(W_FileIO.readable_w),

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	Tue Oct 19 14:46:29 2010
@@ -2,7 +2,7 @@
 from pypy.interpreter.typedef import (
     TypeDef, GetSetProperty, generic_new_descr)
 from pypy.interpreter.gateway import interp2app, Arguments, unwrap_spec
-from pypy.interpreter.error import OperationError
+from pypy.interpreter.error import OperationError, operationerrfmt
 from pypy.rlib.rstring import StringBuilder
 
 def convert_size(space, w_size):
@@ -113,14 +113,15 @@
         has_peek = space.findattr(self, space.wrap("peek"))
 
         builder = StringBuilder()
+        size = 0
 
-        while limit < 0 or len(buffer) < limit:
+        while limit < 0 or size < limit:
             nreadahead = 1
 
             if has_peek:
                 w_readahead = space.call_method(self, "peek", space.wrap(1))
                 if not space.isinstance_w(w_readahead, space.w_str):
-                    raise operationerrorfmt(
+                    raise operationerrfmt(
                         space.w_IOError,
                         "peek() should have returned a bytes object, "
                         "not '%s'", space.type(w_readahead).getname(space, '?'))
@@ -145,9 +146,8 @@
                     nreadahead = n
 
             w_read = space.call_method(self, "read", space.wrap(nreadahead))
-            print "AFA", nreadahead, w_read
             if not space.isinstance_w(w_read, space.w_str):
-                raise operationerrorfmt(
+                raise operationerrfmt(
                     space.w_IOError,
                     "peek() should have returned a bytes object, "
                     "not '%s'", space.type(w_read).getname(space, '?'))
@@ -155,13 +155,12 @@
             if not read:
                 break
 
-            print "AFA APPEND", repr(read)
+            size += len(read)
             builder.append(read)
 
             if read[-1] == '\n':
                 break
 
-        print "AFA RETURN", repr(builder.build())
         return space.wrap(builder.build())
 
 W_IOBase.typedef = TypeDef(

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Tue Oct 19 14:46:29 2010
@@ -30,7 +30,7 @@
 
     def test_open_directory(self):
         import _io
-        raises(OSError, _io.FileIO, self.tmpdir, "rb")
+        raises(IOError, _io.FileIO, self.tmpdir, "rb")
 
     def test_readline(self):
         import _io
@@ -41,3 +41,8 @@
         assert f.readline() == 'c'
         assert f.readline() == ''
         f.close()
+
+    def test_readall(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'rb')
+        assert f.readall() == "a\nb\nc"

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	Tue Oct 19 14:46:29 2010
@@ -38,6 +38,10 @@
 
         assert list(MyFile()) == ["line1", "line2"]
 
+    def test_exception(self):
+        import _io
+        e = _io.UnsupportedOperation("seek")
+
 class AppTestOpen:
     def setup_class(cls):
         tmpfile = udir.join('tmpfile').ensure()


From arigo at codespeak.net  Tue Oct 19 15:41:37 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 19 Oct 2010 15:41:37 +0200 (CEST)
Subject: [pypy-svn] r78082 - pypy/trunk/pypy/jit/metainterp
Message-ID: <20101019134137.C9C55282B90@codespeak.net>

Author: arigo
Date: Tue Oct 19 15:41:35 2010
New Revision: 78082

Modified:
   pypy/trunk/pypy/jit/metainterp/warmstate.py
Log:
Support an infinite threshold (set to -1).


Modified: pypy/trunk/pypy/jit/metainterp/warmstate.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/warmstate.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/warmstate.py	Tue Oct 19 15:41:35 2010
@@ -176,6 +176,9 @@
             meth(default_value)
 
     def set_param_threshold(self, threshold):
+        if threshold < 0:
+            self.increment_threshold = 0   # never reach the THRESHOLD_LIMIT
+            return
         if threshold < 2:
             threshold = 2
         self.increment_threshold = (self.THRESHOLD_LIMIT // threshold) + 1


From arigo at codespeak.net  Tue Oct 19 15:42:35 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 19 Oct 2010 15:42:35 +0200 (CEST)
Subject: [pypy-svn] r78083 - pypy/build/bot2/pypybuildbot
Message-ID: <20101019134235.735E0282B90@codespeak.net>

Author: arigo
Date: Tue Oct 19 15:42:33 2010
New Revision: 78083

Modified:
   pypy/build/bot2/pypybuildbot/builds.py
Log:
Use -1 as the threshold to mean "infinite".
The issue is that some rsre code likely overflows 32 bits when
counting how many times some things loop.


Modified: pypy/build/bot2/pypybuildbot/builds.py
==============================================================================
--- pypy/build/bot2/pypybuildbot/builds.py	(original)
+++ pypy/build/bot2/pypybuildbot/builds.py	Tue Oct 19 15:42:33 2010
@@ -225,7 +225,7 @@
                      '--revision', WithProperties('%(got_revision)s'),
                      '--upload', #'--force-host', 'bigdog',
                      '--branch', WithProperties('%(branch)s'),
-                     '--args', ',--jit threshold=1000000000'],
+                     '--args', ',--jit threshold=-1'],
             workdir='./benchmarks',
             haltOnFailure=True))
         resfile = os.path.expanduser("~/bench_results_nojit/%(got_revision)s.json")


From cfbolz at codespeak.net  Tue Oct 19 16:43:49 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Tue, 19 Oct 2010 16:43:49 +0200 (CEST)
Subject: [pypy-svn] r78085 - in pypy/trunk/pypy/rlib/rsre: . test
Message-ID: <20101019144349.7F295282B90@codespeak.net>

Author: cfbolz
Date: Tue Oct 19 16:43:48 2010
New Revision: 78085

Modified:
   pypy/trunk/pypy/rlib/rsre/rsre_core.py
   pypy/trunk/pypy/rlib/rsre/test/test_zjit.py
Log:
(arigo, cfbolz, antocuni) this probably breaks translation, but makes sure that
the regex JIT actually optimize more


Modified: pypy/trunk/pypy/rlib/rsre/rsre_core.py
==============================================================================
--- pypy/trunk/pypy/rlib/rsre/rsre_core.py	(original)
+++ pypy/trunk/pypy/rlib/rsre/rsre_core.py	Tue Oct 19 16:43:48 2010
@@ -1,5 +1,5 @@
 import sys
-from pypy.rlib.debug import check_nonneg
+from pypy.rlib.debug import check_nonneg, make_sure_not_modified
 from pypy.rlib.unroll import unrolling_iterable
 from pypy.rlib.rsre import rsre_char
 from pypy.tool.sourcetools import func_with_new_name
@@ -469,6 +469,7 @@
     while True:
         op = ctx.pat(ppos)
         ppos += 1
+        make_sure_not_modified(ctx.pattern)
 
         #jit.jit_debug("sre_match", op, ppos, ptr)
         #

Modified: pypy/trunk/pypy/rlib/rsre/test/test_zjit.py
==============================================================================
--- pypy/trunk/pypy/rlib/rsre/test/test_zjit.py	(original)
+++ pypy/trunk/pypy/rlib/rsre/test/test_zjit.py	Tue Oct 19 16:43:48 2010
@@ -9,7 +9,6 @@
 def entrypoint1(r, string, repeat):
     r = array2list(r)
     string = hlstr(string)
-    make_sure_not_modified(r)
     match = None
     for i in range(repeat):
         match = rsre_core.match(r, string)
@@ -21,7 +20,6 @@
 def entrypoint2(r, string, repeat):
     r = array2list(r)
     string = hlstr(string)
-    make_sure_not_modified(r)
     match = None
     for i in range(repeat):
         match = rsre_core.search(r, string)
@@ -118,3 +116,8 @@
             r"Active\s+20\d\d-\d\d-\d\d\s+[[]\d+[]]([^[]+)",
             "Active"*20 + "Active 2010-04-07 [42] Foobar baz boz blah[43]")
         assert res == 6*20
+
+    def test_aorbstar(self):
+        res = self.meta_interp_match("(a|b)*a", "a" * 100)
+        assert res == 100
+        self.check_loops(guard_value=0)


From arigo at codespeak.net  Tue Oct 19 16:56:06 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 19 Oct 2010 16:56:06 +0200 (CEST)
Subject: [pypy-svn] r78086 - pypy/trunk/pypy/module/_sre
Message-ID: <20101019145606.68F4F282B90@codespeak.net>

Author: arigo
Date: Tue Oct 19 16:56:04 2010
New Revision: 78086

Modified:
   pypy/trunk/pypy/module/_sre/interp_sre.py
Log:
Fix translation by making sure that the annotator knows that
"code" is never modified.


Modified: pypy/trunk/pypy/module/_sre/interp_sre.py
==============================================================================
--- pypy/trunk/pypy/module/_sre/interp_sre.py	(original)
+++ pypy/trunk/pypy/module/_sre/interp_sre.py	Tue Oct 19 16:56:04 2010
@@ -290,10 +290,8 @@
 def SRE_Pattern__new__(space, w_subtype, w_pattern, flags, w_code,
               groups=0, w_groupindex=None, w_indexgroup=None):
     n = space.int_w(space.len(w_code))
-    code = [0] * n
-    for i in range(n):
-        x = space.uint_w(space.getitem(w_code, space.wrap(i)))
-        code[i] = intmask(x)
+    code = [intmask(space.uint_w(space.getitem(w_code, space.wrap(i))))
+            for i in range(n)]
     #
     w_srepat = space.allocate_instance(W_SRE_Pattern, w_subtype)
     srepat = space.interp_w(W_SRE_Pattern, w_srepat)


From arigo at codespeak.net  Tue Oct 19 17:16:49 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 19 Oct 2010 17:16:49 +0200 (CEST)
Subject: [pypy-svn] r78089 - in pypy/trunk/pypy/module/pypyjit: . test
Message-ID: <20101019151649.7FF2A282BAD@codespeak.net>

Author: arigo
Date: Tue Oct 19 17:16:48 2010
New Revision: 78089

Modified:
   pypy/trunk/pypy/module/pypyjit/__init__.py
   pypy/trunk/pypy/module/pypyjit/interp_jit.py
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
Log:
Attempt to fix test_blockstack_virtualizable.
The idea was to force a residual call, which is hard to do nowadays.
So I added a debugging API to the pypyjit module doing that.


Modified: pypy/trunk/pypy/module/pypyjit/__init__.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/__init__.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/__init__.py	Tue Oct 19 17:16:48 2010
@@ -6,6 +6,7 @@
 
     interpleveldefs = {
         'set_param':    'interp_jit.set_param',
+        'residual_call': 'interp_jit.residual_call',
     }
 
     def setup_after_space_initialization(self):

Modified: pypy/trunk/pypy/module/pypyjit/interp_jit.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/interp_jit.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/interp_jit.py	Tue Oct 19 17:16:48 2010
@@ -5,10 +5,10 @@
 
 from pypy.tool.pairtype import extendabletype
 from pypy.rlib.rarithmetic import r_uint, intmask
-from pypy.rlib.jit import JitDriver, hint, we_are_jitted
+from pypy.rlib.jit import JitDriver, hint, we_are_jitted, dont_look_inside
 import pypy.interpreter.pyopcode   # for side-effects
 from pypy.interpreter.error import OperationError, operationerrfmt
-from pypy.interpreter.gateway import ObjSpace, Arguments
+from pypy.interpreter.gateway import ObjSpace, Arguments, W_Root
 from pypy.interpreter.pycode import PyCode, CO_GENERATOR
 from pypy.interpreter.pyframe import PyFrame
 from pypy.interpreter.pyopcode import ExitFrame
@@ -131,3 +131,10 @@
                                   "no JIT parameter '%s'", key)
 
 set_param.unwrap_spec = [ObjSpace, Arguments]
+
+ at dont_look_inside
+def residual_call(space, w_callable, args):
+    '''For testing.  Invokes callable(...), but without letting
+    the JIT follow the call.'''
+    return space.call_args(w_callable, args)
+residual_call.unwrap_spec = [ObjSpace, W_Root, Arguments]

Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Tue Oct 19 17:16:48 2010
@@ -560,17 +560,13 @@
 
     def test_blockstack_virtualizable(self):
         self.run_source('''
-        def g(k):
-            s = 0
-            for i in range(k, k+2):
-                s += 1
-            return s
+        import pypyjit
 
         def main():
             i = 0
             while i < 100:
                 try:
-                    g(i)
+                    pypyjit.residual_call(len, [])
                 except:
                     pass
                 i += 1
@@ -1185,6 +1181,17 @@
         cls.pypy_c = option.pypy_c
 
 
+def test_interface_residual_call():
+    space = gettestobjspace(usemodules=['pypyjit'])
+    space.appexec([], """():
+        import pypyjit
+        def f(*args, **kwds):
+            return (args, kwds)
+        res = pypyjit.residual_call(f, 4, x=6)
+        assert res == ((4,), {'x': 6})
+    """)
+
+
 def has_info(pypy_c, option):
     g = os.popen('"%s" --info' % pypy_c, 'r')
     lines = g.readlines()


From arigo at codespeak.net  Tue Oct 19 17:30:10 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 19 Oct 2010 17:30:10 +0200 (CEST)
Subject: [pypy-svn] r78090 - pypy/trunk/lib-python
Message-ID: <20101019153010.5DB92282B90@codespeak.net>

Author: arigo
Date: Tue Oct 19 17:30:08 2010
New Revision: 78090

Modified:
   pypy/trunk/lib-python/conftest.py
Log:
Revert r76359, and skip ctypes tests again.
The comment (re-appearing because of the revert) explains why.
I'm not sure I understand why it was un-skipped but nothing
at all was done to fix the massive number of failures.


Modified: pypy/trunk/lib-python/conftest.py
==============================================================================
--- pypy/trunk/lib-python/conftest.py	(original)
+++ pypy/trunk/lib-python/conftest.py	Tue Oct 19 17:30:08 2010
@@ -464,7 +464,12 @@
     RegrTest('test_coding.py'),
     RegrTest('test_complex_args.py'),
     RegrTest('test_contextlib.py', usemodules="thread"),
-    RegrTest('test_ctypes.py', usemodules="_rawffi"),
+    # we skip test ctypes, since we adapted it massively in order
+    # to test what we want to support. There are real failures,
+    # but it's about missing features that we don't want to support
+    # now
+    RegrTest('test_ctypes.py', usemodules="_rawffi",
+             skip="missing features that we don't want to support now"),
     RegrTest('test_defaultdict.py'),
     RegrTest('test_email_renamed.py'),
     RegrTest('test_exception_variations.py'),


From afa at codespeak.net  Tue Oct 19 17:30:25 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 19 Oct 2010 17:30:25 +0200 (CEST)
Subject: [pypy-svn] r78091 - in pypy/branch/fast-forward/pypy/objspace/std:
	. test
Message-ID: <20101019153025.3B9ED282B90@codespeak.net>

Author: afa
Date: Tue Oct 19 17:30:23 2010
New Revision: 78091

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
   pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
Log:
Implement bytearray.extend


Modified: pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	Tue Oct 19 17:30:23 2010
@@ -350,5 +350,17 @@
         String2Bytearray(space, w_b),
         String2Bytearray(space, w_c)])
 
+# __________________________________________________________
+# Mutability methods
+
+def list_extend__Bytearray_Bytearray(space, w_bytearray, w_other):
+    w_bytearray.data += w_other.data
+    return space.w_None
+
+def list_extend__Bytearray_ANY(space, w_bytearray, w_other):
+    w_bytearray.data += [c for c in space.str_w(w_other)]
+    return space.w_None
+
+
 from pypy.objspace.std import bytearraytype
 register_all(vars(), bytearraytype)

Modified: pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py	Tue Oct 19 17:30:23 2010
@@ -14,6 +14,8 @@
     str_ljust, str_rjust, str_center, str_zfill,
     str_join, str_split, str_rsplit, str_partition, str_rpartition,
     str_splitlines)
+from pypy.objspace.std.listtype import (
+    list_extend)
 
 def _getbytevalue(space, w_value):
     if space.isinstance_w(w_value, space.w_str):

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	Tue Oct 19 17:30:23 2010
@@ -131,3 +131,9 @@
 
         check(b.partition(b'ss'), (b'mi', b'ss', b'issippi'))
         check(b.rpartition(b'ss'), (b'missi', b'ss', b'ippi'))
+
+    def test_extend(self):
+        b = bytearray('abc')
+        b.extend(bytearray('def'))
+        b.extend('ghi')
+        assert b == 'abcdefghi'


From antocuni at codespeak.net  Tue Oct 19 17:47:00 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Tue, 19 Oct 2010 17:47:00 +0200 (CEST)
Subject: [pypy-svn] r78092 - pypy/branch/jitypes
Message-ID: <20101019154700.DB608282B90@codespeak.net>

Author: antocuni
Date: Tue Oct 19 17:46:59 2010
New Revision: 78092

Added:
   pypy/branch/jitypes/   (props changed)
      - copied from r78091, pypy/trunk/
Log:
a branch where to try to make ctypes more jit friendly by using _ffi



From antocuni at codespeak.net  Tue Oct 19 17:48:54 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Tue, 19 Oct 2010 17:48:54 +0200 (CEST)
Subject: [pypy-svn] r78093 - pypy/branch/jitypes/pypy/config
Message-ID: <20101019154854.20858282B90@codespeak.net>

Author: antocuni
Date: Tue Oct 19 17:48:53 2010
New Revision: 78093

Modified:
   pypy/branch/jitypes/pypy/config/pypyoption.py
Log:
enable _ffi by default on this branch


Modified: pypy/branch/jitypes/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/jitypes/pypy/config/pypyoption.py	(original)
+++ pypy/branch/jitypes/pypy/config/pypyoption.py	Tue Oct 19 17:48:53 2010
@@ -30,7 +30,7 @@
       "rctime" , "select", "zipimport", "_lsprof",
      "crypt", "signal", "_rawffi", "termios", "zlib",
      "struct", "md5", "sha", "bz2", "_minimal_curses", "cStringIO",
-     "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array"]
+     "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array", "_ffi"]
 ))
 
 working_oo_modules = default_modules.copy()


From arigo at codespeak.net  Tue Oct 19 18:18:28 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 19 Oct 2010 18:18:28 +0200 (CEST)
Subject: [pypy-svn] r78094 - in pypy/trunk/pypy/module/_rawffi: . test
Message-ID: <20101019161828.A3002282B90@codespeak.net>

Author: arigo
Date: Tue Oct 19 18:18:27 2010
New Revision: 78094

Modified:
   pypy/trunk/pypy/module/_rawffi/structure.py
   pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py
Log:
Support unions with explicit fields, like we support structs
with explicit fields.


Modified: pypy/trunk/pypy/module/_rawffi/structure.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/structure.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/structure.py	Tue Oct 19 18:18:27 2010
@@ -34,7 +34,7 @@
 def round_up(size, alignment):
     return (size + alignment - 1) & -alignment
 
-def size_alignment_pos(fields):
+def size_alignment_pos(fields, is_union=False):
     size = 0
     alignment = 1
     pos = []
@@ -42,16 +42,20 @@
         # fieldtype is a W_Array
         fieldsize = fieldtype.size
         fieldalignment = fieldtype.alignment
-        size = round_up(size, fieldalignment)
         alignment = max(alignment, fieldalignment)
-        pos.append(size)
-        size += intmask(fieldsize)
+        if is_union:
+            pos.append(0)
+            size = max(size, fieldsize)
+        else:
+            size = round_up(size, fieldalignment)
+            pos.append(size)
+            size += intmask(fieldsize)
     size = round_up(size, alignment)
     return size, alignment, pos
 
 
 class W_Structure(W_DataShape):
-    def __init__(self, space, fields, size, alignment):
+    def __init__(self, space, fields, size, alignment, is_union=False):
         name_to_index = {}
         if fields is not None:
             for i in range(len(fields)):
@@ -60,7 +64,7 @@
                     raise operationerrfmt(space.w_ValueError,
                         "duplicate field name %s", name)
                 name_to_index[name] = i
-            size, alignment, pos = size_alignment_pos(fields)
+            size, alignment, pos = size_alignment_pos(fields, is_union)
         else: # opaque case
             fields = []
             pos = []
@@ -69,6 +73,7 @@
         self.alignment = alignment                
         self.ll_positions = pos
         self.name_to_index = name_to_index
+        self.is_union = is_union
 
     def allocate(self, space, length, autofree=False):
         # length is ignored!
@@ -133,15 +138,17 @@
     
 
 
-def descr_new_structure(space, w_type, w_shapeinfo):
+def descr_new_structure(space, w_type, w_shapeinfo, union=0):
+    is_union = bool(union)
     if space.is_true(space.isinstance(w_shapeinfo, space.w_tuple)):
         w_size, w_alignment = space.fixedview(w_shapeinfo, expected_length=2)
         S = W_Structure(space, None, space.int_w(w_size),
-                                     space.int_w(w_alignment))
+                                     space.int_w(w_alignment), is_union)
     else:
         fields = unpack_fields(space, w_shapeinfo)
-        S = W_Structure(space, fields, 0, 0)
+        S = W_Structure(space, fields, 0, 0, is_union)
     return space.wrap(S)
+descr_new_structure.unwrap_spec = [ObjSpace, W_Root, W_Root, int]
 
 W_Structure.typedef = TypeDef(
     'Structure',

Modified: pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py	Tue Oct 19 18:18:27 2010
@@ -945,14 +945,15 @@
         assert a[4] == 't'
 
     def test_union(self):
-        skip("segfaulting")
         import _rawffi
         longsize = _rawffi.sizeof('l')
-        S = _rawffi.Structure((longsize, longsize))
+        S = _rawffi.Structure([('x', 'h'), ('y', 'l')], union=True)
         s = S(autofree=False)
+        s.x = 12345
         lib = _rawffi.CDLL(self.lib_name)
         f = lib.ptr('ret_un_func', [(S, 1)], (S, 1))
         ret = f(s)
+        assert ret.y == 1234500, "ret.y == %d" % (ret.y,)
         s.free()
 
 class AppTestAutoFree:


From afa at codespeak.net  Tue Oct 19 18:25:38 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 19 Oct 2010 18:25:38 +0200 (CEST)
Subject: [pypy-svn] r78095 - in pypy/branch/fast-forward/pypy/objspace/std:
	. test
Message-ID: <20101019162538.986B6282B90@codespeak.net>

Author: afa
Date: Tue Oct 19 18:25:37 2010
New Revision: 78095

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
Log:
Fix bytearray->str conversion


Modified: pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	Tue Oct 19 18:25:37 2010
@@ -104,7 +104,7 @@
 
 # bytearray-to-string delegation
 def delegate_Bytearray2String(space, w_bytearray):
-    return W_StringObject(''.join(w_bytearray.data))
+    return str__Bytearray(space, w_bytearray)
 
 def String2Bytearray(space, w_str):
     data = [c for c in space.str_w(w_str)]
@@ -188,6 +188,9 @@
 
     return space.wrap(buf.build())
 
+def str__Bytearray(space, w_bytearray):
+    return W_StringObject(''.join(w_bytearray.data))
+
 def _convert_idx_params(space, w_self, w_start, w_stop):
     start = slicetype._Eval_SliceIndex(space, w_start)
     stop = slicetype._Eval_SliceIndex(space, w_stop)

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	Tue Oct 19 18:25:37 2010
@@ -27,6 +27,11 @@
         assert repr(bytearray('test')) == "bytearray(b'test')"
         assert repr(bytearray("d'oh")) == r"bytearray(b'd\'oh')"
 
+    def test_str(self):
+        assert str(bytearray()) == ""
+        assert str(bytearray('test')) == "test"
+        assert str(bytearray("d'oh")) == "d'oh"
+
     def test_getitem(self):
         b = bytearray('test')
         assert b[0] == ord('t')


From arigo at codespeak.net  Tue Oct 19 18:41:14 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 19 Oct 2010 18:41:14 +0200 (CEST)
Subject: [pypy-svn] r78096 - pypy/trunk/pypy/module/_rawffi
Message-ID: <20101019164114.83599282BD4@codespeak.net>

Author: arigo
Date: Tue Oct 19 18:41:12 2010
New Revision: 78096

Modified:
   pypy/trunk/pypy/module/_rawffi/structure.py
Log:
We don't actually need to store this as a field.


Modified: pypy/trunk/pypy/module/_rawffi/structure.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/structure.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/structure.py	Tue Oct 19 18:41:12 2010
@@ -73,7 +73,6 @@
         self.alignment = alignment                
         self.ll_positions = pos
         self.name_to_index = name_to_index
-        self.is_union = is_union
 
     def allocate(self, space, length, autofree=False):
         # length is ignored!


From afa at codespeak.net  Tue Oct 19 19:23:16 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 19 Oct 2010 19:23:16 +0200 (CEST)
Subject: [pypy-svn] r78097 - in pypy/branch/fast-forward/pypy/objspace/std:
	. test
Message-ID: <20101019172316.C6795282B90@codespeak.net>

Author: afa
Date: Tue Oct 19 19:23:15 2010
New Revision: 78097

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
   pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
Log:
bytearray.delslice, bytearray.append


Modified: pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	Tue Oct 19 19:23:15 2010
@@ -356,6 +356,11 @@
 # __________________________________________________________
 # Mutability methods
 
+def list_append__Bytearray_ANY(space, w_bytearray, w_item):
+    from pypy.objspace.std.bytearraytype import getbytevalue
+    w_bytearray.data.append(getbytevalue(space, w_item))
+    return space.w_None
+
 def list_extend__Bytearray_Bytearray(space, w_bytearray, w_other):
     w_bytearray.data += w_other.data
     return space.w_None
@@ -364,6 +369,12 @@
     w_bytearray.data += [c for c in space.str_w(w_other)]
     return space.w_None
 
+def delslice__Bytearray_ANY_ANY(space, w_bytearray, w_start, w_stop):
+    length = len(w_bytearray.data)
+    start, stop = normalize_simple_slice(space, length, w_start, w_stop)
+    if start == stop:
+        return
+    del w_bytearray.data[start:stop]
 
 from pypy.objspace.std import bytearraytype
 register_all(vars(), bytearraytype)

Modified: pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/bytearraytype.py	Tue Oct 19 19:23:15 2010
@@ -15,9 +15,9 @@
     str_join, str_split, str_rsplit, str_partition, str_rpartition,
     str_splitlines)
 from pypy.objspace.std.listtype import (
-    list_extend)
+    list_append, list_extend)
 
-def _getbytevalue(space, w_value):
+def getbytevalue(space, w_value):
     if space.isinstance_w(w_value, space.w_str):
         string = space.str_w(w_value)
         if len(string) != 1:
@@ -72,7 +72,7 @@
             if not e.match(space, space.w_StopIteration):
                 raise
             break
-        value = _getbytevalue(space, w_item)
+        value = getbytevalue(space, w_item)
         data.append(value)
 
     return new_bytearray(space, w_bytearraytype, data)

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	Tue Oct 19 19:23:15 2010
@@ -137,8 +137,21 @@
         check(b.partition(b'ss'), (b'mi', b'ss', b'issippi'))
         check(b.rpartition(b'ss'), (b'missi', b'ss', b'ippi'))
 
+    def test_append(self):
+        b = bytearray('abc')
+        b.append('d')
+        b.append(ord('e'))
+        assert b == 'abcde'
+
     def test_extend(self):
         b = bytearray('abc')
         b.extend(bytearray('def'))
         b.extend('ghi')
         assert b == 'abcdefghi'
+
+    def test_delslice(self):
+        b = bytearray('abcdefghi')
+        del b[5:8]
+        assert b == 'abcdei'
+        del b[:3]
+        assert b == 'dei'


From afa at codespeak.net  Tue Oct 19 19:29:13 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 19 Oct 2010 19:29:13 +0200 (CEST)
Subject: [pypy-svn] r78098 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101019172913.494E8282B90@codespeak.net>

Author: afa
Date: Tue Oct 19 19:29:11 2010
New Revision: 78098

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
Log:
_io.FileIO.write()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Tue Oct 19 19:29:11 2010
@@ -277,6 +277,20 @@
     # ______________________________________________
 
     @unwrap_spec('self', ObjSpace, W_Root)
+    def write_w(self, space, w_data):
+        self._check_closed(space)
+        # XXX self._check_writable(space)
+        data = space.str_w(w_data)
+
+        try:
+            n = os.write(self.fd, data)
+        except OSError, e:
+            raise wrap_oserror(space, e,
+                               exception_name='w_IOError')
+
+        return space.wrap(n)
+
+    @unwrap_spec('self', ObjSpace, W_Root)
     def read_w(self, space, w_size=None):
         self._check_closed(space)
         # XXX self._check_readable(space)
@@ -326,6 +340,7 @@
     __init__  = interp2app(W_FileIO.descr_init),
 
     seek = interp2app(W_FileIO.seek_w),
+    write = interp2app(W_FileIO.write_w),
     read = interp2app(W_FileIO.read_w),
     readall = interp2app(W_FileIO.readall_w),
     close = interp2app(W_FileIO.close_w),

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Tue Oct 19 19:29:11 2010
@@ -46,3 +46,15 @@
         import _io
         f = _io.FileIO(self.tmpfile, 'rb')
         assert f.readall() == "a\nb\nc"
+        f.close()
+
+    def test_write(self):
+        import _io
+        filename = self.tmpfile + '_w'
+        f = _io.FileIO(filename, 'wb')
+        f.write("test")
+        # try without flushing
+        f2 = _io.FileIO(filename, 'rb')
+        assert f2.read() == "test"
+        f.close()
+        f2.close()


From afa at codespeak.net  Tue Oct 19 23:52:16 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 19 Oct 2010 23:52:16 +0200 (CEST)
Subject: [pypy-svn] r78099 - pypy/branch/fast-forward/lib-python
Message-ID: <20101019215216.8AD5D282BDD@codespeak.net>

Author: afa
Date: Tue Oct 19 23:52:12 2010
New Revision: 78099

Modified:
   pypy/branch/fast-forward/lib-python/conftest.py
Log:
Skip test_multiprocessing. It leaves subprocesses around
which swamps the buildbot.


Modified: pypy/branch/fast-forward/lib-python/conftest.py
==============================================================================
--- pypy/branch/fast-forward/lib-python/conftest.py	(original)
+++ pypy/branch/fast-forward/lib-python/conftest.py	Tue Oct 19 23:52:12 2010
@@ -318,7 +318,7 @@
     RegrTest('test_multibytecodec.py', skip="unsupported codecs"),
     RegrTest('test_multibytecodec_support.py', skip="not a test"),
     RegrTest('test_multifile.py'),
-    RegrTest('test_multiprocessing.py'),
+    RegrTest('test_multiprocessing.py', skip='FIXME leaves subprocesses'),
     RegrTest('test_mutants.py', core="possibly"),
     RegrTest('test_mutex.py'),
     RegrTest('test_netrc.py'),


From afa at codespeak.net  Wed Oct 20 00:07:41 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 00:07:41 +0200 (CEST)
Subject: [pypy-svn] r78100 - pypy/branch/fast-forward/pypy
Message-ID: <20101019220741.8B645282BE3@codespeak.net>

Author: afa
Date: Wed Oct 20 00:07:40 2010
New Revision: 78100

Modified:
   pypy/branch/fast-forward/pypy/testrunner_cfg.py
Log:
See if I can kill remaining processes this way


Modified: pypy/branch/fast-forward/pypy/testrunner_cfg.py
==============================================================================
--- pypy/branch/fast-forward/pypy/testrunner_cfg.py	(original)
+++ pypy/branch/fast-forward/pypy/testrunner_cfg.py	Wed Oct 20 00:07:40 2010
@@ -1,6 +1,8 @@
 # nightly test configuration for the paraller runner
 import os
 
+os.system('killall pypy-c')
+
 def collect_one_testdir(testdirs, reldir, tests):
     if (reldir.startswith('translator/c/') or 
         reldir.startswith('translator/jvm/') or


From afa at codespeak.net  Wed Oct 20 00:10:19 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 00:10:19 +0200 (CEST)
Subject: [pypy-svn] r78101 - pypy/branch/fast-forward/pypy
Message-ID: <20101019221019.EF413282BE3@codespeak.net>

Author: afa
Date: Wed Oct 20 00:10:18 2010
New Revision: 78101

Modified:
   pypy/branch/fast-forward/pypy/testrunner_cfg.py
Log:
It worked!


Modified: pypy/branch/fast-forward/pypy/testrunner_cfg.py
==============================================================================
--- pypy/branch/fast-forward/pypy/testrunner_cfg.py	(original)
+++ pypy/branch/fast-forward/pypy/testrunner_cfg.py	Wed Oct 20 00:10:18 2010
@@ -1,8 +1,6 @@
 # nightly test configuration for the paraller runner
 import os
 
-os.system('killall pypy-c')
-
 def collect_one_testdir(testdirs, reldir, tests):
     if (reldir.startswith('translator/c/') or 
         reldir.startswith('translator/jvm/') or


From afa at codespeak.net  Wed Oct 20 00:40:46 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 00:40:46 +0200 (CEST)
Subject: [pypy-svn] r78102 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101019224046.C7F2C282BE3@codespeak.net>

Author: afa
Date: Wed Oct 20 00:40:45 2010
New Revision: 78102

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
Log:
Properly wrap OSError in FileIO.seek()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Wed Oct 20 00:40:45 2010
@@ -233,7 +233,11 @@
     @unwrap_spec('self', ObjSpace, r_longlong, int)
     def seek_w(self, space, pos, whence=0):
         self._check_closed(space)
-        pos = os.lseek(self.fd, pos, whence)
+        try:
+            pos = os.lseek(self.fd, pos, whence)
+        except OSError, e:
+            raise wrap_oserror(space, e,
+                               exception_name='w_IOError')
         return space.wrap(pos)
 
     @unwrap_spec('self', ObjSpace)

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Wed Oct 20 00:40:45 2010
@@ -35,7 +35,6 @@
     def test_readline(self):
         import _io
         f = _io.FileIO(self.tmpfile, 'rb')
-        f.seek(0)
         assert f.readline() == 'a\n'
         assert f.readline() == 'b\n'
         assert f.readline() == 'c'
@@ -58,3 +57,10 @@
         assert f2.read() == "test"
         f.close()
         f2.close()
+
+    def test_seek(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'rb')
+        f.seek(0)
+        self.posix.close(f.fileno())
+        raises(IOError, f.seek, 0)


From hakanardo at codespeak.net  Wed Oct 20 07:01:18 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Wed, 20 Oct 2010 07:01:18 +0200 (CEST)
Subject: [pypy-svn] r78103 -
	pypy/branch/jit-unroll-loops/pypy/module/array/benchmark
Message-ID: <20101020050118.D8F9E282B90@codespeak.net>

Author: hakanardo
Date: Wed Oct 20 07:01:16 2010
New Revision: 78103

Added:
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circular.c
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.c
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/looptst.py
      - copied, changed from r77967, pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/loop.py
Removed:
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/loop.py
Modified:
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/Makefile
Log:
circular buffer benchmark

Modified: pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/Makefile
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/Makefile	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/Makefile	Wed Oct 20 07:01:16 2010
@@ -5,4 +5,18 @@
 loop: loop.o
 	gcc -o $@ $^
 sum: sumtst.o sum.o
-	gcc -o $@ $^	
\ No newline at end of file
+	gcc -o $@ $^	
+circular: circulartst.o circular.o
+	gcc -o $@ $^	
+
+run: run-intimg run-sum run-circular
+
+run-%: %
+	@echo $^
+	@echo -n '        C: '
+	@/usr/bin/time -f '%e s' ./$^
+	@echo -n '     pypy: '
+	@/usr/bin/time -f '%e s' ../../../translator/goal/pypy-c $^tst.py 2>&1 | tail -1
+	@echo -n '  cpython: '
+	@/usr/bin/time -f '%e s' python $^tst.py
+	@echo

Added: pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circular.c
==============================================================================
--- (empty file)
+++ pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circular.c	Wed Oct 20 07:01:16 2010
@@ -0,0 +1,10 @@
+#include 
+
+double buf[65536];
+
+double circular(double *buf);
+
+int main() {
+  double sa = circular(buf);
+  //printf("%f\n", sa);
+}

Added: pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.c
==============================================================================
--- (empty file)
+++ pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.c	Wed Oct 20 07:01:16 2010
@@ -0,0 +1,14 @@
+
+double circular(double *buf) {
+  int i;
+  double sa;
+  for (i=0; i<65536; i++) buf[i] = i;
+
+  i = 10;
+  sa = 0;
+  while(i<200000000) {
+    sa += buf[(i-2)&65535] + buf[(i-1)&65535] + buf[i&65535] + buf[(i+1)&65535] + buf[(i+2)&65535];
+    i += 1;
+  }
+  return sa;
+}

Added: pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py
==============================================================================
--- (empty file)
+++ pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py	Wed Oct 20 07:01:16 2010
@@ -0,0 +1,19 @@
+from array import array
+class Circular(array):
+    def __new__(cls):
+        self = array.__new__(cls, 'd', range(65536))
+        return self
+    def __getitem__(self, i):
+        assert self.__len__() == 65536 
+        return array.__getitem__(self, i & 65535)
+
+def main():
+    buf = Circular()
+    i = 10
+    sa = 0
+    while i < 200000000:
+        sa += buf[i-2] + buf[i-1] + buf[i] + buf[i+1] + buf[i+2]
+        i += 1
+    return sa
+
+print main()

Copied: pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/looptst.py (from r77967, pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/loop.py)
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/loop.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/looptst.py	Wed Oct 20 07:01:16 2010
@@ -5,3 +5,5 @@
         s+=i
         i+=1
     return s
+
+print h()


From hakanardo at codespeak.net  Wed Oct 20 07:37:57 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Wed, 20 Oct 2010 07:37:57 +0200 (CEST)
Subject: [pypy-svn] r78104 - pypy/branch/jit-unroll-loops/pypy/jit/tl
Message-ID: <20101020053757.321D8282B90@codespeak.net>

Author: hakanardo
Date: Wed Oct 20 07:37:55 2010
New Revision: 78104

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py
Log:
Optimize setfield_raw/getfield_raw in the same way as the _gc versions 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py	Wed Oct 20 07:37:55 2010
@@ -70,7 +70,7 @@
         buf = Circular()
         i = 10
         sa = 0
-        while i < 20:
+        while i < 20000:
             sa += buf[i-2] + buf[i-1] + buf[i] + buf[i+1] + buf[i+2]
             i += 1
         return sa


From hakanardo at codespeak.net  Wed Oct 20 07:40:35 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Wed, 20 Oct 2010 07:40:35 +0200 (CEST)
Subject: [pypy-svn] r78105 - in pypy/branch/jit-unroll-loops/pypy:
	jit/metainterp/optimizeopt jit/metainterp/test jit/tl
	module/array/benchmark
Message-ID: <20101020054035.42948282B90@codespeak.net>

Author: hakanardo
Date: Wed Oct 20 07:40:33 2010
New Revision: 78105

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py
Log:
Optimize

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py	Wed Oct 20 07:40:33 2010
@@ -107,6 +107,7 @@
             return
         opnum = op.getopnum()
         if (opnum == rop.SETFIELD_GC or
+            opnum == rop.SETFIELD_RAW or
             opnum == rop.SETARRAYITEM_GC or
             opnum == rop.DEBUG_MERGE_POINT):
             return
@@ -236,6 +237,10 @@
         # remember the result of future reads of the field
         self.cache_field_value(op.getdescr(), value, fieldvalue, write=True)
 
+    optimize_GETFIELD_RAW = optimize_GETFIELD_GC
+    optimize_SETFIELD_RAW = optimize_SETFIELD_GC
+    # FIXME: Do we need separate caches for raw and gc?
+
     def optimize_GETARRAYITEM_GC(self, op):
         value = self.getvalue(op.getarg(0))
         indexvalue = self.getvalue(op.getarg(1))

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Wed Oct 20 07:40:33 2010
@@ -73,7 +73,7 @@
                     args.extend(self.getvalue(arg).get_forced_boxes())
                 newop.initarglist(args + inputargs[len(args):])
 
-            #print 'P: ', newop
+            #print 'P: ', str(newop)
             current = len(self.optimizer.newoperations)
             self.emit_operation(newop)
 
@@ -81,7 +81,7 @@
             current = max(current-1, 0)
                 
             for op in self.optimizer.newoperations[current:]:
-                #print 'E: ', op,  self.optimizer.newoperations.index(op)
+                #print 'E: ', str(op)
                 if op.is_guard():
                     descr = op.getdescr()
                     assert isinstance(descr, ResumeGuardDescr)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Wed Oct 20 07:40:33 2010
@@ -3571,6 +3571,25 @@
         """
         self.optimize_loop(ops, 'Not, Not', expected)
 
+    def test_setgetfield_raw(self):
+        ops = """
+        [p4, p7, i30]
+        p16 = getfield_gc(p4, descr=valuedescr)
+        guard_value(p16, ConstPtr(myptr), descr=) []
+        i1 = getfield_raw(p7, descr=nextdescr)
+        i2 = int_add(i1, i30)
+        setfield_raw(p7, 7, descr=nextdescr)
+        setfield_raw(p7, i2, descr=nextdescr)
+        jump(p4, p7, i30)
+        """
+        expected = """
+        [p4, p7, i30, i2]
+        i33 = int_add(i2, i30)
+        setfield_raw(p7, i33, descr=nextdescr)        
+        jump(p4, p7, i30, i33)
+        """
+        self.optimize_loop(ops, 'Not, Not, Not', expected)
+
     def test_addsub_ovf(self):
         ops = """
         [i0]

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py	Wed Oct 20 07:40:33 2010
@@ -34,6 +34,6 @@
     option.view = True
     warmspot.jittify_and_run(interp, graph, [], policy=policy,
                              listops=True, CPUClass=CPUClass,
-                             backendopt=True, inline=True,
+                             backendopt=True, inline=False,
                              optimizer=OPTIMIZER_NO_PERFECTSPEC)
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_demo.py	Wed Oct 20 07:40:33 2010
@@ -57,24 +57,28 @@
 
 ##     #img=array('h',(1,2,3,4))
 ##     print f(3)
+
     from array import array
     class Circular(array):
         def __new__(cls):
             self = array.__new__(cls, 'i', range(16))
             return self
         def __getitem__(self, i):
-            assert self.__len__() == 16 
+            #assert self.__len__() == 16 
             return array.__getitem__(self, i & 15)
 
     def main():
         buf = Circular()
         i = 10
         sa = 0
-        while i < 20000:
-            sa += buf[i-2] + buf[i-1] + buf[i] + buf[i+1] + buf[i+2]
+        while i < 20:
+            #sa += buf[i-2] + buf[i-1] + buf[i] + buf[i+1] + buf[i+2]
+            sa += buf[i]
             i += 1
         return sa
 
+    import pypyjit
+    pypyjit.set_param(threshold=3, inlining=True)
     print main()
     
 except Exception, e:

Modified: pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py	Wed Oct 20 07:40:33 2010
@@ -11,7 +11,7 @@
     buf = Circular()
     i = 10
     sa = 0
-    while i < 200000000:
+    while i < 20000000:
         sa += buf[i-2] + buf[i-1] + buf[i] + buf[i+1] + buf[i+2]
         i += 1
     return sa


From afa at codespeak.net  Wed Oct 20 10:15:37 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 10:15:37 +0200 (CEST)
Subject: [pypy-svn] r78106 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101020081537.4902C282B90@codespeak.net>

Author: afa
Date: Wed Oct 20 10:15:35 2010
New Revision: 78106

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
Log:
FileIO.tell()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Wed Oct 20 10:15:35 2010
@@ -241,6 +241,16 @@
         return space.wrap(pos)
 
     @unwrap_spec('self', ObjSpace)
+    def tell_w(self, space):
+        self._check_closed(space)
+        try:
+            pos = os.lseek(self.fd, 0, 1)
+        except OSError, e:
+            raise wrap_oserror(space, e,
+                               exception_name='w_IOError')
+        return space.wrap(pos)
+
+    @unwrap_spec('self', ObjSpace)
     def readable_w(self, space):
         self._check_closed(space)
         return space.wrap(self.readable)
@@ -344,6 +354,7 @@
     __init__  = interp2app(W_FileIO.descr_init),
 
     seek = interp2app(W_FileIO.seek_w),
+    tell = interp2app(W_FileIO.tell_w),
     write = interp2app(W_FileIO.write_w),
     read = interp2app(W_FileIO.read_w),
     readall = interp2app(W_FileIO.readall_w),

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Wed Oct 20 10:15:35 2010
@@ -64,3 +64,10 @@
         f.seek(0)
         self.posix.close(f.fileno())
         raises(IOError, f.seek, 0)
+
+    def test_tell(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'rb')
+        f.seek(3)
+        assert f.tell() == 3
+        f.close()


From antocuni at codespeak.net  Wed Oct 20 10:22:53 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Wed, 20 Oct 2010 10:22:53 +0200 (CEST)
Subject: [pypy-svn] r78107 - pypy/trunk/pypy/rlib
Message-ID: <20101020082253.BA141282B90@codespeak.net>

Author: antocuni
Date: Wed Oct 20 10:22:52 2010
New Revision: 78107

Modified:
   pypy/trunk/pypy/rlib/libffi.py
Log:
- reformat the if-elif chain to be more readable
- add the cases for slong and ulong: this fixes test_fficall on 64 bit, because without it thr 64 bit integers were never handled



Modified: pypy/trunk/pypy/rlib/libffi.py
==============================================================================
--- pypy/trunk/pypy/rlib/libffi.py	(original)
+++ pypy/trunk/pypy/rlib/libffi.py	Wed Oct 20 10:22:52 2010
@@ -35,43 +35,29 @@
     @staticmethod
     @jit.purefunction
     def getkind(ffi_type):
-        if ffi_type is types.void:
-            return 'v'
-        elif ffi_type is types.double:
-            return 'f'
-        elif ffi_type is types.pointer:
-            return 'i'
-        elif ffi_type is types.uchar:
-            return 'i'
-        elif ffi_type is types.uint8:
-            return 'i'
-        elif ffi_type is types.schar:
-            return 'i'
-        elif ffi_type is types.sint8:
-            return 'i'
-        elif ffi_type is types.uint16:
-            return 'i'
-        elif ffi_type is types.ushort:
-            return 'i'
-        elif ffi_type is types.sint16:
-            return 'i'
-        elif ffi_type is types.sshort:
-            return 'i'
-        elif ffi_type is types.uint:
-            return 'i'
-        elif ffi_type is types.uint32:
-            return 'i'
-        elif ffi_type is types.sint:
-            return 'i'
-        elif ffi_type is types.sint32:
-            return 'i'
-        ## elif ffi_type is types.uint64:
-        ##     return 'i'
-        ## elif ffi_type is types.sint64:
-        ##     return 'i'
+        if   ffi_type is types.void:    return 'v'
+        elif ffi_type is types.double:  return 'f'
+        elif ffi_type is types.pointer: return 'i'
+        #
+        elif ffi_type is types.schar:   return 'i'
+        elif ffi_type is types.uchar:   return 'i'
+        elif ffi_type is types.sshort:  return 'i'
+        elif ffi_type is types.ushort:  return 'i'
+        elif ffi_type is types.sint:    return 'i'
+        elif ffi_type is types.uint:    return 'i'
+        elif ffi_type is types.slong:   return 'i'
+        elif ffi_type is types.ulong:   return 'i'
+        #
+        elif ffi_type is types.sint8:   return 'i'
+        elif ffi_type is types.uint8:   return 'i'
+        elif ffi_type is types.sint16:  return 'i'
+        elif ffi_type is types.uint16:  return 'i'
+        elif ffi_type is types.sint32:  return 'i'
+        elif ffi_type is types.uint32:  return 'i'
+        ## elif ffi_type is types.sint64:  return 'i'
+        ## elif ffi_type is types.uint64:  return 'i'
         raise KeyError
 
-
 types._import()
 
 @specialize.arg(0)


From antocuni at codespeak.net  Wed Oct 20 10:44:42 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Wed, 20 Oct 2010 10:44:42 +0200 (CEST)
Subject: [pypy-svn] r78108 - pypy/branch/debug-test_import
Message-ID: <20101020084442.1D246282B90@codespeak.net>

Author: antocuni
Date: Wed Oct 20 10:44:40 2010
New Revision: 78108

Added:
   pypy/branch/debug-test_import/   (props changed)
      - copied from r78107, pypy/trunk/
Log:
(hopefully) very short lived branch to debug the systematic failure of
module/imp/test/test_import on 64 bit. Unfortunately, I cannot reproduce the
bug outside buildbot, so I'm going to hack the pylib to dump some useful info
in the log




From antocuni at codespeak.net  Wed Oct 20 10:51:56 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Wed, 20 Oct 2010 10:51:56 +0200 (CEST)
Subject: [pypy-svn] r78109 - in pypy/branch/debug-test_import: py/_process
	pypy
Message-ID: <20101020085156.16326282B90@codespeak.net>

Author: antocuni
Date: Wed Oct 20 10:51:55 2010
New Revision: 78109

Modified:
   pypy/branch/debug-test_import/py/_process/cmdexec.py
   pypy/branch/debug-test_import/pypy/testrunner_cfg.py
Log:
run only the relevant tests in this branch, and force cmdexec to print some useful information about the exception


Modified: pypy/branch/debug-test_import/py/_process/cmdexec.py
==============================================================================
--- pypy/branch/debug-test_import/py/_process/cmdexec.py	(original)
+++ pypy/branch/debug-test_import/py/_process/cmdexec.py	Wed Oct 20 10:51:55 2010
@@ -25,8 +25,15 @@
             default_encoding = sys.getdefaultencoding() # jython may not have it
         except AttributeError:
             default_encoding = sys.stdout.encoding or 'UTF-8'
-        out = unicode(out, process.stdout.encoding or default_encoding)
-        err = unicode(err, process.stderr.encoding or default_encoding)
+        try:
+            out = unicode(out, process.stdout.encoding or default_encoding)
+            err = unicode(err, process.stderr.encoding or default_encoding)
+        except UnicodeDecodeError:
+            print 'encoding:', process.stdout.encoding or default_encoding
+            print 'stdout:', repr(stdout)
+            print 'stderr:', repr(stderr)
+            raise
+            
     status = process.poll()
     if status:
         raise ExecutionFailed(status, status, cmd, out, err)

Modified: pypy/branch/debug-test_import/pypy/testrunner_cfg.py
==============================================================================
--- pypy/branch/debug-test_import/pypy/testrunner_cfg.py	(original)
+++ pypy/branch/debug-test_import/pypy/testrunner_cfg.py	Wed Oct 20 10:51:55 2010
@@ -15,5 +15,8 @@
 
 
 _cherrypick = os.getenv('PYPYCHERRYPICK', '')
+
+_cherrypick = 'module/imp'
+
 if _cherrypick:
     cherrypick = _cherrypick.split(':')


From antocuni at codespeak.net  Wed Oct 20 10:54:21 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Wed, 20 Oct 2010 10:54:21 +0200 (CEST)
Subject: [pypy-svn] r78110 - pypy/branch/debug-test_import/py/_process
Message-ID: <20101020085421.B36C0282B90@codespeak.net>

Author: antocuni
Date: Wed Oct 20 10:54:20 2010
New Revision: 78110

Modified:
   pypy/branch/debug-test_import/py/_process/cmdexec.py
Log:
bah, a typo :-(


Modified: pypy/branch/debug-test_import/py/_process/cmdexec.py
==============================================================================
--- pypy/branch/debug-test_import/py/_process/cmdexec.py	(original)
+++ pypy/branch/debug-test_import/py/_process/cmdexec.py	Wed Oct 20 10:54:20 2010
@@ -30,8 +30,8 @@
             err = unicode(err, process.stderr.encoding or default_encoding)
         except UnicodeDecodeError:
             print 'encoding:', process.stdout.encoding or default_encoding
-            print 'stdout:', repr(stdout)
-            print 'stderr:', repr(stderr)
+            print 'stdout:', repr(out)
+            print 'stderr:', repr(err)
             raise
             
     status = process.poll()


From arigo at codespeak.net  Wed Oct 20 11:37:31 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 11:37:31 +0200 (CEST)
Subject: [pypy-svn] r78111 - pypy/trunk/lib_pypy/_ctypes
Message-ID: <20101020093731.5C95A36E3FB@codespeak.net>

Author: arigo
Date: Wed Oct 20 11:37:29 2010
New Revision: 78111

Modified:
   pypy/trunk/lib_pypy/_ctypes/structure.py
   pypy/trunk/lib_pypy/_ctypes/union.py
Log:
My best fix to ctypes ever (which I hope is correct):
kill most of union.py, and just do it by importing from
structure.py.


Modified: pypy/trunk/lib_pypy/_ctypes/structure.py
==============================================================================
--- pypy/trunk/lib_pypy/_ctypes/structure.py	(original)
+++ pypy/trunk/lib_pypy/_ctypes/structure.py	Wed Oct 20 11:37:29 2010
@@ -7,7 +7,7 @@
 def round_up(size, alignment):
     return (size + alignment - 1) & -alignment
 
-def size_alignment_pos(fields):
+def size_alignment_pos(fields, is_union=False):
     import ctypes
     size = 0
     alignment = 1
@@ -15,14 +15,19 @@
     for fieldname, ctype in fields:
         fieldsize = ctypes.sizeof(ctype)
         fieldalignment = ctypes.alignment(ctype)
-        size = round_up(size, fieldalignment)
         alignment = max(alignment, fieldalignment)
-        pos.append(size)
-        size += fieldsize
+        if is_union:
+            pos.append(0)
+            size = max(size, fieldsize)
+        else:
+            size = round_up(size, fieldalignment)
+            pos.append(size)
+            size += fieldsize
     size = round_up(size, alignment)
     return size, alignment, pos
 
-def names_and_fields(_fields_, superclass, zero_offset=False, anon=None):
+def names_and_fields(_fields_, superclass, zero_offset=False, anon=None,
+                     is_union=False):
     if isinstance(_fields_, tuple):
         _fields_ = list(_fields_)
     for _, tp in _fields_:
@@ -36,7 +41,7 @@
     rawfields = [(name, ctype._ffishape)
                  for name, ctype in all_fields]
     if not zero_offset:
-        _, _, pos = size_alignment_pos(all_fields)
+        _, _, pos = size_alignment_pos(all_fields, is_union)
     else:
         pos = [0] * len(all_fields)
     fields = {}
@@ -73,8 +78,8 @@
 
 # ________________________________________________________________
 
-def _set_shape(tp, rawfields):
-    tp._ffistruct = _rawffi.Structure(rawfields)
+def _set_shape(tp, rawfields, is_union=False):
+    tp._ffistruct = _rawffi.Structure(rawfields, is_union)
     tp._ffiargshape = tp._ffishape = (tp._ffistruct, 1)
     tp._fficompositesize = tp._ffistruct.size
 
@@ -92,13 +97,14 @@
             raise AttributeError("Structure or union cannot contain itself")
         self._names, rawfields, self._fieldtypes = names_and_fields(
             value, self.__bases__[0], False,
-            self.__dict__.get('_anonymous_', None))
+            self.__dict__.get('_anonymous_', None), self._is_union)
         _CDataMeta.__setattr__(self, '_fields_', value)
-        _set_shape(self, rawfields)
+        _set_shape(self, rawfields, self._is_union)
         return
     _CDataMeta.__setattr__(self, name, value)
 
-class StructureMeta(_CDataMeta):
+class StructOrUnionMeta(_CDataMeta):
+
     def __new__(self, name, cls, typedict):
         res = type.__new__(self, name, cls, typedict)
         if '_fields_' in typedict:
@@ -109,8 +115,8 @@
                     raise AttributeError("Anonymous field not found")
             res._names, rawfields, res._fieldtypes = names_and_fields(
                 typedict['_fields_'], cls[0], False,
-                typedict.get('_anonymous_', None))
-            _set_shape(res, rawfields)
+                typedict.get('_anonymous_', None), self._is_union)
+            _set_shape(res, rawfields, self._is_union)
 
         return res
 
@@ -150,8 +156,8 @@
         res.__dict__['_index'] = -1
         return res
 
-class Structure(_CData):
-    __metaclass__ = StructureMeta
+class StructOrUnion(_CData):
+    __metaclass__ = StructOrUnionMeta
 
     def __new__(cls, *args, **kwds):
         if not hasattr(cls, '_ffistruct'):
@@ -213,3 +219,10 @@
 
     def _get_buffer_value(self):
         return self._buffer.buffer
+
+
+class StructureMeta(StructOrUnionMeta):
+    _is_union = False
+
+class Structure(StructOrUnion):
+    __metaclass__ = StructureMeta

Modified: pypy/trunk/lib_pypy/_ctypes/union.py
==============================================================================
--- pypy/trunk/lib_pypy/_ctypes/union.py	(original)
+++ pypy/trunk/lib_pypy/_ctypes/union.py	Wed Oct 20 11:37:29 2010
@@ -1,118 +1,7 @@
+from _ctypes import structure
 
+class UnionMeta(structure.StructOrUnionMeta):
+    _is_union = True
 
-import _rawffi
-from _ctypes.basics import _CData, _CDataMeta, store_reference, keepalive_key
-from _ctypes.basics import ensure_objects
-from _ctypes.structure import round_up, names_and_fields, struct_getattr,\
-     struct_setattr
-
-
-def _set_shape(tp):
-    size = tp._sizeofinstances()
-    alignment = tp._alignmentofinstances()
-    tp._ffiopaque = _rawffi.Structure((size, alignment)) # opaque
-    tp._ffiargshape = tp._ffishape = (tp._ffiopaque, 1)
-    tp._fficompositesize = tp._ffiopaque.size
-    # we need to create an array of size one for each
-    # of our elements
-    tp._ffiarrays = {}
-    for name, field in tp._fieldtypes.iteritems():
-        tp._ffiarrays[name] = _rawffi.Array(field.ctype._ffishape)
-        
-class UnionMeta(_CDataMeta):
-    def __new__(self, name, cls, typedict):
-        res = type.__new__(self, name, cls, typedict)
-        if '_fields_' in typedict:
-            res._names, rawfields, res._fieldtypes = names_and_fields(
-                typedict['_fields_'], cls[0], True,
-                typedict.get('_anonymous_', None))
-            _set_shape(res)
-
-        def __init__(self): # don't allow arguments by now
-            if not hasattr(self, '_ffiarrays'):
-                raise TypeError("Cannot instantiate union, has no type")
-            # malloc size
-            size = self.__class__._sizeofinstances()
-            self.__dict__['_objects'] = {}
-            self.__dict__['_buffer'] = self._ffiopaque(autofree=True)
-        res.__init__ = __init__
-        return res
-
-    def _sizeofinstances(self):
-        if not hasattr(self, '_size_'):
-            self._size_ = max([field.size for field in
-                               self._fieldtypes.values()] + [0])
-        return self._size_
-
-    def _alignmentofinstances(self):
-        from ctypes import alignment
-        if not hasattr(self, '_alignment_'):
-            self._alignment_ = max([alignment(field.ctype) for field in
-                                    self._fieldtypes.values()] + [1])
-        return self._alignment_
-    
-    __getattr__ = struct_getattr
-
-    def __setattr__(self, name, value):
-        if name == '_fields_':
-            if self.__dict__.get('_fields_', None):
-                raise AttributeError("_fields_ is final")
-            if self in [v for k, v in value]:
-                raise AttributeError("Union cannot contain itself")
-            self._names, rawfields, self._fieldtypes = names_and_fields(
-                value, self.__bases__[0], True,
-                self.__dict__.get('_anonymous_', None))
-            _CDataMeta.__setattr__(self, '_fields_', value)
-            _set_shape(self)
-        _CDataMeta.__setattr__(self, name, value)
-
-    def _CData_output(self, resarray, base=None, index=-1):
-        res = self.__new__(self)
-        ffiopaque = self._ffiopaque.fromaddress(resarray.buffer)
-        res.__dict__['_buffer'] = ffiopaque
-        res.__dict__['_base'] = base
-        res.__dict__['_index'] = index
-        return res
-    
-    def _CData_retval(self, resbuffer):
-        res = self.__new__(self)
-        res.__dict__['_buffer'] = resbuffer
-        res.__dict__['_base'] = None
-        res.__dict__['_index'] = -1
-        return res
-
-
-class Union(_CData):
+class Union(structure.StructOrUnion):
     __metaclass__ = UnionMeta
-
-    def __getattr__(self, name):
-        try:
-            field = self._fieldtypes[name]
-        except KeyError:
-            raise AttributeError(name)
-        fieldtype = field.ctype
-        val = self._ffiarrays[name].fromaddress(self._buffer.buffer, 1)
-        offset = field.num
-        return fieldtype._CData_output(val, self, offset)
-
-    def __setattr__(self, name, value):
-        try:
-            field = self._fieldtypes[name]
-        except KeyError:
-            raise AttributeError(name)
-        fieldtype = field.ctype
-        cobj = fieldtype.from_param(value)
-        if ensure_objects(cobj) is not None:
-            key = keepalive_key(field.num)
-            store_reference(self, key, cobj._objects)
-        arg = cobj._get_buffer_value()
-        if fieldtype._fficompositesize is not None:
-            from ctypes import memmove
-            dest = self._buffer.buffer
-            memmove(dest, arg, fieldtype._fficompositesize)
-        else:
-            buf = self._ffiarrays[name].fromaddress(self._buffer.buffer, 1)
-            buf[0] = arg
-
-    def _get_buffer_value(self):
-        return self._buffer.buffer


From antocuni at codespeak.net  Wed Oct 20 11:40:41 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Wed, 20 Oct 2010 11:40:41 +0200 (CEST)
Subject: [pypy-svn] r78112 -
	pypy/branch/debug-test_import/pypy/module/imp/test
Message-ID: <20101020094041.E234A282B90@codespeak.net>

Author: antocuni
Date: Wed Oct 20 11:40:40 2010
New Revision: 78112

Modified:
   pypy/branch/debug-test_import/pypy/module/imp/test/test_import.py
Log:
make sure that LANG is unset when calling py.py through py.process.cmdexec.  This is a workaround for pylib bug #130:
http://bitbucket.org/hpk42/py-trunk/issue/130/pyprocesscmdexec-fails-if-the-out-err-contains-non-ascii



Modified: pypy/branch/debug-test_import/pypy/module/imp/test/test_import.py
==============================================================================
--- pypy/branch/debug-test_import/pypy/module/imp/test/test_import.py	(original)
+++ pypy/branch/debug-test_import/pypy/module/imp/test/test_import.py	Wed Oct 20 11:40:40 2010
@@ -790,6 +790,7 @@
     extrapath = udir.ensure("pythonpath", dir=1) 
     extrapath.join("urllib.py").write("print 42\n")
     old = os.environ.get('PYTHONPATH', None)
+    oldlang = os.environ.pop('LANG', None)
     try: 
         os.environ['PYTHONPATH'] = str(extrapath)
         output = py.process.cmdexec('''"%s" "%s" -c "import urllib"''' % 
@@ -798,6 +799,8 @@
     finally: 
         if old: 
             os.environ['PYTHONPATH'] = old 
+        if oldlang:
+            os.environ['LANG'] = oldlang
 
 class AppTestImportHooks(object):
     def test_meta_path(self):


From arigo at codespeak.net  Wed Oct 20 11:44:05 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 11:44:05 +0200 (CEST)
Subject: [pypy-svn] r78113 - in pypy/trunk/pypy: . annotation
	jit/backend/llsupport jit/backend/test jit/backend/x86
	jit/backend/x86/test jit/codewriter jit/metainterp
	jit/metainterp/test module/_socket/test module/cpyext/test
	module/thread rlib rlib/test rpython rpython/lltypesystem
	rpython/lltypesystem/test rpython/memory rpython/memory/gc
	rpython/memory/gctransform rpython/memory/test rpython/test
	tool tool/test translator/c/src translator/c/test
Message-ID: <20101020094405.B9D9D36E3FB@codespeak.net>

Author: arigo
Date: Wed Oct 20 11:44:02 2010
New Revision: 78113

Added:
   pypy/trunk/pypy/tool/leakfinder.py
      - copied unchanged from r78005, pypy/branch/leak-finder/pypy/tool/leakfinder.py
   pypy/trunk/pypy/tool/test/test_leakfinder.py
      - copied unchanged from r78005, pypy/branch/leak-finder/pypy/tool/test/test_leakfinder.py
   pypy/trunk/pypy/translator/c/src/debug_alloc.h
      - copied unchanged from r78005, pypy/branch/leak-finder/pypy/translator/c/src/debug_alloc.h
Modified:
   pypy/trunk/pypy/annotation/builtin.py
   pypy/trunk/pypy/conftest.py
   pypy/trunk/pypy/jit/backend/llsupport/gc.py
   pypy/trunk/pypy/jit/backend/llsupport/llmodel.py
   pypy/trunk/pypy/jit/backend/test/runner_test.py
   pypy/trunk/pypy/jit/backend/x86/assembler.py
   pypy/trunk/pypy/jit/backend/x86/regalloc.py
   pypy/trunk/pypy/jit/backend/x86/runner.py
   pypy/trunk/pypy/jit/backend/x86/test/test_assembler.py
   pypy/trunk/pypy/jit/codewriter/jtransform.py
   pypy/trunk/pypy/jit/metainterp/test/test_basic.py
   pypy/trunk/pypy/jit/metainterp/virtualref.py
   pypy/trunk/pypy/module/_socket/test/test_sock_app.py
   pypy/trunk/pypy/module/cpyext/test/test_cpyext.py
   pypy/trunk/pypy/module/thread/ll_thread.py
   pypy/trunk/pypy/rlib/rgc.py
   pypy/trunk/pypy/rlib/test/test_rdynload.py
   pypy/trunk/pypy/rlib/test/test_rzlib.py
   pypy/trunk/pypy/rpython/llinterp.py
   pypy/trunk/pypy/rpython/lltypesystem/llmemory.py
   pypy/trunk/pypy/rpython/lltypesystem/lloperation.py
   pypy/trunk/pypy/rpython/lltypesystem/lltype.py
   pypy/trunk/pypy/rpython/lltypesystem/rclass.py
   pypy/trunk/pypy/rpython/lltypesystem/rffi.py
   pypy/trunk/pypy/rpython/lltypesystem/test/test_ll2ctypes.py
   pypy/trunk/pypy/rpython/lltypesystem/test/test_llmemory.py
   pypy/trunk/pypy/rpython/lltypesystem/test/test_lltype.py
   pypy/trunk/pypy/rpython/lltypesystem/test/test_rffi.py
   pypy/trunk/pypy/rpython/memory/gc/minimarkpage.py
   pypy/trunk/pypy/rpython/memory/gctransform/asmgcroot.py
   pypy/trunk/pypy/rpython/memory/gctransform/transform.py
   pypy/trunk/pypy/rpython/memory/gcwrapper.py
   pypy/trunk/pypy/rpython/memory/support.py
   pypy/trunk/pypy/rpython/memory/test/test_gc.py
   pypy/trunk/pypy/rpython/rbuiltin.py
   pypy/trunk/pypy/rpython/test/test_llinterp.py
   pypy/trunk/pypy/rpython/test/test_nongc.py
   pypy/trunk/pypy/rpython/test/test_rptr.py
   pypy/trunk/pypy/translator/c/src/g_include.h
   pypy/trunk/pypy/translator/c/src/main.h
   pypy/trunk/pypy/translator/c/test/test_genc.py
   pypy/trunk/pypy/translator/c/test/test_lltyped.py
Log:
Merge branch/leak-finder.  Adds a way for direct tests to detect
when a malloc(flavor='raw') has no corresponding free().  After
translation, in 'make debug' builds, records the raw-flavored
mallocs and frees and print at the end a summary of the still-
allocated objects.  It is normal to have a few (but not a lot).

Note that you don't get tracking if you specify malloc(immortal=True)
or malloc(track_allocation=False).



Modified: pypy/trunk/pypy/annotation/builtin.py
==============================================================================
--- pypy/trunk/pypy/annotation/builtin.py	(original)
+++ pypy/trunk/pypy/annotation/builtin.py	Wed Oct 20 11:44:02 2010
@@ -423,7 +423,7 @@
 from pypy.annotation.model import SomePtr
 from pypy.rpython.lltypesystem import lltype
 
-def malloc(s_T, s_n=None, s_flavor=None, s_zero=None):
+def malloc(s_T, s_n=None, s_flavor=None, s_zero=None, s_track_allocation=None):
     assert (s_n is None or s_n.knowntype == int
             or issubclass(s_n.knowntype, pypy.rlib.rarithmetic.base_int))
     assert s_T.is_constant()
@@ -438,13 +438,15 @@
         r = SomePtr(lltype.typeOf(p))
     else:
         assert s_flavor.is_constant()
+        assert s_track_allocation is None or s_track_allocation.is_constant()
         # not sure how to call malloc() for the example 'p' in the
         # presence of s_extraargs
         r = SomePtr(lltype.Ptr(s_T.const))
     return r
 
-def free(s_p, s_flavor):
+def free(s_p, s_flavor, s_track_allocation=None):
     assert s_flavor.is_constant()
+    assert s_track_allocation is None or s_track_allocation.is_constant()
     # same problem as in malloc(): some flavors are not easy to
     # malloc-by-example
     #T = s_p.ll_ptrtype.TO

Modified: pypy/trunk/pypy/conftest.py
==============================================================================
--- pypy/trunk/pypy/conftest.py	(original)
+++ pypy/trunk/pypy/conftest.py	Wed Oct 20 11:44:02 2010
@@ -7,6 +7,7 @@
 from inspect import isclass, getmro
 from pypy.tool.udir import udir
 from pypy.tool.autopath import pypydir
+from pypy.tool import leakfinder
 
 # pytest settings
 pytest_plugins = "resultlog",
@@ -354,7 +355,14 @@
 
     def runtest(self):
         try:
-            super(IntTestFunction, self).runtest()
+            leakfinder.start_tracking_allocations()
+            try:
+                super(IntTestFunction, self).runtest()
+            finally:
+                if leakfinder.TRACK_ALLOCATIONS:
+                    leaks = leakfinder.stop_tracking_allocations(False)
+                else:
+                    leaks = None   # stop_tracking_allocations() already called
         except OperationError, e:
             check_keyboard_interrupt(e)
             raise
@@ -373,6 +381,8 @@
                 _pygame_imported = True
                 assert option.view, ("should not invoke Pygame "
                                      "if conftest.option.view is False")
+        if leaks:        # check for leaks, but only if the test passed so far
+            raise leakfinder.MallocMismatch(leaks)
 
 class AppTestFunction(PyPyTestFunction):
     def _prunetraceback(self, traceback):

Modified: pypy/trunk/pypy/jit/backend/llsupport/gc.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llsupport/gc.py	(original)
+++ pypy/trunk/pypy/jit/backend/llsupport/gc.py	Wed Oct 20 11:44:02 2010
@@ -158,7 +158,7 @@
         # used to avoid too many duplications in the GCREF_LISTs.
         self.hashtable = lltype.malloc(self.HASHTABLE,
                                        self.HASHTABLE_SIZE+1,
-                                       flavor='raw')
+                                       flavor='raw', track_allocation=False)
         dummy = lltype.direct_ptradd(lltype.direct_arrayitems(self.hashtable),
                                      self.HASHTABLE_SIZE)
         dummy = llmemory.cast_ptr_to_adr(dummy)
@@ -252,14 +252,15 @@
 
     def _enlarge_gcmap(self):
         newlength = 250 + self._gcmap_maxlength * 2
-        newgcmap = lltype.malloc(self.GCMAP_ARRAY, newlength, flavor='raw')
+        newgcmap = lltype.malloc(self.GCMAP_ARRAY, newlength, flavor='raw',
+                                 track_allocation=False)
         oldgcmap = self._gcmap
         for i in range(self._gcmap_curlength):
             newgcmap[i] = oldgcmap[i]
         self._gcmap = newgcmap
         self._gcmap_maxlength = newlength
         if oldgcmap:
-            lltype.free(oldgcmap, flavor='raw')
+            lltype.free(oldgcmap, flavor='raw', track_allocation=False)
 
     def get_basic_shape(self, is_64_bit=False):
         # XXX: Should this code even really know about stack frame layout of
@@ -308,7 +309,8 @@
         # them inside bigger arrays) and we never try to share them.
         length = len(shape)
         compressed = lltype.malloc(self.CALLSHAPE_ARRAY, length,
-                                   flavor='raw')
+                                   flavor='raw',
+                                   track_allocation=False)   # memory leak
         for i in range(length):
             compressed[length-1-i] = rffi.cast(rffi.UCHAR, shape[i])
         return llmemory.cast_ptr_to_adr(compressed)

Modified: pypy/trunk/pypy/jit/backend/llsupport/llmodel.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llsupport/llmodel.py	(original)
+++ pypy/trunk/pypy/jit/backend/llsupport/llmodel.py	Wed Oct 20 11:44:02 2010
@@ -83,7 +83,8 @@
         # read back by the machine code reading at the address given by
         # pos_exception() and pos_exc_value().
         _exception_emulator = lltype.malloc(rffi.CArray(lltype.Signed), 2,
-                                            zero=True, flavor='raw')
+                                            zero=True, flavor='raw',
+                                            immortal=True)
         self._exception_emulator = _exception_emulator
 
         def _store_exception(lle):

Modified: pypy/trunk/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/trunk/pypy/jit/backend/test/runner_test.py	Wed Oct 20 11:44:02 2010
@@ -1330,6 +1330,7 @@
                                    descr=fd)
             res = self.execute_operation(get_op, [s_box], 'int', descr=fd)
             assert res.getint()  == 32
+        lltype.free(s, flavor='raw')
 
     def test_new_with_vtable(self):
         cpu = self.cpu

Modified: pypy/trunk/pypy/jit/backend/x86/assembler.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/assembler.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/assembler.py	Wed Oct 20 11:44:02 2010
@@ -249,7 +249,8 @@
 
     def _build_float_constants(self):
         # 44 bytes: 32 bytes for the data, and up to 12 bytes for alignment
-        addr = lltype.malloc(rffi.CArray(lltype.Char), 44, flavor='raw')
+        addr = lltype.malloc(rffi.CArray(lltype.Char), 44, flavor='raw',
+                             track_allocation=False)
         if not we_are_translated():
             self._keepalive_malloced_float_consts = addr
         float_constants = rffi.cast(lltype.Signed, addr)
@@ -399,7 +400,8 @@
             funcname = "" % len(self.loop_run_counters)
         # invent the counter, so we don't get too confused
         if self._debug:
-            struct = lltype.malloc(DEBUG_COUNTER, flavor='raw')
+            struct = lltype.malloc(DEBUG_COUNTER, flavor='raw',
+                                   track_allocation=False)   # known to leak
             struct.i = 0
             self.loop_run_counters.append((len(self.loop_run_counters), struct))
         return funcname

Modified: pypy/trunk/pypy/jit/backend/x86/regalloc.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/regalloc.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/regalloc.py	Wed Oct 20 11:44:02 2010
@@ -70,8 +70,9 @@
 
     def _get_new_array(self):
         n = self.BASE_CONSTANT_SIZE
+        # known to leak
         self.cur_array = lltype.malloc(rffi.CArray(lltype.Float), n,
-                                       flavor='raw')
+                                       flavor='raw', track_allocation=False)
         self.cur_array_free = n
     _get_new_array._dont_inline_ = True
 

Modified: pypy/trunk/pypy/jit/backend/x86/runner.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/runner.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/runner.py	Wed Oct 20 11:44:02 2010
@@ -113,7 +113,8 @@
         return CPU386.cast_adr_to_int(adr)
 
     all_null_registers = lltype.malloc(rffi.LONGP.TO, 24,
-                                       flavor='raw', zero=True)
+                                       flavor='raw', zero=True,
+                                       immortal=True)
 
     def force(self, addr_of_force_index):
         TP = rffi.CArrayPtr(lltype.Signed)

Modified: pypy/trunk/pypy/jit/backend/x86/test/test_assembler.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/test/test_assembler.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/test/test_assembler.py	Wed Oct 20 11:44:02 2010
@@ -81,7 +81,8 @@
 
     # also test rebuild_faillocs_from_descr(), which should not
     # reproduce the holes at all
-    bytecode = lltype.malloc(rffi.UCHARP.TO, len(mc.content), flavor='raw')
+    bytecode = lltype.malloc(rffi.UCHARP.TO, len(mc.content), flavor='raw',
+                             immortal=True)
     for i in range(len(mc.content)):
         assert 0 <= mc.content[i] <= 255
         bytecode[i] = rffi.cast(rffi.UCHAR, mc.content[i])
@@ -115,7 +116,8 @@
         assert withfloats
         value = random.random() - 0.5
         # make sure it fits into 64 bits
-        tmp = lltype.malloc(rffi.LONGP.TO, 2, flavor='raw')
+        tmp = lltype.malloc(rffi.LONGP.TO, 2, flavor='raw',
+                            track_allocation=False)
         rffi.cast(rffi.DOUBLEP, tmp)[0] = value
         return rffi.cast(rffi.DOUBLEP, tmp)[0], tmp[0], tmp[1]
 
@@ -152,10 +154,12 @@
 
     # prepare the expected target arrays, the descr_bytecode,
     # the 'registers' and the 'stack' arrays according to 'content'
-    xmmregisters = lltype.malloc(rffi.LONGP.TO, 16+ACTUAL_CPU.NUM_REGS+1, flavor='raw')
+    xmmregisters = lltype.malloc(rffi.LONGP.TO, 16+ACTUAL_CPU.NUM_REGS+1,
+                                 flavor='raw', immortal=True)
     registers = rffi.ptradd(xmmregisters, 16)
     stacklen = baseloc + 10
-    stack = lltype.malloc(rffi.LONGP.TO, stacklen, flavor='raw')
+    stack = lltype.malloc(rffi.LONGP.TO, stacklen, flavor='raw',
+                          immortal=True)
     expected_ints = [0] * len(content)
     expected_ptrs = [lltype.nullptr(llmemory.GCREF.TO)] * len(content)
     expected_floats = [0.0] * len(content)
@@ -221,7 +225,7 @@
     descr_bytecode.append(0x00)
     descr_bytecode.append(0xCC)   # end marker
     descr_bytes = lltype.malloc(rffi.UCHARP.TO, len(descr_bytecode),
-                                flavor='raw')
+                                flavor='raw', immortal=True)
     for i in range(len(descr_bytecode)):
         assert 0 <= descr_bytecode[i] <= 255
         descr_bytes[i] = rffi.cast(rffi.UCHAR, descr_bytecode[i])

Modified: pypy/trunk/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/jtransform.py	Wed Oct 20 11:44:02 2010
@@ -433,7 +433,8 @@
                                   op.result)
 
     def rewrite_op_free(self, op):
-        assert op.args[1].value == 'raw'
+        flags = op.args[1].value
+        assert flags['flavor'] == 'raw'
         ARRAY = op.args[0].concretetype.TO
         return self._do_builtin_call(op, 'raw_free', [op.args[0]],
                                      extra = (ARRAY,), extrakey = ARRAY)

Modified: pypy/trunk/pypy/jit/metainterp/test/test_basic.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_basic.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_basic.py	Wed Oct 20 11:44:02 2010
@@ -1780,7 +1780,7 @@
             c = bool(p1)
             d = not bool(p2)
             return 1000*a + 100*b + 10*c + d
-        prebuilt = [lltype.malloc(TP, flavor='raw')] * 2
+        prebuilt = [lltype.malloc(TP, flavor='raw', immortal=True)] * 2
         expected = f(0, 1)
         assert self.interp_operations(f, [0, 1]) == expected
 

Modified: pypy/trunk/pypy/jit/metainterp/virtualref.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/virtualref.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/virtualref.py	Wed Oct 20 11:44:02 2010
@@ -16,7 +16,8 @@
             ('virtualref_index', lltype.Signed),
             ('forced', rclass.OBJECTPTR))
         self.jit_virtual_ref_vtable = lltype.malloc(rclass.OBJECT_VTABLE,
-                                                    zero=True, flavor='raw')
+                                                    zero=True, flavor='raw',
+                                                    immortal=True)
         self.jit_virtual_ref_vtable.name = rclass.alloc_array_name(
             'jit_virtual_ref')
         # build some constants

Modified: pypy/trunk/pypy/module/_socket/test/test_sock_app.py
==============================================================================
--- pypy/trunk/pypy/module/_socket/test/test_sock_app.py	(original)
+++ pypy/trunk/pypy/module/_socket/test/test_sock_app.py	Wed Oct 20 11:44:02 2010
@@ -254,6 +254,7 @@
     s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
     fd = s.fileno()
     w_obj = rsocket.make_address(c_addr, addrlen).as_object(fd, space)
+    lltype.free(c_addr_ll, flavor='raw')
     assert space.is_true(space.eq(w_obj, space.newtuple([
         space.wrap('lo'),
         space.wrap(socket.ntohs(8)),

Modified: pypy/trunk/pypy/module/cpyext/test/test_cpyext.py
==============================================================================
--- pypy/trunk/pypy/module/cpyext/test/test_cpyext.py	(original)
+++ pypy/trunk/pypy/module/cpyext/test/test_cpyext.py	Wed Oct 20 11:44:02 2010
@@ -16,6 +16,7 @@
 from pypy.module.cpyext.pyobject import Py_DecRef, InvalidPointerException
 from pypy.translator.goal import autopath
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 
 @api.cpython_api([], api.PyObject)
 def PyPy_Crash1(space):
@@ -78,7 +79,6 @@
         self.frozen_refcounts[w_obj] = obj.c_ob_refcnt
     #state.print_refcounts()
     self.frozen_ll2callocations = set(ll2ctypes.ALLOCATED.values())
-    lltype.start_tracking_allocations()
 
 class LeakCheckingTest(object):
     def check_and_print_leaks(self):
@@ -126,17 +126,8 @@
         for w_obj in lost_objects_w:
             print >>sys.stderr, "Lost object %r" % (w_obj, )
             leaking = True
-        for llvalue in set(ll2ctypes.ALLOCATED.values()) - self.frozen_ll2callocations:
-            if getattr(llvalue, "_traceback", None): # this means that the allocation should be tracked
-                leaking = True
-                print >>sys.stderr, "Did not deallocate %r (ll2ctypes)" % (llvalue, )
-                print >>sys.stderr, "\t" + "\n\t".join(llvalue._traceback.splitlines())
-        for llvalue in lltype.ALLOCATED.keys():
-            leaking = True
-            print >>sys.stderr, "Did not deallocate %r (llvalue)" % (llvalue, )
-            print >>sys.stderr, "\t" + "\n\t".join(llvalue._traceback.splitlines())
-
-        lltype.stop_tracking_allocations()
+        # the actual low-level leak checking is done by pypy.tool.leakfinder,
+        # enabled automatically by pypy.conftest.
         return leaking
 
 class AppTestCpythonExtensionBase(LeakCheckingTest):

Modified: pypy/trunk/pypy/module/thread/ll_thread.py
==============================================================================
--- pypy/trunk/pypy/module/thread/ll_thread.py	(original)
+++ pypy/trunk/pypy/module/thread/ll_thread.py	Wed Oct 20 11:44:02 2010
@@ -111,7 +111,7 @@
             c_thread_releaselock(self._lock)
 
     def __del__(self):
-        lltype.free(self._lock, flavor='raw')
+        lltype.free(self._lock, flavor='raw', track_allocation=False)
 
 # ____________________________________________________________
 #
@@ -128,10 +128,13 @@
 null_ll_lock = lltype.nullptr(TLOCKP.TO)
 
 def allocate_ll_lock():
-    ll_lock = lltype.malloc(TLOCKP.TO, flavor='raw')
+    # track_allocation=False here; be careful to lltype.free() it.  The
+    # reason it is set to False is that we get it from all app-level
+    # lock objects, as well as from the GIL, which exists at shutdown.
+    ll_lock = lltype.malloc(TLOCKP.TO, flavor='raw', track_allocation=False)
     res = c_thread_lock_init(ll_lock)
     if res == -1:
-        lltype.free(ll_lock, flavor='raw')
+        lltype.free(ll_lock, flavor='raw', track_allocation=False)
         raise error("out of resources")
     return ll_lock
 

Modified: pypy/trunk/pypy/rlib/rgc.py
==============================================================================
--- pypy/trunk/pypy/rlib/rgc.py	(original)
+++ pypy/trunk/pypy/rlib/rgc.py	Wed Oct 20 11:44:02 2010
@@ -170,7 +170,14 @@
         return hop.genop('gc_set_max_heap_size', [v_nbytes],
                          resulttype=lltype.Void)
 
-def can_move(p):    # NB. must not be called with NULL pointers
+def can_move(p):
+    """Check if the GC object 'p' is at an address that can move.
+    Must not be called with None.  With non-moving GCs, it is always False.
+    With some moving GCs like the SemiSpace GC, it is always True.
+    With other moving GCs like the MiniMark GC, it can be True for some
+    time, then False for the same object, when we are sure that it won't
+    move any more.
+    """
     return True
 
 class CanMoveEntry(ExtRegistryEntry):

Modified: pypy/trunk/pypy/rlib/test/test_rdynload.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_rdynload.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_rdynload.py	Wed Oct 20 11:44:02 2010
@@ -5,11 +5,18 @@
 
 class TestDLOperations:
     def test_dlopen(self):
-        py.test.raises(DLOpenError, "dlopen(rffi.str2charp('xxxxxxxxxxxx'))")
-        assert dlopen(rffi.str2charp(get_libc_name()))
+        s = rffi.str2charp('xxxxxxxxxxxx')
+        py.test.raises(DLOpenError, "dlopen(s)")
+        rffi.free_charp(s)
+        #
+        s = rffi.str2charp(get_libc_name())
+        assert dlopen(s)
+        rffi.free_charp(s)
 
     def test_dlsym(self):
-        lib = dlopen(rffi.str2charp(get_libc_name()))
+        s = rffi.str2charp(get_libc_name())
+        lib = dlopen(s)
+        rffi.free_charp(s)
         handle = rffi.cast(lltype.Ptr(lltype.FuncType([lltype.Signed],
                            lltype.Signed)), dlsym(lib, 'abs'))
         assert 1 == handle(1)

Modified: pypy/trunk/pypy/rlib/test/test_rzlib.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_rzlib.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_rzlib.py	Wed Oct 20 11:44:02 2010
@@ -189,6 +189,8 @@
     assert unused3 == len('more_garbage')
     assert data3 == ''
 
+    rzlib.deflateEnd(stream)
+
 
 def test_decompress_max_length():
     """
@@ -205,6 +207,8 @@
     assert finished2 is True
     assert unused2 == 0
 
+    rzlib.deflateEnd(stream)
+
 
 def test_cornercases():
     """
@@ -215,6 +219,7 @@
     bytes += rzlib.compress(stream, "")
     bytes += rzlib.compress(stream, "", rzlib.Z_FINISH)
     assert zlib.decompress(bytes) == ""
+    rzlib.deflateEnd(stream)
 
     stream = rzlib.inflateInit()
     data, finished, unused = rzlib.decompress(stream, "")
@@ -228,3 +233,4 @@
         assert finished is False
         assert unused > 0
         buf = buf[-unused:]
+    rzlib.deflateEnd(stream)

Modified: pypy/trunk/pypy/rpython/llinterp.py
==============================================================================
--- pypy/trunk/pypy/rpython/llinterp.py	(original)
+++ pypy/trunk/pypy/rpython/llinterp.py	Wed Oct 20 11:44:02 2010
@@ -48,8 +48,7 @@
 
     current_interpreter = None
 
-    def __init__(self, typer, tracing=True, exc_data_ptr=None,
-                 malloc_check=True):
+    def __init__(self, typer, tracing=True, exc_data_ptr=None):
         self.bindings = {}
         self.typer = typer
         # 'heap' is module or object that provides malloc, etc for lltype ops
@@ -57,9 +56,7 @@
         self.exc_data_ptr = exc_data_ptr
         self.frame_stack = []
         self.tracer = None
-        self.malloc_check = malloc_check
         self.frame_class = LLFrame
-        self.mallocs = {}
         if tracing:
             self.tracer = Tracer()
 
@@ -163,24 +160,6 @@
             return self.exc_data_ptr
         return None
 
-    def remember_malloc(self, ptr, llframe):
-        # err....
-        self.mallocs[ptr._obj] = llframe
-
-    def remember_free(self, ptr):
-        try:
-            del self.mallocs[ptr._obj]
-        except KeyError:
-            self._rehash_mallocs()
-            del self.mallocs[ptr._obj]
-
-    def _rehash_mallocs(self):
-        # rehashing is needed because some objects' hash may change
-        # when being turned to 
-        items = self.mallocs.items()
-        self.mallocs = {}
-        self.mallocs.update(items)
-
     def _store_exception(self, exc):
         raise PleaseOverwriteStoreException("You just invoked ll2ctypes callback without overwriting _store_exception on llinterpreter")
 
@@ -726,13 +705,13 @@
     def op_malloc(self, obj, flags):
         flavor = flags['flavor']
         zero = flags.get('zero', False)
+        track_allocation = flags.get('track_allocation', True)
         if flavor == "stack":
             result = self.heap.malloc(obj, zero=zero, flavor='raw')
             self.alloca_objects.append(result)
             return result
-        ptr = self.heap.malloc(obj, zero=zero, flavor=flavor)
-        if flavor == 'raw' and self.llinterpreter.malloc_check:
-            self.llinterpreter.remember_malloc(ptr, self)
+        ptr = self.heap.malloc(obj, zero=zero, flavor=flavor,
+                               track_allocation=track_allocation)
         return ptr
 
     def op_malloc_varsize(self, obj, flags, size):
@@ -741,8 +720,6 @@
         assert flavor in ('gc', 'raw')
         try:
             ptr = self.heap.malloc(obj, size, zero=zero, flavor=flavor)
-            if flavor == 'raw' and self.llinterpreter.malloc_check:
-                self.llinterpreter.remember_malloc(ptr, self)
             return ptr
         except MemoryError:
             self.make_llexception()
@@ -759,11 +736,10 @@
         zero = flags.get('zero', False)
         return self.heap.malloc_nonmovable(TYPE, size, zero=zero)
 
-    def op_free(self, obj, flavor):
-        assert isinstance(flavor, str)
-        if flavor == 'raw' and self.llinterpreter.malloc_check:
-            self.llinterpreter.remember_free(obj)
-        self.heap.free(obj, flavor=flavor)
+    def op_free(self, obj, flags):
+        assert flags['flavor'] == 'raw'
+        track_allocation = flags.get('track_allocation', True)
+        self.heap.free(obj, flavor='raw', track_allocation=track_allocation)
 
     def op_shrink_array(self, obj, smallersize):
         return self.heap.shrink_array(obj, smallersize)
@@ -1037,6 +1013,13 @@
     def op_stack_malloc(self, size): # mmh
         raise NotImplementedError("backend only")
 
+    def op_track_alloc_start(self, addr):
+        # we don't do tracking at this level
+        checkadr(addr)
+
+    def op_track_alloc_stop(self, addr):
+        checkadr(addr)
+
     # ____________________________________________________________
     # Overflow-detecting variants
 

Modified: pypy/trunk/pypy/rpython/lltypesystem/llmemory.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/llmemory.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/llmemory.py	Wed Oct 20 11:44:02 2010
@@ -105,11 +105,13 @@
         if (isinstance(self.TYPE, lltype.ContainerType)
             and self.TYPE._gckind == 'gc'):
             assert self.repeat == 1
-            p = lltype.malloc(self.TYPE, flavor='raw', zero=zero)
+            p = lltype.malloc(self.TYPE, flavor='raw', zero=zero,
+                              track_allocation=False)
             return cast_ptr_to_adr(p)
         else:
             T = lltype.FixedSizeArray(self.TYPE, self.repeat)
-            p = lltype.malloc(T, flavor='raw', zero=zero)
+            p = lltype.malloc(T, flavor='raw', zero=zero,
+                              track_allocation=False)
             array_adr = cast_ptr_to_adr(p)
             return array_adr + ArrayItemsOffset(T)
 
@@ -288,7 +290,8 @@
             count = 0
         p = lltype.malloc(parenttype or self.TYPE, count,
                           immortal = self.TYPE._gckind == 'raw',
-                          zero = zero)
+                          zero = zero,
+                          track_allocation = False)
         return cast_ptr_to_adr(p)
 
     def raw_memcopy(self, srcadr, dstadr):

Modified: pypy/trunk/pypy/rpython/lltypesystem/lloperation.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/lloperation.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/lloperation.py	Wed Oct 20 11:44:02 2010
@@ -403,6 +403,8 @@
     'raw_load':             LLOp(sideeffects=False),
     'raw_store':            LLOp(),
     'stack_malloc':         LLOp(), # mmh
+    'track_alloc_start':    LLOp(),
+    'track_alloc_stop':     LLOp(),
     'adr_add':              LLOp(canfold=True),
     'adr_sub':              LLOp(canfold=True),
     'adr_delta':            LLOp(canfold=True),

Modified: pypy/trunk/pypy/rpython/lltypesystem/lltype.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/lltype.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/lltype.py	Wed Oct 20 11:44:02 2010
@@ -1,7 +1,3 @@
-import StringIO
-import traceback
-import sys
-
 import py
 from pypy.rlib.rarithmetic import (r_int, r_uint, intmask, r_singlefloat,
                                    r_ulonglong, r_longlong, base_int,
@@ -10,25 +6,13 @@
 from pypy.tool.uid import Hashable
 from pypy.tool.tls import tlsobject
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 from types import NoneType
 from sys import maxint
 import weakref
 
 TLS = tlsobject()
 
-# Track allocations to detect memory leaks
-# Don't track 'gc' and immortal mallocs
-TRACK_ALLOCATIONS = False
-ALLOCATED = identity_dict()
-
-def start_tracking_allocations():
-    global TRACK_ALLOCATIONS
-    TRACK_ALLOCATIONS = True
-    ALLOCATED.clear()
-
-def stop_tracking_allocations():
-    global TRACK_ALLOCATIONS
-    TRACK_ALLOCATIONS = False
 
 class _uninitialized(object):
     def __init__(self, TYPE):
@@ -1382,41 +1366,21 @@
     __slots__ = ('_TYPE',
                  '_parent_type', '_parent_index', '_keepparent',
                  '_wrparent',
-                 '__weakref__', '_traceback',
-                 '__storage')
+                 '__weakref__',
+                 '_storage')
 
-    def __init__(self, TYPE, track_allocation=None):
+    def __init__(self, TYPE):
         self._wrparent = None
         self._TYPE = TYPE
         self._storage = True    # means "use default storage", as opposed to:
                                 #    None            - container was freed
                                 #     - using ctypes
                                 #                      (see ll2ctypes.py)
-        if track_allocation is not False and TRACK_ALLOCATIONS:
-            self._traceback = self._get_traceback()
-            ALLOCATED[self] = None
-        else:
-            self._traceback = None
-
-    def _get_traceback(self):
-        frame = sys._getframe().f_back.f_back.f_back.f_back
-        sio = StringIO.StringIO()
-        traceback.print_stack(frame, file=sio)
-        return sio.getvalue()
 
     def _free(self):
         self._check()   # no double-frees
         self._storage = None
 
-    def _storage_get(self):
-        return self.__storage
-
-    def _storage_set(self, value):
-        self.__storage = value
-        if value is not True and self in ALLOCATED:
-            del ALLOCATED[self]
-    _storage = property(_storage_get, _storage_set)
-
     def _was_freed(self):
         if self._storage is None:
             return True
@@ -1495,12 +1459,12 @@
 
     __slots__ = ('_hash_cache_', '_compilation_info')
 
-    def __new__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None, track_allocation=None):
+    def __new__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None):
         my_variety = _struct_variety(TYPE._names)
         return object.__new__(my_variety)
 
-    def __init__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None, track_allocation=None):
-        _parentable.__init__(self, TYPE, track_allocation)
+    def __init__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None):
+        _parentable.__init__(self, TYPE)
         if n is not None and TYPE._arrayfld is None:
             raise TypeError("%r is not variable-sized" % (TYPE,))
         if n is None and TYPE._arrayfld is not None:
@@ -1508,8 +1472,7 @@
         first, FIRSTTYPE = TYPE._first_struct()
         for fld, typ in TYPE._flds.items():
             if fld == TYPE._arrayfld:
-                value = _array(typ, n, initialization=initialization, parent=self, parentindex=fld,
-                               track_allocation=track_allocation)
+                value = _array(typ, n, initialization=initialization, parent=self, parentindex=fld)
             else:
                 value = typ._allocate(initialization=initialization, parent=self, parentindex=fld)
             setattr(self, fld, value)
@@ -1570,12 +1533,12 @@
 
     __slots__ = ('items',)
 
-    def __init__(self, TYPE, n, initialization=None, parent=None, parentindex=None, track_allocation=None):
+    def __init__(self, TYPE, n, initialization=None, parent=None, parentindex=None):
         if not isinstance(n, int):
             raise TypeError, "array length must be an int"
         if n < 0:
             raise ValueError, "negative array length"
-        _parentable.__init__(self, TYPE, track_allocation)
+        _parentable.__init__(self, TYPE)
         try:
             myrange = range(n)
         except OverflowError:
@@ -1642,7 +1605,7 @@
     _cache = weakref.WeakKeyDictionary()  # parentarray -> {subarrays}
 
     def __init__(self, TYPE, parent, baseoffset_or_fieldname):
-        _parentable.__init__(self, TYPE, track_allocation=False)
+        _parentable.__init__(self, TYPE)
         self._setparentstructure(parent, baseoffset_or_fieldname)
         # Keep the parent array alive, we share the same allocation.
         # Don't do it if we are inside a GC object, though -- it's someone
@@ -1650,6 +1613,13 @@
         if typeOf(top_container(parent))._gckind == 'raw':
             self._keepparent = parent
 
+    def __str__(self):
+        parent = self._wrparent()
+        if parent is None:
+            return '_subarray at %s in already freed' % (self._parent_index,)
+        return '_subarray at %r in %s' % (self._parent_index,
+                                          parent._TYPE)
+
     def __repr__(self):
         parent = self._wrparent()
         if parent is None:
@@ -1863,8 +1833,9 @@
         return id(self.value)
 
 
-def malloc(T, n=None, flavor='gc', immortal=False, zero=False):
-    assert flavor != 'cpy'
+def malloc(T, n=None, flavor='gc', immortal=False, zero=False,
+           track_allocation=True):
+    assert flavor in ('gc', 'raw')
     if zero or immortal:
         initialization = 'example'
     elif flavor == 'raw':
@@ -1872,9 +1843,9 @@
     else:
         initialization = 'malloc'
     if isinstance(T, Struct):
-        o = _struct(T, n, initialization=initialization, track_allocation=flavor == "raw" and not immortal)
+        o = _struct(T, n, initialization=initialization)
     elif isinstance(T, Array):
-        o = _array(T, n, initialization=initialization, track_allocation=flavor == "raw" and not immortal)
+        o = _array(T, n, initialization=initialization)
     elif isinstance(T, OpaqueType):
         assert n is None
         o = _opaque(T, initialization=initialization)
@@ -1882,15 +1853,19 @@
         raise TypeError, "malloc for Structs and Arrays only"
     if T._gckind != 'gc' and not immortal and flavor.startswith('gc'):
         raise TypeError, "gc flavor malloc of a non-GC non-immortal structure"
+    if flavor == "raw" and not immortal and track_allocation:
+        leakfinder.remember_malloc(o, framedepth=2)
     solid = immortal or not flavor.startswith('gc') # immortal or non-gc case
     return _ptr(Ptr(T), o, solid)
 
-def free(p, flavor):
+def free(p, flavor, track_allocation=True):
     if flavor.startswith('gc'):
         raise TypeError, "gc flavor free"
     T = typeOf(p)
     if not isinstance(T, Ptr) or p._togckind() != 'raw':
         raise TypeError, "free(): only for pointers to non-gc containers"
+    if track_allocation:
+        leakfinder.remember_free(p._obj0)
     p._obj0._free()
 
 def functionptr(TYPE, name, **attrs):

Modified: pypy/trunk/pypy/rpython/lltypesystem/rclass.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/rclass.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/rclass.py	Wed Oct 20 11:44:02 2010
@@ -420,7 +420,7 @@
         return cast_pointer(self.lowleveltype, result)
 
     def create_instance(self):
-        return malloc(self.object_type, flavor=self.gcflavor)
+        return malloc(self.object_type, flavor=self.gcflavor, immortal=True)
 
     def initialize_prebuilt_data(self, value, classdef, result):
         if self.classdef is not None:

Modified: pypy/trunk/pypy/rpython/lltypesystem/rffi.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/rffi.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/rffi.py	Wed Oct 20 11:44:02 2010
@@ -644,10 +644,18 @@
         """
         Either free a non-moving buffer or keep the original storage alive.
         """
-        if rgc.can_move(data):
+        # We cannot rely on rgc.can_move(data) here, because its result
+        # might have changed since get_nonmovingbuffer().  Instead we check
+        # if 'buf' points inside 'data'.  This is only possible if we
+        # followed the 2nd case in get_nonmovingbuffer(); in the first case,
+        # 'buf' points to its own raw-malloced memory.
+        data = llstrtype(data)
+        data_start = cast_ptr_to_adr(data) + \
+            offsetof(STRTYPE, 'chars') + itemoffsetof(STRTYPE.chars, 0)
+        followed_2nd_path = (buf == cast(TYPEP, data_start))
+        keepalive_until_here(data)
+        if not followed_2nd_path:
             lltype.free(buf, flavor='raw')
-        else:
-            keepalive_until_here(data)
 
     # int -> (char*, str)
     def alloc_buffer(count):

Modified: pypy/trunk/pypy/rpython/lltypesystem/test/test_ll2ctypes.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/test/test_ll2ctypes.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/test/test_ll2ctypes.py	Wed Oct 20 11:44:02 2010
@@ -765,6 +765,7 @@
         assert abs(float(b[1]) - 1.1) < 1E-6
         assert isinstance(b[2], rffi.r_singlefloat)
         assert abs(float(b[2]) - 2.2) < 1E-6
+        lltype.free(a, flavor='raw')
 
     def test_different_signatures(self):
         if sys.platform=='win32':
@@ -879,6 +880,7 @@
         qsort(rffi.cast(rffi.VOIDP, a), 5, rffi.sizeof(rffi.INT), compare)
         for i in range(5):
             assert a[i] == i + 1
+        lltype.free(a, flavor='raw')
 
     def test_array_type_bug(self):
         A = lltype.Array(lltype.Signed)

Modified: pypy/trunk/pypy/rpython/lltypesystem/test/test_llmemory.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/test/test_llmemory.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/test/test_llmemory.py	Wed Oct 20 11:44:02 2010
@@ -324,12 +324,14 @@
     p_t = lltype.malloc(T)
     assert p_t.s == lltype.nullptr(S)
     # raw malloc does not
-    p_raw_t = lltype.malloc(T, flavor="raw")
-    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.s")
+    U = lltype.Struct("U", ('x', lltype.Signed))
+    p_raw_t = lltype.malloc(U, flavor="raw")
+    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.x")
+    lltype.free(p_raw_t, flavor="raw")
     # this sort of raw_malloc too
-    p_raw_t = cast_adr_to_ptr(raw_malloc(sizeof(T)), lltype.Ptr(T))
-    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.s")
-    
+    p_raw_t = cast_adr_to_ptr(raw_malloc(sizeof(U)), lltype.Ptr(U))
+    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.x")
+
 
 def test_raw_malloc_signed_bunch():
     adr = raw_malloc(sizeof(lltype.Signed) * 50)
@@ -601,7 +603,8 @@
     a = lltype.malloc(A, flavor='raw')
     src = cast_ptr_to_adr(a) + itemoffsetof(A, 0)
     raw_memclear(src, sizeof(lltype.Signed) * 0)
-    
+    lltype.free(a, flavor="raw")
+
 def test_nonneg():
     S1 = lltype.GcStruct('S1', ('x', lltype.Float))
     A1 = lltype.GcArray(lltype.Float)

Modified: pypy/trunk/pypy/rpython/lltypesystem/test/test_lltype.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/test/test_lltype.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/test/test_lltype.py	Wed Oct 20 11:44:02 2010
@@ -2,6 +2,7 @@
 from pypy.rpython.lltypesystem.lltype import *
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 
 def isweak(p, T):
     try:
@@ -804,22 +805,20 @@
 
 
 class TestTrackAllocation:
-    def setup_method(self, func):
-        start_tracking_allocations()
-
-    def teardown_method(self, func):
-        assert not lltype.ALLOCATED, "Memory was not correctly freed"
-        stop_tracking_allocations()
+    def test_automatic_tracking(self):
+        # calls to start_tracking_allocations/stop_tracking_allocations
+        # should occur automatically from pypy/conftest.py.  Check that.
+        assert leakfinder.TRACK_ALLOCATIONS
 
     def test_track_allocation(self):
         """A malloc'd buffer fills the ALLOCATED dictionary"""
-        assert lltype.TRACK_ALLOCATIONS
-        assert not lltype.ALLOCATED
+        assert leakfinder.TRACK_ALLOCATIONS
+        assert not leakfinder.ALLOCATED
         buf = malloc(Array(Signed), 1, flavor="raw")
-        assert len(lltype.ALLOCATED) == 1
-        assert lltype.ALLOCATED.keys() == [buf._obj]
+        assert len(leakfinder.ALLOCATED) == 1
+        assert leakfinder.ALLOCATED.keys() == [buf._obj]
         free(buf, flavor="raw")
-        assert not lltype.ALLOCATED
+        assert not leakfinder.ALLOCATED
 
     def test_str_from_buffer(self):
         """gc-managed memory does not need to be freed"""
@@ -828,16 +827,22 @@
         for i in range(size): raw_buf[i] = 'a'
         rstr = rffi.str_from_buffer(raw_buf, gc_buf, size, size)
         rffi.keep_buffer_alive_until_here(raw_buf, gc_buf)
-        assert not lltype.ALLOCATED
+        assert not leakfinder.ALLOCATED
 
     def test_leak_traceback(self):
         """Test info stored for allocated items"""
         buf = malloc(Array(Signed), 1, flavor="raw")
-        traceback = lltype.ALLOCATED.keys()[0]._traceback
+        traceback = leakfinder.ALLOCATED.values()[0]
         lines = traceback.splitlines()
         assert 'malloc(' in lines[-1] and 'flavor="raw")' in lines[-1]
 
-        # XXX The traceback should not be too long
+        # The traceback should not be too long
         print traceback
 
         free(buf, flavor="raw")
+
+    def test_no_tracking(self):
+        p1 = malloc(Array(Signed), 1, flavor='raw', track_allocation=False)
+        p2 = malloc(Array(Signed), 1, flavor='raw', track_allocation=False)
+        free(p2, flavor='raw', track_allocation=False)
+        # p1 is not freed

Modified: pypy/trunk/pypy/rpython/lltypesystem/test/test_rffi.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/test/test_rffi.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/test/test_rffi.py	Wed Oct 20 11:44:02 2010
@@ -9,7 +9,7 @@
 from pypy.rpython.lltypesystem.rstr import STR
 from pypy.rpython.lltypesystem import lltype
 from pypy.tool.udir import udir
-from pypy.rpython.test.test_llinterp import interpret, MallocMismatch
+from pypy.rpython.test.test_llinterp import interpret
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.annotation.annrpython import RPythonAnnotator
 from pypy.rpython.rtyper import RPythonTyper

Modified: pypy/trunk/pypy/rpython/memory/gc/minimarkpage.py
==============================================================================
--- pypy/trunk/pypy/rpython/memory/gc/minimarkpage.py	(original)
+++ pypy/trunk/pypy/rpython/memory/gc/minimarkpage.py	Wed Oct 20 11:44:02 2010
@@ -100,11 +100,14 @@
         # allocation of the given size.
         length = small_request_threshold / WORD + 1
         self.page_for_size = lltype.malloc(rffi.CArray(PAGE_PTR), length,
-                                           flavor='raw', zero=True)
+                                           flavor='raw', zero=True,
+                                           immortal=True)
         self.full_page_for_size = lltype.malloc(rffi.CArray(PAGE_PTR), length,
-                                                flavor='raw', zero=True)
+                                                flavor='raw', zero=True,
+                                                immortal=True)
         self.nblocks_for_size = lltype.malloc(rffi.CArray(lltype.Signed),
-                                              length, flavor='raw')
+                                              length, flavor='raw',
+                                              immortal=True)
         self.hdrsize = llmemory.raw_malloc_usage(llmemory.sizeof(PAGE_HEADER))
         assert page_size > self.hdrsize
         self.nblocks_for_size[0] = 0    # unused
@@ -114,11 +117,13 @@
         self.max_pages_per_arena = arena_size // page_size
         self.arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR),
                                           self.max_pages_per_arena,
-                                          flavor='raw', zero=True)
+                                          flavor='raw', zero=True,
+                                          immortal=True)
         # this is used in mass_free() only
         self.old_arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR),
                                               self.max_pages_per_arena,
-                                              flavor='raw', zero=True)
+                                              flavor='raw', zero=True,
+                                              immortal=True)
         #
         # the arena currently consumed; it must have at least one page
         # available, or be NULL.  The arena object that we point to is
@@ -281,7 +286,7 @@
         npages = (arena_end - firstpage) // self.page_size
         #
         # Allocate an ARENA object and initialize it
-        arena = lltype.malloc(ARENA, flavor='raw')
+        arena = lltype.malloc(ARENA, flavor='raw', track_allocation=False)
         arena.base = arena_base
         arena.nfreepages = 0        # they are all uninitialized pages
         arena.totalpages = npages
@@ -332,7 +337,7 @@
                     #
                     # The whole arena is empty.  Free it.
                     llarena.arena_free(arena.base)
-                    lltype.free(arena, flavor='raw')
+                    lltype.free(arena, flavor='raw', track_allocation=False)
                     #
                 else:
                     # Insert 'arena' in the correct arenas_lists[n]

Modified: pypy/trunk/pypy/rpython/memory/gctransform/asmgcroot.py
==============================================================================
--- pypy/trunk/pypy/rpython/memory/gctransform/asmgcroot.py	(original)
+++ pypy/trunk/pypy/rpython/memory/gctransform/asmgcroot.py	Wed Oct 20 11:44:02 2010
@@ -75,7 +75,8 @@
             key = (TYPE, num)
             if key not in sradict:
                 CONTAINER = lltype.FixedSizeArray(TYPE, 1)
-                p = lltype.malloc(CONTAINER, flavor='raw', zero=True)
+                p = lltype.malloc(CONTAINER, flavor='raw', zero=True,
+                                  immortal=True)
                 sradict[key] = Constant(p, lltype.Ptr(CONTAINER))
             sra.append(sradict[key])
         #

Modified: pypy/trunk/pypy/rpython/memory/gctransform/transform.py
==============================================================================
--- pypy/trunk/pypy/rpython/memory/gctransform/transform.py	(original)
+++ pypy/trunk/pypy/rpython/memory/gctransform/transform.py	Wed Oct 20 11:44:02 2010
@@ -430,7 +430,8 @@
         return self.parenttransformer.gct_malloc_varsize(hop)
     
     def gct_free(self, hop):
-        flavor = hop.spaceop.args[1].value
+        flags = hop.spaceop.args[1].value
+        flavor = flags['flavor']
         assert flavor == 'raw'
         return self.parenttransformer.gct_free(hop)
 
@@ -532,6 +533,8 @@
                           resulttype=llmemory.Address)
         if flags.get('zero'):
             hop.genop("raw_memclear", [v_raw, c_size])
+        if flags.get('track_allocation', True):
+            hop.genop("track_alloc_start", [v_raw])
         return v_raw
 
     def gct_fv_stack_malloc(self, hop, flags, TYPE, c_size):
@@ -602,15 +605,20 @@
                                [self.raw_malloc_varsize_ptr, v_length,
                                 c_const_size, c_item_size, c_offset_to_length],
                                resulttype=llmemory.Address)
+        if flags.get('track_allocation', True):
+            hop.genop("track_alloc_start", [v_raw])
         return v_raw
 
     def gct_free(self, hop):
         op = hop.spaceop
-        flavor = op.args[1].value
+        flags = op.args[1].value
+        flavor = flags['flavor']
         v = op.args[0]
         assert flavor != 'cpy', "cannot free CPython objects directly"
         if flavor == 'raw':
             v = hop.genop("cast_ptr_to_adr", [v], resulttype=llmemory.Address)
+            if flags.get('track_allocation', True):
+                hop.genop("track_alloc_stop", [v])
             hop.genop('raw_free', [v])
         else:
             assert False, "%s has no support for free with flavor %r" % (self, flavor)           

Modified: pypy/trunk/pypy/rpython/memory/gcwrapper.py
==============================================================================
--- pypy/trunk/pypy/rpython/memory/gcwrapper.py	(original)
+++ pypy/trunk/pypy/rpython/memory/gcwrapper.py	Wed Oct 20 11:44:02 2010
@@ -42,7 +42,8 @@
     #
     # Interface for the llinterp
     #
-    def malloc(self, TYPE, n=None, flavor='gc', zero=False):
+    def malloc(self, TYPE, n=None, flavor='gc', zero=False,
+               track_allocation=True):
         if flavor == 'gc':
             typeid = self.get_type_id(TYPE)
             addr = self.gc.malloc(typeid, n, zero=zero)
@@ -51,7 +52,8 @@
                 gctypelayout.zero_gc_pointers(result)
             return result
         else:
-            return lltype.malloc(TYPE, n, flavor=flavor, zero=zero)
+            return lltype.malloc(TYPE, n, flavor=flavor, zero=zero,
+                                 track_allocation=track_allocation)
 
     def malloc_nonmovable(self, TYPE, n=None, zero=False):
         typeid = self.get_type_id(TYPE)
@@ -69,9 +71,10 @@
             return self.gc.shrink_array(addr, smallersize)
         return False
 
-    def free(self, TYPE, flavor='gc'):
+    def free(self, TYPE, flavor='gc', track_allocation=True):
         assert flavor != 'gc'
-        return lltype.free(TYPE, flavor=flavor)
+        return lltype.free(TYPE, flavor=flavor,
+                           track_allocation=track_allocation)
 
     def setfield(self, obj, fieldname, fieldvalue):
         STRUCT = lltype.typeOf(obj).TO

Modified: pypy/trunk/pypy/rpython/memory/support.py
==============================================================================
--- pypy/trunk/pypy/rpython/memory/support.py	(original)
+++ pypy/trunk/pypy/rpython/memory/support.py	Wed Oct 20 11:44:02 2010
@@ -30,7 +30,8 @@
                 # we zero-initialize the chunks to make the translation
                 # backends happy, but we don't need to do it at run-time.
                 zero = not we_are_translated()
-                return lltype.malloc(CHUNK, flavor="raw", zero=zero)
+                return lltype.malloc(CHUNK, flavor="raw", zero=zero,
+                                     track_allocation=False)
                 
             result = self.free_list
             self.free_list = result.next
@@ -44,7 +45,7 @@
                 # Don't cache the old chunks but free them immediately.
                 # Helps debugging, and avoids that old chunks full of
                 # addresses left behind by a test end up in genc...
-                lltype.free(chunk, flavor="raw")
+                lltype.free(chunk, flavor="raw", track_allocation=False)
 
     unused_chunks = FreeList()
     cache[chunk_size] = unused_chunks, null_chunk

Modified: pypy/trunk/pypy/rpython/memory/test/test_gc.py
==============================================================================
--- pypy/trunk/pypy/rpython/memory/test/test_gc.py	(original)
+++ pypy/trunk/pypy/rpython/memory/test/test_gc.py	Wed Oct 20 11:44:02 2010
@@ -368,6 +368,14 @@
         res = self.interpret(f, [4, 42])
         assert res == 12
 
+    def test_print_leak(self):
+        def f(n):
+            for i in range(n):
+                print i
+            return 42
+        res = self.interpret(f, [10])
+        assert res == 42
+
     def test_weakref_across_minor_collection(self):
         import weakref
         class A:

Modified: pypy/trunk/pypy/rpython/rbuiltin.py
==============================================================================
--- pypy/trunk/pypy/rpython/rbuiltin.py	(original)
+++ pypy/trunk/pypy/rpython/rbuiltin.py	Wed Oct 20 11:44:02 2010
@@ -345,17 +345,22 @@
 BUILTIN_TYPER[object.__init__] = rtype_object__init__
 # annotation of low-level types
 
-def rtype_malloc(hop, i_flavor=None, i_zero=None):
+def rtype_malloc(hop, i_flavor=None, i_zero=None, i_track_allocation=None):
     assert hop.args_s[0].is_constant()
     vlist = [hop.inputarg(lltype.Void, arg=0)]
     opname = 'malloc'
-    v_flavor, v_zero = parse_kwds(hop, (i_flavor, lltype.Void), (i_zero, None))
+    v_flavor, v_zero, v_track_allocation = parse_kwds(hop,
+        (i_flavor, lltype.Void),
+        (i_zero, None),
+        (i_track_allocation, None))
 
     flags = {'flavor': 'gc'}
     if v_flavor is not None:
         flags['flavor'] = v_flavor.value
     if i_zero is not None:
         flags['zero'] = v_zero.value
+    if i_track_allocation is not None:
+        flags['track_allocation'] = v_track_allocation.value
     vlist.append(hop.inputconst(lltype.Void, flags))
         
     if hop.nb_args == 2:
@@ -366,10 +371,19 @@
     hop.exception_is_here()
     return hop.genop(opname, vlist, resulttype = hop.r_result.lowleveltype)
 
-def rtype_free(hop, i_flavor):
-    assert i_flavor == 1
+def rtype_free(hop, i_flavor, i_track_allocation=None):
+    vlist = [hop.inputarg(hop.args_r[0], arg=0)]
+    v_flavor, v_track_allocation = parse_kwds(hop,
+        (i_flavor, lltype.Void),
+        (i_track_allocation, None))
+    #
+    assert v_flavor is not None and v_flavor.value == 'raw'
+    flags = {'flavor': 'raw'}
+    if i_track_allocation is not None:
+        flags['track_allocation'] = v_track_allocation.value
+    vlist.append(hop.inputconst(lltype.Void, flags))
+    #
     hop.exception_cannot_occur()
-    vlist = hop.inputargs(hop.args_r[0], lltype.Void)
     hop.genop('free', vlist)
 
 def rtype_const_result(hop):
@@ -584,8 +598,9 @@
     vinst, = hop.inputargs(hop.args_r[0])
     flavor = hop.args_r[0].gcflavor
     assert flavor != 'gc'
-    cflavor = hop.inputconst(lltype.Void, flavor)
-    return hop.genop('free', [vinst, cflavor])
+    flags = {'flavor': flavor}
+    cflags = hop.inputconst(lltype.Void, flags)
+    return hop.genop('free', [vinst, cflags])
     
 BUILTIN_TYPER[objectmodel.free_non_gc_object] = rtype_free_non_gc_object
 

Modified: pypy/trunk/pypy/rpython/test/test_llinterp.py
==============================================================================
--- pypy/trunk/pypy/rpython/test/test_llinterp.py	(original)
+++ pypy/trunk/pypy/rpython/test/test_llinterp.py	Wed Oct 20 11:44:02 2010
@@ -12,13 +12,11 @@
 from pypy.annotation.model import lltype_to_annotation
 from pypy.rlib.rarithmetic import r_uint, ovfcheck
 from pypy.rpython.ootypesystem import ootype
+from pypy.tool import leakfinder
 from pypy import conftest
 
 # switch on logging of interp to show more info on failing tests
 
-class MallocMismatch(Exception):
-    pass
-
 def setup_module(mod):
     mod.logstate = py.log._getstate()
     py.log.setconsumer("llinterp", py.log.STDOUT)
@@ -72,7 +70,7 @@
 
 def get_interpreter(func, values, view='auto', viewbefore='auto', policy=None,
                     someobjects=False, type_system="lltype", backendopt=False,
-                    config=None, malloc_check=True, **extraconfigopts):
+                    config=None, **extraconfigopts):
     extra_key = [(key, value) for key, value in extraconfigopts.iteritems()]
     extra_key.sort()
     extra_key = tuple(extra_key)
@@ -97,7 +95,7 @@
                                    viewbefore, policy, type_system=type_system,
                                    backendopt=backendopt, config=config,
                                    **extraconfigopts)
-        interp = LLInterpreter(typer, malloc_check=malloc_check)
+        interp = LLInterpreter(typer)
         _tcache[key] = (t, interp, graph)
         # keep the cache small 
         _lastinterpreted.append(key) 
@@ -115,10 +113,17 @@
     interp, graph = get_interpreter(func, values, view, viewbefore, policy,
                                     someobjects, type_system=type_system,
                                     backendopt=backendopt, config=config,
-                                    malloc_check=malloc_check, **kwargs)
-    result = interp.eval_graph(graph, values)
-    if malloc_check and interp.mallocs:
-        raise MallocMismatch(interp.mallocs)
+                                    **kwargs)
+    if not malloc_check:
+        result = interp.eval_graph(graph, values)
+    else:
+        prev = leakfinder.start_tracking_allocations()
+        try:
+            result = interp.eval_graph(graph, values)
+        finally:
+            leaks = leakfinder.stop_tracking_allocations(False, prev)
+        if leaks:
+            raise leakfinder.MallocMismatch(leaks)
     return result
 
 def interpret_raises(exc, func, values, view='auto', viewbefore='auto',
@@ -418,6 +423,7 @@
             assert result
 
 def test_stack_malloc():
+    py.test.skip("stack-flavored mallocs no longer supported")
     class A(object):
         pass
     def f():
@@ -430,6 +436,7 @@
     assert result == 1
 
 def test_invalid_stack_access():
+    py.test.skip("stack-flavored mallocs no longer supported")
     class A(object):
         pass
     globala = A()
@@ -605,7 +612,7 @@
         if x:
             free(t, flavor='raw')
     interpret(f, [1])
-    py.test.raises(MallocMismatch, "interpret(f, [0])")
+    py.test.raises(leakfinder.MallocMismatch, "interpret(f, [0])")
     
     def f():
         t1 = malloc(T, flavor='raw')

Modified: pypy/trunk/pypy/rpython/test/test_nongc.py
==============================================================================
--- pypy/trunk/pypy/rpython/test/test_nongc.py	(original)
+++ pypy/trunk/pypy/rpython/test/test_nongc.py	Wed Oct 20 11:44:02 2010
@@ -79,7 +79,7 @@
     py.test.raises(TypeError,rtyper.specialize) # results in an invalid cast
 
 def test_isinstance():
-    class A:
+    class A(object):
         _alloc_flavor_ = "raw"
     class B(A):
         pass
@@ -95,7 +95,24 @@
             o = B()
         else:
             o = C()
-        return 100*isinstance(o, A)+10*isinstance(o, B)+1*isinstance(o ,C)
+        res = 100*isinstance(o, A) + 10*isinstance(o, B) + 1*isinstance(o, C)
+        if i == 0:
+            pass
+        elif i == 1:
+            assert isinstance(o, A)
+            free_non_gc_object(o)
+        elif i == 2:
+            assert isinstance(o, B)
+            free_non_gc_object(o)
+        else:
+            assert isinstance(o, C)
+            free_non_gc_object(o)
+        return res
+
+    assert f(1) == 100
+    assert f(2) == 110
+    assert f(3) == 111
+    assert f(0) == 0
 
     a = RPythonAnnotator()
     #does not raise:
@@ -131,10 +148,14 @@
             d = b
         elif i == 2:
             e = c
-        return (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
+        res =  (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
                 0x0008*(a is e) | 0x0010*(b is c) | 0x0020*(b is d) |
                 0x0040*(b is e) | 0x0080*(c is d) | 0x0100*(c is e) |
                 0x0200*(d is e))
+        free_non_gc_object(a)
+        free_non_gc_object(b)
+        free_non_gc_object(c)
+        return res
     a = RPythonAnnotator()
     #does not raise:
     s = a.build_types(f, [int])
@@ -169,10 +190,13 @@
             d = b
         elif i == 2:
             e = c
-        return (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
+        res =  (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
                 0x0008*(a is e) | 0x0010*(b is c) | 0x0020*(b is d) |
                 0x0040*(b is e) | 0x0080*(c is d) | 0x0100*(c is e) |
                 0x0200*(d is e))
+        free_non_gc_object(a)
+        free_non_gc_object(b)
+        return res
     a = RPythonAnnotator()
     #does not raise:
     s = a.build_types(f, [int])

Modified: pypy/trunk/pypy/rpython/test/test_rptr.py
==============================================================================
--- pypy/trunk/pypy/rpython/test/test_rptr.py	(original)
+++ pypy/trunk/pypy/rpython/test/test_rptr.py	Wed Oct 20 11:44:02 2010
@@ -212,10 +212,31 @@
 
     S = Struct('S', ('x', Signed))
     def fn(n):
-        p = malloc(S, flavor='whatever')
+        p = malloc(S, flavor='raw')
         p.x = n
         result = p.x
-        free(p, flavor='whatever')
+        free(p, flavor='raw')
+        return n
+
+    res = interpret(fn, [23])
+    assert res == 23
+
+    S = Struct('S', ('x', Signed))
+    def fn(n):
+        p = malloc(S, flavor='raw', track_allocation=False)
+        p.x = n
+        result = p.x
+        return n
+
+    res = interpret(fn, [23])
+    assert res == 23
+
+    S = Struct('S', ('x', Signed))
+    def fn(n):
+        p = malloc(S, flavor='raw', track_allocation=False)
+        p.x = n
+        result = p.x
+        free(p, flavor='raw', track_allocation=False)
         return n
 
     res = interpret(fn, [23])

Modified: pypy/trunk/pypy/translator/c/src/g_include.h
==============================================================================
--- pypy/trunk/pypy/translator/c/src/g_include.h	(original)
+++ pypy/trunk/pypy/translator/c/src/g_include.h	Wed Oct 20 11:44:02 2010
@@ -53,6 +53,7 @@
 #  include "src/rtyper.h"
 #  include "src/debug_print.h"
 #  include "src/debug_traceback.h"
+#  include "src/debug_alloc.h"
 #ifndef AVR
 #  include "src/ll_os.h"
 #  include "src/ll_strtod.h"

Modified: pypy/trunk/pypy/translator/c/src/main.h
==============================================================================
--- pypy/trunk/pypy/translator/c/src/main.h	(original)
+++ pypy/trunk/pypy/translator/c/src/main.h	Wed Oct 20 11:44:02 2010
@@ -53,10 +53,16 @@
     }
 
     exitcode = STANDALONE_ENTRY_POINT(list);
+
+#ifdef RPY_ASSERT
+    pypy_debug_alloc_results();
+#endif
+
     if (RPyExceptionOccurred()) {
         /* print the RPython traceback */
         pypy_debug_catch_fatal_exception();
     }
+
     return exitcode;
 
  memory_out:

Modified: pypy/trunk/pypy/translator/c/test/test_genc.py
==============================================================================
--- pypy/trunk/pypy/translator/c/test/test_genc.py	(original)
+++ pypy/trunk/pypy/translator/c/test/test_genc.py	Wed Oct 20 11:44:02 2010
@@ -426,6 +426,7 @@
     if py.test.config.option.view:
         t.view()
     assert ' BarStruct ' in t.driver.cbuilder.c_source_filename.read()
+    free(foo, flavor="raw")
 
 def test_recursive_llhelper():
     from pypy.rpython.annlowlevel import llhelper
@@ -473,7 +474,7 @@
         return f(s)
     a_f = A(f, "f")
     a_g = A(g, "g")
-    t = lltype.malloc(STRUCT, flavor="raw")
+    t = lltype.malloc(STRUCT, flavor="raw", immortal=True)
     t.bar = llhelper(FTPTR, a_f.make_func())
     fn = compile(chooser, [bool])
     assert fn(True)

Modified: pypy/trunk/pypy/translator/c/test/test_lltyped.py
==============================================================================
--- pypy/trunk/pypy/translator/c/test/test_lltyped.py	(original)
+++ pypy/trunk/pypy/translator/c/test/test_lltyped.py	Wed Oct 20 11:44:02 2010
@@ -401,6 +401,7 @@
             for i in range(n):
                 p = malloc(S, flavor='raw', zero=True)
                 if p.x != 0 or p.y != 0:
+                    free(p, flavor='raw')
                     return -1
                 p.x = i
                 p.y = i
@@ -418,14 +419,16 @@
         def f(n):
             for length in range(n-1, -1, -1):
                 p = malloc(S, length, flavor='raw', zero=True)
-                if p.x != 0:
-                    return -1
-                p.x = n
-                for j in range(length):
-                    if p.y[j] != 0:
-                        return -3
-                    p.y[j] = n^j
-                free(p, flavor='raw')
+                try:
+                    if p.x != 0:
+                        return -1
+                    p.x = n
+                    for j in range(length):
+                        if p.y[j] != 0:
+                            return -3
+                        p.y[j] = n^j
+                finally:
+                    free(p, flavor='raw')
             return 42
 
         fn = self.getcompiled(f, [int])
@@ -655,7 +658,7 @@
     def test_prebuilt_ll2ctypes_array(self):
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         A = rffi.CArray(Char)
-        a = malloc(A, 6, flavor='raw')
+        a = malloc(A, 6, flavor='raw', immortal=True)
         a[0] = 'a'
         a[1] = 'b'
         a[2] = 'c'
@@ -676,7 +679,7 @@
     def test_ll2ctypes_array_from_c(self):
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         A = rffi.CArray(Char)
-        a = malloc(A, 6, flavor='raw')
+        a = malloc(A, 6, flavor='raw', immortal=True)
         a[0] = 'a'
         a[1] = 'b'
         a[2] = 'c'


From arigo at codespeak.net  Wed Oct 20 11:44:19 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 11:44:19 +0200 (CEST)
Subject: [pypy-svn] r78114 - pypy/branch/leak-finder
Message-ID: <20101020094419.98B83282B90@codespeak.net>

Author: arigo
Date: Wed Oct 20 11:44:18 2010
New Revision: 78114

Removed:
   pypy/branch/leak-finder/
Log:
Remove merged branch.



From arigo at codespeak.net  Wed Oct 20 11:46:06 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 11:46:06 +0200 (CEST)
Subject: [pypy-svn] r78115 - in pypy/branch/jit-sign-descr/pypy/jit/backend:
	llsupport test x86
Message-ID: <20101020094606.C467036E3FB@codespeak.net>

Author: arigo
Date: Wed Oct 20 11:46:04 2010
New Revision: 78115

Modified:
   pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/descr.py
   pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/llmodel.py
   pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/symbolic.py
   pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py
   pypy/branch/jit-sign-descr/pypy/jit/backend/x86/assembler.py
   pypy/branch/jit-sign-descr/pypy/jit/backend/x86/regalloc.py
   pypy/branch/jit-sign-descr/pypy/jit/backend/x86/regloc.py
   pypy/branch/jit-sign-descr/pypy/jit/backend/x86/rx86.py
Log:
Finish this.


Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/descr.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/descr.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/descr.py	Wed Oct 20 11:46:04 2010
@@ -425,7 +425,7 @@
         #
         if TYPE is lltype.Float:
             setattr(Descr, floatattrname, True)
-        elif rffi.cast(TYPE, -1) == -1:
+        elif TYPE is not lltype.Bool and rffi.cast(TYPE, -1) == -1:
             setattr(Descr, signedattrname, True)
         #
         _cache[nameprefix, TYPE] = Descr

Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/llmodel.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/llmodel.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/llmodel.py	Wed Oct 20 11:46:04 2010
@@ -211,7 +211,8 @@
         assert isinstance(fielddescr, BaseFieldDescr)
         ofs = fielddescr.offset
         size = fielddescr.get_field_size(self.translate_support_code)
-        return ofs, size
+        sign = fielddescr.is_field_signed()
+        return ofs, size, sign
     unpack_fielddescr_size._always_inline_ = True
 
     def arraydescrof(self, A):
@@ -226,7 +227,8 @@
         assert isinstance(arraydescr, BaseArrayDescr)
         ofs = arraydescr.get_base_size(self.translate_support_code)
         size = arraydescr.get_item_size(self.translate_support_code)
-        return ofs, size
+        sign = arraydescr.is_item_signed()
+        return ofs, size, sign
     unpack_arraydescr_size._always_inline_ = True
 
     def calldescrof(self, FUNC, ARGS, RESULT, extrainfo=None):
@@ -256,15 +258,21 @@
 
     @specialize.argtype(2)
     def bh_getarrayitem_gc_i(self, arraydescr, gcref, itemindex):
-        ofs, size = self.unpack_arraydescr_size(arraydescr)
+        ofs, size, sign = self.unpack_arraydescr_size(arraydescr)
         # --- start of GC unsafe code (no GC operation!) ---
         items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for STYPE, UTYPE, itemsize in unroll_basic_sizes:
             if size == itemsize:
-                items = rffi.cast(rffi.CArrayPtr(TYPE), items) 
-                val = items[itemindex]
+                if sign:
+                    items = rffi.cast(rffi.CArrayPtr(STYPE), items)
+                    val = items[itemindex]
+                    val = rffi.cast(lltype.Signed, val)
+                else:
+                    items = rffi.cast(rffi.CArrayPtr(UTYPE), items)
+                    val = items[itemindex]
+                    val = rffi.cast(lltype.Signed, val)
                 # --- end of GC unsafe code ---
-                return rffi.cast(lltype.Signed, val)
+                return val
         else:
             raise NotImplementedError("size = %d" % size)
 
@@ -289,10 +297,10 @@
 
     @specialize.argtype(2)
     def bh_setarrayitem_gc_i(self, arraydescr, gcref, itemindex, newvalue):
-        ofs, size = self.unpack_arraydescr_size(arraydescr)
+        ofs, size, sign = self.unpack_arraydescr_size(arraydescr)
         # --- start of GC unsafe code (no GC operation!) ---
         items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for TYPE, _, itemsize in unroll_basic_sizes:
             if size == itemsize:
                 items = rffi.cast(rffi.CArrayPtr(TYPE), items)
                 items[itemindex] = rffi.cast(TYPE, newvalue)
@@ -343,14 +351,22 @@
 
     @specialize.argtype(1)
     def _base_do_getfield_i(self, struct, fielddescr):
-        ofs, size = self.unpack_fielddescr_size(fielddescr)
+        ofs, size, sign = self.unpack_fielddescr_size(fielddescr)
         # --- start of GC unsafe code (no GC operation!) ---
         fieldptr = rffi.ptradd(rffi.cast(rffi.CCHARP, struct), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for STYPE, UTYPE, itemsize in unroll_basic_sizes:
             if size == itemsize:
-                val = rffi.cast(rffi.CArrayPtr(TYPE), fieldptr)[0]
+                # Note that in the common case where size==sizeof(Signed),
+                # both cases of what follows are doing the same thing.
+                # But gcc is clever enough to figure this out :-)
+                if sign:
+                    val = rffi.cast(rffi.CArrayPtr(STYPE), fieldptr)[0]
+                    val = rffi.cast(lltype.Signed, val)
+                else:
+                    val = rffi.cast(rffi.CArrayPtr(UTYPE), fieldptr)[0]
+                    val = rffi.cast(lltype.Signed, val)
                 # --- end of GC unsafe code ---
-                return rffi.cast(lltype.Signed, val)
+                return val
         else:
             raise NotImplementedError("size = %d" % size)
 
@@ -382,10 +398,10 @@
 
     @specialize.argtype(1)
     def _base_do_setfield_i(self, struct, fielddescr, newvalue):
-        ofs, size = self.unpack_fielddescr_size(fielddescr)
+        ofs, size, sign = self.unpack_fielddescr_size(fielddescr)
         # --- start of GC unsafe code (no GC operation!) ---
         fieldptr = rffi.ptradd(rffi.cast(rffi.CCHARP, struct), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for TYPE, _, itemsize in unroll_basic_sizes:
             if size == itemsize:
                 fieldptr = rffi.cast(rffi.CArrayPtr(TYPE), fieldptr)
                 fieldptr[0] = rffi.cast(TYPE, newvalue)

Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/symbolic.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/symbolic.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/llsupport/symbolic.py	Wed Oct 20 11:46:04 2010
@@ -69,8 +69,9 @@
 SIZEOF_INT   = get_size(rffi.INT, False)
 SIZEOF_FLOAT = get_size(lltype.Float, False)
 
-unroll_basic_sizes = unrolling_iterable([(lltype.Signed, WORD),
-                                         (lltype.Char,   SIZEOF_CHAR),
-                                         (rffi.SHORT,    SIZEOF_SHORT),
-                                         (rffi.INT,      SIZEOF_INT)])
+unroll_basic_sizes = unrolling_iterable([
+    (lltype.Signed,   lltype.Unsigned, WORD),
+    (rffi.SIGNEDCHAR, lltype.Char,     SIZEOF_CHAR),
+    (rffi.SHORT,      rffi.USHORT,     SIZEOF_SHORT),
+    (rffi.INT,        rffi.UINT,       SIZEOF_INT)])
 # does not contain Float ^^^ which must be special-cased

Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/test/runner_test.py	Wed Oct 20 11:46:04 2010
@@ -2067,7 +2067,7 @@
             res = self.execute_operation(rop.GETFIELD_GC, [BoxPtr(s_gcref)],
                                          'int', descr=descrfld_x)
             assert res.value == expected, (
-                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
 
     def test_short_result_of_getarrayitem_direct(self):
         # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed

Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/x86/assembler.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/x86/assembler.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/x86/assembler.py	Wed Oct 20 11:46:04 2010
@@ -8,7 +8,8 @@
 from pypy.rpython.annlowlevel import llhelper
 from pypy.tool.uid import fixid
 from pypy.jit.backend.x86.regalloc import (RegAlloc, X86RegisterManager,
-                                           X86XMMRegisterManager, get_ebp_ofs)
+                                           X86XMMRegisterManager, get_ebp_ofs,
+                                           _get_scale)
 
 from pypy.jit.backend.x86.arch import (FRAME_FIXED_SIZE, FORCE_INDEX_OFS, WORD,
                                        IS_X86_32, IS_X86_64)
@@ -22,7 +23,8 @@
                                          X86_64_SCRATCH_REG,
                                          X86_64_XMM_SCRATCH_REG,
                                          RegLoc, StackLoc, ConstFloatLoc,
-                                         ImmedLoc, AddressLoc, imm)
+                                         ImmedLoc, AddressLoc, imm,
+                                         imm0, imm1)
 
 from pypy.rlib.objectmodel import we_are_translated, specialize
 from pypy.jit.backend.x86 import rx86, regloc, codebuf
@@ -440,7 +442,7 @@
             # self.mc.PUSH(eax)
             # adr = rffi.cast(lltype.Signed, self.loop_run_counters[-1][1])
             # self.mc.MOV(eax, heap(adr))
-            # self.mc.ADD(eax, imm(1))
+            # self.mc.ADD(eax, imm1)
             # self.mc.MOV(heap(adr), eax)
             # self.mc.POP(eax)
         return operations
@@ -711,7 +713,7 @@
         self.regalloc_perform_with_guard(None, guard_op, faillocs, arglocs,
                                          resloc, current_depths)
 
-    def load_effective_addr(self, sizereg, baseofs, scale, result, frm=imm(0)):
+    def load_effective_addr(self, sizereg, baseofs, scale, result, frm=imm0):
         self.mc.LEA(result, addr_add(frm, sizereg, baseofs, scale))
 
     def _unaryop(asmop):
@@ -973,28 +975,28 @@
 
     def genop_guard_int_is_true(self, op, guard_op, guard_token, arglocs, resloc):
         guard_opnum = guard_op.getopnum()
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         if guard_opnum == rop.GUARD_TRUE:
             return self.implement_guard(guard_token, 'Z')
         else:
             return self.implement_guard(guard_token, 'NZ')
 
     def genop_int_is_true(self, op, arglocs, resloc):
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         rl = resloc.lowest8bits()
         self.mc.SET_ir(rx86.Conditions['NE'], rl.value)
         self.mc.MOVZX8(resloc, rl)
 
     def genop_guard_int_is_zero(self, op, guard_op, guard_token, arglocs, resloc):
         guard_opnum = guard_op.getopnum()
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         if guard_opnum == rop.GUARD_TRUE:
             return self.implement_guard(guard_token, 'NZ')
         else:
             return self.implement_guard(guard_token, 'Z')
 
     def genop_int_is_zero(self, op, arglocs, resloc):
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         rl = resloc.lowest8bits()
         self.mc.SET_ir(rx86.Conditions['E'], rl.value)
         self.mc.MOVZX8(resloc, rl)
@@ -1050,50 +1052,66 @@
         assert result_loc is eax
         self.call(self.malloc_unicode_func_addr, arglocs, eax)
 
-    def genop_getfield_gc(self, op, arglocs, resloc):
-        base_loc, ofs_loc, size_loc = arglocs
-        assert isinstance(size_loc, ImmedLoc)
+    # ----------
+
+    def load_from_mem(self, resloc, source_addr, size_loc, sign_loc):
         assert isinstance(resloc, RegLoc)
         size = size_loc.value
-
-        source_addr = AddressLoc(base_loc, ofs_loc)
+        sign = sign_loc.value
         if resloc.is_xmm:
             self.mc.MOVSD(resloc, source_addr)
+        elif size == WORD:
+            self.mc.MOV(resloc, source_addr)
+        elif size == 1:
+            if sign:
+                self.mc.MOVSX8(resloc, source_addr)
+            else:
+                self.mc.MOVZX8(resloc, source_addr)
+        elif size == 2:
+            if sign:
+                self.mc.MOVSX16(resloc, source_addr)
+            else:
+                self.mc.MOVZX16(resloc, source_addr)
+        elif IS_X86_64 and size == 4:
+            if sign:
+                self.mc.MOVSX32(resloc, source_addr)
+            else:
+                self.mc.MOV32(resloc, source_addr)    # zero-extending
+        else:
+            not_implemented("load_from_mem size = %d" % size)
+
+    def save_into_mem(self, dest_addr, value_loc, size_loc):
+        size = size_loc.value
+        if isinstance(value_loc, RegLoc) and value_loc.is_xmm:
+            self.mc.MOVSD(dest_addr, value_loc)
         elif size == 1:
-            self.mc.MOVZX8(resloc, source_addr)
+            self.mc.MOV8(dest_addr, value_loc.lowest8bits())
         elif size == 2:
-            self.mc.MOVZX16(resloc, source_addr)
+            self.mc.MOV16(dest_addr, value_loc)
         elif size == 4:
-            # MOV32 is zero-extending on 64-bit, so this is okay
-            self.mc.MOV32(resloc, source_addr)
+            self.mc.MOV32(dest_addr, value_loc)
         elif IS_X86_64 and size == 8:
-            self.mc.MOV(resloc, source_addr)
+            self.mc.MOV(dest_addr, value_loc)
         else:
-            raise NotImplementedError("getfield size = %d" % size)
+            not_implemented("save_into_mem size = %d" % size)
+
+    def genop_getfield_gc(self, op, arglocs, resloc):
+        base_loc, ofs_loc, size_loc, sign_loc = arglocs
+        assert isinstance(size_loc, ImmedLoc)
+        source_addr = AddressLoc(base_loc, ofs_loc)
+        self.load_from_mem(resloc, source_addr, size_loc, sign_loc)
 
     genop_getfield_raw = genop_getfield_gc
     genop_getfield_raw_pure = genop_getfield_gc
     genop_getfield_gc_pure = genop_getfield_gc
 
     def genop_getarrayitem_gc(self, op, arglocs, resloc):
-        base_loc, ofs_loc, scale, ofs = arglocs
+        base_loc, ofs_loc, size_loc, ofs, sign_loc = arglocs
         assert isinstance(ofs, ImmedLoc)
-        assert isinstance(scale, ImmedLoc)
-        src_addr = addr_add(base_loc, ofs_loc, ofs.value, scale.value)
-        if op.result.type == FLOAT:
-            self.mc.MOVSD(resloc, src_addr)
-        else:
-            if scale.value == 0:
-                self.mc.MOVZX8(resloc, src_addr)
-            elif scale.value == 1:
-                self.mc.MOVZX16(resloc, src_addr)
-            elif scale.value == 2:
-                self.mc.MOV32(resloc, src_addr)
-            elif IS_X86_64 and scale.value == 3:
-                self.mc.MOV(resloc, src_addr)
-            else:
-                print "[asmgen]getarrayitem unsupported size: %d" % scale.value
-                raise NotImplementedError()
+        assert isinstance(size_loc, ImmedLoc)
+        scale = _get_scale(size_loc.value)
+        src_addr = addr_add(base_loc, ofs_loc, ofs.value, scale)
+        self.load_from_mem(resloc, src_addr, size_loc, sign_loc)
 
     genop_getarrayitem_gc_pure = genop_getarrayitem_gc
     genop_getarrayitem_raw = genop_getarrayitem_gc
@@ -1101,40 +1119,16 @@
     def genop_discard_setfield_gc(self, op, arglocs):
         base_loc, ofs_loc, size_loc, value_loc = arglocs
         assert isinstance(size_loc, ImmedLoc)
-        size = size_loc.value
         dest_addr = AddressLoc(base_loc, ofs_loc)
-        if isinstance(value_loc, RegLoc) and value_loc.is_xmm:
-            self.mc.MOVSD(dest_addr, value_loc)
-        elif IS_X86_64 and size == 8:
-            self.mc.MOV(dest_addr, value_loc)
-        elif size == 4:
-            self.mc.MOV32(dest_addr, value_loc)
-        elif size == 2:
-            self.mc.MOV16(dest_addr, value_loc)
-        elif size == 1:
-            self.mc.MOV8(dest_addr, value_loc.lowest8bits())
-        else:
-            print "[asmgen]setfield addr size %d" % size
-            raise NotImplementedError("Addr size %d" % size)
+        self.save_into_mem(dest_addr, value_loc, size_loc)
 
     def genop_discard_setarrayitem_gc(self, op, arglocs):
-        base_loc, ofs_loc, value_loc, scale_loc, baseofs = arglocs
+        base_loc, ofs_loc, value_loc, size_loc, baseofs = arglocs
         assert isinstance(baseofs, ImmedLoc)
-        assert isinstance(scale_loc, ImmedLoc)
-        dest_addr = AddressLoc(base_loc, ofs_loc, scale_loc.value, baseofs.value)
-        if op.getarg(2).type == FLOAT:
-            self.mc.MOVSD(dest_addr, value_loc)
-        else:
-            if IS_X86_64 and scale_loc.value == 3:
-                self.mc.MOV(dest_addr, value_loc)
-            elif scale_loc.value == 2:
-                self.mc.MOV32(dest_addr, value_loc)
-            elif scale_loc.value == 1:
-                self.mc.MOV16(dest_addr, value_loc)
-            elif scale_loc.value == 0:
-                self.mc.MOV8(dest_addr, value_loc.lowest8bits())
-            else:
-                raise NotImplementedError("scale = %d" % scale_loc.value)
+        assert isinstance(size_loc, ImmedLoc)
+        scale = _get_scale(size_loc.value)
+        dest_addr = AddressLoc(base_loc, ofs_loc, scale, baseofs.value)
+        self.save_into_mem(dest_addr, value_loc, size_loc)
 
     def genop_discard_strsetitem(self, op, arglocs):
         base_loc, ofs_loc, val_loc = arglocs
@@ -1201,7 +1195,7 @@
 
     def genop_guard_guard_no_exception(self, ign_1, guard_op, guard_token,
                                        locs, ign_2):
-        self.mc.CMP(heap(self.cpu.pos_exception()), imm(0))
+        self.mc.CMP(heap(self.cpu.pos_exception()), imm0)
         return self.implement_guard(guard_token, 'NZ')
 
     def genop_guard_guard_exception(self, ign_1, guard_op, guard_token,
@@ -1213,8 +1207,8 @@
         addr = self.implement_guard(guard_token, 'NE')
         if resloc is not None:
             self.mc.MOV(resloc, heap(self.cpu.pos_exc_value()))
-        self.mc.MOV(heap(self.cpu.pos_exception()), imm(0))
-        self.mc.MOV(heap(self.cpu.pos_exc_value()), imm(0))
+        self.mc.MOV(heap(self.cpu.pos_exception()), imm0)
+        self.mc.MOV(heap(self.cpu.pos_exc_value()), imm0)
         return addr
 
     def _gen_guard_overflow(self, guard_op, guard_token):
@@ -1224,8 +1218,8 @@
         elif guard_opnum == rop.GUARD_OVERFLOW:
             return self.implement_guard(guard_token, 'NO')
         else:
-            print "int_xxx_ovf followed by", guard_op.getopname()
-            raise AssertionError
+            not_implemented("int_xxx_ovf followed by %s" %
+                            guard_op.getopname())
 
     def genop_guard_int_add_ovf(self, op, guard_op, guard_token, arglocs, result_loc):
         self.genop_int_add(op, arglocs, result_loc)
@@ -1288,7 +1282,7 @@
     def genop_guard_guard_nonnull_class(self, ign_1, guard_op,
                                         guard_token, locs, ign_2):
         self.mc.ensure_bytes_available(256)
-        self.mc.CMP(locs[0], imm(1))
+        self.mc.CMP(locs[0], imm1)
         # Patched below
         self.mc.J_il8(rx86.Conditions['B'], 0)
         jb_location = self.mc.get_relative_pos()
@@ -1637,25 +1631,34 @@
         sizeloc = arglocs[0]
         assert isinstance(sizeloc, ImmedLoc)
         size = sizeloc.value
+        signloc = arglocs[1]
 
         if isinstance(op.getarg(0), Const):
             x = imm(op.getarg(0).getint())
         else:
-            x = arglocs[1]
+            x = arglocs[2]
         if x is eax:
             tmp = ecx
         else:
             tmp = eax
         
-        self._emit_call(x, arglocs, 2, tmp=tmp)
+        self._emit_call(x, arglocs, 3, tmp=tmp)
+
+        if IS_X86_32 and isinstance(resloc, StackLoc) and resloc.width == 8:
+            self.mc.FSTP_b(resloc.value)   # float return
+        elif size == WORD:
+            assert resloc is eax or resloc is xmm0    # a full word
+        elif size == 0:
+            pass    # void return
+        else:
+            # use the code in load_from_mem to do the zero- or sign-extension
+            assert resloc is eax
+            if size == 1:
+                srcloc = eax.lowest8bits()
+            else:
+                srcloc = eax
+            self.load_from_mem(eax, srcloc, sizeloc, signloc)
 
-        if isinstance(resloc, StackLoc) and resloc.width == 8 and IS_X86_32:
-            self.mc.FSTP_b(resloc.value)
-        elif size == 1:
-            self.mc.AND_ri(eax.value, 0xff)
-        elif size == 2:
-            self.mc.AND_ri(eax.value, 0xffff)
-    
     def genop_guard_call_may_force(self, op, guard_op, guard_token,
                                    arglocs, result_loc):
         faildescr = guard_op.getdescr()
@@ -1808,20 +1811,16 @@
         self.mc.LEA_rb(resloc.value, FORCE_INDEX_OFS)
 
     def not_implemented_op_discard(self, op, arglocs):
-        msg = "not implemented operation: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation: %s" % op.getopname())
 
     def not_implemented_op(self, op, arglocs, resloc):
-        msg = "not implemented operation with res: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation with res: %s" %
+                        op.getopname())
 
     def not_implemented_op_guard(self, op, guard_op,
                                  failaddr, arglocs, resloc):
-        msg = "not implemented operation (guard): %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation (guard): %s" %
+                        op.getopname())
 
     def mark_gc_roots(self):
         gcrootmap = self.cpu.gc_ll_descr.gcrootmap
@@ -1905,3 +1904,7 @@
 
 def heap(addr):
     return AddressLoc(ImmedLoc(addr), ImmedLoc(0), 0, 0)
+
+def not_implemented(msg):
+    os.write(2, '[x86/asm] %s\n' % msg)
+    raise NotImplementedError(msg)

Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/x86/regalloc.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/x86/regalloc.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/x86/regalloc.py	Wed Oct 20 11:46:04 2010
@@ -2,6 +2,7 @@
 """ Register allocation scheme.
 """
 
+import os
 from pypy.jit.metainterp.history import (Box, Const, ConstInt, ConstPtr,
                                          ResOperation, BoxPtr,
                                          LoopToken, INT, REF, FLOAT)
@@ -40,12 +41,10 @@
             return imm(c.value)
         elif isinstance(c, ConstPtr):
             if we_are_translated() and c.value and rgc.can_move(c.value):
-                print "convert_to_imm: ConstPtr needs special care"
-                raise AssertionError
+                not_implemented("convert_to_imm: ConstPtr needs special care")
             return imm(rffi.cast(lltype.Signed, c.value))
         else:
-            print "convert_to_imm: got a %s" % c
-            raise AssertionError
+            not_implemented("convert_to_imm: got a %s" % c)
 
 class X86_64_RegisterManager(X86RegisterManager):
     # r11 omitted because it's used as scratch
@@ -359,8 +358,8 @@
             if op.is_ovf():
                 if (operations[i + 1].getopnum() != rop.GUARD_NO_OVERFLOW and
                     operations[i + 1].getopnum() != rop.GUARD_OVERFLOW):
-                    print "int_xxx_ovf not followed by guard_(no)_overflow"
-                    raise AssertionError
+                    not_implemented("int_xxx_ovf not followed by "
+                                    "guard_(no)_overflow")
                 return True
             return False
         if (operations[i + 1].getopnum() != rop.GUARD_TRUE and
@@ -412,8 +411,8 @@
                 arg = op.getarg(j)
                 if isinstance(arg, Box):
                     if arg not in start_live:
-                        print "Bogus arg in operation %d at %d" % (op.getopnum(), i)
-                        raise AssertionError
+                        not_implemented("Bogus arg in operation %d at %d" %
+                                        (op.getopnum(), i))
                     longevity[arg] = (start_live[arg], i)
             if op.is_guard():
                 for arg in op.getfailargs():
@@ -421,8 +420,8 @@
                         continue
                     assert isinstance(arg, Box)
                     if arg not in start_live:
-                        print "Bogus arg in guard %d at %d" % (op.getopnum(), i)
-                        raise AssertionError
+                        not_implemented("Bogus arg in guard %d at %d" %
+                                        (op.getopnum(), i))
                     longevity[arg] = (start_live[arg], i)
         for arg in inputargs:
             if arg not in longevity:
@@ -667,7 +666,13 @@
         assert isinstance(calldescr, BaseCallDescr)
         assert len(calldescr.arg_classes) == op.numargs() - 1
         size = calldescr.get_result_size(self.translate_support_code)
-        self._call(op, [imm(size)] + [self.loc(op.getarg(i)) for i in range(op.numargs())],
+        sign = calldescr.is_result_signed()
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self._call(op, [imm(size), sign_loc] +
+                       [self.loc(op.getarg(i)) for i in range(op.numargs())],
                    guard_not_forced_op=guard_not_forced_op)
 
     def consider_call(self, op):
@@ -688,7 +693,7 @@
             self.rm._sync_var(op.getarg(vable_index))
             vable = self.fm.loc(op.getarg(vable_index))
         else:
-            vable = imm(0)
+            vable = imm0
         self._call(op, [imm(size), vable] +
                    [self.loc(op.getarg(i)) for i in range(op.numargs())],
                    guard_not_forced_op=guard_op)
@@ -814,8 +819,9 @@
             arglocs.append(self.loc(op.getarg(0)))
             return self._call(op, arglocs)
         # boehm GC (XXX kill the following code at some point)
-        scale_of_field, basesize, ofs_length, _ = (
+        itemsize, basesize, ofs_length, _, _ = (
             self._unpack_arraydescr(op.getdescr()))
+        scale_of_field = _get_scale(itemsize)
         return self._malloc_varsize(basesize, ofs_length, scale_of_field,
                                     op.getarg(0), op.result)
 
@@ -825,21 +831,19 @@
         ofs = arraydescr.get_base_size(self.translate_support_code)
         size = arraydescr.get_item_size(self.translate_support_code)
         ptr = arraydescr.is_array_of_pointers()
-        scale = 0
-        while (1 << scale) < size:
-            scale += 1
-        assert (1 << scale) == size
-        return scale, ofs, ofs_length, ptr
+        sign = arraydescr.is_item_signed()
+        return size, ofs, ofs_length, ptr, sign
 
     def _unpack_fielddescr(self, fielddescr):
         assert isinstance(fielddescr, BaseFieldDescr)
         ofs = fielddescr.offset
         size = fielddescr.get_field_size(self.translate_support_code)
         ptr = fielddescr.is_pointer_field()
-        return imm(ofs), imm(size), ptr
+        sign = fielddescr.is_field_signed()
+        return imm(ofs), imm(size), ptr, sign
 
     def consider_setfield_gc(self, op):
-        ofs_loc, size_loc, ptr = self._unpack_fielddescr(op.getdescr())
+        ofs_loc, size_loc, _, _ = self._unpack_fielddescr(op.getdescr())
         assert isinstance(size_loc, ImmedLoc)
         if size_loc.value == 1:
             need_lower_byte = True
@@ -866,10 +870,10 @@
     consider_unicodesetitem = consider_strsetitem
 
     def consider_setarrayitem_gc(self, op):
-        scale, ofs, _, ptr = self._unpack_arraydescr(op.getdescr())
+        itemsize, ofs, _, _, _ = self._unpack_arraydescr(op.getdescr())
         args = op.getarglist()
         base_loc  = self.rm.make_sure_var_in_reg(op.getarg(0), args)
-        if scale == 0:
+        if itemsize == 1:
             need_lower_byte = True
         else:
             need_lower_byte = False
@@ -878,30 +882,39 @@
         ofs_loc = self.rm.make_sure_var_in_reg(op.getarg(1), args)
         self.possibly_free_vars(args)
         self.PerformDiscard(op, [base_loc, ofs_loc, value_loc,
-                                 imm(scale), imm(ofs)])
+                                 imm(itemsize), imm(ofs)])
 
     consider_setarrayitem_raw = consider_setarrayitem_gc
 
     def consider_getfield_gc(self, op):
-        ofs_loc, size_loc, _ = self._unpack_fielddescr(op.getdescr())
+        ofs_loc, size_loc, _, sign = self._unpack_fielddescr(op.getdescr())
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
         self.rm.possibly_free_vars(args)
         result_loc = self.force_allocate_reg(op.result)
-        self.Perform(op, [base_loc, ofs_loc, size_loc], result_loc)
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self.Perform(op, [base_loc, ofs_loc, size_loc, sign_loc], result_loc)
 
     consider_getfield_raw = consider_getfield_gc
     consider_getfield_raw_pure = consider_getfield_gc
     consider_getfield_gc_pure = consider_getfield_gc
 
     def consider_getarrayitem_gc(self, op):
-        scale, ofs, _, _ = self._unpack_arraydescr(op.getdescr())
+        itemsize, ofs, _, _, sign = self._unpack_arraydescr(op.getdescr())
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
         ofs_loc = self.rm.make_sure_var_in_reg(op.getarg(1), args)
         self.rm.possibly_free_vars_for_op(op)
         result_loc = self.force_allocate_reg(op.result)
-        self.Perform(op, [base_loc, ofs_loc, imm(scale), imm(ofs)], result_loc)
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self.Perform(op, [base_loc, ofs_loc, imm(itemsize), imm(ofs),
+                          sign_loc], result_loc)
 
     consider_getarrayitem_raw = consider_getarrayitem_gc
     consider_getarrayitem_gc_pure = consider_getarrayitem_gc
@@ -1090,15 +1103,11 @@
         self.Perform(op, [], loc)
 
     def not_implemented_op(self, op):
-        msg = "[regalloc] Not implemented operation: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation: %s" % op.getopname())
 
     def not_implemented_op_with_guard(self, op, guard_op):
-        msg = "[regalloc] Not implemented operation with guard: %s" % (
-            op.getopname(),)
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation with guard: %s" % (
+            op.getopname(),))
 
 oplist = [RegAlloc.not_implemented_op] * rop._LAST
 oplist_with_guard = [RegAlloc.not_implemented_op_with_guard] * rop._LAST
@@ -1132,3 +1141,14 @@
     # Returns (ebp-20), (ebp-24), (ebp-28)...
     # i.e. the n'th word beyond the fixed frame size.
     return -WORD * (FRAME_FIXED_SIZE + position)
+
+def _get_scale(size):
+    assert size == 1 or size == 2 or size == 4 or size == 8
+    if size < 4:
+        return size - 1         # 1, 2 => 0, 1
+    else:
+        return (size >> 2) + 1  # 4, 8 => 2, 3
+
+def not_implemented(msg):
+    os.write(2, '[x86/regalloc] %s\n' % msg)
+    raise NotImplementedError(msg)

Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/x86/regloc.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/x86/regloc.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/x86/regloc.py	Wed Oct 20 11:46:04 2010
@@ -442,8 +442,11 @@
     MOV8 = _binaryop('MOV8')
     MOV16 = _16_bit_binaryop('MOV')
     MOVZX8 = _binaryop('MOVZX8')
+    MOVSX8 = _binaryop('MOVSX8')
     MOVZX16 = _binaryop('MOVZX16')
+    MOVSX16 = _binaryop('MOVSX16')
     MOV32 = _binaryop('MOV32')
+    MOVSX32 = _binaryop('MOVSX32')
     XCHG = _binaryop('XCHG')
 
     PUSH = _unaryop('PUSH')
@@ -473,6 +476,9 @@
     else:
         return ImmedLoc(x)
 
+imm0 = imm(0)
+imm1 = imm(1)
+
 all_extra_instructions = [name for name in LocationCodeBuilder.__dict__
                           if name[0].isupper()]
 all_extra_instructions.sort()

Modified: pypy/branch/jit-sign-descr/pypy/jit/backend/x86/rx86.py
==============================================================================
--- pypy/branch/jit-sign-descr/pypy/jit/backend/x86/rx86.py	(original)
+++ pypy/branch/jit-sign-descr/pypy/jit/backend/x86/rx86.py	Wed Oct 20 11:46:04 2010
@@ -642,7 +642,10 @@
 define_modrm_modes('MOV8_*i', [rex_w, '\xC6', orbyte(0<<3)], [immediate(2, 'b')], regtype='BYTE')
 
 define_modrm_modes('MOVZX8_r*', [rex_w, '\x0F\xB6', register(1, 8)], regtype='BYTE')
+define_modrm_modes('MOVSX8_r*', [rex_w, '\x0F\xBE', register(1, 8)], regtype='BYTE')
 define_modrm_modes('MOVZX16_r*', [rex_w, '\x0F\xB7', register(1, 8)])
+define_modrm_modes('MOVSX16_r*', [rex_w, '\x0F\xBF', register(1, 8)])
+define_modrm_modes('MOVSX32_r*', [rex_w, '\x63', register(1, 8)])
 
 define_modrm_modes('MOVSD_x*', ['\xF2', rex_nw, '\x0F\x10', register(1,8)], regtype='XMM')
 define_modrm_modes('MOVSD_*x', ['\xF2', rex_nw, '\x0F\x11', register(2,8)], regtype='XMM')


From arigo at codespeak.net  Wed Oct 20 11:53:59 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 11:53:59 +0200 (CEST)
Subject: [pypy-svn] r78116 - pypy/trunk/pypy/module/pypyjit/test
Message-ID: <20101020095359.21B4F282BDA@codespeak.net>

Author: arigo
Date: Wed Oct 20 11:53:57 2010
New Revision: 78116

Modified:
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
Log:
Baaah.  This was a CALL_METHOD instead of a CALL_FUNCTION.


Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Wed Oct 20 11:53:57 2010
@@ -560,13 +560,13 @@
 
     def test_blockstack_virtualizable(self):
         self.run_source('''
-        import pypyjit
+        from pypyjit import residual_call
 
         def main():
             i = 0
             while i < 100:
                 try:
-                    pypyjit.residual_call(len, [])
+                    residual_call(len, [])
                 except:
                     pass
                 i += 1


From antocuni at codespeak.net  Wed Oct 20 12:01:04 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Wed, 20 Oct 2010 12:01:04 +0200 (CEST)
Subject: [pypy-svn] r78117 - pypy/trunk/pypy/module/imp/test
Message-ID: <20101020100104.2AC2936E3FB@codespeak.net>

Author: antocuni
Date: Wed Oct 20 12:01:02 2010
New Revision: 78117

Modified:
   pypy/trunk/pypy/module/imp/test/test_import.py
Log:
make sure that LANG is unset when calling py.py through py.process.cmdexec.
This is a workaround for pylib bug #130:
http://bitbucket.org/hpk42/py-trunk/issue/130/pyprocesscmdexec-fails-if-the-out-err-contains-non-ascii



Modified: pypy/trunk/pypy/module/imp/test/test_import.py
==============================================================================
--- pypy/trunk/pypy/module/imp/test/test_import.py	(original)
+++ pypy/trunk/pypy/module/imp/test/test_import.py	Wed Oct 20 12:01:02 2010
@@ -790,6 +790,7 @@
     extrapath = udir.ensure("pythonpath", dir=1) 
     extrapath.join("urllib.py").write("print 42\n")
     old = os.environ.get('PYTHONPATH', None)
+    oldlang = os.environ.pop('LANG', None)
     try: 
         os.environ['PYTHONPATH'] = str(extrapath)
         output = py.process.cmdexec('''"%s" "%s" -c "import urllib"''' % 
@@ -798,6 +799,8 @@
     finally: 
         if old: 
             os.environ['PYTHONPATH'] = old 
+        if oldlang:
+            os.environ['LANG'] = oldlang
 
 class AppTestImportHooks(object):
     def test_meta_path(self):


From antocuni at codespeak.net  Wed Oct 20 12:01:17 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Wed, 20 Oct 2010 12:01:17 +0200 (CEST)
Subject: [pypy-svn] r78118 - pypy/branch/debug-test_import
Message-ID: <20101020100117.72B73282BDA@codespeak.net>

Author: antocuni
Date: Wed Oct 20 12:01:16 2010
New Revision: 78118

Removed:
   pypy/branch/debug-test_import/
Log:
remove merged branch



From arigo at codespeak.net  Wed Oct 20 13:03:15 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 13:03:15 +0200 (CEST)
Subject: [pypy-svn] r78119 -
	pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests
Message-ID: <20101020110315.3BC0536E3FB@codespeak.net>

Author: arigo
Date: Wed Oct 20 13:03:13 2010
New Revision: 78119

Removed:
   pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_find.py
Log:
Revert r52839: this (unmodified) test file from ctypes fails
on some platform (in this case tannit64) because it does not
have the opengl libraries.



From antocuni at codespeak.net  Wed Oct 20 13:04:03 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Wed, 20 Oct 2010 13:04:03 +0200 (CEST)
Subject: [pypy-svn] r78120 - in pypy/trunk/pypy: rlib rpython
Message-ID: <20101020110403.CCBB5282BDA@codespeak.net>

Author: antocuni
Date: Wed Oct 20 13:04:02 2010
New Revision: 78120

Modified:
   pypy/trunk/pypy/rlib/clibffi.py
   pypy/trunk/pypy/rpython/llinterp.py
Log:
(antocuni, arigo): don't track these mallocs, they are automatically freed in the __del__


Modified: pypy/trunk/pypy/rlib/clibffi.py
==============================================================================
--- pypy/trunk/pypy/rlib/clibffi.py	(original)
+++ pypy/trunk/pypy/rlib/clibffi.py	Wed Oct 20 13:04:02 2010
@@ -385,10 +385,12 @@
         self.restype = restype
         self.flags = flags
         argnum = len(argtypes)
-        self.ll_argtypes = lltype.malloc(FFI_TYPE_PP.TO, argnum, flavor='raw')
+        self.ll_argtypes = lltype.malloc(FFI_TYPE_PP.TO, argnum, flavor='raw',
+                                         track_allocation=False) # freed by the __del__
         for i in range(argnum):
             self.ll_argtypes[i] = argtypes[i]
-        self.ll_cif = lltype.malloc(FFI_CIFP.TO, flavor='raw')
+        self.ll_cif = lltype.malloc(FFI_CIFP.TO, flavor='raw',
+                                    track_allocation=False) # freed by the __del__
 
         if _WIN32 and (flags & FUNCFLAG_CDECL == 0):
             cc = FFI_STDCALL

Modified: pypy/trunk/pypy/rpython/llinterp.py
==============================================================================
--- pypy/trunk/pypy/rpython/llinterp.py	(original)
+++ pypy/trunk/pypy/rpython/llinterp.py	Wed Oct 20 13:04:02 2010
@@ -717,9 +717,11 @@
     def op_malloc_varsize(self, obj, flags, size):
         flavor = flags['flavor']
         zero = flags.get('zero', False)
+        track_allocation = flags.get('track_allocation', True)
         assert flavor in ('gc', 'raw')
         try:
-            ptr = self.heap.malloc(obj, size, zero=zero, flavor=flavor)
+            ptr = self.heap.malloc(obj, size, zero=zero, flavor=flavor,
+                                   track_allocation=track_allocation)
             return ptr
         except MemoryError:
             self.make_llexception()


From afa at codespeak.net  Wed Oct 20 13:21:11 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 13:21:11 +0200 (CEST)
Subject: [pypy-svn] r78121 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101020112111.1312E36E3FF@codespeak.net>

Author: afa
Date: Wed Oct 20 13:21:09 2010
New Revision: 78121

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
Log:
Add FileIO.closefd attribute


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Wed Oct 20 13:21:09 2010
@@ -1,6 +1,6 @@
 from pypy.module._io.interp_iobase import W_RawIOBase
 from pypy.interpreter.typedef import (
-    TypeDef, interp_attrproperty_w, GetSetProperty)
+    TypeDef, interp_attrproperty, interp_attrproperty_w, GetSetProperty)
 from pypy.interpreter.gateway import interp2app, unwrap_spec, Arguments
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.error import OperationError, wrap_oserror, wrap_oserror2
@@ -366,6 +366,7 @@
     fileno = interp2app(W_FileIO.fileno_w),
     isatty = interp2app(W_FileIO.isatty_w),
     name = interp_attrproperty_w('w_name', cls=W_FileIO),
+    closefd = interp_attrproperty('closefd', cls=W_FileIO),
     mode = GetSetProperty(W_FileIO.descr_get_mode),
     )
 

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Wed Oct 20 13:21:09 2010
@@ -18,15 +18,18 @@
         f = _io.FileIO(self.tmpfile, 'a')
         assert f.name.endswith('tmpfile')
         assert f.mode == 'wb'
+        assert f.closefd is True
         f.close()
 
     def test_open_fd(self):
         import _io
         os = self.posix
         fd = os.open(self.tmpfile, os.O_RDONLY, 0666)
-        f = _io.FileIO(fd, "rb")
+        f = _io.FileIO(fd, "rb", closefd=False)
         assert f.fileno() == fd
+        assert f.closefd is False
         f.close()
+        os.close(fd)
 
     def test_open_directory(self):
         import _io


From afa at codespeak.net  Wed Oct 20 13:27:35 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 13:27:35 +0200 (CEST)
Subject: [pypy-svn] r78122 - in pypy/branch/fast-forward/pypy/objspace/std:
	. test
Message-ID: <20101020112735.9C964282BDA@codespeak.net>

Author: afa
Date: Wed Oct 20 13:27:34 2010
New Revision: 78122

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
Log:
Implement bytearray.__iadd__, bytearray item/slice assignment.


Modified: pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	Wed Oct 20 13:27:34 2010
@@ -359,15 +359,20 @@
 def list_append__Bytearray_ANY(space, w_bytearray, w_item):
     from pypy.objspace.std.bytearraytype import getbytevalue
     w_bytearray.data.append(getbytevalue(space, w_item))
-    return space.w_None
 
 def list_extend__Bytearray_Bytearray(space, w_bytearray, w_other):
     w_bytearray.data += w_other.data
-    return space.w_None
 
 def list_extend__Bytearray_ANY(space, w_bytearray, w_other):
     w_bytearray.data += [c for c in space.str_w(w_other)]
-    return space.w_None
+
+def inplace_add__Bytearray_Bytearray(space, w_bytearray1, w_bytearray2):
+    list_extend__Bytearray_Bytearray(space, w_bytearray1, w_bytearray2)
+    return w_bytearray1
+
+def inplace_add__Bytearray_ANY(space, w_bytearray1, w_iterable2):
+    list_extend__Bytearray_ANY(space, w_bytearray1, w_iterable2)
+    return w_bytearray1
 
 def delslice__Bytearray_ANY_ANY(space, w_bytearray, w_start, w_stop):
     length = len(w_bytearray.data)
@@ -376,5 +381,24 @@
         return
     del w_bytearray.data[start:stop]
 
+def setitem__Bytearray_ANY_ANY(space, w_bytearray, w_index, w_item):
+    from pypy.objspace.std.bytearraytype import getbytevalue
+    idx = space.getindex_w(w_index, space.w_IndexError, "bytearray index")
+    try:
+        w_bytearray.data[idx] = getbytevalue(space, w_item)
+    except IndexError:
+        raise OperationError(space.w_IndexError,
+                             space.wrap("bytearray index out of range"))
+
+def setitem__Bytearray_Slice_ANY(space, w_bytearray, w_slice, w_other):
+    oldsize = len(w_bytearray.data)
+    start, stop, step, slicelength = w_slice.indices4(space, oldsize)
+    if step != 1:
+        raise OperationError(space.w_NotImplementedError,
+                             space.wrap("fixme: only step=1 for the moment"))
+    assert start >= 0
+    assert stop >= 0
+    w_bytearray.data[start:stop] = [c for c in space.str_w(w_other)]
+
 from pypy.objspace.std import bytearraytype
 register_all(vars(), bytearraytype)

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	Wed Oct 20 13:27:34 2010
@@ -143,6 +143,12 @@
         b.append(ord('e'))
         assert b == 'abcde'
 
+    def test_iadd(self):
+        b = bytearray('abc')
+        b += 'def'
+        assert b == 'abcdef'
+        assert isinstance(b, bytearray)
+
     def test_extend(self):
         b = bytearray('abc')
         b.extend(bytearray('def'))
@@ -155,3 +161,17 @@
         assert b == 'abcdei'
         del b[:3]
         assert b == 'dei'
+
+    def test_setitem(self):
+        b = bytearray('abcdefghi')
+        b[1] = 'B'
+        assert b == 'aBcdefghi'
+
+    def test_setitem_slice(self):
+        b = bytearray('abcdefghi')
+        b[0:3] = 'ABC'
+        assert b == 'ABCdefghi'
+        b[3:3] = '...'
+        assert b == 'ABC...defghi'
+        b[3:6] = '()'
+        assert b == 'ABC()defghi'


From afa at codespeak.net  Wed Oct 20 13:32:47 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 13:32:47 +0200 (CEST)
Subject: [pypy-svn] r78123 - pypy/branch/fast-forward/pypy/module/_io
Message-ID: <20101020113247.92D12282BDA@codespeak.net>

Author: afa
Date: Wed Oct 20 13:32:46 2010
New Revision: 78123

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_io.py
Log:
Add default constructors to all io classes


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py	Wed Oct 20 13:32:46 2010
@@ -45,36 +45,42 @@
     pass
 W_BytesIO.typedef = TypeDef(
     'BytesIO', W_BufferedIOBase.typedef,
+    __new__ = generic_new_descr(W_BytesIO),
     )
 
 class W_BufferedReader(W_BufferedIOBase):
     pass
 W_BufferedReader.typedef = TypeDef(
     'BufferedReader', W_BufferedIOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedReader),
     )
 
 class W_BufferedWriter(W_BufferedIOBase):
     pass
 W_BufferedWriter.typedef = TypeDef(
     'BufferedWriter', W_BufferedIOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedWriter),
     )
 
 class W_BufferedRWPair(W_BufferedIOBase):
     pass
 W_BufferedRWPair.typedef = TypeDef(
     'BufferedRWPair', W_BufferedIOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedRWPair),
     )
 
 class W_BufferedRandom(W_BufferedIOBase):
     pass
 W_BufferedRandom.typedef = TypeDef(
     'BufferedRandom', W_BufferedIOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedRandom),
     )
 
 class W_TextIOWrapper(W_TextIOBase):
     pass
 W_TextIOWrapper.typedef = TypeDef(
     'TextIOWrapper', W_TextIOBase.typedef,
+    __new__ = generic_new_descr(W_TextIOWrapper),
     )
 
 @unwrap_spec(ObjSpace, Arguments)


From arigo at codespeak.net  Wed Oct 20 13:33:38 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 13:33:38 +0200 (CEST)
Subject: [pypy-svn] r78124 - in pypy/trunk/pypy/jit/backend: llsupport
	llsupport/test test x86
Message-ID: <20101020113338.8E6C9282BDA@codespeak.net>

Author: arigo
Date: Wed Oct 20 13:33:36 2010
New Revision: 78124

Modified:
   pypy/trunk/pypy/jit/backend/llsupport/descr.py
   pypy/trunk/pypy/jit/backend/llsupport/ffisupport.py
   pypy/trunk/pypy/jit/backend/llsupport/llmodel.py
   pypy/trunk/pypy/jit/backend/llsupport/symbolic.py
   pypy/trunk/pypy/jit/backend/llsupport/test/test_descr.py
   pypy/trunk/pypy/jit/backend/llsupport/test/test_ffisupport.py
   pypy/trunk/pypy/jit/backend/test/runner_test.py
   pypy/trunk/pypy/jit/backend/x86/assembler.py
   pypy/trunk/pypy/jit/backend/x86/regalloc.py
   pypy/trunk/pypy/jit/backend/x86/regloc.py
   pypy/trunk/pypy/jit/backend/x86/rx86.py
Log:
Merge branch/jit-sign-descr, minus the change to rlib/libffi that will
be redone next.  Implement support for signed shorter-than-LONG integers
in all descrs of the JIT backend, namely for getfield(), getarrayitem()
and call().


Modified: pypy/trunk/pypy/jit/backend/llsupport/descr.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llsupport/descr.py	(original)
+++ pypy/trunk/pypy/jit/backend/llsupport/descr.py	Wed Oct 20 13:33:36 2010
@@ -82,6 +82,7 @@
 
     _is_pointer_field = False   # unless overridden by GcPtrFieldDescr
     _is_float_field = False     # unless overridden by FloatFieldDescr
+    _is_field_signed = False    # unless overridden by XxxFieldDescr
 
     def is_pointer_field(self):
         return self._is_pointer_field
@@ -89,6 +90,9 @@
     def is_float_field(self):
         return self._is_float_field
 
+    def is_field_signed(self):
+        return self._is_field_signed
+
     def repr_of_descr(self):
         return '<%s %s %s>' % (self._clsname, self.name, self.offset)
 
@@ -105,7 +109,7 @@
 def getFieldDescrClass(TYPE):
     return getDescrClass(TYPE, BaseFieldDescr, GcPtrFieldDescr,
                          NonGcPtrFieldDescr, 'Field', 'get_field_size',
-                         '_is_float_field')
+                         '_is_float_field', '_is_field_signed')
 
 def get_field_descr(gccache, STRUCT, fieldname):
     cache = gccache._cache_field
@@ -144,6 +148,7 @@
 
     _is_array_of_pointers = False      # unless overridden by GcPtrArrayDescr
     _is_array_of_floats   = False      # unless overridden by FloatArrayDescr
+    _is_item_signed       = False      # unless overridden by XxxArrayDescr
 
     def is_array_of_pointers(self):
         return self._is_array_of_pointers
@@ -151,6 +156,9 @@
     def is_array_of_floats(self):
         return self._is_array_of_floats
 
+    def is_item_signed(self):
+        return self._is_item_signed
+
     def repr_of_descr(self):
         return '<%s>' % self._clsname
 
@@ -186,12 +194,12 @@
 def getArrayDescrClass(ARRAY):
     return getDescrClass(ARRAY.OF, BaseArrayDescr, GcPtrArrayDescr,
                          NonGcPtrArrayDescr, 'Array', 'get_item_size',
-                         '_is_array_of_floats')
+                         '_is_array_of_floats', '_is_item_signed')
 
 def getArrayNoLengthDescrClass(ARRAY):
     return getDescrClass(ARRAY.OF, BaseArrayNoLengthDescr, GcPtrArrayNoLengthDescr,
                          NonGcPtrArrayNoLengthDescr, 'ArrayNoLength', 'get_item_size',
-                         '_is_array_of_floats')
+                         '_is_array_of_floats', '_is_item_signed')
 
 def get_array_descr(gccache, ARRAY):
     cache = gccache._cache_array
@@ -242,6 +250,9 @@
     def get_result_size(self, translate_support_code):
         raise NotImplementedError
 
+    def is_result_signed(self):
+        return False    # unless overridden
+
     def create_call_stub(self, rtyper, RESULT):
         def process(c):
             arg = 'args_%s[%d]' % (c, seen[c])
@@ -307,6 +318,10 @@
     _return_type = history.INT
     call_stub = staticmethod(lambda func, args_i, args_r, args_f: 0)
 
+    _is_result_signed = False      # can be overridden in XxxCallDescr
+    def is_result_signed(self):
+        return self._is_result_signed
+
 class DynamicIntCallDescr(BaseIntCallDescr):
     """
     calldescr that works for every integer type, by explicitly passing it the
@@ -314,12 +329,17 @@
     """
     _clsname = 'DynamicIntCallDescr'
 
-    def __init__(self, arg_classes, result_size, extrainfo=None):
+    def __init__(self, arg_classes, result_size, result_sign, extrainfo=None):
         BaseIntCallDescr.__init__(self, arg_classes, extrainfo)
-        self._result_size = result_size
+        assert isinstance(result_sign, bool)
+        self._result_size = chr(result_size)
+        self._result_sign = result_sign
 
     def get_result_size(self, translate_support_code):
-        return self._result_size
+        return ord(self._result_size)
+
+    def is_result_signed(self):
+        return self._result_sign
 
 
 class NonGcPtrCallDescr(BaseIntCallDescr):
@@ -356,7 +376,8 @@
         return FloatCallDescr
     return getDescrClass(RESULT, BaseIntCallDescr, GcPtrCallDescr,
                          NonGcPtrCallDescr, 'Call', 'get_result_size',
-                         Ellipsis)  # <= floatattrname should not be used here
+                         Ellipsis,  # <= floatattrname should not be used here
+                         '_is_result_signed')
 
 def get_call_descr(gccache, ARGS, RESULT, extrainfo=None):
     arg_classes = []
@@ -383,7 +404,8 @@
 # ____________________________________________________________
 
 def getDescrClass(TYPE, BaseDescr, GcPtrDescr, NonGcPtrDescr,
-                  nameprefix, methodname, floatattrname, _cache={}):
+                  nameprefix, methodname, floatattrname, signedattrname,
+                  _cache={}):
     if isinstance(TYPE, lltype.Ptr):
         if TYPE.TO._gckind == 'gc':
             return GcPtrDescr
@@ -403,6 +425,8 @@
         #
         if TYPE is lltype.Float:
             setattr(Descr, floatattrname, True)
+        elif TYPE is not lltype.Bool and rffi.cast(TYPE, -1) == -1:
+            setattr(Descr, signedattrname, True)
         #
         _cache[nameprefix, TYPE] = Descr
         return Descr

Modified: pypy/trunk/pypy/jit/backend/llsupport/ffisupport.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llsupport/ffisupport.py	(original)
+++ pypy/trunk/pypy/jit/backend/llsupport/ffisupport.py	Wed Oct 20 13:33:36 2010
@@ -14,7 +14,8 @@
     arg_classes = ''.join(argkinds)
     if reskind == history.INT:
         size = intmask(ffi_result.c_size)
-        return DynamicIntCallDescr(arg_classes, size, extrainfo)
+        signed = is_ffi_type_signed(ffi_result)
+        return DynamicIntCallDescr(arg_classes, size, signed, extrainfo)
     elif reskind == history.REF:
         return  NonGcPtrCallDescr(arg_classes, extrainfo)
     elif reskind == history.FLOAT:
@@ -26,10 +27,15 @@
 def get_ffi_type_kind(ffi_type):
     from pypy.rlib.libffi import types
     kind = types.getkind(ffi_type)
-    if kind == 'i':
+    if kind == 'i' or kind == 'u':
         return history.INT
     elif kind == 'f':
         return history.FLOAT
     elif kind == 'v':
         return history.VOID
-    assert False, "Unsuported kind '%s'" % kind
+    assert False, "Unsupported kind '%s'" % kind
+
+def is_ffi_type_signed(ffi_type):
+    from pypy.rlib.libffi import types
+    kind = types.getkind(ffi_type)
+    return kind != 'u'

Modified: pypy/trunk/pypy/jit/backend/llsupport/llmodel.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llsupport/llmodel.py	(original)
+++ pypy/trunk/pypy/jit/backend/llsupport/llmodel.py	Wed Oct 20 13:33:36 2010
@@ -212,7 +212,8 @@
         assert isinstance(fielddescr, BaseFieldDescr)
         ofs = fielddescr.offset
         size = fielddescr.get_field_size(self.translate_support_code)
-        return ofs, size
+        sign = fielddescr.is_field_signed()
+        return ofs, size, sign
     unpack_fielddescr_size._always_inline_ = True
 
     def arraydescrof(self, A):
@@ -227,7 +228,8 @@
         assert isinstance(arraydescr, BaseArrayDescr)
         ofs = arraydescr.get_base_size(self.translate_support_code)
         size = arraydescr.get_item_size(self.translate_support_code)
-        return ofs, size
+        sign = arraydescr.is_item_signed()
+        return ofs, size, sign
     unpack_arraydescr_size._always_inline_ = True
 
     def calldescrof(self, FUNC, ARGS, RESULT, extrainfo=None):
@@ -257,15 +259,21 @@
 
     @specialize.argtype(2)
     def bh_getarrayitem_gc_i(self, arraydescr, gcref, itemindex):
-        ofs, size = self.unpack_arraydescr_size(arraydescr)
+        ofs, size, sign = self.unpack_arraydescr_size(arraydescr)
         # --- start of GC unsafe code (no GC operation!) ---
         items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for STYPE, UTYPE, itemsize in unroll_basic_sizes:
             if size == itemsize:
-                items = rffi.cast(rffi.CArrayPtr(TYPE), items) 
-                val = items[itemindex]
+                if sign:
+                    items = rffi.cast(rffi.CArrayPtr(STYPE), items)
+                    val = items[itemindex]
+                    val = rffi.cast(lltype.Signed, val)
+                else:
+                    items = rffi.cast(rffi.CArrayPtr(UTYPE), items)
+                    val = items[itemindex]
+                    val = rffi.cast(lltype.Signed, val)
                 # --- end of GC unsafe code ---
-                return rffi.cast(lltype.Signed, val)
+                return val
         else:
             raise NotImplementedError("size = %d" % size)
 
@@ -290,10 +298,10 @@
 
     @specialize.argtype(2)
     def bh_setarrayitem_gc_i(self, arraydescr, gcref, itemindex, newvalue):
-        ofs, size = self.unpack_arraydescr_size(arraydescr)
+        ofs, size, sign = self.unpack_arraydescr_size(arraydescr)
         # --- start of GC unsafe code (no GC operation!) ---
         items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for TYPE, _, itemsize in unroll_basic_sizes:
             if size == itemsize:
                 items = rffi.cast(rffi.CArrayPtr(TYPE), items)
                 items[itemindex] = rffi.cast(TYPE, newvalue)
@@ -344,14 +352,22 @@
 
     @specialize.argtype(1)
     def _base_do_getfield_i(self, struct, fielddescr):
-        ofs, size = self.unpack_fielddescr_size(fielddescr)
+        ofs, size, sign = self.unpack_fielddescr_size(fielddescr)
         # --- start of GC unsafe code (no GC operation!) ---
         fieldptr = rffi.ptradd(rffi.cast(rffi.CCHARP, struct), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for STYPE, UTYPE, itemsize in unroll_basic_sizes:
             if size == itemsize:
-                val = rffi.cast(rffi.CArrayPtr(TYPE), fieldptr)[0]
+                # Note that in the common case where size==sizeof(Signed),
+                # both cases of what follows are doing the same thing.
+                # But gcc is clever enough to figure this out :-)
+                if sign:
+                    val = rffi.cast(rffi.CArrayPtr(STYPE), fieldptr)[0]
+                    val = rffi.cast(lltype.Signed, val)
+                else:
+                    val = rffi.cast(rffi.CArrayPtr(UTYPE), fieldptr)[0]
+                    val = rffi.cast(lltype.Signed, val)
                 # --- end of GC unsafe code ---
-                return rffi.cast(lltype.Signed, val)
+                return val
         else:
             raise NotImplementedError("size = %d" % size)
 
@@ -383,10 +399,10 @@
 
     @specialize.argtype(1)
     def _base_do_setfield_i(self, struct, fielddescr, newvalue):
-        ofs, size = self.unpack_fielddescr_size(fielddescr)
+        ofs, size, sign = self.unpack_fielddescr_size(fielddescr)
         # --- start of GC unsafe code (no GC operation!) ---
         fieldptr = rffi.ptradd(rffi.cast(rffi.CCHARP, struct), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for TYPE, _, itemsize in unroll_basic_sizes:
             if size == itemsize:
                 fieldptr = rffi.cast(rffi.CArrayPtr(TYPE), fieldptr)
                 fieldptr[0] = rffi.cast(TYPE, newvalue)

Modified: pypy/trunk/pypy/jit/backend/llsupport/symbolic.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llsupport/symbolic.py	(original)
+++ pypy/trunk/pypy/jit/backend/llsupport/symbolic.py	Wed Oct 20 13:33:36 2010
@@ -69,8 +69,9 @@
 SIZEOF_INT   = get_size(rffi.INT, False)
 SIZEOF_FLOAT = get_size(lltype.Float, False)
 
-unroll_basic_sizes = unrolling_iterable([(lltype.Signed, WORD),
-                                         (lltype.Char,   SIZEOF_CHAR),
-                                         (rffi.SHORT,    SIZEOF_SHORT),
-                                         (rffi.INT,      SIZEOF_INT)])
+unroll_basic_sizes = unrolling_iterable([
+    (lltype.Signed,   lltype.Unsigned, WORD),
+    (rffi.SIGNEDCHAR, lltype.Char,     SIZEOF_CHAR),
+    (rffi.SHORT,      rffi.USHORT,     SIZEOF_SHORT),
+    (rffi.INT,        rffi.UINT,       SIZEOF_INT)])
 # does not contain Float ^^^ which must be special-cased

Modified: pypy/trunk/pypy/jit/backend/llsupport/test/test_descr.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llsupport/test/test_descr.py	(original)
+++ pypy/trunk/pypy/jit/backend/llsupport/test/test_descr.py	Wed Oct 20 13:33:36 2010
@@ -83,6 +83,18 @@
         assert     descr_f.is_float_field()
 
 
+def test_get_field_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        S = lltype.GcStruct('S', ('x', RESTYPE))
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            descr_x = get_field_descr(c2, S, 'x')
+            assert descr_x.is_field_signed() == signed
+
+
 def test_get_array_descr():
     U = lltype.Struct('U')
     T = lltype.GcStruct('T')
@@ -164,6 +176,25 @@
     assert descr.get_base_size(False) == 0
     assert descr.get_ofs_length(False) == -1
 
+
+def test_get_array_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        A = lltype.GcArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            arraydescr = get_array_descr(c2, A)
+            assert arraydescr.is_item_signed() == signed
+        #
+        RA = rffi.CArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            arraydescr = get_array_descr(c2, RA)
+            assert arraydescr.is_item_signed() == signed
+
+
 def test_get_call_descr_not_translated():
     c0 = GcCache(False)
     descr1 = get_call_descr(c0, [lltype.Char, lltype.Signed], lltype.Char)
@@ -219,6 +250,17 @@
     extrainfo = descr3.get_extra_info()
     assert extrainfo is None
 
+def test_get_call_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        A = lltype.GcArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            descr1 = get_call_descr(c2, [], RESTYPE)
+            assert descr1.is_result_signed() == signed
+
 
 def test_repr_of_descr():
     c0 = GcCache(False)

Modified: pypy/trunk/pypy/jit/backend/llsupport/test/test_ffisupport.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llsupport/test/test_ffisupport.py	(original)
+++ pypy/trunk/pypy/jit/backend/llsupport/test/test_ffisupport.py	Wed Oct 20 13:33:36 2010
@@ -12,6 +12,12 @@
     descr = get_call_descr_dynamic([], types.sint8)
     assert isinstance(descr, DynamicIntCallDescr)
     assert descr.get_result_size(False) == 1
+    assert descr.is_result_signed() == True
+
+    descr = get_call_descr_dynamic([], types.uint8)
+    assert isinstance(descr, DynamicIntCallDescr)
+    assert descr.get_result_size(False) == 1
+    assert descr.is_result_signed() == False
 
     descr = get_call_descr_dynamic([], types.float)
     assert descr is None # single floats are not supported so far

Modified: pypy/trunk/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/trunk/pypy/jit/backend/test/runner_test.py	Wed Oct 20 13:33:36 2010
@@ -15,6 +15,7 @@
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.llinterp import LLException
 from pypy.jit.codewriter import heaptracker
+from pypy.rlib.rarithmetic import intmask
 
 
 class Runner(object):
@@ -2028,6 +2029,196 @@
         assert self.cpu.get_latest_value_float(0) == 13.5
         assert called
 
+    def test_short_result_of_getfield_direct(self):
+        # Test that a getfield that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            S = lltype.GcStruct('S', ('x', RESTYPE))
+            descrfld_x = cpu.fielddescrof(S, 'x')
+            s = lltype.malloc(S)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            s.x = rffi.cast(RESTYPE, value)
+            x = cpu.bh_getfield_gc_i(lltype.cast_opaque_ptr(llmemory.GCREF, s),
+                                     descrfld_x)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_getfield_compiled(self):
+        # Test that a getfield that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            S = lltype.GcStruct('S', ('x', RESTYPE))
+            descrfld_x = cpu.fielddescrof(S, 'x')
+            s = lltype.malloc(S)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            s.x = rffi.cast(RESTYPE, value)
+            s_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, s)
+            res = self.execute_operation(rop.GETFIELD_GC, [BoxPtr(s_gcref)],
+                                         'int', descr=descrfld_x)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
+    def test_short_result_of_getarrayitem_direct(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = lltype.GcArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            x = cpu.bh_getarrayitem_gc_i(
+                descrarray, lltype.cast_opaque_ptr(llmemory.GCREF, a), 3)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_getarrayitem_compiled(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = lltype.GcArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, a)
+            res = self.execute_operation(rop.GETARRAYITEM_GC,
+                                         [BoxPtr(a_gcref), BoxInt(3)],
+                                         'int', descr=descrarray)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
+    def test_short_result_of_getarrayitem_raw_direct(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = rffi.CArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5, flavor='raw')
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_rawint = heaptracker.adr2int(llmemory.cast_ptr_to_adr(a))
+            x = cpu.bh_getarrayitem_raw_i(descrarray, a_rawint, 3)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+            lltype.free(a, flavor='raw')
+
+    def test_short_result_of_getarrayitem_raw_compiled(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = rffi.CArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5, flavor='raw')
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_rawint = heaptracker.adr2int(llmemory.cast_ptr_to_adr(a))
+            res = self.execute_operation(rop.GETARRAYITEM_RAW,
+                                         [BoxInt(a_rawint), BoxInt(3)],
+                                         'int', descr=descrarray)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+            lltype.free(a, flavor='raw')
+
+    def test_short_result_of_call_direct(self):
+        # Test that calling a function that returns a CHAR, SHORT or INT,
+        # signed or unsigned, properly gets zero-extended or sign-extended.
+        from pypy.translator.tool.cbuild import ExternalCompilationInfo
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            # Tested with a function that intentionally does not cast the
+            # result to RESTYPE, but makes sure that we return the whole
+            # value in eax or rax.
+            eci = ExternalCompilationInfo(separate_module_sources=["""
+                long fn_test_result_of_call(long x)
+                {
+                    return x + 1;
+                }
+            """])
+            f = rffi.llexternal('fn_test_result_of_call', [lltype.Signed],
+                                RESTYPE, compilation_info=eci, _nowrapper=True)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value + 1))
+            assert intmask(f(value)) == expected
+            #
+            FUNC = self.FuncType([lltype.Signed], RESTYPE)
+            FPTR = self.Ptr(FUNC)
+            calldescr = self.cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
+            x = self.cpu.bh_call_i(self.get_funcbox(self.cpu, f).value,
+                                   calldescr, [value], None, None)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_call_compiled(self):
+        # Test that calling a function that returns a CHAR, SHORT or INT,
+        # signed or unsigned, properly gets zero-extended or sign-extended.
+        from pypy.translator.tool.cbuild import ExternalCompilationInfo
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            # Tested with a function that intentionally does not cast the
+            # result to RESTYPE, but makes sure that we return the whole
+            # value in eax or rax.
+            eci = ExternalCompilationInfo(separate_module_sources=["""
+                long fn_test_result_of_call(long x)
+                {
+                    return x + 1;
+                }
+            """])
+            f = rffi.llexternal('fn_test_result_of_call', [lltype.Signed],
+                                RESTYPE, compilation_info=eci, _nowrapper=True)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value + 1))
+            assert intmask(f(value)) == expected
+            #
+            FUNC = self.FuncType([lltype.Signed], RESTYPE)
+            FPTR = self.Ptr(FUNC)
+            calldescr = self.cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
+            funcbox = self.get_funcbox(self.cpu, f)
+            res = self.execute_operation(rop.CALL, [funcbox, BoxInt(value)],
+                                         'int', descr=calldescr)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
 
 class OOtypeBackendTest(BaseBackendTest):
 

Modified: pypy/trunk/pypy/jit/backend/x86/assembler.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/assembler.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/assembler.py	Wed Oct 20 13:33:36 2010
@@ -8,7 +8,8 @@
 from pypy.rpython.annlowlevel import llhelper
 from pypy.tool.uid import fixid
 from pypy.jit.backend.x86.regalloc import (RegAlloc, X86RegisterManager,
-                                           X86XMMRegisterManager, get_ebp_ofs)
+                                           X86XMMRegisterManager, get_ebp_ofs,
+                                           _get_scale)
 
 from pypy.jit.backend.x86.arch import (FRAME_FIXED_SIZE, FORCE_INDEX_OFS, WORD,
                                        IS_X86_32, IS_X86_64)
@@ -22,7 +23,8 @@
                                          X86_64_SCRATCH_REG,
                                          X86_64_XMM_SCRATCH_REG,
                                          RegLoc, StackLoc, ConstFloatLoc,
-                                         ImmedLoc, AddressLoc, imm)
+                                         ImmedLoc, AddressLoc, imm,
+                                         imm0, imm1)
 
 from pypy.rlib.objectmodel import we_are_translated, specialize
 from pypy.jit.backend.x86 import rx86, regloc, codebuf
@@ -442,7 +444,7 @@
             # self.mc.PUSH(eax)
             # adr = rffi.cast(lltype.Signed, self.loop_run_counters[-1][1])
             # self.mc.MOV(eax, heap(adr))
-            # self.mc.ADD(eax, imm(1))
+            # self.mc.ADD(eax, imm1)
             # self.mc.MOV(heap(adr), eax)
             # self.mc.POP(eax)
         return operations
@@ -713,7 +715,7 @@
         self.regalloc_perform_with_guard(None, guard_op, faillocs, arglocs,
                                          resloc, current_depths)
 
-    def load_effective_addr(self, sizereg, baseofs, scale, result, frm=imm(0)):
+    def load_effective_addr(self, sizereg, baseofs, scale, result, frm=imm0):
         self.mc.LEA(result, addr_add(frm, sizereg, baseofs, scale))
 
     def _unaryop(asmop):
@@ -975,28 +977,28 @@
 
     def genop_guard_int_is_true(self, op, guard_op, guard_token, arglocs, resloc):
         guard_opnum = guard_op.getopnum()
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         if guard_opnum == rop.GUARD_TRUE:
             return self.implement_guard(guard_token, 'Z')
         else:
             return self.implement_guard(guard_token, 'NZ')
 
     def genop_int_is_true(self, op, arglocs, resloc):
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         rl = resloc.lowest8bits()
         self.mc.SET_ir(rx86.Conditions['NE'], rl.value)
         self.mc.MOVZX8(resloc, rl)
 
     def genop_guard_int_is_zero(self, op, guard_op, guard_token, arglocs, resloc):
         guard_opnum = guard_op.getopnum()
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         if guard_opnum == rop.GUARD_TRUE:
             return self.implement_guard(guard_token, 'NZ')
         else:
             return self.implement_guard(guard_token, 'Z')
 
     def genop_int_is_zero(self, op, arglocs, resloc):
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         rl = resloc.lowest8bits()
         self.mc.SET_ir(rx86.Conditions['E'], rl.value)
         self.mc.MOVZX8(resloc, rl)
@@ -1052,50 +1054,66 @@
         assert result_loc is eax
         self.call(self.malloc_unicode_func_addr, arglocs, eax)
 
-    def genop_getfield_gc(self, op, arglocs, resloc):
-        base_loc, ofs_loc, size_loc = arglocs
-        assert isinstance(size_loc, ImmedLoc)
+    # ----------
+
+    def load_from_mem(self, resloc, source_addr, size_loc, sign_loc):
         assert isinstance(resloc, RegLoc)
         size = size_loc.value
-
-        source_addr = AddressLoc(base_loc, ofs_loc)
+        sign = sign_loc.value
         if resloc.is_xmm:
             self.mc.MOVSD(resloc, source_addr)
+        elif size == WORD:
+            self.mc.MOV(resloc, source_addr)
+        elif size == 1:
+            if sign:
+                self.mc.MOVSX8(resloc, source_addr)
+            else:
+                self.mc.MOVZX8(resloc, source_addr)
+        elif size == 2:
+            if sign:
+                self.mc.MOVSX16(resloc, source_addr)
+            else:
+                self.mc.MOVZX16(resloc, source_addr)
+        elif IS_X86_64 and size == 4:
+            if sign:
+                self.mc.MOVSX32(resloc, source_addr)
+            else:
+                self.mc.MOV32(resloc, source_addr)    # zero-extending
+        else:
+            not_implemented("load_from_mem size = %d" % size)
+
+    def save_into_mem(self, dest_addr, value_loc, size_loc):
+        size = size_loc.value
+        if isinstance(value_loc, RegLoc) and value_loc.is_xmm:
+            self.mc.MOVSD(dest_addr, value_loc)
         elif size == 1:
-            self.mc.MOVZX8(resloc, source_addr)
+            self.mc.MOV8(dest_addr, value_loc.lowest8bits())
         elif size == 2:
-            self.mc.MOVZX16(resloc, source_addr)
+            self.mc.MOV16(dest_addr, value_loc)
         elif size == 4:
-            # MOV32 is zero-extending on 64-bit, so this is okay
-            self.mc.MOV32(resloc, source_addr)
+            self.mc.MOV32(dest_addr, value_loc)
         elif IS_X86_64 and size == 8:
-            self.mc.MOV(resloc, source_addr)
+            self.mc.MOV(dest_addr, value_loc)
         else:
-            raise NotImplementedError("getfield size = %d" % size)
+            not_implemented("save_into_mem size = %d" % size)
+
+    def genop_getfield_gc(self, op, arglocs, resloc):
+        base_loc, ofs_loc, size_loc, sign_loc = arglocs
+        assert isinstance(size_loc, ImmedLoc)
+        source_addr = AddressLoc(base_loc, ofs_loc)
+        self.load_from_mem(resloc, source_addr, size_loc, sign_loc)
 
     genop_getfield_raw = genop_getfield_gc
     genop_getfield_raw_pure = genop_getfield_gc
     genop_getfield_gc_pure = genop_getfield_gc
 
     def genop_getarrayitem_gc(self, op, arglocs, resloc):
-        base_loc, ofs_loc, scale, ofs = arglocs
+        base_loc, ofs_loc, size_loc, ofs, sign_loc = arglocs
         assert isinstance(ofs, ImmedLoc)
-        assert isinstance(scale, ImmedLoc)
-        src_addr = addr_add(base_loc, ofs_loc, ofs.value, scale.value)
-        if op.result.type == FLOAT:
-            self.mc.MOVSD(resloc, src_addr)
-        else:
-            if scale.value == 0:
-                self.mc.MOVZX8(resloc, src_addr)
-            elif scale.value == 1:
-                self.mc.MOVZX16(resloc, src_addr)
-            elif scale.value == 2:
-                self.mc.MOV32(resloc, src_addr)
-            elif IS_X86_64 and scale.value == 3:
-                self.mc.MOV(resloc, src_addr)
-            else:
-                print "[asmgen]getarrayitem unsupported size: %d" % scale.value
-                raise NotImplementedError()
+        assert isinstance(size_loc, ImmedLoc)
+        scale = _get_scale(size_loc.value)
+        src_addr = addr_add(base_loc, ofs_loc, ofs.value, scale)
+        self.load_from_mem(resloc, src_addr, size_loc, sign_loc)
 
     genop_getarrayitem_gc_pure = genop_getarrayitem_gc
     genop_getarrayitem_raw = genop_getarrayitem_gc
@@ -1103,40 +1121,16 @@
     def genop_discard_setfield_gc(self, op, arglocs):
         base_loc, ofs_loc, size_loc, value_loc = arglocs
         assert isinstance(size_loc, ImmedLoc)
-        size = size_loc.value
         dest_addr = AddressLoc(base_loc, ofs_loc)
-        if isinstance(value_loc, RegLoc) and value_loc.is_xmm:
-            self.mc.MOVSD(dest_addr, value_loc)
-        elif IS_X86_64 and size == 8:
-            self.mc.MOV(dest_addr, value_loc)
-        elif size == 4:
-            self.mc.MOV32(dest_addr, value_loc)
-        elif size == 2:
-            self.mc.MOV16(dest_addr, value_loc)
-        elif size == 1:
-            self.mc.MOV8(dest_addr, value_loc.lowest8bits())
-        else:
-            print "[asmgen]setfield addr size %d" % size
-            raise NotImplementedError("Addr size %d" % size)
+        self.save_into_mem(dest_addr, value_loc, size_loc)
 
     def genop_discard_setarrayitem_gc(self, op, arglocs):
-        base_loc, ofs_loc, value_loc, scale_loc, baseofs = arglocs
+        base_loc, ofs_loc, value_loc, size_loc, baseofs = arglocs
         assert isinstance(baseofs, ImmedLoc)
-        assert isinstance(scale_loc, ImmedLoc)
-        dest_addr = AddressLoc(base_loc, ofs_loc, scale_loc.value, baseofs.value)
-        if op.getarg(2).type == FLOAT:
-            self.mc.MOVSD(dest_addr, value_loc)
-        else:
-            if IS_X86_64 and scale_loc.value == 3:
-                self.mc.MOV(dest_addr, value_loc)
-            elif scale_loc.value == 2:
-                self.mc.MOV32(dest_addr, value_loc)
-            elif scale_loc.value == 1:
-                self.mc.MOV16(dest_addr, value_loc)
-            elif scale_loc.value == 0:
-                self.mc.MOV8(dest_addr, value_loc.lowest8bits())
-            else:
-                raise NotImplementedError("scale = %d" % scale_loc.value)
+        assert isinstance(size_loc, ImmedLoc)
+        scale = _get_scale(size_loc.value)
+        dest_addr = AddressLoc(base_loc, ofs_loc, scale, baseofs.value)
+        self.save_into_mem(dest_addr, value_loc, size_loc)
 
     def genop_discard_strsetitem(self, op, arglocs):
         base_loc, ofs_loc, val_loc = arglocs
@@ -1203,7 +1197,7 @@
 
     def genop_guard_guard_no_exception(self, ign_1, guard_op, guard_token,
                                        locs, ign_2):
-        self.mc.CMP(heap(self.cpu.pos_exception()), imm(0))
+        self.mc.CMP(heap(self.cpu.pos_exception()), imm0)
         return self.implement_guard(guard_token, 'NZ')
 
     def genop_guard_guard_exception(self, ign_1, guard_op, guard_token,
@@ -1215,8 +1209,8 @@
         addr = self.implement_guard(guard_token, 'NE')
         if resloc is not None:
             self.mc.MOV(resloc, heap(self.cpu.pos_exc_value()))
-        self.mc.MOV(heap(self.cpu.pos_exception()), imm(0))
-        self.mc.MOV(heap(self.cpu.pos_exc_value()), imm(0))
+        self.mc.MOV(heap(self.cpu.pos_exception()), imm0)
+        self.mc.MOV(heap(self.cpu.pos_exc_value()), imm0)
         return addr
 
     def _gen_guard_overflow(self, guard_op, guard_token):
@@ -1226,8 +1220,8 @@
         elif guard_opnum == rop.GUARD_OVERFLOW:
             return self.implement_guard(guard_token, 'NO')
         else:
-            print "int_xxx_ovf followed by", guard_op.getopname()
-            raise AssertionError
+            not_implemented("int_xxx_ovf followed by %s" %
+                            guard_op.getopname())
 
     def genop_guard_int_add_ovf(self, op, guard_op, guard_token, arglocs, result_loc):
         self.genop_int_add(op, arglocs, result_loc)
@@ -1290,7 +1284,7 @@
     def genop_guard_guard_nonnull_class(self, ign_1, guard_op,
                                         guard_token, locs, ign_2):
         self.mc.ensure_bytes_available(256)
-        self.mc.CMP(locs[0], imm(1))
+        self.mc.CMP(locs[0], imm1)
         # Patched below
         self.mc.J_il8(rx86.Conditions['B'], 0)
         jb_location = self.mc.get_relative_pos()
@@ -1639,25 +1633,34 @@
         sizeloc = arglocs[0]
         assert isinstance(sizeloc, ImmedLoc)
         size = sizeloc.value
+        signloc = arglocs[1]
 
         if isinstance(op.getarg(0), Const):
             x = imm(op.getarg(0).getint())
         else:
-            x = arglocs[1]
+            x = arglocs[2]
         if x is eax:
             tmp = ecx
         else:
             tmp = eax
         
-        self._emit_call(x, arglocs, 2, tmp=tmp)
+        self._emit_call(x, arglocs, 3, tmp=tmp)
+
+        if IS_X86_32 and isinstance(resloc, StackLoc) and resloc.width == 8:
+            self.mc.FSTP_b(resloc.value)   # float return
+        elif size == WORD:
+            assert resloc is eax or resloc is xmm0    # a full word
+        elif size == 0:
+            pass    # void return
+        else:
+            # use the code in load_from_mem to do the zero- or sign-extension
+            assert resloc is eax
+            if size == 1:
+                srcloc = eax.lowest8bits()
+            else:
+                srcloc = eax
+            self.load_from_mem(eax, srcloc, sizeloc, signloc)
 
-        if isinstance(resloc, StackLoc) and resloc.width == 8 and IS_X86_32:
-            self.mc.FSTP_b(resloc.value)
-        elif size == 1:
-            self.mc.AND_ri(eax.value, 0xff)
-        elif size == 2:
-            self.mc.AND_ri(eax.value, 0xffff)
-    
     def genop_guard_call_may_force(self, op, guard_op, guard_token,
                                    arglocs, result_loc):
         faildescr = guard_op.getdescr()
@@ -1810,20 +1813,16 @@
         self.mc.LEA_rb(resloc.value, FORCE_INDEX_OFS)
 
     def not_implemented_op_discard(self, op, arglocs):
-        msg = "not implemented operation: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation: %s" % op.getopname())
 
     def not_implemented_op(self, op, arglocs, resloc):
-        msg = "not implemented operation with res: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation with res: %s" %
+                        op.getopname())
 
     def not_implemented_op_guard(self, op, guard_op,
                                  failaddr, arglocs, resloc):
-        msg = "not implemented operation (guard): %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation (guard): %s" %
+                        op.getopname())
 
     def mark_gc_roots(self):
         gcrootmap = self.cpu.gc_ll_descr.gcrootmap
@@ -1907,3 +1906,7 @@
 
 def heap(addr):
     return AddressLoc(ImmedLoc(addr), ImmedLoc(0), 0, 0)
+
+def not_implemented(msg):
+    os.write(2, '[x86/asm] %s\n' % msg)
+    raise NotImplementedError(msg)

Modified: pypy/trunk/pypy/jit/backend/x86/regalloc.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/regalloc.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/regalloc.py	Wed Oct 20 13:33:36 2010
@@ -2,6 +2,7 @@
 """ Register allocation scheme.
 """
 
+import os
 from pypy.jit.metainterp.history import (Box, Const, ConstInt, ConstPtr,
                                          ResOperation, BoxPtr,
                                          LoopToken, INT, REF, FLOAT)
@@ -40,12 +41,10 @@
             return imm(c.value)
         elif isinstance(c, ConstPtr):
             if we_are_translated() and c.value and rgc.can_move(c.value):
-                print "convert_to_imm: ConstPtr needs special care"
-                raise AssertionError
+                not_implemented("convert_to_imm: ConstPtr needs special care")
             return imm(rffi.cast(lltype.Signed, c.value))
         else:
-            print "convert_to_imm: got a %s" % c
-            raise AssertionError
+            not_implemented("convert_to_imm: got a %s" % c)
 
 class X86_64_RegisterManager(X86RegisterManager):
     # r11 omitted because it's used as scratch
@@ -360,8 +359,8 @@
             if op.is_ovf():
                 if (operations[i + 1].getopnum() != rop.GUARD_NO_OVERFLOW and
                     operations[i + 1].getopnum() != rop.GUARD_OVERFLOW):
-                    print "int_xxx_ovf not followed by guard_(no)_overflow"
-                    raise AssertionError
+                    not_implemented("int_xxx_ovf not followed by "
+                                    "guard_(no)_overflow")
                 return True
             return False
         if (operations[i + 1].getopnum() != rop.GUARD_TRUE and
@@ -413,8 +412,8 @@
                 arg = op.getarg(j)
                 if isinstance(arg, Box):
                     if arg not in start_live:
-                        print "Bogus arg in operation %d at %d" % (op.getopnum(), i)
-                        raise AssertionError
+                        not_implemented("Bogus arg in operation %d at %d" %
+                                        (op.getopnum(), i))
                     longevity[arg] = (start_live[arg], i)
             if op.is_guard():
                 for arg in op.getfailargs():
@@ -422,8 +421,8 @@
                         continue
                     assert isinstance(arg, Box)
                     if arg not in start_live:
-                        print "Bogus arg in guard %d at %d" % (op.getopnum(), i)
-                        raise AssertionError
+                        not_implemented("Bogus arg in guard %d at %d" %
+                                        (op.getopnum(), i))
                     longevity[arg] = (start_live[arg], i)
         for arg in inputargs:
             if arg not in longevity:
@@ -668,7 +667,13 @@
         assert isinstance(calldescr, BaseCallDescr)
         assert len(calldescr.arg_classes) == op.numargs() - 1
         size = calldescr.get_result_size(self.translate_support_code)
-        self._call(op, [imm(size)] + [self.loc(op.getarg(i)) for i in range(op.numargs())],
+        sign = calldescr.is_result_signed()
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self._call(op, [imm(size), sign_loc] +
+                       [self.loc(op.getarg(i)) for i in range(op.numargs())],
                    guard_not_forced_op=guard_not_forced_op)
 
     def consider_call(self, op):
@@ -689,7 +694,7 @@
             self.rm._sync_var(op.getarg(vable_index))
             vable = self.fm.loc(op.getarg(vable_index))
         else:
-            vable = imm(0)
+            vable = imm0
         self._call(op, [imm(size), vable] +
                    [self.loc(op.getarg(i)) for i in range(op.numargs())],
                    guard_not_forced_op=guard_op)
@@ -815,8 +820,9 @@
             arglocs.append(self.loc(op.getarg(0)))
             return self._call(op, arglocs)
         # boehm GC (XXX kill the following code at some point)
-        scale_of_field, basesize, ofs_length, _ = (
+        itemsize, basesize, ofs_length, _, _ = (
             self._unpack_arraydescr(op.getdescr()))
+        scale_of_field = _get_scale(itemsize)
         return self._malloc_varsize(basesize, ofs_length, scale_of_field,
                                     op.getarg(0), op.result)
 
@@ -826,21 +832,19 @@
         ofs = arraydescr.get_base_size(self.translate_support_code)
         size = arraydescr.get_item_size(self.translate_support_code)
         ptr = arraydescr.is_array_of_pointers()
-        scale = 0
-        while (1 << scale) < size:
-            scale += 1
-        assert (1 << scale) == size
-        return scale, ofs, ofs_length, ptr
+        sign = arraydescr.is_item_signed()
+        return size, ofs, ofs_length, ptr, sign
 
     def _unpack_fielddescr(self, fielddescr):
         assert isinstance(fielddescr, BaseFieldDescr)
         ofs = fielddescr.offset
         size = fielddescr.get_field_size(self.translate_support_code)
         ptr = fielddescr.is_pointer_field()
-        return imm(ofs), imm(size), ptr
+        sign = fielddescr.is_field_signed()
+        return imm(ofs), imm(size), ptr, sign
 
     def consider_setfield_gc(self, op):
-        ofs_loc, size_loc, ptr = self._unpack_fielddescr(op.getdescr())
+        ofs_loc, size_loc, _, _ = self._unpack_fielddescr(op.getdescr())
         assert isinstance(size_loc, ImmedLoc)
         if size_loc.value == 1:
             need_lower_byte = True
@@ -867,10 +871,10 @@
     consider_unicodesetitem = consider_strsetitem
 
     def consider_setarrayitem_gc(self, op):
-        scale, ofs, _, ptr = self._unpack_arraydescr(op.getdescr())
+        itemsize, ofs, _, _, _ = self._unpack_arraydescr(op.getdescr())
         args = op.getarglist()
         base_loc  = self.rm.make_sure_var_in_reg(op.getarg(0), args)
-        if scale == 0:
+        if itemsize == 1:
             need_lower_byte = True
         else:
             need_lower_byte = False
@@ -879,30 +883,39 @@
         ofs_loc = self.rm.make_sure_var_in_reg(op.getarg(1), args)
         self.possibly_free_vars(args)
         self.PerformDiscard(op, [base_loc, ofs_loc, value_loc,
-                                 imm(scale), imm(ofs)])
+                                 imm(itemsize), imm(ofs)])
 
     consider_setarrayitem_raw = consider_setarrayitem_gc
 
     def consider_getfield_gc(self, op):
-        ofs_loc, size_loc, _ = self._unpack_fielddescr(op.getdescr())
+        ofs_loc, size_loc, _, sign = self._unpack_fielddescr(op.getdescr())
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
         self.rm.possibly_free_vars(args)
         result_loc = self.force_allocate_reg(op.result)
-        self.Perform(op, [base_loc, ofs_loc, size_loc], result_loc)
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self.Perform(op, [base_loc, ofs_loc, size_loc, sign_loc], result_loc)
 
     consider_getfield_raw = consider_getfield_gc
     consider_getfield_raw_pure = consider_getfield_gc
     consider_getfield_gc_pure = consider_getfield_gc
 
     def consider_getarrayitem_gc(self, op):
-        scale, ofs, _, _ = self._unpack_arraydescr(op.getdescr())
+        itemsize, ofs, _, _, sign = self._unpack_arraydescr(op.getdescr())
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
         ofs_loc = self.rm.make_sure_var_in_reg(op.getarg(1), args)
         self.rm.possibly_free_vars_for_op(op)
         result_loc = self.force_allocate_reg(op.result)
-        self.Perform(op, [base_loc, ofs_loc, imm(scale), imm(ofs)], result_loc)
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self.Perform(op, [base_loc, ofs_loc, imm(itemsize), imm(ofs),
+                          sign_loc], result_loc)
 
     consider_getarrayitem_raw = consider_getarrayitem_gc
     consider_getarrayitem_gc_pure = consider_getarrayitem_gc
@@ -1091,15 +1104,11 @@
         self.Perform(op, [], loc)
 
     def not_implemented_op(self, op):
-        msg = "[regalloc] Not implemented operation: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation: %s" % op.getopname())
 
     def not_implemented_op_with_guard(self, op, guard_op):
-        msg = "[regalloc] Not implemented operation with guard: %s" % (
-            op.getopname(),)
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation with guard: %s" % (
+            op.getopname(),))
 
 oplist = [RegAlloc.not_implemented_op] * rop._LAST
 oplist_with_guard = [RegAlloc.not_implemented_op_with_guard] * rop._LAST
@@ -1133,3 +1142,14 @@
     # Returns (ebp-20), (ebp-24), (ebp-28)...
     # i.e. the n'th word beyond the fixed frame size.
     return -WORD * (FRAME_FIXED_SIZE + position)
+
+def _get_scale(size):
+    assert size == 1 or size == 2 or size == 4 or size == 8
+    if size < 4:
+        return size - 1         # 1, 2 => 0, 1
+    else:
+        return (size >> 2) + 1  # 4, 8 => 2, 3
+
+def not_implemented(msg):
+    os.write(2, '[x86/regalloc] %s\n' % msg)
+    raise NotImplementedError(msg)

Modified: pypy/trunk/pypy/jit/backend/x86/regloc.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/regloc.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/regloc.py	Wed Oct 20 13:33:36 2010
@@ -442,8 +442,11 @@
     MOV8 = _binaryop('MOV8')
     MOV16 = _16_bit_binaryop('MOV')
     MOVZX8 = _binaryop('MOVZX8')
+    MOVSX8 = _binaryop('MOVSX8')
     MOVZX16 = _binaryop('MOVZX16')
+    MOVSX16 = _binaryop('MOVSX16')
     MOV32 = _binaryop('MOV32')
+    MOVSX32 = _binaryop('MOVSX32')
     XCHG = _binaryop('XCHG')
 
     PUSH = _unaryop('PUSH')
@@ -473,6 +476,9 @@
     else:
         return ImmedLoc(x)
 
+imm0 = imm(0)
+imm1 = imm(1)
+
 all_extra_instructions = [name for name in LocationCodeBuilder.__dict__
                           if name[0].isupper()]
 all_extra_instructions.sort()

Modified: pypy/trunk/pypy/jit/backend/x86/rx86.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/rx86.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/rx86.py	Wed Oct 20 13:33:36 2010
@@ -642,7 +642,10 @@
 define_modrm_modes('MOV8_*i', [rex_w, '\xC6', orbyte(0<<3)], [immediate(2, 'b')], regtype='BYTE')
 
 define_modrm_modes('MOVZX8_r*', [rex_w, '\x0F\xB6', register(1, 8)], regtype='BYTE')
+define_modrm_modes('MOVSX8_r*', [rex_w, '\x0F\xBE', register(1, 8)], regtype='BYTE')
 define_modrm_modes('MOVZX16_r*', [rex_w, '\x0F\xB7', register(1, 8)])
+define_modrm_modes('MOVSX16_r*', [rex_w, '\x0F\xBF', register(1, 8)])
+define_modrm_modes('MOVSX32_r*', [rex_w, '\x63', register(1, 8)])
 
 define_modrm_modes('MOVSD_x*', ['\xF2', rex_nw, '\x0F\x10', register(1,8)], regtype='XMM')
 define_modrm_modes('MOVSD_*x', ['\xF2', rex_nw, '\x0F\x11', register(2,8)], regtype='XMM')


From arigo at codespeak.net  Wed Oct 20 13:35:29 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 13:35:29 +0200 (CEST)
Subject: [pypy-svn] r78125 - in pypy/trunk/pypy: module/_ffi
	module/_ffi/test rlib
Message-ID: <20101020113529.2A13D282BDA@codespeak.net>

Author: arigo
Date: Wed Oct 20 13:35:27 2010
New Revision: 78125

Modified:
   pypy/trunk/pypy/module/_ffi/interp_ffi.py
   pypy/trunk/pypy/module/_ffi/test/test__ffi.py
   pypy/trunk/pypy/rlib/libffi.py
Log:
Adapt libffi.py and module/_ffi to get the same signed/unsigned
distinction.  Add tests.


Modified: pypy/trunk/pypy/module/_ffi/interp_ffi.py
==============================================================================
--- pypy/trunk/pypy/module/_ffi/interp_ffi.py	(original)
+++ pypy/trunk/pypy/module/_ffi/interp_ffi.py	Wed Oct 20 13:35:27 2010
@@ -11,6 +11,7 @@
 from pypy.rlib import jit
 from pypy.rlib import libffi
 from pypy.rlib.rdynload import DLOpenError
+from pypy.rlib.rarithmetic import intmask
 
 class W_FFIType(Wrappable):
     def __init__(self, name, ffitype):
@@ -74,6 +75,8 @@
             kind = libffi.types.getkind(argtype)
             if kind == 'i':
                 argchain.arg(space.int_w(w_arg))
+            elif kind == 'u':
+                argchain.arg(intmask(space.uint_w(w_arg)))
             elif kind == 'f':
                 argchain.arg(space.float_w(w_arg))
             else:
@@ -88,6 +91,9 @@
         if reskind == 'i':
             intres = self.func.call(argchain, rffi.LONG)
             return space.wrap(intres)
+        elif reskind == 'u':
+            intres = self.func.call(argchain, rffi.ULONG)
+            return space.wrap(intres)
         elif reskind == 'f':
             floatres = self.func.call(argchain, rffi.DOUBLE)
             return space.wrap(floatres)

Modified: pypy/trunk/pypy/module/_ffi/test/test__ffi.py
==============================================================================
--- pypy/trunk/pypy/module/_ffi/test/test__ffi.py	(original)
+++ pypy/trunk/pypy/module/_ffi/test/test__ffi.py	Wed Oct 20 13:35:27 2010
@@ -97,6 +97,36 @@
         assert set_dummy(42) is None
         assert get_dummy() == 42
 
+    def test_unsigned_long_args(self):
+        """
+            unsigned long sum_xy_ul(unsigned long x, unsigned long y)
+            {
+                return x+y;
+            }
+        """
+        import sys
+        from _ffi import CDLL, types
+        libfoo = CDLL(self.libfoo_name)
+        sum_xy = libfoo.getfunc('sum_xy_ul', [types.ulong, types.ulong],
+                                types.ulong)
+        assert sum_xy(sys.maxint, 12) == sys.maxint+12
+        assert sum_xy(sys.maxint+1, 12) == sys.maxint+13
+
+    def test_unsigned_short_args(self):
+        """
+            unsigned short sum_xy_us(unsigned short x, unsigned short y)
+            {
+                return x+y;
+            }
+        """
+        import sys
+        from _ffi import CDLL, types
+        libfoo = CDLL(self.libfoo_name)
+        sum_xy = libfoo.getfunc('sum_xy_us', [types.ushort, types.ushort],
+                                types.ushort)
+        assert sum_xy(32000, 8000) == 40000
+        assert sum_xy(60000, 30000) == 90000 % 65536
+
     def test_TypeError_numargs(self):
         from _ffi import CDLL, types
         libfoo = CDLL(self.libfoo_name)

Modified: pypy/trunk/pypy/rlib/libffi.py
==============================================================================
--- pypy/trunk/pypy/rlib/libffi.py	(original)
+++ pypy/trunk/pypy/rlib/libffi.py	Wed Oct 20 13:35:27 2010
@@ -35,27 +35,33 @@
     @staticmethod
     @jit.purefunction
     def getkind(ffi_type):
+        """Returns 'v' for void, 'f' for float, 'i' for signed integer,
+        and 'u' for unsigned integer.
+        """
         if   ffi_type is types.void:    return 'v'
         elif ffi_type is types.double:  return 'f'
         elif ffi_type is types.pointer: return 'i'
         #
         elif ffi_type is types.schar:   return 'i'
-        elif ffi_type is types.uchar:   return 'i'
+        elif ffi_type is types.uchar:   return 'u'
         elif ffi_type is types.sshort:  return 'i'
-        elif ffi_type is types.ushort:  return 'i'
+        elif ffi_type is types.ushort:  return 'u'
         elif ffi_type is types.sint:    return 'i'
-        elif ffi_type is types.uint:    return 'i'
+        elif ffi_type is types.uint:    return 'u'
         elif ffi_type is types.slong:   return 'i'
-        elif ffi_type is types.ulong:   return 'i'
+        elif ffi_type is types.ulong:   return 'u'
         #
         elif ffi_type is types.sint8:   return 'i'
-        elif ffi_type is types.uint8:   return 'i'
+        elif ffi_type is types.uint8:   return 'u'
         elif ffi_type is types.sint16:  return 'i'
-        elif ffi_type is types.uint16:  return 'i'
+        elif ffi_type is types.uint16:  return 'u'
         elif ffi_type is types.sint32:  return 'i'
-        elif ffi_type is types.uint32:  return 'i'
+        elif ffi_type is types.uint32:  return 'u'
+        ## we only support integers that fit in a lltype.Signed (==rffi.LONG)
+        ## (on 64-bit platforms, types.sint64 is types.slong and the case is
+        ## caught above)
         ## elif ffi_type is types.sint64:  return 'i'
-        ## elif ffi_type is types.uint64:  return 'i'
+        ## elif ffi_type is types.uint64:  return 'u'
         raise KeyError
 
 types._import()


From arigo at codespeak.net  Wed Oct 20 13:43:06 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 13:43:06 +0200 (CEST)
Subject: [pypy-svn] r78126 - pypy/trunk/pypy/rlib
Message-ID: <20101020114306.A88C8282BDA@codespeak.net>

Author: arigo
Date: Wed Oct 20 13:43:05 2010
New Revision: 78126

Modified:
   pypy/trunk/pypy/rlib/clibffi.py
Log:
'track_allocation=False' must also be given on lltype.free().


Modified: pypy/trunk/pypy/rlib/clibffi.py
==============================================================================
--- pypy/trunk/pypy/rlib/clibffi.py	(original)
+++ pypy/trunk/pypy/rlib/clibffi.py	Wed Oct 20 13:43:05 2010
@@ -414,10 +414,10 @@
 
     def __del__(self):
         if self.ll_cif:
-            lltype.free(self.ll_cif, flavor='raw')
+            lltype.free(self.ll_cif, flavor='raw', track_allocation=False)
             self.ll_cif = lltype.nullptr(FFI_CIFP.TO)
         if self.ll_argtypes:
-            lltype.free(self.ll_argtypes, flavor='raw')
+            lltype.free(self.ll_argtypes, flavor='raw', track_allocation=False)
             self.ll_argtypes = lltype.nullptr(FFI_TYPE_PP.TO)
 
 # as long as CallbackFuncPtr is kept alive, the underlaying userdata


From arigo at codespeak.net  Wed Oct 20 13:43:32 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 13:43:32 +0200 (CEST)
Subject: [pypy-svn] r78127 - pypy/branch/jit-sign-descr
Message-ID: <20101020114332.9E9EF282BDA@codespeak.net>

Author: arigo
Date: Wed Oct 20 13:43:31 2010
New Revision: 78127

Removed:
   pypy/branch/jit-sign-descr/
Log:
Remove merged branch.



From afa at codespeak.net  Wed Oct 20 14:25:40 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 14:25:40 +0200 (CEST)
Subject: [pypy-svn] r78129 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101020122540.CFBC2282BDA@codespeak.net>

Author: afa
Date: Wed Oct 20 14:25:39 2010
New Revision: 78129

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
Log:
FileIO.truncate()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Wed Oct 20 14:25:39 2010
@@ -7,7 +7,7 @@
 from pypy.rlib.rarithmetic import r_longlong
 from pypy.rlib.rstring import StringBuilder
 from os import O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_TRUNC
-import os, stat, errno
+import sys, os, stat, errno
 
 O_BINARY = getattr(os, "O_BINARY", 0)
 O_APPEND = getattr(os, "O_APPEND", 0)
@@ -347,6 +347,25 @@
             total += len(chunk)
         return space.wrap(builder.build())
 
+    if sys.platform == "win32":
+        def _truncate(self, size):
+            from pypy.rlib.streamio import ftruncate_win32
+            ftruncate_win32(self.fd, size)
+    else:
+        def _truncate(self, size):
+            os.ftruncate(self.fd, size)
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def truncate_w(self, space, w_size=None):
+        if space.is_w(w_size, space.w_None):
+            w_size = self.tell_w(space)
+
+        try:
+            self._truncate(space.r_longlong_w(w_size))
+        except OSError, e:
+            raise wrap_oserror(space, e)
+
+        return w_size
 
 W_FileIO.typedef = TypeDef(
     'FileIO', W_RawIOBase.typedef,
@@ -358,6 +377,7 @@
     write = interp2app(W_FileIO.write_w),
     read = interp2app(W_FileIO.read_w),
     readall = interp2app(W_FileIO.readall_w),
+    truncate = interp2app(W_FileIO.truncate_w),
     close = interp2app(W_FileIO.close_w),
 
     readable = interp2app(W_FileIO.readable_w),

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Wed Oct 20 14:25:39 2010
@@ -74,3 +74,20 @@
         f.seek(3)
         assert f.tell() == 3
         f.close()
+
+    def test_truncate(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'wb')
+        assert f.truncate(100) == 100 # grow the file
+        f.close()
+        f = _io.FileIO(self.tmpfile)
+        assert len(f.read()) == 100
+        f.close()
+        #
+        f = _io.FileIO(self.tmpfile, 'wb')
+        f.seek(50)
+        assert f.truncate() == 50
+        f.close()
+        f = _io.FileIO(self.tmpfile)
+        assert len(f.read()) == 50
+        f.close()


From afa at codespeak.net  Wed Oct 20 14:45:07 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 14:45:07 +0200 (CEST)
Subject: [pypy-svn] r78130 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101020124507.83A90282BDA@codespeak.net>

Author: afa
Date: Wed Oct 20 14:45:05 2010
New Revision: 78130

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
Log:
FileIO.readlines()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	Wed Oct 20 14:45:05 2010
@@ -163,6 +163,29 @@
 
         return space.wrap(builder.build())
 
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def readlines_w(self, space, w_hint=None):
+        hint = convert_size(space, w_hint)
+
+        if hint <= 0:
+            return space.newlist(space.unpackiterable(self))
+
+        lines_w = []
+        length = 0
+        while True:
+            w_line = space.call_method(self, "readline")
+            line_length = space.int_w(space.len(w_line))
+            if line_length == 0: # done
+                break
+
+            lines_w.append(w_line)
+
+            length += line_length
+            if length > hint:
+                break
+
+        return space.newlist(lines_w)
+
 W_IOBase.typedef = TypeDef(
     '_IOBase',
     __new__ = generic_new_descr(W_IOBase),
@@ -180,6 +203,7 @@
     closed = GetSetProperty(W_IOBase.closed_get_w),
 
     readline = interp2app(W_IOBase.readline_w),
+    readlines = interp2app(W_IOBase.readlines_w),
     )
 
 class W_RawIOBase(W_IOBase):

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Wed Oct 20 14:45:05 2010
@@ -44,6 +44,14 @@
         assert f.readline() == ''
         f.close()
 
+    def test_readlines(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'rb')
+        assert f.readlines() == ["a\n", "b\n", "c"]
+        f.seek(0)
+        assert f.readlines(3) == ["a\n", "b\n"]
+        f.close()
+
     def test_readall(self):
         import _io
         f = _io.FileIO(self.tmpfile, 'rb')


From afa at codespeak.net  Wed Oct 20 15:14:17 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 15:14:17 +0200 (CEST)
Subject: [pypy-svn] r78131 - in pypy/branch/fast-forward/pypy/objspace/std:
	. test
Message-ID: <20101020131417.CDBC8282BDA@codespeak.net>

Author: afa
Date: Wed Oct 20 15:14:16 2010
New Revision: 78131

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
Log:
buffer interface for bytearray objects


Modified: pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	Wed Oct 20 15:14:16 2010
@@ -11,6 +11,7 @@
 from pypy.objspace.std.sliceobject import W_SliceObject, normalize_simple_slice
 from pypy.objspace.std import slicetype
 from pypy.interpreter import gateway
+from pypy.interpreter.buffer import RWBuffer
 
 class W_BytearrayObject(W_Object):
     from pypy.objspace.std.bytearraytype import bytearray_typedef as typedef
@@ -400,5 +401,25 @@
     assert stop >= 0
     w_bytearray.data[start:stop] = [c for c in space.str_w(w_other)]
 
+# __________________________________________________________
+# Buffer interface
+
+class BytearrayBuffer(RWBuffer):
+    def __init__(self, data):
+        self.data = data
+
+    def getlength(self):
+        return len(self.data)
+
+    def getitem(self, index):
+        return self.data[index]
+
+    def setitem(self, index, char):
+        self.data[index] = char
+
+def buffer__Bytearray(space, self):
+    b = BytearrayBuffer(self.data)
+    return space.wrap(b)
+
 from pypy.objspace.std import bytearraytype
 register_all(vars(), bytearraytype)

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	Wed Oct 20 15:14:16 2010
@@ -175,3 +175,12 @@
         assert b == 'ABC...defghi'
         b[3:6] = '()'
         assert b == 'ABC()defghi'
+
+    def test_buffer(self):
+        b = bytearray('abcdefghi')
+        buf = buffer(b)
+        assert buf[2] == 'c'
+        buf[3] = 'D'
+        assert b == 'abcDefghi'
+        buf[4:6] = 'EF'
+        assert b == 'abcDEFghi'


From afa at codespeak.net  Wed Oct 20 15:18:59 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 15:18:59 +0200 (CEST)
Subject: [pypy-svn] r78132 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101020131859.635C136E3FB@codespeak.net>

Author: afa
Date: Wed Oct 20 15:18:58 2010
New Revision: 78132

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
Log:
FileIO.readinto()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Wed Oct 20 15:18:58 2010
@@ -321,6 +321,21 @@
 
         return space.wrap(s)
 
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def readinto_w(self, space, w_buffer):
+        self._check_closed(space)
+
+        # XXX check readable
+        rwbuffer = space.rwbuffer_w(w_buffer)
+        length = rwbuffer.getlength()
+        try:
+            buf = os.read(self.fd, length)
+        except OSError, e:
+            raise wrap_oserror(space, e,
+                               exception_name='w_IOError')
+        rwbuffer.setslice(0, buf)
+        return space.wrap(len(buf))
+
     @unwrap_spec('self', ObjSpace)
     def readall_w(self, space):
         self._check_closed(space)
@@ -376,6 +391,7 @@
     tell = interp2app(W_FileIO.tell_w),
     write = interp2app(W_FileIO.write_w),
     read = interp2app(W_FileIO.read_w),
+    readinto = interp2app(W_FileIO.readinto_w),
     readall = interp2app(W_FileIO.readall_w),
     truncate = interp2app(W_FileIO.truncate_w),
     close = interp2app(W_FileIO.close_w),

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Wed Oct 20 15:18:58 2010
@@ -85,17 +85,32 @@
 
     def test_truncate(self):
         import _io
-        f = _io.FileIO(self.tmpfile, 'wb')
+        f = _io.FileIO(self.tmpfile, 'r+b')
         assert f.truncate(100) == 100 # grow the file
         f.close()
         f = _io.FileIO(self.tmpfile)
         assert len(f.read()) == 100
         f.close()
         #
-        f = _io.FileIO(self.tmpfile, 'wb')
+        f = _io.FileIO(self.tmpfile, 'r+b')
         f.seek(50)
         assert f.truncate() == 50
         f.close()
         f = _io.FileIO(self.tmpfile)
         assert len(f.read()) == 50
         f.close()
+
+    def test_readinto(self):
+        import _io
+        a = bytearray('x' * 10)
+        f = _io.FileIO(self.tmpfile, 'r+')
+        assert f.readinto(a) == 10
+        f.close()
+        assert a == 'a\nb\nc\0\0\0\0\0'
+        #
+        a = bytearray('x' * 10)
+        f = _io.FileIO(self.tmpfile, 'r+')
+        f.truncate(3)
+        assert f.readinto(a) == 3
+        f.close()
+        assert a == 'a\nbxxxxxxx'


From arigo at codespeak.net  Wed Oct 20 15:54:18 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 15:54:18 +0200 (CEST)
Subject: [pypy-svn] r78133 - in pypy/trunk/pypy/translator: . c c/src c/test
Message-ID: <20101020135418.6C230282BDA@codespeak.net>

Author: arigo
Date: Wed Oct 20 15:54:16 2010
New Revision: 78133

Modified:
   pypy/trunk/pypy/translator/c/funcgen.py
   pypy/trunk/pypy/translator/c/src/stack.h
   pypy/trunk/pypy/translator/c/test/test_genc.py
   pypy/trunk/pypy/translator/c/test/test_standalone.py
   pypy/trunk/pypy/translator/transform.py
Log:
(antocuni, arigo)

Hopefully fix lib-python/.../test_descr on 64-bit platforms.  The issue
is that we can get a cycle of tail recursions (more easily on x86-64
than on x86-32), and in such cycles, the stack overflow detection does
not work as a way to interrupt the cycle.  Thus we can get caught in
what seems like an infinite recursion (but is not in C).



Modified: pypy/trunk/pypy/translator/c/funcgen.py
==============================================================================
--- pypy/trunk/pypy/translator/c/funcgen.py	(original)
+++ pypy/trunk/pypy/translator/c/funcgen.py	Wed Oct 20 15:54:16 2010
@@ -427,7 +427,7 @@
         r = self.expr(op.result)
         return 'OP_CALL_ARGS((%s), %s);' % (', '.join(args), r)
 
-    def generic_call(self, FUNC, fnexpr, args_v, v_result):
+    def generic_call(self, FUNC, fnexpr, args_v, v_result, targets=None):
         args = []
         assert len(args_v) == len(FUNC.TO.ARGS)
         for v, ARGTYPE in zip(args_v, FUNC.TO.ARGS):
@@ -444,17 +444,23 @@
             # skip assignment of 'void' return value
             r = self.expr(v_result)
             line = '%s = %s' % (r, line)
+        if targets:
+            for func in targets:
+                graph = getattr(func._obj, 'graph', None)
+                if getattr(graph, 'inhibit_tail_call', False):
+                    line += '\nPYPY_INHIBIT_TAIL_CALL();'
+                    break
         return line
 
     def OP_DIRECT_CALL(self, op):
         fn = op.args[0]
         return self.generic_call(fn.concretetype, self.expr(fn),
-                                 op.args[1:], op.result)
+                                 op.args[1:], op.result, [fn.value])
 
     def OP_INDIRECT_CALL(self, op):
         fn = op.args[0]
         return self.generic_call(fn.concretetype, self.expr(fn),
-                                 op.args[1:-1], op.result)
+                                 op.args[1:-1], op.result, op.args[-1].value)
 
     def OP_ADR_CALL(self, op):
         ARGTYPES = [v.concretetype for v in op.args[1:]]

Modified: pypy/trunk/pypy/translator/c/src/stack.h
==============================================================================
--- pypy/trunk/pypy/translator/c/src/stack.h	(original)
+++ pypy/trunk/pypy/translator/c/src/stack.h	Wed Oct 20 15:54:16 2010
@@ -33,6 +33,12 @@
 		&& LL_stack_too_big_slowpath());
 }
 
+#ifdef __GNUC__
+#  define PYPY_INHIBIT_TAIL_CALL()   asm("/* inhibit_tail_call */")
+#else
+#  define PYPY_INHIBIT_TAIL_CALL()   /* add hints for other compilers here */
+#endif
+
 
 #ifndef PYPY_NOT_MAIN_FILE
 #include 

Modified: pypy/trunk/pypy/translator/c/test/test_genc.py
==============================================================================
--- pypy/trunk/pypy/translator/c/test/test_genc.py	(original)
+++ pypy/trunk/pypy/translator/c/test/test_genc.py	Wed Oct 20 15:54:16 2010
@@ -478,3 +478,23 @@
     t.bar = llhelper(FTPTR, a_f.make_func())
     fn = compile(chooser, [bool])
     assert fn(True)
+
+def test_inhibit_tail_call():
+    from pypy.rpython.lltypesystem import lltype
+    def foobar_fn(n):
+        return 42
+    foobar_fn._dont_inline_ = True
+    def main(n):
+        return foobar_fn(n)
+    #
+    t = Translation(main, [int], backend="c")
+    t.rtype()
+    t.context._graphof(foobar_fn).inhibit_tail_call = True
+    t.source_c()
+    lines = t.driver.cbuilder.c_source_filename.readlines()
+    for i, line in enumerate(lines):
+        if '= pypy_g_foobar_fn' in line:
+            break
+    else:
+        assert 0, "the call was not found in the C source"
+    assert 'PYPY_INHIBIT_TAIL_CALL();' in lines[i+1]

Modified: pypy/trunk/pypy/translator/c/test/test_standalone.py
==============================================================================
--- pypy/trunk/pypy/translator/c/test/test_standalone.py	(original)
+++ pypy/trunk/pypy/translator/c/test/test_standalone.py	Wed Oct 20 15:54:16 2010
@@ -16,11 +16,16 @@
 class StandaloneTests(object):
     config = None
 
-    def compile(self, entry_point, debug=True, shared=False):
+    def compile(self, entry_point, debug=True, shared=False,
+                stackcheck=False):
         t = TranslationContext(self.config)
         t.buildannotator().build_types(entry_point, [s_list_of_strings])
         t.buildrtyper().specialize()
 
+        if stackcheck:
+            from pypy.translator.transform import insert_ll_stackcheck
+            insert_ll_stackcheck(t)
+
         t.config.translation.shared = shared
 
         cbuilder = CStandaloneBuilder(t, entry_point, t.config)
@@ -630,6 +635,22 @@
             else:
                 os.environ['CC'] = old_cc
 
+    def test_inhibit_tail_call(self):
+        # the point is to check that the f()->f() recursion stops
+        from pypy.rlib.rstackovf import StackOverflow
+        def f(n):
+            if n <= 0:
+                return 42
+            return f(n+1)
+        def entry_point(argv):
+            try:
+                return f(1)
+            except StackOverflow:
+                print 'hi!'
+                return 0
+        t, cbuilder = self.compile(entry_point, stackcheck=True)
+        out = cbuilder.cmdexec("")
+        assert out.strip() == "hi!"
 
 class TestMaemo(TestStandalone):
     def setup_class(cls):

Modified: pypy/trunk/pypy/translator/transform.py
==============================================================================
--- pypy/trunk/pypy/translator/transform.py	(original)
+++ pypy/trunk/pypy/translator/transform.py	Wed Oct 20 15:54:16 2010
@@ -221,15 +221,19 @@
     stack_check_ptr_const = Constant(stack_check_ptr, lltype.typeOf(stack_check_ptr))
     edges = set()
     insert_in = set()
+    block2graph = {}
     for caller in translator.graphs:
         for block, callee in find_calls_from(translator, caller):
             if getattr(getattr(callee, 'func', None),
                        'insert_stack_check_here', False):
                 insert_in.add(callee.startblock)
+                block2graph[callee.startblock] = callee
                 continue
             if block is not caller.startblock:
                 edges.add((caller.startblock, block))
+                block2graph[caller.startblock] = caller
             edges.add((block, callee.startblock))
+            block2graph[block] = caller
 
     edgelist = [Edge(block1, block2) for (block1, block2) in edges]
     edgedict = make_edge_dict(edgelist)
@@ -241,6 +245,10 @@
         v.concretetype = lltype.Void
         unwind_op = SpaceOperation('direct_call', [stack_check_ptr_const], v)
         block.operations.insert(0, unwind_op)
+        # prevents cycles of tail calls from occurring -- such cycles would
+        # not consume any stack, so would turn into potentially infinite loops
+        graph = block2graph[block]
+        graph.inhibit_tail_call = True
     return len(insert_in)
 
 


From afa at codespeak.net  Wed Oct 20 16:06:24 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 16:06:24 +0200 (CEST)
Subject: [pypy-svn] r78134 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101020140624.DA12036E3FB@codespeak.net>

Author: afa
Date: Wed Oct 20 16:06:23 2010
New Revision: 78134

Added:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py   (contents, props changed)
   pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py   (contents, props changed)
Modified:
   pypy/branch/fast-forward/pypy/module/_io/__init__.py
   pypy/branch/fast-forward/pypy/module/_io/interp_io.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py   (contents, props changed)
Log:
Move classes to their own files


Modified: pypy/branch/fast-forward/pypy/module/_io/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/__init__.py	Wed Oct 20 16:06:23 2010
@@ -11,17 +11,17 @@
         'BlockingIOError': 'interp_io.W_BlockingIOError',
         '_IOBase': 'interp_iobase.W_IOBase',
         '_RawIOBase': 'interp_iobase.W_RawIOBase',
-        '_BufferedIOBase': 'interp_io.W_BufferedIOBase',
-        '_TextIOBase': 'interp_io.W_TextIOBase',
+        '_BufferedIOBase': 'interp_bufferedio.W_BufferedIOBase',
+        '_TextIOBase': 'interp_textio.W_TextIOBase',
 
         'FileIO': 'interp_fileio.W_FileIO',
-        'BytesIO': 'interp_io.W_BytesIO',
+        'BytesIO': 'interp_bytesio.W_BytesIO',
         'StringIO': 'interp_stringio.W_StringIO',
-        'BufferedReader': 'interp_io.W_BufferedReader',
-        'BufferedWriter': 'interp_io.W_BufferedWriter',
-        'BufferedRWPair': 'interp_io.W_BufferedRWPair',
-        'BufferedRandom': 'interp_io.W_BufferedRandom',
-        'TextIOWrapper': 'interp_io.W_TextIOWrapper',
+        'BufferedReader': 'interp_bufferedio.W_BufferedReader',
+        'BufferedWriter': 'interp_bufferedio.W_BufferedWriter',
+        'BufferedRWPair': 'interp_bufferedio.W_BufferedRWPair',
+        'BufferedRandom': 'interp_bufferedio.W_BufferedRandom',
+        'TextIOWrapper': 'interp_textio.W_TextIOWrapper',
 
         'open': 'interp_io.open',
         'IncrementalNewlineDecoder': 'space.w_None',

Added: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- (empty file)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Wed Oct 20 16:06:23 2010
@@ -0,0 +1,39 @@
+from pypy.module._io.interp_iobase import W_IOBase
+from pypy.interpreter.typedef import (
+    TypeDef, generic_new_descr)
+
+class W_BufferedIOBase(W_IOBase):
+    pass
+W_BufferedIOBase.typedef = TypeDef(
+    '_BufferedIOBase', W_IOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedIOBase),
+    )
+
+class W_BufferedReader(W_BufferedIOBase):
+    pass
+W_BufferedReader.typedef = TypeDef(
+    'BufferedReader', W_BufferedIOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedReader),
+    )
+
+class W_BufferedWriter(W_BufferedIOBase):
+    pass
+W_BufferedWriter.typedef = TypeDef(
+    'BufferedWriter', W_BufferedIOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedWriter),
+    )
+
+class W_BufferedRWPair(W_BufferedIOBase):
+    pass
+W_BufferedRWPair.typedef = TypeDef(
+    'BufferedRWPair', W_BufferedIOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedRWPair),
+    )
+
+class W_BufferedRandom(W_BufferedIOBase):
+    pass
+W_BufferedRandom.typedef = TypeDef(
+    'BufferedRandom', W_BufferedIOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedRandom),
+    )
+

Added: pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py
==============================================================================
--- (empty file)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py	Wed Oct 20 16:06:23 2010
@@ -0,0 +1,11 @@
+from pypy.module._io.interp_bufferedio import W_BufferedIOBase
+from pypy.interpreter.typedef import (
+    TypeDef, generic_new_descr)
+
+class W_BytesIO(W_BufferedIOBase):
+    pass
+W_BytesIO.typedef = TypeDef(
+    'BytesIO', W_BufferedIOBase.typedef,
+    __new__ = generic_new_descr(W_BytesIO),
+    )
+

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py	Wed Oct 20 16:06:23 2010
@@ -26,14 +26,6 @@
     characters_written = interp_attrproperty('written', W_BlockingIOError),
     )
 
-class W_BufferedIOBase(W_IOBase):
-    pass
-
-W_BufferedIOBase.typedef = TypeDef(
-    '_BufferedIOBase', W_IOBase.typedef,
-    __new__ = generic_new_descr(W_BufferedIOBase),
-    )
-
 class W_TextIOBase(W_IOBase):
     pass
 W_TextIOBase.typedef = TypeDef(
@@ -41,41 +33,6 @@
     __new__ = generic_new_descr(W_TextIOBase),
     )
 
-class W_BytesIO(W_BufferedIOBase):
-    pass
-W_BytesIO.typedef = TypeDef(
-    'BytesIO', W_BufferedIOBase.typedef,
-    __new__ = generic_new_descr(W_BytesIO),
-    )
-
-class W_BufferedReader(W_BufferedIOBase):
-    pass
-W_BufferedReader.typedef = TypeDef(
-    'BufferedReader', W_BufferedIOBase.typedef,
-    __new__ = generic_new_descr(W_BufferedReader),
-    )
-
-class W_BufferedWriter(W_BufferedIOBase):
-    pass
-W_BufferedWriter.typedef = TypeDef(
-    'BufferedWriter', W_BufferedIOBase.typedef,
-    __new__ = generic_new_descr(W_BufferedWriter),
-    )
-
-class W_BufferedRWPair(W_BufferedIOBase):
-    pass
-W_BufferedRWPair.typedef = TypeDef(
-    'BufferedRWPair', W_BufferedIOBase.typedef,
-    __new__ = generic_new_descr(W_BufferedRWPair),
-    )
-
-class W_BufferedRandom(W_BufferedIOBase):
-    pass
-W_BufferedRandom.typedef = TypeDef(
-    'BufferedRandom', W_BufferedIOBase.typedef,
-    __new__ = generic_new_descr(W_BufferedRandom),
-    )
-
 class W_TextIOWrapper(W_TextIOBase):
     pass
 W_TextIOWrapper.typedef = TypeDef(

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Wed Oct 20 16:06:23 2010
@@ -1,116 +1,116 @@
-from pypy.conftest import gettestobjspace
-from pypy.tool.udir import udir
-import os
-
-class AppTestFileIO:
-    def setup_class(cls):
-        cls.space = gettestobjspace(usemodules=['_io'])
-        tmpfile = udir.join('tmpfile')
-        tmpfile.write("a\nb\nc", mode='wb')
-        cls.w_tmpfile = cls.space.wrap(str(tmpfile))
-        cls.w_tmpdir = cls.space.wrap(str(udir))
-        cls.w_posix = cls.space.appexec([], """():
-            import %s as m;
-            return m""" % os.name)
-
-    def test_constructor(self):
-        import _io
-        f = _io.FileIO(self.tmpfile, 'a')
-        assert f.name.endswith('tmpfile')
-        assert f.mode == 'wb'
-        assert f.closefd is True
-        f.close()
-
-    def test_open_fd(self):
-        import _io
-        os = self.posix
-        fd = os.open(self.tmpfile, os.O_RDONLY, 0666)
-        f = _io.FileIO(fd, "rb", closefd=False)
-        assert f.fileno() == fd
-        assert f.closefd is False
-        f.close()
-        os.close(fd)
-
-    def test_open_directory(self):
-        import _io
-        raises(IOError, _io.FileIO, self.tmpdir, "rb")
-
-    def test_readline(self):
-        import _io
-        f = _io.FileIO(self.tmpfile, 'rb')
-        assert f.readline() == 'a\n'
-        assert f.readline() == 'b\n'
-        assert f.readline() == 'c'
-        assert f.readline() == ''
-        f.close()
-
-    def test_readlines(self):
-        import _io
-        f = _io.FileIO(self.tmpfile, 'rb')
-        assert f.readlines() == ["a\n", "b\n", "c"]
-        f.seek(0)
-        assert f.readlines(3) == ["a\n", "b\n"]
-        f.close()
-
-    def test_readall(self):
-        import _io
-        f = _io.FileIO(self.tmpfile, 'rb')
-        assert f.readall() == "a\nb\nc"
-        f.close()
-
-    def test_write(self):
-        import _io
-        filename = self.tmpfile + '_w'
-        f = _io.FileIO(filename, 'wb')
-        f.write("test")
-        # try without flushing
-        f2 = _io.FileIO(filename, 'rb')
-        assert f2.read() == "test"
-        f.close()
-        f2.close()
-
-    def test_seek(self):
-        import _io
-        f = _io.FileIO(self.tmpfile, 'rb')
-        f.seek(0)
-        self.posix.close(f.fileno())
-        raises(IOError, f.seek, 0)
-
-    def test_tell(self):
-        import _io
-        f = _io.FileIO(self.tmpfile, 'rb')
-        f.seek(3)
-        assert f.tell() == 3
-        f.close()
-
-    def test_truncate(self):
-        import _io
-        f = _io.FileIO(self.tmpfile, 'r+b')
-        assert f.truncate(100) == 100 # grow the file
-        f.close()
-        f = _io.FileIO(self.tmpfile)
-        assert len(f.read()) == 100
-        f.close()
-        #
-        f = _io.FileIO(self.tmpfile, 'r+b')
-        f.seek(50)
-        assert f.truncate() == 50
-        f.close()
-        f = _io.FileIO(self.tmpfile)
-        assert len(f.read()) == 50
-        f.close()
-
-    def test_readinto(self):
-        import _io
-        a = bytearray('x' * 10)
-        f = _io.FileIO(self.tmpfile, 'r+')
-        assert f.readinto(a) == 10
-        f.close()
-        assert a == 'a\nb\nc\0\0\0\0\0'
-        #
-        a = bytearray('x' * 10)
-        f = _io.FileIO(self.tmpfile, 'r+')
-        f.truncate(3)
-        assert f.readinto(a) == 3
-        f.close()
-        assert a == 'a\nbxxxxxxx'
+from pypy.conftest import gettestobjspace
+from pypy.tool.udir import udir
+import os
+
+class AppTestFileIO:
+    def setup_class(cls):
+        cls.space = gettestobjspace(usemodules=['_io'])
+        tmpfile = udir.join('tmpfile')
+        tmpfile.write("a\nb\nc", mode='wb')
+        cls.w_tmpfile = cls.space.wrap(str(tmpfile))
+        cls.w_tmpdir = cls.space.wrap(str(udir))
+        cls.w_posix = cls.space.appexec([], """():
+            import %s as m;
+            return m""" % os.name)
+
+    def test_constructor(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'a')
+        assert f.name.endswith('tmpfile')
+        assert f.mode == 'wb'
+        assert f.closefd is True
+        f.close()
+
+    def test_open_fd(self):
+        import _io
+        os = self.posix
+        fd = os.open(self.tmpfile, os.O_RDONLY, 0666)
+        f = _io.FileIO(fd, "rb", closefd=False)
+        assert f.fileno() == fd
+        assert f.closefd is False
+        f.close()
+        os.close(fd)
+
+    def test_open_directory(self):
+        import _io
+        raises(IOError, _io.FileIO, self.tmpdir, "rb")
+
+    def test_readline(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'rb')
+        assert f.readline() == 'a\n'
+        assert f.readline() == 'b\n'
+        assert f.readline() == 'c'
+        assert f.readline() == ''
+        f.close()
+
+    def test_readlines(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'rb')
+        assert f.readlines() == ["a\n", "b\n", "c"]
+        f.seek(0)
+        assert f.readlines(3) == ["a\n", "b\n"]
+        f.close()
+
+    def test_readall(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'rb')
+        assert f.readall() == "a\nb\nc"
+        f.close()
+
+    def test_write(self):
+        import _io
+        filename = self.tmpfile + '_w'
+        f = _io.FileIO(filename, 'wb')
+        f.write("test")
+        # try without flushing
+        f2 = _io.FileIO(filename, 'rb')
+        assert f2.read() == "test"
+        f.close()
+        f2.close()
+
+    def test_seek(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'rb')
+        f.seek(0)
+        self.posix.close(f.fileno())
+        raises(IOError, f.seek, 0)
+
+    def test_tell(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'rb')
+        f.seek(3)
+        assert f.tell() == 3
+        f.close()
+
+    def test_truncate(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'r+b')
+        assert f.truncate(100) == 100 # grow the file
+        f.close()
+        f = _io.FileIO(self.tmpfile)
+        assert len(f.read()) == 100
+        f.close()
+        #
+        f = _io.FileIO(self.tmpfile, 'r+b')
+        f.seek(50)
+        assert f.truncate() == 50
+        f.close()
+        f = _io.FileIO(self.tmpfile)
+        assert len(f.read()) == 50
+        f.close()
+
+    def test_readinto(self):
+        import _io
+        a = bytearray('x' * 10)
+        f = _io.FileIO(self.tmpfile, 'r+')
+        assert f.readinto(a) == 10
+        f.close()
+        assert a == 'a\nb\nc\0\0\0\0\0'
+        #
+        a = bytearray('x' * 10)
+        f = _io.FileIO(self.tmpfile, 'r+')
+        f.truncate(3)
+        assert f.readinto(a) == 3
+        f.close()
+        assert a == 'a\nbxxxxxxx'


From hakanardo at codespeak.net  Wed Oct 20 16:29:47 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Wed, 20 Oct 2010 16:29:47 +0200 (CEST)
Subject: [pypy-svn] r78135 -
	pypy/branch/jit-unroll-loops/pypy/module/array/benchmark
Message-ID: <20101020142947.B5E01282BDA@codespeak.net>

Author: hakanardo
Date: Wed Oct 20 16:29:46 2010
New Revision: 78135

Added:
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/result.txt
Modified:
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/Makefile
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py
Log:
some benchmark results

Modified: pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/Makefile
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/Makefile	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/Makefile	Wed Oct 20 16:29:46 2010
@@ -13,10 +13,12 @@
 
 run-%: %
 	@echo $^
-	@echo -n '        C: '
-	@/usr/bin/time -f '%e s' ./$^
-	@echo -n '     pypy: '
+	@echo -n '         C: '
+	@/usr/bin/time -f '%e s' ./$^ 2>&1 | tail -1
+	@echo -n '      pypy: '
 	@/usr/bin/time -f '%e s' ../../../translator/goal/pypy-c $^tst.py 2>&1 | tail -1
-	@echo -n '  cpython: '
-	@/usr/bin/time -f '%e s' python $^tst.py
+	@echo -n 'pypy-trunk: '
+	@/usr/bin/time -f '%e s' ../../../../../trunk/pypy/translator/goal/pypy-c $^tst.py 2>&1 | tail -1
+	@echo -n '   cpython: '
+	@/usr/bin/time -f '%e s' python $^tst.py 2>&1 | tail -1
 	@echo

Modified: pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/circulartst.py	Wed Oct 20 16:29:46 2010
@@ -4,14 +4,14 @@
         self = array.__new__(cls, 'd', range(65536))
         return self
     def __getitem__(self, i):
-        assert self.__len__() == 65536 
+        assert len(self) == 65536 
         return array.__getitem__(self, i & 65535)
 
 def main():
     buf = Circular()
     i = 10
     sa = 0
-    while i < 20000000:
+    while i < 200000000:
         sa += buf[i-2] + buf[i-1] + buf[i] + buf[i+1] + buf[i+2]
         i += 1
     return sa

Added: pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/result.txt
==============================================================================
--- (empty file)
+++ pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/result.txt	Wed Oct 20 16:29:46 2010
@@ -0,0 +1,18 @@
+intimg
+         C: 0.29 s
+      pypy: 0.99 s
+pypy-trunk: 1.33 s
+   cpython: 68.20 s
+
+sum
+         C: 0.61 s
+      pypy: 0.59 s
+pypy-trunk: 0.74 s
+   cpython: 24.77 s
+
+circular
+         C: 0.47 s
+      pypy: 1.48 s
+pypy-trunk: 21.13 s
+   cpython: 869.56 s
+


From arigo at codespeak.net  Wed Oct 20 16:46:49 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 16:46:49 +0200 (CEST)
Subject: [pypy-svn] r78136 - pypy/trunk/pypy/translator/c
Message-ID: <20101020144649.80E1F36E3FB@codespeak.net>

Author: arigo
Date: Wed Oct 20 16:46:48 2010
New Revision: 78136

Modified:
   pypy/trunk/pypy/translator/c/funcgen.py
Log:
Fix.


Modified: pypy/trunk/pypy/translator/c/funcgen.py
==============================================================================
--- pypy/trunk/pypy/translator/c/funcgen.py	(original)
+++ pypy/trunk/pypy/translator/c/funcgen.py	Wed Oct 20 16:46:48 2010
@@ -445,8 +445,7 @@
             r = self.expr(v_result)
             line = '%s = %s' % (r, line)
         if targets:
-            for func in targets:
-                graph = getattr(func._obj, 'graph', None)
+            for graph in targets:
                 if getattr(graph, 'inhibit_tail_call', False):
                     line += '\nPYPY_INHIBIT_TAIL_CALL();'
                     break
@@ -454,8 +453,12 @@
 
     def OP_DIRECT_CALL(self, op):
         fn = op.args[0]
+        try:
+            targets = [fn.value._obj.graph]
+        except AttributeError:
+            targets = None
         return self.generic_call(fn.concretetype, self.expr(fn),
-                                 op.args[1:], op.result, [fn.value])
+                                 op.args[1:], op.result, targets)
 
     def OP_INDIRECT_CALL(self, op):
         fn = op.args[0]


From afa at codespeak.net  Wed Oct 20 16:53:02 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 16:53:02 +0200 (CEST)
Subject: [pypy-svn] r78137 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101020145302.3BAA1282BDA@codespeak.net>

Author: afa
Date: Wed Oct 20 16:53:00 2010
New Revision: 78137

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
Log:
FileIO.__repr__, and make FileIO.name a readwrite attribute


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Wed Oct 20 16:53:00 2010
@@ -1,6 +1,7 @@
 from pypy.module._io.interp_iobase import W_RawIOBase
 from pypy.interpreter.typedef import (
-    TypeDef, interp_attrproperty, interp_attrproperty_w, GetSetProperty)
+    TypeDef, interp_attrproperty, interp_attrproperty_w, GetSetProperty,
+    make_weakref_descr)
 from pypy.interpreter.gateway import interp2app, unwrap_spec, Arguments
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.error import OperationError, wrap_oserror, wrap_oserror2
@@ -9,6 +10,27 @@
 from os import O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_TRUNC
 import sys, os, stat, errno
 
+def interp_member_w(name, cls, doc=None):
+    "NOT_RPYTHON: initialization-time only"
+    def fget(space, obj):
+        w_value = getattr(obj, name)
+        if w_value is None:
+            raise OperationError(space.w_AttributeError,
+                                 space.wrap(name))
+        else:
+            return w_value
+    def fset(space, obj, w_value):
+        setattr(obj, name, w_value)
+    def fdel(space, obj):
+        w_value = getattr(obj, name)
+        if w_value is None:
+            raise OperationError(space.w_AttributeError,
+                                 space.wrap(name))
+        setattr(obj, name, None)
+
+    return GetSetProperty(fget, fset, fdel, cls=cls, doc=doc)
+
+
 O_BINARY = getattr(os, "O_BINARY", 0)
 O_APPEND = getattr(os, "O_APPEND", 0)
 
@@ -288,6 +310,21 @@
             raise wrap_oserror(space, e)
         return space.wrap(res)
 
+    @unwrap_spec('self', ObjSpace)
+    def repr_w(self, space):
+        if self.fd < 0:
+            return space.wrap("<_io.FileIO [closed]>")
+
+        if self.w_name is None:
+            return space.wrap(
+                "<_io.FileIO fd=%d mode='%s'>" % (
+                    self.fd, self._mode()))
+        else:
+            w_repr = space.repr(self.w_name)
+            return space.wrap(
+                "<_io.FileIO name=%s mode='%s'>" % (
+                    space.str_w(w_repr), self._mode()))
+
     # ______________________________________________
 
     @unwrap_spec('self', ObjSpace, W_Root)
@@ -386,6 +423,8 @@
     'FileIO', W_RawIOBase.typedef,
     __new__  = interp2app(W_FileIO.descr_new.im_func),
     __init__  = interp2app(W_FileIO.descr_init),
+    __repr__ = interp2app(W_FileIO.repr_w),
+    __weakref__ = make_weakref_descr(W_FileIO),
 
     seek = interp2app(W_FileIO.seek_w),
     tell = interp2app(W_FileIO.tell_w),
@@ -401,7 +440,7 @@
     seekable = interp2app(W_FileIO.seekable_w),
     fileno = interp2app(W_FileIO.fileno_w),
     isatty = interp2app(W_FileIO.isatty_w),
-    name = interp_attrproperty_w('w_name', cls=W_FileIO),
+    name = interp_member_w('w_name', cls=W_FileIO),
     closefd = interp_attrproperty('closefd', cls=W_FileIO),
     mode = GetSetProperty(W_FileIO.descr_get_mode),
     )

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_fileio.py	Wed Oct 20 16:53:00 2010
@@ -21,6 +21,14 @@
         assert f.closefd is True
         f.close()
 
+    def test_weakrefable(self):
+        import _io
+        from weakref import proxy
+        f = _io.FileIO(self.tmpfile)
+        p = proxy(f)
+        assert p.mode == 'rb'
+        f.close()
+
     def test_open_fd(self):
         import _io
         os = self.posix
@@ -114,3 +122,15 @@
         assert f.readinto(a) == 3
         f.close()
         assert a == 'a\nbxxxxxxx'
+
+    def test_repr(self):
+        import _io
+        f = _io.FileIO(self.tmpfile, 'r')
+        assert repr(f) == ("<_io.FileIO name=%r mode='%s'>"
+                           % (f.name, f.mode))
+        del f.name
+        assert repr(f) == ("<_io.FileIO fd=%r mode='%s'>"
+                           % (f.fileno(), f.mode))
+        f.close()
+        assert repr(f) == "<_io.FileIO [closed]>"
+


From arigo at codespeak.net  Wed Oct 20 16:53:19 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 16:53:19 +0200 (CEST)
Subject: [pypy-svn] r78138 - pypy/branch/cleanup-dict-impl
Message-ID: <20101020145319.062F4282BDA@codespeak.net>

Author: arigo
Date: Wed Oct 20 16:53:18 2010
New Revision: 78138

Added:
   pypy/branch/cleanup-dict-impl/
      - copied from r78136, pypy/trunk/
Log:
(cfbolz, arigo)
A branch to clean up the various superfluous dict implementations. 



From afa at codespeak.net  Wed Oct 20 17:09:27 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 17:09:27 +0200 (CEST)
Subject: [pypy-svn] r78139 - pypy/branch/fast-forward/pypy/module/_io
Message-ID: <20101020150927.5B0DC282BDA@codespeak.net>

Author: afa
Date: Wed Oct 20 17:09:25 2010
New Revision: 78139

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_io.py
   pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py
Log:
Kill duplicates


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py	Wed Oct 20 17:09:25 2010
@@ -26,20 +26,6 @@
     characters_written = interp_attrproperty('written', W_BlockingIOError),
     )
 
-class W_TextIOBase(W_IOBase):
-    pass
-W_TextIOBase.typedef = TypeDef(
-    '_TextIOBase', W_IOBase.typedef,
-    __new__ = generic_new_descr(W_TextIOBase),
-    )
-
-class W_TextIOWrapper(W_TextIOBase):
-    pass
-W_TextIOWrapper.typedef = TypeDef(
-    'TextIOWrapper', W_TextIOBase.typedef,
-    __new__ = generic_new_descr(W_TextIOWrapper),
-    )
-
 @unwrap_spec(ObjSpace, Arguments)
 def open(space, __args__):
     # XXX cheat!

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py	Wed Oct 20 17:09:25 2010
@@ -1,4 +1,4 @@
-from pypy.module._io.interp_io import W_TextIOBase
+from pypy.module._io.interp_textio import W_TextIOBase
 from pypy.interpreter.typedef import TypeDef
 from pypy.interpreter.gateway import interp2app, unwrap_spec
 from pypy.interpreter.error import operationerrfmt


From arigo at codespeak.net  Wed Oct 20 17:19:50 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 17:19:50 +0200 (CEST)
Subject: [pypy-svn] r78140 - in pypy/branch/cleanup-dict-impl/pypy: config
	config/test doc/config interpreter module/__builtin__
	objspace objspace/std objspace/std/test
Message-ID: <20101020151950.683D4282BE7@codespeak.net>

Author: arigo
Date: Wed Oct 20 17:19:47 2010
New Revision: 78140

Removed:
   pypy/branch/cleanup-dict-impl/pypy/doc/config/objspace.std.withshadowtracking.txt
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_shadowtracking.py
Modified:
   pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py
   pypy/branch/cleanup-dict-impl/pypy/config/test/test_pypyoption.py
   pypy/branch/cleanup-dict-impl/pypy/interpreter/baseobjspace.py
   pypy/branch/cleanup-dict-impl/pypy/interpreter/typedef.py
   pypy/branch/cleanup-dict-impl/pypy/module/__builtin__/interp_classobj.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/descroperation.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/callmethod.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/celldict.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/dictmultiobject.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/inlinedict.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/objspace.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/proxyobject.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/sharingdict.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_dictmultiobject.py
Log:
(cfbolz, arigo) Kill shadow tracking.


Modified: pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py	Wed Oct 20 17:19:47 2010
@@ -239,14 +239,12 @@
 
         BoolOption("withinlineddict",
                    "make instances more compact by revoming a level of indirection",
-                   default=False,
-                   requires=[("objspace.std.withshadowtracking", False)]),
+                   default=False),
 
         BoolOption("withmapdict",
                    "make instances really small but slow without the JIT",
                    default=False,
-                   requires=[("objspace.std.withshadowtracking", False),
-                             ("objspace.std.withinlineddict", False),
+                   requires=[("objspace.std.withinlineddict", False),
                              ("objspace.std.withsharingdict", False),
                              ("objspace.std.getattributeshortcut", True),
                              ("objspace.std.withtypeversion", True),
@@ -265,12 +263,6 @@
                    # weakrefs needed, because of get_subclasses()
                    requires=[("translation.rweakref", True)]),
 
-        BoolOption("withshadowtracking",
-                   "track whether an instance attribute shadows a type"
-                   " attribute",
-                   default=False,
-                   requires=[("objspace.std.withtypeversion", True),
-                             ("translation.rweakref", True)]),
         BoolOption("withmethodcache",
                    "try to cache method lookups",
                    default=False,

Modified: pypy/branch/cleanup-dict-impl/pypy/config/test/test_pypyoption.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/config/test/test_pypyoption.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/config/test/test_pypyoption.py	Wed Oct 20 17:19:47 2010
@@ -59,7 +59,6 @@
 
     assert not conf.objspace.std.withtypeversion
     assert not conf.objspace.std.withmethodcache
-    assert not conf.objspace.std.withshadowtracking
 
 def test_check_documentation():
     def check_file_exists(fn):

Modified: pypy/branch/cleanup-dict-impl/pypy/interpreter/baseobjspace.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/interpreter/baseobjspace.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/interpreter/baseobjspace.py	Wed Oct 20 17:19:47 2010
@@ -36,13 +36,10 @@
             return space.finditem_str(w_dict, attr)
         return None
 
-    def getdictvalue_attr_is_in_class(self, space, attr):
-        return self.getdictvalue(space, attr)
-
-    def setdictvalue(self, space, attr, w_value, shadows_type=True):
+    def setdictvalue(self, space, attr, w_value):
         w_dict = self.getdict()
         if w_dict is not None:
-            space.setitem_str(w_dict, attr, w_value, shadows_type)
+            space.setitem_str(w_dict, attr, w_value)
             return True
         return False
 
@@ -657,7 +654,7 @@
         """shortcut for space.int_w(space.hash(w_obj))"""
         return self.int_w(self.hash(w_obj))
 
-    def setitem_str(self, w_obj, key, w_value, shadows_type=True):
+    def setitem_str(self, w_obj, key, w_value):
         return self.setitem(w_obj, self.wrap(key), w_value)
 
     def finditem_str(self, w_obj, key):

Modified: pypy/branch/cleanup-dict-impl/pypy/interpreter/typedef.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/interpreter/typedef.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/interpreter/typedef.py	Wed Oct 20 17:19:47 2010
@@ -290,15 +290,6 @@
             def setclass(self, space, w_subtype):
                 # only used by descr_set___class__
                 self.w__class__ = w_subtype
-                if space.config.objspace.std.withshadowtracking:
-                    self.w__dict__.set_shadows_anything()
-
-            def getdictvalue_attr_is_in_class(self, space, name):
-                w_dict = self.w__dict__
-                if space.config.objspace.std.withshadowtracking:
-                    if not w_dict.shadows_anything():
-                        return None
-                return space.finditem_str(w_dict, name)
 
         add(Proto)
 

Modified: pypy/branch/cleanup-dict-impl/pypy/module/__builtin__/interp_classobj.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/module/__builtin__/interp_classobj.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/module/__builtin__/interp_classobj.py	Wed Oct 20 17:19:47 2010
@@ -410,8 +410,7 @@
         if w_meth is not None:
             space.call_function(w_meth, w_name, w_value)
         else:
-            # bit obscure: appease normalization
-            self.setdictvalue(space, name, w_value, True)
+            self.setdictvalue(space, name, w_value)
 
     def descr_delattr(self, space, w_name):
         name = unwrap_attr(space, w_name)

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/descroperation.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/descroperation.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/descroperation.py	Wed Oct 20 17:19:47 2010
@@ -64,9 +64,7 @@
                     w_type = space.type(w_obj)
                     return space.get_and_call_function(w_get, w_descr, w_obj,
                                                        w_type)
-            w_value = w_obj.getdictvalue_attr_is_in_class(space, name)
-        else:
-            w_value = w_obj.getdictvalue(space, name)
+        w_value = w_obj.getdictvalue(space, name)
         if w_value is not None:
             return w_value
         if w_descr is not None:
@@ -76,13 +74,11 @@
     def descr__setattr__(space, w_obj, w_name, w_value):
         name = space.str_w(w_name)
         w_descr = space.lookup(w_obj, name)
-        shadows_type = False
         if w_descr is not None:
             if space.is_data_descr(w_descr):
                 space.set(w_descr, w_obj, w_value)
                 return
-            shadows_type = True
-        if w_obj.setdictvalue(space, name, w_value, shadows_type):
+        if w_obj.setdictvalue(space, name, w_value):
             return
         raiseattrerror(space, w_obj, name, w_descr)
 

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/callmethod.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/callmethod.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/callmethod.py	Wed Oct 20 17:19:47 2010
@@ -44,7 +44,7 @@
         else:
             typ = type(w_descr)
             if typ is function.Function or typ is function.FunctionWithFixedCode:
-                w_value = w_obj.getdictvalue_attr_is_in_class(space, name)
+                w_value = w_obj.getdictvalue(space, name)
                 if w_value is None:
                     # fast method path: a function object in the class,
                     # nothing in the instance
@@ -103,7 +103,7 @@
         w_descr = space.lookup(w_obj, methname)
         typ = type(w_descr)
         if typ is function.Function or typ is function.FunctionWithFixedCode:
-            w_value = w_obj.getdictvalue_attr_is_in_class(space, methname)
+            w_value = w_obj.getdictvalue(space, methname)
             if w_value is None:
                 # fast method path: a function object in the class,
                 # nothing in the instance

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/celldict.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/celldict.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/celldict.py	Wed Oct 20 17:19:47 2010
@@ -47,7 +47,7 @@
         else:
             self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
-    def impl_setitem_str(self, name, w_value, shadows_type=True):
+    def impl_setitem_str(self, name, w_value):
         self.getcell(name, True).w_value = w_value
 
     def impl_delitem(self, w_key):

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/dictmultiobject.py	Wed Oct 20 17:19:47 2010
@@ -53,10 +53,6 @@
             from pypy.objspace.std.sharingdict import SharedDictImplementation
             assert w_type is None
             return SharedDictImplementation(space)
-        elif (space.config.objspace.std.withshadowtracking and instance and
-                classofinstance is not None):
-            assert w_type is None
-            return ShadowDetectingDictImplementation(space, classofinstance)
         elif instance or strdict or module:
             assert w_type is None
             return StrDictImplementation(space)
@@ -112,7 +108,7 @@
         #return w_value or None
         raise NotImplementedError("abstract base class")
 
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         raise NotImplementedError("abstract base class")
 
     def impl_setitem(self,  w_key, w_value):
@@ -165,20 +161,13 @@
         key = OPTIMIZED_BUILTINS[i]
         return self.impl_getitem_str(key)
 
-    # this method will only be seen whan a certain config option is used
-    def impl_shadows_anything(self):
-        return True
-
-    def impl_set_shadows_anything(self):
-        pass
-
     # _________________________________________________________________
     # fallback implementation methods
 
     def impl_fallback_setitem(self, w_key, w_value):
         self.r_dict_content[w_key] = w_value
 
-    def impl_fallback_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_fallback_setitem_str(self, key, w_value):
         return self.impl_fallback_setitem(self.space.wrap(key), w_value)
 
     def impl_fallback_delitem(self, w_key):
@@ -211,18 +200,12 @@
         key = OPTIMIZED_BUILTINS[i]
         return self.impl_fallback_getitem_str(key)
 
-    def impl_fallback_shadows_anything(self):
-        return True
-
-    def impl_fallback_set_shadows_anything(self):
-        pass
-
 
 implementation_methods = [
     ("getitem", 1),
     ("getitem_str", 1),
     ("length", 0),
-    ("setitem_str", 3),
+    ("setitem_str", 2),
     ("setitem", 2),
     ("delitem", 1),
     ("iter", 0),
@@ -231,8 +214,6 @@
     ("keys", 0),
     ("clear", 0),
     ("get_builtin_indexed", 1),
-    ("shadows_anything", 0),
-    ("set_shadows_anything", 0),
 ]
 
 
@@ -312,7 +293,7 @@
         else:
             self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         self.content[key] = w_value
 
     def impl_delitem(self, w_key):
@@ -388,47 +369,12 @@
             return None, None
 
 
-class ShadowDetectingDictImplementation(StrDictImplementation):
-    def __init__(self, space, w_type):
-        StrDictImplementation.__init__(self, space)
-        self.w_type = w_type
-        self.original_version_tag = w_type.version_tag()
-        if self.original_version_tag is None:
-            self._shadows_anything = True
-        else:
-            self._shadows_anything = False
-
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
-        if shadows_type:
-            self._shadows_anything = True
-        StrDictImplementation.impl_setitem_str(
-            self, key, w_value, shadows_type)
-
-    def impl_setitem(self, w_key, w_value):
-        space = self.space
-        if space.is_w(space.type(w_key), space.w_str):
-            if not self._shadows_anything:
-                w_obj = self.w_type.lookup(space.str_w(w_key))
-                if w_obj is not None:
-                    self._shadows_anything = True
-            StrDictImplementation.impl_setitem_str(
-                self, self.space.str_w(w_key), w_value, False)
-        else:
-            self._as_rdict().impl_fallback_setitem(w_key, w_value)
-
-    def impl_shadows_anything(self):
-        return (self._shadows_anything or 
-                self.w_type.version_tag() is not self.original_version_tag)
-
-    def impl_set_shadows_anything(self):
-        self._shadows_anything = True
-
 class WaryDictImplementation(StrDictImplementation):
     def __init__(self, space):
         StrDictImplementation.__init__(self, space)
         self.shadowed = [None] * len(BUILTIN_TO_INDEX)
 
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         i = BUILTIN_TO_INDEX.get(key, -1)
         if i != -1:
             self.shadowed[i] = w_value
@@ -558,7 +504,7 @@
         self.info.writes += 1
         self.content[w_key] = w_value
         self.info.maxcontents = max(self.info.maxcontents, len(self.content))
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         self.info.setitem_strs += 1
         self.impl_setitem(self.space.wrap(key), w_value)
     def impl_delitem(self, w_key):

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/inlinedict.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/inlinedict.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/inlinedict.py	Wed Oct 20 17:19:47 2010
@@ -91,14 +91,8 @@
             w_dict = self.getdict()
             return w_dict.getitem_str(attr)
 
-        def getdictvalue_attr_is_in_class(self, space, attr):
-            return self.getdictvalue(space, attr)
-
-        def setdictvalue(self, space, attr, w_value, shadows_type=True):
+        def setdictvalue(self, space, attr, w_value):
             if self._inlined_dict_valid():
-                # XXX so far we ignore shadows_type, which is a small
-                # performance-degradation if the JIT is not used (i.e. shadow
-                # tracking does not work). Maybe we don't care.
                 self.impl_setitem_str(attr, w_value)
                 return True
             w_dict = self.getdict()

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py	Wed Oct 20 17:19:47 2010
@@ -291,7 +291,7 @@
     def getdictvalue(self, space, attrname):
         return self._get_mapdict_map().read(self, (attrname, DICT))
 
-    def setdictvalue(self, space, attrname, w_value, shadows_type=True):
+    def setdictvalue(self, space, attrname, w_value):
         return self._get_mapdict_map().write(self, (attrname, DICT), w_value)
 
     def deldictvalue(self, space, w_name):
@@ -506,8 +506,8 @@
     def impl_getitem_str(self, key):
         return self.w_obj.getdictvalue(self.space, key)
 
-    def impl_setitem_str(self,  key, w_value, shadows_type=True):
-        flag = self.w_obj.setdictvalue(self.space, key, w_value, shadows_type)
+    def impl_setitem_str(self,  key, w_value):
+        flag = self.w_obj.setdictvalue(self.space, key, w_value)
         assert flag
 
     def impl_setitem(self,  w_key, w_value):
@@ -588,8 +588,6 @@
 # ____________________________________________________________
 # Magic caching
 
-# XXX we also would like getdictvalue_attr_is_in_class() above
-
 class CacheEntry(object):
     map = None
     version_tag = None

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/objspace.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/objspace.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/objspace.py	Wed Oct 20 17:19:47 2010
@@ -437,7 +437,7 @@
             if is_data:
                 w_get = self.lookup(w_descr, "__get__")
             if w_get is None:
-                w_value = w_obj.getdictvalue_attr_is_in_class(self, name)
+                w_value = w_obj.getdictvalue(self, name)
                 if w_value is not None:
                     return w_value
                 if not is_data:
@@ -489,14 +489,12 @@
             return w_obj.getitem(w_key)
         return ObjSpace.finditem(self, w_obj, w_key)
 
-    def setitem_str(self, w_obj, key, w_value, shadows_type=True):
+    def setitem_str(self, w_obj, key, w_value):
         """ Same as setitem, but takes string instead of any wrapped object
-
-        XXX what shadows_type means???
         """
         if (isinstance(w_obj, W_DictMultiObject) and
                 not w_obj.user_overridden_class):
-            w_obj.setitem_str(key, w_value, shadows_type)
+            w_obj.setitem_str(key, w_value)
         else:
             self.setitem(w_obj, self.wrap(key), w_value)
 

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/proxyobject.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/proxyobject.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/proxyobject.py	Wed Oct 20 17:19:47 2010
@@ -43,7 +43,7 @@
                     raise
                 return None
         
-        def setdictvalue(self, space, attr, w_value, shadows_type=True):
+        def setdictvalue(self, space, attr, w_value):
             try:
                 space.call_function(self.w_controller, space.wrap('__setattr__'),
                    space.wrap(attr), w_value)

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/sharingdict.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/sharingdict.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/sharingdict.py	Wed Oct 20 17:19:47 2010
@@ -87,7 +87,7 @@
             self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
     @unroll_safe
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         i = self.structure.lookup_position(key)
         if i != -1:
             self.entries[i] = w_value

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_dictmultiobject.py	Wed Oct 20 17:19:47 2010
@@ -632,7 +632,6 @@
             withsharingdict = False
             withsmalldicts = False
             withcelldict = False
-            withshadowtracking = False
         class opcodes:
             CALL_LIKELY_BUILTIN = False
 


From arigo at codespeak.net  Wed Oct 20 17:29:14 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 17:29:14 +0200 (CEST)
Subject: [pypy-svn] r78141 - in pypy/branch/cleanup-dict-impl/pypy: config
	interpreter jit/tl objspace/std objspace/std/test
Message-ID: <20101020152914.17402282BE7@codespeak.net>

Author: arigo
Date: Wed Oct 20 17:29:13 2010
New Revision: 78141

Removed:
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/inlinedict.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_inlinedict.py
Modified:
   pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py
   pypy/branch/cleanup-dict-impl/pypy/interpreter/typedef.py
   pypy/branch/cleanup-dict-impl/pypy/jit/tl/pypyjit.py
Log:
(cfbolz, arigo) Kill inlinedict.


Modified: pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py	Wed Oct 20 17:29:13 2010
@@ -237,15 +237,10 @@
                    "about dictionaries",
                    default=False),
 
-        BoolOption("withinlineddict",
-                   "make instances more compact by revoming a level of indirection",
-                   default=False),
-
         BoolOption("withmapdict",
                    "make instances really small but slow without the JIT",
                    default=False,
-                   requires=[("objspace.std.withinlineddict", False),
-                             ("objspace.std.withsharingdict", False),
+                   requires=[("objspace.std.withsharingdict", False),
                              ("objspace.std.getattributeshortcut", True),
                              ("objspace.std.withtypeversion", True),
                        ]),
@@ -336,7 +331,6 @@
         config.objspace.std.suggest(newshortcut=True)        
         if type_system != 'ootype':
             config.objspace.std.suggest(withsharingdict=True)
-        config.objspace.std.suggest(withinlineddict=True)
 
     # extra costly optimizations only go in level 3
     if level == '3':

Modified: pypy/branch/cleanup-dict-impl/pypy/interpreter/typedef.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/interpreter/typedef.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/interpreter/typedef.py	Wed Oct 20 17:29:13 2010
@@ -262,16 +262,7 @@
                 return self.slots_w[index]
         add(Proto)
 
-    wantdict = "dict" in features
-    if wantdict and config.objspace.std.withinlineddict:
-        from pypy.objspace.std.objectobject import W_ObjectObject
-        from pypy.objspace.std.inlinedict import make_mixin
-        if supercls is W_ObjectObject:
-            Mixin = make_mixin(config)
-            add(Mixin)
-            wantdict = False
-
-    if wantdict:
+    if "dict" in features:
         base_user_setup = supercls.user_setup.im_func
         if "user_setup" in body:
             base_user_setup = body["user_setup"]

Modified: pypy/branch/cleanup-dict-impl/pypy/jit/tl/pypyjit.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/jit/tl/pypyjit.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/jit/tl/pypyjit.py	Wed Oct 20 17:29:13 2010
@@ -44,7 +44,6 @@
 config.objspace.usemodules._ffi = True
 #
 set_pypy_opt_level(config, level='jit')
-config.objspace.std.withinlineddict = True
 
 if BACKEND == 'c':
     config.objspace.std.multimethods = 'mrd'


From arigo at codespeak.net  Wed Oct 20 17:42:08 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 17:42:08 +0200 (CEST)
Subject: [pypy-svn] r78142 - in pypy/branch/cleanup-dict-impl/pypy: config
	config/test module/__builtin__/test objspace/std objspace/std/test
Message-ID: <20101020154208.42918282BE7@codespeak.net>

Author: arigo
Date: Wed Oct 20 17:42:06 2010
New Revision: 78142

Removed:
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/sharingdict.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_sharingdict.py
Modified:
   pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py
   pypy/branch/cleanup-dict-impl/pypy/config/test/test_pypyoption.py
   pypy/branch/cleanup-dict-impl/pypy/module/__builtin__/test/test_classobj.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/dictmultiobject.py
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_dictmultiobject.py
Log:
(cfbolz, arigo) Kill sharingdict.


Modified: pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/config/pypyoption.py	Wed Oct 20 17:42:06 2010
@@ -228,10 +228,6 @@
                    requires=[("objspace.opcodes.CALL_LIKELY_BUILTIN", False),
                              ("objspace.honor__builtins__", False)]),
 
-        BoolOption("withsharingdict",
-                   "use dictionaries that share the keys part",
-                   default=False),
-
         BoolOption("withdictmeasurement",
                    "create huge files with masses of information "
                    "about dictionaries",
@@ -240,8 +236,7 @@
         BoolOption("withmapdict",
                    "make instances really small but slow without the JIT",
                    default=False,
-                   requires=[("objspace.std.withsharingdict", False),
-                             ("objspace.std.getattributeshortcut", True),
+                   requires=[("objspace.std.getattributeshortcut", True),
                              ("objspace.std.withtypeversion", True),
                        ]),
 
@@ -329,8 +324,6 @@
         config.objspace.std.suggest(optimized_list_getitem=True)
         config.objspace.std.suggest(getattributeshortcut=True)
         config.objspace.std.suggest(newshortcut=True)        
-        if type_system != 'ootype':
-            config.objspace.std.suggest(withsharingdict=True)
 
     # extra costly optimizations only go in level 3
     if level == '3':
@@ -359,7 +352,7 @@
     # extra optimizations with the JIT
     if level == 'jit':
         config.objspace.std.suggest(withcelldict=True)
-        #config.objspace.std.suggest(withmapdict=True)
+        config.objspace.std.suggest(withmapdict=True)
 
 
 def enable_allworkingmodules(config):

Modified: pypy/branch/cleanup-dict-impl/pypy/config/test/test_pypyoption.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/config/test/test_pypyoption.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/config/test/test_pypyoption.py	Wed Oct 20 17:42:06 2010
@@ -47,7 +47,7 @@
 def test_set_pypy_opt_level():
     conf = get_pypy_config()
     set_pypy_opt_level(conf, '2')
-    assert conf.objspace.std.withsharingdict
+    assert conf.objspace.std.newshortcut
     conf = get_pypy_config()
     set_pypy_opt_level(conf, '0')
     assert not conf.objspace.std.newshortcut

Modified: pypy/branch/cleanup-dict-impl/pypy/module/__builtin__/test/test_classobj.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/module/__builtin__/test/test_classobj.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/module/__builtin__/test/test_classobj.py	Wed Oct 20 17:42:06 2010
@@ -954,28 +954,7 @@
         raises(TypeError, descr.__delete__, a)
 
 
-class AppTestOldStyleSharing(AppTestOldstyle):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withsharingdict": True})
-        if option.runappdirect:
-            py.test.skip("can only be run on py.py")
-        def is_sharing(space, w_inst):
-            from pypy.objspace.std.sharingdict import SharedDictImplementation
-            w_d = w_inst.getdict()
-            return space.wrap(isinstance(w_d, SharedDictImplementation) and w_d.r_dict_content is None)
-        cls.w_is_sharing = cls.space.wrap(gateway.interp2app(is_sharing))
-
-
-    def test_real_sharing(self):
-        class A:
-            def __init__(self):
-                self.x = 42
-        A1, A2, A3 = A(), A(), A()
-        assert self.is_sharing(A3)
-        assert self.is_sharing(A2)
-        assert self.is_sharing(A1)
-
-class AppTestOldStyleModDict(object):
+class AppTestOldStyleClassStrDict(object):
     def setup_class(cls):
         if option.runappdirect:
             py.test.skip("can only be run on py.py")

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/dictmultiobject.py	Wed Oct 20 17:42:06 2010
@@ -49,10 +49,6 @@
         elif space.config.objspace.std.withdictmeasurement:
             assert w_type is None
             return MeasuringDictImplementation(space)
-        elif space.config.objspace.std.withsharingdict and instance:
-            from pypy.objspace.std.sharingdict import SharedDictImplementation
-            assert w_type is None
-            return SharedDictImplementation(space)
         elif instance or strdict or module:
             assert w_type is None
             return StrDictImplementation(space)

Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_dictmultiobject.py	Wed Oct 20 17:42:06 2010
@@ -4,7 +4,6 @@
      StrDictImplementation
 
 from pypy.objspace.std.celldict import ModuleDictImplementation
-from pypy.objspace.std.sharingdict import SharedDictImplementation
 from pypy.conftest import gettestobjspace
 
 
@@ -510,32 +509,6 @@
         assert getattr(a, s) == 42
 
 
-class TestW_DictSharing(TestW_DictObject):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withsharingdict": True})
-
-class AppTest_DictSharing(AppTest_DictObject):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withsharingdict": True})
-
-    def test_values_does_not_share(self):
-        class A(object):
-            pass
-        a = A()
-        a.abc = 12
-        l = a.__dict__.values()
-        assert l == [12]
-        l[0] = 24
-        assert a.abc == 12
-
-    def test_items(self):
-        class A(object):
-            pass
-        a = A()
-        a.abc = 12
-        a.__dict__.items() == [("abc", 12)]
-
-
 class AppTestModuleDict(object):
     def setup_class(cls):
         cls.space = gettestobjspace(**{"objspace.std.withcelldict": True})
@@ -629,7 +602,6 @@
     class objspace:
         class std:
             withdictmeasurement = False
-            withsharingdict = False
             withsmalldicts = False
             withcelldict = False
         class opcodes:
@@ -766,9 +738,6 @@
     string = "int"
     string2 = "isinstance"
 
-class TestSharedDictImplementation(BaseTestRDictImplementation):
-    ImplementionClass = SharedDictImplementation
-
 
 class BaseTestDevolvedDictImplementation(BaseTestRDictImplementation):
     def fill_impl(self):
@@ -790,8 +759,6 @@
     string = "int"
     string2 = "isinstance"
 
-class TestDevolvedSharedDictImplementation(BaseTestDevolvedDictImplementation):
-    ImplementionClass = SharedDictImplementation
 
 def test_module_uses_strdict():
     fakespace = FakeSpace()


From arigo at codespeak.net  Wed Oct 20 17:45:14 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 17:45:14 +0200 (CEST)
Subject: [pypy-svn] r78143 -
	pypy/branch/cleanup-dict-impl/pypy/module/pypyjit/test
Message-ID: <20101020154514.69CC9282BDA@codespeak.net>

Author: arigo
Date: Wed Oct 20 17:45:13 2010
New Revision: 78143

Modified:
   pypy/branch/cleanup-dict-impl/pypy/module/pypyjit/test/test_pypy_c.py
Log:
(cfbolz, arigo) Now withmapdict is the default with the JIT.


Modified: pypy/branch/cleanup-dict-impl/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/module/pypyjit/test/test_pypy_c.py	Wed Oct 20 17:45:13 2010
@@ -278,7 +278,7 @@
         assert len(ops) == 2
         assert not ops[0].get_opnames("call")
         assert not ops[0].get_opnames("new")
-        assert len(ops[0].get_opnames("guard")) <= 3     # we get 2 withmapdict
+        assert len(ops[0].get_opnames("guard")) <= 2
         assert not ops[1] # second LOOKUP_METHOD folded away
 
         ops = self.get_by_bytecode("CALL_METHOD")
@@ -323,8 +323,8 @@
         assert len(ops) == 2
         assert not ops[0].get_opnames("call")
         assert not ops[0].get_opnames("new")
-        assert len(ops[0].get_opnames("guard")) <= 3    # we get 2 withmapdict
-        assert len(ops[0].get_opnames("getfield")) <= 5 # we get <5 withmapdict
+        assert len(ops[0].get_opnames("guard")) <= 2
+        assert len(ops[0].get_opnames("getfield")) <= 4
         assert not ops[1] # second LOOKUP_METHOD folded away
 
     def test_default_and_kw(self):


From arigo at codespeak.net  Wed Oct 20 17:59:13 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 20 Oct 2010 17:59:13 +0200 (CEST)
Subject: [pypy-svn] r78144 -
	pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests
Message-ID: <20101020155913.BCB9E282BDA@codespeak.net>

Author: arigo
Date: Wed Oct 20 17:59:12 2010
New Revision: 78144

Modified:
   pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py
Log:
Fix for 64-bits.


Modified: pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py
==============================================================================
--- pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py	(original)
+++ pypy/trunk/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py	Wed Oct 20 17:59:12 2010
@@ -10,7 +10,7 @@
     """
     def test_array_of_pointers(self):
         # tests array item assignements & pointer.contents = ...
-        A = POINTER(c_int) * 24
+        A = POINTER(c_long) * 24
         a = A()
         l = c_long(2)
         p = pointer(l)


From afa at codespeak.net  Wed Oct 20 18:24:24 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 18:24:24 +0200 (CEST)
Subject: [pypy-svn] r78145 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101020162424.76AEB282BDA@codespeak.net>

Author: afa
Date: Wed Oct 20 18:24:22 2010
New Revision: 78145

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
Log:
Add finalizer for IOBase (and all subclasses)


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	Wed Oct 20 18:24:22 2010
@@ -16,6 +16,7 @@
         # XXX: IOBase thinks it has to maintain its own internal state in
         # `__IOBase_closed` and call flush() by itself, but it is redundant
         # with whatever behaviour a non-trivial derived class will implement.
+        self.space = space
         self.__IOBase_closed = False
 
     def _closed(self, space):
@@ -26,6 +27,20 @@
             return True
         return False
 
+    def __del__(self):
+        space = self.space
+        w_closed = space.findattr(self, space.wrap('closed'))
+        try:
+            # If `closed` doesn't exist or can't be evaluated as bool, then
+            # the object is probably in an unusable state, so ignore.
+            if w_closed is not None and not space.is_true(w_closed):
+                space.call_method(self, "close")
+        except OperationError:
+            # Silencing I/O errors is bad, but printing spurious tracebacks is
+            # equally as bad, and potentially more frequent (because of
+            # shutdown issues).
+            pass
+
     def _CLOSED(self):
         # Use this macro whenever you want to check the internal `closed`
         # status of the IOBase object rather than the virtual `closed`

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	Wed Oct 20 18:24:22 2010
@@ -42,6 +42,25 @@
         import _io
         e = _io.UnsupportedOperation("seek")
 
+    def test_destructor(self):
+        import io
+        io.IOBase()
+
+        record = []
+        class MyIO(io.IOBase):
+            def __del__(self):
+                record.append(1)
+                super(MyIO, self).__del__()
+            def close(self):
+                record.append(2)
+                super(MyIO, self).close()
+            def flush(self):
+                record.append(3)
+                super(MyIO, self).flush()
+        MyIO()
+        import gc; gc.collect()
+        assert record == [1, 2, 3]
+
 class AppTestOpen:
     def setup_class(cls):
         tmpfile = udir.join('tmpfile').ensure()


From afa at codespeak.net  Wed Oct 20 20:03:58 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 20:03:58 +0200 (CEST)
Subject: [pypy-svn] r78147 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101020180358.72176282BE8@codespeak.net>

Author: afa
Date: Wed Oct 20 20:03:56 2010
New Revision: 78147

Added:
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
Log:
Start a BufferedIO


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Wed Oct 20 20:03:56 2010
@@ -1,19 +1,80 @@
-from pypy.module._io.interp_iobase import W_IOBase
 from pypy.interpreter.typedef import (
     TypeDef, generic_new_descr)
+from pypy.interpreter.gateway import interp2app, unwrap_spec
+from pypy.interpreter.baseobjspace import ObjSpace, W_Root
+from pypy.interpreter.error import OperationError
+from pypy.rpython.lltypesystem import lltype, rffi
+from pypy.module._io.interp_iobase import W_IOBase
+from pypy.module._io.interp_io import DEFAULT_BUFFER_SIZE
+from pypy.module.thread.os_lock import Lock
 
 class W_BufferedIOBase(W_IOBase):
-    pass
+    def __init__(self, space):
+        W_IOBase.__init__(self, space)
+        self.buffer = None
+        self.lock = None
+
+    def _init(self, space):
+        if self.buffer_size <= 0:
+            raise OperationError(space.w_ValueError, space.wrap(
+                "buffer size must be strictly positive"))
+
+        if self.buffer:
+            lltype.free(self.buffer, flavor='raw')
+        self.buffer = lltype.malloc(rffi.CCHARP.TO, self.buffer_size,
+                                    flavor='raw')
+
+        ## XXX cannot free a Lock?
+        ## if self.lock:
+        ##     self.lock.free()
+        self.lock = Lock(space)
+
+        try:
+            self._raw_tell(space)
+        except OperationError:
+            pass
+
+    def _raw_tell(self, space):
+        w_pos = space.call_method(self.raw, "tell")
+        pos = space.r_longlong_w(w_pos)
+        if pos < 0:
+            raise OperationError(space.w_IOError, space.wrap(
+                "raw stream returned invalid position"))
+
+        self.abs_pos = pos
+        return pos
+
 W_BufferedIOBase.typedef = TypeDef(
     '_BufferedIOBase', W_IOBase.typedef,
     __new__ = generic_new_descr(W_BufferedIOBase),
     )
 
 class W_BufferedReader(W_BufferedIOBase):
-    pass
+    def __init__(self, space):
+        W_BufferedIOBase.__init__(self, space)
+        self.ok = False
+        self.detached = False
+
+    @unwrap_spec('self', ObjSpace, W_Root, int)
+    def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE):
+        raw = space.interp_w(W_IOBase, w_raw)
+        raw.check_readable_w(space)
+
+        self.raw = raw
+        self.buffer_size = buffer_size
+        self.readable = True
+        self.writable = False
+
+        self._init(space)
+        self._reset_buf()
+
+    def _reset_buf(self):
+        self.read_end = -1
+
 W_BufferedReader.typedef = TypeDef(
     'BufferedReader', W_BufferedIOBase.typedef,
     __new__ = generic_new_descr(W_BufferedReader),
+    __init__  = interp2app(W_BufferedReader.descr_init),
     )
 
 class W_BufferedWriter(W_BufferedIOBase):

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	Wed Oct 20 20:03:56 2010
@@ -110,6 +110,20 @@
         return space.w_False
 
     @unwrap_spec('self', ObjSpace)
+    def check_readable_w(self, space):
+        if not space.is_true(space.call_method(self, 'readable')):
+            raise OperationError(
+                space.w_IOError,
+                space.wrap("file or stream is not readable"))
+
+    @unwrap_spec('self', ObjSpace)
+    def check_writable_w(self, space):
+        if not space.is_true(space.call_method(self, 'writable')):
+            raise OperationError(
+                space.w_IOError,
+                space.wrap("file or stream is not writable"))
+
+    @unwrap_spec('self', ObjSpace)
     def check_seekable_w(self, space):
         if not space.is_true(space.call_method(self, 'seekable')):
             raise OperationError(
@@ -214,6 +228,8 @@
     readable = interp2app(W_IOBase.readable_w),
     writable = interp2app(W_IOBase.writable_w),
     seekable = interp2app(W_IOBase.seekable_w),
+    _checkReadable = interp2app(W_IOBase.check_readable_w),
+    _checkWritable = interp2app(W_IOBase.check_writable_w),
     _checkSeekable = interp2app(W_IOBase.check_seekable_w),
     closed = GetSetProperty(W_IOBase.closed_get_w),
 

Added: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- (empty file)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Wed Oct 20 20:03:56 2010
@@ -0,0 +1,14 @@
+from pypy.conftest import gettestobjspace
+from pypy.tool.udir import udir
+
+class AppTestBufferedIO:
+    def setup_class(cls):
+        cls.space = gettestobjspace(usemodules=['_io'])
+        tmpfile = udir.join('tmpfile')
+        tmpfile.write("a\nb\nc", mode='wb')
+        cls.w_tmpfile = cls.space.wrap(str(tmpfile))
+
+    def test_simple(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile)
+        _io.BufferedReader(raw)


From afa at codespeak.net  Wed Oct 20 23:44:18 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 20 Oct 2010 23:44:18 +0200 (CEST)
Subject: [pypy-svn] r78148 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101020214418.87F84282BDC@codespeak.net>

Author: afa
Date: Wed Oct 20 23:44:16 2010
New Revision: 78148

Added:
   pypy/branch/fast-forward/pypy/module/_io/interp_textio.py   (contents, props changed)
Modified:
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py   (contents, props changed)
Log:
Add missing file


Added: pypy/branch/fast-forward/pypy/module/_io/interp_textio.py
==============================================================================
--- (empty file)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_textio.py	Wed Oct 20 23:44:16 2010
@@ -0,0 +1,17 @@
+from pypy.module._io.interp_iobase import W_IOBase
+from pypy.interpreter.typedef import (
+    TypeDef, generic_new_descr)
+
+class W_TextIOBase(W_IOBase):
+    pass
+W_TextIOBase.typedef = TypeDef(
+    '_TextIOBase', W_IOBase.typedef,
+    __new__ = generic_new_descr(W_TextIOBase),
+    )
+
+class W_TextIOWrapper(W_TextIOBase):
+    pass
+W_TextIOWrapper.typedef = TypeDef(
+    'TextIOWrapper', W_TextIOBase.typedef,
+    __new__ = generic_new_descr(W_TextIOWrapper),
+    )

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Wed Oct 20 23:44:16 2010
@@ -1,14 +1,14 @@
-from pypy.conftest import gettestobjspace
-from pypy.tool.udir import udir
-
-class AppTestBufferedIO:
-    def setup_class(cls):
-        cls.space = gettestobjspace(usemodules=['_io'])
-        tmpfile = udir.join('tmpfile')
-        tmpfile.write("a\nb\nc", mode='wb')
-        cls.w_tmpfile = cls.space.wrap(str(tmpfile))
-
-    def test_simple(self):
-        import _io
-        raw = _io.FileIO(self.tmpfile)
-        _io.BufferedReader(raw)
+from pypy.conftest import gettestobjspace
+from pypy.tool.udir import udir
+
+class AppTestBufferedIO:
+    def setup_class(cls):
+        cls.space = gettestobjspace(usemodules=['_io'])
+        tmpfile = udir.join('tmpfile')
+        tmpfile.write("a\nb\nc", mode='wb')
+        cls.w_tmpfile = cls.space.wrap(str(tmpfile))
+
+    def test_simple(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile)
+        _io.BufferedReader(raw)


From afa at codespeak.net  Thu Oct 21 00:38:44 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 21 Oct 2010 00:38:44 +0200 (CEST)
Subject: [pypy-svn] r78149 - pypy/branch/fast-forward/pypy/module/_io
Message-ID: <20101020223844.2035536E3D6@codespeak.net>

Author: afa
Date: Thu Oct 21 00:38:42 2010
New Revision: 78149

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
Log:
Fix translation


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Thu Oct 21 00:38:42 2010
@@ -11,7 +11,7 @@
 class W_BufferedIOBase(W_IOBase):
     def __init__(self, space):
         W_IOBase.__init__(self, space)
-        self.buffer = None
+        self.buffer = lltype.nullptr(rffi.CCHARP.TO)
         self.lock = None
 
     def _init(self, space):


From trundle at codespeak.net  Thu Oct 21 01:44:13 2010
From: trundle at codespeak.net (trundle at codespeak.net)
Date: Thu, 21 Oct 2010 01:44:13 +0200 (CEST)
Subject: [pypy-svn] r78150 - in pypy/branch/fast-forward: lib-python
	pypy/interpreter pypy/objspace/flow
Message-ID: <20101020234413.E3CD8282BDC@codespeak.net>

Author: trundle
Date: Thu Oct 21 01:44:12 2010
New Revision: 78150

Modified:
   pypy/branch/fast-forward/lib-python/TODO
   pypy/branch/fast-forward/pypy/interpreter/pyopcode.py
   pypy/branch/fast-forward/pypy/objspace/flow/objspace.py
Log:
Catching a string exception now raises a DeprecationWarning.


Modified: pypy/branch/fast-forward/lib-python/TODO
==============================================================================
--- pypy/branch/fast-forward/lib-python/TODO	(original)
+++ pypy/branch/fast-forward/lib-python/TODO	Thu Oct 21 01:44:12 2010
@@ -32,9 +32,6 @@
 - Ast objects should be picklable, see in pypy/module/_ast/test/test_ast.py:
   test_pickle()
 
-- catching a string should emit a DeprecationWarning ("catching of
-  string exceptions is deprecated")
-
 - missing builtin: memoryview
 
 Longer tasks

Modified: pypy/branch/fast-forward/pypy/interpreter/pyopcode.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyopcode.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyopcode.py	Thu Oct 21 01:44:12 2010
@@ -747,6 +747,16 @@
         return self.space.not_(self.space.is_(w_1, w_2))
 
     def cmp_exc_match(self, w_1, w_2):
+        if self.space.is_true(self.space.isinstance(w_2, self.space.w_tuple)):
+            for w_t in self.space.fixedview(w_2):
+                if self.space.is_true(self.space.isinstance(w_t,
+                                                            self.space.w_str)):
+                    self.space.warn("catching of string exceptions is "
+                                    "deprecated",
+                                    self.space.w_DeprecationWarning)
+        elif self.space.is_true(self.space.isinstance(w_2, self.space.w_str)):
+            self.space.warn("catching of string exceptions is deprecated",
+                            self.space.w_DeprecationWarning)
         return self.space.newbool(self.space.exception_match(w_1, w_2))
 
     def COMPARE_OP(self, testnum, next_instr):

Modified: pypy/branch/fast-forward/pypy/objspace/flow/objspace.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/flow/objspace.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/flow/objspace.py	Thu Oct 21 01:44:12 2010
@@ -77,6 +77,9 @@
         self.specialcases = {}
         #self.make_builtins()
         #self.make_sys()
+        # w_str is needed because cmp_exc_match of frames checks against it,
+        # as string exceptions are deprecated
+        self.w_str = Constant(str)
         # objects which should keep their SomeObjectness
         self.not_really_const = NOT_REALLY_CONST
 


From hakanardo at codespeak.net  Thu Oct 21 08:20:15 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Thu, 21 Oct 2010 08:20:15 +0200 (CEST)
Subject: [pypy-svn] r78151 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101021062015.4FFD3282BDC@codespeak.net>

Author: hakanardo
Date: Thu Oct 21 08:20:12 2010
New Revision: 78151

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
pure_operations inherited from first iteration

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Thu Oct 21 08:20:12 2010
@@ -43,6 +43,11 @@
             if not v.is_constant() and v.box:
                 v.fromstart = True
 
+        for op in self.optimizer.pure_operations.values():
+            v = self.getvalue(op.result)
+            v.fromstart = True
+            
+
         self.snapshot_map ={None: None}
         
         inputargs = []

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Thu Oct 21 08:20:12 2010
@@ -121,6 +121,7 @@
                 fieldvalue = self._fields[ofs]
                 fieldvalue.get_args_for_fail(modifier)
 
+    # FIXME: circular references
     def get_forced_boxes(self):
         if self.box is None:
             lst = self._get_field_descr_list()

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Thu Oct 21 08:20:12 2010
@@ -1131,8 +1131,8 @@
         jump(i1)
         """
         expected = """
-        [i]
-        jump(5)
+        []
+        jump()
         """
         self.node.value = 5
         self.optimize_loop(ops, 'Not', expected)
@@ -3590,6 +3590,23 @@
         """
         self.optimize_loop(ops, 'Not, Not, Not', expected)
 
+    def test_pure(self):
+        ops = """
+        [p42]
+        p53 = getfield_gc(ConstPtr(myptr), descr=nextdescr)
+        p59 = getfield_gc_pure(p53, descr=valuedescr)
+        i61 = call(1, p59, descr=nonwritedescr)
+        jump(p42)
+        """
+        expected = """
+        [p42, p59]
+        i61 = call(1, p59, descr=nonwritedescr)
+        jump(p42, p59)
+
+        """
+        self.node.value = 5
+        self.optimize_loop(ops, 'Not', expected)
+
     def test_addsub_ovf(self):
         ops = """
         [i0]


From afa at codespeak.net  Thu Oct 21 08:45:30 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 21 Oct 2010 08:45:30 +0200 (CEST)
Subject: [pypy-svn] r78152 - in pypy/branch/fast-forward/pypy/module/array:
	. test
Message-ID: <20101021064530.0AA06282BDC@codespeak.net>

Author: afa
Date: Thu Oct 21 08:45:29 2010
New Revision: 78152

Modified:
   pypy/branch/fast-forward/pypy/module/array/interp_array.py
   pypy/branch/fast-forward/pypy/module/array/test/test_array.py
Log:
Add overflow checks in array module


Modified: pypy/branch/fast-forward/pypy/module/array/interp_array.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/array/interp_array.py	(original)
+++ pypy/branch/fast-forward/pypy/module/array/interp_array.py	Thu Oct 21 08:45:29 2010
@@ -6,6 +6,7 @@
 from pypy.rlib.jit import dont_look_inside
 from pypy.rlib import rgc
 from pypy.rlib.unroll import unrolling_iterable
+from pypy.rlib.rarithmetic import ovfcheck
 from pypy.rlib.rstruct.runpack import runpack
 from pypy.interpreter.argument import Arguments, Signature
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root, Wrappable
@@ -488,7 +489,11 @@
             raise
         a = mytype.w_class(space)
         repeat = max(repeat, 0)
-        a.setlen(self.len * repeat)
+        try:
+            newlen = ovfcheck(self.len * repeat)
+        except OverflowError:
+            raise MemoryError
+        a.setlen(newlen)
         for r in range(repeat):
             for i in range(self.len):
                 a.buffer[r * self.len + i] = self.buffer[i]
@@ -506,7 +511,11 @@
             raise
         oldlen = self.len
         repeat = max(repeat, 0)
-        self.setlen(self.len * repeat)
+        try:
+            newlen = ovfcheck(self.len * repeat)
+        except OverflowError:
+            raise MemoryError
+        self.setlen(newlen)
         for r in range(1, repeat):
             for i in range(oldlen):
                 self.buffer[r * oldlen + i] = self.buffer[i]
@@ -544,7 +553,10 @@
             raise OperationError(space.w_TypeError, space.wrap(msg))
         n = space.int_w(w_n)
 
-        size = self.itemsize * n
+        try:
+            size = ovfcheck(self.itemsize * n)
+        except OverflowError:
+            raise MemoryError
         w_item = space.call_method(w_f, 'read', space.wrap(size))
         item = space.str_w(w_item)
         if len(item) < size:

Modified: pypy/branch/fast-forward/pypy/module/array/test/test_array.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/array/test/test_array.py	(original)
+++ pypy/branch/fast-forward/pypy/module/array/test/test_array.py	Thu Oct 21 08:45:29 2010
@@ -1,4 +1,5 @@
 from pypy.conftest import gettestobjspace
+import sys
 import py
 import py.test
 
@@ -590,6 +591,9 @@
         a += self.array('i', (7,))
         assert repr(a) == "array('i', [1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 7])"
 
+        raises(MemoryError, "a * self.maxint")
+        raises(MemoryError, "a *= self.maxint")
+
         raises(TypeError, "a = self.array('i') + 2")
         raises(TypeError, "self.array('i') + self.array('b')")
         a = self.array('i')
@@ -812,7 +816,8 @@
         cls.array = array.array
         import struct
         cls.struct = struct
-        cls.tempfile = str(py.test.ensuretemp('array').join('tmpfile'))        
+        cls.tempfile = str(py.test.ensuretemp('array').join('tmpfile'))
+        cls.maxint = sys.maxint
 
 class AppTestArray(BaseArrayTests):
     def setup_class(cls):
@@ -831,6 +836,7 @@
         """)
         cls.w_tempfile = cls.space.wrap(
             str(py.test.ensuretemp('array').join('tmpfile')))
+        cls.w_maxint = cls.space.wrap(sys.maxint)
 
 
 


From afa at codespeak.net  Thu Oct 21 08:59:15 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 21 Oct 2010 08:59:15 +0200 (CEST)
Subject: [pypy-svn] r78153 - pypy/branch/fast-forward/pypy/module/select
Message-ID: <20101021065915.268D5282BDC@codespeak.net>

Author: afa
Date: Thu Oct 21 08:59:13 2010
New Revision: 78153

Modified:
   pypy/branch/fast-forward/pypy/module/select/interp_select.py
Log:
Add Poll.modify()


Modified: pypy/branch/fast-forward/pypy/module/select/interp_select.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/select/interp_select.py	(original)
+++ pypy/branch/fast-forward/pypy/module/select/interp_select.py	Thu Oct 21 08:59:13 2010
@@ -2,8 +2,10 @@
 from pypy.interpreter.typedef import TypeDef
 from pypy.interpreter.baseobjspace import Wrappable
 from pypy.interpreter.gateway import W_Root, ObjSpace, interp2app
-from pypy.interpreter.error import OperationError, operationerrfmt
+from pypy.interpreter.error import (
+    OperationError, operationerrfmt, wrap_oserror)
 from pypy.rlib import rpoll
+import errno
 
 defaultevents = rpoll.POLLIN | rpoll.POLLOUT | rpoll.POLLPRI
 
@@ -41,6 +43,14 @@
         self.fddict[fd] = events
     register.unwrap_spec = ['self', ObjSpace, W_Root, int]
 
+    def modify(self, space, w_fd, events):
+        fd = as_fd_w(space, w_fd)
+        if fd not in self.fddict:
+            raise wrap_oserror(space, OSError(errno.ENOENT, "poll.modify"),
+                               exception_name='w_IOError')
+        self.fddict[fd] = events
+    modify.unwrap_spec = ['self', ObjSpace, W_Root, int]
+
     def unregister(self, space, w_fd):
         fd = as_fd_w(space, w_fd)
         try:
@@ -81,7 +91,7 @@
     poll.unwrap_spec = ['self', ObjSpace, W_Root]
 
 pollmethods = {}
-for methodname in 'register unregister poll'.split():
+for methodname in 'register modify unregister poll'.split():
     method = getattr(Poll, methodname)
     assert hasattr(method,'unwrap_spec'), methodname
     assert method.im_func.func_code.co_argcount == len(method.unwrap_spec), methodname


From afa at codespeak.net  Thu Oct 21 09:00:47 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 21 Oct 2010 09:00:47 +0200 (CEST)
Subject: [pypy-svn] r78154 - pypy/branch/fast-forward/lib-python
Message-ID: <20101021070047.59434282BDC@codespeak.net>

Author: afa
Date: Thu Oct 21 09:00:45 2010
New Revision: 78154

Modified:
   pypy/branch/fast-forward/lib-python/TODO
Log:
Add a TODO item


Modified: pypy/branch/fast-forward/lib-python/TODO
==============================================================================
--- pypy/branch/fast-forward/lib-python/TODO	(original)
+++ pypy/branch/fast-forward/lib-python/TODO	Thu Oct 21 09:00:45 2010
@@ -26,6 +26,8 @@
 
 - Fix fcntl.fcntl(fd, fcntl.F_NOTIFY, fcntl.DN_MULTISHOT) on 32bit platform.
 
+- missing functions in itertools: combinations, product, compress...
+
 Medium tasks
 ------------
 


From arigo at codespeak.net  Thu Oct 21 11:22:13 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 11:22:13 +0200 (CEST)
Subject: [pypy-svn] r78156 - pypy/trunk/pypy/rlib
Message-ID: <20101021092213.1F30F282BDC@codespeak.net>

Author: arigo
Date: Thu Oct 21 11:22:12 2010
New Revision: 78156

Modified:
   pypy/trunk/pypy/rlib/jit.py
Log:
Fix order-dependency annotation issues.


Modified: pypy/trunk/pypy/rlib/jit.py
==============================================================================
--- pypy/trunk/pypy/rlib/jit.py	(original)
+++ pypy/trunk/pypy/rlib/jit.py	Thu Oct 21 11:22:12 2010
@@ -380,6 +380,11 @@
 
     def compute_result_annotation(self, **kwds_s):
         from pypy.annotation import model as annmodel
+
+        if self.instance.__name__ == 'jit_merge_point':
+            if not self.annotate_hooks(**kwds_s):
+                return None      # wrong order, try again later
+
         driver = self.instance.im_self
         keys = kwds_s.keys()
         keys.sort()
@@ -408,22 +413,19 @@
                                    key[2:])
             cache[key] = s_value
 
-        if self.instance.__name__ == 'jit_merge_point':
-            self.annotate_hooks(**kwds_s)
-            
         return annmodel.s_None
 
     def annotate_hooks(self, **kwds_s):
         driver = self.instance.im_self
         s_jitcell = self.bookkeeper.valueoftype(BaseJitCell)
-        self.annotate_hook(driver.get_jitcell_at, driver.greens, **kwds_s)
-        self.annotate_hook(driver.set_jitcell_at, driver.greens, [s_jitcell],
-                           **kwds_s)
-        self.annotate_hook(driver.get_printable_location, driver.greens, **kwds_s)
+        h = self.annotate_hook
+        return (h(driver.get_jitcell_at, driver.greens, **kwds_s)
+            and h(driver.set_jitcell_at, driver.greens, [s_jitcell], **kwds_s)
+            and h(driver.get_printable_location, driver.greens, **kwds_s))
 
     def annotate_hook(self, func, variables, args_s=[], **kwds_s):
         if func is None:
-            return
+            return True
         bk = self.bookkeeper
         s_func = bk.immutablevalue(func)
         uniquekey = 'jitdriver.%s' % func.func_name
@@ -435,9 +437,11 @@
                 objname, fieldname = name.split('.')
                 s_instance = kwds_s['s_' + objname]
                 s_arg = s_instance.classdef.about_attribute(fieldname)
-                assert s_arg is not None
+                if s_arg is None:
+                    return False     # wrong order, try again later
             args_s.append(s_arg)
         bk.emulate_pbc_call(uniquekey, s_func, args_s)
+        return True
 
     def specialize_call(self, hop, **kwds_i):
         # XXX to be complete, this could also check that the concretetype


From arigo at codespeak.net  Thu Oct 21 11:23:06 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 11:23:06 +0200 (CEST)
Subject: [pypy-svn] r78157 - pypy/trunk/pypy/rpython/lltypesystem
Message-ID: <20101021092306.D4EC7282BDC@codespeak.net>

Author: arigo
Date: Thu Oct 21 11:23:05 2010
New Revision: 78157

Modified:
   pypy/trunk/pypy/rpython/lltypesystem/llarena.py
   pypy/trunk/pypy/rpython/lltypesystem/rffi.py
Log:
A non-rpython version of str2charp that allocates an immortal string.


Modified: pypy/trunk/pypy/rpython/lltypesystem/llarena.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/llarena.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/llarena.py	Thu Oct 21 11:23:05 2010
@@ -440,7 +440,7 @@
                                    [rffi.INT],
                                    rffi.INT,
                                    sandboxsafe=True, _nowrapper=True)
-    _dev_zero = rffi.str2charp('/dev/zero')   # prebuilt
+    _dev_zero = rffi.str2charp_immortal('/dev/zero')   # prebuilt
 
     def clear_large_memory_chunk(baseaddr, size):
         # on some Unixy platforms, reading from /dev/zero is the fastest way

Modified: pypy/trunk/pypy/rpython/lltypesystem/rffi.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/rffi.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/rffi.py	Thu Oct 21 11:23:05 2010
@@ -607,6 +607,15 @@
         return array
     str2charp._annenforceargs_ = [strtype]
 
+    def str2charp_immortal(s):
+        "NOT_RPYTHON"
+        array = lltype.malloc(TYPEP.TO, len(s) + 1, flavor='raw',
+                              immortal=True)
+        for i in range(len(s)):
+            array[i] = s[i]
+        array[len(s)] = lastchar
+        return array
+
     def free_charp(cp):
         lltype.free(cp, flavor='raw')
 
@@ -725,19 +734,19 @@
         l = [cp[i] for i in range(size)]
         return emptystr.join(l)
 
-    return (str2charp, free_charp, charp2str,
+    return (str2charp, str2charp_immortal, free_charp, charp2str,
             get_nonmovingbuffer, free_nonmovingbuffer,
             alloc_buffer, str_from_buffer, keep_buffer_alive_until_here,
             charp2strn, charpsize2str,
             )
 
-(str2charp, free_charp, charp2str,
+(str2charp, str2charp_immortal, free_charp, charp2str,
  get_nonmovingbuffer, free_nonmovingbuffer,
  alloc_buffer, str_from_buffer, keep_buffer_alive_until_here,
  charp2strn, charpsize2str,
  ) = make_string_mappings(str)
 
-(unicode2wcharp, free_wcharp, wcharp2unicode,
+(unicode2wcharp, unicode2wcharp_immortal, free_wcharp, wcharp2unicode,
  get_nonmoving_unicodebuffer, free_nonmoving_unicodebuffer,
  alloc_unicodebuffer, unicode_from_buffer, keep_unicodebuffer_alive_until_here,
  wcharp2unicoden, wcharpsize2unicode,


From arigo at codespeak.net  Thu Oct 21 11:45:03 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 11:45:03 +0200 (CEST)
Subject: [pypy-svn] r78158 - in pypy/trunk/pypy/rpython/lltypesystem: . test
Message-ID: <20101021094503.CA5EC282BDC@codespeak.net>

Author: arigo
Date: Thu Oct 21 11:45:02 2010
New Revision: 78158

Modified:
   pypy/trunk/pypy/rpython/lltypesystem/ll2ctypes.py
   pypy/trunk/pypy/rpython/lltypesystem/test/test_rffi.py
Log:
Test and fix.


Modified: pypy/trunk/pypy/rpython/lltypesystem/ll2ctypes.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/ll2ctypes.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/ll2ctypes.py	Thu Oct 21 11:45:02 2010
@@ -823,6 +823,8 @@
         except (ValueError, OverflowError):
             for tc in 'HIL':
                 if array(tc).itemsize == array('u').itemsize:
+                    import struct
+                    cobj &= 256 ** struct.calcsize(tc) - 1
                     llobj = array('u', array(tc, (cobj,)).tostring())[0]
                     break
             else:

Modified: pypy/trunk/pypy/rpython/lltypesystem/test/test_rffi.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/test/test_rffi.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/test/test_rffi.py	Thu Oct 21 11:45:02 2010
@@ -787,3 +787,10 @@
     mixann.getgraph(f2, [], s_None)
     mixann.finish()
 
+def test_force_cast_unichar():
+    x = cast(lltype.UniChar, -1)
+    assert isinstance(x, unicode)
+    if sys.maxunicode == 65535:
+        assert cast(LONG, x) == 65535
+    else:
+        assert cast(LONG, cast(INT, x)) == -1


From arigo at codespeak.net  Thu Oct 21 12:44:57 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 12:44:57 +0200 (CEST)
Subject: [pypy-svn] r78159 - pypy/trunk/pypy/module/_rawffi/test
Message-ID: <20101021104457.8CC7A282BDC@codespeak.net>

Author: arigo
Date: Thu Oct 21 12:44:56 2010
New Revision: 78159

Modified:
   pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py
Log:
- add a missing "const"
- add the symbol "ret_un_func" to the list of exported symbols (needed
  on Mac OS/X, for example)


Modified: pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py
==============================================================================
--- pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py	(original)
+++ pypy/trunk/pypy/module/_rawffi/test/test__rawffi.py	Thu Oct 21 12:44:56 2010
@@ -70,7 +70,7 @@
            return s[num];
         }
 
-        char *char_check(char x, char y)
+        const char *char_check(char x, char y)
         {
            if (y == static_str[0])
               return static_str;
@@ -185,6 +185,7 @@
                      sum_x_y
                      give perturb get_s2a check_s2a
                      AAA_first_ordinal_function
+                     ret_un_func
                   """.split()
         eci = ExternalCompilationInfo(export_symbols=symbols)
         return str(platform.compile([c_file], eci, 'x', standalone=False))


From cfbolz at codespeak.net  Thu Oct 21 12:57:01 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 21 Oct 2010 12:57:01 +0200 (CEST)
Subject: [pypy-svn] r78160 - pypy/extradoc/talk/pepm2011
Message-ID: <20101021105701.96049282BDC@codespeak.net>

Author: cfbolz
Date: Thu Oct 21 12:57:00 2010
New Revision: 78160

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
an attempt to rewrite section 4


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 21 12:57:00 2010
@@ -29,6 +29,8 @@
   fancyvrb=true,
   showstringspaces=false,
   %keywords={def,while,if,elif,return,class,get,set,new,guard_class}
+  numberstyle = \tiny,
+  numbersep = 0pt,
 }
 
 
@@ -388,12 +390,7 @@
 
 
 \begin{figure}
-\lstset{
-    numbers = right,
-    numberstyle = \tiny,
-    numbersep = 0pt
-}
-\begin{lstlisting}[mathescape]
+\begin{lstlisting}[mathescape,numbers = right]
 # arguments to the trace: $p_{0}$, $p_{1}$
 # inside f: res.add(y)
 guard_class($p_{1}$, BoxedInteger)
@@ -472,8 +469,8 @@
     \item \lstinline{new} creates a new object.
     \item \lstinline{get} reads an attribute of an object.
     \item \lstinline{set} writes to an attribute of an object.
-    \item \lstinline{guard_class} precedes an (inlined) method call and is
-    followed by the trace of the called method.
+    \item \lstinline{guard_class} is a precise type check and precedes an
+    (inlined) method call and is followed by the trace of the called method.
     \item \lstinline{int_add} and \lstinline{int_gt} are integer addition and
     comparison (``greater than''), respectively.
 \end{itemize}
@@ -564,41 +561,63 @@
 \label{sec:statics}
 
 The main insight to improve the code shown in the last section is that objects
-in category 1 do not survive very long -- they are used only inside the loop
-and there is no other outside reference to them. Therefore the optimizer
-identifies objects in category 1 and removes the allocation of these objects,
-and all operations manipulating them.
+in category 1 do not survive very long -- they are used only inside the loop and
+there is no other outside reference to them. The optimizer identifies objects in
+category 1 and removes the allocation of these objects, and all operations
+manipulating them.
 
 This is a process that is usually called \emph{escape analysis}
-\cite{goldberg_higher_1990}. In this paper we will
-perform escape analysis by using partial evaluation. The use of partial evaluation is a
-bit peculiar in that it receives no static input arguments for the trace,
-but it is only used to optimize operations within the trace.
-
-To optimize the trace, it is traversed from beginning to end. Every
-operation in the input trace is either removed, or new operations are
-produced. Whenever a \lstinline{new} operation is seen, the operation it is
-removed optimistically and a \emph{static object}\footnote{Here ``static'' is
-meant in the sense of partial evaluation, \ie known at partial evaluation time,
-not in the sense of static allocation or static method.} is constructed and
-associated with the result variable. The static object describes the shape of
-the original object, \ie where the values that would be stored in the fields of
-the allocated object come from, as well as the type of the object.
-
-When a \lstinline{set} that writes into a static object is optimized, the
-corresponding shape description is updated and the operation is removed. This
-means that the operation was done at partial evaluation time. When the
-optimizer encounters a \lstinline{get} from a static object, the result is read
-from the shape description, and the operation is also removed. Equivalently, a
-\lstinline{guard_class} on a variable that has a shape description can be
-removed as well, because the shape description stores the type and thus the
-outcome of the type check the guard does is statically known. Operations that
-have dynamic (\ie non-static) objects as arguments are just left untouched by
-the optimizer.
-
-In the example from Section~\ref{sub:example}, the following operations
-of Figure~\ref{fig:unopt-trace} (lines 10-17) produce two
-static objects, and can be completely removed from the optimized trace:
+\cite{goldberg_higher_1990}. In this paper we will perform escape analysis by
+using partial evaluation. The use of partial evaluation is a bit peculiar in
+that it receives no static input arguments for the trace, but it is only used to
+optimize operations within the trace. This section will give an informal account
+of this process by examining the example trace in Figure~\ref{fig:unopt-trace}.
+The final trace after optimization can be seen in Figure~\ref{fig:step1} (the
+line numbers are the lines of the unoptimized trace where the operation comes
+from).
+
+\begin{figure}
+\begin{lstlisting}[mathescape,numbers=right,escapechar=|,numberstyle = \tiny,numbersep=0pt, numberblanklines=false]
+# arguments to the trace: $p_{0}$, $p_{1}$ |\setcounter{lstnumber}{2}|
+guard_class($p_1$, BoxedInteger)           |\setcounter{lstnumber}{4}|
+$i_2$ = get($p_1$, intval)
+guard_class($p_0$, BoxedInteger)           |\setcounter{lstnumber}{7}|
+$i_3$ = get($p_0$, intval)
+$i_4$ = int_add($i_2$, $i_3$)              |\setcounter{lstnumber}{25}|
+$i_9$ = int_add($i_4$, -100)               |\setcounter{lstnumber}{35}|
+
+guard_class($p_0$, BoxedInteger)           |\setcounter{lstnumber}{38}|
+$i_{12}$ = get($p_0$, intval)              |\setcounter{lstnumber}{42}|
+$i_{14}$ = int_add($i_{12}$, -1)           |\setcounter{lstnumber}{50}|
+
+$i_{17}$ = int_gt($i_{14}$, 0)             |\setcounter{lstnumber}{53}|
+guard_true($i_{17}$)                       |\setcounter{lstnumber}{42}|
+
+$p_{15}$ = new(BoxedInteger)               |\setcounter{lstnumber}{45}|
+set($p_{15}$, intval, $i_{14}$)            |\setcounter{lstnumber}{26}|
+$p_{10}$ = new(BoxedInteger)               |\setcounter{lstnumber}{28}|
+set($p_{10}$, intval, $i_9$)               |\setcounter{lstnumber}{53}|
+
+jump($p_{15}$, $p_{10}$)
+\end{lstlisting}
+
+\caption{Resulting Trace After Allocation Removal}
+\label{fig:step1}
+\end{figure}
+
+To optimize the trace, it is traversed from beginning to end and an output trace
+is produced at the same time. Every operation in the input trace is either
+removed, or put into the output trace. Sometimes new operations need to be
+produced as well. The optimizer can only remove operations that manipulate
+objects that have been allocated within the trace, all others are copied to the
+output trace untouched.
+
+Looking at the example trace of Figure~\ref{fig:unopt-trace}, this is what
+happens with the operations in lines 1-9. They are manipulating objects that
+existed before the trace because they are passed in as arguments. Therefore the
+optimizer just puts them into the output trace.
+
+The following operations (lines 10-17) are more interesting:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 $p_{5}$ = new(BoxedInteger)
@@ -607,6 +626,22 @@
 set($p_{6}$, intval, -100)
 \end{lstlisting}
 
+When the optimizer encounters a \lstinline{new}, it removes it optimistically,
+and assumes that the object is in category 1. When the object escapes later, it
+will be allocated at that point. The optimizer needs to keep track
+of how the object that the operation creates looks like at various points in
+the trace. This is done with the help of a \emph{static object}\footnote{Here
+``static'' is meant in the sense of partial evaluation, \ie known at partial
+evaluation time, not in the sense of ``static allocation'' or ``static
+method''.}. The static object describes the shape of the object that would have
+been allocated, \ie the type of the object and where the values that would be
+stored in the fields of the allocated object come from.
+
+In the snippet above, the two \lstinline{new} operations are removed and two
+static objects are constructed. The \lstinline{set} operations manipulate a
+static object, therefore they can be removed as well and their effect is
+remembered in the static objects.
+
 The static object associated with $p_{5}$ would store the knowledge that it is a
 \lstinline{BoxedInteger} whose \lstinline{intval} field contains $i_{4}$; the
 one associated with $p_{6}$ would store that it is a \lstinline{BoxedInteger}
@@ -625,36 +660,40 @@
 $i_{9}$ = int_add($i_{7}$, $i_{8}$)
 \end{lstlisting}
 
-First, the \lstinline{guard_class} operations can be removed, because the classes of $p_{5}$ and
-$p_{6}$ are known to be \lstinline{BoxedInteger}. Second, the \lstinline{get} operations can be removed
-and $i_{7}$ and $i_{8}$ are just replaced by $i_{4}$ and -100. The only
-remaining operation in the optimized trace would be:
+The \lstinline{guard_class} operations can be removed, since their argument is a
+static object with the matching type \lstinline{BoxedInteger}. The
+\lstinline{get} operations can be removed as well, because each of them reads a
+field out of a static object. The results of the get operation are replaced with
+what the static object stores in these fields: $i_{7}$ and $i_{8}$ are just
+replaced by $i_{4}$ and -100. The only operation put into the optimized trace
+is the addition:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 $i_{9}$ = int_add($i_{4}$, -100)
 \end{lstlisting}
 
-The rest of the trace from Figure~\ref{fig:unopt-trace} is optimized similarly.
-
-
-So far we have only described what happens when static objects are used in guards and in
-operations that read and write fields. When the static
-object is used in any other operation, it cannot remain static. For example, when
-a static object is stored in a globally accessible place, the object has to
-be allocated, as it might live longer than one iteration of the loop and as
-arbitrary \lstinline{set} operations could change it due to aliasing. This
-means that the static
-object needs to be turned into a dynamic one, \ie lifted. This makes it
-necessary to put operations into the residual code that allocate the
-static object at runtime.
-
-This is what happens at the end of the trace in Figure~\ref{fig:unopt-trace}, when the \lstinline{jump} operation
-is optimized. The arguments of the jump are at this point static objects. Before the
-jump is emitted, they are \emph{lifted}. This means that the optimizer produces code
-that allocates a new object of the right type and sets its fields to the field
-values that the static object has (if the static object points to other static
-objects, those need to be lifted as well, recursively). This means that instead of a simple jump,
-the following operations are emitted:
+The rest of the trace from Figure~\ref{fig:unopt-trace} is optimized in a
+similar vein. The operations in lines 27-35 produce two more static objects and
+are removed. Those in line 36-39 are just put into the output trace because they
+manipulate objects that are allocated before the trace. Lines 40-42 are removed
+because they operate on a static object. Line 43 is put into the output trace.
+Lines 44-46 produce a new static object and are removed, lines 48-51 manipulate
+that static object and are removed as well. Lines 52-54 are put into the output
+trace.
+
+The last operation (line 55) is an interesting case. It is the \lstinline{jump}
+operation that passes control back to the beginning of the trace. The two
+arguments to this operation are at this point static objects. However, because
+they passed into the next iteration of the loop they live longer than the trace
+and therefore cannot remain static. They need to be turned into a dynamic
+(runtime) object before the actual \lstinline{jump} operation. This process of
+turning a static object into a dynamic one is called \emph{lifting}.
+
+Lifting a static object puts \lstinline{new} and \lstinline{set} operations into
+the output trace. Those operations produce an object at runtime that has the
+same shape that the static object describes. This process is a bit delicate,
+because the static objects could form an arbitrary graph structure. In our
+example is is simple, though:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 $p_{15}$ = new(BoxedInteger)
@@ -664,47 +703,21 @@
 jump($p_{15}$, $p_{10}$)
 \end{lstlisting}
 
-Observe how the operations for creating these two instances have been moved to later point in the
-trace. 
-At first sight, it may look like for these operations we didn't gain much, as
-the objects are still allocated in the end. However, our optimizations were still
-worthwhile, because some operations that have been performed
-on the lifted static objects have been removed (some \lstinline{get} operations
-and \lstinline{guard_class} operations).
-
-\begin{figure}
-\begin{lstlisting}[mathescape,numbers=right,escapechar=|,numberstyle = \tiny,numbersep=0pt, numberblanklines=false]
-# arguments to the trace: $p_{0}$, $p_{1}$ |\setcounter{lstnumber}{2}|
-guard_class($p_1$, BoxedInteger)           |\setcounter{lstnumber}{4}|
-$i_2$ = get($p_1$, intval)
-guard_class($p_0$, BoxedInteger)           |\setcounter{lstnumber}{7}|
-$i_3$ = get($p_0$, intval)
-$i_4$ = int_add($i_2$, $i_3$)              |\setcounter{lstnumber}{25}|
-$i_9$ = int_add($i_4$, -100)               |\setcounter{lstnumber}{35}|
-
-guard_class($p_0$, BoxedInteger)           |\setcounter{lstnumber}{38}|
-$i_{12}$ = get($p_0$, intval)              |\setcounter{lstnumber}{42}|
-$i_{14}$ = int_add($i_{12}$, -1)           |\setcounter{lstnumber}{50}|
-
-$i_{17}$ = int_gt($i_{14}$, 0)             |\setcounter{lstnumber}{53}|
-guard_true($i_{17}$)                       |\setcounter{lstnumber}{42}|
-
-$p_{15}$ = new(BoxedInteger)               |\setcounter{lstnumber}{45}|
-set($p_{15}$, intval, $i_{14}$)            |\setcounter{lstnumber}{26}|
-$p_{10}$ = new(BoxedInteger)               |\setcounter{lstnumber}{28}|
-set($p_{10}$, intval, $i_9$)               |\setcounter{lstnumber}{53}|
 
-jump($p_{15}$, $p_{10}$)
-\end{lstlisting}
-\caption{Resulting Trace After Allocation Removal}
-\label{fig:step1}
-\end{figure}
+Observe how the operations for creating these two instances have been moved to a
+later point in the trace. This is worthwhile even though the objects have to be
+allocated in the end because some \lstinline{get} operations and
+\lstinline{guard_class} operations on the lifted static objects could be
+removed.
+
+A bit more generally, lifting needs to occur if a static object is used in any
+operation apart from \lstinline{get}, \lstinline{set}, and \lstinline{guard}.
+It also needs to occur if \lstinline{set} is used to store a static object into
+a non-static one.
 
 The final optimized trace of the example can be seen in Figure~\ref{fig:step1}.
 The optimized trace contains only two allocations, instead of the original five,
-and only three \lstinline{guard_class} operations, from the original seven. The
-line numbers are the lines where the operations occurred in the original trace
-in Figure~\ref{fig:unopt-trace}.
+and only three \lstinline{guard_class} operations, from the original seven.
 
 \section{Formal Description of the Algorithm}
 \label{sec:formal}
@@ -849,9 +862,12 @@
 To optimize the simple traces of the last section, we use online partial
 evaluation. The partial evaluator optimizes one operation of a trace at a
 time. Every operation in the unoptimized trace is replaced by a list of
-operations in the optimized trace. This list is empty if the operation
-can be optimized away. The optimization rules can be seen in
-Figure~\ref{fig:optimization}.
+operations in the optimized trace. This list is empty if the operation can be
+optimized away. The optimization rules can be seen in
+Figure~\ref{fig:optimization}. Lists are written using angular brackets $<...>$,
+list concatenation is expressed using two colons: $l_1::l_2$.
+
+XXX input/output of optimizer
 
 The state of the optimizer is stored in an environment $E$ and a \emph{static
 heap} $S$. The environment is a partial function from variables in the
@@ -955,7 +971,8 @@
 not escape) are completely removed; moreover, objects in category 2
 (\ie escaping) are still partially optimized: all the operations in between the
 creation of the object and the point where it escapes that involve the object
-are removed.
+are removed. Objects in category 3 and 4 are also partially optimized, their
+allocation is delayed till the end of the trace.
 
 The optimization is particularly effective for chains of operations.
 For example, it is typical for an interpreter to generate sequences of
@@ -1097,7 +1114,7 @@
 result. The errors were computed using a confidence interval with a 95\%
 confidence level \cite{georges_statistically_2007}. The results are reported in
 Figure~\ref{fig:times}. In addition to the run times the table also reports the
-speedup that PyPy achieves when the optimization is turned on.
+speedup that PyPy achieves when the optimization is turned on. XXX sounds sucky
 
 With the optimization turned on, PyPy's Python interpreter outperforms CPython
 in all benchmarks except spambayes (which heavily relies on regular expression


From arigo at codespeak.net  Thu Oct 21 13:23:06 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 13:23:06 +0200 (CEST)
Subject: [pypy-svn] r78161 - in pypy/trunk/pypy/rlib/rsre: . test
Message-ID: <20101021112306.5557D282BDC@codespeak.net>

Author: arigo
Date: Thu Oct 21 13:23:04 2010
New Revision: 78161

Modified:
   pypy/trunk/pypy/rlib/rsre/rsre_core.py
   pypy/trunk/pypy/rlib/rsre/test/test_match.py
Log:
Fix the tests, at least on Mac.


Modified: pypy/trunk/pypy/rlib/rsre/rsre_core.py
==============================================================================
--- pypy/trunk/pypy/rlib/rsre/rsre_core.py	(original)
+++ pypy/trunk/pypy/rlib/rsre/rsre_core.py	Thu Oct 21 13:23:04 2010
@@ -170,6 +170,8 @@
     def __init__(self, pattern, string, match_start, end, flags):
         AbstractMatchContext.__init__(self, pattern, match_start, end, flags)
         self._string = string
+        if not we_are_translated() and isinstance(string, unicode):
+            self.flags |= rsre_char.SRE_FLAG_UNICODE   # for rsre_re.py
 
     def str(self, index):
         check_nonneg(index)

Modified: pypy/trunk/pypy/rlib/rsre/test/test_match.py
==============================================================================
--- pypy/trunk/pypy/rlib/rsre/test/test_match.py	(original)
+++ pypy/trunk/pypy/rlib/rsre/test/test_match.py	Thu Oct 21 13:23:04 2010
@@ -1,24 +1,49 @@
-import _sre, re, sre_compile
-from pypy.rlib.rsre import rsre_core
+import re
+from pypy.rlib.rsre import rsre_core, rsre_char
 
 
-def get_code(regexp, flags=0, allargs=False):
-    class GotIt(Exception):
-        pass
-    def my_compile(pattern, flags, code, *args):
-        print code
-        raise GotIt(code, flags, args)
-    saved = _sre.compile
-    try:
-        _sre.compile = my_compile
-        try:
-            sre_compile.compile(regexp, flags)
-        except GotIt, e:
-            pass
+def get_hacked_sre_compile(my_compile):
+    """Return a copy of the sre_compile module for which the _sre
+    module is a custom module that has _sre.compile == my_compile
+    and CODESIZE == rsre_char.CODESIZE.
+    """
+    import sre_compile, __builtin__, new
+    sre_hacked = new.module("_sre_hacked")
+    sre_hacked.compile = my_compile
+    sre_hacked.MAGIC = sre_compile.MAGIC
+    sre_hacked.CODESIZE = rsre_char.CODESIZE
+    sre_hacked.getlower = rsre_char.getlower
+    def my_import(name, *args):
+        if name == '_sre':
+            return sre_hacked
         else:
-            raise ValueError("did not reach _sre.compile()!")
+            return default_import(name, *args)
+    src = sre_compile.__file__
+    if src.lower().endswith('.pyc') or src.lower().endswith('.pyo'):
+        src = src[:-1]
+    mod = new.module("sre_compile_hacked")
+    default_import = __import__
+    try:
+        __builtin__.__import__ = my_import
+        execfile(src, mod.__dict__)
     finally:
-        _sre.compile = saved
+        __builtin__.__import__ = default_import
+    return mod
+
+class GotIt(Exception):
+    pass
+def my_compile(pattern, flags, code, *args):
+    print code
+    raise GotIt(code, flags, args)
+sre_compile_hacked = get_hacked_sre_compile(my_compile)
+
+def get_code(regexp, flags=0, allargs=False):
+    try:
+        sre_compile_hacked.compile(regexp, flags)
+    except GotIt, e:
+        pass
+    else:
+        raise ValueError("did not reach _sre.compile()!")
     if allargs:
         return e.args
     else:


From antocuni at codespeak.net  Thu Oct 21 13:23:20 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 21 Oct 2010 13:23:20 +0200 (CEST)
Subject: [pypy-svn] r78162 - pypy/extradoc/talk/pepm2011
Message-ID: <20101021112320.7E45A282BDC@codespeak.net>

Author: antocuni
Date: Thu Oct 21 13:23:18 2010
New Revision: 78162

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
review the new section 4


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 21 13:23:18 2010
@@ -560,7 +560,7 @@
 \section{Allocation Removal in Traces}
 \label{sec:statics}
 
-The main insight to improve the code shown in the last section is that objects
+The main insight to improve the code shown in section \ref{sub:example} is that objects
 in category 1 do not survive very long -- they are used only inside the loop and
 there is no other outside reference to them. The optimizer identifies objects in
 category 1 and removes the allocation of these objects, and all operations
@@ -609,15 +609,16 @@
 is produced at the same time. Every operation in the input trace is either
 removed, or put into the output trace. Sometimes new operations need to be
 produced as well. The optimizer can only remove operations that manipulate
-objects that have been allocated within the trace, all others are copied to the
+objects that have been allocated within the trace, while all other operations are copied to the
 output trace untouched.
 
-Looking at the example trace of Figure~\ref{fig:unopt-trace}, this is what
-happens with the operations in lines 1-9. They are manipulating objects that
-existed before the trace because they are passed in as arguments. Therefore the
-optimizer just puts them into the output trace.
+Looking at the example trace of Figure~\ref{fig:unopt-trace}, the operations
+in lines 1-9 are manipulating objects which existed before the trace and that
+are passed in as arguments: therefore the optimizer just puts them into the
+output trace.
 
 The following operations (lines 10-17) are more interesting:
+\anto{what about putting the line numbers also here? Moreover, figure 4 is put just after this listing, it could be confused}
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 $p_{5}$ = new(BoxedInteger)
@@ -626,8 +627,9 @@
 set($p_{6}$, intval, -100)
 \end{lstlisting}
 
+\anto{I don't like ``how the object looks like''. I don't know what else to say, though :-(} 
 When the optimizer encounters a \lstinline{new}, it removes it optimistically,
-and assumes that the object is in category 1. When the object escapes later, it
+and assumes that the object is in category 1. If later the optimizer finds that the object escapes, it
 will be allocated at that point. The optimizer needs to keep track
 of how the object that the operation creates looks like at various points in
 the trace. This is done with the help of a \emph{static object}\footnote{Here
@@ -664,7 +666,7 @@
 static object with the matching type \lstinline{BoxedInteger}. The
 \lstinline{get} operations can be removed as well, because each of them reads a
 field out of a static object. The results of the get operation are replaced with
-what the static object stores in these fields: $i_{7}$ and $i_{8}$ are just
+what the static object stores in these fields: all the occurences of $i_{7}$ and $i_{8}$ in the trace are just
 replaced by $i_{4}$ and -100. The only operation put into the optimized trace
 is the addition:
 


From arigo at codespeak.net  Thu Oct 21 13:24:11 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 13:24:11 +0200 (CEST)
Subject: [pypy-svn] r78163 - pypy/trunk/pypy/jit/backend/test
Message-ID: <20101021112411.4D166282BDC@codespeak.net>

Author: arigo
Date: Thu Oct 21 13:24:09 2010
New Revision: 78163

Modified:
   pypy/trunk/pypy/jit/backend/test/runner_test.py
Log:
Fix test.


Modified: pypy/trunk/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/trunk/pypy/jit/backend/test/runner_test.py	Thu Oct 21 13:24:09 2010
@@ -519,7 +519,8 @@
 
     def test_call_to_c_function(self):
         from pypy.rlib.libffi import CDLL, types, ArgChain
-        libc = CDLL('libc.so.6')
+        from pypy.rpython.lltypesystem.ll2ctypes import libc_name
+        libc = CDLL(libc_name)
         c_tolower = libc.getpointer('tolower', [types.uchar], types.sint)
         argchain = ArgChain().arg(ord('A'))
         assert c_tolower.call(argchain, rffi.INT) == ord('a')


From arigo at codespeak.net  Thu Oct 21 13:29:49 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 13:29:49 +0200 (CEST)
Subject: [pypy-svn] r78164 - pypy/trunk/pypy/translator/platform
Message-ID: <20101021112949.59E27282BDC@codespeak.net>

Author: arigo
Date: Thu Oct 21 13:29:47 2010
New Revision: 78164

Modified:
   pypy/trunk/pypy/translator/platform/darwin.py
Log:
Use $CC on mac too.  Fixes test_gcc_options.


Modified: pypy/trunk/pypy/translator/platform/darwin.py
==============================================================================
--- pypy/trunk/pypy/translator/platform/darwin.py	(original)
+++ pypy/trunk/pypy/translator/platform/darwin.py	Thu Oct 21 13:29:47 2010
@@ -14,7 +14,10 @@
     
     def __init__(self, cc=None):
         if cc is None:
-            cc = 'gcc'
+            try:
+                cc = os.environ['CC']
+            except KeyError:
+                cc = 'gcc'
         self.cc = cc
 
     def _args_for_shared(self, args):


From antocuni at codespeak.net  Thu Oct 21 13:46:17 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 21 Oct 2010 13:46:17 +0200 (CEST)
Subject: [pypy-svn] r78165 - pypy/trunk/pypy/rlib/test
Message-ID: <20101021114617.671A7282BDC@codespeak.net>

Author: antocuni
Date: Thu Oct 21 13:46:15 2010
New Revision: 78165

Modified:
   pypy/trunk/pypy/rlib/test/test_libffi.py
Log:
checkin this test which passes on 32bit, but I expect it to fail on 64


Modified: pypy/trunk/pypy/rlib/test/test_libffi.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_libffi.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_libffi.py	Thu Oct 21 13:46:15 2010
@@ -173,6 +173,19 @@
         res = self.call(func, [chr(20), 22], rffi.LONG)
         assert res == 42
 
+    def test_unsigned_short_args(self):
+        """
+            unsigned short sum_xy_us(unsigned short x, unsigned short y)
+            {
+                return x+y;
+            }
+        """
+        libfoo = self.get_libfoo()
+        func = (libfoo, 'sum_xy_us', [types.ushort, types.ushort], types.ushort)
+        res = self.call(func, [32000, 8000], rffi.USHORT)
+        assert res == 40000
+
+
     def test_pointer_as_argument(self):
         """#include 
             long inc(long* x)


From cfbolz at codespeak.net  Thu Oct 21 13:46:48 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 21 Oct 2010 13:46:48 +0200 (CEST)
Subject: [pypy-svn] r78166 - pypy/extradoc/talk/pepm2011
Message-ID: <20101021114648.EE1C0282BDC@codespeak.net>

Author: cfbolz
Date: Thu Oct 21 13:46:47 2010
New Revision: 78166

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
fix some of anto's issues, will try to add line numbers now


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 21 13:46:47 2010
@@ -560,7 +560,7 @@
 \section{Allocation Removal in Traces}
 \label{sec:statics}
 
-The main insight to improve the code shown in section \ref{sub:example} is that objects
+The main insight to improve the code shown in Section~\ref{sub:example} is that objects
 in category 1 do not survive very long -- they are used only inside the loop and
 there is no other outside reference to them. The optimizer identifies objects in
 category 1 and removes the allocation of these objects, and all operations
@@ -576,35 +576,6 @@
 line numbers are the lines of the unoptimized trace where the operation comes
 from).
 
-\begin{figure}
-\begin{lstlisting}[mathescape,numbers=right,escapechar=|,numberstyle = \tiny,numbersep=0pt, numberblanklines=false]
-# arguments to the trace: $p_{0}$, $p_{1}$ |\setcounter{lstnumber}{2}|
-guard_class($p_1$, BoxedInteger)           |\setcounter{lstnumber}{4}|
-$i_2$ = get($p_1$, intval)
-guard_class($p_0$, BoxedInteger)           |\setcounter{lstnumber}{7}|
-$i_3$ = get($p_0$, intval)
-$i_4$ = int_add($i_2$, $i_3$)              |\setcounter{lstnumber}{25}|
-$i_9$ = int_add($i_4$, -100)               |\setcounter{lstnumber}{35}|
-
-guard_class($p_0$, BoxedInteger)           |\setcounter{lstnumber}{38}|
-$i_{12}$ = get($p_0$, intval)              |\setcounter{lstnumber}{42}|
-$i_{14}$ = int_add($i_{12}$, -1)           |\setcounter{lstnumber}{50}|
-
-$i_{17}$ = int_gt($i_{14}$, 0)             |\setcounter{lstnumber}{53}|
-guard_true($i_{17}$)                       |\setcounter{lstnumber}{42}|
-
-$p_{15}$ = new(BoxedInteger)               |\setcounter{lstnumber}{45}|
-set($p_{15}$, intval, $i_{14}$)            |\setcounter{lstnumber}{26}|
-$p_{10}$ = new(BoxedInteger)               |\setcounter{lstnumber}{28}|
-set($p_{10}$, intval, $i_9$)               |\setcounter{lstnumber}{53}|
-
-jump($p_{15}$, $p_{10}$)
-\end{lstlisting}
-
-\caption{Resulting Trace After Allocation Removal}
-\label{fig:step1}
-\end{figure}
-
 To optimize the trace, it is traversed from beginning to end and an output trace
 is produced at the same time. Every operation in the input trace is either
 removed, or put into the output trace. Sometimes new operations need to be
@@ -618,7 +589,7 @@
 output trace.
 
 The following operations (lines 10-17) are more interesting:
-\anto{what about putting the line numbers also here? Moreover, figure 4 is put just after this listing, it could be confused}
+% XXX line numbers
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt]
 $p_{5}$ = new(BoxedInteger)
@@ -627,12 +598,11 @@
 set($p_{6}$, intval, -100)
 \end{lstlisting}
 
-\anto{I don't like ``how the object looks like''. I don't know what else to say, though :-(} 
 When the optimizer encounters a \lstinline{new}, it removes it optimistically,
 and assumes that the object is in category 1. If later the optimizer finds that the object escapes, it
 will be allocated at that point. The optimizer needs to keep track
-of how the object that the operation creates looks like at various points in
-the trace. This is done with the help of a \emph{static object}\footnote{Here
+of the state of the object that the operation would have created.
+This is done with the help of a \emph{static object}\footnote{Here
 ``static'' is meant in the sense of partial evaluation, \ie known at partial
 evaluation time, not in the sense of ``static allocation'' or ``static
 method''.}. The static object describes the shape of the object that would have
@@ -721,6 +691,36 @@
 The optimized trace contains only two allocations, instead of the original five,
 and only three \lstinline{guard_class} operations, from the original seven.
 
+\begin{figure}
+\begin{lstlisting}[mathescape,numbers=right,escapechar=|,numberstyle = \tiny,numbersep=0pt, numberblanklines=false]
+# arguments to the trace: $p_{0}$, $p_{1}$ |\setcounter{lstnumber}{2}|
+guard_class($p_1$, BoxedInteger)           |\setcounter{lstnumber}{4}|
+$i_2$ = get($p_1$, intval)
+guard_class($p_0$, BoxedInteger)           |\setcounter{lstnumber}{7}|
+$i_3$ = get($p_0$, intval)
+$i_4$ = int_add($i_2$, $i_3$)              |\setcounter{lstnumber}{25}|
+$i_9$ = int_add($i_4$, -100)               |\setcounter{lstnumber}{35}|
+
+guard_class($p_0$, BoxedInteger)           |\setcounter{lstnumber}{38}|
+$i_{12}$ = get($p_0$, intval)              |\setcounter{lstnumber}{42}|
+$i_{14}$ = int_add($i_{12}$, -1)           |\setcounter{lstnumber}{50}|
+
+$i_{17}$ = int_gt($i_{14}$, 0)             |\setcounter{lstnumber}{53}|
+guard_true($i_{17}$)                       |\setcounter{lstnumber}{42}|
+
+$p_{15}$ = new(BoxedInteger)               |\setcounter{lstnumber}{45}|
+set($p_{15}$, intval, $i_{14}$)            |\setcounter{lstnumber}{26}|
+$p_{10}$ = new(BoxedInteger)               |\setcounter{lstnumber}{28}|
+set($p_{10}$, intval, $i_9$)               |\setcounter{lstnumber}{53}|
+
+jump($p_{15}$, $p_{10}$)
+\end{lstlisting}
+
+\caption{Resulting Trace After Allocation Removal}
+\label{fig:step1}
+\end{figure}
+
+
 \section{Formal Description of the Algorithm}
 \label{sec:formal}
 


From arigo at codespeak.net  Thu Oct 21 13:49:18 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 13:49:18 +0200 (CEST)
Subject: [pypy-svn] r78167 - pypy/trunk/pypy/module/array
Message-ID: <20101021114918.3B8CF282BDC@codespeak.net>

Author: arigo
Date: Thu Oct 21 13:49:16 2010
New Revision: 78167

Modified:
   pypy/trunk/pypy/module/array/interp_array.py
Log:
Reorder a bit the code in item_w().  It should help the JIT
generate less operations, because it doesn't need to compare
the item with the lower and upper bounds explicitly.  It only
needs to do the casting (which requires masking) and then
comparing the result with the original value.


Modified: pypy/trunk/pypy/module/array/interp_array.py
==============================================================================
--- pypy/trunk/pypy/module/array/interp_array.py	(original)
+++ pypy/trunk/pypy/module/array/interp_array.py	Thu Oct 21 13:49:16 2010
@@ -192,32 +192,30 @@
                           mytype.bytes
                     raise OperationError(space.w_OverflowError,
                                          space.wrap(msg))
-            elif mytype.unwrap == 'str_w' or mytype.unwrap == 'unicode_w':
+                return rffi.cast(mytype.itemtype, item)
+            if mytype.unwrap == 'str_w' or mytype.unwrap == 'unicode_w':
                 if len(item) != 1:
                     msg = 'array item must be char'
                     raise OperationError(space.w_TypeError, space.wrap(msg))
                 item = item[0]
-
+                return rffi.cast(mytype.itemtype, item)
+            #
+            # "regular" case: it fits in an rpython integer (lltype.Signed)
+            result = rffi.cast(mytype.itemtype, item)
             if mytype.canoverflow:
-                msg = None
-                if mytype.signed:
-                    if item < -1 << (mytype.bytes * 8 - 1):
+                if rffi.cast(lltype.Signed, result) != item:
+                    # overflow.  build the correct message
+                    if item < 0:
                         msg = ('signed %d-byte integer is less than minimum' %
                                mytype.bytes)
-                    elif item > (1 << (mytype.bytes * 8 - 1)) - 1:
+                    else:
                         msg = ('signed %d-byte integer is greater than maximum'
                                % mytype.bytes)
-                else:
-                    if item < 0:
-                        msg = ('unsigned %d-byte integer is less than minimum'
-                               % mytype.bytes)
-                    elif item > (1 << (mytype.bytes * 8)) - 1:
-                        msg = ('unsigned %d-byte integer is greater'
-                               ' than maximum' % mytype.bytes)
-                if msg is not None:
+                    if not mytype.signed:
+                        msg = 'un' + msg      # 'signed' => 'unsigned'
                     raise OperationError(space.w_OverflowError,
                                          space.wrap(msg))
-            return rffi.cast(mytype.itemtype, item)
+            return result
 
         def __del__(self):
             self.setlen(0)


From arigo at codespeak.net  Thu Oct 21 13:54:09 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 13:54:09 +0200 (CEST)
Subject: [pypy-svn] r78168 - pypy/trunk/pypy/module/pypyjit/test
Message-ID: <20101021115409.3E372282BDC@codespeak.net>

Author: arigo
Date: Thu Oct 21 13:54:07 2010
New Revision: 78168

Modified:
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
Log:
Fix the test on 64 bits.


Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Thu Oct 21 13:54:07 2010
@@ -608,16 +608,17 @@
         #     call that can raise is not exchanged into getarrayitem_gc
 
     def test_overflow_checking(self):
+        startvalue = sys.maxint - 2147483647
         self.run_source('''
         def main():
             def f(a,b):
                 if a < 0: return -1
                 return a-b
-            total = 0
+            total = %d
             for i in range(100000):
                 total += f(i, 5)
             return total
-        ''', 170, ([], 4999450000L))
+        ''' % startvalue, 170, ([], startvalue + 4999450000L))
 
     def test_boolrewrite_invers(self):
         for a, b, res, ops in (('2000', '2000', 20001000, 51),


From cfbolz at codespeak.net  Thu Oct 21 14:02:27 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 21 Oct 2010 14:02:27 +0200 (CEST)
Subject: [pypy-svn] r78169 - pypy/extradoc/talk/pepm2011
Message-ID: <20101021120227.14F7B282BDC@codespeak.net>

Author: cfbolz
Date: Thu Oct 21 14:02:26 2010
New Revision: 78169

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
add line numbers


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 21 14:02:26 2010
@@ -30,7 +30,7 @@
   showstringspaces=false,
   %keywords={def,while,if,elif,return,class,get,set,new,guard_class}
   numberstyle = \tiny,
-  numbersep = 0pt,
+  numbersep = -20pt,
 }
 
 
@@ -591,10 +591,10 @@
 The following operations (lines 10-17) are more interesting:
 % XXX line numbers
 
-\begin{lstlisting}[mathescape,xleftmargin=20pt]
-$p_{5}$ = new(BoxedInteger)
-set($p_{5}$, intval, $i_{4}$)
-$p_{6}$ = new(BoxedInteger)
+\begin{lstlisting}[mathescape,xleftmargin=20pt,numbers=right,escapechar=|, firstnumber=10]
+$p_{5}$ = new(BoxedInteger)   |\setcounter{lstnumber}{11}|
+set($p_{5}$, intval, $i_{4}$) |\setcounter{lstnumber}{14}|
+$p_{6}$ = new(BoxedInteger)   |\setcounter{lstnumber}{16}|
 set($p_{6}$, intval, -100)
 \end{lstlisting}
 
@@ -622,12 +622,10 @@
 The subsequent operations (line 19-29) in Figure~\ref{fig:unopt-trace}, which
 use $p_{5}$ and $p_{6}$, can then be optimized using that knowledge:
 
-\begin{lstlisting}[mathescape,xleftmargin=20pt]
-guard_class($p_{5}$, BoxedInteger)
-# inside BoxedInteger.add
+\begin{lstlisting}[mathescape,xleftmargin=20pt,numbers=right,escapechar=|, firstnumber=20]
+guard_class($p_{5}$, BoxedInteger)  |\setcounter{lstnumber}{21}|
 $i_{7}$ = get($p_{5}$, intval)
-guard_class($p_{6}$, BoxedInteger)
-# inside BoxedInteger.add__int
+guard_class($p_{6}$, BoxedInteger)  |\setcounter{lstnumber}{24}|
 $i_{8}$ = get($p_{6}$, intval)
 $i_{9}$ = int_add($i_{7}$, $i_{8}$)
 \end{lstlisting}
@@ -640,7 +638,7 @@
 replaced by $i_{4}$ and -100. The only operation put into the optimized trace
 is the addition:
 
-\begin{lstlisting}[mathescape,xleftmargin=20pt]
+\begin{lstlisting}[mathescape,xleftmargin=20pt,numbers=right,escapechar=|, firstnumber=26]
 $i_{9}$ = int_add($i_{4}$, -100)
 \end{lstlisting}
 
@@ -667,11 +665,11 @@
 because the static objects could form an arbitrary graph structure. In our
 example is is simple, though:
 
-\begin{lstlisting}[mathescape,xleftmargin=20pt]
-$p_{15}$ = new(BoxedInteger)
-set($p_{15}$, intval, $i_{14}$)
-$p_{10}$ = new(BoxedInteger)
-set($p_{10}$, intval, $i_{9}$)
+\begin{lstlisting}[mathescape,xleftmargin=20pt,numbers=right,escapechar=|, firstnumber=44]
+$p_{15}$ = new(BoxedInteger)               |\setcounter{lstnumber}{45}|
+set($p_{15}$, intval, $i_{14}$)            |\setcounter{lstnumber}{26}|
+$p_{10}$ = new(BoxedInteger)               |\setcounter{lstnumber}{28}|
+set($p_{10}$, intval, $i_9$)               |\setcounter{lstnumber}{54}|
 jump($p_{15}$, $p_{10}$)
 \end{lstlisting}
 
@@ -692,7 +690,7 @@
 and only three \lstinline{guard_class} operations, from the original seven.
 
 \begin{figure}
-\begin{lstlisting}[mathescape,numbers=right,escapechar=|,numberstyle = \tiny,numbersep=0pt, numberblanklines=false]
+\begin{lstlisting}[mathescape,numbers=right,escapechar=|, numberblanklines=false]
 # arguments to the trace: $p_{0}$, $p_{1}$ |\setcounter{lstnumber}{2}|
 guard_class($p_1$, BoxedInteger)           |\setcounter{lstnumber}{4}|
 $i_2$ = get($p_1$, intval)


From arigo at codespeak.net  Thu Oct 21 14:14:42 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 14:14:42 +0200 (CEST)
Subject: [pypy-svn] r78170 - pypy/extradoc/talk/pepm2011
Message-ID: <20101021121442.AAE3A282BDC@codespeak.net>

Author: arigo
Date: Thu Oct 21 14:14:41 2010
New Revision: 78170

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
A typo, and a semicolon (sorry, couldn't resist).


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 21 14:14:41 2010
@@ -611,7 +611,7 @@
 
 In the snippet above, the two \lstinline{new} operations are removed and two
 static objects are constructed. The \lstinline{set} operations manipulate a
-static object, therefore they can be removed as well and their effect is
+static object, therefore they can be removed as well; their effect is
 remembered in the static objects.
 
 The static object associated with $p_{5}$ would store the knowledge that it is a
@@ -663,7 +663,7 @@
 the output trace. Those operations produce an object at runtime that has the
 same shape that the static object describes. This process is a bit delicate,
 because the static objects could form an arbitrary graph structure. In our
-example is is simple, though:
+example it is simple, though:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt,numbers=right,escapechar=|, firstnumber=44]
 $p_{15}$ = new(BoxedInteger)               |\setcounter{lstnumber}{45}|


From cfbolz at codespeak.net  Thu Oct 21 14:23:50 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 21 Oct 2010 14:23:50 +0200 (CEST)
Subject: [pypy-svn] r78171 - pypy/extradoc/talk/pepm2011
Message-ID: <20101021122350.AD95B282BDC@codespeak.net>

Author: cfbolz
Date: Thu Oct 21 14:23:49 2010
New Revision: 78171

Modified:
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
did those


Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 21 14:23:49 2010
@@ -589,7 +589,6 @@
 output trace.
 
 The following operations (lines 10-17) are more interesting:
-% XXX line numbers
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt,numbers=right,escapechar=|, firstnumber=10]
 $p_{5}$ = new(BoxedInteger)   |\setcounter{lstnumber}{11}|


From antocuni at codespeak.net  Thu Oct 21 14:26:50 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 21 Oct 2010 14:26:50 +0200 (CEST)
Subject: [pypy-svn] r78172 - in pypy/trunk/pypy/rlib: . test
Message-ID: <20101021122650.2BC2E282BDC@codespeak.net>

Author: antocuni
Date: Thu Oct 21 14:26:48 2010
New Revision: 78172

Modified:
   pypy/trunk/pypy/rlib/libffi.py
   pypy/trunk/pypy/rlib/test/test_libffi.py
Log:
make rlib.libffi more robust when calling a function that it's declared to return a smaller unsigned, but then called expecting a ULONG. This fixes test_unsigned_short_args in module/_ffi


Modified: pypy/trunk/pypy/rlib/libffi.py
==============================================================================
--- pypy/trunk/pypy/rlib/libffi.py	(original)
+++ pypy/trunk/pypy/rlib/libffi.py	Thu Oct 21 14:26:48 2010
@@ -188,6 +188,9 @@
         #
         if _fits_into_long(RESULT):
             res = self._do_call_int(self.funcsym, ll_args)
+            if self.restype.c_size < types.slong.c_size:
+                # mask res to keep only the bits we are interested in
+                res &= ~(-1 << self.restype.c_size*8)
         elif RESULT is rffi.DOUBLE:
             return self._do_call_float(self.funcsym, ll_args)
         elif RESULT is lltype.Void:

Modified: pypy/trunk/pypy/rlib/test/test_libffi.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_libffi.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_libffi.py	Thu Oct 21 14:26:48 2010
@@ -182,7 +182,9 @@
         """
         libfoo = self.get_libfoo()
         func = (libfoo, 'sum_xy_us', [types.ushort, types.ushort], types.ushort)
-        res = self.call(func, [32000, 8000], rffi.USHORT)
+        # the ULONG in the next line is not a typo: we really want to test
+        # that we get the correct value even if we cast it to a larger type
+        res = self.call(func, [32000, 8000], rffi.ULONG)
         assert res == 40000
 
 


From cfbolz at codespeak.net  Thu Oct 21 14:50:45 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Thu, 21 Oct 2010 14:50:45 +0200 (CEST)
Subject: [pypy-svn] r78173 - pypy/extradoc/talk/pepm2011
Message-ID: <20101021125045.BC7A2282BDC@codespeak.net>

Author: cfbolz
Date: Thu Oct 21 14:50:44 2010
New Revision: 78173

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
some more tweaks by stephan


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Thu Oct 21 14:50:44 2010
@@ -573,22 +573,21 @@
 optimize operations within the trace. This section will give an informal account
 of this process by examining the example trace in Figure~\ref{fig:unopt-trace}.
 The final trace after optimization can be seen in Figure~\ref{fig:step1} (the
-line numbers are the lines of the unoptimized trace where the operation comes
-from).
+line numbers are the lines of the unoptimized trace where the operation originates).
 
-To optimize the trace, it is traversed from beginning to end and an output trace
-is produced at the same time. Every operation in the input trace is either
-removed, or put into the output trace. Sometimes new operations need to be
+To optimize the trace, it is traversed from beginning to end while an output
+trace is produced. Every operation in the input trace is either
+removed or put into the output trace. Sometimes new operations need to be
 produced as well. The optimizer can only remove operations that manipulate
 objects that have been allocated within the trace, while all other operations are copied to the
-output trace untouched.
+output trace unchanged.
 
 Looking at the example trace of Figure~\ref{fig:unopt-trace}, the operations
-in lines 1-9 are manipulating objects which existed before the trace and that
+in lines 1--9 are manipulating objects which existed before the trace and that
 are passed in as arguments: therefore the optimizer just puts them into the
 output trace.
 
-The following operations (lines 10-17) are more interesting:
+The following operations (lines 10--17) are more interesting:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt,numbers=right,escapechar=|, firstnumber=10]
 $p_{5}$ = new(BoxedInteger)   |\setcounter{lstnumber}{11}|
@@ -618,7 +617,7 @@
 one associated with $p_{6}$ would store that it is a \lstinline{BoxedInteger}
 whose \lstinline{intval} field contains the constant -100.
 
-The subsequent operations (line 19-29) in Figure~\ref{fig:unopt-trace}, which
+The subsequent operations (line 20--26) in Figure~\ref{fig:unopt-trace}, which
 use $p_{5}$ and $p_{6}$, can then be optimized using that knowledge:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt,numbers=right,escapechar=|, firstnumber=20]
@@ -634,7 +633,7 @@
 \lstinline{get} operations can be removed as well, because each of them reads a
 field out of a static object. The results of the get operation are replaced with
 what the static object stores in these fields: all the occurences of $i_{7}$ and $i_{8}$ in the trace are just
-replaced by $i_{4}$ and -100. The only operation put into the optimized trace
+replaced by $i_{4}$ and -100. The only operation copied into the optimized trace
 is the addition:
 
 \begin{lstlisting}[mathescape,xleftmargin=20pt,numbers=right,escapechar=|, firstnumber=26]
@@ -642,25 +641,25 @@
 \end{lstlisting}
 
 The rest of the trace from Figure~\ref{fig:unopt-trace} is optimized in a
-similar vein. The operations in lines 27-35 produce two more static objects and
-are removed. Those in line 36-39 are just put into the output trace because they
-manipulate objects that are allocated before the trace. Lines 40-42 are removed
+similar vein. The operations in lines 27--35 produce two more static objects and
+are removed. Those in line 36--39 are just put into the output trace because they
+manipulate objects that are allocated before the trace. Lines 40--42 are removed
 because they operate on a static object. Line 43 is put into the output trace.
-Lines 44-46 produce a new static object and are removed, lines 48-51 manipulate
-that static object and are removed as well. Lines 52-54 are put into the output
+Lines 44--46 produce a new static object and are removed, lines 48--51 manipulate
+that static object and are removed as well. Lines 52--54 are put into the output
 trace.
 
 The last operation (line 55) is an interesting case. It is the \lstinline{jump}
 operation that passes control back to the beginning of the trace. The two
-arguments to this operation are at this point static objects. However, because
-they passed into the next iteration of the loop they live longer than the trace
-and therefore cannot remain static. They need to be turned into a dynamic
-(runtime) object before the actual \lstinline{jump} operation. This process of
+arguments to this operation at this point are static objects. However, because
+they are passed into the next iteration of the loop they live longer than the trace
+and therefore cannot remain static. They need to be turned into dynamic
+(runtime) objects before the actual \lstinline{jump} operation. This process of
 turning a static object into a dynamic one is called \emph{lifting}.
 
 Lifting a static object puts \lstinline{new} and \lstinline{set} operations into
 the output trace. Those operations produce an object at runtime that has the
-same shape that the static object describes. This process is a bit delicate,
+shape described by the static object. This process is a bit delicate,
 because the static objects could form an arbitrary graph structure. In our
 example it is simple, though:
 
@@ -679,14 +678,15 @@
 \lstinline{guard_class} operations on the lifted static objects could be
 removed.
 
-A bit more generally, lifting needs to occur if a static object is used in any
+More generally, lifting needs to occur if a static object is used in any
 operation apart from \lstinline{get}, \lstinline{set}, and \lstinline{guard}.
 It also needs to occur if \lstinline{set} is used to store a static object into
 a non-static one.
 
 The final optimized trace of the example can be seen in Figure~\ref{fig:step1}.
-The optimized trace contains only two allocations, instead of the original five,
-and only three \lstinline{guard_class} operations, from the original seven.
+The optimized trace contains only two allocations, instead of the original
+five, and only three \lstinline{guard_class} operations, compared to the
+original seven.
 
 \begin{figure}
 \begin{lstlisting}[mathescape,numbers=right,escapechar=|, numberblanklines=false]
@@ -866,10 +866,12 @@
 Figure~\ref{fig:optimization}. Lists are written using angular brackets $<...>$,
 list concatenation is expressed using two colons: $l_1::l_2$.
 
-XXX input/output of optimizer
-
 The state of the optimizer is stored in an environment $E$ and a \emph{static
-heap} $S$. The environment is a partial function from variables in the
+heap} $S$. Each step of the optimizer takes an operation, an environment and a
+static heap and produces a list of operations, a new environment and a new
+static heap.
+
+The environment is a partial function from variables in the
 unoptimized trace $V$ to variables in the optimized trace $V^*$ (which are
 themselves written with a
 $\ ^*$ for clarity). The reason for introducing new variables in the optimized
@@ -1112,8 +1114,8 @@
 code. The arithmetic mean of the times of the last 30 runs were used as the
 result. The errors were computed using a confidence interval with a 95\%
 confidence level \cite{georges_statistically_2007}. The results are reported in
-Figure~\ref{fig:times}. In addition to the run times the table also reports the
-speedup that PyPy achieves when the optimization is turned on. XXX sounds sucky
+Figure~\ref{fig:times}. For each implementation the table also reports the
+speedup that PyPy with optimization achieves over it.
 
 With the optimization turned on, PyPy's Python interpreter outperforms CPython
 in all benchmarks except spambayes (which heavily relies on regular expression


From afa at codespeak.net  Thu Oct 21 16:14:14 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 21 Oct 2010 16:14:14 +0200 (CEST)
Subject: [pypy-svn] r78174 - in pypy/trunk/pypy: annotation/test interpreter
	objspace/flow/test rpython/test translator/c/test
Message-ID: <20101021141414.6E553282BE0@codespeak.net>

Author: afa
Date: Thu Oct 21 16:14:12 2010
New Revision: 78174

Modified:
   pypy/trunk/pypy/annotation/test/test_annrpython.py
   pypy/trunk/pypy/interpreter/pyopcode.py
   pypy/trunk/pypy/objspace/flow/test/test_objspace.py
   pypy/trunk/pypy/rpython/test/test_llinterp.py
   pypy/trunk/pypy/translator/c/test/test_typed.py
Log:
Add support for "with" statements in RPython.
The goal is to simplify resource management, like temporary allocations, 
scoped locks, etc::

    with lltype.alloc_buffer(30) as tmpbuf:
        [...use tmpbuf...]

The __exit__ function is always called, even when an exception is raised.

Actually there was not much to do (a space.wrap() was missing)
... at least on top of CPython2.5
It's more complex for other versions of CPython, because this opcode
change implementations every release...


Modified: pypy/trunk/pypy/annotation/test/test_annrpython.py
==============================================================================
--- pypy/trunk/pypy/annotation/test/test_annrpython.py	(original)
+++ pypy/trunk/pypy/annotation/test/test_annrpython.py	Thu Oct 21 16:14:12 2010
@@ -3337,6 +3337,26 @@
         s = a.build_types(f, [])
         assert isinstance(s, annmodel.SomeChar)
 
+    def test_context_manager(self):
+        class C:
+            def __init__(self):
+                pass
+            def __enter__(self):
+                self.x = 1
+            def __exit__(self, *args):
+                self.x = 3
+        def f():
+            c = C()
+            with c:
+                pass
+            return c.x
+
+        a = self.RPythonAnnotator()
+        s = a.build_types(f, [])
+        assert isinstance(s, annmodel.SomeInteger)
+        # not a constant: both __enter__ and __exit__ have been annotated
+        assert not s.is_constant()
+
 
 def g(n):
     return [0,1,2,n]

Modified: pypy/trunk/pypy/interpreter/pyopcode.py
==============================================================================
--- pypy/trunk/pypy/interpreter/pyopcode.py	(original)
+++ pypy/trunk/pypy/interpreter/pyopcode.py	Thu Oct 21 16:14:12 2010
@@ -873,15 +873,34 @@
 
     def WITH_CLEANUP(self, oparg, next_instr):
         # see comment in END_FINALLY for stack state
-        w_exitfunc = self.popvalue()
-        w_unroller = self.peekvalue(2)
+        # This opcode changed a lot between CPython versions
+        if self.pycode.magic >= 0xa0df2ef:
+            # Implementation since 2.7a0: 62191 (introduce SETUP_WITH)
+            raise NotImplementedError("WITH_CLEANUP for CPython 2.7")
+        elif self.pycode.magic >= 0xa0df2d1:
+            # implementation since 2.6a1: 62161 (WITH_CLEANUP optimization)
+            self.popvalue()
+            self.popvalue()
+            w_unroller = self.popvalue()
+            w_exitfunc = self.popvalue()
+            self.pushvalue(w_unroller)
+            self.pushvalue(self.space.w_None)
+            self.pushvalue(self.space.w_None)
+        elif self.pycode.magic >= 0xa0df28c:
+            # Implemementation since 2.5a0: 62092 (changed WITH_CLEANUP opcode)
+            w_exitfunc = self.popvalue()
+            w_unroller = self.peekvalue(2)
+        else:
+            raise NotImplementedError("WITH_CLEANUP for CPython <= 2.4")
+
         unroller = self.space.interpclass_w(w_unroller)
         if isinstance(unroller, SApplicationException):
             operr = unroller.operr
+            w_traceback = self.space.wrap(operr.application_traceback)
             w_result = self.space.call_function(w_exitfunc,
                                                 operr.w_type,
                                                 operr.get_w_value(self.space),
-                                                operr.application_traceback)
+                                                w_traceback)
             if self.space.is_true(w_result):
                 # __exit__() returned True -> Swallow the exception.
                 self.settopvalue(self.space.w_None, 2)

Modified: pypy/trunk/pypy/objspace/flow/test/test_objspace.py
==============================================================================
--- pypy/trunk/pypy/objspace/flow/test/test_objspace.py	(original)
+++ pypy/trunk/pypy/objspace/flow/test/test_objspace.py	Thu Oct 21 16:14:12 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 import new
 import py
 from pypy.objspace.flow.model import Constant, Block, Link, Variable, traverse
@@ -828,6 +829,25 @@
         simplify_graph(graph)
         assert self.all_operations(graph) == {'getitem': 1}
 
+    def test_context_manager(self):
+        def f(c, x):
+            with x:
+                pass
+        graph = self.codetest(f)
+        # 2 method calls: x.__enter__() and x.__exit__(None, None, None)
+        assert self.all_operations(graph) == {'getattr': 2,
+                                              'simple_call': 2}
+        #
+        def g(): pass
+        def f(c, x):
+            with x:
+                g()
+        graph = self.codetest(f)
+        assert self.all_operations(graph) == {
+            'getattr': 2,     # __enter__ and __exit__
+            'simple_call': 4, # __enter__, g and 2 possible calls to __exit__
+            'is_true': 1}     # check the result of __exit__()
+
     def monkey_patch_code(self, code, stacksize, flags, codestring, names, varnames):
         c = code
         return new.code(c.co_argcount, c.co_nlocals, stacksize, flags,

Modified: pypy/trunk/pypy/rpython/test/test_llinterp.py
==============================================================================
--- pypy/trunk/pypy/rpython/test/test_llinterp.py	(original)
+++ pypy/trunk/pypy/rpython/test/test_llinterp.py	Thu Oct 21 16:14:12 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 import py
 import sys
 from pypy.rpython.lltypesystem.lltype import typeOf, pyobjectptr, Ptr,\
@@ -5,6 +6,7 @@
 from pypy.rpython.lltypesystem.lloperation import llop
 from pypy.rpython.llinterp import LLInterpreter, LLException, log
 from pypy.rpython.rmodel import inputconst
+from pypy.rpython.annlowlevel import hlstr
 from pypy.translator.translator import TranslationContext, graphof
 from pypy.rpython.rint import signed_repr
 from pypy.rpython.lltypesystem import rstr, lltype
@@ -622,3 +624,24 @@
 
     interpret(f, [])
 
+def test_context_manager():
+    state = []
+    class C:
+        def __enter__(self):
+            state.append('acquire')
+            return self
+        def __exit__(self, *args):
+            if args[1] is not None:
+                state.append('raised')
+            state.append('release')
+    def f():
+        try:
+            with C() as c:
+                state.append('use')
+                raise ValueError
+        except ValueError:
+            pass
+        return ', '.join(state)
+    res = interpret(f, [])
+    assert hlstr(res) == 'acquire, use, raised, release'
+

Modified: pypy/trunk/pypy/translator/c/test/test_typed.py
==============================================================================
--- pypy/trunk/pypy/translator/c/test/test_typed.py	(original)
+++ pypy/trunk/pypy/translator/c/test/test_typed.py	Thu Oct 21 16:14:12 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 import autopath
 import sys
 import math
@@ -823,3 +824,29 @@
             while int(x + frac) >= -sys.maxint-1:
                 x -= 1
             assert f(x + frac) == -666
+
+    def test_context_manager(self):
+        state = []
+        class C:
+            def __init__(self, name):
+                self.name = name
+            def __enter__(self):
+                state.append('acquire')
+                return self
+            def __exit__(self, *args):
+                if args[1] is not None:
+                    state.append('raised')
+                state.append('release')
+
+        def func():
+            try:
+                with C('hello') as c:
+                    state.append(c.name)
+                    raise ValueError
+            except ValueError:
+                pass
+            return ', '.join(state)
+        f = self.getcompiled(func, [])
+        res = f()
+        assert res == 'acquire, hello, raised, release'
+


From afa at codespeak.net  Thu Oct 21 17:01:39 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 21 Oct 2010 17:01:39 +0200 (CEST)
Subject: [pypy-svn] r78175 - in pypy/trunk/pypy: interpreter objspace/flow
Message-ID: <20101021150139.4DD6B282BE0@codespeak.net>

Author: afa
Date: Thu Oct 21 17:01:37 2010
New Revision: 78175

Modified:
   pypy/trunk/pypy/interpreter/pyopcode.py
   pypy/trunk/pypy/objspace/flow/flowcontext.py
Log:
Implement the SETUP_WITH opcode for the flow object space running on code compiled by cpython 2.7.
Now all previous tests run on top of cpython 2.5 to 2.7.

The implementation was borrowed from the fast-forward branch, and slightly simplified
to adapt to the RPython type system, which has virtual methods but no __mro__ or special descriptors.
This is why this code is not in pyopcode.py


Modified: pypy/trunk/pypy/interpreter/pyopcode.py
==============================================================================
--- pypy/trunk/pypy/interpreter/pyopcode.py	(original)
+++ pypy/trunk/pypy/interpreter/pyopcode.py	Thu Oct 21 17:01:37 2010
@@ -874,10 +874,9 @@
     def WITH_CLEANUP(self, oparg, next_instr):
         # see comment in END_FINALLY for stack state
         # This opcode changed a lot between CPython versions
-        if self.pycode.magic >= 0xa0df2ef:
+        if (self.pycode.magic >= 0xa0df2ef
             # Implementation since 2.7a0: 62191 (introduce SETUP_WITH)
-            raise NotImplementedError("WITH_CLEANUP for CPython 2.7")
-        elif self.pycode.magic >= 0xa0df2d1:
+            or self.pycode.magic >= 0xa0df2d1):
             # implementation since 2.6a1: 62161 (WITH_CLEANUP optimization)
             self.popvalue()
             self.popvalue()
@@ -1249,10 +1248,18 @@
         frame.pushvalue(frame.space.w_None)
         return self.handlerposition   # jump to the handler
 
+class WithBlock(FinallyBlock):
+
+    def really_handle(self, frame, unroller):
+        if (frame.space.full_exceptions and
+            isinstance(unroller, SApplicationException)):
+            unroller.operr.normalize_exception(frame.space)
+        return FinallyBlock.really_handle(self, frame, unroller)
 
 block_classes = {'SETUP_LOOP': LoopBlock,
                  'SETUP_EXCEPT': ExceptBlock,
-                 'SETUP_FINALLY': FinallyBlock}
+                 'SETUP_FINALLY': FinallyBlock,
+                 'SETUP_WITH': WithBlock}
 
 ### helpers written at the application-level ###
 # Some of these functions are expected to be generally useful if other

Modified: pypy/trunk/pypy/objspace/flow/flowcontext.py
==============================================================================
--- pypy/trunk/pypy/objspace/flow/flowcontext.py	(original)
+++ pypy/trunk/pypy/objspace/flow/flowcontext.py	Thu Oct 21 17:01:37 2010
@@ -478,10 +478,19 @@
 
     # `with` statement
 
-    def SETUP_WITH(self, oparg, next_instr):
-        raise NotImplementedError("SETUP_WITH")
+    def SETUP_WITH(self, offsettoend, next_instr):
+        # A simpler version than the 'real' 2.7 one:
+        # directly call manager.__enter__(), don't use special lookup functions
+        # which are too complex for the flow object space.
+        from pypy.interpreter.pyopcode import WithBlock
+        w_manager = self.peekvalue()
+        w_exit = self.space.getattr(w_manager, self.space.wrap("__exit__"))
+        self.settopvalue(w_exit)
+        w_result = self.space.call_method(w_manager, "__enter__")
+        block = WithBlock(self, next_instr + offsettoend)
+        self.append_block(block)
+        self.pushvalue(w_result)
 
-    
     def make_arguments(self, nargs):
         return ArgumentsForTranslation(self.space, self.peekvalues(nargs))
     def argument_factory(self, *args):


From afa at codespeak.net  Thu Oct 21 17:08:51 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 21 Oct 2010 17:08:51 +0200 (CEST)
Subject: [pypy-svn] r78176 - pypy/trunk/pypy/objspace/flow
Message-ID: <20101021150851.74330282BE0@codespeak.net>

Author: afa
Date: Thu Oct 21 17:08:49 2010
New Revision: 78176

Modified:
   pypy/trunk/pypy/objspace/flow/flowcontext.py
Log:
Move SETUP_WITH out of the Unimplemented section


Modified: pypy/trunk/pypy/objspace/flow/flowcontext.py
==============================================================================
--- pypy/trunk/pypy/objspace/flow/flowcontext.py	(original)
+++ pypy/trunk/pypy/objspace/flow/flowcontext.py	Thu Oct 21 17:08:49 2010
@@ -460,21 +460,6 @@
         else:
             v = self.peekvalue(oparg - 1)
         self.space.call_method(v, 'append', w)
-    
-    # XXX Unimplemented 2.7 opcodes ----------------
-
-    # Set literals, set comprehensions
-
-    def BUILD_SET(self, oparg, next_instr):
-        raise NotImplementedError("BUILD_SET")
-
-    def SET_ADD(self, oparg, next_instr):
-        raise NotImplementedError("SET_ADD")
-
-    # Dict comprehensions
-
-    def MAP_ADD(self, oparg, next_instr):
-        raise NotImplementedError("MAP_ADD")
 
     # `with` statement
 
@@ -491,6 +476,21 @@
         self.append_block(block)
         self.pushvalue(w_result)
 
+    # XXX Unimplemented 2.7 opcodes ----------------
+
+    # Set literals, set comprehensions
+
+    def BUILD_SET(self, oparg, next_instr):
+        raise NotImplementedError("BUILD_SET")
+
+    def SET_ADD(self, oparg, next_instr):
+        raise NotImplementedError("SET_ADD")
+
+    # Dict comprehensions
+
+    def MAP_ADD(self, oparg, next_instr):
+        raise NotImplementedError("MAP_ADD")
+
     def make_arguments(self, nargs):
         return ArgumentsForTranslation(self.space, self.peekvalues(nargs))
     def argument_factory(self, *args):


From afa at codespeak.net  Thu Oct 21 17:11:00 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 21 Oct 2010 17:11:00 +0200 (CEST)
Subject: [pypy-svn] r78177 - pypy/trunk/pypy/objspace/flow
Message-ID: <20101021151100.A38B6282BE0@codespeak.net>

Author: afa
Date: Thu Oct 21 17:10:59 2010
New Revision: 78177

Modified:
   pypy/trunk/pypy/objspace/flow/flowcontext.py
Log:
Improve comment


Modified: pypy/trunk/pypy/objspace/flow/flowcontext.py
==============================================================================
--- pypy/trunk/pypy/objspace/flow/flowcontext.py	(original)
+++ pypy/trunk/pypy/objspace/flow/flowcontext.py	Thu Oct 21 17:10:59 2010
@@ -466,7 +466,7 @@
     def SETUP_WITH(self, offsettoend, next_instr):
         # A simpler version than the 'real' 2.7 one:
         # directly call manager.__enter__(), don't use special lookup functions
-        # which are too complex for the flow object space.
+        # which don't make sense on the RPython type system.
         from pypy.interpreter.pyopcode import WithBlock
         w_manager = self.peekvalue()
         w_exit = self.space.getattr(w_manager, self.space.wrap("__exit__"))


From antocuni at codespeak.net  Thu Oct 21 17:13:14 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 21 Oct 2010 17:13:14 +0200 (CEST)
Subject: [pypy-svn] r78178 - pypy/trunk/pypy/rlib/test
Message-ID: <20101021151314.95A88282BE0@codespeak.net>

Author: antocuni
Date: Thu Oct 21 17:13:12 2010
New Revision: 78178

Modified:
   pypy/trunk/pypy/rlib/test/test_libffi.py
Log:
(antocuni, arigo): fix the test: it relied on the fact that dummy was 0 at the beginning of the test, but this is not necessarly true if the same shared library is used by someone else (like, metainterp/test/test_ffi_call)


Modified: pypy/trunk/pypy/rlib/test/test_libffi.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_libffi.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_libffi.py	Thu Oct 21 17:13:12 2010
@@ -249,7 +249,7 @@
 
     def test_void_result(self):
         """
-            int dummy = 0;
+            int dummy;
             void set_dummy(int val) { dummy = val; }
             int get_dummy() { return dummy; }
         """
@@ -257,11 +257,10 @@
         set_dummy = (libfoo, 'set_dummy', [types.sint], types.void)
         get_dummy = (libfoo, 'get_dummy', [], types.sint)
         #
-        res = self.call(get_dummy, [], rffi.LONG)
-        assert res == 0
+        initval = self.call(get_dummy, [], rffi.LONG)
         #
-        res = self.call(set_dummy, [42], lltype.Void, init_result=None)
+        res = self.call(set_dummy, [initval+1], lltype.Void, init_result=None)
         assert res is None
         #
         res = self.call(get_dummy, [], rffi.LONG)
-        assert res == 42
+        assert res == initval+1


From antocuni at codespeak.net  Thu Oct 21 17:30:46 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Thu, 21 Oct 2010 17:30:46 +0200 (CEST)
Subject: [pypy-svn] r78179 - in pypy/trunk/pypy: module/_ffi rlib rlib/test
Message-ID: <20101021153046.CAF65282BF0@codespeak.net>

Author: antocuni
Date: Thu Oct 21 17:30:45 2010
New Revision: 78179

Modified:
   pypy/trunk/pypy/module/_ffi/interp_ffi.py
   pypy/trunk/pypy/rlib/libffi.py
   pypy/trunk/pypy/rlib/test/test_libffi.py
Log:
(antocuni, arigo around): finally fix test_unsigned_short_arguments on 64
bits, by handling all the possible cases separately.



Modified: pypy/trunk/pypy/module/_ffi/interp_ffi.py
==============================================================================
--- pypy/trunk/pypy/module/_ffi/interp_ffi.py	(original)
+++ pypy/trunk/pypy/module/_ffi/interp_ffi.py	Thu Oct 21 17:30:45 2010
@@ -89,11 +89,9 @@
         argchain = self.build_argchain(space, self.func.argtypes, args_w)
         reskind = libffi.types.getkind(self.func.restype)
         if reskind == 'i':
-            intres = self.func.call(argchain, rffi.LONG)
-            return space.wrap(intres)
+            return self._call_int(space, argchain)
         elif reskind == 'u':
-            intres = self.func.call(argchain, rffi.ULONG)
-            return space.wrap(intres)
+            return self._call_uint(space, argchain)
         elif reskind == 'f':
             floatres = self.func.call(argchain, rffi.DOUBLE)
             return space.wrap(floatres)
@@ -102,6 +100,58 @@
             assert voidres is None
             return space.w_None
 
+    def _call_int(self, space, argchain):
+        # if the declared return type of the function is smaller than LONG,
+        # the result buffer may contains garbage in its higher bits.  To get
+        # the correct value, and to be sure to handle the signed/unsigned case
+        # correctly, we need to cast the result to the correct type.  After
+        # that, we cast it back to LONG, because this is what we want to pass
+        # to space.wrap in order to get a nice applevel .
+        #
+        restype = self.func.restype
+        call = self.func.call
+        if restype is libffi.types.slong:
+            intres = call(argchain, rffi.LONG)
+        elif restype is libffi.types.sint:
+            intres = rffi.cast(rffi.LONG, call(argchain, rffi.INT))
+        elif restype is libffi.types.sshort:
+            intres = rffi.cast(rffi.LONG, call(argchain, rffi.SHORT))
+        elif restype is libffi.types.schar:
+            intres = rffi.cast(rffi.LONG, call(argchain, rffi.SIGNEDCHAR))
+        else:
+            raise OperationError(space.w_ValueError,
+                                 space.wrap('Unsupported restype'))
+        return space.wrap(intres)
+
+    def _call_uint(self, space, argchain):
+        # the same comment as above apply. Moreover, we need to be careful
+        # when the return type is ULONG, because the value might not fit into
+        # a signed LONG: this is the only case in which we cast the result to
+        # something different than LONG; as a result, the applevel value will
+        # be a .
+        #
+        # Note that we check for ULONG before UINT: this is needed on 32bit
+        # machines, where they are they same: if we checked for UINT before
+        # ULONG, we would cast to the wrong type.  Note that this also means
+        # that on 32bit the UINT case will never be entered (because it is
+        # handled by the ULONG case).
+        restype = self.func.restype
+        call = self.func.call
+        if restype is libffi.types.ulong:
+            # special case
+            uintres = call(argchain, rffi.ULONG)
+            return space.wrap(uintres)
+        elif restype is libffi.types.uint:
+            intres = rffi.cast(rffi.LONG, call(argchain, rffi.UINT))
+        elif restype is libffi.types.ushort:
+            intres = rffi.cast(rffi.LONG, call(argchain, rffi.USHORT))
+        elif restype is libffi.types.uchar:
+            intres = rffi.cast(rffi.LONG, call(argchain, rffi.UCHAR))
+        else:
+            raise OperationError(space.w_ValueError,
+                                 space.wrap('Unsupported restype'))
+        return space.wrap(intres)
+
     @unwrap_spec('self', ObjSpace)
     def getaddr(self, space):
         """

Modified: pypy/trunk/pypy/rlib/libffi.py
==============================================================================
--- pypy/trunk/pypy/rlib/libffi.py	(original)
+++ pypy/trunk/pypy/rlib/libffi.py	Thu Oct 21 17:30:45 2010
@@ -188,9 +188,6 @@
         #
         if _fits_into_long(RESULT):
             res = self._do_call_int(self.funcsym, ll_args)
-            if self.restype.c_size < types.slong.c_size:
-                # mask res to keep only the bits we are interested in
-                res &= ~(-1 << self.restype.c_size*8)
         elif RESULT is rffi.DOUBLE:
             return self._do_call_float(self.funcsym, ll_args)
         elif RESULT is lltype.Void:

Modified: pypy/trunk/pypy/rlib/test/test_libffi.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_libffi.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_libffi.py	Thu Oct 21 17:30:45 2010
@@ -182,9 +182,7 @@
         """
         libfoo = self.get_libfoo()
         func = (libfoo, 'sum_xy_us', [types.ushort, types.ushort], types.ushort)
-        # the ULONG in the next line is not a typo: we really want to test
-        # that we get the correct value even if we cast it to a larger type
-        res = self.call(func, [32000, 8000], rffi.ULONG)
+        res = self.call(func, [32000, 8000], rffi.USHORT)
         assert res == 40000
 
 


From arigo at codespeak.net  Thu Oct 21 18:05:06 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 21 Oct 2010 18:05:06 +0200 (CEST)
Subject: [pypy-svn] r78181 -
	pypy/branch/cleanup-dict-impl/pypy/jit/codewriter
Message-ID: <20101021160506.A9671282BE0@codespeak.net>

Author: arigo
Date: Thu Oct 21 18:05:05 2010
New Revision: 78181

Modified:
   pypy/branch/cleanup-dict-impl/pypy/jit/codewriter/jtransform.py
Log:
Support for pypy.rlib.rerased: uses cast_opaque_ptr.

Modified: pypy/branch/cleanup-dict-impl/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/jit/codewriter/jtransform.py	Thu Oct 21 18:05:05 2010
@@ -172,6 +172,7 @@
 
     def rewrite_op_same_as(self, op): pass
     def rewrite_op_cast_pointer(self, op): pass
+    def rewrite_op_cast_opaque_ptr(self, op): pass   # rlib.rerased
     def rewrite_op_cast_primitive(self, op): pass
     def rewrite_op_cast_bool_to_int(self, op): pass
     def rewrite_op_cast_bool_to_uint(self, op): pass


From afa at codespeak.net  Thu Oct 21 18:09:39 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 21 Oct 2010 18:09:39 +0200 (CEST)
Subject: [pypy-svn] r78182 - in pypy/trunk/pypy/rpython/lltypesystem: . test
Message-ID: <20101021160939.928EF282BE0@codespeak.net>

Author: afa
Date: Thu Oct 21 18:09:35 2010
New Revision: 78182

Modified:
   pypy/trunk/pypy/rpython/lltypesystem/lltype.py
   pypy/trunk/pypy/rpython/lltypesystem/test/test_lltype.py
Log:
Attempt to define a "scoped allocator" to use in a 'with' statement.


Modified: pypy/trunk/pypy/rpython/lltypesystem/lltype.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/lltype.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/lltype.py	Thu Oct 21 18:09:35 2010
@@ -1868,6 +1868,32 @@
         leakfinder.remember_free(p._obj0)
     p._obj0._free()
 
+def _make_scoped_allocator(T):
+    class ScopedAlloc:
+        def __init__(self, n=None, zero=False):
+            self.buf = malloc(T, n=n, flavor='raw', zero=zero)
+
+        def __enter__(self):
+            return self.buf
+
+        def __exit__(self, *args):
+            free(self.buf, flavor='raw')
+
+    ScopedAlloc.__name__ = 'ScopedAlloc_%s' % (T,)
+    return ScopedAlloc
+_make_scoped_allocator._annspecialcase_ = 'specialize:memo'
+
+def scoped_alloc(T, n=None, zero=False):
+    """Returns a context manager which handles allocation and
+    deallocation of temporary memory. Use it in a with statement::
+
+        with scoped_alloc(Array(Signed), 1) as array:
+            ...use array...
+        ...it's freed now.
+    """
+    return _make_scoped_allocator(T)(n=n, zero=zero)
+scoped_alloc._annspecialcase_ = 'specialize:arg(0)'
+
 def functionptr(TYPE, name, **attrs):
     if not isinstance(TYPE, FuncType):
         raise TypeError, "functionptr() for FuncTypes only"

Modified: pypy/trunk/pypy/rpython/lltypesystem/test/test_lltype.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/test/test_lltype.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/test/test_lltype.py	Thu Oct 21 18:09:35 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 import py
 from pypy.rpython.lltypesystem.lltype import *
 from pypy.rpython.lltypesystem import lltype, rffi
@@ -846,3 +847,9 @@
         p2 = malloc(Array(Signed), 1, flavor='raw', track_allocation=False)
         free(p2, flavor='raw', track_allocation=False)
         # p1 is not freed
+
+    def test_scoped_allocator(self):
+        with scoped_alloc(Array(Signed), 1) as array:
+            array[0] = -42
+            x = array[0]
+        assert x == -42


From hakanardo at codespeak.net  Thu Oct 21 21:31:19 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Thu, 21 Oct 2010 21:31:19 +0200 (CEST)
Subject: [pypy-svn] r78187 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101021193119.D624A282BDC@codespeak.net>

Author: hakanardo
Date: Thu Oct 21 21:31:17 2010
New Revision: 78187

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_exception.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitdriver.py
Log:
Some tests ajusted to reflect the new optimization

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py	Thu Oct 21 21:31:17 2010
@@ -1453,7 +1453,8 @@
             return x
         res = self.meta_interp(f, [299], listops=True)
         assert res == f(299)
-        self.check_loops(guard_class=0, guard_value=3)
+        self.check_loops(guard_class=0, guard_value=1)        
+        self.check_loops(guard_class=0, guard_value=4, everywhere=True)
 
     def test_merge_guardnonnull_guardclass(self):
         from pypy.rlib.objectmodel import instantiate
@@ -1482,7 +1483,10 @@
         res = self.meta_interp(f, [299], listops=True)
         assert res == f(299)
         self.check_loops(guard_class=0, guard_nonnull=0,
-                         guard_nonnull_class=2, guard_isnull=1)
+                         guard_nonnull_class=1, guard_isnull=0)
+        self.check_loops(guard_class=0, guard_nonnull=0,
+                         guard_nonnull_class=3, guard_isnull=1,
+                         everywhere=True)
 
     def test_merge_guardnonnull_guardvalue(self):
         from pypy.rlib.objectmodel import instantiate
@@ -1509,8 +1513,11 @@
             return x
         res = self.meta_interp(f, [299], listops=True)
         assert res == f(299)
-        self.check_loops(guard_class=0, guard_nonnull=0, guard_value=2,
-                         guard_nonnull_class=0, guard_isnull=1)
+        self.check_loops(guard_class=0, guard_nonnull=0, guard_value=1,
+                         guard_nonnull_class=0, guard_isnull=0)
+        self.check_loops(guard_class=0, guard_nonnull=0, guard_value=3,
+                         guard_nonnull_class=0, guard_isnull=1,
+                         everywhere=True)
 
     def test_merge_guardnonnull_guardvalue_2(self):
         from pypy.rlib.objectmodel import instantiate
@@ -1537,8 +1544,11 @@
             return x
         res = self.meta_interp(f, [299], listops=True)
         assert res == f(299)
-        self.check_loops(guard_class=0, guard_nonnull=0, guard_value=2,
-                         guard_nonnull_class=0, guard_isnull=1)
+        self.check_loops(guard_class=0, guard_nonnull=0, guard_value=1,
+                         guard_nonnull_class=0, guard_isnull=0)
+        self.check_loops(guard_class=0, guard_nonnull=0, guard_value=3,
+                         guard_nonnull_class=0, guard_isnull=1,
+                         everywhere=True)
 
     def test_merge_guardnonnull_guardclass_guardvalue(self):
         from pypy.rlib.objectmodel import instantiate
@@ -1568,8 +1578,11 @@
             return x
         res = self.meta_interp(f, [399], listops=True)
         assert res == f(399)
-        self.check_loops(guard_class=0, guard_nonnull=0, guard_value=3,
-                         guard_nonnull_class=0, guard_isnull=1)
+        self.check_loops(guard_class=0, guard_nonnull=0, guard_value=1,
+                         guard_nonnull_class=0, guard_isnull=0)
+        self.check_loops(guard_class=0, guard_nonnull=0, guard_value=4,
+                         guard_nonnull_class=0, guard_isnull=1,
+                         everywhere=True)
 
     def test_residual_call_doesnt_lose_info(self):
         myjitdriver = JitDriver(greens = [], reds = ['x', 'y', 'l'])
@@ -1595,7 +1608,8 @@
                 y.v = g(y.v) - y.v/y.v + lc/l[0] - 1
             return y.v
         res = self.meta_interp(f, [20], listops=True)
-        self.check_loops(getfield_gc=1, getarrayitem_gc=0)
+        self.check_loops(getfield_gc=0, getarrayitem_gc=0)
+        self.check_loops(getfield_gc=1, getarrayitem_gc=0, everywhere=True)
 
     def test_guard_isnull_nonnull(self):
         myjitdriver = JitDriver(greens = [], reds = ['x', 'res'])
@@ -1625,7 +1639,7 @@
         assert res == 42
         self.check_loops(guard_nonnull=1, guard_isnull=1)
 
-    def test_loop_invariant(self):
+    def test_loop_invariant1(self):
         myjitdriver = JitDriver(greens = [], reds = ['x', 'res'])
         class A(object):
             pass
@@ -1650,7 +1664,8 @@
             return res
         res = self.meta_interp(g, [21])
         assert res == 3 * 21
-        self.check_loops(call=1)
+        self.check_loops(call=0)
+        self.check_loops(call=1, everywhere=True)
 
     def test_bug_optimizeopt_mutates_ops(self):
         myjitdriver = JitDriver(greens = [], reds = ['x', 'res', 'const', 'a'])

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py	Thu Oct 21 21:31:17 2010
@@ -5,7 +5,7 @@
 from pypy.jit.metainterp.compile import ResumeGuardDescr
 from pypy.jit.metainterp.compile import ResumeGuardCountersInt
 from pypy.jit.metainterp.compile import compile_tmp_callback
-from pypy.jit.metainterp import optimize, jitprof, typesystem, compile
+from pypy.jit.metainterp import optimize_nopspec, jitprof, typesystem, compile
 from pypy.jit.metainterp.test.oparser import parse
 from pypy.jit.metainterp.test.test_optimizefindnode import LLtypeMixin
 
@@ -41,9 +41,12 @@
         pass
 
 class FakeState:
-    optimize_loop = staticmethod(optimize.optimize_loop)
+    optimize_loop = staticmethod(optimize_nopspec.optimize_loop)
     debug_level = 0
 
+    def attach_unoptimized_bridge_from_interp(*args):
+        pass
+
 class FakeGlobalData:
     loopnumbering = 0
 
@@ -87,11 +90,11 @@
     loop_tokens = []
     loop_token = compile_new_loop(metainterp, loop_tokens, [], 0)
     assert loop_tokens == [loop_token]
-    assert loop_token.number == 1
-    assert staticdata.globaldata.loopnumbering == 2
+    assert loop_token.number == 2
+    assert staticdata.globaldata.loopnumbering == 3
     #
-    assert len(cpu.seen) == 1
-    assert cpu.seen[0][2] == loop_token
+    assert len(cpu.seen) == 2
+    assert cpu.seen[1][2] == loop_token
     #
     del cpu.seen[:]
     metainterp = FakeMetaInterp()
@@ -105,7 +108,7 @@
     assert loop_token_2 is loop_token
     assert loop_tokens == [loop_token]
     assert len(cpu.seen) == 0
-    assert staticdata.globaldata.loopnumbering == 2    
+    assert staticdata.globaldata.loopnumbering == 3
 
 
 def test_resume_guard_counters():

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_exception.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_exception.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_exception.py	Thu Oct 21 21:31:17 2010
@@ -516,10 +516,9 @@
         assert res == -1
         self.check_tree_loop_count(2)      # the loop and the entry path
         # we get:
-        #    ENTER    - compile the new loop
-        #    ENTER    - compile the entry bridge
+        #    ENTER    - compile the new loop and the entry bridge
         #    ENTER    - compile the leaving path (raising MyError)
-        self.check_enter_count(3)
+        self.check_enter_count(2)
 
 
     def test_bridge_from_interpreter_exc_2(self):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitdriver.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitdriver.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitdriver.py	Thu Oct 21 21:31:17 2010
@@ -42,7 +42,7 @@
         # at the generated machine code
         self.check_loop_count(5)
         self.check_tree_loop_count(4)    # 2 x loop, 2 x enter bridge
-        self.check_enter_count(7)
+        self.check_enter_count(5)
 
     def test_inline(self):
         # this is not an example of reasonable code: loop1() is unrolled


From trundle at codespeak.net  Fri Oct 22 00:00:17 2010
From: trundle at codespeak.net (trundle at codespeak.net)
Date: Fri, 22 Oct 2010 00:00:17 +0200 (CEST)
Subject: [pypy-svn] r78188 - pypy/branch/fast-forward/pypy/module/exceptions
Message-ID: <20101021220017.7CCB1282BDC@codespeak.net>

Author: trundle
Date: Fri Oct 22 00:00:15 2010
New Revision: 78188

Modified:
   pypy/branch/fast-forward/pypy/module/exceptions/__init__.py
   pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py
Log:
Add BufferError, make GeneratorExit inherit from BaseException and update
exception hierarchy in docstring.

test_pep352 passes now.


Modified: pypy/branch/fast-forward/pypy/module/exceptions/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/exceptions/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/exceptions/__init__.py	Fri Oct 22 00:00:15 2010
@@ -9,6 +9,7 @@
         'AssertionError' : 'interp_exceptions.W_AssertionError',
         'AttributeError' : 'interp_exceptions.W_AttributeError',
         'BaseException' : 'interp_exceptions.W_BaseException',
+        'BufferError' : 'interp_exceptions.W_BufferError',
         'BytesWarning'  : 'interp_exceptions.W_BytesWarning',
         'DeprecationWarning' : 'interp_exceptions.W_DeprecationWarning',
         'EOFError' : 'interp_exceptions.W_EOFError',

Modified: pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py	(original)
+++ pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py	Fri Oct 22 00:00:15 2010
@@ -23,10 +23,11 @@
 BaseException
  +-- SystemExit
  +-- KeyboardInterrupt
+ +-- GeneratorExit
  +-- Exception
-      +-- GeneratorExit
       +-- StopIteration
       +-- StandardError
+      |    +-- BufferError
       |    +-- ArithmeticError
       |    |    +-- FloatingPointError
       |    |    +-- OverflowError
@@ -55,20 +56,20 @@
       |    +-- SystemError
       |    +-- TypeError
       |    +-- ValueError
-      |    |    +-- UnicodeError
-      |    |         +-- UnicodeDecodeError
-      |    |         +-- UnicodeEncodeError
-      |    |         +-- UnicodeTranslateError
+      |         +-- UnicodeError
+      |              +-- UnicodeDecodeError
+      |              +-- UnicodeEncodeError
+      |              +-- UnicodeTranslateError
       +-- Warning
-           +-- BytesWarning
            +-- DeprecationWarning
            +-- PendingDeprecationWarning
            +-- RuntimeWarning
            +-- SyntaxWarning
            +-- UserWarning
            +-- FutureWarning
-           +-- ImportWarning
-           +-- UnicodeWarning
+	   +-- ImportWarning
+	   +-- UnicodeWarning
+	   +-- BytesWarning
 """
 
 from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root
@@ -262,12 +263,15 @@
 W_Exception = _new_exception('Exception', W_BaseException,
                          """Common base class for all non-exit exceptions.""")
 
-W_GeneratorExit = _new_exception('GeneratorExit', W_Exception,
+W_GeneratorExit = _new_exception('GeneratorExit', W_BaseException,
                           """Request that a generator exit.""")
 
 W_StandardError = _new_exception('StandardError', W_Exception,
                          """Base class for all standard Python exceptions.""")
 
+W_BufferError = _new_exception('BufferError', W_StandardError,
+                         """Buffer error.""")
+
 W_ValueError = _new_exception('ValueError', W_StandardError,
                          """Inappropriate argument value (of correct type).""")
 


From afa at codespeak.net  Fri Oct 22 09:57:57 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 22 Oct 2010 09:57:57 +0200 (CEST)
Subject: [pypy-svn] r78189 - pypy/branch/fast-forward/pypy/module/exceptions
Message-ID: <20101022075757.C057D36E0CB@codespeak.net>

Author: afa
Date: Fri Oct 22 09:57:54 2010
New Revision: 78189

Modified:
   pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py
Log:
Untabify


Modified: pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py	(original)
+++ pypy/branch/fast-forward/pypy/module/exceptions/interp_exceptions.py	Fri Oct 22 09:57:54 2010
@@ -67,9 +67,9 @@
            +-- SyntaxWarning
            +-- UserWarning
            +-- FutureWarning
-	   +-- ImportWarning
-	   +-- UnicodeWarning
-	   +-- BytesWarning
+           +-- ImportWarning
+           +-- UnicodeWarning
+           +-- BytesWarning
 """
 
 from pypy.interpreter.baseobjspace import ObjSpace, Wrappable, W_Root


From dan at codespeak.net  Fri Oct 22 10:24:52 2010
From: dan at codespeak.net (dan at codespeak.net)
Date: Fri, 22 Oct 2010 10:24:52 +0200 (CEST)
Subject: [pypy-svn] r78190 - pypy/branch/ootype-virtualrefs
Message-ID: <20101022082452.4D064282BDC@codespeak.net>

Author: dan
Date: Fri Oct 22 10:24:50 2010
New Revision: 78190

Added:
   pypy/branch/ootype-virtualrefs/   (props changed)
      - copied from r78189, pypy/trunk/
Log:
Hopefully this branch will lead to fixing ootype translation.



From dan at codespeak.net  Fri Oct 22 10:32:43 2010
From: dan at codespeak.net (dan at codespeak.net)
Date: Fri, 22 Oct 2010 10:32:43 +0200 (CEST)
Subject: [pypy-svn] r78191 - in pypy/branch/ootype-virtualrefs/pypy: rlib
	translator/goal
Message-ID: <20101022083243.0C7BD282BDC@codespeak.net>

Author: dan
Date: Fri Oct 22 10:32:41 2010
New Revision: 78191

Modified:
   pypy/branch/ootype-virtualrefs/pypy/rlib/_jit_vref.py
   pypy/branch/ootype-virtualrefs/pypy/translator/goal/targetpypystandalone.py
Log:
SomeVRef now has a more correct Repr on ootypesystem

Modified: pypy/branch/ootype-virtualrefs/pypy/rlib/_jit_vref.py
==============================================================================
--- pypy/branch/ootype-virtualrefs/pypy/rlib/_jit_vref.py	(original)
+++ pypy/branch/ootype-virtualrefs/pypy/rlib/_jit_vref.py	Fri Oct 22 10:32:41 2010
@@ -24,7 +24,10 @@
         return self.s_instance
 
     def rtyper_makerepr(self, rtyper):
-        return vrefrepr
+        if rtyper.type_system.name == 'lltypesystem':
+            return vrefrepr
+        elif rtyper.type_system.name == 'ootypesystem':
+            return oovrefrepr
 
     def rtyper_makekey(self):
         return self.__class__,
@@ -54,4 +57,10 @@
                              " prebuilt virtual_ref")
         return lltype.nullptr(OBJECTPTR.TO)
 
+from pypy.rpython.ootypesystem.rclass import OBJECT
+
+class OOVRefRepr(VRefRepr):
+    lowleveltype = OBJECT
+
 vrefrepr = VRefRepr()
+oovrefrepr = OOVRefRepr()

Modified: pypy/branch/ootype-virtualrefs/pypy/translator/goal/targetpypystandalone.py
==============================================================================
--- pypy/branch/ootype-virtualrefs/pypy/translator/goal/targetpypystandalone.py	(original)
+++ pypy/branch/ootype-virtualrefs/pypy/translator/goal/targetpypystandalone.py	Fri Oct 22 10:32:41 2010
@@ -105,7 +105,6 @@
             print 'Translation to cli and jvm is known to be broken at the moment'
             print 'Please try the "cli-jit" branch at:'
             print 'http://codespeak.net/svn/pypy/branch/cli-jit/'
-            sys.exit(1)
 
         self.translateconfig = translateconfig
         # set up the objspace optimizations based on the --opt argument


From dan at codespeak.net  Fri Oct 22 10:46:09 2010
From: dan at codespeak.net (dan at codespeak.net)
Date: Fri, 22 Oct 2010 10:46:09 +0200 (CEST)
Subject: [pypy-svn] r78192 - pypy/branch/ootype-virtualrefs/pypy/rlib/test
Message-ID: <20101022084609.A84F7282BDC@codespeak.net>

Author: dan
Date: Fri Oct 22 10:46:07 2010
New Revision: 78192

Modified:
   pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py
Log:
Made tests run on both lltypesystem and ootypesystem.

Modified: pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py
==============================================================================
--- pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py	(original)
+++ pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py	Fri Oct 22 10:46:07 2010
@@ -8,6 +8,7 @@
 from pypy.rpython.lltypesystem.rclass import OBJECTPTR
 from pypy.rpython.lltypesystem import lltype
 
+from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 
 class X(object):
     pass
@@ -79,37 +80,44 @@
     assert s.s_instance.can_be_None
     assert s.s_instance.classdef == a.bookkeeper.getuniqueclassdef(X)
 
-def test_rtype_1():
-    def f():
-        return virtual_ref(X())
-    x = interpret(f, [])
-    assert lltype.typeOf(x) == OBJECTPTR
+class BaseTestVRef(BaseRtypingTest):
+    def test_rtype_1(self):
+        def f():
+            return virtual_ref(X())
+        x = self.interpret(f, [])
+        self.is_of_instance_type(x)
 
-def test_rtype_2():
-    def f():
-        x1 = X()
-        vref = virtual_ref(x1)
-        x2 = vref()
-        virtual_ref_finish(x2)
-        return x2
-    x = interpret(f, [])
-    assert lltype.castable(OBJECTPTR, lltype.typeOf(x)) > 0
+    def test_rtype_2(self):
+        def f():
+            x1 = X()
+            vref = virtual_ref(x1)
+            x2 = vref()
+            virtual_ref_finish(x2)
+            return x2
+        x = self.interpret(f, [])
+        assert lltype.castable(OBJECTPTR, lltype.typeOf(x)) > 0
+
+    def test_rtype_3(self):
+        def f(n):
+            if n > 0:
+                return virtual_ref(Y())
+            else:
+                return non_virtual_ref(Z())
+        x = self.interpret(f, [-5])
+        assert lltype.typeOf(x) == OBJECTPTR
+
+    def test_rtype_4(self):
+        def f(n):
+            if n > 0:
+                return virtual_ref(X())
+            else:
+                return vref_None
+        x = self.interpret(f, [-5])
+        assert lltype.typeOf(x) == OBJECTPTR
+        assert not x
 
-def test_rtype_3():
-    def f(n):
-        if n > 0:
-            return virtual_ref(Y())
-        else:
-            return non_virtual_ref(Z())
-    x = interpret(f, [-5])
-    assert lltype.typeOf(x) == OBJECTPTR
+class TestLLtype(BaseTestVRef, LLRtypeMixin):
+    pass
 
-def test_rtype_4():
-    def f(n):
-        if n > 0:
-            return virtual_ref(X())
-        else:
-            return vref_None
-    x = interpret(f, [-5])
-    assert lltype.typeOf(x) == OBJECTPTR
-    assert not x
+class TestOOtype(BaseTestVRef, OORtypeMixin):
+    pass


From dan at codespeak.net  Fri Oct 22 11:45:43 2010
From: dan at codespeak.net (dan at codespeak.net)
Date: Fri, 22 Oct 2010 11:45:43 +0200 (CEST)
Subject: [pypy-svn] r78194 - pypy/branch/ootype-virtualrefs/pypy/rlib/test
Message-ID: <20101022094543.5B813282BDC@codespeak.net>

Author: dan
Date: Fri Oct 22 11:45:41 2010
New Revision: 78194

Modified:
   pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py
Log:
Fixed the tests a bit more.

Modified: pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py
==============================================================================
--- pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py	(original)
+++ pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py	Fri Oct 22 11:45:41 2010
@@ -85,7 +85,7 @@
         def f():
             return virtual_ref(X())
         x = self.interpret(f, [])
-        self.is_of_instance_type(x)
+        assert self.is_of_instance_type(x)
 
     def test_rtype_2(self):
         def f():
@@ -104,7 +104,7 @@
             else:
                 return non_virtual_ref(Z())
         x = self.interpret(f, [-5])
-        assert lltype.typeOf(x) == OBJECTPTR
+        assert self.is_of_instance_type(x)
 
     def test_rtype_4(self):
         def f(n):
@@ -113,7 +113,7 @@
             else:
                 return vref_None
         x = self.interpret(f, [-5])
-        assert lltype.typeOf(x) == OBJECTPTR
+        assert self.is_of_instance_type(x)
         assert not x
 
 class TestLLtype(BaseTestVRef, LLRtypeMixin):


From arigo at codespeak.net  Fri Oct 22 13:09:22 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 13:09:22 +0200 (CEST)
Subject: [pypy-svn] r78195 - in pypy/trunk/pypy/module: array pypyjit/test
Message-ID: <20101022110922.EE9D5282BDC@codespeak.net>

Author: arigo
Date: Fri Oct 22 13:09:20 2010
New Revision: 78195

Modified:
   pypy/trunk/pypy/module/array/interp_array.py
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
Log:
On 64-bit machines, consider 'I' to fit in a Signed too.


Modified: pypy/trunk/pypy/module/array/interp_array.py
==============================================================================
--- pypy/trunk/pypy/module/array/interp_array.py	(original)
+++ pypy/trunk/pypy/module/array/interp_array.py	Fri Oct 22 13:09:20 2010
@@ -323,7 +323,7 @@
         idx, stop, step = space.decode_index(w_idx, self.len)
         assert step == 0
         item = self.buffer[idx]
-        if mytype.typecode in 'bBhHil':
+        if mytype.canoverflow:    # if it fits in a Signed
             item = rffi.cast(lltype.Signed, item)
         elif mytype.typecode == 'f':
             item = float(item)

Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Fri Oct 22 13:09:20 2010
@@ -837,6 +837,9 @@
         # XXX this test is way too imprecise in what it is actually testing
         # it should count the number of guards instead
         for tc, maxops in zip('ilILd', (67, 67, 69, 69, 61)):
+            print
+            print '='*65
+            print '='*20, 'running test for tc=%r' % (tc,), '='*20
             res = 73574560
             if tc in 'IL':
                 res = long(res)


From arigo at codespeak.net  Fri Oct 22 13:27:37 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 13:27:37 +0200 (CEST)
Subject: [pypy-svn] r78196 -
	pypy/branch/cleanup-dict-impl/pypy/module/pypyjit/test
Message-ID: <20101022112737.777D9282BDC@codespeak.net>

Author: arigo
Date: Fri Oct 22 13:27:36 2010
New Revision: 78196

Modified:
   pypy/branch/cleanup-dict-impl/pypy/module/pypyjit/test/test_pypy_c.py
Log:
Dump the loop as loaded from the log file, instead of dumping it
after being parsed and reformatted (the descrs are lost).


Modified: pypy/branch/cleanup-dict-impl/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/module/pypyjit/test/test_pypy_c.py	Fri Oct 22 13:27:36 2010
@@ -136,9 +136,10 @@
         assert opslogfile.check()
         log = logparser.parse_log_file(str(opslogfile))
         parts = logparser.extract_category(log, 'jit-log-opt-')
+        self.rawloops = [part for part in parts
+                         if not from_entry_bridge(part, parts)]
         # skip entry bridges, they can contain random things
-        self.loops = [parse(part, no_namespace=True) for part in parts
-                          if not from_entry_bridge(part, parts)]
+        self.loops = [parse(part, no_namespace=True) for part in self.rawloops]
         self.sliced_loops = [] # contains all bytecodes of all loops
         self.total_ops = 0
         for loop in self.loops:
@@ -162,12 +163,11 @@
         return [ops for ops in self.sliced_loops if ops.bytecode == name]
 
     def print_loops(self):
-        for loop in self.loops:
+        for rawloop in self.rawloops:
             print
             print '@' * 79
             print
-            for op in loop.operations:
-                print op
+            print rawloop.rstrip()
         print
         print '@' * 79
 


From cfbolz at codespeak.net  Fri Oct 22 13:50:51 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 22 Oct 2010 13:50:51 +0200 (CEST)
Subject: [pypy-svn] r78197 - pypy/extradoc/talk/pepm2011
Message-ID: <20101022115051.AE9E4282BDC@codespeak.net>

Author: cfbolz
Date: Fri Oct 22 13:50:50 2010
New Revision: 78197

Modified:
   pypy/extradoc/talk/pepm2011/paper.bib
Log:
capitalization of things, use dls paper for spur


Modified: pypy/extradoc/talk/pepm2011/paper.bib
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.bib	(original)
+++ pypy/extradoc/talk/pepm2011/paper.bib	Fri Oct 22 13:50:50 2010
@@ -1,6 +1,6 @@
 ?
 @article{choi_escape_1999,
-	title = {Escape analysis for Java},
+	title = {Escape analysis for {J}ava},
 	volume = {34},
 	url = {http://portal.acm.org/citation.cfm?id=320386},
 	doi = {10.1145/320385.320386},
@@ -59,22 +59,8 @@
 	pages = {328--347}
 },
 
- at techreport{michael_bebenita_spur:_2010,
-	title = {{SPUR:} A {Trace-Based} {JIT} Compiler for {CIL}},
-	abstract = {Tracing just-in-time compilers {(TJITs)} determine frequently executed traces (hot paths and loops) in running programs and focus their optimization effort by emitting optimized machine code specialized to these traces. Prior work has established this strategy to be especially beneficial for dynamic languages such as {JavaScript,} where the {TJIT} interfaces with the interpreter and produces machine code from the {JavaScript} trace.
-
-This direct coupling with a {JavaScript} interpreter makes it difficult to harness the power of a {TJIT} for other components that are not written in {JavaScript,} e.g., the {DOM} implementation or the layout engine inside a browser. Furthermore, if a {TJIT} is tied to a particular high-level language interpreter, it is difficult to reuse it for other input languages as the optimizations are likely targeted at specific idioms of the source language.
-
-To address these issues, we designed and implemented a {TJIT} for Microsoft?s Common Intermediate Language {CIL} (the target language of C\#, {VisualBasic,} F\#, and many other languages). Working on {CIL} enables {TJIT} optimizations for any program compiled to this platform. In addition, to validate that the performance gains of a {TJIT} for {JavaScript} do not depend on specific idioms of {JavaScript} that are lost in the translation to {CIL,} we provide a performance evaluation of our {JavaScript} runtime which translates {JavaScript} to {CIL} and then runs on top of our {CIL} {TJIT.}},
-	number = {{MSR-TR-2010-27}},
-	institution = {Microsoft Research},
-	author = {Michael Bebenita and Florian Brandner and Manuel Fahndrich and Francesco Logozzo and Wolfram Schulte and Nikolai Tillmann and Herman Venter},
-	month = mar,
-	year = {2010},
-},
-
 @article{bruno_blanchet_escape_2003,
-	title = {Escape analysis for Java: Theory and practice},
+	title = {Escape analysis for {J}ava: Theory and practice},
 	volume = {25},
 	shorttitle = {Escape analysis for {Java{\textless}sup{\textgreater}TM{\textless}/sup{\textgreater}}},
 	url = {http://portal.acm.org/citation.cfm?id=945885.945886},
@@ -89,7 +75,7 @@
 
 @incollection{mazur_practical_2001,
 	series = {Lecture Notes in Computer Science},
-	title = {Practical Aspects for a Working Compile Time Garbage Collection System for Mercury},
+	title = {Practical Aspects for a Working Compile Time Garbage Collection System for {M}ercury},
 	volume = {2237},
 	url = {http://dx.doi.org/10.1007/3-540-45635-X_15},
 	abstract = {Compile-time garbage collection {(CTGC)} is still a very uncommon feature within compilers. In previous work we have developed a compile-time structure reuse system for Mercury, a logic programming language. This system indicates which datastructures can safely be reused at run-time. As preliminary experiments were promising, we have continued this work and have now a working and well performing near-to-ship {CTGC-system} built into the Melbourne Mercury Compiler {(MMC).} In this paper we present the multiple design decisions leading to this system, we report the results of using {CTGC} for a set of benchmarks, including a real-world program, and finally we discuss further possible improvements. Benchmarks show substantial memory savings and a noticeable reduction in execution time.},
@@ -114,7 +100,7 @@
 },
 
 @phdthesis{cuni_high_2010,
-	title = {High performance implementation of Python for {CLI/.NET} with {JIT} compiler generation for dynamic languages.},
+	title = {High performance implementation of {P}ython for {CLI/.NET} with {JIT} compiler generation for dynamic languages.},
 	school = {Dipartimento di Informatica e Scienze {dell'Informazione,} University of Genova},
 	author = {Antonio Cuni},
 	year = {2010},
@@ -123,7 +109,7 @@
 
 @inproceedings{carl_friedrich_bolz_towards_2010,
 	address = {Hagenberg, Austria},
-	title = {Towards a Jitting {VM} for Prolog execution},
+	title = {Towards a Jitting {VM} for {P}rolog execution},
 	isbn = {978-1-4503-0132-9},
 	url = {http://portal.acm.org/citation.cfm?id=1836102},
 	doi = {10.1145/1836089.1836102},
@@ -135,6 +121,35 @@
 	pages = {99--108}
 },
 
+ at inproceedings{mogensen_constructor_1993,
+	address = {Copenhagen, Denmark},
+	title = {Constructor specialization},
+	isbn = {0-89791-594-1},
+	url = {http://portal.acm.org/citation.cfm?id=154633},
+	doi = {10.1145/154630.154633},
+	abstract = {In the section on ?challenging problems? in the proceedings from the first international workshop on partial evaluation and mixed computation {[BEJ88]} a question is stated: {?Can} {PE} be used to generate new specialized data types, in a way analogous to generating specialized functions?. Since then little has been done to address this problem. In {[Lau89],} new types are indeed generated, but they are all simpler versions of the types in the original program. It is, e.g. not possible to have types with more constructors than the types in the original program. I propose to alleviate this by means of constructor specialization. Constructors are specialized with respect to the static parts of their arguments, just like residual functions. I show how this is done and argue that it makes it possible to get good results from partial evaluation in cases where the traditional methods fail to produce satisfactory results. The discussion is centered around a small subset of Standard {ML,} but the idea applies equally well to other languages having user defined constructors, e.g. Haskell and Prolog.},
+	booktitle = {Proceedings of the 1993 {ACM} {SIGPLAN} symposium on Partial evaluation and semantics-based program manipulation},
+	publisher = {{ACM}},
+	author = {Torben ? Mogensen},
+	year = {1993},
+	pages = {22--32}
+},
+
+ at inproceedings{bebenita_spur:_2010,
+	address = {{Reno/Tahoe,} Nevada, {USA}},
+	title = {{SPUR:} a trace-based {JIT} compiler for {CIL}},
+	isbn = {978-1-4503-0203-6},
+	shorttitle = {{SPUR}},
+	url = {http://portal.acm.org/citation.cfm?id=1869459.1869517&coll=GUIDE&dl=GUIDE&type=series&idx=SERIES318&part=series&WantType=Proceedings&title=OOPSLA%2FSPLASH&CFID=106280261&CFTOKEN=29377718},
+	doi = {10.1145/1869459.1869517},
+	abstract = {Tracing just-in-time compilers {(TJITs)} determine frequently executed traces (hot paths and loops) in running programs and focus their optimization effort by emitting optimized machine code specialized to these traces. Prior work has established this strategy to be especially beneficial for dynamic languages such as {JavaScript,} where the {TJIT} interfaces with the interpreter and produces machine code from the {JavaScript} trace.},
+	booktitle = {Proceedings of the {ACM} international conference on Object oriented programming systems languages and applications},
+	publisher = {{ACM}},
+	author = {Michael Bebenita and Florian Brandner and Manuel Fahndrich and Francesco Logozzo and Wolfram Schulte and Nikolai Tillmann and Herman Venter},
+	year = {2010},
+	pages = {708--725}
+},
+
 @inproceedings{chang_tracing_2009,
 	address = {Washington, {DC,} {USA}},
 	title = {Tracing for Web 3.0: Trace Compilation for the Next Generation Web Applications},
@@ -166,7 +181,7 @@
 },
 
 @article{futamura_partial_1999,
-	title = {Partial Evaluation of Computation Process - An Approach to a {Compiler-Compiler}},
+	title = {Partial Evaluation of Computation Process - An Approach to a Compiler-Compiler},
 	volume = {12},
 	url = {http://citeseer.ist.psu.edu/futamura99partial.html},
 	number = {4},
@@ -201,7 +216,7 @@
 },
 
 @article{blanchet_escape_1999,
-	title = {Escape Analysis for Object Oriented Languages. Application to Java},
+	title = {Escape Analysis for Object Oriented Languages. Application to {J}ava},
 	url = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.47.2277},
 	doi = {10.1.1.47.2277},
 	journal = {In Proceedings of the 14th Annual Conference on {Object-Oriented} Programming Systems, Languages and Applications},
@@ -211,7 +226,7 @@
 },
 
 @article{georges_statistically_2007,
-	title = {Statistically rigorous java performance evaluation},
+	title = {Statistically rigorous {J}ava performance evaluation},
 	volume = {42},
 	url = {http://portal.acm.org/citation.cfm?id=1297105.1297033},
 	doi = {10.1145/1297105.1297033},
@@ -232,7 +247,7 @@
 },
 
 @inproceedings{andreas_gal_trace-based_2009,
-	title = {Trace-based {Just-in-Time} Type Specialization for Dynamic Languages},
+	title = {Trace-based Just-in-Time Type Specialization for Dynamic Languages},
 	booktitle = {{PLDI}},
 	author = {Andreas Gal and Brendan Eich and Mike Shaver and David Anderson and Blake Kaplan and Graydon Hoare and David Mandelin and Boris Zbarsky and Jason Orendorff and Michael Bebenita and Mason Chang and Michael Franz and Edwin Smith and Rick Reitmaier and Mohammad Haghighat},
 	year = {2009},
@@ -254,7 +269,7 @@
 },
 
 @techreport{mason_chang_efficient_2007,
-	title = {Efficient {Just-In-Time} Execution of Dynamically Typed Languages
+	title = {Efficient Just-In-Time Execution of Dynamically Typed Languages
 Via Code Specialization Using Precise Runtime Type Inference},
 	abstract = {Dynamically typed languages such as {JavaScript} present a challenge to just-in-time compilers. In contrast to statically typed languages such as {JVML,} in which there are specific opcodes for common operations on primitive types (such as iadd for integer addition), all operations in dynamically typed language such as {JavaScript} are late-bound. Often enough, types cannot be inferred with certainty ahead of execution. As a result, just-in-time compilers for dynamically typed languages have tended to perform worse than their statically-typed counterparts. We present a new approach to compiling dynamically typed languages in which code traces observed during execution are dynamically specialized for each actually observed run-time type. For most benchmark programs, our prototype {JavaScript} virtual machine outperforms every other {JavaScript} platform known to us.},
 	number = {{ICS-TR-07-10}},
@@ -316,7 +331,7 @@
 
 @inproceedings{rigo_representation-based_2004,
 	address = {Verona, Italy},
-	title = {Representation-based just-in-time specialization and the Psyco prototype for Python},
+	title = {Representation-based just-in-time specialization and the {P}syco prototype for {P}ython},
 	isbn = {1-58113-835-0},
 	url = {http://portal.acm.org/citation.cfm?id=1014010},
 	doi = {10.1145/1014007.1014010},
@@ -329,7 +344,7 @@
 },
 
 @incollection{carl_friedrich_bolz_back_2008,
-	title = {Back to the Future in One Week ? Implementing a Smalltalk {VM} in {PyPy}},
+	title = {Back to the Future in One Week ? Implementing a {S}malltalk {VM} in {PyPy}},
 	url = {http://dx.doi.org/10.1007/978-3-540-89275-5_7},
 	abstract = {We report on our experiences with the Spy project, including implementation details and benchmark results. Spy is a re-implementation of the Squeak (i.e. Smalltalk-80) {VM} using the {PyPy} toolchain. The {PyPy} project allows code written in {RPython,} a subset of Python, to be translated
 to a multitude of different backends and architectures. During the translation, many aspects of the implementation can be
@@ -342,13 +357,3 @@
 	year = {2008},
 	pages = {123--139}
 }
- at inproceedings{Mogensen:PEPM93,
-   author =    {Torben Mogensen},
-   title =     {Constructor Specialization},
-   booktitle = {Proceedings of PEPM'93, the ACM Sigplan Symposium on Partial 
-                Evaluation and Semantics-Based Program Manipulation},
-   year =      1993,
-   month =     {June},
-   publisher = {ACM Press},
-   pages =     {22--32},
-}
\ No newline at end of file


From cfbolz at codespeak.net  Fri Oct 22 13:56:31 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 22 Oct 2010 13:56:31 +0200 (CEST)
Subject: [pypy-svn] r78198 - pypy/extradoc/talk/pepm2011
Message-ID: <20101022115631.8B68B282BDC@codespeak.net>

Author: cfbolz
Date: Fri Oct 22 13:56:29 2010
New Revision: 78198

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/math.lyx
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
various fixes, as well as two bugs in the math


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/math.lyx
==============================================================================
--- pypy/extradoc/talk/pepm2011/math.lyx	(original)
+++ pypy/extradoc/talk/pepm2011/math.lyx	Fri Oct 22 13:56:29 2010
@@ -552,7 +552,7 @@
 \begin_inset Text
 
 \begin_layout Plain Layout
-\begin_inset Formula ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathtt{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$
+\begin_inset Formula ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathtt{new}\left(\mathrm{type}\left(S\left(v^{*}\right)\right)\right)\right\rangle ::ops,S^{\prime}}}$
 \end_inset
 
 
@@ -575,7 +575,7 @@
 \begin_inset Text
 
 \begin_layout Plain Layout
-\begin_inset Formula ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathtt{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathtt{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$
+\begin_inset Formula ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathtt{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathtt{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime\prime}}}$
 \end_inset
 
 

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 22 13:56:29 2010
@@ -235,7 +235,7 @@
 the translation to C, PyPy's tools can generate a tracing just-in-time compiler for the
 language that the interpreter is implementing. This process is mostly
 automatic; it only needs to be guided by the language implementer using a small number of
-source-code hints. Mostly-automatically generating a JIT compiler has many advantages
+source-code hints in the interpreter. Mostly-automatically generating a JIT compiler has many advantages
 over writing one manually, an error-prone and tedious process.
 By construction, the generated JIT has the same semantics as the interpreter.
 Optimizations can be shared between different languages implemented with PyPy.
@@ -289,7 +289,7 @@
 during tracing, because if
 it isn't, the rest of the trace would not be valid.
 
-When generating machine code, every guard is be turned into a quick check to
+When generating machine code, every guard is turned into a quick check to
 see whether the assumption still holds. When such a guard is hit during the
 execution of the machine code and the assumption does not hold, the execution of
 the machine code is stopped, and interpreter continues to run from that point
@@ -297,9 +297,9 @@
 loop end condition also takes the form of a guard.
 
 If one specific guard fails often enough, the tracing JIT will generate a new
-trace that starts exactly at the position of the failing guard. The existing
-assembler is patched to jump to the new trace when the guard fails
-\cite{andreas_gal_incremental_2006}.  This approach guarantees that all the
+trace that starts exactly at the position of the failing guard
+\cite{andreas_gal_incremental_2006}. The existing assembler is patched to jump
+to the new trace when the guard fails.  This approach guarantees that all the
 hot paths in the program will eventually be traced and compiled into efficient
 code.
 
@@ -377,7 +377,7 @@
 implement the numeric tower needs two method calls per arithmetic operation,
 which is costly due to the method dispatch.
 
-Let us now consider a simple interpreter function \lstinline{f} that uses the
+Let us now consider a simple ``interpreter'' function \lstinline{f} that uses the
 object model (see the bottom of Figure~\ref{fig:objmodel}).
 The loop in \lstinline{f} iterates \lstinline{y} times, and computes something in the process.
 Simply running this function is slow, because there are lots of virtual method
@@ -460,7 +460,8 @@
 corresponding to the stack level of the function that contains the traced
 operation. The trace is in single-assignment form, meaning that each variable is
 assigned a value exactly once. The arguments $p_0$ and $p_1$ of the loop correspond
-to the live variables \lstinline{y} and \lstinline{res} in the original function.
+to the live variables \lstinline{y} and \lstinline{res} in the while-loop of
+the original function.
 
 The operations in the trace correspond to the operations in the RPython program
 in Figure~\ref{fig:objmodel}:
@@ -473,11 +474,13 @@
     (inlined) method call and is followed by the trace of the called method.
     \item \lstinline{int_add} and \lstinline{int_gt} are integer addition and
     comparison (``greater than''), respectively.
+    \item \lstinline{guard_true} checks that a boolean is true.
 \end{itemize}
 
 The method calls in the trace are always preceded by a \lstinline{guard_class}
 operation, to check that the class of the receiver is the same as the one that
-was observed during tracing.\footnote{\lstinline{guard_class} performs a precise
+was observed during tracing.\footnote{\lstinline{guard_class} XXX lstinline too large in footnotes
+performs a precise
 class check, not checking for subclasses.} These guards make the trace specific
 to the situation where \lstinline{y} is really a \lstinline{BoxedInteger}. When
 the trace is turned into machine code and afterwards executed with
@@ -575,16 +578,16 @@
 The final trace after optimization can be seen in Figure~\ref{fig:step1} (the
 line numbers are the lines of the unoptimized trace where the operation originates).
 
-To optimize the trace, it is traversed from beginning to end while an output
+To optimize the trace, it is traversed from beginning to end and an output
 trace is produced. Every operation in the input trace is either
-removed or put into the output trace. Sometimes new operations need to be
+removed or copied into the output trace. Sometimes new operations need to be
 produced as well. The optimizer can only remove operations that manipulate
 objects that have been allocated within the trace, while all other operations are copied to the
 output trace unchanged.
 
 Looking at the example trace of Figure~\ref{fig:unopt-trace}, the operations
 in lines 1--9 are manipulating objects which existed before the trace and that
-are passed in as arguments: therefore the optimizer just puts them into the
+are passed in as arguments: therefore the optimizer just copies them into the
 output trace.
 
 The following operations (lines 10--17) are more interesting:
@@ -608,11 +611,11 @@
 stored in the fields of the allocated object come from.
 
 In the snippet above, the two \lstinline{new} operations are removed and two
-static objects are constructed. The \lstinline{set} operations manipulate a
-static object, therefore they can be removed as well; their effect is
+static objects are constructed. The \lstinline{set} operations manipulate
+static objects, therefore they can be removed as well; their effect is
 remembered in the static objects.
 
-The static object associated with $p_{5}$ would store the knowledge that it is a
+After the operations the static object associated with $p_{5}$ would store the knowledge that it is a
 \lstinline{BoxedInteger} whose \lstinline{intval} field contains $i_{4}$; the
 one associated with $p_{6}$ would store that it is a \lstinline{BoxedInteger}
 whose \lstinline{intval} field contains the constant -100.
@@ -628,8 +631,8 @@
 $i_{9}$ = int_add($i_{7}$, $i_{8}$)
 \end{lstlisting}
 
-The \lstinline{guard_class} operations can be removed, since their argument is a
-static object with the matching type \lstinline{BoxedInteger}. The
+The \lstinline{guard_class} operations can be removed, since their arguments are
+static objects with the matching type \lstinline{BoxedInteger}. The
 \lstinline{get} operations can be removed as well, because each of them reads a
 field out of a static object. The results of the get operation are replaced with
 what the static object stores in these fields: all the occurences of $i_{7}$ and $i_{8}$ in the trace are just
@@ -642,11 +645,11 @@
 
 The rest of the trace from Figure~\ref{fig:unopt-trace} is optimized in a
 similar vein. The operations in lines 27--35 produce two more static objects and
-are removed. Those in line 36--39 are just put into the output trace because they
+are removed. Those in line 36--39 are just copied into the output trace because they
 manipulate objects that are allocated before the trace. Lines 40--42 are removed
-because they operate on a static object. Line 43 is put into the output trace.
+because they operate on a static object. Line 43 is copied into the output trace.
 Lines 44--46 produce a new static object and are removed, lines 48--51 manipulate
-that static object and are removed as well. Lines 52--54 are put into the output
+that static object and are removed as well. Lines 52--54 are copied into the output
 trace.
 
 The last operation (line 55) is an interesting case. It is the \lstinline{jump}
@@ -759,16 +762,16 @@
 
 In this section we want to give a formal description of the semantics of the
 traces and of the optimizer and liken the optimization to partial evaluation.
-We focus on the operations for manipulating dynamically allocated objects,
+We focus on the operations for manipulating heap allocated objects,
 as those are the only ones that are actually optimized. We also consider only
-objects with two fields in this section, generalizing to arbitrary many fields
+objects with two fields $L$ and $R$ in this section, generalizing to arbitrary many fields
 is straightforward.
 
 Traces are lists of operations. The operations considered here are
 \lstinline{new}, \lstinline{get}, \lstinline{set} and \lstinline{guard_class}.
 The values of all
 variables are locations (\ie pointers). Locations are mapped to objects, which
-are represented by triples of a type $T$, and two locations that represent the
+are represented by triples $(T,l_1,l_2)$ of a type $T$, and two locations that represent the
 fields of the object. When a new object is created, the fields are initialized
 to null, but we require that they are initialized to a real
 location before being read, otherwise the trace is malformed (this condition is
@@ -776,7 +779,7 @@
 
 We use some abbreviations when dealing with object triples. To read the type of
 an object, $\mathrm{type}((T,l_1,l_2))=T$ is used. Reading a field $F$ from an
-object is written $(T,l_1,l_2)_F$ which either returns $l_1$ if $F=L$ or $l_2$
+object is written $(T,l_1,l_2)_F$ which either is $l_1$ if $F=L$ or $l_2$
 if $F=R$. To set field $F$ to a new location $l$, we use the notation
 $(T,l_1,l_2)!_Fl$, which yields a new triple $(T,l,l_2)$ if $F=L$ or a new
 triple $(T,l_1,l)$ if $F=R$.
@@ -827,8 +830,8 @@
 \emph{guard} & ${\displaystyle \frac{E(v)\in\mathrm{dom}(S),\,\mathrm{type}(S(E(v)))=T}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\left\langle \,\right\rangle ,E,S}}$\tabularnewline[3em]
  & ${\displaystyle \frac{E(v)\notin\mathrm{dom}(S)\vee\mathrm{type}(S(E(v)))\neq T,\,\left(E(v),S\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{\mathtt{guard\_class}(v,T),E,S\overset{\mathrm{opt}}{\Longrightarrow}\mathrm{ops}::\left\langle \mathtt{guard\_class}(E\left(v\right),T)\right\rangle ,E,S^{\prime}}}$\tabularnewline[3em]
 \emph{lifting} & ${\displaystyle \frac{v^{*}\notin\mathrm{dom}(S)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle \,\right\rangle ,S}}$\tabularnewline[3em]
- & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathtt{new}\left(T\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em]
- & ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathtt{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathtt{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime}}}$\tabularnewline[3em]
+ & ${\displaystyle \frac{v^{*}\in\mathrm{dom}(S),\,\left(v^{*},S\right)\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\left(\mathrm{ops},S^{\prime}\right)}{v^{*},S\overset{\mathrm{lift}}{\Longrightarrow}\left\langle v^{*}=\mathtt{new}\left(\mathrm{type}\left(S\left(v^{*}\right)\right)\right)\right\rangle ::ops,S^{\prime}}}$\tabularnewline[3em]
+ & ${\displaystyle \frac{\left(S\left(v^{*}\right)_{L},S\setminus\left\{ v^{*}\mapsto S\left(v^{*}\right)\right\} \right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{L},S^{\prime}\right),\,\left(S\left(v^{*}\right)_{R},S^{\prime}\right)\overset{\mathrm{lift}}{\Longrightarrow}\left(\mathrm{ops}_{R},S^{\prime\prime}\right)}{v^{*},S\overset{\mathrm{liftfields}}{=\!=\!\Longrightarrow}\mathrm{ops}_{L}::ops_{R}::\left\langle \mathtt{set}\left(v^{*},L,S\left(v^{*}\right)_{L}\right),\,\mathtt{set}\left(v^{*},R,S\left(v^{*}\right)_{R}\right)\right\rangle ,S^{\prime\prime}}}$\tabularnewline[3em]
 \end{tabular}
 \end{center}
 
@@ -958,7 +961,7 @@
 \subsection{Analysis of the Algorithm}
 \label{sub:analysis}
 
-While we do not offer a formal proof of it, it should be relatively clear
+While we do not offer a formal proof of it, it can argue informally
 that the algorithm presented above is sound: it works by delaying (and
 often completely removing) some operations.  The algorithm runs in a
 single pass over the list of operations.  We can check that although
@@ -1014,7 +1017,7 @@
 To evaluate our allocation removal algorithm, we look at the effectiveness when
 used in the generated tracing JIT of PyPy's Python interpreter. This interpreter
 is a full implementation of Python 2.5 language semantics and is about 30'000
-lines of code.
+lines of RPython code.
 
 The
 benchmarks we used are small-to-medium Python programs, some synthetic
@@ -1058,7 +1061,7 @@
 CPython 2.6.6\footnote{\texttt{http://python.org}}, which uses a bytecode-based
 interpreter. Furthermore we compared against
 Psyco\cite{rigo_representation-based_2004} 1.6,
-an extension to CPython which is a
+a (hand-written) extension module to CPython which is a
 just-in-time compiler that produces machine code at run-time. It is not based
 on traces. Finally, we used two versions of PyPy's Python interpreter (revision
 77823 of SVN trunk\footnote{\texttt{http://codespeak.net/svn/pypy/trunk}}): one
@@ -1178,17 +1181,17 @@
 loop to only allocate it once, instead of every iteration. No details are given
 for this optimization. The fact that the object is still allocated and needs to
 be written to means that only the allocations are optimized away, but not the
-reads and writes out of/into the object.
+reads out of and writes into the object.
 
 SPUR, a tracing JIT for C\# seems to be able to remove allocations in a similar
 way to the approach described here, as hinted at in the technical report
-\cite{michael_bebenita_spur:_2010}. However, no details for the approach and its
+\cite{bebenita_spur:_2010}. However, no details for the approach and its
 implementation are given.
 
 Psyco \cite{rigo_representation-based_2004} is a (non-tracing) JIT for Python
 that implements a more ad-hoc version of the allocation removal described here.
 Our static objects could be related to what are called \emph{virtual} objects
-in Psyco.  It is a hand-written extension module for CPython. Historically,
+in Psyco.  Historically,
 PyPy's JIT can be seen as some successor of Psyco for a general context (one of
 the authors of this paper is the author of Psyco).
 


From david at codespeak.net  Fri Oct 22 13:59:45 2010
From: david at codespeak.net (david at codespeak.net)
Date: Fri, 22 Oct 2010 13:59:45 +0200 (CEST)
Subject: [pypy-svn] r78199 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . test
Message-ID: <20101022115945.7632C282BDC@codespeak.net>

Author: david
Date: Fri Oct 22 13:59:44 2010
New Revision: 78199

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
Log:
Add instr. to call coprocessor

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	Fri Oct 22 13:59:44 2010
@@ -1,6 +1,6 @@
 from pypy.jit.backend.arm import conditions as cond
 from pypy.jit.backend.arm import instructions
-
+# move table lookup out of generated functions
 def define_load_store_func(name, table):
     #  XXX W and P bits are not encoded yet
     n = (0x1 << 26
@@ -92,6 +92,19 @@
                         | reg_operation(rd, rn, rm, imm, s, shifttype))
     return f
 
+def define_supervisor_and_coproc(name, table):
+    n = (0x3 << 26 | (table['op1'] & 0x3F) << 20 | (table['op'] & 0x1) << 4)
+    def f(self, coproc, opc1, rt, crn, crm, opc2=0, cond=cond.AL):
+        self.write32(n
+                    | cond << 28
+                    | (opc1 & 0x7) << 21
+                    | (crn & 0xF) << 16
+                    | (rt & 0xF) << 12
+                    | (coproc & 0xF) << 8
+                    | (opc2 & 0x7) << 5
+                    | (crm & 0xF))
+    return f
+
 def imm_operation(rt, rn, imm):
     return ((rn & 0xFF) << 16
     | (rt & 0xFF) << 12
@@ -119,3 +132,6 @@
 
     for key, val in instructions.data_proc_imm.iteritems():
         define_instruction(define_data_proc_imm, key, val, target)
+
+    for key, val in instructions.supervisor_and_coproc.iteritems():
+        define_instruction(define_supervisor_and_coproc, key, val, target)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	Fri Oct 22 13:59:44 2010
@@ -52,3 +52,7 @@
     'BIC_ri': {'op': 0x1C, 'rncond':'', 'result':True, 'base':True},
     'MVN_ri': {'op': 0x1E, 'rncond':'', 'result':True, 'base':False},
 }
+
+supervisor_and_coproc = {
+    'MCR': {'op1': 0x20, 'op': 1, 'rn':0, 'coproc':0},
+}

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	Fri Oct 22 13:59:44 2010
@@ -114,6 +114,11 @@
         self.cb.CMP(r.r3, 123)
         self.assert_equal('CMP r3, #123')
 
+    def test_mcr(self):
+        self.cb.MCR(15, 0, r.r1, 7, 10,0)
+
+        self.assert_equal('MCR P15, 0, r1, c7, c10, 0')
+
 
 class TestInstrCodeBuilderForGeneratedInstr(ASMTest):
     def setup_method(self, ffuu_method):


From cfbolz at codespeak.net  Fri Oct 22 14:02:38 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 22 Oct 2010 14:02:38 +0200 (CEST)
Subject: [pypy-svn] r78200 - pypy/extradoc/talk/pepm2011
Message-ID: <20101022120238.D4DA936E0CB@codespeak.net>

Author: cfbolz
Date: Fri Oct 22 14:02:37 2010
New Revision: 78200

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.bib
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
fix citation


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.bib
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.bib	(original)
+++ pypy/extradoc/talk/pepm2011/paper.bib	Fri Oct 22 14:02:37 2010
@@ -130,7 +130,7 @@
 	abstract = {In the section on ?challenging problems? in the proceedings from the first international workshop on partial evaluation and mixed computation {[BEJ88]} a question is stated: {?Can} {PE} be used to generate new specialized data types, in a way analogous to generating specialized functions?. Since then little has been done to address this problem. In {[Lau89],} new types are indeed generated, but they are all simpler versions of the types in the original program. It is, e.g. not possible to have types with more constructors than the types in the original program. I propose to alleviate this by means of constructor specialization. Constructors are specialized with respect to the static parts of their arguments, just like residual functions. I show how this is done and argue that it makes it possible to get good results from partial evaluation in cases where the traditional methods fail to produce satisfactory results. The discussion is centered around a small subset of Standard {ML,} but the idea applies equally well to other languages having user defined constructors, e.g. Haskell and Prolog.},
 	booktitle = {Proceedings of the 1993 {ACM} {SIGPLAN} symposium on Partial evaluation and semantics-based program manipulation},
 	publisher = {{ACM}},
-	author = {Torben ? Mogensen},
+	author = {Torben Mogensen},
 	year = {1993},
 	pages = {22--32}
 },

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 22 14:02:37 2010
@@ -765,10 +765,9 @@
 We focus on the operations for manipulating heap allocated objects,
 as those are the only ones that are actually optimized. We also consider only
 objects with two fields $L$ and $R$ in this section, generalizing to arbitrary many fields
-is straightforward.
-
-Traces are lists of operations. The operations considered here are
+is straightforward. Traces are lists of operations. The operations considered here are
 \lstinline{new}, \lstinline{get}, \lstinline{set} and \lstinline{guard_class}.
+
 The values of all
 variables are locations (\ie pointers). Locations are mapped to objects, which
 are represented by triples $(T,l_1,l_2)$ of a type $T$, and two locations that represent the
@@ -814,6 +813,8 @@
 continues without changing heap and environment. Otherwise, execution is
 stopped.
 
+% subsection Operational Semantics of Traces (end)
+
 \subsection{Optimizing Traces}
 \label{sub:formalopt}
 
@@ -1210,7 +1211,7 @@
 residual program. This is similar to what our optimization does for an
 imperative language.
 In functional programming this idea was introduced as
- constructor specialisation by Mogensen \cite{Mogensen:PEPM93}.
+ constructor specialisation by Mogensen \cite{mogensen_constructor_1993}.
 
 %xxx:
 %partially static data structures: kenichi asai's thesis?


From cfbolz at codespeak.net  Fri Oct 22 14:06:46 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 22 Oct 2010 14:06:46 +0200 (CEST)
Subject: [pypy-svn] r78201 - pypy/extradoc/talk/pepm2011
Message-ID: <20101022120646.C5138282BDC@codespeak.net>

Author: cfbolz
Date: Fri Oct 22 14:06:45 2010
New Revision: 78201

Modified:
   pypy/extradoc/talk/pepm2011/escape-tracing.pdf
   pypy/extradoc/talk/pepm2011/paper.tex
Log:
kill XXX, remove line overflow


Modified: pypy/extradoc/talk/pepm2011/escape-tracing.pdf
==============================================================================
Binary files. No diff available.

Modified: pypy/extradoc/talk/pepm2011/paper.tex
==============================================================================
--- pypy/extradoc/talk/pepm2011/paper.tex	(original)
+++ pypy/extradoc/talk/pepm2011/paper.tex	Fri Oct 22 14:06:45 2010
@@ -477,9 +477,9 @@
     \item \lstinline{guard_true} checks that a boolean is true.
 \end{itemize}
 
-The method calls in the trace are always preceded by a \lstinline{guard_class}
+Method calls in the trace are preceded by a \lstinline{guard_class}
 operation, to check that the class of the receiver is the same as the one that
-was observed during tracing.\footnote{\lstinline{guard_class} XXX lstinline too large in footnotes
+was observed during tracing.\footnote{\lstinline{guard_class}
 performs a precise
 class check, not checking for subclasses.} These guards make the trace specific
 to the situation where \lstinline{y} is really a \lstinline{BoxedInteger}. When


From afa at codespeak.net  Fri Oct 22 14:12:02 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 22 Oct 2010 14:12:02 +0200 (CEST)
Subject: [pypy-svn] r78202 - pypy/trunk/pypy/translator/c/test
Message-ID: <20101022121202.4C92B36E0CB@codespeak.net>

Author: afa
Date: Fri Oct 22 14:12:00 2010
New Revision: 78202

Modified:
   pypy/trunk/pypy/translator/c/test/test_typed.py
Log:
Show how variable calls to __exit__ confuse the annotator


Modified: pypy/trunk/pypy/translator/c/test/test_typed.py
==============================================================================
--- pypy/trunk/pypy/translator/c/test/test_typed.py	(original)
+++ pypy/trunk/pypy/translator/c/test/test_typed.py	Fri Oct 22 14:12:00 2010
@@ -838,15 +838,16 @@
                     state.append('raised')
                 state.append('release')
 
-        def func():
+        def func(n):
             try:
                 with C('hello') as c:
                     state.append(c.name)
-                    raise ValueError
+                    if 1:
+                        raise ValueError
             except ValueError:
                 pass
             return ', '.join(state)
-        f = self.getcompiled(func, [])
-        res = f()
+        f = self.getcompiled(func, [int])
+        res = f(1)
         assert res == 'acquire, hello, raised, release'
 


From afa at codespeak.net  Fri Oct 22 14:14:14 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 22 Oct 2010 14:14:14 +0200 (CEST)
Subject: [pypy-svn] r78203 - pypy/trunk/pypy/translator/c/test
Message-ID: <20101022121414.5160B282BDC@codespeak.net>

Author: afa
Date: Fri Oct 22 14:14:12 2010
New Revision: 78203

Modified:
   pypy/trunk/pypy/translator/c/test/test_typed.py
Log:
The actual (failing) test


Modified: pypy/trunk/pypy/translator/c/test/test_typed.py
==============================================================================
--- pypy/trunk/pypy/translator/c/test/test_typed.py	(original)
+++ pypy/trunk/pypy/translator/c/test/test_typed.py	Fri Oct 22 14:14:12 2010
@@ -842,7 +842,7 @@
             try:
                 with C('hello') as c:
                     state.append(c.name)
-                    if 1:
+                    if n:
                         raise ValueError
             except ValueError:
                 pass


From cfbolz at codespeak.net  Fri Oct 22 14:24:57 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 22 Oct 2010 14:24:57 +0200 (CEST)
Subject: [pypy-svn] r78204 - pypy/extradoc/talk/pepm2011
Message-ID: <20101022122457.5EB3E282BDC@codespeak.net>

Author: cfbolz
Date: Fri Oct 22 14:24:56 2010
New Revision: 78204

Added:
   pypy/extradoc/talk/pepm2011/bolz-allocation-removal.pdf   (contents, props changed)
Log:
paper as submitted


Added: pypy/extradoc/talk/pepm2011/bolz-allocation-removal.pdf
==============================================================================
Binary file. No diff available.


From arigo at codespeak.net  Fri Oct 22 14:28:41 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 14:28:41 +0200 (CEST)
Subject: [pypy-svn] r78205 - pypy/build/bot2/pypybuildbot
Message-ID: <20101022122841.0CFE3282BDC@codespeak.net>

Author: arigo
Date: Fri Oct 22 14:28:40 2010
New Revision: 78205

Modified:
   pypy/build/bot2/pypybuildbot/summary.py
Log:
Increase the size of the cache.  Right now it needs 84 entries;
it used to be 60, so let's bump it to 150.


Modified: pypy/build/bot2/pypybuildbot/summary.py
==============================================================================
--- pypy/build/bot2/pypybuildbot/summary.py	(original)
+++ pypy/build/bot2/pypybuildbot/summary.py	Fri Oct 22 14:28:40 2010
@@ -295,7 +295,7 @@
 
 N = 5
 
-outcome_set_cache = RevisionOutcomeSetCache(10*(N+1))
+outcome_set_cache = RevisionOutcomeSetCache(25*(N+1))
 
 
 def colsizes(namekeys):


From afa at codespeak.net  Fri Oct 22 14:49:31 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 22 Oct 2010 14:49:31 +0200 (CEST)
Subject: [pypy-svn] r78207 - in pypy/trunk/pypy: interpreter objspace/flow
Message-ID: <20101022124931.F235C282BDC@codespeak.net>

Author: afa
Date: Fri Oct 22 14:49:30 2010
New Revision: 78207

Modified:
   pypy/trunk/pypy/interpreter/pyopcode.py
   pypy/trunk/pypy/objspace/flow/flowcontext.py
Log:
Fix the test: in RPython, the 'exc_type' parameter of __exit__ is not usable.
'exc_value' probably contains everything you need.


Modified: pypy/trunk/pypy/interpreter/pyopcode.py
==============================================================================
--- pypy/trunk/pypy/interpreter/pyopcode.py	(original)
+++ pypy/trunk/pypy/interpreter/pyopcode.py	Fri Oct 22 14:49:30 2010
@@ -164,6 +164,9 @@
             next_instr = block.handle(self, unroller)
             return next_instr
 
+    def call_contextmanager_exit_function(self, w_func, w_typ, w_val, w_tb):
+        return self.space.call_function(w_func, w_typ, w_val, w_tb)
+
     @jit.unroll_safe
     def dispatch_bytecode(self, co_code, next_instr, ec):
         space = self.space
@@ -886,7 +889,7 @@
             self.pushvalue(self.space.w_None)
             self.pushvalue(self.space.w_None)
         elif self.pycode.magic >= 0xa0df28c:
-            # Implemementation since 2.5a0: 62092 (changed WITH_CLEANUP opcode)
+            # Implementation since 2.5a0: 62092 (changed WITH_CLEANUP opcode)
             w_exitfunc = self.popvalue()
             w_unroller = self.peekvalue(2)
         else:
@@ -896,18 +899,20 @@
         if isinstance(unroller, SApplicationException):
             operr = unroller.operr
             w_traceback = self.space.wrap(operr.application_traceback)
-            w_result = self.space.call_function(w_exitfunc,
-                                                operr.w_type,
-                                                operr.get_w_value(self.space),
-                                                w_traceback)
+            w_result = self.call_contextmanager_exit_function(
+                w_exitfunc,
+                operr.w_type,
+                operr.get_w_value(self.space),
+                w_traceback)
             if self.space.is_true(w_result):
                 # __exit__() returned True -> Swallow the exception.
                 self.settopvalue(self.space.w_None, 2)
         else:
-            self.space.call_function(w_exitfunc,
-                                     self.space.w_None,
-                                     self.space.w_None,
-                                     self.space.w_None)
+            self.call_contextmanager_exit_function(
+                w_exitfunc,
+                self.space.w_None,
+                self.space.w_None,
+                self.space.w_None)
 
     @jit.unroll_safe
     def call_function(self, oparg, w_star=None, w_starstar=None):

Modified: pypy/trunk/pypy/objspace/flow/flowcontext.py
==============================================================================
--- pypy/trunk/pypy/objspace/flow/flowcontext.py	(original)
+++ pypy/trunk/pypy/objspace/flow/flowcontext.py	Fri Oct 22 14:49:30 2010
@@ -502,3 +502,10 @@
             raise operr
         return pyframe.PyFrame.handle_operation_error(self, ec, operr,
                                                       *args, **kwds)
+
+    def call_contextmanager_exit_function(self, w_func, w_typ, w_val, w_tb):
+        # The annotator won't allow to merge exception types with None.
+        # XXX return an object which will break translation when it is used
+        w_typ = self.space.w_None
+        return self.space.call_function(w_func, w_typ, w_val, w_tb)
+


From arigo at codespeak.net  Fri Oct 22 14:55:00 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 14:55:00 +0200 (CEST)
Subject: [pypy-svn] r78208 - in pypy/trunk/pypy: jit/codewriter
	jit/metainterp jit/metainterp/test rlib
Message-ID: <20101022125500.41E85282BDC@codespeak.net>

Author: arigo
Date: Fri Oct 22 14:54:58 2010
New Revision: 78208

Modified:
   pypy/trunk/pypy/jit/codewriter/jtransform.py
   pypy/trunk/pypy/jit/metainterp/test/test_jitdriver.py
   pypy/trunk/pypy/jit/metainterp/warmspot.py
   pypy/trunk/pypy/rlib/jit.py
Log:
Add the ability to deactivate some jitdrivers.
Far better solution than somehow hacking in the source to
remove the calls to jitdriver.jit_merge_point().


Modified: pypy/trunk/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/jtransform.py	Fri Oct 22 14:54:58 2010
@@ -831,6 +831,8 @@
     def rewrite_op_jit_marker(self, op):
         key = op.args[0].value
         jitdriver = op.args[1].value
+        if not jitdriver.active:
+            return []
         return getattr(self, 'handle_jit_marker__%s' % key)(op, jitdriver)
 
     def handle_jit_marker__jit_merge_point(self, op, jitdriver):

Modified: pypy/trunk/pypy/jit/metainterp/test/test_jitdriver.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_jitdriver.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_jitdriver.py	Fri Oct 22 14:54:58 2010
@@ -72,6 +72,33 @@
         # we expect no loop at all for 'loop1': it should always be inlined
         self.check_tree_loop_count(2)    # 1 x loop, 1 x enter bridge
 
+    def test_inactive_jitdriver(self):
+        myjitdriver1 = JitDriver(greens=[], reds=['n', 'm'],
+                                 get_printable_location = getloc1)
+        myjitdriver2 = JitDriver(greens=['g'], reds=['r'],
+                                 get_printable_location = getloc2)
+        #
+        myjitdriver1.active = False    # <===
+        #
+        def loop1(n, m):
+            while n > 0:
+                myjitdriver1.can_enter_jit(n=n, m=m)
+                myjitdriver1.jit_merge_point(n=n, m=m)
+                n -= m
+            return n
+        #
+        def loop2(g, r):
+            while r > 0:
+                myjitdriver2.can_enter_jit(g=g, r=r)
+                myjitdriver2.jit_merge_point(g=g, r=r)
+                r += loop1(r, g) + (-1)
+            return r
+        #
+        res = self.meta_interp(loop2, [4, 40], repeat=7, inline=True)
+        assert res == loop2(4, 40)
+        # we expect no int_sub, but a residual call
+        self.check_loops(int_sub=0, call=1)
+
 
 class TestLLtype(MultipleJitDriversTests, LLJitMixin):
     pass

Modified: pypy/trunk/pypy/jit/metainterp/warmspot.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/warmspot.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/warmspot.py	Fri Oct 22 14:54:58 2010
@@ -110,7 +110,8 @@
             for i in range(len(block.operations)):
                 op = block.operations[i]
                 if (op.opname == 'jit_marker' and
-                    op.args[0].value == marker_name):
+                    op.args[0].value == marker_name and
+                    op.args[1].value.active):   # the jitdriver
                     results.append((graph, block, i))
     return results
 

Modified: pypy/trunk/pypy/rlib/jit.py
==============================================================================
--- pypy/trunk/pypy/rlib/jit.py	(original)
+++ pypy/trunk/pypy/rlib/jit.py	Fri Oct 22 14:54:58 2010
@@ -274,8 +274,9 @@
     several independent JITting interpreters in it.
     """
 
+    active = True          # if set to False, this JitDriver is ignored
     virtualizables = []
-    
+
     def __init__(self, greens=None, reds=None, virtualizables=None,
                  get_jitcell_at=None, set_jitcell_at=None,
                  get_printable_location=None, confirm_enter_jit=None,


From arigo at codespeak.net  Fri Oct 22 14:55:52 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 14:55:52 +0200 (CEST)
Subject: [pypy-svn] r78209 - pypy/trunk/pypy/rlib/rsre
Message-ID: <20101022125552.B7AF536E0CB@codespeak.net>

Author: arigo
Date: Fri Oct 22 14:55:51 2010
New Revision: 78209

Modified:
   pypy/trunk/pypy/rlib/rsre/rsre_jit.py
Log:
Temporarily(?) deactive the rsre jit drivers,
to see the benchmark run results tomorrow.


Modified: pypy/trunk/pypy/rlib/rsre/rsre_jit.py
==============================================================================
--- pypy/trunk/pypy/rlib/rsre/rsre_jit.py	(original)
+++ pypy/trunk/pypy/rlib/rsre/rsre_jit.py	Fri Oct 22 14:55:51 2010
@@ -2,6 +2,7 @@
 
 
 class RSreJitDriver(JitDriver):
+    active = False        # XXX temporary?
 
     def __init__(self, name, debugprint, **kwds):
         JitDriver.__init__(self, **kwds)


From arigo at codespeak.net  Fri Oct 22 15:47:24 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 15:47:24 +0200 (CEST)
Subject: [pypy-svn] r78210 - pypy/build/bot2/pypybuildbot
Message-ID: <20101022134724.505F3282BDC@codespeak.net>

Author: arigo
Date: Fri Oct 22 15:47:22 2010
New Revision: 78210

Modified:
   pypy/build/bot2/pypybuildbot/master.py
   pypy/build/bot2/pypybuildbot/summary.py
Log:
Raaaaaaah.  Grrr grr gr.  Killed the lines in master.py.


Modified: pypy/build/bot2/pypybuildbot/master.py
==============================================================================
--- pypy/build/bot2/pypybuildbot/master.py	(original)
+++ pypy/build/bot2/pypybuildbot/master.py	Fri Oct 22 15:47:22 2010
@@ -317,14 +317,3 @@
     'buildbotURL': 'http://codespeak.net:%d/'%(httpPortNumber),
     'projectURL': 'http://codespeak.net/pypy/',
     'projectName': 'PyPy'}
-
-CACHESIZE = 80 # cache size for build outcomes
-
-estimated = (sum([len(_sched.listBuilderNames())
-             for _sched in BuildmasterConfig['schedulers']]) * 6)
-
-if estimated > CACHESIZE:
-    raise ValueError("master.py CACHESIZE (%d) is too small considered"
-                     " a builder*scheduler combinations based estimate (%d)"
-                     % (CACHESIZE, estimated))
-summary.outcome_set_cache.cachesize = CACHESIZE

Modified: pypy/build/bot2/pypybuildbot/summary.py
==============================================================================
--- pypy/build/bot2/pypybuildbot/summary.py	(original)
+++ pypy/build/bot2/pypybuildbot/summary.py	Fri Oct 22 15:47:22 2010
@@ -295,7 +295,7 @@
 
 N = 5
 
-outcome_set_cache = RevisionOutcomeSetCache(25*(N+1))
+outcome_set_cache = RevisionOutcomeSetCache(32*(N+1))
 
 
 def colsizes(namekeys):


From arigo at codespeak.net  Fri Oct 22 16:05:04 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 16:05:04 +0200 (CEST)
Subject: [pypy-svn] r78211 - pypy/trunk/pypy/module/array
Message-ID: <20101022140504.6BA6436E35D@codespeak.net>

Author: arigo
Date: Fri Oct 22 16:05:03 2010
New Revision: 78211

Modified:
   pypy/trunk/pypy/module/array/interp_array.py
Log:
Revert r78195: CPython returns longs for arrays of 'I' even on 64-bit machines,
and I don't want to consider the deep implications of that.


Modified: pypy/trunk/pypy/module/array/interp_array.py
==============================================================================
--- pypy/trunk/pypy/module/array/interp_array.py	(original)
+++ pypy/trunk/pypy/module/array/interp_array.py	Fri Oct 22 16:05:03 2010
@@ -323,7 +323,7 @@
         idx, stop, step = space.decode_index(w_idx, self.len)
         assert step == 0
         item = self.buffer[idx]
-        if mytype.canoverflow:    # if it fits in a Signed
+        if mytype.typecode in 'bBhHil':
             item = rffi.cast(lltype.Signed, item)
         elif mytype.typecode == 'f':
             item = float(item)


From arigo at codespeak.net  Fri Oct 22 16:09:58 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 16:09:58 +0200 (CEST)
Subject: [pypy-svn] r78212 - pypy/trunk/pypy/module/pypyjit/test
Message-ID: <20101022140958.CE98D282BDC@codespeak.net>

Author: arigo
Date: Fri Oct 22 16:09:57 2010
New Revision: 78212

Modified:
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
Log:
70 operations look fine on 64-bit machines.


Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Fri Oct 22 16:09:57 2010
@@ -836,7 +836,7 @@
     def test_array_intimg(self):
         # XXX this test is way too imprecise in what it is actually testing
         # it should count the number of guards instead
-        for tc, maxops in zip('ilILd', (67, 67, 69, 69, 61)):
+        for tc, maxops in zip('ilILd', (67, 67, 70, 70, 61)):
             print
             print '='*65
             print '='*20, 'running test for tc=%r' % (tc,), '='*20


From arigo at codespeak.net  Fri Oct 22 16:13:08 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 16:13:08 +0200 (CEST)
Subject: [pypy-svn] r78213 - pypy/branch/cleanup-dict-impl/pypy/objspace/std
Message-ID: <20101022141308.E225E282BDC@codespeak.net>

Author: arigo
Date: Fri Oct 22 16:13:07 2010
New Revision: 78213

Modified:
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py
Log:
Simplify the code for getting the correct class.
It should also help the JIT get rid of a few operations.


Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py	Fri Oct 22 16:13:07 2010
@@ -1,4 +1,4 @@
-from pypy.rlib import jit, objectmodel
+from pypy.rlib import jit, objectmodel, debug
 
 from pypy.interpreter.baseobjspace import W_Root
 from pypy.objspace.std.dictmultiobject import W_DictMultiObject
@@ -10,6 +10,9 @@
 # attribute shapes
 
 NUM_DIGITS = 4
+NUM_DIGITS_POW2 = 1 << NUM_DIGITS
+# note: we use "x * NUM_DIGITS_POW2" instead of "x << NUM_DIGITS" because
+# we want to propagate knowledge that the result cannot be negative
 
 class AbstractAttribute(object):
     _immutable_fields_ = ['w_cls']
@@ -69,8 +72,10 @@
         attr = self._get_new_attr(selector[0], selector[1])
         oldattr = obj._get_mapdict_map()
         if not jit.we_are_jitted():
-            oldattr._size_estimate += attr.size_estimate() - oldattr.size_estimate()
-            assert oldattr.size_estimate() >= oldattr.length()
+            size_est = (oldattr._size_estimate + attr.size_estimate()
+                                               - oldattr.size_estimate())
+            assert size_est >= (oldattr.length() * NUM_DIGITS_POW2)
+            oldattr._size_estimate = size_est
         if attr.length() > obj._mapdict_storage_length():
             # note that attr.size_estimate() is always at least attr.length()
             new_storage = [None] * attr.size_estimate()
@@ -188,7 +193,7 @@
         self.selector = selector
         self.position = back.length()
         self.back = back
-        self._size_estimate = self.length() << NUM_DIGITS
+        self._size_estimate = self.length() * NUM_DIGITS_POW2
 
     def _copy_attr(self, obj, new_obj):
         w_value = self.read(obj, self.selector)
@@ -384,12 +389,11 @@
     map = w_type.terminator
     classes = memo_get_subclass_of_correct_size(space, cls)
     size = map.size_estimate()
-    if not size:
-        size = 1
-    try:
-        return classes[size - 1]
-    except IndexError:
-        return classes[-1]
+    debug.check_nonneg(size)
+    if size < len(classes):
+        return classes[size]
+    else:
+        return classes[len(classes)-1]
 get_subclass_of_correct_size._annspecialcase_ = "specialize:arg(1)"
 
 NUM_SUBCLASSES = 10 # XXX tweak this number
@@ -403,6 +407,7 @@
         result = []
         for i in range(1, NUM_SUBCLASSES+1):
             result.append(_make_subclass_size_n(supercls, i))
+        result.insert(0, result[0])    # for 0, use the same class as for 1
         _subclass_cache[key] = result
         return result
 memo_get_subclass_of_correct_size._annspecialcase_ = "specialize:memo"


From afa at codespeak.net  Fri Oct 22 16:31:52 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 22 Oct 2010 16:31:52 +0200 (CEST)
Subject: [pypy-svn] r78214 - in pypy/trunk/pypy: objspace/flow
	translator/c/test
Message-ID: <20101022143152.6739B282BDC@codespeak.net>

Author: afa
Date: Fri Oct 22 16:31:50 2010
New Revision: 78214

Modified:
   pypy/trunk/pypy/objspace/flow/flowcontext.py
   pypy/trunk/pypy/translator/c/test/test_typed.py
Log:
Replace the exc_typ argument of the __exit__ method with an object
that is likely to break annotation if used in any way 
(except maybe with the 'is' operator)


Modified: pypy/trunk/pypy/objspace/flow/flowcontext.py
==============================================================================
--- pypy/trunk/pypy/objspace/flow/flowcontext.py	(original)
+++ pypy/trunk/pypy/objspace/flow/flowcontext.py	Fri Oct 22 16:31:50 2010
@@ -504,8 +504,10 @@
                                                       *args, **kwds)
 
     def call_contextmanager_exit_function(self, w_func, w_typ, w_val, w_tb):
-        # The annotator won't allow to merge exception types with None.
-        # XXX return an object which will break translation when it is used
-        w_typ = self.space.w_None
+        if w_typ is not self.space.w_None:
+            # The annotator won't allow to merge exception types with None.
+            # Replace it with an object which will break translation when used
+            # (except maybe with 'exc_typ is None')
+            w_typ = self.space.wrap(self.space)
         return self.space.call_function(w_func, w_typ, w_val, w_tb)
 

Modified: pypy/trunk/pypy/translator/c/test/test_typed.py
==============================================================================
--- pypy/trunk/pypy/translator/c/test/test_typed.py	(original)
+++ pypy/trunk/pypy/translator/c/test/test_typed.py	Fri Oct 22 16:31:50 2010
@@ -833,21 +833,32 @@
             def __enter__(self):
                 state.append('acquire')
                 return self
-            def __exit__(self, *args):
-                if args[1] is not None:
+            def __exit__(self, typ, value, tb):
+                if typ is not None:
+                    if value is None:
+                        raise RuntimeError('test failed')
                     state.append('raised')
+                else:
+                    if value is not None:
+                        raise RuntimeError('test failed')
                 state.append('release')
 
         def func(n):
+            del state[:]
             try:
                 with C('hello') as c:
                     state.append(c.name)
-                    if n:
+                    if n == 1:
                         raise ValueError
-            except ValueError:
+                    elif n == 2:
+                        raise TypeError
+            except (ValueError, TypeError):
                 pass
             return ', '.join(state)
         f = self.getcompiled(func, [int])
+        res = f(0)
+        assert res == 'acquire, hello, release'
         res = f(1)
         assert res == 'acquire, hello, raised, release'
-
+        res = f(2)
+        assert res == 'acquire, hello, raised, release'


From arigo at codespeak.net  Fri Oct 22 16:50:18 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 16:50:18 +0200 (CEST)
Subject: [pypy-svn] r78215 - pypy/trunk/pypy/translator/goal
Message-ID: <20101022145018.0B000282BDC@codespeak.net>

Author: arigo
Date: Fri Oct 22 16:50:17 2010
New Revision: 78215

Modified:
   pypy/trunk/pypy/translator/goal/app_main.py
Log:
Bah.  There is also the bug that we cannot combine options,
like "pypy-c -iE"


Modified: pypy/trunk/pypy/translator/goal/app_main.py
==============================================================================
--- pypy/trunk/pypy/translator/goal/app_main.py	(original)
+++ pypy/trunk/pypy/translator/goal/app_main.py	Fri Oct 22 16:50:17 2010
@@ -255,7 +255,7 @@
             break
         elif arg == '-u':
             unbuffered = True
-        elif arg == '-O':
+        elif arg == '-O' or arg == '-OO':
             pass
         elif arg == '--version' or arg == '-V':
             print "Python", sys.version


From arigo at codespeak.net  Fri Oct 22 17:12:00 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 17:12:00 +0200 (CEST)
Subject: [pypy-svn] r78216 - pypy/trunk/pypy/translator/goal
Message-ID: <20101022151200.F0C2F36E363@codespeak.net>

Author: arigo
Date: Fri Oct 22 17:11:59 2010
New Revision: 78216

Modified:
   pypy/trunk/pypy/translator/goal/app_main.py
Log:
Print an error message here.


Modified: pypy/trunk/pypy/translator/goal/app_main.py
==============================================================================
--- pypy/trunk/pypy/translator/goal/app_main.py	(original)
+++ pypy/trunk/pypy/translator/goal/app_main.py	Fri Oct 22 17:11:59 2010
@@ -386,8 +386,9 @@
                 if python_startup:
                     try:
                         startup = open(python_startup).read()
-                    except IOError:
-                        pass
+                    except IOError, e:
+                        print >> sys.stderr, "Could not open PYTHONSTARTUP"
+                        print >> sys.stderr, "IOError:", e
                     else:
                         def run_it():
                             co_python_startup = compile(startup,


From arigo at codespeak.net  Fri Oct 22 17:13:03 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 22 Oct 2010 17:13:03 +0200 (CEST)
Subject: [pypy-svn] r78217 - pypy/trunk/pypy/translator/goal
Message-ID: <20101022151303.7F644282BDC@codespeak.net>

Author: arigo
Date: Fri Oct 22 17:13:02 2010
New Revision: 78217

Modified:
   pypy/trunk/pypy/translator/goal/app_main.py
Log:
Use the proper gc-friendly style of explicitly close()ing the file.


Modified: pypy/trunk/pypy/translator/goal/app_main.py
==============================================================================
--- pypy/trunk/pypy/translator/goal/app_main.py	(original)
+++ pypy/trunk/pypy/translator/goal/app_main.py	Fri Oct 22 17:13:02 2010
@@ -385,7 +385,9 @@
                 python_startup = os.getenv('PYTHONSTARTUP')
                 if python_startup:
                     try:
-                        startup = open(python_startup).read()
+                        f = open(python_startup)
+                        startup = f.read()
+                        f.close()
                     except IOError, e:
                         print >> sys.stderr, "Could not open PYTHONSTARTUP"
                         print >> sys.stderr, "IOError:", e


From fijal at codespeak.net  Fri Oct 22 18:19:57 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Fri, 22 Oct 2010 18:19:57 +0200 (CEST)
Subject: [pypy-svn] r78218 - in pypy/trunk/pypy: jit/backend/test
	jit/backend/x86/test jit/metainterp/test jit/tool
	module/pypyjit/test
Message-ID: <20101022161957.01D29282BDC@codespeak.net>

Author: fijal
Date: Fri Oct 22 18:19:52 2010
New Revision: 78218

Added:
   pypy/trunk/pypy/jit/tool/oparser.py
      - copied, changed from r78182, pypy/trunk/pypy/jit/metainterp/test/oparser.py
Removed:
   pypy/trunk/pypy/jit/metainterp/test/oparser.py
Modified:
   pypy/trunk/pypy/jit/backend/test/runner_test.py
   pypy/trunk/pypy/jit/backend/x86/test/test_gc_integration.py
   pypy/trunk/pypy/jit/backend/x86/test/test_regalloc.py
   pypy/trunk/pypy/jit/backend/x86/test/test_runner.py
   pypy/trunk/pypy/jit/metainterp/test/test_compile.py
   pypy/trunk/pypy/jit/metainterp/test/test_logger.py
   pypy/trunk/pypy/jit/metainterp/test/test_oparser.py
   pypy/trunk/pypy/jit/metainterp/test/test_optimizefindnode.py
   pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py
   pypy/trunk/pypy/jit/tool/loopviewer.py
   pypy/trunk/pypy/jit/tool/showstats.py
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
Log:
Refactor a bit oparser. Also move it to tool instead of metainterp/test
(it's rather widely used by now)


Modified: pypy/trunk/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/trunk/pypy/jit/backend/test/runner_test.py	Fri Oct 22 18:19:52 2010
@@ -9,7 +9,7 @@
                                          ConstObj, BoxFloat, ConstFloat)
 from pypy.jit.metainterp.resoperation import ResOperation, rop
 from pypy.jit.metainterp.typesystem import deref
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.rpython.lltypesystem import lltype, llmemory, rstr, rffi, rclass
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython.annlowlevel import llhelper

Modified: pypy/trunk/pypy/jit/backend/x86/test/test_gc_integration.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/test/test_gc_integration.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/test/test_gc_integration.py	Fri Oct 22 18:19:52 2010
@@ -12,7 +12,7 @@
 from pypy.jit.backend.detect_cpu import getcpuclass
 from pypy.jit.backend.x86.regalloc import RegAlloc
 from pypy.jit.backend.x86.arch import WORD, FRAME_FIXED_SIZE
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.rpython.lltypesystem import lltype, llmemory, rffi
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import rclass, rstr

Modified: pypy/trunk/pypy/jit/backend/x86/test/test_regalloc.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/test/test_regalloc.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/test/test_regalloc.py	Fri Oct 22 18:19:52 2010
@@ -11,7 +11,7 @@
 from pypy.jit.backend.x86.regalloc import RegAlloc, X86RegisterManager,\
      FloatConstants, is_comparison_or_ovf_op
 from pypy.jit.backend.x86.arch import IS_X86_32, IS_X86_64
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.rpython.lltypesystem import lltype, llmemory, rffi
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import rclass, rstr

Modified: pypy/trunk/pypy/jit/backend/x86/test/test_runner.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/test/test_runner.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/test/test_runner.py	Fri Oct 22 18:19:52 2010
@@ -10,7 +10,7 @@
 from pypy.jit.metainterp.resoperation import rop
 from pypy.jit.metainterp.executor import execute
 from pypy.jit.backend.test.runner_test import LLtypeBackendTest
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.tool.udir import udir
 import ctypes
 import sys

Modified: pypy/trunk/pypy/jit/metainterp/test/test_compile.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_compile.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_compile.py	Fri Oct 22 18:19:52 2010
@@ -6,8 +6,8 @@
 from pypy.jit.metainterp.compile import ResumeGuardCountersInt
 from pypy.jit.metainterp.compile import compile_tmp_callback
 from pypy.jit.metainterp import optimize, jitprof, typesystem, compile
-from pypy.jit.metainterp.test.oparser import parse
 from pypy.jit.metainterp.test.test_optimizefindnode import LLtypeMixin
+from pypy.jit.tool.oparser import parse
 
 
 def test_insert_loop_token():

Modified: pypy/trunk/pypy/jit/metainterp/test/test_logger.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_logger.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_logger.py	Fri Oct 22 18:19:52 2010
@@ -1,6 +1,6 @@
 import sys
 from pypy.rlib import debug
-from pypy.jit.metainterp.test.oparser import pure_parse
+from pypy.jit.tool.oparser import pure_parse
 from pypy.jit.metainterp import logger
 from pypy.jit.metainterp.typesystem import llhelper
 from StringIO import StringIO

Modified: pypy/trunk/pypy/jit/metainterp/test/test_oparser.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_oparser.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_oparser.py	Fri Oct 22 18:19:52 2010
@@ -1,7 +1,7 @@
 
 from pypy.rpython.lltypesystem import lltype, llmemory
 
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.jit.metainterp.resoperation import rop
 from pypy.jit.metainterp.history import AbstractDescr, BoxInt, LoopToken,\
      BoxFloat

Modified: pypy/trunk/pypy/jit/metainterp/test/test_optimizefindnode.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_optimizefindnode.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_optimizefindnode.py	Fri Oct 22 18:19:52 2010
@@ -18,7 +18,7 @@
 from pypy.jit.metainterp.specnode import ConstantSpecNode
 from pypy.jit.codewriter.effectinfo import EffectInfo
 from pypy.jit.codewriter.heaptracker import register_known_gctype, adr2int
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 
 def test_sort_descrs():
     class PseudoDescr(AbstractDescr):

Modified: pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py	Fri Oct 22 18:19:52 2010
@@ -12,7 +12,7 @@
 from pypy.jit.metainterp.jitprof import EmptyProfiler
 from pypy.jit.metainterp import executor, compile, resume, history
 from pypy.jit.metainterp.resoperation import rop, opname, ResOperation
-from pypy.jit.metainterp.test.oparser import pure_parse
+from pypy.jit.tool.oparser import pure_parse
 
 ##class FakeFrame(object):
 ##    parent_resumedata_snapshot = None

Modified: pypy/trunk/pypy/jit/tool/loopviewer.py
==============================================================================
--- pypy/trunk/pypy/jit/tool/loopviewer.py	(original)
+++ pypy/trunk/pypy/jit/tool/loopviewer.py	Fri Oct 22 18:19:52 2010
@@ -9,7 +9,7 @@
 import optparse
 from pprint import pprint
 from pypy.tool import logparser
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.jit.metainterp.history import ConstInt
 from pypy.rpython.lltypesystem import llmemory, lltype
 

Copied: pypy/trunk/pypy/jit/tool/oparser.py (from r78182, pypy/trunk/pypy/jit/metainterp/test/oparser.py)
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/oparser.py	(original)
+++ pypy/trunk/pypy/jit/tool/oparser.py	Fri Oct 22 18:19:52 2010
@@ -184,24 +184,9 @@
                 self.newvar(arg)
             return self.vars[arg]
 
-    def parse_op(self, line):
-        num = line.find('(')
-        if num == -1:
-            raise ParseError("invalid line: %s" % line)
-        opname = line[:num]
-        try:
-            opnum = getattr(rop, opname.upper())
-        except AttributeError:
-            if opname == 'escape':
-                opnum = ESCAPE_OP.OPNUM
-            else:
-                raise ParseError("unknown op: %s" % opname)
-        endnum = line.rfind(')')
-        if endnum == -1:
-            raise ParseError("invalid line: %s" % line)
+    def parse_args(self, opname, argspec):
         args = []
         descr = None
-        argspec = line[num + 1:endnum]
         if argspec.strip():
             if opname == 'debug_merge_point':
                 allargs = [argspec]
@@ -219,6 +204,24 @@
                     args.append(self.getvar(arg))
                 except KeyError:
                     raise ParseError("Unknown var: %s" % arg)
+        return args, descr
+
+    def parse_op(self, line):
+        num = line.find('(')
+        if num == -1:
+            raise ParseError("invalid line: %s" % line)
+        opname = line[:num]
+        try:
+            opnum = getattr(rop, opname.upper())
+        except AttributeError:
+            if opname == 'escape':
+                opnum = ESCAPE_OP.OPNUM
+            else:
+                raise ParseError("unknown op: %s" % opname)
+        endnum = line.rfind(')')
+        if endnum == -1:
+            raise ParseError("invalid line: %s" % line)
+        args, descr = self.parse_args(opname, line[num + 1:endnum])
         if rop._GUARD_FIRST <= opnum <= rop._GUARD_LAST:
             i = line.find('[', endnum) + 1
             j = line.find(']', i)

Modified: pypy/trunk/pypy/jit/tool/showstats.py
==============================================================================
--- pypy/trunk/pypy/jit/tool/showstats.py	(original)
+++ pypy/trunk/pypy/jit/tool/showstats.py	Fri Oct 22 18:19:52 2010
@@ -4,7 +4,7 @@
 import autopath
 import sys, py
 from pypy.tool import logparser
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.jit.metainterp.resoperation import rop
 from pypy.rpython.lltypesystem import lltype, llmemory
 

Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Fri Oct 22 18:19:52 2010
@@ -131,7 +131,7 @@
         return result
 
     def parse_loops(self, opslogfile):
-        from pypy.jit.metainterp.test.oparser import parse
+        from pypy.jit.tool.oparser import parse
         from pypy.tool import logparser
         assert opslogfile.check()
         log = logparser.parse_log_file(str(opslogfile))


From dan at codespeak.net  Fri Oct 22 19:32:41 2010
From: dan at codespeak.net (dan at codespeak.net)
Date: Fri, 22 Oct 2010 19:32:41 +0200 (CEST)
Subject: [pypy-svn] r78223 - pypy/branch/ootype-virtualrefs/pypy/rlib/test
Message-ID: <20101022173241.5D9BA282BDC@codespeak.net>

Author: dan
Date: Fri Oct 22 19:32:39 2010
New Revision: 78223

Modified:
   pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py
Log:
Properly test interp() results.

Modified: pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py
==============================================================================
--- pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py	(original)
+++ pypy/branch/ootype-virtualrefs/pypy/rlib/test/test__jit_vref.py	Fri Oct 22 19:32:39 2010
@@ -6,6 +6,7 @@
 from pypy.annotation.annrpython import RPythonAnnotator
 from pypy.rpython.test.test_llinterp import interpret
 from pypy.rpython.lltypesystem.rclass import OBJECTPTR
+from pypy.rpython.ootypesystem.rclass import OBJECT
 from pypy.rpython.lltypesystem import lltype
 
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
@@ -85,7 +86,7 @@
         def f():
             return virtual_ref(X())
         x = self.interpret(f, [])
-        assert self.is_of_instance_type(x)
+        assert lltype.typeOf(x) == self.OBJECTTYPE
 
     def test_rtype_2(self):
         def f():
@@ -95,7 +96,7 @@
             virtual_ref_finish(x2)
             return x2
         x = self.interpret(f, [])
-        assert lltype.castable(OBJECTPTR, lltype.typeOf(x)) > 0
+        assert self.castable(self.OBJECTTYPE, x)
 
     def test_rtype_3(self):
         def f(n):
@@ -104,7 +105,7 @@
             else:
                 return non_virtual_ref(Z())
         x = self.interpret(f, [-5])
-        assert self.is_of_instance_type(x)
+        assert lltype.typeOf(x) == self.OBJECTTYPE
 
     def test_rtype_4(self):
         def f(n):
@@ -113,11 +114,15 @@
             else:
                 return vref_None
         x = self.interpret(f, [-5])
-        assert self.is_of_instance_type(x)
+        assert lltype.typeOf(x) == self.OBJECTTYPE
         assert not x
 
 class TestLLtype(BaseTestVRef, LLRtypeMixin):
-    pass
+    OBJECTTYPE = OBJECTPTR
+    def castable(self, TO, var):
+        return lltype.castable(TO, lltype.typeOf(var)) > 0
 
 class TestOOtype(BaseTestVRef, OORtypeMixin):
-    pass
+    OBJECTTYPE = OBJECT 
+    def castable(self, TO, var):
+        return ootype.isSubclass(TO, lltype.typeOf(var))


From fijal at codespeak.net  Fri Oct 22 19:46:58 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Fri, 22 Oct 2010 19:46:58 +0200 (CEST)
Subject: [pypy-svn] r78224 - in pypy/trunk/pypy/rpython: lltypesystem test
Message-ID: <20101022174658.DA3F5282BDC@codespeak.net>

Author: fijal
Date: Fri Oct 22 19:46:57 2010
New Revision: 78224

Modified:
   pypy/trunk/pypy/rpython/lltypesystem/lltype.py
   pypy/trunk/pypy/rpython/test/test_llinterp.py
Log:
A test and a fix


Modified: pypy/trunk/pypy/rpython/lltypesystem/lltype.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/lltype.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/lltype.py	Fri Oct 22 19:46:57 2010
@@ -1871,7 +1871,10 @@
 def _make_scoped_allocator(T):
     class ScopedAlloc:
         def __init__(self, n=None, zero=False):
-            self.buf = malloc(T, n=n, flavor='raw', zero=zero)
+            if n is None:
+                self.buf = malloc(T, flavor='raw', zero=zero)
+            else:
+                self.buf = malloc(T, n, flavor='raw', zero=zero)
 
         def __enter__(self):
             return self.buf

Modified: pypy/trunk/pypy/rpython/test/test_llinterp.py
==============================================================================
--- pypy/trunk/pypy/rpython/test/test_llinterp.py	(original)
+++ pypy/trunk/pypy/rpython/test/test_llinterp.py	Fri Oct 22 19:46:57 2010
@@ -645,3 +645,16 @@
     res = interpret(f, [])
     assert hlstr(res) == 'acquire, use, raised, release'
 
+
+def test_scoped_allocator():
+    from pypy.rpython.lltypesystem.lltype import scoped_alloc, Array, Signed
+    T = Array(Signed)
+    
+    def f():
+        x = 0
+        with scoped_alloc(T, 1) as array:
+            array[0] = -42
+            x = array[0]
+        assert x == -42
+
+    res = interpret(f, [])


From afa at codespeak.net  Fri Oct 22 20:14:22 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 22 Oct 2010 20:14:22 +0200 (CEST)
Subject: [pypy-svn] r78225 - in pypy/trunk/pypy: module/_winreg
	rpython/lltypesystem
Message-ID: <20101022181422.00056282BDC@codespeak.net>

Author: afa
Date: Fri Oct 22 20:14:21 2010
New Revision: 78225

Modified:
   pypy/trunk/pypy/module/_winreg/interp_winreg.py
   pypy/trunk/pypy/rpython/lltypesystem/rffi.py
Log:
Start using the "with" statement to manage temporary allocations.

Only the _winreg module for now... it's now 60 lines shorter!


Modified: pypy/trunk/pypy/module/_winreg/interp_winreg.py
==============================================================================
--- pypy/trunk/pypy/module/_winreg/interp_winreg.py	(original)
+++ pypy/trunk/pypy/module/_winreg/interp_winreg.py	Fri Oct 22 20:14:21 2010
@@ -212,13 +212,10 @@
         subkey = None
     else:
         subkey = space.str_w(w_subkey)
-    dataptr = rffi.str2charp(value)
-    try:
+    with rffi.scoped_str2charp(value) as dataptr:
         ret = rwinreg.RegSetValue(hkey, subkey, rwinreg.REG_SZ, dataptr, len(value))
-    finally:
-        rffi.free_charp(dataptr)
-    if ret != 0:
-        raiseWindowsError(space, ret, 'RegSetValue')
+        if ret != 0:
+            raiseWindowsError(space, ret, 'RegSetValue')
 SetValue.unwrap_spec = [ObjSpace, W_Root, W_Root, int, str]
 
 def QueryValue(space, w_hkey, w_subkey):
@@ -237,23 +234,15 @@
         subkey = None
     else:
         subkey = space.str_w(w_subkey)
-    bufsize_p = lltype.malloc(rwin32.PLONG.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwin32.PLONG.TO, 1) as bufsize_p:
         ret = rwinreg.RegQueryValue(hkey, subkey, None, bufsize_p)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegQueryValue')
-        buf = lltype.malloc(rffi.CCHARP.TO, bufsize_p[0], flavor='raw')
-        try:
+        with lltype.scoped_alloc(rffi.CCHARP.TO, bufsize_p[0]) as buf:
             ret = rwinreg.RegQueryValue(hkey, subkey, buf, bufsize_p)
             if ret != 0:
                 raiseWindowsError(space, ret, 'RegQueryValue')
             return space.wrap(rffi.charp2strn(buf, bufsize_p[0] - 1))
-        finally:
-            lltype.free(buf, flavor='raw')
-    finally:
-        lltype.free(bufsize_p, flavor='raw')
-    if ret != 0:
-        raiseWindowsError(space, ret, 'RegQueryValue')
 QueryValue.unwrap_spec = [ObjSpace, W_Root, W_Root]
 
 def convert_to_regdata(space, w_value, typ):
@@ -412,16 +401,14 @@
 value_name is a string indicating the value to query"""
     hkey = hkey_w(w_hkey, space)
     null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
-    retDataSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retDataSize:
         ret = rwinreg.RegQueryValueEx(hkey, subkey, null_dword, null_dword,
                                       None, retDataSize)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegQueryValueEx')
-        databuf = lltype.malloc(rffi.CCHARP.TO, retDataSize[0], flavor='raw')
-        try:
-            retType = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-            try:
+
+        with lltype.scoped_alloc(rffi.CCHARP.TO, retDataSize[0]) as databuf:
+            with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retType:
 
                 ret = rwinreg.RegQueryValueEx(hkey, subkey, null_dword,
                                               retType, databuf, retDataSize)
@@ -432,12 +419,6 @@
                                          retDataSize[0], retType[0]),
                     space.wrap(retType[0]),
                     ])
-            finally:
-                lltype.free(retType, flavor='raw')
-        finally:
-            lltype.free(databuf, flavor='raw')
-    finally:
-        lltype.free(retDataSize, flavor='raw')
 
 QueryValueEx.unwrap_spec = [ObjSpace, W_Root, str]
 
@@ -454,14 +435,11 @@
 The return value is the handle of the opened key.
 If the function fails, an exception is raised."""
     hkey = hkey_w(w_hkey, space)
-    rethkey = lltype.malloc(rwinreg.PHKEY.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwinreg.PHKEY.TO, 1) as rethkey:
         ret = rwinreg.RegCreateKey(hkey, subkey, rethkey)
         if ret != 0:
             raiseWindowsError(space, ret, 'CreateKey')
         return space.wrap(W_HKEY(rethkey[0]))
-    finally:
-        lltype.free(rethkey, flavor='raw')
 CreateKey.unwrap_spec = [ObjSpace, W_Root, str]
 
 def DeleteKey(space, w_hkey, subkey):
@@ -504,14 +482,11 @@
 The result is a new handle to the specified key
 If the function fails, an EnvironmentError exception is raised."""
     hkey = hkey_w(w_hkey, space)
-    rethkey = lltype.malloc(rwinreg.PHKEY.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwinreg.PHKEY.TO, 1) as rethkey:
         ret = rwinreg.RegOpenKeyEx(hkey, subkey, res, sam, rethkey)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegOpenKeyEx')
         return space.wrap(W_HKEY(rethkey[0]))
-    finally:
-        lltype.free(rethkey, flavor='raw')
 OpenKey.unwrap_spec = [ObjSpace, W_Root, str, int, rffi.r_uint]
 
 def EnumValue(space, w_hkey, index):
@@ -531,10 +506,8 @@
     hkey = hkey_w(w_hkey, space)
     null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
 
-    retValueSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-    try:
-        retDataSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-        try:
+    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retValueSize:
+        with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retDataSize:
             ret = rwinreg.RegQueryInfoKey(
                 hkey, None, null_dword, null_dword,
                 null_dword, null_dword, null_dword,
@@ -546,14 +519,9 @@
             retValueSize[0] += 1
             retDataSize[0] += 1
 
-            valuebuf = lltype.malloc(rffi.CCHARP.TO, retValueSize[0],
-                                     flavor='raw')
-            try:
-                databuf = lltype.malloc(rffi.CCHARP.TO, retDataSize[0],
-                                        flavor='raw')
-                try:
-                    retType = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-                    try:
+            with lltype.scoped_alloc(rffi.CCHARP.TO, retValueSize[0]) as valuebuf:
+                with lltype.scoped_alloc(rffi.CCHARP.TO, retDataSize[0]) as databuf:
+                    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retType:
                         ret = rwinreg.RegEnumValue(
                             hkey, index, valuebuf, retValueSize,
                             null_dword, retType, databuf, retDataSize)
@@ -566,16 +534,6 @@
                                                  retDataSize[0], retType[0]),
                             space.wrap(retType[0]),
                             ])
-                    finally:
-                        lltype.free(retType, flavor='raw')
-                finally:
-                    lltype.free(databuf, flavor='raw')
-            finally:
-                lltype.free(valuebuf, flavor='raw')
-        finally:
-            lltype.free(retDataSize, flavor='raw')
-    finally:
-        lltype.free(retValueSize, flavor='raw')
 
 EnumValue.unwrap_spec = [ObjSpace, W_Root, int]
 
@@ -592,10 +550,8 @@
     null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
 
     # max key name length is 255
-    buf = lltype.malloc(rffi.CCHARP.TO, 256, flavor='raw')
-    try:
-        retValueSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-        try:
+    with lltype.scoped_alloc(rffi.CCHARP.TO, 256) as buf:
+        with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retValueSize:
             retValueSize[0] = 256 # includes NULL terminator
             ret = rwinreg.RegEnumKeyEx(hkey, index, buf, retValueSize,
                                        null_dword, None, null_dword,
@@ -603,10 +559,6 @@
             if ret != 0:
                 raiseWindowsError(space, ret, 'RegEnumKeyEx')
             return space.wrap(rffi.charp2str(buf))
-        finally:
-            lltype.free(retValueSize, flavor='raw')
-    finally:
-        lltype.free(buf, flavor='raw')
 
 EnumKey.unwrap_spec = [ObjSpace, W_Root, int]
 
@@ -621,12 +573,9 @@
 A long integer that identifies when the key was last modified (if available)
  as 100's of nanoseconds since Jan 1, 1600."""
     hkey = hkey_w(w_hkey, space)
-    nSubKeys = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-    try:
-        nValues = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-        try:
-            ft = lltype.malloc(rwin32.PFILETIME.TO, 1, flavor='raw')
-            try:
+    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as nSubKeys:
+        with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as nValues:
+            with lltype.scoped_alloc(rwin32.PFILETIME.TO, 1) as ft:
                 null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
                 ret = rwinreg.RegQueryInfoKey(
                     hkey, None, null_dword, null_dword,
@@ -640,12 +589,6 @@
                 return space.newtuple([space.wrap(nSubKeys[0]),
                                        space.wrap(nValues[0]),
                                        space.wrap(l)])
-            finally:
-                lltype.free(ft, flavor='raw')
-        finally:
-            lltype.free(nValues, flavor='raw')
-    finally:
-        lltype.free(nSubKeys, flavor='raw')
 QueryInfoKey.unwrap_spec = [ObjSpace, W_Root]
 
 def str_or_None_w(space, w_obj):
@@ -666,12 +609,9 @@
 If the function fails, an EnvironmentError exception is raised."""
     machine = str_or_None_w(space, w_machine)
     hkey = hkey_w(w_hkey, space)
-    rethkey = lltype.malloc(rwinreg.PHKEY.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwinreg.PHKEY.TO, 1) as rethkey:
         ret = rwinreg.RegConnectRegistry(machine, hkey, rethkey)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegConnectRegistry')
         return space.wrap(W_HKEY(rethkey[0]))
-    finally:
-        lltype.free(rethkey, flavor='raw')
 ConnectRegistry.unwrap_spec = [ObjSpace, W_Root, W_Root]

Modified: pypy/trunk/pypy/rpython/lltypesystem/rffi.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/rffi.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/rffi.py	Fri Oct 22 20:14:21 2010
@@ -933,3 +933,11 @@
     """
     return cast(lltype.Signed, getattr(pdst, fieldname))
 getintfield._annspecialcase_ = 'specialize:ll_and_arg(1)'
+
+class scoped_str2charp:
+    def __init__(self, value):
+        self.buf = str2charp(value)
+    def __enter__(self):
+        return self.buf
+    def __exit__(self, *args):
+        free_charp(self.buf)


From afa at codespeak.net  Fri Oct 22 22:30:52 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 22 Oct 2010 22:30:52 +0200 (CEST)
Subject: [pypy-svn] r78226 - pypy/trunk/pypy/module/bz2
Message-ID: <20101022203052.EDE5F282BDC@codespeak.net>

Author: afa
Date: Fri Oct 22 22:30:49 2010
New Revision: 78226

Modified:
   pypy/trunk/pypy/module/bz2/interp_bz2.py
Log:
Use the "with" memory manager in bz2 module


Modified: pypy/trunk/pypy/module/bz2/interp_bz2.py
==============================================================================
--- pypy/trunk/pypy/module/bz2/interp_bz2.py	(original)
+++ pypy/trunk/pypy/module/bz2/interp_bz2.py	Fri Oct 22 22:30:49 2010
@@ -225,6 +225,11 @@
         if self.current_size > 0:
             rffi.keep_buffer_alive_until_here(self.raw_buf, self.gc_buf)
 
+    def __enter__(self):
+        return self
+    def __exit__(self, *args):
+        self.free()
+
 # ____________________________________________________________
 #
 # Make the BZ2File type by internally inheriting from W_File.
@@ -531,33 +536,30 @@
         if not self.running:
             raise OperationError(self.space.w_ValueError,
                 self.space.wrap("this object was already flushed"))
-        
-        out = OutBuffer(self.bzs)
+
         in_bufsize = datasize
-        in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-        for i in range(datasize):
-            in_buf[i] = data[i]
 
-        try:
-        
-            self.bzs.c_next_in = in_buf
-            rffi.setintfield(self.bzs, 'c_avail_in', in_bufsize)
+        with OutBuffer(self.bzs) as out:
+            with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
 
-            while True:
-                bzerror = BZ2_bzCompress(self.bzs, BZ_RUN)
-                if bzerror != BZ_RUN_OK:
-                    _catch_bz2_error(self.space, bzerror)
+                for i in range(datasize):
+                    in_buf[i] = data[i]
 
-                if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
-                    break
-                elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
-                    out.prepare_next_chunk()
+                self.bzs.c_next_in = in_buf
+                rffi.setintfield(self.bzs, 'c_avail_in', in_bufsize)
 
-            res = out.make_result_string()
-            return self.space.wrap(res)
-        finally:
-            lltype.free(in_buf, flavor='raw')
-            out.free()
+                while True:
+                    bzerror = BZ2_bzCompress(self.bzs, BZ_RUN)
+                    if bzerror != BZ_RUN_OK:
+                        _catch_bz2_error(self.space, bzerror)
+
+                    if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
+                        break
+                    elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
+
+                res = out.make_result_string()
+                return self.space.wrap(res)
 
     compress.unwrap_spec = ['self', 'bufferstr']
     
@@ -566,9 +568,8 @@
             raise OperationError(self.space.w_ValueError,
                 self.space.wrap("this object was already flushed"))
         self.running = False
-        
-        out = OutBuffer(self.bzs)
-        try:
+
+        with OutBuffer(self.bzs) as out:
             while True:
                 bzerror = BZ2_bzCompress(self.bzs, BZ_FINISH)
                 if bzerror == BZ_STREAM_END:
@@ -581,8 +582,6 @@
 
             res = out.make_result_string()
             return self.space.wrap(res)
-        finally:
-            out.free()
     flush.unwrap_spec = ['self']
 
 W_BZ2Compressor.typedef = TypeDef("BZ2Compressor",
@@ -641,38 +640,37 @@
         if not self.running:
             raise OperationError(self.space.w_EOFError,
                 self.space.wrap("end of stream was already found"))
-        
+
         in_bufsize = len(data)
-        in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-        for i in range(in_bufsize):
-            in_buf[i] = data[i]
 
-        out = OutBuffer(self.bzs)
-        try:
+        with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
+            for i in range(in_bufsize):
+                in_buf[i] = data[i]
             self.bzs.c_next_in = in_buf
             rffi.setintfield(self.bzs, 'c_avail_in', in_bufsize)
 
-            while True:
-                bzerror = BZ2_bzDecompress(self.bzs)
-                if bzerror == BZ_STREAM_END:
-                    if rffi.getintfield(self.bzs, 'c_avail_in') != 0:
-                        unused = [self.bzs.c_next_in[i] for i in range(rffi.getintfield(self.bzs, 'c_avail_in'))]
-                        self.unused_data = "".join(unused)
-                    self.running = False
-                    break
-                if bzerror != BZ_OK:
-                    _catch_bz2_error(self.space, bzerror)
-
-                if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
-                    break
-                elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
-                    out.prepare_next_chunk()
+            with OutBuffer(self.bzs) as out:
+                while True:
+                    bzerror = BZ2_bzDecompress(self.bzs)
+                    if bzerror == BZ_STREAM_END:
+                        if rffi.getintfield(self.bzs, 'c_avail_in') != 0:
+                            unused = [self.bzs.c_next_in[i]
+                                      for i in range(
+                                          rffi.getintfield(self.bzs,
+                                                           'c_avail_in'))]
+                            self.unused_data = "".join(unused)
+                        self.running = False
+                        break
+                    if bzerror != BZ_OK:
+                        _catch_bz2_error(self.space, bzerror)
+
+                    if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
+                        break
+                    elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
 
-            res = out.make_result_string()
-            return self.space.wrap(res)
-        finally:
-            lltype.free(in_buf, flavor='raw')
-            out.free()
+                res = out.make_result_string()
+                return self.space.wrap(res)
 
     decompress.unwrap_spec = ['self', 'bufferstr']
 
@@ -695,43 +693,39 @@
     if compresslevel < 1 or compresslevel > 9:
         raise OperationError(space.w_ValueError,
             space.wrap("compresslevel must be between 1 and 9"))
-            
-    bzs = lltype.malloc(bz_stream.TO, flavor='raw', zero=True)
-    in_bufsize = len(data)
-    # conforming to bz2 manual, this is large enough to fit compressed
-    # data in one shot. We will check it later anyway.
-    out = OutBuffer(bzs, in_bufsize + (in_bufsize / 100 + 1) + 600)
-
-    in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-    for i in range(in_bufsize):
-        in_buf[i] = data[i]
-
-    try:
-        bzs.c_next_in = in_buf
-        rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
 
-        bzerror = BZ2_bzCompressInit(bzs, compresslevel, 0, 0)
-        if bzerror != BZ_OK:
-            _catch_bz2_error(space, bzerror)
+    with lltype.scoped_alloc(bz_stream.TO, zero=True) as bzs:
+        in_bufsize = len(data)
+
+        with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
+            for i in range(in_bufsize):
+                in_buf[i] = data[i]
+            bzs.c_next_in = in_buf
+            rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
+
+            # conforming to bz2 manual, this is large enough to fit compressed
+            # data in one shot. We will check it later anyway.
+            with OutBuffer(bzs,
+                           in_bufsize + (in_bufsize / 100 + 1) + 600) as out:
+
+                bzerror = BZ2_bzCompressInit(bzs, compresslevel, 0, 0)
+                if bzerror != BZ_OK:
+                    _catch_bz2_error(space, bzerror)
 
-        while True:
-            bzerror = BZ2_bzCompress(bzs, BZ_FINISH)
-            if bzerror == BZ_STREAM_END:
-                break
-            elif bzerror != BZ_FINISH_OK:
+                while True:
+                    bzerror = BZ2_bzCompress(bzs, BZ_FINISH)
+                    if bzerror == BZ_STREAM_END:
+                        break
+                    elif bzerror != BZ_FINISH_OK:
+                        BZ2_bzCompressEnd(bzs)
+                        _catch_bz2_error(space, bzerror)
+
+                    if rffi.getintfield(bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
+
+                res = out.make_result_string()
                 BZ2_bzCompressEnd(bzs)
-                _catch_bz2_error(space, bzerror)
-            
-            if rffi.getintfield(bzs, 'c_avail_out') == 0:
-                out.prepare_next_chunk()
-
-        res = out.make_result_string()
-        BZ2_bzCompressEnd(bzs)
-        return space.wrap(res)
-    finally:
-        lltype.free(bzs, flavor='raw')
-        lltype.free(in_buf, flavor='raw')
-        out.free()
+                return space.wrap(res)
 compress.unwrap_spec = [ObjSpace, 'bufferstr', int]
 
 def decompress(space, data):
@@ -744,40 +738,34 @@
     if in_bufsize == 0:
         return space.wrap("")
 
-    bzs = lltype.malloc(bz_stream.TO, flavor='raw', zero=True)
-    in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-    for i in range(in_bufsize):
-        in_buf[i] = data[i]
-
-    out = OutBuffer(bzs)
-    try:
-        bzs.c_next_in = in_buf
-        rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
-    
-        bzerror = BZ2_bzDecompressInit(bzs, 0, 0)
-        if bzerror != BZ_OK:
-            _catch_bz2_error(space, bzerror)
-        
-        while True:
-            bzerror = BZ2_bzDecompress(bzs)
-            if bzerror == BZ_STREAM_END:
-                break
-            if bzerror != BZ_OK:
-                BZ2_bzDecompressEnd(bzs)
-                _catch_bz2_error(space, bzerror)
-        
-            if rffi.getintfield(bzs, 'c_avail_in') == 0:
+    with lltype.scoped_alloc(bz_stream.TO, zero=True) as bzs:
+        with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
+            for i in range(in_bufsize):
+                in_buf[i] = data[i]
+            bzs.c_next_in = in_buf
+            rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
+
+            with OutBuffer(bzs) as out:
+                bzerror = BZ2_bzDecompressInit(bzs, 0, 0)
+                if bzerror != BZ_OK:
+                    _catch_bz2_error(space, bzerror)
+
+                while True:
+                    bzerror = BZ2_bzDecompress(bzs)
+                    if bzerror == BZ_STREAM_END:
+                        break
+                    if bzerror != BZ_OK:
+                        BZ2_bzDecompressEnd(bzs)
+                    _catch_bz2_error(space, bzerror)
+
+                    if rffi.getintfield(bzs, 'c_avail_in') == 0:
+                        BZ2_bzDecompressEnd(bzs)
+                        raise OperationError(space.w_ValueError, space.wrap(
+                            "couldn't find end of stream"))
+                    elif rffi.getintfield(bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
+
+                res = out.make_result_string()
                 BZ2_bzDecompressEnd(bzs)
-                raise OperationError(space.w_ValueError,
-                                     space.wrap("couldn't find end of stream"))
-            elif rffi.getintfield(bzs, 'c_avail_out') == 0:
-                out.prepare_next_chunk()
-
-        res = out.make_result_string()
-        BZ2_bzDecompressEnd(bzs)
-        return space.wrap(res)
-    finally:
-        lltype.free(bzs, flavor='raw')
-        lltype.free(in_buf, flavor='raw')
-        out.free()
+                return space.wrap(res)
 decompress.unwrap_spec = [ObjSpace, 'bufferstr']


From afa at codespeak.net  Fri Oct 22 22:32:10 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 22 Oct 2010 22:32:10 +0200 (CEST)
Subject: [pypy-svn] r78227 -
	pypy/branch/fast-forward/pypy/module/exceptions/test
Message-ID: <20101022203210.D90AB282BDC@codespeak.net>

Author: afa
Date: Fri Oct 22 22:32:09 2010
New Revision: 78227

Modified:
   pypy/branch/fast-forward/pypy/module/exceptions/test/test_exc.py
Log:
A failing test that shows that the __doc__ attribute is broken.


Modified: pypy/branch/fast-forward/pypy/module/exceptions/test/test_exc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/exceptions/test/test_exc.py	(original)
+++ pypy/branch/fast-forward/pypy/module/exceptions/test/test_exc.py	Fri Oct 22 22:32:09 2010
@@ -227,6 +227,7 @@
             if isinstance(e, type) and issubclass(e, exceptions.BaseException):
                 assert e.__doc__, e
                 assert e.__module__ == 'exceptions', e
+        assert 'run-time' in RuntimeError.__doc__
 
     def test_reduce(self):
         from exceptions import LookupError, EnvironmentError


From afa at codespeak.net  Fri Oct 22 23:09:54 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 22 Oct 2010 23:09:54 +0200 (CEST)
Subject: [pypy-svn] r78228 - in pypy/branch/fast-forward: dotviewer
	lib-python lib_pypy/_ctypes pypy pypy/annotation
	pypy/annotation/test pypy/config pypy/doc pypy/doc/config
	pypy/interpreter pypy/interpreter/pyparser
	pypy/jit/backend/llgraph pypy/jit/backend/llsupport
	pypy/jit/backend/llsupport/test pypy/jit/backend/test
	pypy/jit/backend/x86 pypy/jit/backend/x86/test
	pypy/jit/codewriter pypy/jit/codewriter/test
	pypy/jit/metainterp pypy/jit/metainterp/optimizeopt
	pypy/jit/metainterp/test pypy/jit/tl pypy/jit/tool
	pypy/module/__builtin__ pypy/module/__builtin__/test
	pypy/module/_ffi pypy/module/_ffi/test pypy/module/_rawffi
	pypy/module/_rawffi/test pypy/module/_socket/test
	pypy/module/_sre pypy/module/_weakref pypy/module/_winreg
	pypy/module/array pypy/module/bz2 pypy/module/cpyext
	pypy/module/cpyext/test pypy/module/gc pypy/module/imp/test
	pypy/module/pypyjit pypy/module/pypyjit/test
	pypy/module/signal pypy/module/sys
	pypy/module/test_lib_pypy/ctypes_tests pypy/module/thread
	pypy/objspace/flow pypy/objspace/flow/test pypy/objspace/std
	pypy/objspace/std/test pypy/rlib pypy/rlib/rsre
	pypy/rlib/rsre/test pypy/rlib/test pypy/rpython
	pypy/rpython/lltypesystem pypy/rpython/lltypesystem/test
	pypy/rpython/memory pypy/rpython/memory/gc
	pypy/rpython/memory/gctransform pypy/rpython/memory/test
	pypy/rpython/module pypy/rpython/test pypy/tool
	pypy/tool/test pypy/translator pypy/translator/c
	pypy/translator/c/gcc pypy/translator/c/gcc/test/elf
	pypy/translator/c/gcc/test/elf64 pypy/translator/c/src
	pypy/translator/c/test pypy/translator/goal pypy/translator/jvm/test
	pypy/translator/oosupport/test_template pypy/translator/platform
Message-ID: <20101022210954.7F7A1282BDC@codespeak.net>

Author: afa
Date: Fri Oct 22 23:09:43 2010
New Revision: 78228

Added:
   pypy/branch/fast-forward/pypy/doc/config/objspace.std.withmapdict.txt
      - copied unchanged from r78227, pypy/trunk/pypy/doc/config/objspace.std.withmapdict.txt
   pypy/branch/fast-forward/pypy/doc/config/objspace.std.withstrbuf.txt
      - copied unchanged from r78227, pypy/trunk/pypy/doc/config/objspace.std.withstrbuf.txt
   pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._ffi.txt
      - copied unchanged from r78227, pypy/trunk/pypy/doc/config/objspace.usemodules._ffi.txt
   pypy/branch/fast-forward/pypy/jit/backend/llsupport/ffisupport.py
      - copied unchanged from r78227, pypy/trunk/pypy/jit/backend/llsupport/ffisupport.py
   pypy/branch/fast-forward/pypy/jit/backend/llsupport/test/test_ffisupport.py
      - copied unchanged from r78227, pypy/trunk/pypy/jit/backend/llsupport/test/test_ffisupport.py
   pypy/branch/fast-forward/pypy/jit/metainterp/greenfield.py
      - copied unchanged from r78227, pypy/trunk/pypy/jit/metainterp/greenfield.py
   pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/fficall.py
      - copied unchanged from r78227, pypy/trunk/pypy/jit/metainterp/optimizeopt/fficall.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_fficall.py
      - copied unchanged from r78227, pypy/trunk/pypy/jit/metainterp/test/test_fficall.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_greenfield.py
      - copied unchanged from r78227, pypy/trunk/pypy/jit/metainterp/test/test_greenfield.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_optimizefficall.py
      - copied unchanged from r78227, pypy/trunk/pypy/jit/metainterp/test/test_optimizefficall.py
   pypy/branch/fast-forward/pypy/jit/tool/oparser.py
      - copied unchanged from r78227, pypy/trunk/pypy/jit/tool/oparser.py
   pypy/branch/fast-forward/pypy/module/_ffi/
      - copied from r78227, pypy/trunk/pypy/module/_ffi/
   pypy/branch/fast-forward/pypy/module/_ffi/__init__.py
      - copied unchanged from r78227, pypy/trunk/pypy/module/_ffi/__init__.py
   pypy/branch/fast-forward/pypy/module/_ffi/interp_ffi.py
      - copied unchanged from r78227, pypy/trunk/pypy/module/_ffi/interp_ffi.py
   pypy/branch/fast-forward/pypy/module/_ffi/test/
      - copied from r78227, pypy/trunk/pypy/module/_ffi/test/
   pypy/branch/fast-forward/pypy/module/_ffi/test/__init__.py
      - copied unchanged from r78227, pypy/trunk/pypy/module/_ffi/test/__init__.py
   pypy/branch/fast-forward/pypy/module/_ffi/test/test__ffi.py
      - copied unchanged from r78227, pypy/trunk/pypy/module/_ffi/test/test__ffi.py
   pypy/branch/fast-forward/pypy/objspace/std/mapdict.py
      - copied unchanged from r78227, pypy/trunk/pypy/objspace/std/mapdict.py
   pypy/branch/fast-forward/pypy/objspace/std/strbufobject.py
      - copied unchanged from r78227, pypy/trunk/pypy/objspace/std/strbufobject.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_mapdict.py
      - copied unchanged from r78227, pypy/trunk/pypy/objspace/std/test/test_mapdict.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_strbufobject.py
      - copied unchanged from r78227, pypy/trunk/pypy/objspace/std/test/test_strbufobject.py
   pypy/branch/fast-forward/pypy/rlib/clibffi.py
      - copied unchanged from r78227, pypy/trunk/pypy/rlib/clibffi.py
   pypy/branch/fast-forward/pypy/rlib/libffi.py
      - copied unchanged from r78227, pypy/trunk/pypy/rlib/libffi.py
   pypy/branch/fast-forward/pypy/rlib/rerased.py
      - copied unchanged from r78227, pypy/trunk/pypy/rlib/rerased.py
   pypy/branch/fast-forward/pypy/rlib/rsre/rsre_jit.py
      - copied unchanged from r78227, pypy/trunk/pypy/rlib/rsre/rsre_jit.py
   pypy/branch/fast-forward/pypy/rlib/rsre/test/conftest.py
      - copied unchanged from r78227, pypy/trunk/pypy/rlib/rsre/test/conftest.py
   pypy/branch/fast-forward/pypy/rlib/rsre/test/test_zjit.py
      - copied unchanged from r78227, pypy/trunk/pypy/rlib/rsre/test/test_zjit.py
   pypy/branch/fast-forward/pypy/rlib/test/test_clibffi.py
      - copied unchanged from r78227, pypy/trunk/pypy/rlib/test/test_clibffi.py
   pypy/branch/fast-forward/pypy/rlib/test/test_libffi.py
      - copied unchanged from r78227, pypy/trunk/pypy/rlib/test/test_libffi.py
   pypy/branch/fast-forward/pypy/rlib/test/test_rerased.py
      - copied unchanged from r78227, pypy/trunk/pypy/rlib/test/test_rerased.py
   pypy/branch/fast-forward/pypy/tool/leakfinder.py
      - copied unchanged from r78227, pypy/trunk/pypy/tool/leakfinder.py
   pypy/branch/fast-forward/pypy/tool/test/test_leakfinder.py
      - copied unchanged from r78227, pypy/trunk/pypy/tool/test/test_leakfinder.py
   pypy/branch/fast-forward/pypy/translator/c/gcc/test/elf/track10.s
      - copied unchanged from r78227, pypy/trunk/pypy/translator/c/gcc/test/elf/track10.s
   pypy/branch/fast-forward/pypy/translator/c/gcc/test/elf/track11.s
      - copied unchanged from r78227, pypy/trunk/pypy/translator/c/gcc/test/elf/track11.s
   pypy/branch/fast-forward/pypy/translator/c/gcc/test/elf64/track_loadconst.s
      - copied unchanged from r78227, pypy/trunk/pypy/translator/c/gcc/test/elf64/track_loadconst.s
   pypy/branch/fast-forward/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s
      - copied unchanged from r78227, pypy/trunk/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s
   pypy/branch/fast-forward/pypy/translator/c/src/debug_alloc.h
      - copied unchanged from r78227, pypy/trunk/pypy/translator/c/src/debug_alloc.h
Removed:
   pypy/branch/fast-forward/pypy/jit/metainterp/test/oparser.py
   pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_find.py
   pypy/branch/fast-forward/pypy/rpython/rspecialcase.py
   pypy/branch/fast-forward/pypy/rpython/test/test_rspecialcase.py
Modified:
   pypy/branch/fast-forward/dotviewer/drawgraph.py
   pypy/branch/fast-forward/lib-python/conftest.py
   pypy/branch/fast-forward/lib_pypy/_ctypes/structure.py
   pypy/branch/fast-forward/lib_pypy/_ctypes/union.py
   pypy/branch/fast-forward/pypy/   (props changed)
   pypy/branch/fast-forward/pypy/annotation/builtin.py
   pypy/branch/fast-forward/pypy/annotation/model.py
   pypy/branch/fast-forward/pypy/annotation/policy.py
   pypy/branch/fast-forward/pypy/annotation/test/test_annrpython.py
   pypy/branch/fast-forward/pypy/annotation/test/test_model.py
   pypy/branch/fast-forward/pypy/config/pypyoption.py
   pypy/branch/fast-forward/pypy/conftest.py
   pypy/branch/fast-forward/pypy/doc/docindex.txt
   pypy/branch/fast-forward/pypy/interpreter/baseobjspace.py
   pypy/branch/fast-forward/pypy/interpreter/pycode.py
   pypy/branch/fast-forward/pypy/interpreter/pyopcode.py
   pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py
   pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py
   pypy/branch/fast-forward/pypy/interpreter/typedef.py
   pypy/branch/fast-forward/pypy/jit/backend/llgraph/llimpl.py
   pypy/branch/fast-forward/pypy/jit/backend/llgraph/runner.py
   pypy/branch/fast-forward/pypy/jit/backend/llsupport/descr.py
   pypy/branch/fast-forward/pypy/jit/backend/llsupport/gc.py
   pypy/branch/fast-forward/pypy/jit/backend/llsupport/llmodel.py
   pypy/branch/fast-forward/pypy/jit/backend/llsupport/symbolic.py
   pypy/branch/fast-forward/pypy/jit/backend/llsupport/test/test_descr.py
   pypy/branch/fast-forward/pypy/jit/backend/test/runner_test.py
   pypy/branch/fast-forward/pypy/jit/backend/test/test_ll_random.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/assembler.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/regalloc.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/regloc.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/runner.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/rx86.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_assembler.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_gc_integration.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_regalloc.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_runner.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_string.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_zrpy_gc.py
   pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_ztranslation.py
   pypy/branch/fast-forward/pypy/jit/codewriter/call.py
   pypy/branch/fast-forward/pypy/jit/codewriter/codewriter.py
   pypy/branch/fast-forward/pypy/jit/codewriter/effectinfo.py
   pypy/branch/fast-forward/pypy/jit/codewriter/jtransform.py
   pypy/branch/fast-forward/pypy/jit/codewriter/support.py
   pypy/branch/fast-forward/pypy/jit/codewriter/test/test_codewriter.py
   pypy/branch/fast-forward/pypy/jit/codewriter/test/test_effectinfo.py
   pypy/branch/fast-forward/pypy/jit/codewriter/test/test_jtransform.py
   pypy/branch/fast-forward/pypy/jit/metainterp/blackhole.py
   pypy/branch/fast-forward/pypy/jit/metainterp/compile.py
   pypy/branch/fast-forward/pypy/jit/metainterp/executor.py
   pypy/branch/fast-forward/pypy/jit/metainterp/graphpage.py
   pypy/branch/fast-forward/pypy/jit/metainterp/history.py
   pypy/branch/fast-forward/pypy/jit/metainterp/jitdriver.py
   pypy/branch/fast-forward/pypy/jit/metainterp/optimize_nopspec.py   (contents, props changed)
   pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/__init__.py
   pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/intbounds.py
   pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/rewrite.py
   pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/string.py
   pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/virtualize.py
   pypy/branch/fast-forward/pypy/jit/metainterp/pyjitpl.py
   pypy/branch/fast-forward/pypy/jit/metainterp/resoperation.py
   pypy/branch/fast-forward/pypy/jit/metainterp/resume.py
   pypy/branch/fast-forward/pypy/jit/metainterp/simple_optimize.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_basic.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_compile.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_jitdriver.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_logger.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_loop_nopspec.py   (props changed)
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_oparser.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_optimizefindnode.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_optimizeopt.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_resume.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_string.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_virtualref.py
   pypy/branch/fast-forward/pypy/jit/metainterp/test/test_warmspot.py
   pypy/branch/fast-forward/pypy/jit/metainterp/virtualref.py
   pypy/branch/fast-forward/pypy/jit/metainterp/warmspot.py
   pypy/branch/fast-forward/pypy/jit/metainterp/warmstate.py
   pypy/branch/fast-forward/pypy/jit/tl/jittest.py   (props changed)
   pypy/branch/fast-forward/pypy/jit/tl/pypyjit.py
   pypy/branch/fast-forward/pypy/jit/tl/pypyjit_child.py
   pypy/branch/fast-forward/pypy/jit/tool/loopviewer.py
   pypy/branch/fast-forward/pypy/jit/tool/showstats.py
   pypy/branch/fast-forward/pypy/jit/tool/traceviewer.py
   pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py
   pypy/branch/fast-forward/pypy/module/__builtin__/test/test_classobj.py
   pypy/branch/fast-forward/pypy/module/_rawffi/__init__.py
   pypy/branch/fast-forward/pypy/module/_rawffi/array.py
   pypy/branch/fast-forward/pypy/module/_rawffi/callback.py
   pypy/branch/fast-forward/pypy/module/_rawffi/interp_rawffi.py
   pypy/branch/fast-forward/pypy/module/_rawffi/structure.py
   pypy/branch/fast-forward/pypy/module/_rawffi/test/test__rawffi.py
   pypy/branch/fast-forward/pypy/module/_socket/test/test_sock_app.py
   pypy/branch/fast-forward/pypy/module/_sre/interp_sre.py
   pypy/branch/fast-forward/pypy/module/_weakref/interp__weakref.py
   pypy/branch/fast-forward/pypy/module/_winreg/interp_winreg.py
   pypy/branch/fast-forward/pypy/module/array/interp_array.py
   pypy/branch/fast-forward/pypy/module/bz2/interp_bz2.py
   pypy/branch/fast-forward/pypy/module/cpyext/classobject.py
   pypy/branch/fast-forward/pypy/module/cpyext/test/test_cpyext.py
   pypy/branch/fast-forward/pypy/module/cpyext/test/test_unicodeobject.py
   pypy/branch/fast-forward/pypy/module/gc/referents.py
   pypy/branch/fast-forward/pypy/module/imp/test/test_import.py
   pypy/branch/fast-forward/pypy/module/pypyjit/__init__.py
   pypy/branch/fast-forward/pypy/module/pypyjit/interp_jit.py
   pypy/branch/fast-forward/pypy/module/pypyjit/policy.py
   pypy/branch/fast-forward/pypy/module/pypyjit/test/test_pypy_c.py
   pypy/branch/fast-forward/pypy/module/signal/interp_signal.py
   pypy/branch/fast-forward/pypy/module/sys/__init__.py
   pypy/branch/fast-forward/pypy/module/sys/state.py
   pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c
   pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py
   pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py
   pypy/branch/fast-forward/pypy/module/thread/ll_thread.py
   pypy/branch/fast-forward/pypy/objspace/flow/flowcontext.py
   pypy/branch/fast-forward/pypy/objspace/flow/test/test_objspace.py
   pypy/branch/fast-forward/pypy/objspace/std/celldict.py
   pypy/branch/fast-forward/pypy/objspace/std/dictmultiobject.py
   pypy/branch/fast-forward/pypy/objspace/std/fake.py
   pypy/branch/fast-forward/pypy/objspace/std/model.py
   pypy/branch/fast-forward/pypy/objspace/std/objspace.py
   pypy/branch/fast-forward/pypy/objspace/std/sharingdict.py
   pypy/branch/fast-forward/pypy/objspace/std/stringobject.py
   pypy/branch/fast-forward/pypy/objspace/std/stringtype.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_dictmultiobject.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_shadowtracking.py
   pypy/branch/fast-forward/pypy/objspace/std/typeobject.py
   pypy/branch/fast-forward/pypy/rlib/_rsocket_rffi.py
   pypy/branch/fast-forward/pypy/rlib/jit.py
   pypy/branch/fast-forward/pypy/rlib/rgc.py
   pypy/branch/fast-forward/pypy/rlib/rmmap.py
   pypy/branch/fast-forward/pypy/rlib/rsre/rsre_char.py
   pypy/branch/fast-forward/pypy/rlib/rsre/rsre_core.py
   pypy/branch/fast-forward/pypy/rlib/rsre/test/test_match.py
   pypy/branch/fast-forward/pypy/rlib/rstring.py
   pypy/branch/fast-forward/pypy/rlib/test/test_jit.py
   pypy/branch/fast-forward/pypy/rlib/test/test_rdynload.py
   pypy/branch/fast-forward/pypy/rlib/test/test_rsocket.py
   pypy/branch/fast-forward/pypy/rlib/test/test_rstring.py
   pypy/branch/fast-forward/pypy/rlib/test/test_rzlib.py
   pypy/branch/fast-forward/pypy/rpython/llinterp.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/ll2ctypes.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/llarena.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/llmemory.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/lloperation.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/lltype.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/rbuilder.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/rclass.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/rffi.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_ll2ctypes.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_llmemory.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_lltype.py
   pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_rffi.py
   pypy/branch/fast-forward/pypy/rpython/memory/gc/minimarkpage.py
   pypy/branch/fast-forward/pypy/rpython/memory/gctransform/asmgcroot.py
   pypy/branch/fast-forward/pypy/rpython/memory/gctransform/transform.py
   pypy/branch/fast-forward/pypy/rpython/memory/gcwrapper.py
   pypy/branch/fast-forward/pypy/rpython/memory/support.py
   pypy/branch/fast-forward/pypy/rpython/memory/test/test_gc.py
   pypy/branch/fast-forward/pypy/rpython/module/ll_time.py
   pypy/branch/fast-forward/pypy/rpython/rbuilder.py
   pypy/branch/fast-forward/pypy/rpython/rbuiltin.py
   pypy/branch/fast-forward/pypy/rpython/rpbc.py
   pypy/branch/fast-forward/pypy/rpython/rtyper.py
   pypy/branch/fast-forward/pypy/rpython/test/test_llinterp.py
   pypy/branch/fast-forward/pypy/rpython/test/test_nongc.py
   pypy/branch/fast-forward/pypy/rpython/test/test_rbuilder.py
   pypy/branch/fast-forward/pypy/rpython/test/test_rclass.py
   pypy/branch/fast-forward/pypy/rpython/test/test_rptr.py
   pypy/branch/fast-forward/pypy/translator/c/funcgen.py
   pypy/branch/fast-forward/pypy/translator/c/gcc/instruction.py
   pypy/branch/fast-forward/pypy/translator/c/gcc/test/elf/track5.s
   pypy/branch/fast-forward/pypy/translator/c/gcc/trackgcroot.py
   pypy/branch/fast-forward/pypy/translator/c/src/g_include.h
   pypy/branch/fast-forward/pypy/translator/c/src/main.h
   pypy/branch/fast-forward/pypy/translator/c/src/signals.h
   pypy/branch/fast-forward/pypy/translator/c/src/stack.h
   pypy/branch/fast-forward/pypy/translator/c/test/test_genc.py
   pypy/branch/fast-forward/pypy/translator/c/test/test_lltyped.py
   pypy/branch/fast-forward/pypy/translator/c/test/test_newgc.py
   pypy/branch/fast-forward/pypy/translator/c/test/test_standalone.py
   pypy/branch/fast-forward/pypy/translator/c/test/test_typed.py
   pypy/branch/fast-forward/pypy/translator/goal/ann_override.py
   pypy/branch/fast-forward/pypy/translator/goal/app_main.py
   pypy/branch/fast-forward/pypy/translator/jvm/test/test_class.py
   pypy/branch/fast-forward/pypy/translator/oosupport/test_template/class_.py
   pypy/branch/fast-forward/pypy/translator/platform/darwin.py
   pypy/branch/fast-forward/pypy/translator/transform.py
Log:
Merge from trunk:
svn merge -r77543:78227 ../trunk/ .


Modified: pypy/branch/fast-forward/dotviewer/drawgraph.py
==============================================================================
--- pypy/branch/fast-forward/dotviewer/drawgraph.py	(original)
+++ pypy/branch/fast-forward/dotviewer/drawgraph.py	Fri Oct 22 23:09:43 2010
@@ -423,20 +423,43 @@
         else:
             for line in lines:
                 raw_line = line.replace('\\l','').replace('\r','') or ' '
-                img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
-                w, h = img.get_size()
-                if w>wmax: wmax = w
-                if raw_line.strip():
-                    if line.endswith('\\l'):
-                        def cmd(img=img, y=hmax):
-                            img.draw(xleft, ytop+y)
-                    elif line.endswith('\r'):
-                        def cmd(img=img, y=hmax, w=w):
-                            img.draw(xright-w, ytop+y)
-                    else:
-                        def cmd(img=img, y=hmax, w=w):
-                            img.draw(xcenter-w//2, ytop+y)
+                if '\f' in raw_line:   # grayed out parts of the line
+                    imgs = []
+                    graytext = True
+                    h = 16
+                    w_total = 0
+                    for linepart in raw_line.split('\f'):
+                        graytext = not graytext
+                        if not linepart.strip():
+                            continue
+                        if graytext:
+                            fgcolor = (128, 160, 160)
+                        else:
+                            fgcolor = (0, 0, 0)
+                        img = TextSnippet(self, linepart, fgcolor, bgcolor)
+                        imgs.append((w_total, img))
+                        w, h = img.get_size()
+                        w_total += w
+                    if w_total > wmax: wmax = w_total
+                    def cmd(imgs=imgs, y=hmax):
+                        for x, img in imgs:
+                            img.draw(xleft+x, ytop+y)
                     commands.append(cmd)
+                else:
+                    img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
+                    w, h = img.get_size()
+                    if w>wmax: wmax = w
+                    if raw_line.strip():
+                        if line.endswith('\\l'):
+                            def cmd(img=img, y=hmax):
+                                img.draw(xleft, ytop+y)
+                        elif line.endswith('\r'):
+                            def cmd(img=img, y=hmax, w=w):
+                                img.draw(xright-w, ytop+y)
+                        else:
+                            def cmd(img=img, y=hmax, w=w):
+                                img.draw(xcenter-w//2, ytop+y)
+                        commands.append(cmd)
                 hmax += h
                 #hmax += 8
 

Modified: pypy/branch/fast-forward/lib-python/conftest.py
==============================================================================
--- pypy/branch/fast-forward/lib-python/conftest.py	(original)
+++ pypy/branch/fast-forward/lib-python/conftest.py	Fri Oct 22 23:09:43 2010
@@ -496,7 +496,12 @@
     RegrTest('test_coding.py'),
     RegrTest('test_complex_args.py'),
     RegrTest('test_contextlib.py', usemodules="thread"),
-    RegrTest('test_ctypes.py', usemodules="_rawffi"),
+    # we skip test ctypes, since we adapted it massively in order
+    # to test what we want to support. There are real failures,
+    # but it's about missing features that we don't want to support
+    # now
+    RegrTest('test_ctypes.py', usemodules="_rawffi",
+             skip="missing features that we don't want to support now"),
     RegrTest('test_defaultdict.py'),
     RegrTest('test_email_renamed.py'),
     RegrTest('test_exception_variations.py'),

Modified: pypy/branch/fast-forward/lib_pypy/_ctypes/structure.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/_ctypes/structure.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/_ctypes/structure.py	Fri Oct 22 23:09:43 2010
@@ -7,7 +7,7 @@
 def round_up(size, alignment):
     return (size + alignment - 1) & -alignment
 
-def size_alignment_pos(fields):
+def size_alignment_pos(fields, is_union=False):
     import ctypes
     size = 0
     alignment = 1
@@ -15,14 +15,19 @@
     for fieldname, ctype in fields:
         fieldsize = ctypes.sizeof(ctype)
         fieldalignment = ctypes.alignment(ctype)
-        size = round_up(size, fieldalignment)
         alignment = max(alignment, fieldalignment)
-        pos.append(size)
-        size += fieldsize
+        if is_union:
+            pos.append(0)
+            size = max(size, fieldsize)
+        else:
+            size = round_up(size, fieldalignment)
+            pos.append(size)
+            size += fieldsize
     size = round_up(size, alignment)
     return size, alignment, pos
 
-def names_and_fields(_fields_, superclass, zero_offset=False, anon=None):
+def names_and_fields(_fields_, superclass, zero_offset=False, anon=None,
+                     is_union=False):
     if isinstance(_fields_, tuple):
         _fields_ = list(_fields_)
     for _, tp in _fields_:
@@ -36,7 +41,7 @@
     rawfields = [(name, ctype._ffishape)
                  for name, ctype in all_fields]
     if not zero_offset:
-        _, _, pos = size_alignment_pos(all_fields)
+        _, _, pos = size_alignment_pos(all_fields, is_union)
     else:
         pos = [0] * len(all_fields)
     fields = {}
@@ -73,8 +78,8 @@
 
 # ________________________________________________________________
 
-def _set_shape(tp, rawfields):
-    tp._ffistruct = _rawffi.Structure(rawfields)
+def _set_shape(tp, rawfields, is_union=False):
+    tp._ffistruct = _rawffi.Structure(rawfields, is_union)
     tp._ffiargshape = tp._ffishape = (tp._ffistruct, 1)
     tp._fficompositesize = tp._ffistruct.size
 
@@ -92,13 +97,14 @@
             raise AttributeError("Structure or union cannot contain itself")
         self._names, rawfields, self._fieldtypes = names_and_fields(
             value, self.__bases__[0], False,
-            self.__dict__.get('_anonymous_', None))
+            self.__dict__.get('_anonymous_', None), self._is_union)
         _CDataMeta.__setattr__(self, '_fields_', value)
-        _set_shape(self, rawfields)
+        _set_shape(self, rawfields, self._is_union)
         return
     _CDataMeta.__setattr__(self, name, value)
 
-class StructureMeta(_CDataMeta):
+class StructOrUnionMeta(_CDataMeta):
+
     def __new__(self, name, cls, typedict):
         res = type.__new__(self, name, cls, typedict)
         if '_fields_' in typedict:
@@ -109,8 +115,8 @@
                     raise AttributeError("Anonymous field not found")
             res._names, rawfields, res._fieldtypes = names_and_fields(
                 typedict['_fields_'], cls[0], False,
-                typedict.get('_anonymous_', None))
-            _set_shape(res, rawfields)
+                typedict.get('_anonymous_', None), self._is_union)
+            _set_shape(res, rawfields, self._is_union)
 
         return res
 
@@ -150,8 +156,8 @@
         res.__dict__['_index'] = -1
         return res
 
-class Structure(_CData):
-    __metaclass__ = StructureMeta
+class StructOrUnion(_CData):
+    __metaclass__ = StructOrUnionMeta
 
     def __new__(cls, *args, **kwds):
         if not hasattr(cls, '_ffistruct'):
@@ -213,3 +219,10 @@
 
     def _get_buffer_value(self):
         return self._buffer.buffer
+
+
+class StructureMeta(StructOrUnionMeta):
+    _is_union = False
+
+class Structure(StructOrUnion):
+    __metaclass__ = StructureMeta

Modified: pypy/branch/fast-forward/lib_pypy/_ctypes/union.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/_ctypes/union.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/_ctypes/union.py	Fri Oct 22 23:09:43 2010
@@ -1,118 +1,7 @@
+from _ctypes import structure
 
+class UnionMeta(structure.StructOrUnionMeta):
+    _is_union = True
 
-import _rawffi
-from _ctypes.basics import _CData, _CDataMeta, store_reference, keepalive_key
-from _ctypes.basics import ensure_objects
-from _ctypes.structure import round_up, names_and_fields, struct_getattr,\
-     struct_setattr
-
-
-def _set_shape(tp):
-    size = tp._sizeofinstances()
-    alignment = tp._alignmentofinstances()
-    tp._ffiopaque = _rawffi.Structure((size, alignment)) # opaque
-    tp._ffiargshape = tp._ffishape = (tp._ffiopaque, 1)
-    tp._fficompositesize = tp._ffiopaque.size
-    # we need to create an array of size one for each
-    # of our elements
-    tp._ffiarrays = {}
-    for name, field in tp._fieldtypes.iteritems():
-        tp._ffiarrays[name] = _rawffi.Array(field.ctype._ffishape)
-        
-class UnionMeta(_CDataMeta):
-    def __new__(self, name, cls, typedict):
-        res = type.__new__(self, name, cls, typedict)
-        if '_fields_' in typedict:
-            res._names, rawfields, res._fieldtypes = names_and_fields(
-                typedict['_fields_'], cls[0], True,
-                typedict.get('_anonymous_', None))
-            _set_shape(res)
-
-        def __init__(self): # don't allow arguments by now
-            if not hasattr(self, '_ffiarrays'):
-                raise TypeError("Cannot instantiate union, has no type")
-            # malloc size
-            size = self.__class__._sizeofinstances()
-            self.__dict__['_objects'] = {}
-            self.__dict__['_buffer'] = self._ffiopaque(autofree=True)
-        res.__init__ = __init__
-        return res
-
-    def _sizeofinstances(self):
-        if not hasattr(self, '_size_'):
-            self._size_ = max([field.size for field in
-                               self._fieldtypes.values()] + [0])
-        return self._size_
-
-    def _alignmentofinstances(self):
-        from ctypes import alignment
-        if not hasattr(self, '_alignment_'):
-            self._alignment_ = max([alignment(field.ctype) for field in
-                                    self._fieldtypes.values()] + [1])
-        return self._alignment_
-    
-    __getattr__ = struct_getattr
-
-    def __setattr__(self, name, value):
-        if name == '_fields_':
-            if self.__dict__.get('_fields_', None):
-                raise AttributeError("_fields_ is final")
-            if self in [v for k, v in value]:
-                raise AttributeError("Union cannot contain itself")
-            self._names, rawfields, self._fieldtypes = names_and_fields(
-                value, self.__bases__[0], True,
-                self.__dict__.get('_anonymous_', None))
-            _CDataMeta.__setattr__(self, '_fields_', value)
-            _set_shape(self)
-        _CDataMeta.__setattr__(self, name, value)
-
-    def _CData_output(self, resarray, base=None, index=-1):
-        res = self.__new__(self)
-        ffiopaque = self._ffiopaque.fromaddress(resarray.buffer)
-        res.__dict__['_buffer'] = ffiopaque
-        res.__dict__['_base'] = base
-        res.__dict__['_index'] = index
-        return res
-    
-    def _CData_retval(self, resbuffer):
-        res = self.__new__(self)
-        res.__dict__['_buffer'] = resbuffer
-        res.__dict__['_base'] = None
-        res.__dict__['_index'] = -1
-        return res
-
-
-class Union(_CData):
+class Union(structure.StructOrUnion):
     __metaclass__ = UnionMeta
-
-    def __getattr__(self, name):
-        try:
-            field = self._fieldtypes[name]
-        except KeyError:
-            raise AttributeError(name)
-        fieldtype = field.ctype
-        val = self._ffiarrays[name].fromaddress(self._buffer.buffer, 1)
-        offset = field.num
-        return fieldtype._CData_output(val, self, offset)
-
-    def __setattr__(self, name, value):
-        try:
-            field = self._fieldtypes[name]
-        except KeyError:
-            raise AttributeError(name)
-        fieldtype = field.ctype
-        cobj = fieldtype.from_param(value)
-        if ensure_objects(cobj) is not None:
-            key = keepalive_key(field.num)
-            store_reference(self, key, cobj._objects)
-        arg = cobj._get_buffer_value()
-        if fieldtype._fficompositesize is not None:
-            from ctypes import memmove
-            dest = self._buffer.buffer
-            memmove(dest, arg, fieldtype._fficompositesize)
-        else:
-            buf = self._ffiarrays[name].fromaddress(self._buffer.buffer, 1)
-            buf[0] = arg
-
-    def _get_buffer_value(self):
-        return self._buffer.buffer

Modified: pypy/branch/fast-forward/pypy/annotation/builtin.py
==============================================================================
--- pypy/branch/fast-forward/pypy/annotation/builtin.py	(original)
+++ pypy/branch/fast-forward/pypy/annotation/builtin.py	Fri Oct 22 23:09:43 2010
@@ -423,7 +423,7 @@
 from pypy.annotation.model import SomePtr
 from pypy.rpython.lltypesystem import lltype
 
-def malloc(s_T, s_n=None, s_flavor=None, s_zero=None):
+def malloc(s_T, s_n=None, s_flavor=None, s_zero=None, s_track_allocation=None):
     assert (s_n is None or s_n.knowntype == int
             or issubclass(s_n.knowntype, pypy.rlib.rarithmetic.base_int))
     assert s_T.is_constant()
@@ -438,13 +438,15 @@
         r = SomePtr(lltype.typeOf(p))
     else:
         assert s_flavor.is_constant()
+        assert s_track_allocation is None or s_track_allocation.is_constant()
         # not sure how to call malloc() for the example 'p' in the
         # presence of s_extraargs
         r = SomePtr(lltype.Ptr(s_T.const))
     return r
 
-def free(s_p, s_flavor):
+def free(s_p, s_flavor, s_track_allocation=None):
     assert s_flavor.is_constant()
+    assert s_track_allocation is None or s_track_allocation.is_constant()
     # same problem as in malloc(): some flavors are not easy to
     # malloc-by-example
     #T = s_p.ll_ptrtype.TO

Modified: pypy/branch/fast-forward/pypy/annotation/model.py
==============================================================================
--- pypy/branch/fast-forward/pypy/annotation/model.py	(original)
+++ pypy/branch/fast-forward/pypy/annotation/model.py	Fri Oct 22 23:09:43 2010
@@ -584,11 +584,11 @@
         
 NUMBER = object()
 annotation_to_ll_map = [
+    (SomeSingleFloat(), lltype.SingleFloat),
     (s_None, lltype.Void),   # also matches SomeImpossibleValue()
     (s_Bool, lltype.Bool),
     (SomeInteger(knowntype=r_ulonglong), NUMBER),    
     (SomeFloat(), lltype.Float),
-    (SomeSingleFloat(), lltype.SingleFloat),
     (SomeLongFloat(), lltype.LongFloat),
     (SomeChar(), lltype.Char),
     (SomeUnicodeCodePoint(), lltype.UniChar),

Modified: pypy/branch/fast-forward/pypy/annotation/policy.py
==============================================================================
--- pypy/branch/fast-forward/pypy/annotation/policy.py	(original)
+++ pypy/branch/fast-forward/pypy/annotation/policy.py	Fri Oct 22 23:09:43 2010
@@ -1,4 +1,4 @@
-# base annotation policy for overrides and specialization
+# base annotation policy for specialization
 from pypy.annotation.specialize import default_specialize as default
 from pypy.annotation.specialize import specialize_argvalue, specialize_argtype, specialize_arglistitemtype
 from pypy.annotation.specialize import memo
@@ -41,7 +41,7 @@
         if directive is None:
             return pol.default_specialize
 
-        # specialize|override:name[(args)]
+        # specialize[(args)]
         directive_parts = directive.split('(', 1)
         if len(directive_parts) == 1:
             [name] = directive_parts
@@ -60,14 +60,6 @@
         except AttributeError:
             raise AttributeError("%r specialize tag not defined in annotation"
                                  "policy %s" % (name, pol))
-        if directive.startswith('override:'):
-            # different signature: override__xyz(*args_s)
-            if parms:
-                raise Exception, "override:* specialisations don't support parameters"
-            def specialize_override(funcdesc, args_s):
-                funcdesc.overridden = True
-                return specializer(*args_s)
-            return specialize_override
         else:
             if not parms:
                 return specializer
@@ -92,9 +84,5 @@
         from pypy.rpython.annlowlevel import LowLevelAnnotatorPolicy
         return LowLevelAnnotatorPolicy.specialize__ll_and_arg(*args)
 
-    def override__ignore(pol, *args):
-        bk = getbookkeeper()
-        return bk.immutablevalue(None)
-
 class StrictAnnotatorPolicy(AnnotatorPolicy):
     allow_someobjects = False

Modified: pypy/branch/fast-forward/pypy/annotation/test/test_annrpython.py
==============================================================================
--- pypy/branch/fast-forward/pypy/annotation/test/test_annrpython.py	(original)
+++ pypy/branch/fast-forward/pypy/annotation/test/test_annrpython.py	Fri Oct 22 23:09:43 2010
@@ -766,28 +766,6 @@
         s = a.build_types(f, [list])
         assert s.classdef is a.bookkeeper.getuniqueclassdef(IndexError)  # KeyError ignored because l is a list
 
-    def test_overrides(self):
-        excs = []
-        def record_exc(e):
-            """NOT_RPYTHON"""
-            excs.append(sys.exc_info)
-        record_exc._annspecialcase_ = "override:record_exc"
-        def g():
-            pass
-        def f():
-            try:
-                g()
-            except Exception, e:
-                record_exc(e)
-        class MyAnnotatorPolicy(policy.AnnotatorPolicy):
-
-            def override__record_exc(pol, s_e):
-                return a.bookkeeper.immutablevalue(None)
-            
-        a = self.RPythonAnnotator(policy=MyAnnotatorPolicy())
-        s = a.build_types(f, [])
-        assert s.const is None
-
     def test_freeze_protocol(self):
         class Stuff:
             def __init__(self, flag):
@@ -3359,6 +3337,26 @@
         s = a.build_types(f, [])
         assert isinstance(s, annmodel.SomeChar)
 
+    def test_context_manager(self):
+        class C:
+            def __init__(self):
+                pass
+            def __enter__(self):
+                self.x = 1
+            def __exit__(self, *args):
+                self.x = 3
+        def f():
+            c = C()
+            with c:
+                pass
+            return c.x
+
+        a = self.RPythonAnnotator()
+        s = a.build_types(f, [])
+        assert isinstance(s, annmodel.SomeInteger)
+        # not a constant: both __enter__ and __exit__ have been annotated
+        assert not s.is_constant()
+
 
 def g(n):
     return [0,1,2,n]

Modified: pypy/branch/fast-forward/pypy/annotation/test/test_model.py
==============================================================================
--- pypy/branch/fast-forward/pypy/annotation/test/test_model.py	(original)
+++ pypy/branch/fast-forward/pypy/annotation/test/test_model.py	Fri Oct 22 23:09:43 2010
@@ -128,7 +128,7 @@
     assert isinstance(s_p, SomeOOInstance) and s_p.ootype == C
 
 def test_annotation_to_lltype():
-    from pypy.rlib.rarithmetic import r_uint
+    from pypy.rlib.rarithmetic import r_uint, r_singlefloat
     s_i = SomeInteger()
     s_pos = SomeInteger(nonneg=True)
     s_1 = SomeInteger(nonneg=True); s_1.const = 1
@@ -151,6 +151,9 @@
     C = ootype.Instance('C', ROOT, {})
     ref = SomeOOInstance(C)
     assert annotation_to_lltype(ref) == C
+    s_singlefloat = SomeSingleFloat()
+    s_singlefloat.const = r_singlefloat(0.0)
+    assert annotation_to_lltype(s_singlefloat) == lltype.SingleFloat
     
 def test_ll_union():
     PS1 = lltype.Ptr(lltype.GcStruct('s'))

Modified: pypy/branch/fast-forward/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/fast-forward/pypy/config/pypyoption.py	(original)
+++ pypy/branch/fast-forward/pypy/config/pypyoption.py	Fri Oct 22 23:09:43 2010
@@ -73,9 +73,9 @@
     }
 
 module_import_dependencies = {
-    # no _rawffi if importing pypy.rlib.libffi raises ImportError
+    # no _rawffi if importing pypy.rlib.clibffi raises ImportError
     # or CompilationError
-    "_rawffi"   : ["pypy.rlib.libffi"],
+    "_rawffi"   : ["pypy.rlib.clibffi"],
 
     "zlib"      : ["pypy.rlib.rzlib"],
     "bz2"       : ["pypy.module.bz2.interp_bz2"],
@@ -198,6 +198,9 @@
         BoolOption("withstrslice", "use strings optimized for slicing",
                    default=False),
 
+        BoolOption("withstrbuf", "use strings optimized for addition (ver 2)",
+                   default=False),
+
         BoolOption("withprebuiltchar",
                    "use prebuilt single-character string objects",
                    default=False),
@@ -210,7 +213,8 @@
         BoolOption("withrope", "use ropes as the string implementation",
                    default=False,
                    requires=[("objspace.std.withstrslice", False),
-                             ("objspace.std.withstrjoin", False)],
+                             ("objspace.std.withstrjoin", False),
+                             ("objspace.std.withstrbuf", False)],
                    suggests=[("objspace.std.withprebuiltchar", True),
                              ("objspace.std.sharesmallstr", True)]),
 
@@ -238,6 +242,16 @@
                    default=False,
                    requires=[("objspace.std.withshadowtracking", False)]),
 
+        BoolOption("withmapdict",
+                   "make instances really small but slow without the JIT",
+                   default=False,
+                   requires=[("objspace.std.withshadowtracking", False),
+                             ("objspace.std.withinlineddict", False),
+                             ("objspace.std.withsharingdict", False),
+                             ("objspace.std.getattributeshortcut", True),
+                             ("objspace.std.withtypeversion", True),
+                       ]),
+
         BoolOption("withrangelist",
                    "enable special range list implementation that does not "
                    "actually create the full list until the resulting "
@@ -343,7 +357,7 @@
         config.objspace.std.suggest(withprebuiltint=True)
         config.objspace.std.suggest(withrangelist=True)
         config.objspace.std.suggest(withprebuiltchar=True)
-        config.objspace.std.suggest(withinlineddict=True)
+        config.objspace.std.suggest(withmapdict=True)
         config.objspace.std.suggest(withstrslice=True)
         config.objspace.std.suggest(withstrjoin=True)
         # xxx other options? ropes maybe?
@@ -359,6 +373,7 @@
     # extra optimizations with the JIT
     if level == 'jit':
         config.objspace.std.suggest(withcelldict=True)
+        #config.objspace.std.suggest(withmapdict=True)
 
 
 def enable_allworkingmodules(config):

Modified: pypy/branch/fast-forward/pypy/conftest.py
==============================================================================
--- pypy/branch/fast-forward/pypy/conftest.py	(original)
+++ pypy/branch/fast-forward/pypy/conftest.py	Fri Oct 22 23:09:43 2010
@@ -7,6 +7,7 @@
 from inspect import isclass, getmro
 from pypy.tool.udir import udir
 from pypy.tool.autopath import pypydir
+from pypy.tool import leakfinder
 
 # pytest settings
 pytest_plugins = "resultlog",
@@ -358,7 +359,14 @@
 
     def runtest(self):
         try:
-            super(IntTestFunction, self).runtest()
+            leakfinder.start_tracking_allocations()
+            try:
+                super(IntTestFunction, self).runtest()
+            finally:
+                if leakfinder.TRACK_ALLOCATIONS:
+                    leaks = leakfinder.stop_tracking_allocations(False)
+                else:
+                    leaks = None   # stop_tracking_allocations() already called
         except OperationError, e:
             check_keyboard_interrupt(e)
             raise
@@ -377,6 +385,8 @@
                 _pygame_imported = True
                 assert option.view, ("should not invoke Pygame "
                                      "if conftest.option.view is False")
+        if leaks:        # check for leaks, but only if the test passed so far
+            raise leakfinder.MallocMismatch(leaks)
 
 class AppTestFunction(PyPyTestFunction):
     def _prunetraceback(self, traceback):

Modified: pypy/branch/fast-forward/pypy/doc/docindex.txt
==============================================================================
--- pypy/branch/fast-forward/pypy/doc/docindex.txt	(original)
+++ pypy/branch/fast-forward/pypy/doc/docindex.txt	Fri Oct 22 23:09:43 2010
@@ -84,7 +84,7 @@
 
 PyPy's own tests `summary`_, daily updated, run through BuildBot infrastructure.
 You can also find CPython's compliance tests run with compiled ``pypy-c``
-exeuctables there.
+executables there.
 
 information dating from early 2007: 
 

Modified: pypy/branch/fast-forward/pypy/interpreter/baseobjspace.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/baseobjspace.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/baseobjspace.py	Fri Oct 22 23:09:43 2010
@@ -168,6 +168,20 @@
     def _call_builtin_destructor(self):
         pass     # method overridden in typedef.py
 
+    # hooks that the mapdict implementations needs:
+    def _get_mapdict_map(self):
+        return None
+    def _set_mapdict_map(self, map):
+        raise NotImplementedError
+    def _mapdict_read_storage(self, index):
+        raise NotImplementedError
+    def _mapdict_write_storage(self, index, value):
+        raise NotImplementedError
+    def _mapdict_storage_length(self):
+        raise NotImplementedError
+    def _set_mapdict_storage_and_map(self, storage, map):
+        raise NotImplementedError
+
 
 class Wrappable(W_Root):
     """A subclass of Wrappable is an internal, interpreter-level class

Modified: pypy/branch/fast-forward/pypy/interpreter/pycode.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pycode.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pycode.py	Fri Oct 22 23:09:43 2010
@@ -117,6 +117,10 @@
 
         self._compute_flatcall()
 
+        if self.space.config.objspace.std.withmapdict:
+            from pypy.objspace.std.mapdict import init_mapdict_cache
+            init_mapdict_cache(self)
+
     def _freeze_(self):
         if (self.magic == cpython_magic and
             '__pypy__' not in sys.builtin_module_names):

Modified: pypy/branch/fast-forward/pypy/interpreter/pyopcode.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyopcode.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyopcode.py	Fri Oct 22 23:09:43 2010
@@ -164,6 +164,9 @@
             next_instr = block.handle(self, unroller)
             return next_instr
 
+    def call_contextmanager_exit_function(self, w_func, w_typ, w_val, w_tb):
+        return self.space.call_function(w_func, w_typ, w_val, w_tb)
+
     @jit.unroll_safe
     def dispatch_bytecode(self, co_code, next_instr, ec):
         space = self.space
@@ -710,9 +713,14 @@
 
     def LOAD_ATTR(self, nameindex, next_instr):
         "obj.attributename"
-        w_attributename = self.getname_w(nameindex)
         w_obj = self.popvalue()
-        w_value = self.space.getattr(w_obj, w_attributename)
+        if (self.space.config.objspace.std.withmapdict
+            and not jit.we_are_jitted()):
+            from pypy.objspace.std.mapdict import LOAD_ATTR_caching
+            w_value = LOAD_ATTR_caching(self.getcode(), w_obj, nameindex)
+        else:
+            w_attributename = self.getname_w(nameindex)
+            w_value = self.space.getattr(w_obj, w_attributename)
         self.pushvalue(w_value)
     LOAD_ATTR._always_inline_ = True
 
@@ -904,28 +912,47 @@
         self.pushvalue(w_result)
 
     def WITH_CLEANUP(self, oparg, next_instr):
-        self.dropvalues(2)
-        w_unroller = self.popvalue()
+        # see comment in END_FINALLY for stack state
+        # This opcode changed a lot between CPython versions
+        if (self.pycode.magic >= 0xa0df2ef
+            # Implementation since 2.7a0: 62191 (introduce SETUP_WITH)
+            or self.pycode.magic >= 0xa0df2d1):
+            # implementation since 2.6a1: 62161 (WITH_CLEANUP optimization)
+            self.popvalue()
+            self.popvalue()
+            w_unroller = self.popvalue()
+            w_exitfunc = self.popvalue()
+            self.pushvalue(w_unroller)
+            self.pushvalue(self.space.w_None)
+            self.pushvalue(self.space.w_None)
+        elif self.pycode.magic >= 0xa0df28c:
+            # Implementation since 2.5a0: 62092 (changed WITH_CLEANUP opcode)
+            w_exitfunc = self.popvalue()
+            w_unroller = self.peekvalue(2)
+        else:
+            raise NotImplementedError("WITH_CLEANUP for CPython <= 2.4")
+
         unroller = self.space.interpclass_w(w_unroller)
         w_exit = self.popvalue()
         is_app_exc = (unroller is not None and
                       isinstance(unroller, SApplicationException))
         if is_app_exc:
             operr = unroller.operr
-            w_type = operr.w_type
-            w_value = operr.get_w_value(self.space)
-            w_tb = self.space.wrap(operr.application_traceback)
-        else:
-            w_type = w_value = w_tb = self.space.w_None
-        w_suppress = self.space.call_function(w_exit, w_type, w_value, w_tb)
-        if is_app_exc and self.space.is_true(w_suppress):
-            self.pushvalue(self.space.w_None)
-            self.pushvalue(self.space.w_None)
-            self.pushvalue(self.space.w_None)
+            w_traceback = self.space.wrap(operr.application_traceback)
+            w_suppress = self.call_contextmanager_exit_function(
+                w_exitfunc,
+                operr.w_type,
+                operr.get_w_value(self.space),
+                w_traceback)
+            if self.space.is_true(w_suppress):
+                # __exit__() returned True -> Swallow the exception.
+                self.settopvalue(self.space.w_None, 2)
         else:
-            self.pushvalue(w_unroller)
-            self.pushvalue(w_value)
-            self.pushvalue(w_type)
+            self.call_contextmanager_exit_function(
+                w_exitfunc,
+                self.space.w_None,
+                self.space.w_None,
+                self.space.w_None)
 
     @jit.unroll_safe
     def call_function(self, oparg, w_star=None, w_starstar=None):
@@ -1345,10 +1372,10 @@
             unroller.operr.normalize_exception(frame.space)
         return FinallyBlock.really_handle(self, frame, unroller)
 
-
 block_classes = {'SETUP_LOOP': LoopBlock,
                  'SETUP_EXCEPT': ExceptBlock,
-                 'SETUP_FINALLY': FinallyBlock}
+                 'SETUP_FINALLY': FinallyBlock,
+                 'SETUP_WITH': WithBlock}
 
 ### helpers written at the application-level ###
 # Some of these functions are expected to be generally useful if other

Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py	Fri Oct 22 23:09:43 2010
@@ -268,7 +268,8 @@
         i += 1
     import StringIO
     print "states = ["
-    for state in dfa.states:
+    for numstate, state in enumerate(dfa.states):
+        print "    #", numstate
         s = StringIO.StringIO()
         i = 0
         for k, v in sorted(state.items()):
@@ -284,14 +285,17 @@
                 s.write(', ')
         s.write('},')
         i = 0
-        for line in textwrap.wrap(s.getvalue(), width=35):
+        if len(state) <= 4:
+            text = [s.getvalue()]
+        else:
+            text = textwrap.wrap(s.getvalue(), width=36)
+        for line in text:
             line = line.replace('::', ': ')
             if i == 0:
                 print '    {' + line
             else:
                 print '     ' + line
             i += 1
-        print
     print "    ]"
     print "%s = automata.%s(states, accepts)" % (name, dfa_class)
     print
@@ -302,16 +306,8 @@
     endDFAMap = makePyEndDFAMap()
     output("double3DFA", "NonGreedyDFA", endDFAMap['"""'])
     output("single3DFA", "NonGreedyDFA", endDFAMap["'''"])
-    output("doubleDFA", "DFA", endDFAMap['"'])
     output("singleDFA", "DFA", endDFAMap["'"])
-    print "endDFAs = {\"'\" : singleDFA,"
-    print "           '\"' : doubleDFA,"
-    print "           'r' : None,"
-    print "           'R' : None,"
-    print "           'u' : None,"
-    print "           'U' : None,"
-    print "           'b' : None,"
-    print "           'B' : None}"
+    output("doubleDFA", "DFA", endDFAMap['"'])
 
 # ______________________________________________________________________
 

Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py	Fri Oct 22 23:09:43 2010
@@ -30,6 +30,7 @@
            True, False, False, False, False, True, False,
            False, False, True]
 states = [
+    # 0
     {'\t': 0, '\n': 13, '\x0c': 0,
      '\r': 14, ' ': 0, '!': 10, '"': 16,
      '#': 18, '%': 12, '&': 12, "'": 15,
@@ -55,7 +56,7 @@
      'u': 2, 'v': 1, 'w': 1, 'x': 1,
      'y': 1, 'z': 1, '{': 13, '|': 12,
      '}': 13, '~': 13},
-
+    # 1
     {'0': 1, '1': 1, '2': 1, '3': 1,
      '4': 1, '5': 1, '6': 1, '7': 1,
      '8': 1, '9': 1, 'A': 1, 'B': 1,
@@ -72,7 +73,7 @@
      'p': 1, 'q': 1, 'r': 1, 's': 1,
      't': 1, 'u': 1, 'v': 1, 'w': 1,
      'x': 1, 'y': 1, 'z': 1},
-
+    # 2
     {'"': 16, "'": 15, '0': 1, '1': 1,
      '2': 1, '3': 1, '4': 1, '5': 1,
      '6': 1, '7': 1, '8': 1, '9': 1,
@@ -90,7 +91,7 @@
      'r': 3, 's': 1, 't': 1, 'u': 1,
      'v': 1, 'w': 1, 'x': 1, 'y': 1,
      'z': 1},
-
+    # 3
     {'"': 16, "'": 15, '0': 1, '1': 1,
      '2': 1, '3': 1, '4': 1, '5': 1,
      '6': 1, '7': 1, '8': 1, '9': 1,
@@ -108,215 +109,188 @@
      'r': 1, 's': 1, 't': 1, 'u': 1,
      'v': 1, 'w': 1, 'x': 1, 'y': 1,
      'z': 1},
-
+    # 4
     {'.': 24, '0': 21, '1': 21, '2': 21,
      '3': 21, '4': 21, '5': 21, '6': 21,
      '7': 21, '8': 23, '9': 23, 'B': 22,
      'E': 25, 'J': 13, 'L': 13, 'O': 20,
      'X': 19, 'b': 22, 'e': 25, 'j': 13,
      'l': 13, 'o': 20, 'x': 19},
-
+    # 5
     {'.': 24, '0': 5, '1': 5, '2': 5,
      '3': 5, '4': 5, '5': 5, '6': 5,
      '7': 5, '8': 5, '9': 5, 'E': 25,
      'J': 13, 'L': 13, 'e': 25, 'j': 13,
      'l': 13},
-
+    # 6
     {'0': 26, '1': 26, '2': 26, '3': 26,
      '4': 26, '5': 26, '6': 26, '7': 26,
      '8': 26, '9': 26},
-
+    # 7
     {'*': 12, '=': 13},
-
+    # 8
     {'=': 13, '>': 12},
-
+    # 9
     {'<': 12, '=': 13, '>': 13},
-
+    # 10
     {'=': 13},
-
+    # 11
     {'/': 12, '=': 13},
-
+    # 12
     {'=': 13},
-
+    # 13
     {},
-
+    # 14
     {'\n': 13},
-
-    {automata.DEFAULT: 30, '\n': 27,
-     "'": 28, '\\': 29},
-
-    {automata.DEFAULT: 33, '\n': 27,
-     '"': 31, '\\': 32},
-
+    # 15
+    {automata.DEFAULT: 30, '\n': 27, "'": 28, '\\': 29},
+    # 16
+    {automata.DEFAULT: 33, '\n': 27, '"': 31, '\\': 32},
+    # 17
     {'\n': 13, '\r': 14},
-
-    {automata.DEFAULT: 18, '\n': 27,
-     '\r': 27},
-
+    # 18
+    {automata.DEFAULT: 18, '\n': 27, '\r': 27},
+    # 19
     {'0': 19, '1': 19, '2': 19, '3': 19,
      '4': 19, '5': 19, '6': 19, '7': 19,
      '8': 19, '9': 19, 'A': 19, 'B': 19,
      'C': 19, 'D': 19, 'E': 19, 'F': 19,
      'L': 13, 'a': 19, 'b': 19, 'c': 19,
-     'd': 19, 'e': 19, 'f': 19,
-     'l': 13},
-
+     'd': 19, 'e': 19, 'f': 19, 'l': 13},
+    # 20
     {'0': 20, '1': 20, '2': 20, '3': 20,
      '4': 20, '5': 20, '6': 20, '7': 20,
      'L': 13, 'l': 13},
-
+    # 21
     {'.': 24, '0': 21, '1': 21, '2': 21,
      '3': 21, '4': 21, '5': 21, '6': 21,
      '7': 21, '8': 23, '9': 23, 'E': 25,
      'J': 13, 'L': 13, 'e': 25, 'j': 13,
      'l': 13},
-
-    {'0': 22, '1': 22, 'L': 13,
-     'l': 13},
-
+    # 22
+    {'0': 22, '1': 22, 'L': 13, 'l': 13},
+    # 23
     {'.': 24, '0': 23, '1': 23, '2': 23,
      '3': 23, '4': 23, '5': 23, '6': 23,
      '7': 23, '8': 23, '9': 23, 'E': 25,
      'J': 13, 'e': 25, 'j': 13},
-
+    # 24
     {'0': 24, '1': 24, '2': 24, '3': 24,
      '4': 24, '5': 24, '6': 24, '7': 24,
      '8': 24, '9': 24, 'E': 34, 'J': 13,
      'e': 34, 'j': 13},
-
+    # 25
     {'+': 35, '-': 35, '0': 36, '1': 36,
      '2': 36, '3': 36, '4': 36, '5': 36,
-     '6': 36, '7': 36, '8': 36,
-     '9': 36},
-
+     '6': 36, '7': 36, '8': 36, '9': 36},
+    # 26
     {'0': 26, '1': 26, '2': 26, '3': 26,
      '4': 26, '5': 26, '6': 26, '7': 26,
      '8': 26, '9': 26, 'E': 34, 'J': 13,
      'e': 34, 'j': 13},
-
+    # 27
     {},
-
+    # 28
     {"'": 13},
-
-    {automata.DEFAULT: 37, '\n': 13,
-     '\r': 14},
-
-    {automata.DEFAULT: 30, '\n': 27,
-     "'": 13, '\\': 29},
-
+    # 29
+    {automata.DEFAULT: 37, '\n': 13, '\r': 14},
+    # 30
+    {automata.DEFAULT: 30, '\n': 27, "'": 13, '\\': 29},
+    # 31
     {'"': 13},
-
-    {automata.DEFAULT: 38, '\n': 13,
-     '\r': 14},
-
-    {automata.DEFAULT: 33, '\n': 27,
-     '"': 13, '\\': 32},
-
+    # 32
+    {automata.DEFAULT: 38, '\n': 13, '\r': 14},
+    # 33
+    {automata.DEFAULT: 33, '\n': 27, '"': 13, '\\': 32},
+    # 34
     {'+': 39, '-': 39, '0': 40, '1': 40,
      '2': 40, '3': 40, '4': 40, '5': 40,
-     '6': 40, '7': 40, '8': 40,
-     '9': 40},
-
+     '6': 40, '7': 40, '8': 40, '9': 40},
+    # 35
     {'0': 36, '1': 36, '2': 36, '3': 36,
      '4': 36, '5': 36, '6': 36, '7': 36,
      '8': 36, '9': 36},
-
+    # 36
     {'0': 36, '1': 36, '2': 36, '3': 36,
      '4': 36, '5': 36, '6': 36, '7': 36,
-     '8': 36, '9': 36, 'J': 13,
-     'j': 13},
-
-    {automata.DEFAULT: 37, '\n': 27,
-     "'": 13, '\\': 29},
-
-    {automata.DEFAULT: 38, '\n': 27,
-     '"': 13, '\\': 32},
-
+     '8': 36, '9': 36, 'J': 13, 'j': 13},
+    # 37
+    {automata.DEFAULT: 37, '\n': 27, "'": 13, '\\': 29},
+    # 38
+    {automata.DEFAULT: 38, '\n': 27, '"': 13, '\\': 32},
+    # 39
     {'0': 40, '1': 40, '2': 40, '3': 40,
      '4': 40, '5': 40, '6': 40, '7': 40,
      '8': 40, '9': 40},
-
+    # 40
     {'0': 40, '1': 40, '2': 40, '3': 40,
      '4': 40, '5': 40, '6': 40, '7': 40,
-     '8': 40, '9': 40, 'J': 13,
-     'j': 13},
-
+     '8': 40, '9': 40, 'J': 13, 'j': 13},
     ]
 pseudoDFA = automata.DFA(states, accepts)
 
 accepts = [False, False, False, False, False, True]
 states = [
-    {automata.DEFAULT: 0, '"': 1,
-     '\\': 2},
-
-    {automata.DEFAULT: 4, '"': 3,
-     '\\': 2},
-
+    # 0
+    {automata.DEFAULT: 0, '"': 1, '\\': 2},
+    # 1
+    {automata.DEFAULT: 4, '"': 3, '\\': 2},
+    # 2
     {automata.DEFAULT: 4},
-
-    {automata.DEFAULT: 4, '"': 5,
-     '\\': 2},
-
-    {automata.DEFAULT: 4, '"': 1,
-     '\\': 2},
-
-    {automata.DEFAULT: 4, '"': 5,
-     '\\': 2},
-
+    # 3
+    {automata.DEFAULT: 4, '"': 5, '\\': 2},
+    # 4
+    {automata.DEFAULT: 4, '"': 1, '\\': 2},
+    # 5
+    {automata.DEFAULT: 4, '"': 5, '\\': 2},
     ]
 double3DFA = automata.NonGreedyDFA(states, accepts)
 
 accepts = [False, False, False, False, False, True]
 states = [
-    {automata.DEFAULT: 0, "'": 1,
-     '\\': 2},
-
-    {automata.DEFAULT: 4, "'": 3,
-     '\\': 2},
-
+    # 0
+    {automata.DEFAULT: 0, "'": 1, '\\': 2},
+    # 1
+    {automata.DEFAULT: 4, "'": 3, '\\': 2},
+    # 2
     {automata.DEFAULT: 4},
-
-    {automata.DEFAULT: 4, "'": 5,
-     '\\': 2},
-
-    {automata.DEFAULT: 4, "'": 1,
-     '\\': 2},
-
-    {automata.DEFAULT: 4, "'": 5,
-     '\\': 2},
-
+    # 3
+    {automata.DEFAULT: 4, "'": 5, '\\': 2},
+    # 4
+    {automata.DEFAULT: 4, "'": 1, '\\': 2},
+    # 5
+    {automata.DEFAULT: 4, "'": 5, '\\': 2},
     ]
 single3DFA = automata.NonGreedyDFA(states, accepts)
 
 accepts = [False, True, False, False]
 states = [
-    {automata.DEFAULT: 0, '"': 1,
-     '\\': 2},
-
+    # 0
+    {automata.DEFAULT: 0, "'": 1, '\\': 2},
+    # 1
     {},
-
+    # 2
     {automata.DEFAULT: 3},
-
-    {automata.DEFAULT: 3, '"': 1,
-     '\\': 2},
-
+    # 3
+    {automata.DEFAULT: 3, "'": 1, '\\': 2},
     ]
-doubleDFA = automata.DFA(states, accepts)
+singleDFA = automata.DFA(states, accepts)
 
 accepts = [False, True, False, False]
 states = [
-    {automata.DEFAULT: 0, "'": 1,
-     '\\': 2},
-
+    # 0
+    {automata.DEFAULT: 0, '"': 1, '\\': 2},
+    # 1
     {},
-
+    # 2
     {automata.DEFAULT: 3},
-
-    {automata.DEFAULT: 3, "'": 1,
-     '\\': 2},
-
+    # 3
+    {automata.DEFAULT: 3, '"': 1, '\\': 2},
     ]
-singleDFA = automata.DFA(states, accepts)
+doubleDFA = automata.DFA(states, accepts)
+
+#_______________________________________________________________________
+# End of automatically generated DFA's
 
 endDFAs = {"'" : singleDFA,
            '"' : doubleDFA,

Modified: pypy/branch/fast-forward/pypy/interpreter/typedef.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/typedef.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/typedef.py	Fri Oct 22 23:09:43 2010
@@ -127,6 +127,13 @@
     typedef = cls.typedef
     if wants_dict and typedef.hasdict:
         wants_dict = False
+    if config.objspace.std.withmapdict and not typedef.hasdict:
+        # mapdict only works if the type does not already have a dict
+        if wants_del:
+            parentcls = get_unique_interplevel_subclass(config, cls, True, True,
+                                                        False, True)
+            return _usersubclswithfeature(config, parentcls, "del")
+        return _usersubclswithfeature(config, cls, "user", "dict", "weakref", "slots")
     # Forest of if's - see the comment above.
     if wants_del:
         if wants_dict:
@@ -180,10 +187,20 @@
 
     def add(Proto):
         for key, value in Proto.__dict__.items():
-            if not key.startswith('__') or key == '__del__':
+            if (not key.startswith('__') and not key.startswith('_mixin_') 
+                    or key == '__del__'):
+                if hasattr(value, "func_name"):
+                    value = func_with_new_name(value, value.func_name)
                 body[key] = value
 
+    if (config.objspace.std.withmapdict and "dict" in features):
+        from pypy.objspace.std.mapdict import BaseMapdictObject, ObjectMixin
+        add(BaseMapdictObject)
+        add(ObjectMixin)
+        features = ()
+
     if "user" in features:     # generic feature needed by all subcls
+
         class Proto(object):
             user_overridden_class = True
 
@@ -249,6 +266,9 @@
             wantdict = False
 
     if wantdict:
+        base_user_setup = supercls.user_setup.im_func
+        if "user_setup" in body:
+            base_user_setup = body["user_setup"]
         class Proto(object):
             def getdict(self):
                 return self.w__dict__
@@ -257,11 +277,9 @@
                 self.w__dict__ = check_new_dictionary(space, w_dict)
             
             def user_setup(self, space, w_subtype):
-                self.space = space
-                self.w__class__ = w_subtype
                 self.w__dict__ = space.newdict(
                     instance=True, classofinstance=w_subtype)
-                self.user_setup_slots(w_subtype.nslots)
+                base_user_setup(self, space, w_subtype)
 
             def setclass(self, space, w_subtype):
                 # only used by descr_set___class__

Modified: pypy/branch/fast-forward/pypy/jit/backend/llgraph/llimpl.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/llgraph/llimpl.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/llgraph/llimpl.py	Fri Oct 22 23:09:43 2010
@@ -10,7 +10,7 @@
                                          BoxInt, BoxPtr, BoxObj, BoxFloat,
                                          REF, INT, FLOAT)
 from pypy.jit.codewriter import heaptracker
-from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr
+from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr, rffi
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython.module.support import LLSupport, OOSupport
 from pypy.rpython.llinterp import LLException
@@ -305,12 +305,12 @@
     loop = _from_opaque(loop)
     loop.operations.append(Operation(opnum))
 
-def compile_add_descr(loop, ofs, type):
+def compile_add_descr(loop, ofs, type, arg_types):
     from pypy.jit.backend.llgraph.runner import Descr
     loop = _from_opaque(loop)
     op = loop.operations[-1]
     assert isinstance(type, str) and len(type) == 1
-    op.descr = Descr(ofs, type)
+    op.descr = Descr(ofs, type, arg_types=arg_types)
 
 def compile_add_loop_token(loop, descr):
     if we_are_translated():
@@ -801,7 +801,7 @@
             else:
                 raise TypeError(x)
         try:
-            return _do_call_common(func, args_in_order)
+            return _do_call_common(func, args_in_order, calldescr)
         except LLException, lle:
             _last_exception = lle
             d = {'v': None,
@@ -1018,6 +1018,9 @@
     if isinstance(TYPE, lltype.Ptr):
         if isinstance(x, (int, long, llmemory.AddressAsInt)):
             x = llmemory.cast_int_to_adr(x)
+        if TYPE is rffi.VOIDP:
+            # assume that we want a "C-style" cast, without typechecking the value
+            return rffi.cast(TYPE, x)
         return llmemory.cast_adr_to_ptr(x, TYPE)
     elif TYPE == llmemory.Address:
         if isinstance(x, (int, long, llmemory.AddressAsInt)):
@@ -1411,10 +1414,26 @@
 def do_call_pushfloat(x):
     _call_args_f.append(x)
 
-def _do_call_common(f, args_in_order=None):
+kind2TYPE = {
+    'i': lltype.Signed,
+    'f': lltype.Float,
+    'v': lltype.Void,
+    }
+
+def _do_call_common(f, args_in_order=None, calldescr=None):
     ptr = llmemory.cast_int_to_adr(f).ptr
-    FUNC = lltype.typeOf(ptr).TO
-    ARGS = FUNC.ARGS
+    PTR = lltype.typeOf(ptr)
+    if PTR == rffi.VOIDP:
+        # it's a pointer to a C function, so we don't have a precise
+        # signature: create one from the descr
+        ARGS = map(kind2TYPE.get, calldescr.arg_types)
+        RESULT = kind2TYPE[calldescr.typeinfo]
+        FUNC = lltype.FuncType(ARGS, RESULT)
+        func_to_call = rffi.cast(lltype.Ptr(FUNC), ptr)
+    else:
+        FUNC = PTR.TO
+        ARGS = FUNC.ARGS
+        func_to_call = ptr._obj._callable
     args = cast_call_args(ARGS, _call_args_i, _call_args_r, _call_args_f,
                           args_in_order)
     del _call_args_i[:]
@@ -1426,7 +1445,7 @@
         result = llinterp.eval_graph(ptr._obj.graph, args)
         # ^^^ may raise, in which case we get an LLException
     else:
-        result = ptr._obj._callable(*args)
+        result = func_to_call(*args)
     return result
 
 def do_call_void(f):

Modified: pypy/branch/fast-forward/pypy/jit/backend/llgraph/runner.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/llgraph/runner.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/llgraph/runner.py	Fri Oct 22 23:09:43 2010
@@ -154,7 +154,7 @@
             llimpl.compile_add(c, op.getopnum())
             descr = op.getdescr()
             if isinstance(descr, Descr):
-                llimpl.compile_add_descr(c, descr.ofs, descr.typeinfo)
+                llimpl.compile_add_descr(c, descr.ofs, descr.typeinfo, descr.arg_types)
             if isinstance(descr, history.LoopToken) and op.getopnum() != rop.JUMP:
                 llimpl.compile_add_loop_token(c, descr)
             if self.is_oo and isinstance(descr, (OODescr, MethDescr)):
@@ -297,6 +297,18 @@
         return self.getdescr(0, token[0], extrainfo=extrainfo,
                              arg_types=''.join(arg_types))
 
+    def calldescrof_dynamic(self, ffi_args, ffi_result, extrainfo=None):
+        from pypy.jit.backend.llsupport.ffisupport import get_ffi_type_kind
+        arg_types = []
+        for arg in ffi_args:
+            kind = get_ffi_type_kind(arg)
+            if kind != history.VOID:
+                arg_types.append(kind)
+        reskind = get_ffi_type_kind(ffi_result)
+        return self.getdescr(0, reskind, extrainfo=extrainfo,
+                             arg_types=''.join(arg_types))
+
+
     def grab_exc_value(self):
         return llimpl.grab_exc_value()
 

Modified: pypy/branch/fast-forward/pypy/jit/backend/llsupport/descr.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/llsupport/descr.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/llsupport/descr.py	Fri Oct 22 23:09:43 2010
@@ -82,6 +82,7 @@
 
     _is_pointer_field = False   # unless overridden by GcPtrFieldDescr
     _is_float_field = False     # unless overridden by FloatFieldDescr
+    _is_field_signed = False    # unless overridden by XxxFieldDescr
 
     def is_pointer_field(self):
         return self._is_pointer_field
@@ -89,6 +90,9 @@
     def is_float_field(self):
         return self._is_float_field
 
+    def is_field_signed(self):
+        return self._is_field_signed
+
     def repr_of_descr(self):
         return '<%s %s %s>' % (self._clsname, self.name, self.offset)
 
@@ -105,7 +109,7 @@
 def getFieldDescrClass(TYPE):
     return getDescrClass(TYPE, BaseFieldDescr, GcPtrFieldDescr,
                          NonGcPtrFieldDescr, 'Field', 'get_field_size',
-                         '_is_float_field')
+                         '_is_float_field', '_is_field_signed')
 
 def get_field_descr(gccache, STRUCT, fieldname):
     cache = gccache._cache_field
@@ -144,6 +148,7 @@
 
     _is_array_of_pointers = False      # unless overridden by GcPtrArrayDescr
     _is_array_of_floats   = False      # unless overridden by FloatArrayDescr
+    _is_item_signed       = False      # unless overridden by XxxArrayDescr
 
     def is_array_of_pointers(self):
         return self._is_array_of_pointers
@@ -151,6 +156,9 @@
     def is_array_of_floats(self):
         return self._is_array_of_floats
 
+    def is_item_signed(self):
+        return self._is_item_signed
+
     def repr_of_descr(self):
         return '<%s>' % self._clsname
 
@@ -186,12 +194,12 @@
 def getArrayDescrClass(ARRAY):
     return getDescrClass(ARRAY.OF, BaseArrayDescr, GcPtrArrayDescr,
                          NonGcPtrArrayDescr, 'Array', 'get_item_size',
-                         '_is_array_of_floats')
+                         '_is_array_of_floats', '_is_item_signed')
 
 def getArrayNoLengthDescrClass(ARRAY):
     return getDescrClass(ARRAY.OF, BaseArrayNoLengthDescr, GcPtrArrayNoLengthDescr,
                          NonGcPtrArrayNoLengthDescr, 'ArrayNoLength', 'get_item_size',
-                         '_is_array_of_floats')
+                         '_is_array_of_floats', '_is_item_signed')
 
 def get_array_descr(gccache, ARRAY):
     cache = gccache._cache_array
@@ -242,6 +250,9 @@
     def get_result_size(self, translate_support_code):
         raise NotImplementedError
 
+    def is_result_signed(self):
+        return False    # unless overridden
+
     def create_call_stub(self, rtyper, RESULT):
         def process(c):
             arg = 'args_%s[%d]' % (c, seen[c])
@@ -307,6 +318,30 @@
     _return_type = history.INT
     call_stub = staticmethod(lambda func, args_i, args_r, args_f: 0)
 
+    _is_result_signed = False      # can be overridden in XxxCallDescr
+    def is_result_signed(self):
+        return self._is_result_signed
+
+class DynamicIntCallDescr(BaseIntCallDescr):
+    """
+    calldescr that works for every integer type, by explicitly passing it the
+    size of the result. Used only by get_call_descr_dynamic
+    """
+    _clsname = 'DynamicIntCallDescr'
+
+    def __init__(self, arg_classes, result_size, result_sign, extrainfo=None):
+        BaseIntCallDescr.__init__(self, arg_classes, extrainfo)
+        assert isinstance(result_sign, bool)
+        self._result_size = chr(result_size)
+        self._result_sign = result_sign
+
+    def get_result_size(self, translate_support_code):
+        return ord(self._result_size)
+
+    def is_result_signed(self):
+        return self._result_sign
+
+
 class NonGcPtrCallDescr(BaseIntCallDescr):
     _clsname = 'NonGcPtrCallDescr'
     def get_result_size(self, translate_support_code):
@@ -341,7 +376,8 @@
         return FloatCallDescr
     return getDescrClass(RESULT, BaseIntCallDescr, GcPtrCallDescr,
                          NonGcPtrCallDescr, 'Call', 'get_result_size',
-                         Ellipsis)  # <= floatattrname should not be used here
+                         Ellipsis,  # <= floatattrname should not be used here
+                         '_is_result_signed')
 
 def get_call_descr(gccache, ARGS, RESULT, extrainfo=None):
     arg_classes = []
@@ -368,7 +404,8 @@
 # ____________________________________________________________
 
 def getDescrClass(TYPE, BaseDescr, GcPtrDescr, NonGcPtrDescr,
-                  nameprefix, methodname, floatattrname, _cache={}):
+                  nameprefix, methodname, floatattrname, signedattrname,
+                  _cache={}):
     if isinstance(TYPE, lltype.Ptr):
         if TYPE.TO._gckind == 'gc':
             return GcPtrDescr
@@ -388,6 +425,8 @@
         #
         if TYPE is lltype.Float:
             setattr(Descr, floatattrname, True)
+        elif TYPE is not lltype.Bool and rffi.cast(TYPE, -1) == -1:
+            setattr(Descr, signedattrname, True)
         #
         _cache[nameprefix, TYPE] = Descr
         return Descr

Modified: pypy/branch/fast-forward/pypy/jit/backend/llsupport/gc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/llsupport/gc.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/llsupport/gc.py	Fri Oct 22 23:09:43 2010
@@ -158,7 +158,7 @@
         # used to avoid too many duplications in the GCREF_LISTs.
         self.hashtable = lltype.malloc(self.HASHTABLE,
                                        self.HASHTABLE_SIZE+1,
-                                       flavor='raw')
+                                       flavor='raw', track_allocation=False)
         dummy = lltype.direct_ptradd(lltype.direct_arrayitems(self.hashtable),
                                      self.HASHTABLE_SIZE)
         dummy = llmemory.cast_ptr_to_adr(dummy)
@@ -252,14 +252,15 @@
 
     def _enlarge_gcmap(self):
         newlength = 250 + self._gcmap_maxlength * 2
-        newgcmap = lltype.malloc(self.GCMAP_ARRAY, newlength, flavor='raw')
+        newgcmap = lltype.malloc(self.GCMAP_ARRAY, newlength, flavor='raw',
+                                 track_allocation=False)
         oldgcmap = self._gcmap
         for i in range(self._gcmap_curlength):
             newgcmap[i] = oldgcmap[i]
         self._gcmap = newgcmap
         self._gcmap_maxlength = newlength
         if oldgcmap:
-            lltype.free(oldgcmap, flavor='raw')
+            lltype.free(oldgcmap, flavor='raw', track_allocation=False)
 
     def get_basic_shape(self, is_64_bit=False):
         # XXX: Should this code even really know about stack frame layout of
@@ -308,7 +309,8 @@
         # them inside bigger arrays) and we never try to share them.
         length = len(shape)
         compressed = lltype.malloc(self.CALLSHAPE_ARRAY, length,
-                                   flavor='raw')
+                                   flavor='raw',
+                                   track_allocation=False)   # memory leak
         for i in range(length):
             compressed[length-1-i] = rffi.cast(rffi.UCHAR, shape[i])
         return llmemory.cast_ptr_to_adr(compressed)

Modified: pypy/branch/fast-forward/pypy/jit/backend/llsupport/llmodel.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/llsupport/llmodel.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/llsupport/llmodel.py	Fri Oct 22 23:09:43 2010
@@ -17,6 +17,7 @@
 from pypy.jit.backend.llsupport.descr import get_call_descr
 from pypy.jit.backend.llsupport.descr import BaseIntCallDescr, GcPtrCallDescr
 from pypy.jit.backend.llsupport.descr import FloatCallDescr, VoidCallDescr
+from pypy.jit.backend.llsupport.ffisupport import get_call_descr_dynamic
 from pypy.rpython.annlowlevel import cast_instance_to_base_ptr
 
 
@@ -82,7 +83,8 @@
         # read back by the machine code reading at the address given by
         # pos_exception() and pos_exc_value().
         _exception_emulator = lltype.malloc(rffi.CArray(lltype.Signed), 2,
-                                            zero=True, flavor='raw')
+                                            zero=True, flavor='raw',
+                                            immortal=True)
         self._exception_emulator = _exception_emulator
 
         def _store_exception(lle):
@@ -210,7 +212,8 @@
         assert isinstance(fielddescr, BaseFieldDescr)
         ofs = fielddescr.offset
         size = fielddescr.get_field_size(self.translate_support_code)
-        return ofs, size
+        sign = fielddescr.is_field_signed()
+        return ofs, size, sign
     unpack_fielddescr_size._always_inline_ = True
 
     def arraydescrof(self, A):
@@ -225,12 +228,16 @@
         assert isinstance(arraydescr, BaseArrayDescr)
         ofs = arraydescr.get_base_size(self.translate_support_code)
         size = arraydescr.get_item_size(self.translate_support_code)
-        return ofs, size
+        sign = arraydescr.is_item_signed()
+        return ofs, size, sign
     unpack_arraydescr_size._always_inline_ = True
 
     def calldescrof(self, FUNC, ARGS, RESULT, extrainfo=None):
         return get_call_descr(self.gc_ll_descr, ARGS, RESULT, extrainfo)
 
+    def calldescrof_dynamic(self, ffi_args, ffi_result, extrainfo=None):
+        return get_call_descr_dynamic(ffi_args, ffi_result, extrainfo)
+
     def get_overflow_error(self):
         ovf_vtable = self.cast_adr_to_int(self._ovf_error_vtable)
         ovf_inst = lltype.cast_opaque_ptr(llmemory.GCREF,
@@ -252,15 +259,21 @@
 
     @specialize.argtype(2)
     def bh_getarrayitem_gc_i(self, arraydescr, gcref, itemindex):
-        ofs, size = self.unpack_arraydescr_size(arraydescr)
+        ofs, size, sign = self.unpack_arraydescr_size(arraydescr)
         # --- start of GC unsafe code (no GC operation!) ---
         items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for STYPE, UTYPE, itemsize in unroll_basic_sizes:
             if size == itemsize:
-                items = rffi.cast(rffi.CArrayPtr(TYPE), items) 
-                val = items[itemindex]
+                if sign:
+                    items = rffi.cast(rffi.CArrayPtr(STYPE), items)
+                    val = items[itemindex]
+                    val = rffi.cast(lltype.Signed, val)
+                else:
+                    items = rffi.cast(rffi.CArrayPtr(UTYPE), items)
+                    val = items[itemindex]
+                    val = rffi.cast(lltype.Signed, val)
                 # --- end of GC unsafe code ---
-                return rffi.cast(lltype.Signed, val)
+                return val
         else:
             raise NotImplementedError("size = %d" % size)
 
@@ -285,10 +298,10 @@
 
     @specialize.argtype(2)
     def bh_setarrayitem_gc_i(self, arraydescr, gcref, itemindex, newvalue):
-        ofs, size = self.unpack_arraydescr_size(arraydescr)
+        ofs, size, sign = self.unpack_arraydescr_size(arraydescr)
         # --- start of GC unsafe code (no GC operation!) ---
         items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for TYPE, _, itemsize in unroll_basic_sizes:
             if size == itemsize:
                 items = rffi.cast(rffi.CArrayPtr(TYPE), items)
                 items[itemindex] = rffi.cast(TYPE, newvalue)
@@ -339,14 +352,22 @@
 
     @specialize.argtype(1)
     def _base_do_getfield_i(self, struct, fielddescr):
-        ofs, size = self.unpack_fielddescr_size(fielddescr)
+        ofs, size, sign = self.unpack_fielddescr_size(fielddescr)
         # --- start of GC unsafe code (no GC operation!) ---
         fieldptr = rffi.ptradd(rffi.cast(rffi.CCHARP, struct), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for STYPE, UTYPE, itemsize in unroll_basic_sizes:
             if size == itemsize:
-                val = rffi.cast(rffi.CArrayPtr(TYPE), fieldptr)[0]
+                # Note that in the common case where size==sizeof(Signed),
+                # both cases of what follows are doing the same thing.
+                # But gcc is clever enough to figure this out :-)
+                if sign:
+                    val = rffi.cast(rffi.CArrayPtr(STYPE), fieldptr)[0]
+                    val = rffi.cast(lltype.Signed, val)
+                else:
+                    val = rffi.cast(rffi.CArrayPtr(UTYPE), fieldptr)[0]
+                    val = rffi.cast(lltype.Signed, val)
                 # --- end of GC unsafe code ---
-                return rffi.cast(lltype.Signed, val)
+                return val
         else:
             raise NotImplementedError("size = %d" % size)
 
@@ -378,10 +399,10 @@
 
     @specialize.argtype(1)
     def _base_do_setfield_i(self, struct, fielddescr, newvalue):
-        ofs, size = self.unpack_fielddescr_size(fielddescr)
+        ofs, size, sign = self.unpack_fielddescr_size(fielddescr)
         # --- start of GC unsafe code (no GC operation!) ---
         fieldptr = rffi.ptradd(rffi.cast(rffi.CCHARP, struct), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for TYPE, _, itemsize in unroll_basic_sizes:
             if size == itemsize:
                 fieldptr = rffi.cast(rffi.CArrayPtr(TYPE), fieldptr)
                 fieldptr[0] = rffi.cast(TYPE, newvalue)

Modified: pypy/branch/fast-forward/pypy/jit/backend/llsupport/symbolic.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/llsupport/symbolic.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/llsupport/symbolic.py	Fri Oct 22 23:09:43 2010
@@ -69,8 +69,9 @@
 SIZEOF_INT   = get_size(rffi.INT, False)
 SIZEOF_FLOAT = get_size(lltype.Float, False)
 
-unroll_basic_sizes = unrolling_iterable([(lltype.Signed, WORD),
-                                         (lltype.Char,   SIZEOF_CHAR),
-                                         (rffi.SHORT,    SIZEOF_SHORT),
-                                         (rffi.INT,      SIZEOF_INT)])
+unroll_basic_sizes = unrolling_iterable([
+    (lltype.Signed,   lltype.Unsigned, WORD),
+    (rffi.SIGNEDCHAR, lltype.Char,     SIZEOF_CHAR),
+    (rffi.SHORT,      rffi.USHORT,     SIZEOF_SHORT),
+    (rffi.INT,        rffi.UINT,       SIZEOF_INT)])
 # does not contain Float ^^^ which must be special-cased

Modified: pypy/branch/fast-forward/pypy/jit/backend/llsupport/test/test_descr.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/llsupport/test/test_descr.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/llsupport/test/test_descr.py	Fri Oct 22 23:09:43 2010
@@ -83,6 +83,18 @@
         assert     descr_f.is_float_field()
 
 
+def test_get_field_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        S = lltype.GcStruct('S', ('x', RESTYPE))
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            descr_x = get_field_descr(c2, S, 'x')
+            assert descr_x.is_field_signed() == signed
+
+
 def test_get_array_descr():
     U = lltype.Struct('U')
     T = lltype.GcStruct('T')
@@ -164,6 +176,25 @@
     assert descr.get_base_size(False) == 0
     assert descr.get_ofs_length(False) == -1
 
+
+def test_get_array_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        A = lltype.GcArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            arraydescr = get_array_descr(c2, A)
+            assert arraydescr.is_item_signed() == signed
+        #
+        RA = rffi.CArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            arraydescr = get_array_descr(c2, RA)
+            assert arraydescr.is_item_signed() == signed
+
+
 def test_get_call_descr_not_translated():
     c0 = GcCache(False)
     descr1 = get_call_descr(c0, [lltype.Char, lltype.Signed], lltype.Char)
@@ -219,6 +250,17 @@
     extrainfo = descr3.get_extra_info()
     assert extrainfo is None
 
+def test_get_call_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        A = lltype.GcArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            descr1 = get_call_descr(c2, [], RESTYPE)
+            assert descr1.is_result_signed() == signed
+
 
 def test_repr_of_descr():
     c0 = GcCache(False)

Modified: pypy/branch/fast-forward/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/test/runner_test.py	Fri Oct 22 23:09:43 2010
@@ -9,12 +9,13 @@
                                          ConstObj, BoxFloat, ConstFloat)
 from pypy.jit.metainterp.resoperation import ResOperation, rop
 from pypy.jit.metainterp.typesystem import deref
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.rpython.lltypesystem import lltype, llmemory, rstr, rffi, rclass
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.llinterp import LLException
 from pypy.jit.codewriter import heaptracker
+from pypy.rlib.rarithmetic import intmask
 
 
 class Runner(object):
@@ -421,6 +422,7 @@
             assert x == 3.5 - 42
 
     def test_call(self):
+        from pypy.rlib.libffi import types
 
         def func_int(a, b):
             return a + b
@@ -428,23 +430,31 @@
             return chr(ord(c) + ord(c1))
 
         functions = [
-            (func_int, lltype.Signed, 655360),
-            (func_int, rffi.SHORT, 1213),
-            (func_char, lltype.Char, 12)
+            (func_int, lltype.Signed, types.sint, 655360),
+            (func_int, rffi.SHORT, types.sint16, 1213),
+            (func_char, lltype.Char, types.uchar, 12)
             ]
 
-        for func, TP, num in functions:
+        for func, TP, ffi_type, num in functions:
             cpu = self.cpu
             #
             FPTR = self.Ptr(self.FuncType([TP, TP], TP))
             func_ptr = llhelper(FPTR, func)
             FUNC = deref(FPTR)
-            calldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
             funcbox = self.get_funcbox(cpu, func_ptr)
+            # first, try it with the "normal" calldescr
+            calldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
             res = self.execute_operation(rop.CALL,
                                          [funcbox, BoxInt(num), BoxInt(num)],
                                          'int', descr=calldescr)
             assert res.value == 2 * num
+            # then, try it with the dynamic calldescr
+            dyn_calldescr = cpu.calldescrof_dynamic([ffi_type, ffi_type], ffi_type)
+            res = self.execute_operation(rop.CALL,
+                                         [funcbox, BoxInt(num), BoxInt(num)],
+                                         'int', descr=dyn_calldescr)
+            assert res.value == 2 * num
+            
 
         if cpu.supports_floats:
             def func(f0, f1, f2, f3, f4, f5, f6, i0, i1, f7, f8, f9):
@@ -507,6 +517,24 @@
                                          'int', descr=calldescr)
             assert res.value == func_ints(*args)
 
+    def test_call_to_c_function(self):
+        from pypy.rlib.libffi import CDLL, types, ArgChain
+        from pypy.rpython.lltypesystem.ll2ctypes import libc_name
+        libc = CDLL(libc_name)
+        c_tolower = libc.getpointer('tolower', [types.uchar], types.sint)
+        argchain = ArgChain().arg(ord('A'))
+        assert c_tolower.call(argchain, rffi.INT) == ord('a')
+
+        func_adr = llmemory.cast_ptr_to_adr(c_tolower.funcsym)
+        funcbox = ConstInt(heaptracker.adr2int(func_adr))
+        calldescr = self.cpu.calldescrof_dynamic([types.uchar], types.sint)
+        res = self.execute_operation(rop.CALL,
+                                     [funcbox, BoxInt(ord('A'))],
+                                     'int',
+                                     descr=calldescr)
+        assert res.value == ord('a')
+
+
     def test_field_basic(self):
         t_box, T_box = self.alloc_instance(self.T)
         fielddescr = self.cpu.fielddescrof(self.S, 'value')
@@ -833,6 +861,23 @@
                                                         length_box], 'void')
                                 assert self.look_string(r_box) == "!??cdef?!"
 
+    def test_copyunicodecontent(self):
+        s_box = self.alloc_unicode(u"abcdef")
+        for s_box in [s_box, s_box.constbox()]:
+            for srcstart_box in [BoxInt(2), ConstInt(2)]:
+                for dststart_box in [BoxInt(3), ConstInt(3)]:
+                    for length_box in [BoxInt(4), ConstInt(4)]:
+                        for r_box_is_const in [False, True]:
+                            r_box = self.alloc_unicode(u"!???????!")
+                            if r_box_is_const:
+                                r_box = r_box.constbox()
+                                self.execute_operation(rop.COPYUNICODECONTENT,
+                                                       [s_box, r_box,
+                                                        srcstart_box,
+                                                        dststart_box,
+                                                        length_box], 'void')
+                                assert self.look_unicode(r_box) == u"!??cdef?!"
+
     def test_do_unicode_basic(self):
         u = self.cpu.bh_newunicode(5)
         self.cpu.bh_unicodesetitem(u, 4, 123)
@@ -1227,6 +1272,10 @@
         u_box = BoxPtr(lltype.cast_opaque_ptr(llmemory.GCREF, u))
         return u_box
 
+    def look_unicode(self, unicode_box):
+        u = unicode_box.getref(lltype.Ptr(rstr.UNICODE))
+        return u''.join(u.chars)
+
 
     def test_casts(self):
         py.test.skip("xxx fix or kill")
@@ -1283,6 +1332,7 @@
                                    descr=fd)
             res = self.execute_operation(get_op, [s_box], 'int', descr=fd)
             assert res.getint()  == 32
+        lltype.free(s, flavor='raw')
 
     def test_new_with_vtable(self):
         cpu = self.cpu
@@ -1980,6 +2030,196 @@
         assert self.cpu.get_latest_value_float(0) == 13.5
         assert called
 
+    def test_short_result_of_getfield_direct(self):
+        # Test that a getfield that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            S = lltype.GcStruct('S', ('x', RESTYPE))
+            descrfld_x = cpu.fielddescrof(S, 'x')
+            s = lltype.malloc(S)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            s.x = rffi.cast(RESTYPE, value)
+            x = cpu.bh_getfield_gc_i(lltype.cast_opaque_ptr(llmemory.GCREF, s),
+                                     descrfld_x)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_getfield_compiled(self):
+        # Test that a getfield that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            S = lltype.GcStruct('S', ('x', RESTYPE))
+            descrfld_x = cpu.fielddescrof(S, 'x')
+            s = lltype.malloc(S)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            s.x = rffi.cast(RESTYPE, value)
+            s_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, s)
+            res = self.execute_operation(rop.GETFIELD_GC, [BoxPtr(s_gcref)],
+                                         'int', descr=descrfld_x)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
+    def test_short_result_of_getarrayitem_direct(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = lltype.GcArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            x = cpu.bh_getarrayitem_gc_i(
+                descrarray, lltype.cast_opaque_ptr(llmemory.GCREF, a), 3)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_getarrayitem_compiled(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = lltype.GcArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, a)
+            res = self.execute_operation(rop.GETARRAYITEM_GC,
+                                         [BoxPtr(a_gcref), BoxInt(3)],
+                                         'int', descr=descrarray)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
+    def test_short_result_of_getarrayitem_raw_direct(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = rffi.CArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5, flavor='raw')
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_rawint = heaptracker.adr2int(llmemory.cast_ptr_to_adr(a))
+            x = cpu.bh_getarrayitem_raw_i(descrarray, a_rawint, 3)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+            lltype.free(a, flavor='raw')
+
+    def test_short_result_of_getarrayitem_raw_compiled(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = rffi.CArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5, flavor='raw')
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_rawint = heaptracker.adr2int(llmemory.cast_ptr_to_adr(a))
+            res = self.execute_operation(rop.GETARRAYITEM_RAW,
+                                         [BoxInt(a_rawint), BoxInt(3)],
+                                         'int', descr=descrarray)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+            lltype.free(a, flavor='raw')
+
+    def test_short_result_of_call_direct(self):
+        # Test that calling a function that returns a CHAR, SHORT or INT,
+        # signed or unsigned, properly gets zero-extended or sign-extended.
+        from pypy.translator.tool.cbuild import ExternalCompilationInfo
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            # Tested with a function that intentionally does not cast the
+            # result to RESTYPE, but makes sure that we return the whole
+            # value in eax or rax.
+            eci = ExternalCompilationInfo(separate_module_sources=["""
+                long fn_test_result_of_call(long x)
+                {
+                    return x + 1;
+                }
+            """])
+            f = rffi.llexternal('fn_test_result_of_call', [lltype.Signed],
+                                RESTYPE, compilation_info=eci, _nowrapper=True)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value + 1))
+            assert intmask(f(value)) == expected
+            #
+            FUNC = self.FuncType([lltype.Signed], RESTYPE)
+            FPTR = self.Ptr(FUNC)
+            calldescr = self.cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
+            x = self.cpu.bh_call_i(self.get_funcbox(self.cpu, f).value,
+                                   calldescr, [value], None, None)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_call_compiled(self):
+        # Test that calling a function that returns a CHAR, SHORT or INT,
+        # signed or unsigned, properly gets zero-extended or sign-extended.
+        from pypy.translator.tool.cbuild import ExternalCompilationInfo
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            # Tested with a function that intentionally does not cast the
+            # result to RESTYPE, but makes sure that we return the whole
+            # value in eax or rax.
+            eci = ExternalCompilationInfo(separate_module_sources=["""
+                long fn_test_result_of_call(long x)
+                {
+                    return x + 1;
+                }
+            """])
+            f = rffi.llexternal('fn_test_result_of_call', [lltype.Signed],
+                                RESTYPE, compilation_info=eci, _nowrapper=True)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value + 1))
+            assert intmask(f(value)) == expected
+            #
+            FUNC = self.FuncType([lltype.Signed], RESTYPE)
+            FPTR = self.Ptr(FUNC)
+            calldescr = self.cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
+            funcbox = self.get_funcbox(self.cpu, f)
+            res = self.execute_operation(rop.CALL, [funcbox, BoxInt(value)],
+                                         'int', descr=calldescr)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
 
 class OOtypeBackendTest(BaseBackendTest):
 

Modified: pypy/branch/fast-forward/pypy/jit/backend/test/test_ll_random.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/test/test_ll_random.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/test/test_ll_random.py	Fri Oct 22 23:09:43 2010
@@ -599,7 +599,7 @@
     OPERATIONS.append(StrLenOperation(rop.STRLEN))
     OPERATIONS.append(UnicodeLenOperation(rop.UNICODELEN))
     OPERATIONS.append(CopyStrContentOperation(rop.COPYSTRCONTENT))
-    #OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT))
+    OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT))
 
 for i in range(2):
     OPERATIONS.append(GuardClassOperation(rop.GUARD_CLASS))

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/assembler.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/assembler.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/assembler.py	Fri Oct 22 23:09:43 2010
@@ -8,7 +8,8 @@
 from pypy.rpython.annlowlevel import llhelper
 from pypy.tool.uid import fixid
 from pypy.jit.backend.x86.regalloc import (RegAlloc, X86RegisterManager,
-                                           X86XMMRegisterManager, get_ebp_ofs)
+                                           X86XMMRegisterManager, get_ebp_ofs,
+                                           _get_scale)
 
 from pypy.jit.backend.x86.arch import (FRAME_FIXED_SIZE, FORCE_INDEX_OFS, WORD,
                                        IS_X86_32, IS_X86_64)
@@ -22,7 +23,8 @@
                                          X86_64_SCRATCH_REG,
                                          X86_64_XMM_SCRATCH_REG,
                                          RegLoc, StackLoc, ConstFloatLoc,
-                                         ImmedLoc, AddressLoc, imm)
+                                         ImmedLoc, AddressLoc, imm,
+                                         imm0, imm1)
 
 from pypy.rlib.objectmodel import we_are_translated, specialize
 from pypy.jit.backend.x86 import rx86, regloc, codebuf
@@ -244,12 +246,13 @@
             f = open_file_as_stream(output_log, "w")
             for i in range(len(self.loop_run_counters)):
                 name, struct = self.loop_run_counters[i]
-                f.write(str(struct.i) + " " * (8 - len(str(struct.i))) + name + "\n")
+                f.write(str(name) + ":" +  str(struct.i) + "\n")
             f.close()
 
     def _build_float_constants(self):
         # 44 bytes: 32 bytes for the data, and up to 12 bytes for alignment
-        addr = lltype.malloc(rffi.CArray(lltype.Char), 44, flavor='raw')
+        addr = lltype.malloc(rffi.CArray(lltype.Char), 44, flavor='raw',
+                             track_allocation=False)
         if not we_are_translated():
             self._keepalive_malloced_float_consts = addr
         float_constants = rffi.cast(lltype.Signed, addr)
@@ -399,9 +402,10 @@
             funcname = "" % len(self.loop_run_counters)
         # invent the counter, so we don't get too confused
         if self._debug:
-            struct = lltype.malloc(DEBUG_COUNTER, flavor='raw')
+            struct = lltype.malloc(DEBUG_COUNTER, flavor='raw',
+                                   track_allocation=False)   # known to leak
             struct.i = 0
-            self.loop_run_counters.append((funcname, struct))
+            self.loop_run_counters.append((len(self.loop_run_counters), struct))
         return funcname
         
     def patch_jump_for_descr(self, faildescr, adr_new_target):
@@ -440,7 +444,7 @@
             # self.mc.PUSH(eax)
             # adr = rffi.cast(lltype.Signed, self.loop_run_counters[-1][1])
             # self.mc.MOV(eax, heap(adr))
-            # self.mc.ADD(eax, imm(1))
+            # self.mc.ADD(eax, imm1)
             # self.mc.MOV(heap(adr), eax)
             # self.mc.POP(eax)
         return operations
@@ -711,7 +715,7 @@
         self.regalloc_perform_with_guard(None, guard_op, faillocs, arglocs,
                                          resloc, current_depths)
 
-    def load_effective_addr(self, sizereg, baseofs, scale, result, frm=imm(0)):
+    def load_effective_addr(self, sizereg, baseofs, scale, result, frm=imm0):
         self.mc.LEA(result, addr_add(frm, sizereg, baseofs, scale))
 
     def _unaryop(asmop):
@@ -973,28 +977,28 @@
 
     def genop_guard_int_is_true(self, op, guard_op, guard_token, arglocs, resloc):
         guard_opnum = guard_op.getopnum()
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         if guard_opnum == rop.GUARD_TRUE:
             return self.implement_guard(guard_token, 'Z')
         else:
             return self.implement_guard(guard_token, 'NZ')
 
     def genop_int_is_true(self, op, arglocs, resloc):
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         rl = resloc.lowest8bits()
         self.mc.SET_ir(rx86.Conditions['NE'], rl.value)
         self.mc.MOVZX8(resloc, rl)
 
     def genop_guard_int_is_zero(self, op, guard_op, guard_token, arglocs, resloc):
         guard_opnum = guard_op.getopnum()
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         if guard_opnum == rop.GUARD_TRUE:
             return self.implement_guard(guard_token, 'NZ')
         else:
             return self.implement_guard(guard_token, 'Z')
 
     def genop_int_is_zero(self, op, arglocs, resloc):
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         rl = resloc.lowest8bits()
         self.mc.SET_ir(rx86.Conditions['E'], rl.value)
         self.mc.MOVZX8(resloc, rl)
@@ -1050,50 +1054,66 @@
         assert result_loc is eax
         self.call(self.malloc_unicode_func_addr, arglocs, eax)
 
-    def genop_getfield_gc(self, op, arglocs, resloc):
-        base_loc, ofs_loc, size_loc = arglocs
-        assert isinstance(size_loc, ImmedLoc)
+    # ----------
+
+    def load_from_mem(self, resloc, source_addr, size_loc, sign_loc):
         assert isinstance(resloc, RegLoc)
         size = size_loc.value
-
-        source_addr = AddressLoc(base_loc, ofs_loc)
+        sign = sign_loc.value
         if resloc.is_xmm:
             self.mc.MOVSD(resloc, source_addr)
+        elif size == WORD:
+            self.mc.MOV(resloc, source_addr)
+        elif size == 1:
+            if sign:
+                self.mc.MOVSX8(resloc, source_addr)
+            else:
+                self.mc.MOVZX8(resloc, source_addr)
+        elif size == 2:
+            if sign:
+                self.mc.MOVSX16(resloc, source_addr)
+            else:
+                self.mc.MOVZX16(resloc, source_addr)
+        elif IS_X86_64 and size == 4:
+            if sign:
+                self.mc.MOVSX32(resloc, source_addr)
+            else:
+                self.mc.MOV32(resloc, source_addr)    # zero-extending
+        else:
+            not_implemented("load_from_mem size = %d" % size)
+
+    def save_into_mem(self, dest_addr, value_loc, size_loc):
+        size = size_loc.value
+        if isinstance(value_loc, RegLoc) and value_loc.is_xmm:
+            self.mc.MOVSD(dest_addr, value_loc)
         elif size == 1:
-            self.mc.MOVZX8(resloc, source_addr)
+            self.mc.MOV8(dest_addr, value_loc.lowest8bits())
         elif size == 2:
-            self.mc.MOVZX16(resloc, source_addr)
+            self.mc.MOV16(dest_addr, value_loc)
         elif size == 4:
-            # MOV32 is zero-extending on 64-bit, so this is okay
-            self.mc.MOV32(resloc, source_addr)
+            self.mc.MOV32(dest_addr, value_loc)
         elif IS_X86_64 and size == 8:
-            self.mc.MOV(resloc, source_addr)
+            self.mc.MOV(dest_addr, value_loc)
         else:
-            raise NotImplementedError("getfield size = %d" % size)
+            not_implemented("save_into_mem size = %d" % size)
+
+    def genop_getfield_gc(self, op, arglocs, resloc):
+        base_loc, ofs_loc, size_loc, sign_loc = arglocs
+        assert isinstance(size_loc, ImmedLoc)
+        source_addr = AddressLoc(base_loc, ofs_loc)
+        self.load_from_mem(resloc, source_addr, size_loc, sign_loc)
 
     genop_getfield_raw = genop_getfield_gc
     genop_getfield_raw_pure = genop_getfield_gc
     genop_getfield_gc_pure = genop_getfield_gc
 
     def genop_getarrayitem_gc(self, op, arglocs, resloc):
-        base_loc, ofs_loc, scale, ofs = arglocs
+        base_loc, ofs_loc, size_loc, ofs, sign_loc = arglocs
         assert isinstance(ofs, ImmedLoc)
-        assert isinstance(scale, ImmedLoc)
-        src_addr = addr_add(base_loc, ofs_loc, ofs.value, scale.value)
-        if op.result.type == FLOAT:
-            self.mc.MOVSD(resloc, src_addr)
-        else:
-            if scale.value == 0:
-                self.mc.MOVZX8(resloc, src_addr)
-            elif scale.value == 1:
-                self.mc.MOVZX16(resloc, src_addr)
-            elif scale.value == 2:
-                self.mc.MOV32(resloc, src_addr)
-            elif IS_X86_64 and scale.value == 3:
-                self.mc.MOV(resloc, src_addr)
-            else:
-                print "[asmgen]getarrayitem unsupported size: %d" % scale.value
-                raise NotImplementedError()
+        assert isinstance(size_loc, ImmedLoc)
+        scale = _get_scale(size_loc.value)
+        src_addr = addr_add(base_loc, ofs_loc, ofs.value, scale)
+        self.load_from_mem(resloc, src_addr, size_loc, sign_loc)
 
     genop_getarrayitem_gc_pure = genop_getarrayitem_gc
     genop_getarrayitem_raw = genop_getarrayitem_gc
@@ -1101,40 +1121,16 @@
     def genop_discard_setfield_gc(self, op, arglocs):
         base_loc, ofs_loc, size_loc, value_loc = arglocs
         assert isinstance(size_loc, ImmedLoc)
-        size = size_loc.value
         dest_addr = AddressLoc(base_loc, ofs_loc)
-        if isinstance(value_loc, RegLoc) and value_loc.is_xmm:
-            self.mc.MOVSD(dest_addr, value_loc)
-        elif IS_X86_64 and size == 8:
-            self.mc.MOV(dest_addr, value_loc)
-        elif size == 4:
-            self.mc.MOV32(dest_addr, value_loc)
-        elif size == 2:
-            self.mc.MOV16(dest_addr, value_loc)
-        elif size == 1:
-            self.mc.MOV8(dest_addr, value_loc.lowest8bits())
-        else:
-            print "[asmgen]setfield addr size %d" % size
-            raise NotImplementedError("Addr size %d" % size)
+        self.save_into_mem(dest_addr, value_loc, size_loc)
 
     def genop_discard_setarrayitem_gc(self, op, arglocs):
-        base_loc, ofs_loc, value_loc, scale_loc, baseofs = arglocs
+        base_loc, ofs_loc, value_loc, size_loc, baseofs = arglocs
         assert isinstance(baseofs, ImmedLoc)
-        assert isinstance(scale_loc, ImmedLoc)
-        dest_addr = AddressLoc(base_loc, ofs_loc, scale_loc.value, baseofs.value)
-        if op.getarg(2).type == FLOAT:
-            self.mc.MOVSD(dest_addr, value_loc)
-        else:
-            if IS_X86_64 and scale_loc.value == 3:
-                self.mc.MOV(dest_addr, value_loc)
-            elif scale_loc.value == 2:
-                self.mc.MOV32(dest_addr, value_loc)
-            elif scale_loc.value == 1:
-                self.mc.MOV16(dest_addr, value_loc)
-            elif scale_loc.value == 0:
-                self.mc.MOV8(dest_addr, value_loc.lowest8bits())
-            else:
-                raise NotImplementedError("scale = %d" % scale_loc.value)
+        assert isinstance(size_loc, ImmedLoc)
+        scale = _get_scale(size_loc.value)
+        dest_addr = AddressLoc(base_loc, ofs_loc, scale, baseofs.value)
+        self.save_into_mem(dest_addr, value_loc, size_loc)
 
     def genop_discard_strsetitem(self, op, arglocs):
         base_loc, ofs_loc, val_loc = arglocs
@@ -1201,7 +1197,7 @@
 
     def genop_guard_guard_no_exception(self, ign_1, guard_op, guard_token,
                                        locs, ign_2):
-        self.mc.CMP(heap(self.cpu.pos_exception()), imm(0))
+        self.mc.CMP(heap(self.cpu.pos_exception()), imm0)
         return self.implement_guard(guard_token, 'NZ')
 
     def genop_guard_guard_exception(self, ign_1, guard_op, guard_token,
@@ -1213,8 +1209,8 @@
         addr = self.implement_guard(guard_token, 'NE')
         if resloc is not None:
             self.mc.MOV(resloc, heap(self.cpu.pos_exc_value()))
-        self.mc.MOV(heap(self.cpu.pos_exception()), imm(0))
-        self.mc.MOV(heap(self.cpu.pos_exc_value()), imm(0))
+        self.mc.MOV(heap(self.cpu.pos_exception()), imm0)
+        self.mc.MOV(heap(self.cpu.pos_exc_value()), imm0)
         return addr
 
     def _gen_guard_overflow(self, guard_op, guard_token):
@@ -1224,8 +1220,8 @@
         elif guard_opnum == rop.GUARD_OVERFLOW:
             return self.implement_guard(guard_token, 'NO')
         else:
-            print "int_xxx_ovf followed by", guard_op.getopname()
-            raise AssertionError
+            not_implemented("int_xxx_ovf followed by %s" %
+                            guard_op.getopname())
 
     def genop_guard_int_add_ovf(self, op, guard_op, guard_token, arglocs, result_loc):
         self.genop_int_add(op, arglocs, result_loc)
@@ -1288,7 +1284,7 @@
     def genop_guard_guard_nonnull_class(self, ign_1, guard_op,
                                         guard_token, locs, ign_2):
         self.mc.ensure_bytes_available(256)
-        self.mc.CMP(locs[0], imm(1))
+        self.mc.CMP(locs[0], imm1)
         # Patched below
         self.mc.J_il8(rx86.Conditions['B'], 0)
         jb_location = self.mc.get_relative_pos()
@@ -1637,25 +1633,34 @@
         sizeloc = arglocs[0]
         assert isinstance(sizeloc, ImmedLoc)
         size = sizeloc.value
+        signloc = arglocs[1]
 
         if isinstance(op.getarg(0), Const):
             x = imm(op.getarg(0).getint())
         else:
-            x = arglocs[1]
+            x = arglocs[2]
         if x is eax:
             tmp = ecx
         else:
             tmp = eax
         
-        self._emit_call(x, arglocs, 2, tmp=tmp)
+        self._emit_call(x, arglocs, 3, tmp=tmp)
+
+        if IS_X86_32 and isinstance(resloc, StackLoc) and resloc.width == 8:
+            self.mc.FSTP_b(resloc.value)   # float return
+        elif size == WORD:
+            assert resloc is eax or resloc is xmm0    # a full word
+        elif size == 0:
+            pass    # void return
+        else:
+            # use the code in load_from_mem to do the zero- or sign-extension
+            assert resloc is eax
+            if size == 1:
+                srcloc = eax.lowest8bits()
+            else:
+                srcloc = eax
+            self.load_from_mem(eax, srcloc, sizeloc, signloc)
 
-        if isinstance(resloc, StackLoc) and resloc.width == 8 and IS_X86_32:
-            self.mc.FSTP_b(resloc.value)
-        elif size == 1:
-            self.mc.AND_ri(eax.value, 0xff)
-        elif size == 2:
-            self.mc.AND_ri(eax.value, 0xffff)
-    
     def genop_guard_call_may_force(self, op, guard_op, guard_token,
                                    arglocs, result_loc):
         faildescr = guard_op.getdescr()
@@ -1808,20 +1813,16 @@
         self.mc.LEA_rb(resloc.value, FORCE_INDEX_OFS)
 
     def not_implemented_op_discard(self, op, arglocs):
-        msg = "not implemented operation: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation: %s" % op.getopname())
 
     def not_implemented_op(self, op, arglocs, resloc):
-        msg = "not implemented operation with res: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation with res: %s" %
+                        op.getopname())
 
     def not_implemented_op_guard(self, op, guard_op,
                                  failaddr, arglocs, resloc):
-        msg = "not implemented operation (guard): %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation (guard): %s" %
+                        op.getopname())
 
     def mark_gc_roots(self):
         gcrootmap = self.cpu.gc_ll_descr.gcrootmap
@@ -1905,3 +1906,7 @@
 
 def heap(addr):
     return AddressLoc(ImmedLoc(addr), ImmedLoc(0), 0, 0)
+
+def not_implemented(msg):
+    os.write(2, '[x86/asm] %s\n' % msg)
+    raise NotImplementedError(msg)

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/regalloc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/regalloc.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/regalloc.py	Fri Oct 22 23:09:43 2010
@@ -2,6 +2,7 @@
 """ Register allocation scheme.
 """
 
+import os
 from pypy.jit.metainterp.history import (Box, Const, ConstInt, ConstPtr,
                                          ResOperation, BoxPtr,
                                          LoopToken, INT, REF, FLOAT)
@@ -40,12 +41,10 @@
             return imm(c.value)
         elif isinstance(c, ConstPtr):
             if we_are_translated() and c.value and rgc.can_move(c.value):
-                print "convert_to_imm: ConstPtr needs special care"
-                raise AssertionError
+                not_implemented("convert_to_imm: ConstPtr needs special care")
             return imm(rffi.cast(lltype.Signed, c.value))
         else:
-            print "convert_to_imm: got a %s" % c
-            raise AssertionError
+            not_implemented("convert_to_imm: got a %s" % c)
 
 class X86_64_RegisterManager(X86RegisterManager):
     # r11 omitted because it's used as scratch
@@ -70,8 +69,9 @@
 
     def _get_new_array(self):
         n = self.BASE_CONSTANT_SIZE
+        # known to leak
         self.cur_array = lltype.malloc(rffi.CArray(lltype.Float), n,
-                                       flavor='raw')
+                                       flavor='raw', track_allocation=False)
         self.cur_array_free = n
     _get_new_array._dont_inline_ = True
 
@@ -359,8 +359,8 @@
             if op.is_ovf():
                 if (operations[i + 1].getopnum() != rop.GUARD_NO_OVERFLOW and
                     operations[i + 1].getopnum() != rop.GUARD_OVERFLOW):
-                    print "int_xxx_ovf not followed by guard_(no)_overflow"
-                    raise AssertionError
+                    not_implemented("int_xxx_ovf not followed by "
+                                    "guard_(no)_overflow")
                 return True
             return False
         if (operations[i + 1].getopnum() != rop.GUARD_TRUE and
@@ -412,8 +412,8 @@
                 arg = op.getarg(j)
                 if isinstance(arg, Box):
                     if arg not in start_live:
-                        print "Bogus arg in operation %d at %d" % (op.getopnum(), i)
-                        raise AssertionError
+                        not_implemented("Bogus arg in operation %d at %d" %
+                                        (op.getopnum(), i))
                     longevity[arg] = (start_live[arg], i)
             if op.is_guard():
                 for arg in op.getfailargs():
@@ -421,8 +421,8 @@
                         continue
                     assert isinstance(arg, Box)
                     if arg not in start_live:
-                        print "Bogus arg in guard %d at %d" % (op.getopnum(), i)
-                        raise AssertionError
+                        not_implemented("Bogus arg in guard %d at %d" %
+                                        (op.getopnum(), i))
                     longevity[arg] = (start_live[arg], i)
         for arg in inputargs:
             if arg not in longevity:
@@ -667,7 +667,13 @@
         assert isinstance(calldescr, BaseCallDescr)
         assert len(calldescr.arg_classes) == op.numargs() - 1
         size = calldescr.get_result_size(self.translate_support_code)
-        self._call(op, [imm(size)] + [self.loc(op.getarg(i)) for i in range(op.numargs())],
+        sign = calldescr.is_result_signed()
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self._call(op, [imm(size), sign_loc] +
+                       [self.loc(op.getarg(i)) for i in range(op.numargs())],
                    guard_not_forced_op=guard_not_forced_op)
 
     def consider_call(self, op):
@@ -688,7 +694,7 @@
             self.rm._sync_var(op.getarg(vable_index))
             vable = self.fm.loc(op.getarg(vable_index))
         else:
-            vable = imm(0)
+            vable = imm0
         self._call(op, [imm(size), vable] +
                    [self.loc(op.getarg(i)) for i in range(op.numargs())],
                    guard_not_forced_op=guard_op)
@@ -778,15 +784,11 @@
             loc = self.loc(op.getarg(0))
             return self._call(op, [loc])
         # boehm GC (XXX kill the following code at some point)
-        ofs_items, itemsize, ofs = symbolic.get_array_token(rstr.UNICODE, self.translate_support_code)
-        if itemsize == 4:
-            return self._malloc_varsize(ofs_items, ofs, 2, op.getarg(0),
-                                        op.result)
-        elif itemsize == 2:
-            return self._malloc_varsize(ofs_items, ofs, 1, op.getarg(0),
-                                        op.result)
-        else:
-            assert False, itemsize
+        ofs_items, _, ofs = symbolic.get_array_token(rstr.UNICODE,
+                                                   self.translate_support_code)
+        scale = self._get_unicode_item_scale()
+        return self._malloc_varsize(ofs_items, ofs, scale, op.getarg(0),
+                                    op.result)
 
     def _malloc_varsize(self, ofs_items, ofs_length, scale, v, res_v):
         # XXX kill this function at some point
@@ -818,8 +820,9 @@
             arglocs.append(self.loc(op.getarg(0)))
             return self._call(op, arglocs)
         # boehm GC (XXX kill the following code at some point)
-        scale_of_field, basesize, ofs_length, _ = (
+        itemsize, basesize, ofs_length, _, _ = (
             self._unpack_arraydescr(op.getdescr()))
+        scale_of_field = _get_scale(itemsize)
         return self._malloc_varsize(basesize, ofs_length, scale_of_field,
                                     op.getarg(0), op.result)
 
@@ -829,21 +832,19 @@
         ofs = arraydescr.get_base_size(self.translate_support_code)
         size = arraydescr.get_item_size(self.translate_support_code)
         ptr = arraydescr.is_array_of_pointers()
-        scale = 0
-        while (1 << scale) < size:
-            scale += 1
-        assert (1 << scale) == size
-        return scale, ofs, ofs_length, ptr
+        sign = arraydescr.is_item_signed()
+        return size, ofs, ofs_length, ptr, sign
 
     def _unpack_fielddescr(self, fielddescr):
         assert isinstance(fielddescr, BaseFieldDescr)
         ofs = fielddescr.offset
         size = fielddescr.get_field_size(self.translate_support_code)
         ptr = fielddescr.is_pointer_field()
-        return imm(ofs), imm(size), ptr
+        sign = fielddescr.is_field_signed()
+        return imm(ofs), imm(size), ptr, sign
 
     def consider_setfield_gc(self, op):
-        ofs_loc, size_loc, ptr = self._unpack_fielddescr(op.getdescr())
+        ofs_loc, size_loc, _, _ = self._unpack_fielddescr(op.getdescr())
         assert isinstance(size_loc, ImmedLoc)
         if size_loc.value == 1:
             need_lower_byte = True
@@ -870,10 +871,10 @@
     consider_unicodesetitem = consider_strsetitem
 
     def consider_setarrayitem_gc(self, op):
-        scale, ofs, _, ptr = self._unpack_arraydescr(op.getdescr())
+        itemsize, ofs, _, _, _ = self._unpack_arraydescr(op.getdescr())
         args = op.getarglist()
         base_loc  = self.rm.make_sure_var_in_reg(op.getarg(0), args)
-        if scale == 0:
+        if itemsize == 1:
             need_lower_byte = True
         else:
             need_lower_byte = False
@@ -882,30 +883,39 @@
         ofs_loc = self.rm.make_sure_var_in_reg(op.getarg(1), args)
         self.possibly_free_vars(args)
         self.PerformDiscard(op, [base_loc, ofs_loc, value_loc,
-                                 imm(scale), imm(ofs)])
+                                 imm(itemsize), imm(ofs)])
 
     consider_setarrayitem_raw = consider_setarrayitem_gc
 
     def consider_getfield_gc(self, op):
-        ofs_loc, size_loc, _ = self._unpack_fielddescr(op.getdescr())
+        ofs_loc, size_loc, _, sign = self._unpack_fielddescr(op.getdescr())
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
         self.rm.possibly_free_vars(args)
         result_loc = self.force_allocate_reg(op.result)
-        self.Perform(op, [base_loc, ofs_loc, size_loc], result_loc)
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self.Perform(op, [base_loc, ofs_loc, size_loc, sign_loc], result_loc)
 
     consider_getfield_raw = consider_getfield_gc
     consider_getfield_raw_pure = consider_getfield_gc
     consider_getfield_gc_pure = consider_getfield_gc
 
     def consider_getarrayitem_gc(self, op):
-        scale, ofs, _, _ = self._unpack_arraydescr(op.getdescr())
+        itemsize, ofs, _, _, sign = self._unpack_arraydescr(op.getdescr())
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
         ofs_loc = self.rm.make_sure_var_in_reg(op.getarg(1), args)
         self.rm.possibly_free_vars_for_op(op)
         result_loc = self.force_allocate_reg(op.result)
-        self.Perform(op, [base_loc, ofs_loc, imm(scale), imm(ofs)], result_loc)
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self.Perform(op, [base_loc, ofs_loc, imm(itemsize), imm(ofs),
+                          sign_loc], result_loc)
 
     consider_getarrayitem_raw = consider_getarrayitem_gc
     consider_getarrayitem_gc_pure = consider_getarrayitem_gc
@@ -959,6 +969,12 @@
     consider_unicodegetitem = consider_strgetitem
 
     def consider_copystrcontent(self, op):
+        self._consider_copystrcontent(op, is_unicode=False)
+
+    def consider_copyunicodecontent(self, op):
+        self._consider_copystrcontent(op, is_unicode=True)
+
+    def _consider_copystrcontent(self, op, is_unicode):
         # compute the source address
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(args[0], args)
@@ -970,7 +986,8 @@
         srcaddr_box = TempBox()
         forbidden_vars = [args[1], args[3], args[4], srcaddr_box]
         srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box, forbidden_vars)
-        self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc)
+        self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc,
+                                        is_unicode=is_unicode)
         # compute the destination address
         base_loc = self.rm.make_sure_var_in_reg(args[1], forbidden_vars)
         ofs_loc = self.rm.make_sure_var_in_reg(args[3], forbidden_vars)
@@ -980,25 +997,57 @@
         forbidden_vars = [args[4], srcaddr_box]
         dstaddr_box = TempBox()
         dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box, forbidden_vars)
-        self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc)
+        self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc,
+                                        is_unicode=is_unicode)
+        # compute the length in bytes
+        length_box = args[4]
+        length_loc = self.loc(length_box)
+        if is_unicode:
+            self.rm.possibly_free_var(length_box)
+            forbidden_vars = [srcaddr_box, dstaddr_box]
+            bytes_box = TempBox()
+            bytes_loc = self.rm.force_allocate_reg(bytes_box, forbidden_vars)
+            scale = self._get_unicode_item_scale()
+            if not (isinstance(length_loc, ImmedLoc) or
+                    isinstance(length_loc, RegLoc)):
+                self.assembler.mov(length_loc, bytes_loc)
+                length_loc = bytes_loc
+            self.assembler.load_effective_addr(length_loc, 0, scale, bytes_loc)
+            length_box = bytes_box
+            length_loc = bytes_loc
         # call memcpy()
-        length_loc = self.loc(args[4])
         self.rm.before_call()
         self.xrm.before_call()
         self.assembler._emit_call(imm(self.assembler.memcpy_addr),
                                   [dstaddr_loc, srcaddr_loc, length_loc])
-        self.rm.possibly_free_var(args[4])
+        self.rm.possibly_free_var(length_box)
         self.rm.possibly_free_var(dstaddr_box)
         self.rm.possibly_free_var(srcaddr_box)
 
-    def _gen_address_inside_string(self, baseloc, ofsloc, resloc):
+    def _gen_address_inside_string(self, baseloc, ofsloc, resloc, is_unicode):
         cpu = self.assembler.cpu
-        ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR,
+        if is_unicode:
+            ofs_items, _, _ = symbolic.get_array_token(rstr.UNICODE,
                                                   self.translate_support_code)
-        assert itemsize == 1
-        self.assembler.load_effective_addr(ofsloc, ofs_items, 0,
+            scale = self._get_unicode_item_scale()
+        else:
+            ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR,
+                                                  self.translate_support_code)
+            assert itemsize == 1
+            scale = 0
+        self.assembler.load_effective_addr(ofsloc, ofs_items, scale,
                                            resloc, baseloc)
 
+    def _get_unicode_item_scale(self):
+        _, itemsize, _ = symbolic.get_array_token(rstr.UNICODE,
+                                                  self.translate_support_code)
+        if itemsize == 4:
+            return 2
+        elif itemsize == 2:
+            return 1
+        else:
+            raise AssertionError("bad unicode item size")
+
     def consider_jump(self, op):
         assembler = self.assembler
         assert self.jump_target_descr is None
@@ -1033,6 +1082,9 @@
     def consider_debug_merge_point(self, op):
         pass
 
+    def consider_jit_debug(self, op):
+        pass
+
     def get_mark_gc_roots(self, gcrootmap):
         shape = gcrootmap.get_basic_shape(IS_X86_64)
         for v, val in self.fm.frame_bindings.items():
@@ -1052,15 +1104,11 @@
         self.Perform(op, [], loc)
 
     def not_implemented_op(self, op):
-        msg = "[regalloc] Not implemented operation: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation: %s" % op.getopname())
 
     def not_implemented_op_with_guard(self, op, guard_op):
-        msg = "[regalloc] Not implemented operation with guard: %s" % (
-            op.getopname(),)
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation with guard: %s" % (
+            op.getopname(),))
 
 oplist = [RegAlloc.not_implemented_op] * rop._LAST
 oplist_with_guard = [RegAlloc.not_implemented_op_with_guard] * rop._LAST
@@ -1094,3 +1142,14 @@
     # Returns (ebp-20), (ebp-24), (ebp-28)...
     # i.e. the n'th word beyond the fixed frame size.
     return -WORD * (FRAME_FIXED_SIZE + position)
+
+def _get_scale(size):
+    assert size == 1 or size == 2 or size == 4 or size == 8
+    if size < 4:
+        return size - 1         # 1, 2 => 0, 1
+    else:
+        return (size >> 2) + 1  # 4, 8 => 2, 3
+
+def not_implemented(msg):
+    os.write(2, '[x86/regalloc] %s\n' % msg)
+    raise NotImplementedError(msg)

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/regloc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/regloc.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/regloc.py	Fri Oct 22 23:09:43 2010
@@ -442,8 +442,11 @@
     MOV8 = _binaryop('MOV8')
     MOV16 = _16_bit_binaryop('MOV')
     MOVZX8 = _binaryop('MOVZX8')
+    MOVSX8 = _binaryop('MOVSX8')
     MOVZX16 = _binaryop('MOVZX16')
+    MOVSX16 = _binaryop('MOVSX16')
     MOV32 = _binaryop('MOV32')
+    MOVSX32 = _binaryop('MOVSX32')
     XCHG = _binaryop('XCHG')
 
     PUSH = _unaryop('PUSH')
@@ -473,6 +476,9 @@
     else:
         return ImmedLoc(x)
 
+imm0 = imm(0)
+imm1 = imm(1)
+
 all_extra_instructions = [name for name in LocationCodeBuilder.__dict__
                           if name[0].isupper()]
 all_extra_instructions.sort()

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/runner.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/runner.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/runner.py	Fri Oct 22 23:09:43 2010
@@ -113,7 +113,8 @@
         return CPU386.cast_adr_to_int(adr)
 
     all_null_registers = lltype.malloc(rffi.LONGP.TO, 24,
-                                       flavor='raw', zero=True)
+                                       flavor='raw', zero=True,
+                                       immortal=True)
 
     def force(self, addr_of_force_index):
         TP = rffi.CArrayPtr(lltype.Signed)

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/rx86.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/rx86.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/rx86.py	Fri Oct 22 23:09:43 2010
@@ -642,7 +642,10 @@
 define_modrm_modes('MOV8_*i', [rex_w, '\xC6', orbyte(0<<3)], [immediate(2, 'b')], regtype='BYTE')
 
 define_modrm_modes('MOVZX8_r*', [rex_w, '\x0F\xB6', register(1, 8)], regtype='BYTE')
+define_modrm_modes('MOVSX8_r*', [rex_w, '\x0F\xBE', register(1, 8)], regtype='BYTE')
 define_modrm_modes('MOVZX16_r*', [rex_w, '\x0F\xB7', register(1, 8)])
+define_modrm_modes('MOVSX16_r*', [rex_w, '\x0F\xBF', register(1, 8)])
+define_modrm_modes('MOVSX32_r*', [rex_w, '\x63', register(1, 8)])
 
 define_modrm_modes('MOVSD_x*', ['\xF2', rex_nw, '\x0F\x10', register(1,8)], regtype='XMM')
 define_modrm_modes('MOVSD_*x', ['\xF2', rex_nw, '\x0F\x11', register(2,8)], regtype='XMM')

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_assembler.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_assembler.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_assembler.py	Fri Oct 22 23:09:43 2010
@@ -81,7 +81,8 @@
 
     # also test rebuild_faillocs_from_descr(), which should not
     # reproduce the holes at all
-    bytecode = lltype.malloc(rffi.UCHARP.TO, len(mc.content), flavor='raw')
+    bytecode = lltype.malloc(rffi.UCHARP.TO, len(mc.content), flavor='raw',
+                             immortal=True)
     for i in range(len(mc.content)):
         assert 0 <= mc.content[i] <= 255
         bytecode[i] = rffi.cast(rffi.UCHAR, mc.content[i])
@@ -115,7 +116,8 @@
         assert withfloats
         value = random.random() - 0.5
         # make sure it fits into 64 bits
-        tmp = lltype.malloc(rffi.LONGP.TO, 2, flavor='raw')
+        tmp = lltype.malloc(rffi.LONGP.TO, 2, flavor='raw',
+                            track_allocation=False)
         rffi.cast(rffi.DOUBLEP, tmp)[0] = value
         return rffi.cast(rffi.DOUBLEP, tmp)[0], tmp[0], tmp[1]
 
@@ -152,10 +154,12 @@
 
     # prepare the expected target arrays, the descr_bytecode,
     # the 'registers' and the 'stack' arrays according to 'content'
-    xmmregisters = lltype.malloc(rffi.LONGP.TO, 16+ACTUAL_CPU.NUM_REGS+1, flavor='raw')
+    xmmregisters = lltype.malloc(rffi.LONGP.TO, 16+ACTUAL_CPU.NUM_REGS+1,
+                                 flavor='raw', immortal=True)
     registers = rffi.ptradd(xmmregisters, 16)
     stacklen = baseloc + 10
-    stack = lltype.malloc(rffi.LONGP.TO, stacklen, flavor='raw')
+    stack = lltype.malloc(rffi.LONGP.TO, stacklen, flavor='raw',
+                          immortal=True)
     expected_ints = [0] * len(content)
     expected_ptrs = [lltype.nullptr(llmemory.GCREF.TO)] * len(content)
     expected_floats = [0.0] * len(content)
@@ -221,7 +225,7 @@
     descr_bytecode.append(0x00)
     descr_bytecode.append(0xCC)   # end marker
     descr_bytes = lltype.malloc(rffi.UCHARP.TO, len(descr_bytecode),
-                                flavor='raw')
+                                flavor='raw', immortal=True)
     for i in range(len(descr_bytecode)):
         assert 0 <= descr_bytecode[i] <= 255
         descr_bytes[i] = rffi.cast(rffi.UCHAR, descr_bytecode[i])

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_gc_integration.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_gc_integration.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_gc_integration.py	Fri Oct 22 23:09:43 2010
@@ -12,7 +12,7 @@
 from pypy.jit.backend.detect_cpu import getcpuclass
 from pypy.jit.backend.x86.regalloc import RegAlloc
 from pypy.jit.backend.x86.arch import WORD, FRAME_FIXED_SIZE
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.rpython.lltypesystem import lltype, llmemory, rffi
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import rclass, rstr

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_regalloc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_regalloc.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_regalloc.py	Fri Oct 22 23:09:43 2010
@@ -11,7 +11,7 @@
 from pypy.jit.backend.x86.regalloc import RegAlloc, X86RegisterManager,\
      FloatConstants, is_comparison_or_ovf_op
 from pypy.jit.backend.x86.arch import IS_X86_32, IS_X86_64
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.rpython.lltypesystem import lltype, llmemory, rffi
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import rclass, rstr

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_runner.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_runner.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_runner.py	Fri Oct 22 23:09:43 2010
@@ -10,7 +10,7 @@
 from pypy.jit.metainterp.resoperation import rop
 from pypy.jit.metainterp.executor import execute
 from pypy.jit.backend.test.runner_test import LLtypeBackendTest
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.tool.udir import udir
 import ctypes
 import sys
@@ -506,8 +506,8 @@
         self.cpu.execute_token(ops.token)
         # check debugging info
         name, struct = self.cpu.assembler.loop_run_counters[0]
-        assert name == 'xyz'
+        assert name == 0       # 'xyz'
         assert struct.i == 10
         self.cpu.finish_once()
         lines = py.path.local(self.logfile + ".count").readlines()
-        assert lines[0] == '10      xyz\n'
+        assert lines[0] == '0:10\n'  # '10      xyz\n'

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_string.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_string.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_string.py	Fri Oct 22 23:09:43 2010
@@ -2,8 +2,12 @@
 from pypy.jit.metainterp.test import test_string
 from pypy.jit.backend.x86.test.test_basic import Jit386Mixin
 
-class TestString(Jit386Mixin, test_string.StringTests):
+class TestString(Jit386Mixin, test_string.TestLLtype):
     # for the individual tests see
     # ====> ../../../metainterp/test/test_string.py
-    CALL = 'call'
-    CALL_PURE = 'call_pure'
+    pass
+
+class TestUnicode(Jit386Mixin, test_string.TestLLtypeUnicode):
+    # for the individual tests see
+    # ====> ../../../metainterp/test/test_string.py
+    pass

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_zrpy_gc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_zrpy_gc.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_zrpy_gc.py	Fri Oct 22 23:09:43 2010
@@ -191,6 +191,33 @@
     def run_orig(self, name, n, x):
         self.main_allfuncs(name, n, x)
 
+    def define_libffi_workaround(cls):
+        # XXX: this is a workaround for a bug in database.py.  It seems that
+        # the problem is triggered by optimizeopt/fficall.py, and in
+        # particular by the ``cast_base_ptr_to_instance(Func, llfunc)``: in
+        # these tests, that line is the only place where libffi.Func is
+        # referenced.
+        #
+        # The problem occurs because the gctransformer tries to annotate a
+        # low-level helper to call the __del__ of libffi.Func when it's too
+        # late.
+        #
+        # This workaround works by forcing the annotator (and all the rest of
+        # the toolchain) to see libffi.Func in a "proper" context, not just as
+        # the target of cast_base_ptr_to_instance.  Note that the function
+        # below is *never* called by any actual test, it's just annotated.
+        #
+        from pypy.rlib.libffi import get_libc_name, CDLL, types, ArgChain
+        libc_name = get_libc_name()
+        def f(n, x, *args):
+            libc = CDLL(libc_name)
+            ptr = libc.getpointer('labs', [types.slong], types.slong)
+            chain = ArgChain()
+            chain.arg(n)
+            n = ptr.call(chain, lltype.Signed)
+            return (n, x) + args
+        return None, f, None
+
     def define_compile_framework_1(cls):
         # a moving GC.  Supports malloc_varsize_nonmovable.  Simple test, works
         # without write_barriers and root stack enumeration.

Modified: pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_ztranslation.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_ztranslation.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/backend/x86/test/test_ztranslation.py	Fri Oct 22 23:09:43 2010
@@ -1,4 +1,4 @@
-import py, os
+import py, os, sys
 from pypy.tool.udir import udir
 from pypy.rlib.jit import JitDriver, OPTIMIZER_FULL, unroll_parameters
 from pypy.rlib.jit import PARAMETERS, dont_look_inside
@@ -63,8 +63,32 @@
                 if k - abs(j):  raise ValueError
                 if k - abs(-j): raise ValueError
             return total * 10
+        #
+        from pypy.rpython.lltypesystem import lltype, rffi
+        from pypy.rlib.libffi import types, CDLL, ArgChain
+        from pypy.rlib.test.test_libffi import get_libm_name
+        libm_name = get_libm_name(sys.platform)
+        jitdriver2 = JitDriver(greens=[], reds = ['i', 'func', 'res', 'x'])
+        def libffi_stuff(i, j):
+            lib = CDLL(libm_name)
+            func = lib.getpointer('fabs', [types.double], types.double)
+            res = 0.0
+            x = float(j)
+            while i > 0:
+                jitdriver2.jit_merge_point(i=i, res=res, func=func, x=x)
+                jitdriver2.can_enter_jit(i=i, res=res, func=func, x=x)
+                func = hint(func, promote=True)
+                argchain = ArgChain()
+                argchain.arg(x)
+                res = func.call(argchain, rffi.DOUBLE)
+                i -= 1
+            return res
+        #
+        def main(i, j):
+            return f(i, j) + libffi_stuff(i, j)
+        expected = f(40, -49)
         res = self.meta_interp(f, [40, -49])
-        assert res == f(40, -49)
+        assert res == expected
 
     def test_direct_assembler_call_translates(self):
         class Thing(object):

Modified: pypy/branch/fast-forward/pypy/jit/codewriter/call.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/codewriter/call.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/codewriter/call.py	Fri Oct 22 23:09:43 2010
@@ -277,3 +277,11 @@
             return seen.pop()
         else:
             return None
+
+    def could_be_green_field(self, GTYPE, fieldname):
+        GTYPE_fieldname = (GTYPE, fieldname)
+        for jd in self.jitdrivers_sd:
+            if jd.greenfield_info is not None:
+                if GTYPE_fieldname in jd.greenfield_info.green_fields:
+                    return True
+        return False

Modified: pypy/branch/fast-forward/pypy/jit/codewriter/codewriter.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/codewriter/codewriter.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/codewriter/codewriter.py	Fri Oct 22 23:09:43 2010
@@ -95,18 +95,18 @@
             print '%s:' % (ssarepr.name,)
             print format_assembler(ssarepr)
         else:
-            dir = udir.ensure("jitcodes", dir=1)
-            if portal_jitdriver:
-                name = "%02d_portal_runner" % (portal_jitdriver.index,)
-            elif ssarepr.name and ssarepr.name != '?':
-                name = ssarepr.name
-            else:
-                name = 'unnamed' % id(ssarepr)
-            i = 1
-            extra = ''
-            while name+extra in self._seen_files:
-                i += 1
-                extra = '.%d' % i
-            self._seen_files.add(name+extra)
-            dir.join(name+extra).write(format_assembler(ssarepr))
             log.dot()
+        dir = udir.ensure("jitcodes", dir=1)
+        if portal_jitdriver:
+            name = "%02d_portal_runner" % (portal_jitdriver.index,)
+        elif ssarepr.name and ssarepr.name != '?':
+            name = ssarepr.name
+        else:
+            name = 'unnamed' % id(ssarepr)
+        i = 1
+        extra = ''
+        while name+extra in self._seen_files:
+            i += 1
+            extra = '.%d' % i
+        self._seen_files.add(name+extra)
+        dir.join(name+extra).write(format_assembler(ssarepr))

Modified: pypy/branch/fast-forward/pypy/jit/codewriter/effectinfo.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/codewriter/effectinfo.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/codewriter/effectinfo.py	Fri Oct 22 23:09:43 2010
@@ -18,19 +18,34 @@
     # the 'oopspecindex' field is one of the following values:
     OS_NONE                     = 0    # normal case, no oopspec
     OS_ARRAYCOPY                = 1    # "list.ll_arraycopy"
-    OS_STR_CONCAT               = 2    # "stroruni.concat"
-    OS_UNI_CONCAT               = 3    # "stroruni.concat"
-    OS_STR_SLICE                = 4    # "stroruni.slice"
-    OS_UNI_SLICE                = 5    # "stroruni.slice"
-    OS_STR_EQUAL                = 6    # "stroruni.equal"
-    OS_UNI_EQUAL                = 7    # "stroruni.equal"
-    OS_STREQ_SLICE_CHECKNULL    = 8    # s2!=NULL and s1[x:x+length]==s2
-    OS_STREQ_SLICE_NONNULL      = 9    # s1[x:x+length]==s2   (assert s2!=NULL)
-    OS_STREQ_SLICE_CHAR         = 10   # s1[x:x+length]==char
-    OS_STREQ_NONNULL            = 11   # s1 == s2    (assert s1!=NULL,s2!=NULL)
-    OS_STREQ_NONNULL_CHAR       = 12   # s1 == char  (assert s1!=NULL)
-    OS_STREQ_CHECKNULL_CHAR     = 13   # s1!=NULL and s1==char
-    OS_STREQ_LENGTHOK           = 14   # s1 == s2    (assert len(s1)==len(s2))
+    OS_STR2UNICODE              = 2    # "str.str2unicode"
+    #
+    OS_STR_CONCAT               = 22   # "stroruni.concat"
+    OS_STR_SLICE                = 23   # "stroruni.slice"
+    OS_STR_EQUAL                = 24   # "stroruni.equal"
+    OS_STREQ_SLICE_CHECKNULL    = 25   # s2!=NULL and s1[x:x+length]==s2
+    OS_STREQ_SLICE_NONNULL      = 26   # s1[x:x+length]==s2   (assert s2!=NULL)
+    OS_STREQ_SLICE_CHAR         = 27   # s1[x:x+length]==char
+    OS_STREQ_NONNULL            = 28   # s1 == s2    (assert s1!=NULL,s2!=NULL)
+    OS_STREQ_NONNULL_CHAR       = 29   # s1 == char  (assert s1!=NULL)
+    OS_STREQ_CHECKNULL_CHAR     = 30   # s1!=NULL and s1==char
+    OS_STREQ_LENGTHOK           = 31   # s1 == s2    (assert len(s1)==len(s2))
+    #
+    OS_UNI_CONCAT               = 42   #
+    OS_UNI_SLICE                = 43   #
+    OS_UNI_EQUAL                = 44   #
+    OS_UNIEQ_SLICE_CHECKNULL    = 45   #
+    OS_UNIEQ_SLICE_NONNULL      = 46   #
+    OS_UNIEQ_SLICE_CHAR         = 47   #
+    OS_UNIEQ_NONNULL            = 48   #   the same for unicode
+    OS_UNIEQ_NONNULL_CHAR       = 49   #   (must be the same amount as for
+    OS_UNIEQ_CHECKNULL_CHAR     = 50   #   STR, in the same order)
+    OS_UNIEQ_LENGTHOK           = 51   #
+    _OS_offset_uni              = OS_UNI_CONCAT - OS_STR_CONCAT
+    #
+    OS_LIBFFI_PREPARE           = 60
+    OS_LIBFFI_PUSH_ARG          = 61
+    OS_LIBFFI_CALL              = 62
 
     def __new__(cls, readonly_descrs_fields,
                 write_descrs_fields, write_descrs_arrays,

Modified: pypy/branch/fast-forward/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/codewriter/jtransform.py	Fri Oct 22 23:09:43 2010
@@ -316,8 +316,14 @@
             prepare = self._handle_list_call
         elif oopspec_name.startswith('stroruni.'):
             prepare = self._handle_stroruni_call
+        elif oopspec_name == 'str.str2unicode':
+            prepare = self._handle_str2unicode_call
         elif oopspec_name.startswith('virtual_ref'):
             prepare = self._handle_virtual_ref_call
+        elif oopspec_name.startswith('jit.'):
+            prepare = self._handle_jit_call
+        elif oopspec_name.startswith('libffi_'):
+            prepare = self._handle_libffi_call
         else:
             prepare = self.prepare_builtin_call
         try:
@@ -427,7 +433,8 @@
                                   op.result)
 
     def rewrite_op_free(self, op):
-        assert op.args[1].value == 'raw'
+        flags = op.args[1].value
+        assert flags['flavor'] == 'raw'
         ARRAY = op.args[0].concretetype.TO
         return self._do_builtin_call(op, 'raw_free', [op.args[0]],
                                      extra = (ARRAY,), extrakey = ARRAY)
@@ -519,7 +526,12 @@
         # check for deepfrozen structures that force constant-folding
         immut = v_inst.concretetype.TO._immutable_field(c_fieldname.value)
         if immut:
-            pure = '_pure'
+            if (self.callcontrol is not None and
+                self.callcontrol.could_be_green_field(v_inst.concretetype.TO,
+                                                      c_fieldname.value)):
+                pure = '_greenfield'
+            else:
+                pure = '_pure'
             if immut == "[*]":
                 self.immutable_arrays[op.result] = True
         else:
@@ -819,6 +831,8 @@
     def rewrite_op_jit_marker(self, op):
         key = op.args[0].value
         jitdriver = op.args[1].value
+        if not jitdriver.active:
+            return []
         return getattr(self, 'handle_jit_marker__%s' % key)(op, jitdriver)
 
     def handle_jit_marker__jit_merge_point(self, op, jitdriver):
@@ -852,6 +866,15 @@
                     (self.graph,))
         return []
 
+    def _handle_jit_call(self, op, oopspec_name, args):
+        if oopspec_name == 'jit.debug':
+            return SpaceOperation('jit_debug', args, None)
+        elif oopspec_name == 'jit.assert_green':
+            kind = getkind(args[0].concretetype)
+            return SpaceOperation('%s_assert_green' % kind, args, None)
+        else:
+            raise AssertionError("missing support for %r" % oopspec_name)
+
     # ----------
     # Lists.
 
@@ -1028,8 +1051,10 @@
     # ----------
     # Strings and Unicodes.
 
-    def _handle_oopspec_call(self, op, args, oopspecindex):
+    def _handle_oopspec_call(self, op, args, oopspecindex, extraeffect=None):
         calldescr = self.callcontrol.getcalldescr(op, oopspecindex)
+        if extraeffect:
+            calldescr.get_extra_info().extraeffect = extraeffect
         if isinstance(op.args[0].value, str):
             pass  # for tests only
         else:
@@ -1055,28 +1080,32 @@
                             [c_func] + [varoftype(T) for T in argtypes],
                             varoftype(resulttype))
         calldescr = self.callcontrol.getcalldescr(op, oopspecindex)
-        func = heaptracker.adr2int(
-            llmemory.cast_ptr_to_adr(c_func.value))
+        if isinstance(c_func.value, str):    # in tests only
+            func = c_func.value
+        else:
+            func = heaptracker.adr2int(
+                llmemory.cast_ptr_to_adr(c_func.value))
         _callinfo_for_oopspec[oopspecindex] = calldescr, func
 
     def _handle_stroruni_call(self, op, oopspec_name, args):
-        if args[0].concretetype.TO == rstr.STR:
+        SoU = args[0].concretetype     # Ptr(STR) or Ptr(UNICODE)
+        if SoU.TO == rstr.STR:
             dict = {"stroruni.concat": EffectInfo.OS_STR_CONCAT,
                     "stroruni.slice":  EffectInfo.OS_STR_SLICE,
                     "stroruni.equal":  EffectInfo.OS_STR_EQUAL,
                     }
-        elif args[0].concretetype.TO == rstr.UNICODE:
+            CHR = lltype.Char
+        elif SoU.TO == rstr.UNICODE:
             dict = {"stroruni.concat": EffectInfo.OS_UNI_CONCAT,
                     "stroruni.slice":  EffectInfo.OS_UNI_SLICE,
                     "stroruni.equal":  EffectInfo.OS_UNI_EQUAL,
                     }
+            CHR = lltype.UniChar
         else:
             assert 0, "args[0].concretetype must be STR or UNICODE"
         #
         if oopspec_name == "stroruni.equal":
-            SoU = args[0].concretetype     # Ptr(STR) or Ptr(UNICODE)
             for otherindex, othername, argtypes, resulttype in [
-
                 (EffectInfo.OS_STREQ_SLICE_CHECKNULL,
                      "str.eq_slice_checknull",
                      [SoU, lltype.Signed, lltype.Signed, SoU],
@@ -1087,7 +1116,7 @@
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_SLICE_CHAR,
                      "str.eq_slice_char",
-                     [SoU, lltype.Signed, lltype.Signed, lltype.Char],
+                     [SoU, lltype.Signed, lltype.Signed, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_NONNULL,
                      "str.eq_nonnull",
@@ -1095,22 +1124,27 @@
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_NONNULL_CHAR,
                      "str.eq_nonnull_char",
-                     [SoU, lltype.Char],
+                     [SoU, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_CHECKNULL_CHAR,
                      "str.eq_checknull_char",
-                     [SoU, lltype.Char],
+                     [SoU, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_LENGTHOK,
                      "str.eq_lengthok",
                      [SoU, SoU],
                      lltype.Signed),
                 ]:
+                if args[0].concretetype.TO == rstr.UNICODE:
+                    otherindex += EffectInfo._OS_offset_uni
                 self._register_extra_helper(otherindex, othername,
                                             argtypes, resulttype)
         #
         return self._handle_oopspec_call(op, args, dict[oopspec_name])
 
+    def _handle_str2unicode_call(self, op, oopspec_name, args):
+        return self._handle_oopspec_call(op, args, EffectInfo.OS_STR2UNICODE)
+
     # ----------
     # VirtualRefs.
 
@@ -1121,6 +1155,23 @@
                                           vrefinfo.JIT_VIRTUAL_REF)
         return SpaceOperation(oopspec_name, list(args), op.result)
 
+    # -----------
+    # rlib.libffi
+
+    def _handle_libffi_call(self, op, oopspec_name, args):
+        if oopspec_name == 'libffi_prepare_call':
+            oopspecindex = EffectInfo.OS_LIBFFI_PREPARE
+            extraeffect = EffectInfo.EF_CANNOT_RAISE
+        elif oopspec_name.startswith('libffi_push_'):
+            oopspecindex = EffectInfo.OS_LIBFFI_PUSH_ARG
+            extraeffect = EffectInfo.EF_CANNOT_RAISE
+        elif oopspec_name.startswith('libffi_call_'):
+            oopspecindex = EffectInfo.OS_LIBFFI_CALL
+            extraeffect = EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE
+        else:
+            assert False, 'unsupported oopspec: %s' % oopspec_name
+        return self._handle_oopspec_call(op, args, oopspecindex, extraeffect)
+
     def rewrite_op_jit_force_virtual(self, op):
         return self._do_builtin_call(op)
 

Modified: pypy/branch/fast-forward/pypy/jit/codewriter/support.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/codewriter/support.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/codewriter/support.py	Fri Oct 22 23:09:43 2010
@@ -1,5 +1,5 @@
 import sys
-from pypy.rpython.lltypesystem import lltype, rclass
+from pypy.rpython.lltypesystem import lltype, rclass, rffi
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython import rlist
 from pypy.rpython.lltypesystem import rstr as ll_rstr, rdict as ll_rdict
@@ -8,6 +8,7 @@
 from pypy.rpython.ootypesystem import rdict as oo_rdict
 from pypy.rpython.llinterp import LLInterpreter
 from pypy.rpython.extregistry import ExtRegistryEntry
+from pypy.rpython.annlowlevel import cast_base_ptr_to_instance
 from pypy.translator.simplify import get_funcobj
 from pypy.translator.unsimplify import split_block
 from pypy.objspace.flow.model import Constant
@@ -60,7 +61,7 @@
     return rtyper.annotator.translator.graphs[0]
 
 def split_before_jit_merge_point(graph, portalblock, portalopindex):
-    """Find the block with 'jit_merge_point' and split just before,
+    """Split the block just before the 'jit_merge_point',
     making sure the input args are in the canonical order.
     """
     # split the block just before the jit_merge_point()
@@ -217,6 +218,33 @@
     else:
         return x
 
+
+# libffi support
+# --------------
+
+def func(llfunc):
+    from pypy.rlib.libffi import Func
+    return cast_base_ptr_to_instance(Func, llfunc)
+
+def _ll_1_libffi_prepare_call(llfunc):
+    return func(llfunc)._prepare()
+
+def _ll_4_libffi_push_int(llfunc, value, ll_args, i):
+    return func(llfunc)._push_int(value, ll_args, i)
+
+def _ll_4_libffi_push_float(llfunc, value, ll_args, i):
+    return func(llfunc)._push_float(value, ll_args, i)
+
+def _ll_3_libffi_call_int(llfunc, funcsym, ll_args):
+    return func(llfunc)._do_call(funcsym, ll_args, rffi.LONG)
+
+def _ll_3_libffi_call_float(llfunc, funcsym, ll_args):
+    return func(llfunc)._do_call(funcsym, ll_args, rffi.DOUBLE)
+
+def _ll_3_libffi_call_void(llfunc, funcsym, ll_args):
+    return func(llfunc)._do_call(funcsym, ll_args, lltype.Void)
+
+
 # in the following calls to builtins, the JIT is allowed to look inside:
 inline_calls_to = [
     ('int_floordiv_ovf_zer', [lltype.Signed, lltype.Signed], lltype.Signed),

Modified: pypy/branch/fast-forward/pypy/jit/codewriter/test/test_codewriter.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/codewriter/test/test_codewriter.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/codewriter/test/test_codewriter.py	Fri Oct 22 23:09:43 2010
@@ -45,6 +45,7 @@
         self.portal_graph = portal_graph
         self.portal_runner_ptr = "???"
         self.virtualizable_info = None
+        self.greenfield_info = None
 
 
 def test_loop():

Modified: pypy/branch/fast-forward/pypy/jit/codewriter/test/test_effectinfo.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/codewriter/test/test_effectinfo.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/codewriter/test/test_effectinfo.py	Fri Oct 22 23:09:43 2010
@@ -1,7 +1,8 @@
 from pypy.rpython.lltypesystem.rclass import OBJECT
 from pypy.rpython.lltypesystem import lltype
 from pypy.rpython.ootypesystem import ootype
-from pypy.jit.codewriter.effectinfo import effectinfo_from_writeanalyze
+from pypy.jit.codewriter.effectinfo import effectinfo_from_writeanalyze,\
+    EffectInfo
 
 class FakeCPU:
     def fielddescrof(self, T, fieldname):
@@ -9,6 +10,14 @@
     def arraydescrof(self, A):
         return ('arraydescr', A)
 
+def test_no_oopspec_duplicate():
+    # check that all the various EffectInfo.OS_* have unique values
+    oopspecs = set()
+    for name, value in EffectInfo.__dict__.iteritems():
+        if name.startswith('OS_'):
+            assert value not in oopspecs
+            oopspecs.add(value)
+
 def test_include_read_field():
     S = lltype.GcStruct("S", ("a", lltype.Signed))
     effects = frozenset([("readstruct", lltype.Ptr(S), "a")])

Modified: pypy/branch/fast-forward/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/codewriter/test/test_jtransform.py	Fri Oct 22 23:09:43 2010
@@ -77,7 +77,32 @@
 class FakeBuiltinCallControl:
     def guess_call_kind(self, op):
         return 'builtin'
-    def getcalldescr(self, op, oopspecindex):
+    def getcalldescr(self, op, oopspecindex=None):
+        assert oopspecindex is not None    # in this test
+        EI = effectinfo.EffectInfo
+        if oopspecindex != EI.OS_ARRAYCOPY:
+            PSTR = lltype.Ptr(rstr.STR)
+            PUNICODE = lltype.Ptr(rstr.UNICODE)
+            INT = lltype.Signed
+            UNICHAR = lltype.UniChar
+            argtypes = {
+             EI.OS_STR2UNICODE:([PSTR], PUNICODE),
+             EI.OS_STR_CONCAT: ([PSTR, PSTR], PSTR),
+             EI.OS_STR_SLICE:  ([PSTR, INT, INT], PSTR),
+             EI.OS_UNI_CONCAT: ([PUNICODE, PUNICODE], PUNICODE),
+             EI.OS_UNI_SLICE:  ([PUNICODE, INT, INT], PUNICODE),
+             EI.OS_UNI_EQUAL:  ([PUNICODE, PUNICODE], lltype.Bool),
+             EI.OS_UNIEQ_SLICE_CHECKNULL:([PUNICODE, INT, INT, PUNICODE], INT),
+             EI.OS_UNIEQ_SLICE_NONNULL:  ([PUNICODE, INT, INT, PUNICODE], INT),
+             EI.OS_UNIEQ_SLICE_CHAR:     ([PUNICODE, INT, INT, UNICHAR], INT),
+             EI.OS_UNIEQ_NONNULL:        ([PUNICODE, PUNICODE], INT),
+             EI.OS_UNIEQ_NONNULL_CHAR:   ([PUNICODE, UNICHAR], INT),
+             EI.OS_UNIEQ_CHECKNULL_CHAR: ([PUNICODE, UNICHAR], INT),
+             EI.OS_UNIEQ_LENGTHOK:       ([PUNICODE, PUNICODE], INT),
+            }
+            argtypes = argtypes[oopspecindex]
+            assert argtypes[0] == [v.concretetype for v in op.args[1:]]
+            assert argtypes[1] == op.result.concretetype
         return 'calldescr-%d' % oopspecindex
     def calldescr_canraise(self, calldescr):
         return False
@@ -662,6 +687,79 @@
     assert block.operations[1].result is None
     assert block.exits[0].args == [v1]
 
+def test_jit_merge_point_1():
+    class FakeJitDriverSD:
+        index = 42
+        class jitdriver:
+            greens = ['green1', 'green2', 'voidgreen3']
+            reds = ['red1', 'red2', 'voidred3']
+    jd = FakeJitDriverSD()
+    v1 = varoftype(lltype.Signed)
+    v2 = varoftype(lltype.Signed)
+    vvoid1 = varoftype(lltype.Void)
+    v3 = varoftype(lltype.Signed)
+    v4 = varoftype(lltype.Signed)
+    vvoid2 = varoftype(lltype.Void)
+    v5 = varoftype(lltype.Void)
+    op = SpaceOperation('jit_marker',
+                        [Constant('jit_merge_point', lltype.Void),
+                         Constant(jd.jitdriver, lltype.Void),
+                         v1, v2, vvoid1, v3, v4, vvoid2], v5)
+    tr = Transformer()
+    tr.portal_jd = jd
+    oplist = tr.rewrite_operation(op)
+    assert len(oplist) == 6
+    assert oplist[0].opname == '-live-'
+    assert oplist[1].opname == 'int_guard_value'
+    assert oplist[1].args   == [v1]
+    assert oplist[2].opname == '-live-'
+    assert oplist[3].opname == 'int_guard_value'
+    assert oplist[3].args   == [v2]
+    assert oplist[4].opname == 'jit_merge_point'
+    assert oplist[4].args[0].value == 42
+    assert list(oplist[4].args[1]) == [v1, v2]
+    assert list(oplist[4].args[4]) == [v3, v4]
+    assert oplist[5].opname == '-live-'
+
+def test_getfield_gc():
+    S = lltype.GcStruct('S', ('x', lltype.Char))
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
+def test_getfield_gc_pure():
+    S = lltype.GcStruct('S', ('x', lltype.Char),
+                        hints={'immutable': True})
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i_pure'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
+def test_getfield_gc_greenfield():
+    class FakeCC:
+        def get_vinfo(self, v):
+            return None
+        def could_be_green_field(self, S1, name1):
+            assert S1 is S
+            assert name1 == 'x'
+            return True
+    S = lltype.GcStruct('S', ('x', lltype.Char),
+                        hints={'immutable': True})
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU(), FakeCC()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i_greenfield'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
 def test_int_abs():
     v1 = varoftype(lltype.Signed)
     v2 = varoftype(lltype.Signed)
@@ -766,6 +864,46 @@
     assert op1.args[3] == ListOfKind('ref', [v1])
     assert op1.result == v4
 
+def test_str2unicode():
+    # test that the oopspec is present and correctly transformed
+    PSTR = lltype.Ptr(rstr.STR)
+    PUNICODE = lltype.Ptr(rstr.UNICODE)
+    FUNC = lltype.FuncType([PSTR], PUNICODE)
+    func = lltype.functionptr(FUNC, 'll_str2unicode',
+                            _callable=rstr.LLHelpers.ll_str2unicode)
+    v1 = varoftype(PSTR)
+    v2 = varoftype(PUNICODE)
+    op = SpaceOperation('direct_call', [const(func), v1], v2)
+    tr = Transformer(FakeCPU(), FakeBuiltinCallControl())
+    op1 = tr.rewrite_operation(op)
+    assert op1.opname == 'residual_call_r_r'
+    assert op1.args[0].value == func
+    assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_STR2UNICODE
+    assert op1.args[2] == ListOfKind('ref', [v1])
+    assert op1.result == v2
+
+def test_unicode_eq_checknull_char():
+    # test that the oopspec is present and correctly transformed
+    PUNICODE = lltype.Ptr(rstr.UNICODE)
+    FUNC = lltype.FuncType([PUNICODE, PUNICODE], lltype.Bool)
+    func = lltype.functionptr(FUNC, 'll_streq',
+                              _callable=rstr.LLHelpers.ll_streq)
+    v1 = varoftype(PUNICODE)
+    v2 = varoftype(PUNICODE)
+    v3 = varoftype(lltype.Bool)
+    op = SpaceOperation('direct_call', [const(func), v1, v2], v3)
+    tr = Transformer(FakeCPU(), FakeBuiltinCallControl())
+    op1 = tr.rewrite_operation(op)
+    assert op1.opname == 'residual_call_r_i'
+    assert op1.args[0].value == func
+    assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_UNI_EQUAL
+    assert op1.args[2] == ListOfKind('ref', [v1, v2])
+    assert op1.result == v3
+    # test that the OS_UNIEQ_* functions are registered
+    cifo = effectinfo._callinfo_for_oopspec
+    assert effectinfo.EffectInfo.OS_UNIEQ_SLICE_NONNULL in cifo
+    assert effectinfo.EffectInfo.OS_UNIEQ_CHECKNULL_CHAR in cifo
+
 def test_list_ll_arraycopy():
     from pypy.rlib.rgc import ll_arraycopy
     LIST = lltype.GcArray(lltype.Signed)

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/blackhole.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/blackhole.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/blackhole.py	Fri Oct 22 23:09:43 2010
@@ -760,6 +760,20 @@
     def bhimpl_debug_fatalerror(msg):
         llop.debug_fatalerror(lltype.Void, msg)
 
+    @arguments("r", "i", "i", "i", "i")
+    def bhimpl_jit_debug(string, arg1=0, arg2=0, arg3=0, arg4=0):
+        pass
+
+    @arguments("i")
+    def bhimpl_int_assert_green(x):
+        pass
+    @arguments("r")
+    def bhimpl_ref_assert_green(x):
+        pass
+    @arguments("f")
+    def bhimpl_float_assert_green(x):
+        pass
+
     # ----------
     # the main hints and recursive calls
 
@@ -1073,6 +1087,10 @@
     bhimpl_getfield_vable_r = bhimpl_getfield_gc_r
     bhimpl_getfield_vable_f = bhimpl_getfield_gc_f
 
+    bhimpl_getfield_gc_i_greenfield = bhimpl_getfield_gc_i
+    bhimpl_getfield_gc_r_greenfield = bhimpl_getfield_gc_r
+    bhimpl_getfield_gc_f_greenfield = bhimpl_getfield_gc_f
+
     @arguments("cpu", "i", "d", returns="i")
     def bhimpl_getfield_raw_i(cpu, struct, fielddescr):
         return cpu.bh_getfield_raw_i(struct, fielddescr)

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/compile.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/compile.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/compile.py	Fri Oct 22 23:09:43 2010
@@ -370,7 +370,8 @@
         from pypy.jit.metainterp.resume import force_from_resumedata
         metainterp_sd = self.metainterp_sd
         vinfo = self.jitdriver_sd.virtualizable_info
-        all_virtuals = force_from_resumedata(metainterp_sd, self, vinfo)
+        ginfo = self.jitdriver_sd.greenfield_info
+        all_virtuals = force_from_resumedata(metainterp_sd, self, vinfo, ginfo)
         # The virtualizable data was stored on the real virtualizable above.
         # Handle all_virtuals: keep them for later blackholing from the
         # future failure of the GUARD_NOT_FORCED

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/executor.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/executor.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/executor.py	Fri Oct 22 23:09:43 2010
@@ -80,6 +80,9 @@
 do_call_loopinvariant = do_call
 do_call_may_force = do_call
 
+def do_call_c(cpu, metainterp, argboxes, descr):
+    raise NotImplementedError("Should never be called directly")
+
 def do_getarrayitem_gc(cpu, _, arraybox, indexbox, arraydescr):
     array = arraybox.getref_base()
     index = indexbox.getint()
@@ -304,6 +307,7 @@
                          rop.CALL_ASSEMBLER,
                          rop.COND_CALL_GC_WB,
                          rop.DEBUG_MERGE_POINT,
+                         rop.JIT_DEBUG,
                          rop.SETARRAYITEM_RAW,
                          ):      # list of opcodes never executed by pyjitpl
                 continue

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/graphpage.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/graphpage.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/graphpage.py	Fri Oct 22 23:09:43 2010
@@ -153,7 +153,7 @@
         opindex = opstartindex
         while True:
             op = operations[opindex]
-            lines.append(repr(op))
+            lines.append(op.repr(graytext=True))
             if is_interesting_guard(op):
                 tgt = op.getdescr()._debug_suboperations[0]
                 tgt_g, tgt_i = self.all_operations[tgt]

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/history.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/history.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/history.py	Fri Oct 22 23:09:43 2010
@@ -698,6 +698,21 @@
     return result
 _const_ptr_for_string = {}
 
+def get_const_ptr_for_unicode(s):
+    from pypy.rpython.annlowlevel import llunicode
+    if not we_are_translated():
+        try:
+            return _const_ptr_for_unicode[s]
+        except KeyError:
+            pass
+    if isinstance(s, str):
+        s = unicode(s)
+    result = ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, llunicode(s)))
+    if not we_are_translated():
+        _const_ptr_for_unicode[s] = result
+    return result
+_const_ptr_for_unicode = {}
+
 # ____________________________________________________________
 
 # The TreeLoop class contains a loop or a generalized loop, i.e. a tree

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/jitdriver.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/jitdriver.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/jitdriver.py	Fri Oct 22 23:09:43 2010
@@ -13,8 +13,10 @@
     #    self.num_red_args      ... pypy.jit.metainterp.warmspot
     #    self.result_type       ... pypy.jit.metainterp.warmspot
     #    self.virtualizable_info... pypy.jit.metainterp.warmspot
+    #    self.greenfield_info   ... pypy.jit.metainterp.warmspot
     #    self.warmstate         ... pypy.jit.metainterp.warmspot
     #    self.handle_jitexc_from_bh pypy.jit.metainterp.warmspot
+    #    self.no_loop_header    ... pypy.jit.metainterp.warmspot
     #    self.portal_finishtoken... pypy.jit.metainterp.pyjitpl
     #    self.index             ... pypy.jit.codewriter.call
     #    self.mainjitcode       ... pypy.jit.codewriter.call

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/optimize_nopspec.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/optimize_nopspec.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/optimize_nopspec.py	Fri Oct 22 23:09:43 2010
@@ -14,6 +14,9 @@
 def _optimize_loop(metainterp_sd, old_loop_tokens, loop):
     cpu = metainterp_sd.cpu
     metainterp_sd.logger_noopt.log_loop(loop.inputargs, loop.operations)
+    # XXX the following lines are probably still needed, to discard invalid
+    # loops. bit silly to run a full perfect specialization and throw the
+    # result away.
     finder = PerfectSpecializationFinder(cpu)
     finder.find_nodes_loop(loop, False)
     if old_loop_tokens:
@@ -31,6 +34,7 @@
 def _optimize_bridge(metainterp_sd, old_loop_tokens, bridge):
     cpu = metainterp_sd.cpu    
     metainterp_sd.logger_noopt.log_loop(bridge.inputargs, bridge.operations)
+    # XXX same comment as above applies
     finder = BridgeSpecializationFinder(cpu)
     finder.find_nodes_bridge(bridge)
     if old_loop_tokens:

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/__init__.py	Fri Oct 22 23:09:43 2010
@@ -3,6 +3,7 @@
 from pypy.jit.metainterp.optimizeopt.intbounds import OptIntBounds
 from pypy.jit.metainterp.optimizeopt.virtualize import OptVirtualize
 from pypy.jit.metainterp.optimizeopt.heap import OptHeap
+from pypy.jit.metainterp.optimizeopt.fficall import OptFfiCall
 from pypy.jit.metainterp.optimizeopt.string import OptString
 
 def optimize_loop_1(metainterp_sd, loop, virtuals=True):
@@ -16,6 +17,7 @@
                      OptVirtualize(),
                      OptString(),
                      OptHeap(),
+                     OptFfiCall(),
                     ]
     optimizer = Optimizer(metainterp_sd, loop, optimizations, virtuals)
     optimizer.propagate_all_forward()

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/intbounds.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/intbounds.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/intbounds.py	Fri Oct 22 23:09:43 2010
@@ -191,6 +191,7 @@
         v1.intbound.make_ge(IntLowerBound(0))
 
     optimize_STRLEN = optimize_ARRAYLEN_GC
+    optimize_UNICODELEN = optimize_ARRAYLEN_GC
 
     def make_int_lt(self, box1, box2):
         v1 = self.getvalue(box1)

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/rewrite.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/rewrite.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/rewrite.py	Fri Oct 22 23:09:43 2010
@@ -3,6 +3,7 @@
 from pypy.jit.metainterp.history import ConstInt
 from pypy.jit.metainterp.optimizeutil import _findall
 from pypy.jit.metainterp.resoperation import rop, ResOperation
+from pypy.jit.codewriter.effectinfo import EffectInfo
 
 class OptRewrite(Optimization):
     """Rewrite operations into equivalent, cheaper operations.
@@ -245,6 +246,9 @@
     def optimize_CALL_LOOPINVARIANT(self, op):
         funcvalue = self.getvalue(op.getarg(0))
         if not funcvalue.is_constant():
+            # XXX this code path is never executed in tests nor in production.
+            # in fact, it can't even happen since residual_call in codewriter
+            # expects a compile-time constant
             self.emit_operation(op)
             return
         key = make_hashable_int(op.getarg(0).getint())
@@ -323,8 +327,37 @@
 ##            return
 ##        self.emit_operation(op)
 
-optimize_ops = _findall(OptRewrite, 'optimize_')
-        
+    def optimize_CALL(self, op):
+        # dispatch based on 'oopspecindex' to a method that handles
+        # specifically the given oopspec call.  For non-oopspec calls,
+        # oopspecindex is just zero.
+        effectinfo = op.getdescr().get_extra_info()
+        if effectinfo is not None:
+            oopspecindex = effectinfo.oopspecindex
+            if oopspecindex == EffectInfo.OS_ARRAYCOPY:
+                if self._optimize_CALL_ARRAYCOPY(op):
+                    return
+        self.emit_operation(op)
 
+    def _optimize_CALL_ARRAYCOPY(self, op):
+        source_value = self.getvalue(op.getarg(1))
+        dest_value = self.getvalue(op.getarg(2))
+        source_start_box = self.get_constant_box(op.getarg(3))
+        dest_start_box = self.get_constant_box(op.getarg(4))
+        length = self.get_constant_box(op.getarg(5))
+        if (source_value.is_virtual() and source_start_box and dest_start_box
+            and length and dest_value.is_virtual()):
+            # XXX optimize the case where dest value is not virtual,
+            #     but we still can avoid a mess
+            source_start = source_start_box.getint()
+            dest_start = dest_start_box.getint()
+            for index in range(length.getint()):
+                val = source_value.getitem(index + source_start)
+                dest_value.setitem(index + dest_start, val)
+            return True
+        if length and length.getint() == 0:
+            return True # 0-length arraycopy
+        return False
+
+optimize_ops = _findall(OptRewrite, 'optimize_')
 
-        

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/string.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/string.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/string.py	Fri Oct 22 23:09:43 2010
@@ -3,6 +3,7 @@
 from pypy.jit.metainterp.history import Box, BoxInt, BoxPtr
 from pypy.jit.metainterp.history import Const, ConstInt, ConstPtr
 from pypy.jit.metainterp.history import get_const_ptr_for_string
+from pypy.jit.metainterp.history import get_const_ptr_for_unicode
 from pypy.jit.metainterp.resoperation import rop, ResOperation
 from pypy.jit.metainterp.optimizeopt import optimizer, virtualize
 from pypy.jit.metainterp.optimizeopt.optimizer import CONST_0, CONST_1
@@ -11,56 +12,106 @@
 from pypy.jit.codewriter.effectinfo import EffectInfo, callinfo_for_oopspec
 from pypy.jit.codewriter import heaptracker
 from pypy.rlib.unroll import unrolling_iterable
+from pypy.rlib.objectmodel import specialize, we_are_translated
+
+
+class StrOrUnicode(object):
+    def __init__(self, LLTYPE, hlstr, emptystr, chr,
+                 NEWSTR, STRLEN, STRGETITEM, STRSETITEM, COPYSTRCONTENT,
+                 OS_offset):
+        self.LLTYPE = LLTYPE
+        self.hlstr = hlstr
+        self.emptystr = emptystr
+        self.chr = chr
+        self.NEWSTR = NEWSTR
+        self.STRLEN = STRLEN
+        self.STRGETITEM = STRGETITEM
+        self.STRSETITEM = STRSETITEM
+        self.COPYSTRCONTENT = COPYSTRCONTENT
+        self.OS_offset = OS_offset
+
+    def _freeze_(self):
+        return True
+
+mode_string = StrOrUnicode(rstr.STR, annlowlevel.hlstr, '', chr,
+                           rop.NEWSTR, rop.STRLEN, rop.STRGETITEM,
+                           rop.STRSETITEM, rop.COPYSTRCONTENT, 0)
+mode_unicode = StrOrUnicode(rstr.UNICODE, annlowlevel.hlunicode, u'', unichr,
+                            rop.NEWUNICODE, rop.UNICODELEN, rop.UNICODEGETITEM,
+                            rop.UNICODESETITEM, rop.COPYUNICODECONTENT,
+                            EffectInfo._OS_offset_uni)
+
+# ____________________________________________________________
 
 
 class __extend__(optimizer.OptValue):
     """New methods added to the base class OptValue for this file."""
 
-    def getstrlen(self, newoperations):
-        s = self.get_constant_string()
-        if s is not None:
-            return ConstInt(len(s))
+    def getstrlen(self, newoperations, mode):
+        if mode is mode_string:
+            s = self.get_constant_string_spec(mode_string)
+            if s is not None:
+                return ConstInt(len(s))
         else:
-            if newoperations is None:
-                return None
-            self.ensure_nonnull()
-            box = self.force_box()
-            lengthbox = BoxInt()
-            newoperations.append(ResOperation(rop.STRLEN, [box], lengthbox))
-            return lengthbox
+            s = self.get_constant_string_spec(mode_unicode)
+            if s is not None:
+                return ConstInt(len(s))
+        if newoperations is None:
+            return None
+        self.ensure_nonnull()
+        box = self.force_box()
+        lengthbox = BoxInt()
+        newoperations.append(ResOperation(mode.STRLEN, [box], lengthbox))
+        return lengthbox
 
-    def get_constant_string(self):
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
         if self.is_constant():
-            s = self.box.getref(lltype.Ptr(rstr.STR))
-            return annlowlevel.hlstr(s)
+            s = self.box.getref(lltype.Ptr(mode.LLTYPE))
+            return mode.hlstr(s)
         else:
             return None
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
         # Copies the pointer-to-string 'self' into the target string
         # given by 'targetbox', at the specified offset.  Returns the offset
         # at the end of the copy.
-        lengthbox = self.getstrlen(newoperations)
+        lengthbox = self.getstrlen(newoperations, mode)
         srcbox = self.force_box()
         return copy_str_content(newoperations, srcbox, targetbox,
-                                CONST_0, offsetbox, lengthbox)
+                                CONST_0, offsetbox, lengthbox, mode)
 
 
 class VAbstractStringValue(virtualize.AbstractVirtualValue):
-    _attrs_ = ()
+    _attrs_ = ('mode',)
+
+    def __init__(self, optimizer, keybox, source_op, mode):
+        virtualize.AbstractVirtualValue.__init__(self, optimizer, keybox,
+                                                 source_op)
+        self.mode = mode
 
     def _really_force(self):
-        s = self.get_constant_string()
-        if s is not None:
-            c_s = get_const_ptr_for_string(s)
-            self.make_constant(c_s)
-            return
+        if self.mode is mode_string:
+            s = self.get_constant_string_spec(mode_string)
+            if s is not None:
+                c_s = get_const_ptr_for_string(s)
+                self.make_constant(c_s)
+                return
+        else:
+            s = self.get_constant_string_spec(mode_unicode)
+            if s is not None:
+                c_s = get_const_ptr_for_unicode(s)
+                self.make_constant(c_s)
+                return
         assert self.source_op is not None
         self.box = box = self.source_op.result
         newoperations = self.optimizer.newoperations
-        lengthbox = self.getstrlen(newoperations)
-        newoperations.append(ResOperation(rop.NEWSTR, [lengthbox], box))
-        self.string_copy_parts(newoperations, box, CONST_0)
+        lengthbox = self.getstrlen(newoperations, self.mode)
+        op = ResOperation(self.mode.NEWSTR, [lengthbox], box)
+        if not we_are_translated():
+            op.name = 'FORCE'
+        newoperations.append(op)
+        self.string_copy_parts(newoperations, box, CONST_0, self.mode)
 
 
 class VStringPlainValue(VAbstractStringValue):
@@ -74,7 +125,7 @@
         assert 0 <= start <= stop <= len(longerlist)
         self._chars = longerlist[start:stop]
 
-    def getstrlen(self, _):
+    def getstrlen(self, _, mode):
         if self._lengthbox is None:
             self._lengthbox = ConstInt(len(self._chars))
         return self._lengthbox
@@ -86,18 +137,21 @@
         assert isinstance(charvalue, optimizer.OptValue)
         self._chars[index] = charvalue
 
-    def get_constant_string(self):
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
         for c in self._chars:
             if c is optimizer.CVAL_UNINITIALIZED_ZERO or not c.is_constant():
                 return None
-        return ''.join([chr(c.box.getint()) for c in self._chars])
+        return mode.emptystr.join([mode.chr(c.box.getint())
+                                   for c in self._chars])
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
         for i in range(len(self._chars)):
             charbox = self._chars[i].force_box()
-            newoperations.append(ResOperation(rop.STRSETITEM, [targetbox,
-                                                               offsetbox,
-                                                               charbox], None))
+            newoperations.append(ResOperation(mode.STRSETITEM, [targetbox,
+                                                                offsetbox,
+                                                                charbox],
+                                              None))
             offsetbox = _int_add(newoperations, offsetbox, CONST_1)
         return offsetbox
 
@@ -109,7 +163,7 @@
                 value.get_args_for_fail(modifier)
 
     def _make_virtual(self, modifier):
-        return modifier.make_vstrplain()
+        return modifier.make_vstrplain(self.mode is mode_unicode)
 
 
 class VStringConcatValue(VAbstractStringValue):
@@ -120,23 +174,24 @@
         self.right = right
         self.lengthbox = lengthbox
 
-    def getstrlen(self, _):
+    def getstrlen(self, _, mode):
         return self.lengthbox
 
-    def get_constant_string(self):
-        s1 = self.left.get_constant_string()
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
+        s1 = self.left.get_constant_string_spec(mode)
         if s1 is None:
             return None
-        s2 = self.right.get_constant_string()
+        s2 = self.right.get_constant_string_spec(mode)
         if s2 is None:
             return None
         return s1 + s2
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
         offsetbox = self.left.string_copy_parts(newoperations, targetbox,
-                                                offsetbox)
+                                                offsetbox, mode)
         offsetbox = self.right.string_copy_parts(newoperations, targetbox,
-                                                 offsetbox)
+                                                 offsetbox, mode)
         return offsetbox
 
     def get_args_for_fail(self, modifier):
@@ -150,7 +205,7 @@
             self.right.get_args_for_fail(modifier)
 
     def _make_virtual(self, modifier):
-        return modifier.make_vstrconcat()
+        return modifier.make_vstrconcat(self.mode is mode_unicode)
 
 
 class VStringSliceValue(VAbstractStringValue):
@@ -162,12 +217,13 @@
         self.vstart = vstart
         self.vlength = vlength
 
-    def getstrlen(self, _):
+    def getstrlen(self, _, mode):
         return self.vlength.force_box()
 
-    def get_constant_string(self):
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
         if self.vstart.is_constant() and self.vlength.is_constant():
-            s1 = self.vstr.get_constant_string()
+            s1 = self.vstr.get_constant_string_spec(mode)
             if s1 is None:
                 return None
             start = self.vstart.box.getint()
@@ -177,12 +233,12 @@
             return s1[start : start + length]
         return None
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
-        lengthbox = self.getstrlen(newoperations)
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
+        lengthbox = self.getstrlen(newoperations, mode)
         return copy_str_content(newoperations,
                                 self.vstr.force_box(), targetbox,
                                 self.vstart.force_box(), offsetbox,
-                                lengthbox)
+                                lengthbox, mode)
 
     def get_args_for_fail(self, modifier):
         if self.box is None and not modifier.already_seen_virtual(self.keybox):
@@ -195,11 +251,11 @@
             self.vlength.get_args_for_fail(modifier)
 
     def _make_virtual(self, modifier):
-        return modifier.make_vstrslice()
+        return modifier.make_vstrslice(self.mode is mode_unicode)
 
 
 def copy_str_content(newoperations, srcbox, targetbox,
-                     srcoffsetbox, offsetbox, lengthbox):
+                     srcoffsetbox, offsetbox, lengthbox, mode):
     if isinstance(srcbox, ConstPtr) and isinstance(srcoffsetbox, Const):
         M = 5
     else:
@@ -208,17 +264,18 @@
         # up to M characters are done "inline", i.e. with STRGETITEM/STRSETITEM
         # instead of just a COPYSTRCONTENT.
         for i in range(lengthbox.value):
-            charbox = _strgetitem(newoperations, srcbox, srcoffsetbox)
+            charbox = _strgetitem(newoperations, srcbox, srcoffsetbox, mode)
             srcoffsetbox = _int_add(newoperations, srcoffsetbox, CONST_1)
-            newoperations.append(ResOperation(rop.STRSETITEM, [targetbox,
-                                                               offsetbox,
-                                                               charbox], None))
+            newoperations.append(ResOperation(mode.STRSETITEM, [targetbox,
+                                                                offsetbox,
+                                                                charbox],
+                                              None))
             offsetbox = _int_add(newoperations, offsetbox, CONST_1)
     else:
         nextoffsetbox = _int_add(newoperations, offsetbox, lengthbox)
-        op = ResOperation(rop.COPYSTRCONTENT, [srcbox, targetbox,
-                                               srcoffsetbox, offsetbox,
-                                               lengthbox], None)
+        op = ResOperation(mode.COPYSTRCONTENT, [srcbox, targetbox,
+                                                srcoffsetbox, offsetbox,
+                                                lengthbox], None)
         newoperations.append(op)
         offsetbox = nextoffsetbox
     return offsetbox
@@ -245,12 +302,16 @@
     newoperations.append(ResOperation(rop.INT_SUB, [box1, box2], resbox))
     return resbox
 
-def _strgetitem(newoperations, strbox, indexbox):
+def _strgetitem(newoperations, strbox, indexbox, mode):
     if isinstance(strbox, ConstPtr) and isinstance(indexbox, ConstInt):
-        s = strbox.getref(lltype.Ptr(rstr.STR))
-        return ConstInt(ord(s.chars[indexbox.getint()]))
+        if mode is mode_string:
+            s = strbox.getref(lltype.Ptr(rstr.STR))
+            return ConstInt(ord(s.chars[indexbox.getint()]))
+        else:
+            s = strbox.getref(lltype.Ptr(rstr.UNICODE))
+            return ConstInt(ord(s.chars[indexbox.getint()]))
     resbox = BoxInt()
-    newoperations.append(ResOperation(rop.STRGETITEM, [strbox, indexbox],
+    newoperations.append(ResOperation(mode.STRGETITEM, [strbox, indexbox],
                                       resbox))
     return resbox
 
@@ -258,62 +319,34 @@
 class OptString(optimizer.Optimization):
     "Handling of strings and unicodes."
 
-    def make_vstring_plain(self, box, source_op=None):
-        vvalue = VStringPlainValue(self.optimizer, box, source_op)
+    def make_vstring_plain(self, box, source_op, mode):
+        vvalue = VStringPlainValue(self.optimizer, box, source_op, mode)
         self.make_equal_to(box, vvalue)
         return vvalue
 
-    def make_vstring_concat(self, box, source_op=None):
-        vvalue = VStringConcatValue(self.optimizer, box, source_op)
+    def make_vstring_concat(self, box, source_op, mode):
+        vvalue = VStringConcatValue(self.optimizer, box, source_op, mode)
         self.make_equal_to(box, vvalue)
         return vvalue
 
-    def make_vstring_slice(self, box, source_op=None):
-        vvalue = VStringSliceValue(self.optimizer, box, source_op)
+    def make_vstring_slice(self, box, source_op, mode):
+        vvalue = VStringSliceValue(self.optimizer, box, source_op, mode)
         self.make_equal_to(box, vvalue)
         return vvalue
 
-    def optimize_CALL(self, op):
-        # dispatch based on 'oopspecindex' to a method that handles
-        # specifically the given oopspec call.  For non-oopspec calls,
-        # oopspecindex is just zero.
-        effectinfo = op.getdescr().get_extra_info()
-        if effectinfo is not None:
-            oopspecindex = effectinfo.oopspecindex
-            for value, meth in opt_call_oopspec_ops:
-                if oopspecindex == value:
-                    if meth(self, op):
-                        return
-        self.emit_operation(op)
-
-    def opt_call_oopspec_ARRAYCOPY(self, op):
-        source_value = self.getvalue(op.getarg(1))
-        dest_value = self.getvalue(op.getarg(2))
-        source_start_box = self.get_constant_box(op.getarg(3))
-        dest_start_box = self.get_constant_box(op.getarg(4))
-        length = self.get_constant_box(op.getarg(5))
-        if (source_value.is_virtual() and source_start_box and dest_start_box
-            and length and dest_value.is_virtual()):
-            # XXX optimize the case where dest value is not virtual,
-            #     but we still can avoid a mess
-            source_start = source_start_box.getint()
-            dest_start = dest_start_box.getint()
-            for index in range(length.getint()):
-                val = source_value.getitem(index + source_start)
-                dest_value.setitem(index + dest_start, val)
-            return True
-        if length and length.getint() == 0:
-            return True # 0-length arraycopy
-        return False
-
     def optimize_NEWSTR(self, op):
+        self._optimize_NEWSTR(op, mode_string)
+    def optimize_NEWUNICODE(self, op):
+        self._optimize_NEWSTR(op, mode_unicode)
+
+    def _optimize_NEWSTR(self, op, mode):
         length_box = self.get_constant_box(op.getarg(0))
         if length_box:
             # if the original 'op' did not have a ConstInt as argument,
             # build a new one with the ConstInt argument
             if not isinstance(op.getarg(0), ConstInt):
-                op = ResOperation(rop.NEWSTR, [length_box], op.result)
-            vvalue = self.make_vstring_plain(op.result, op)
+                op = ResOperation(mode.NEWSTR, [length_box], op.result)
+            vvalue = self.make_vstring_plain(op.result, op, mode)
             vvalue.setup(length_box.getint())
         else:
             self.getvalue(op.result).ensure_nonnull()
@@ -329,13 +362,20 @@
         value.ensure_nonnull()
         self.emit_operation(op)
 
+    optimize_UNICODESETITEM = optimize_STRSETITEM
+
     def optimize_STRGETITEM(self, op):
+        self._optimize_STRGETITEM(op, mode_string)
+    def optimize_UNICODEGETITEM(self, op):
+        self._optimize_STRGETITEM(op, mode_unicode)
+
+    def _optimize_STRGETITEM(self, op, mode):
         value = self.getvalue(op.getarg(0))
         vindex = self.getvalue(op.getarg(1))
-        vresult = self.strgetitem(value, vindex)
+        vresult = self.strgetitem(value, vindex, mode)
         self.make_equal_to(op.result, vresult)
 
-    def strgetitem(self, value, vindex):
+    def strgetitem(self, value, vindex, mode):
         value.ensure_nonnull()
         #
         if value.is_virtual() and isinstance(value, VStringSliceValue):
@@ -350,28 +390,71 @@
                 return value.getitem(vindex.box.getint())
         #
         resbox = _strgetitem(self.optimizer.newoperations,
-                             value.force_box(),vindex.force_box())
+                             value.force_box(),vindex.force_box(), mode)
         return self.getvalue(resbox)
 
     def optimize_STRLEN(self, op):
+        self._optimize_STRLEN(op, mode_string)
+    def optimize_UNICODELEN(self, op):
+        self._optimize_STRLEN(op, mode_unicode)
+
+    def _optimize_STRLEN(self, op, mode):
         value = self.getvalue(op.getarg(0))
-        lengthbox = value.getstrlen(self.optimizer.newoperations)
+        lengthbox = value.getstrlen(self.optimizer.newoperations, mode)
         self.make_equal_to(op.result, self.getvalue(lengthbox))
 
-    def opt_call_oopspec_STR_CONCAT(self, op):
+    def optimize_CALL(self, op):
+        # dispatch based on 'oopspecindex' to a method that handles
+        # specifically the given oopspec call.  For non-oopspec calls,
+        # oopspecindex is just zero.
+        effectinfo = op.getdescr().get_extra_info()
+        if effectinfo is not None:
+            oopspecindex = effectinfo.oopspecindex
+            for value, meth in opt_call_oopspec_ops:
+                if oopspecindex == value:      # a match with the OS_STR_xxx
+                    if meth(self, op, mode_string):
+                        return
+                    break
+                if oopspecindex == value + EffectInfo._OS_offset_uni:
+                    # a match with the OS_UNI_xxx
+                    if meth(self, op, mode_unicode):
+                        return
+                    break
+            if oopspecindex == EffectInfo.OS_STR2UNICODE:
+                if self.opt_call_str_STR2UNICODE(op):
+                    return
+        self.emit_operation(op)
+
+    def opt_call_str_STR2UNICODE(self, op):
+        # Constant-fold unicode("constant string").
+        # More generally, supporting non-constant but virtual cases is
+        # not obvious, because of the exception UnicodeDecodeError that
+        # can be raised by ll_str2unicode()
+        varg = self.getvalue(op.getarg(1))
+        s = varg.get_constant_string_spec(mode_string)
+        if s is None:
+            return False
+        try:
+            u = unicode(s)
+        except UnicodeDecodeError:
+            return False
+        self.make_constant(op.result, get_const_ptr_for_unicode(u))
+        return True
+
+    def opt_call_stroruni_STR_CONCAT(self, op, mode):
         vleft = self.getvalue(op.getarg(1))
         vright = self.getvalue(op.getarg(2))
         vleft.ensure_nonnull()
         vright.ensure_nonnull()
         newoperations = self.optimizer.newoperations
-        len1box = vleft.getstrlen(newoperations)
-        len2box = vright.getstrlen(newoperations)
+        len1box = vleft.getstrlen(newoperations, mode)
+        len2box = vright.getstrlen(newoperations, mode)
         lengthbox = _int_add(newoperations, len1box, len2box)
-        value = self.make_vstring_concat(op.result, op)
+        value = self.make_vstring_concat(op.result, op, mode)
         value.setup(vleft, vright, lengthbox)
         return True
 
-    def opt_call_oopspec_STR_SLICE(self, op):
+    def opt_call_stroruni_STR_SLICE(self, op, mode):
         newoperations = self.optimizer.newoperations
         vstr = self.getvalue(op.getarg(1))
         vstart = self.getvalue(op.getarg(2))
@@ -380,7 +463,7 @@
         if (isinstance(vstr, VStringPlainValue) and vstart.is_constant()
             and vstop.is_constant()):
             # slicing with constant bounds of a VStringPlainValue
-            value = self.make_vstring_plain(op.result, op)
+            value = self.make_vstring_plain(op.result, op, mode)
             value.setup_slice(vstr._chars, vstart.box.getint(),
                                            vstop.box.getint())
             return True
@@ -398,16 +481,16 @@
                                 vstart.force_box())
             vstart = self.getvalue(startbox)
         #
-        value = self.make_vstring_slice(op.result, op)
+        value = self.make_vstring_slice(op.result, op, mode)
         value.setup(vstr, vstart, self.getvalue(lengthbox))
         return True
 
-    def opt_call_oopspec_STR_EQUAL(self, op):
+    def opt_call_stroruni_STR_EQUAL(self, op, mode):
         v1 = self.getvalue(op.getarg(1))
         v2 = self.getvalue(op.getarg(2))
         #
-        l1box = v1.getstrlen(None)
-        l2box = v2.getstrlen(None)
+        l1box = v1.getstrlen(None, mode)
+        l2box = v2.getstrlen(None, mode)
         if (l1box is not None and l2box is not None and
             isinstance(l1box, ConstInt) and
             isinstance(l2box, ConstInt) and
@@ -416,13 +499,13 @@
             self.make_constant(op.result, CONST_0)
             return True
         #
-        if self.handle_str_equal_level1(v1, v2, op.result):
+        if self.handle_str_equal_level1(v1, v2, op.result, mode):
             return True
-        if self.handle_str_equal_level1(v2, v1, op.result):
+        if self.handle_str_equal_level1(v2, v1, op.result, mode):
             return True
-        if self.handle_str_equal_level2(v1, v2, op.result):
+        if self.handle_str_equal_level2(v1, v2, op.result, mode):
             return True
-        if self.handle_str_equal_level2(v2, v1, op.result):
+        if self.handle_str_equal_level2(v2, v1, op.result, mode):
             return True
         #
         if v1.is_nonnull() and v2.is_nonnull():
@@ -434,37 +517,37 @@
             else:
                 do = EffectInfo.OS_STREQ_NONNULL
             self.generate_modified_call(do, [v1.force_box(),
-                                             v2.force_box()], op.result)
+                                             v2.force_box()], op.result, mode)
             return True
         return False
 
-    def handle_str_equal_level1(self, v1, v2, resultbox):
-        l2box = v2.getstrlen(None)
+    def handle_str_equal_level1(self, v1, v2, resultbox, mode):
+        l2box = v2.getstrlen(None, mode)
         if isinstance(l2box, ConstInt):
             if l2box.value == 0:
-                lengthbox = v1.getstrlen(self.optimizer.newoperations)
+                lengthbox = v1.getstrlen(self.optimizer.newoperations, mode)
                 seo = self.optimizer.send_extra_operation
                 seo(ResOperation(rop.INT_EQ, [lengthbox, CONST_0], resultbox))
                 return True
             if l2box.value == 1:
-                l1box = v1.getstrlen(None)
+                l1box = v1.getstrlen(None, mode)
                 if isinstance(l1box, ConstInt) and l1box.value == 1:
                     # comparing two single chars
-                    vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO)
-                    vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO)
+                    vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO, mode)
+                    vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO, mode)
                     seo = self.optimizer.send_extra_operation
                     seo(ResOperation(rop.INT_EQ, [vchar1.force_box(),
                                                   vchar2.force_box()],
                                      resultbox))
                     return True
                 if isinstance(v1, VStringSliceValue):
-                    vchar = self.strgetitem(v2, optimizer.CVAL_ZERO)
+                    vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode)
                     do = EffectInfo.OS_STREQ_SLICE_CHAR
                     self.generate_modified_call(do, [v1.vstr.force_box(),
                                                      v1.vstart.force_box(),
                                                      v1.vlength.force_box(),
                                                      vchar.force_box()],
-                                                resultbox)
+                                                resultbox, mode)
                     return True
         #
         if v2.is_null():
@@ -482,17 +565,18 @@
         #
         return False
 
-    def handle_str_equal_level2(self, v1, v2, resultbox):
-        l2box = v2.getstrlen(None)
+    def handle_str_equal_level2(self, v1, v2, resultbox, mode):
+        l2box = v2.getstrlen(None, mode)
         if isinstance(l2box, ConstInt):
             if l2box.value == 1:
-                vchar = self.strgetitem(v2, optimizer.CVAL_ZERO)
+                vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode)
                 if v1.is_nonnull():
                     do = EffectInfo.OS_STREQ_NONNULL_CHAR
                 else:
                     do = EffectInfo.OS_STREQ_CHECKNULL_CHAR
                 self.generate_modified_call(do, [v1.force_box(),
-                                                 vchar.force_box()], resultbox)
+                                                 vchar.force_box()], resultbox,
+                                            mode)
                 return True
         #
         if v1.is_virtual() and isinstance(v1, VStringSliceValue):
@@ -503,11 +587,12 @@
             self.generate_modified_call(do, [v1.vstr.force_box(),
                                              v1.vstart.force_box(),
                                              v1.vlength.force_box(),
-                                             v2.force_box()], resultbox)
+                                             v2.force_box()], resultbox, mode)
             return True
         return False
 
-    def generate_modified_call(self, oopspecindex, args, result):
+    def generate_modified_call(self, oopspecindex, args, result, mode):
+        oopspecindex += mode.OS_offset
         calldescr, func = callinfo_for_oopspec(oopspecindex)
         op = ResOperation(rop.CALL, [ConstInt(func)] + args, result,
                           descr=calldescr)
@@ -525,7 +610,7 @@
 optimize_ops = _findall(OptString, 'optimize_')
 
 def _findall_call_oopspec():
-    prefix = 'opt_call_oopspec_'
+    prefix = 'opt_call_stroruni_'
     result = []
     for name in dir(OptString):
         if name.startswith(prefix):

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/optimizeopt/virtualize.py	Fri Oct 22 23:09:43 2010
@@ -74,6 +74,8 @@
         assert self.source_op is not None
         # ^^^ This case should not occur any more (see test_bug_3).
         #
+        if not we_are_translated():
+            self.source_op.name = 'FORCE ' + self.source_op.name
         newoperations = self.optimizer.newoperations
         newoperations.append(self.source_op)
         self.box = box = self.source_op.result
@@ -134,6 +136,11 @@
         fielddescrs = self._get_field_descr_list()
         return modifier.make_virtual(self.known_class, fielddescrs)
 
+    def __repr__(self):
+        cls_name = self.known_class.value.adr.ptr._obj._TYPE._name
+        field_names = [field.name for field in self._fields]
+        return "" % (cls_name, field_names)
+
 class VStructValue(AbstractVirtualStructValue):
 
     def __init__(self, optimizer, structdescr, keybox, source_op=None):
@@ -165,6 +172,8 @@
 
     def _really_force(self):
         assert self.source_op is not None
+        if not we_are_translated():
+            self.source_op.name = 'FORCE ' + self.source_op.name
         newoperations = self.optimizer.newoperations
         newoperations.append(self.source_op)
         self.box = box = self.source_op.result

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/pyjitpl.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/pyjitpl.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/pyjitpl.py	Fri Oct 22 23:09:43 2010
@@ -1,4 +1,4 @@
-import py, os
+import py, os, sys
 from pypy.rpython.lltypesystem import lltype, llmemory, rclass
 from pypy.rlib.objectmodel import we_are_translated
 from pypy.rlib.unroll import unrolling_iterable
@@ -498,6 +498,22 @@
     opimpl_getfield_gc_r_pure = _opimpl_getfield_gc_pure_any
     opimpl_getfield_gc_f_pure = _opimpl_getfield_gc_pure_any
 
+    @arguments("orgpc", "box", "descr")
+    def _opimpl_getfield_gc_greenfield_any(self, pc, box, fielddescr):
+        ginfo = self.metainterp.jitdriver_sd.greenfield_info
+        if (ginfo is not None and fielddescr in ginfo.green_field_descrs
+            and not self._nonstandard_virtualizable(pc, box)):
+            # fetch the result, but consider it as a Const box and don't
+            # record any operation
+            resbox = executor.execute(self.metainterp.cpu, self.metainterp,
+                                      rop.GETFIELD_GC_PURE, fielddescr, box)
+            return resbox.constbox()
+        # fall-back
+        return self.execute_with_descr(rop.GETFIELD_GC_PURE, fielddescr, box)
+    opimpl_getfield_gc_i_greenfield = _opimpl_getfield_gc_greenfield_any
+    opimpl_getfield_gc_r_greenfield = _opimpl_getfield_gc_greenfield_any
+    opimpl_getfield_gc_f_greenfield = _opimpl_getfield_gc_greenfield_any
+
     @arguments("box", "descr", "box")
     def _opimpl_setfield_gc_any(self, box, fielddescr, valuebox):
         self.execute_with_descr(rop.SETFIELD_GC, fielddescr, box, valuebox)
@@ -529,7 +545,8 @@
     def _nonstandard_virtualizable(self, pc, box):
         # returns True if 'box' is actually not the "standard" virtualizable
         # that is stored in metainterp.virtualizable_boxes[-1]
-        if self.metainterp.jitdriver_sd.virtualizable_info is None:
+        if (self.metainterp.jitdriver_sd.virtualizable_info is None and
+            self.metainterp.jitdriver_sd.greenfield_info is None):
             return True      # can occur in case of multiple JITs
         standard_box = self.metainterp.virtualizable_boxes[-1]
         if standard_box is box:
@@ -799,12 +816,16 @@
 
     @arguments("orgpc", "int", "boxes3", "boxes3")
     def opimpl_jit_merge_point(self, orgpc, jdindex, greenboxes, redboxes):
+        any_operation = len(self.metainterp.history.operations) > 0
         jitdriver_sd = self.metainterp.staticdata.jitdrivers_sd[jdindex]
         self.verify_green_args(jitdriver_sd, greenboxes)
         # xxx we may disable the following line in some context later
         self.debug_merge_point(jitdriver_sd, greenboxes)
         if self.metainterp.seen_loop_header_for_jdindex < 0:
-            return
+            if not jitdriver_sd.no_loop_header or not any_operation:
+                return
+            # automatically add a loop_header if there is none
+            self.metainterp.seen_loop_header_for_jdindex = jdindex
         #
         assert self.metainterp.seen_loop_header_for_jdindex == jdindex, (
             "found a loop_header for a JitDriver that does not match "
@@ -893,6 +914,40 @@
         msg = box.getref(lltype.Ptr(rstr.STR))
         lloperation.llop.debug_fatalerror(msg)
 
+    @arguments("box", "box", "box", "box", "box")
+    def opimpl_jit_debug(self, stringbox, arg1box, arg2box, arg3box, arg4box):
+        from pypy.rpython.lltypesystem import rstr
+        from pypy.rpython.annlowlevel import hlstr
+        msg = stringbox.getref(lltype.Ptr(rstr.STR))
+        debug_print('jit_debug:', hlstr(msg),
+                    arg1box.getint(), arg2box.getint(),
+                    arg3box.getint(), arg4box.getint())
+        args = [stringbox, arg1box, arg2box, arg3box, arg4box]
+        i = 4
+        while i > 0 and args[i].getint() == -sys.maxint-1:
+            i -= 1
+        assert i >= 0
+        op = self.metainterp.history.record(rop.JIT_DEBUG, args[:i+1], None)
+        self.metainterp.attach_debug_info(op)
+
+    @arguments("box")
+    def _opimpl_assert_green(self, box):
+        if not isinstance(box, Const):
+            msg = "assert_green failed at %s:%d" % (
+                self.jitcode.name,
+                self.pc)
+            if we_are_translated():
+                from pypy.rpython.annlowlevel import llstr
+                from pypy.rpython.lltypesystem import lloperation
+                lloperation.llop.debug_fatalerror(lltype.Void, llstr(msg))
+            else:
+                from pypy.rlib.jit import AssertGreenFailed
+                raise AssertGreenFailed(msg)
+
+    opimpl_int_assert_green   = _opimpl_assert_green
+    opimpl_ref_assert_green   = _opimpl_assert_green
+    opimpl_float_assert_green = _opimpl_assert_green
+
     @arguments("box")
     def opimpl_virtual_ref(self, box):
         # Details on the content of metainterp.virtualref_boxes:
@@ -998,7 +1053,8 @@
         guard_op = metainterp.history.record(opnum, moreargs, None,
                                              descr=resumedescr)
         virtualizable_boxes = None
-        if metainterp.jitdriver_sd.virtualizable_info is not None:
+        if (metainterp.jitdriver_sd.virtualizable_info is not None or
+            metainterp.jitdriver_sd.greenfield_info is not None):
             virtualizable_boxes = metainterp.virtualizable_boxes
         saved_pc = self.pc
         if resumepc >= 0:
@@ -1646,6 +1702,7 @@
                                               duplicates)
             live_arg_boxes += self.virtualizable_boxes
             live_arg_boxes.pop()
+        #
         assert len(self.virtualref_boxes) == 0, "missing virtual_ref_finish()?"
         # Called whenever we reach the 'loop_header' hint.
         # First, attempt to make a bridge:
@@ -1832,6 +1889,7 @@
         f.setup_call(original_boxes)
         assert self.in_recursion == 0
         self.virtualref_boxes = []
+        self.initialize_withgreenfields(original_boxes)
         self.initialize_virtualizable(original_boxes)
 
     def initialize_state_from_guard_failure(self, resumedescr):
@@ -1856,6 +1914,14 @@
             self.virtualizable_boxes.append(virtualizable_box)
             self.initialize_virtualizable_enter()
 
+    def initialize_withgreenfields(self, original_boxes):
+        ginfo = self.jitdriver_sd.greenfield_info
+        if ginfo is not None:
+            assert self.jitdriver_sd.virtualizable_info is None
+            index = (self.jitdriver_sd.num_green_args +
+                     ginfo.red_index)
+            self.virtualizable_boxes = [original_boxes[index]]
+
     def initialize_virtualizable_enter(self):
         vinfo = self.jitdriver_sd.virtualizable_info
         virtualizable_box = self.virtualizable_boxes[-1]
@@ -1949,8 +2015,10 @@
 
     def rebuild_state_after_failure(self, resumedescr):
         vinfo = self.jitdriver_sd.virtualizable_info
+        ginfo = self.jitdriver_sd.greenfield_info
         self.framestack = []
-        boxlists = resume.rebuild_from_resumedata(self, resumedescr, vinfo)
+        boxlists = resume.rebuild_from_resumedata(self, resumedescr, vinfo,
+                                                  ginfo)
         inputargs_and_holes, virtualizable_boxes, virtualref_boxes = boxlists
         #
         # virtual refs: make the vrefs point to the freshly allocated virtuals
@@ -1975,6 +2043,12 @@
             assert not virtualizable.vable_token
             # fill the virtualizable with the local boxes
             self.synchronize_virtualizable()
+        #
+        elif self.jitdriver_sd.greenfield_info:
+            self.virtualizable_boxes = virtualizable_boxes
+        else:
+            assert not virtualizable_boxes
+        #
         return inputargs_and_holes
 
     def check_synchronized_virtualizable(self):
@@ -2048,7 +2122,8 @@
         for i in range(len(boxes)):
             if boxes[i] is oldbox:
                 boxes[i] = newbox
-        if self.jitdriver_sd.virtualizable_info is not None:
+        if (self.jitdriver_sd.virtualizable_info is not None or
+            self.jitdriver_sd.greenfield_info is not None):
             boxes = self.virtualizable_boxes
             for i in range(len(boxes)):
                 if boxes[i] is oldbox:

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/resoperation.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/resoperation.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/resoperation.py	Fri Oct 22 23:09:43 2010
@@ -93,7 +93,7 @@
     def __repr__(self):
         return self.repr()
 
-    def repr(self):
+    def repr(self, graytext=False):
         # RPython-friendly version
         if self.result is not None:
             sres = '%s = ' % (self.result,)
@@ -101,6 +101,8 @@
             sres = ''
         if self.name:
             prefix = "%s:%s   " % (self.name, self.pc)
+            if graytext:
+                prefix = "\f%s\f" % prefix
         else:
             prefix = ""
         args = self.getarglist()
@@ -457,13 +459,14 @@
     #'RUNTIMENEW/1',     # ootype operation    
     'COND_CALL_GC_WB/2d', # [objptr, newvalue]   (for the write barrier)
     'DEBUG_MERGE_POINT/1',      # debugging only
+    'JIT_DEBUG/*',              # debugging only
     'VIRTUAL_REF_FINISH/2',   # removed before it's passed to the backend
     'COPYSTRCONTENT/5',       # src, dst, srcstart, dststart, length
     'COPYUNICODECONTENT/5',
 
     '_CANRAISE_FIRST', # ----- start of can_raise operations -----
     'CALL/*d',
-    'CALL_ASSEMBLER/*d',
+    'CALL_ASSEMBLER/*d',  # call already compiled assembler
     'CALL_MAY_FORCE/*d',
     'CALL_LOOPINVARIANT/*d',
     #'OOSEND',                     # ootype operation

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/resume.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/resume.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/resume.py	Fri Oct 22 23:09:43 2010
@@ -255,13 +255,19 @@
     def make_varray(self, arraydescr):
         return VArrayInfo(arraydescr)
 
-    def make_vstrplain(self):
+    def make_vstrplain(self, is_unicode=False):
+        if is_unicode:
+            return VUniPlainInfo()
         return VStrPlainInfo()
 
-    def make_vstrconcat(self):
+    def make_vstrconcat(self, is_unicode=False):
+        if is_unicode:
+            return VUniConcatInfo()
         return VStrConcatInfo()
 
-    def make_vstrslice(self):
+    def make_vstrslice(self, is_unicode=False):
+        if is_unicode:
+            return VUniSliceInfo()
         return VStrSliceInfo()
 
     def register_virtual_fields(self, virtualbox, fieldboxes):
@@ -550,6 +556,60 @@
         for i in self.fieldnums:
             debug_print("\t\t", str(untag(i)))
 
+
+class VUniPlainInfo(AbstractVirtualInfo):
+    """Stands for the unicode string made out of the characters of all
+    fieldnums."""
+
+    @specialize.argtype(1)
+    def allocate(self, decoder, index):
+        length = len(self.fieldnums)
+        string = decoder.allocate_unicode(length)
+        decoder.virtuals_cache[index] = string
+        for i in range(length):
+            decoder.unicode_setitem(string, i, self.fieldnums[i])
+        return string
+
+    def debug_prints(self):
+        debug_print("\tvuniplaininfo length", len(self.fieldnums))
+
+
+class VUniConcatInfo(AbstractVirtualInfo):
+    """Stands for the unicode string made out of the concatenation of two
+    other unicode strings."""
+
+    @specialize.argtype(1)
+    def allocate(self, decoder, index):
+        # xxx for blackhole resuming, this will build all intermediate
+        # strings and throw them away immediately, which is a bit sub-
+        # efficient.  Not sure we care.
+        left, right = self.fieldnums
+        string = decoder.concat_unicodes(left, right)
+        decoder.virtuals_cache[index] = string
+        return string
+
+    def debug_prints(self):
+        debug_print("\tvuniconcatinfo")
+        for i in self.fieldnums:
+            debug_print("\t\t", str(untag(i)))
+
+
+class VUniSliceInfo(AbstractVirtualInfo):
+    """Stands for the unicode string made out of slicing another
+    unicode string."""
+
+    @specialize.argtype(1)
+    def allocate(self, decoder, index):
+        largerstr, start, length = self.fieldnums
+        string = decoder.slice_unicode(largerstr, start, length)
+        decoder.virtuals_cache[index] = string
+        return string
+
+    def debug_prints(self):
+        debug_print("\tvunisliceinfo")
+        for i in self.fieldnums:
+            debug_print("\t\t", str(untag(i)))
+
 # ____________________________________________________________
 
 class AbstractResumeDataReader(object):
@@ -629,9 +689,11 @@
 
 # ---------- when resuming for pyjitpl.py, make boxes ----------
 
-def rebuild_from_resumedata(metainterp, storage, virtualizable_info):
+def rebuild_from_resumedata(metainterp, storage, virtualizable_info,
+                            greenfield_info):
     resumereader = ResumeDataBoxReader(storage, metainterp)
-    boxes = resumereader.consume_vref_and_vable_boxes(virtualizable_info)
+    boxes = resumereader.consume_vref_and_vable_boxes(virtualizable_info,
+                                                      greenfield_info)
     virtualizable_boxes, virtualref_boxes = boxes
     frameinfo = storage.rd_frame_info_list
     while True:
@@ -676,15 +738,18 @@
         assert (end & 1) == 0
         return [self.decode_ref(nums[i]) for i in range(end)]
 
-    def consume_vref_and_vable_boxes(self, vinfo):
+    def consume_vref_and_vable_boxes(self, vinfo, ginfo):
         nums = self.cur_numb.nums
         self.cur_numb = self.cur_numb.prev
-        if vinfo is None:
-            virtualizable_boxes = None
-            end = len(nums)
-        else:
+        if vinfo is not None:
             virtualizable_boxes = self.consume_virtualizable_boxes(vinfo, nums)
             end = len(nums) - len(virtualizable_boxes)
+        elif ginfo is not None:
+            virtualizable_boxes = [self.decode_ref(nums[-1])]
+            end = len(nums) - 1
+        else:
+            virtualizable_boxes = None
+            end = len(nums)
         virtualref_boxes = self.consume_virtualref_boxes(nums, end)
         return virtualizable_boxes, virtualref_boxes
 
@@ -725,6 +790,32 @@
         return self.metainterp.execute_and_record_varargs(
             rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr)
 
+    def allocate_unicode(self, length):
+        return self.metainterp.execute_and_record(rop.NEWUNICODE,
+                                                  None, ConstInt(length))
+
+    def unicode_setitem(self, strbox, index, charnum):
+        charbox = self.decode_box(charnum, INT)
+        self.metainterp.execute_and_record(rop.UNICODESETITEM, None,
+                                           strbox, ConstInt(index), charbox)
+
+    def concat_unicodes(self, str1num, str2num):
+        calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_CONCAT)
+        str1box = self.decode_box(str1num, REF)
+        str2box = self.decode_box(str2num, REF)
+        return self.metainterp.execute_and_record_varargs(
+            rop.CALL, [ConstInt(func), str1box, str2box], calldescr)
+
+    def slice_unicode(self, strnum, startnum, lengthnum):
+        calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_SLICE)
+        strbox = self.decode_box(strnum, REF)
+        startbox = self.decode_box(startnum, INT)
+        lengthbox = self.decode_box(lengthnum, INT)
+        stopbox = self.metainterp.execute_and_record(rop.INT_ADD, None,
+                                                     startbox, lengthbox)
+        return self.metainterp.execute_and_record_varargs(
+            rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr)
+
     def setfield(self, descr, structbox, fieldnum):
         if descr.is_pointer_field():
             kind = REF
@@ -815,8 +906,9 @@
     resumereader = ResumeDataDirectReader(blackholeinterpbuilder.cpu, storage,
                                           all_virtuals)
     vinfo = jitdriver_sd.virtualizable_info
+    ginfo = jitdriver_sd.greenfield_info
     vrefinfo = blackholeinterpbuilder.metainterp_sd.virtualref_info
-    resumereader.consume_vref_and_vable(vrefinfo, vinfo)
+    resumereader.consume_vref_and_vable(vrefinfo, vinfo, ginfo)
     #
     # First get a chain of blackhole interpreters whose length is given
     # by the depth of rd_frame_info_list.  The first one we get must be
@@ -846,11 +938,11 @@
     resumereader.done()
     return firstbh
 
-def force_from_resumedata(metainterp_sd, storage, vinfo=None):
+def force_from_resumedata(metainterp_sd, storage, vinfo, ginfo):
     resumereader = ResumeDataDirectReader(metainterp_sd.cpu, storage)
     resumereader.handling_async_forcing()
     vrefinfo = metainterp_sd.virtualref_info
-    resumereader.consume_vref_and_vable(vrefinfo, vinfo)
+    resumereader.consume_vref_and_vable(vrefinfo, vinfo, ginfo)
     return resumereader.force_all_virtuals()
 
 class ResumeDataDirectReader(AbstractResumeDataReader):
@@ -925,11 +1017,12 @@
         return specialize_value(TYPE, x)
     load_value_of_type._annspecialcase_ = 'specialize:arg(1)'
 
-    def consume_vref_and_vable(self, vrefinfo, vinfo):
+    def consume_vref_and_vable(self, vrefinfo, vinfo, ginfo):
         nums = self.cur_numb.nums
         self.cur_numb = self.cur_numb.prev
         if self.resume_after_guard_not_forced != 2:
             end_vref = self.consume_vable_info(vinfo, nums)
+            if ginfo is not None: end_vref -= 1
             self.consume_virtualref_info(vrefinfo, nums, end_vref)
 
     def allocate_with_vtable(self, known_class):
@@ -967,6 +1060,31 @@
         result = funcptr(str, start, start + length)
         return lltype.cast_opaque_ptr(llmemory.GCREF, result)
 
+    def allocate_unicode(self, length):
+        return self.cpu.bh_newunicode(length)
+
+    def unicode_setitem(self, str, index, charnum):
+        char = self.decode_int(charnum)
+        self.cpu.bh_unicodesetitem(str, index, char)
+
+    def concat_unicodes(self, str1num, str2num):
+        str1 = self.decode_ref(str1num)
+        str2 = self.decode_ref(str2num)
+        str1 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str1)
+        str2 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str2)
+        funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_CONCAT)
+        result = funcptr(str1, str2)
+        return lltype.cast_opaque_ptr(llmemory.GCREF, result)
+
+    def slice_unicode(self, strnum, startnum, lengthnum):
+        str = self.decode_ref(strnum)
+        start = self.decode_int(startnum)
+        length = self.decode_int(lengthnum)
+        str = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str)
+        funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_SLICE)
+        result = funcptr(str, start, start + length)
+        return lltype.cast_opaque_ptr(llmemory.GCREF, result)
+
     def setfield(self, descr, struct, fieldnum):
         if descr.is_pointer_field():
             newvalue = self.decode_ref(fieldnum)

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/simple_optimize.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/simple_optimize.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/simple_optimize.py	Fri Oct 22 23:09:43 2010
@@ -9,11 +9,13 @@
 
 def transform(op):
     from pypy.jit.metainterp.history import AbstractDescr
-    # Rename CALL_PURE to CALL.
+    # Rename CALL_PURE and CALL_LOOPINVARIANT to CALL.
     # Simplify the VIRTUAL_REF_* so that they don't show up in the backend.
     if op.getopnum() == rop.CALL_PURE:
         op = ResOperation(rop.CALL, op.getarglist()[1:], op.result,
                           op.getdescr())
+    elif op.getopnum() == rop.CALL_LOOPINVARIANT:
+        op = op.copy_and_change(rop.CALL)
     elif op.getopnum() == rop.VIRTUAL_REF:
         op = ResOperation(rop.SAME_AS, [op.getarg(0)], op.result)
     elif op.getopnum() == rop.VIRTUAL_REF_FINISH:

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_basic.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_basic.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_basic.py	Fri Oct 22 23:09:43 2010
@@ -2,6 +2,7 @@
 import sys
 from pypy.rlib.jit import JitDriver, we_are_jitted, hint, dont_look_inside
 from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_SIMPLE, loop_invariant
+from pypy.rlib.jit import jit_debug, assert_green, AssertGreenFailed
 from pypy.jit.metainterp.warmspot import ll_meta_interp, get_stats
 from pypy.jit.backend.llgraph import runner
 from pypy.jit.metainterp import pyjitpl, history
@@ -44,6 +45,7 @@
         num_green_args = 0
         portal_graph = graphs[0]
         virtualizable_info = None
+        greenfield_info = None
         result_type = result_kind
         portal_runner_ptr = "???"
 
@@ -1644,6 +1646,33 @@
         res = self.interp_operations(f, [10, 3.5])
         assert res == 3.5
 
+    def test_jit_debug(self):
+        myjitdriver = JitDriver(greens = [], reds = ['x'])
+        class A:
+            pass
+        def f(x):
+            while x > 0:
+                myjitdriver.can_enter_jit(x=x)
+                myjitdriver.jit_merge_point(x=x)
+                jit_debug("hi there:", x)
+                jit_debug("foobar")
+                x -= 1
+            return x
+        res = self.meta_interp(f, [8])
+        assert res == 0
+        self.check_loops(jit_debug=2)
+
+    def test_assert_green(self):
+        def f(x, promote):
+            if promote:
+                x = hint(x, promote=True)
+            assert_green(x)
+            return x
+        res = self.interp_operations(f, [8, 1])
+        assert res == 8
+        py.test.raises(AssertGreenFailed, self.interp_operations, f, [8, 0])
+
+
 class TestOOtype(BasicTests, OOJitMixin):
 
     def test_oohash(self):
@@ -1751,7 +1780,7 @@
             c = bool(p1)
             d = not bool(p2)
             return 1000*a + 100*b + 10*c + d
-        prebuilt = [lltype.malloc(TP, flavor='raw')] * 2
+        prebuilt = [lltype.malloc(TP, flavor='raw', immortal=True)] * 2
         expected = f(0, 1)
         assert self.interp_operations(f, [0, 1]) == expected
 

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_compile.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_compile.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_compile.py	Fri Oct 22 23:09:43 2010
@@ -6,8 +6,8 @@
 from pypy.jit.metainterp.compile import ResumeGuardCountersInt
 from pypy.jit.metainterp.compile import compile_tmp_callback
 from pypy.jit.metainterp import optimize, jitprof, typesystem, compile
-from pypy.jit.metainterp.test.oparser import parse
 from pypy.jit.metainterp.test.test_optimizefindnode import LLtypeMixin
+from pypy.jit.tool.oparser import parse
 
 
 def test_insert_loop_token():

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_jitdriver.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_jitdriver.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_jitdriver.py	Fri Oct 22 23:09:43 2010
@@ -72,6 +72,33 @@
         # we expect no loop at all for 'loop1': it should always be inlined
         self.check_tree_loop_count(2)    # 1 x loop, 1 x enter bridge
 
+    def test_inactive_jitdriver(self):
+        myjitdriver1 = JitDriver(greens=[], reds=['n', 'm'],
+                                 get_printable_location = getloc1)
+        myjitdriver2 = JitDriver(greens=['g'], reds=['r'],
+                                 get_printable_location = getloc2)
+        #
+        myjitdriver1.active = False    # <===
+        #
+        def loop1(n, m):
+            while n > 0:
+                myjitdriver1.can_enter_jit(n=n, m=m)
+                myjitdriver1.jit_merge_point(n=n, m=m)
+                n -= m
+            return n
+        #
+        def loop2(g, r):
+            while r > 0:
+                myjitdriver2.can_enter_jit(g=g, r=r)
+                myjitdriver2.jit_merge_point(g=g, r=r)
+                r += loop1(r, g) + (-1)
+            return r
+        #
+        res = self.meta_interp(loop2, [4, 40], repeat=7, inline=True)
+        assert res == loop2(4, 40)
+        # we expect no int_sub, but a residual call
+        self.check_loops(int_sub=0, call=1)
+
 
 class TestLLtype(MultipleJitDriversTests, LLJitMixin):
     pass

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_logger.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_logger.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_logger.py	Fri Oct 22 23:09:43 2010
@@ -1,6 +1,6 @@
 import sys
 from pypy.rlib import debug
-from pypy.jit.metainterp.test.oparser import pure_parse
+from pypy.jit.tool.oparser import pure_parse
 from pypy.jit.metainterp import logger
 from pypy.jit.metainterp.typesystem import llhelper
 from StringIO import StringIO

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_oparser.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_oparser.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_oparser.py	Fri Oct 22 23:09:43 2010
@@ -1,7 +1,7 @@
 
 from pypy.rpython.lltypesystem import lltype, llmemory
 
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.jit.metainterp.resoperation import rop
 from pypy.jit.metainterp.history import AbstractDescr, BoxInt, LoopToken,\
      BoxFloat

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_optimizefindnode.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_optimizefindnode.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_optimizefindnode.py	Fri Oct 22 23:09:43 2010
@@ -18,7 +18,7 @@
 from pypy.jit.metainterp.specnode import ConstantSpecNode
 from pypy.jit.codewriter.effectinfo import EffectInfo
 from pypy.jit.codewriter.heaptracker import register_known_gctype, adr2int
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 
 def test_sort_descrs():
     class PseudoDescr(AbstractDescr):
@@ -117,33 +117,32 @@
                             EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE))
     arraycopydescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
                  EffectInfo([], [], [], oopspecindex=EffectInfo.OS_ARRAYCOPY))
-    strconcatdescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_CONCAT))
-    slicedescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_SLICE))
-    strequaldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_EQUAL))
-    streq_slice_checknull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_CHECKNULL))
-    streq_slice_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_NONNULL))
-    streq_slice_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_CHAR))
-    streq_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_NONNULL))
-    streq_nonnull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_NONNULL_CHAR))
-    streq_checknull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_CHECKNULL_CHAR))
-    streq_lengthok_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_LENGTHOK))
+
+    for _name, _os in [
+        ('strconcatdescr',               'OS_STR_CONCAT'),
+        ('strslicedescr',                'OS_STR_SLICE'),
+        ('strequaldescr',                'OS_STR_EQUAL'),
+        ('streq_slice_checknull_descr',  'OS_STREQ_SLICE_CHECKNULL'),
+        ('streq_slice_nonnull_descr',    'OS_STREQ_SLICE_NONNULL'),
+        ('streq_slice_char_descr',       'OS_STREQ_SLICE_CHAR'),
+        ('streq_nonnull_descr',          'OS_STREQ_NONNULL'),
+        ('streq_nonnull_char_descr',     'OS_STREQ_NONNULL_CHAR'),
+        ('streq_checknull_char_descr',   'OS_STREQ_CHECKNULL_CHAR'),
+        ('streq_lengthok_descr',         'OS_STREQ_LENGTHOK'),
+        ]:
+        _oopspecindex = getattr(EffectInfo, _os)
+        locals()[_name] = \
+            cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=_oopspecindex))
+        #
+        _oopspecindex = getattr(EffectInfo, _os.replace('STR', 'UNI'))
+        locals()[_name.replace('str', 'unicode')] = \
+            cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=_oopspecindex))
+
+    s2u_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR2UNICODE))
+    #
 
     class LoopToken(AbstractDescr):
         pass

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_optimizeopt.py	Fri Oct 22 23:09:43 2010
@@ -12,7 +12,7 @@
 from pypy.jit.metainterp.jitprof import EmptyProfiler
 from pypy.jit.metainterp import executor, compile, resume, history
 from pypy.jit.metainterp.resoperation import rop, opname, ResOperation
-from pypy.jit.metainterp.test.oparser import pure_parse
+from pypy.jit.tool.oparser import pure_parse
 
 ##class FakeFrame(object):
 ##    parent_resumedata_snapshot = None
@@ -132,14 +132,21 @@
 # ____________________________________________________________
 
 def equaloplists(oplist1, oplist2, strict_fail_args=True, remap={}):
-    print '-'*20, 'Comparing lists', '-'*20
+    # try to use the full width of the terminal to display the list
+    # unfortunately, does not work with the default capture method of py.test
+    # (which is fd), you you need to use either -s or --capture=sys, else you
+    # get the standard 80 columns width
+    totwidth = py.io.get_terminal_width()
+    width = totwidth / 2 - 1
+    print ' Comparing lists '.center(totwidth, '-')
+    print '%s| %s' % ('optimized'.center(width), 'expected'.center(width))
     for op1, op2 in zip(oplist1, oplist2):
         txt1 = str(op1)
         txt2 = str(op2)
         while txt1 or txt2:
-            print '%-39s| %s' % (txt1[:39], txt2[:39])
-            txt1 = txt1[39:]
-            txt2 = txt2[39:]
+            print '%s| %s' % (txt1[:width].ljust(width), txt2[:width])
+            txt1 = txt1[width:]
+            txt2 = txt2[width:]
         assert op1.getopnum() == op2.getopnum()
         assert op1.numargs() == op2.numargs()
         for i in range(op1.numargs()):
@@ -262,6 +269,10 @@
         expected = self.parse(optops)
         print '\n'.join([str(o) for o in loop.operations])
         self.assert_equal(loop, expected)
+        return loop
+
+
+class OptimizeOptTest(BaseTestOptimizeOpt):
 
     def test_simple(self):
         ops = """
@@ -2643,7 +2654,7 @@
             ''', rop.GUARD_TRUE)
 
 
-class TestLLtype(BaseTestOptimizeOpt, LLtypeMixin):
+class TestLLtype(OptimizeOptTest, LLtypeMixin):
 
     def test_residual_call_does_not_invalidate_caches(self):
         ops = """
@@ -3487,7 +3498,7 @@
         i0 = strlen(p0)
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not', expected)
 
     def test_addsub_const(self):
         ops = """
@@ -3893,6 +3904,15 @@
         """
         self.optimize_loop(ops, 'Not, Not', expected)
 
+    # ----------
+    def optimize_strunicode_loop(self, ops, spectext, optops):
+        # check with the arguments passed in
+        self.optimize_loop(ops, spectext, optops)
+        # check with replacing 'str' with 'unicode' everywhere
+        self.optimize_loop(ops.replace('str','unicode').replace('s"', 'u"'),
+                           spectext,
+                           optops.replace('str','unicode').replace('s"', 'u"'))
+
     def test_newstr_1(self):
         ops = """
         [i0]
@@ -3905,7 +3925,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not', expected)
 
     def test_newstr_2(self):
         ops = """
@@ -3921,7 +3941,7 @@
         [i0, i1]
         jump(i1, i0)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_concat_1(self):
         ops = """
@@ -3942,7 +3962,7 @@
         copystrcontent(p2, p3, 0, i4, i5)
         jump(p2, p3)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_concat_vstr2_str(self):
         ops = """
@@ -3965,7 +3985,7 @@
         copystrcontent(p2, p3, 0, 2, i4)
         jump(i1, i0, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_concat_str_vstr2(self):
         ops = """
@@ -3989,7 +4009,7 @@
         i6 = int_add(i5, 1)      # will be killed by the backend
         jump(i1, i0, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_concat_str_str_str(self):
         ops = """
@@ -4016,12 +4036,12 @@
         copystrcontent(p3, p5, 0, i12b, i3b)
         jump(p2, p3, p5)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_concat_str_cstr1(self):
         ops = """
         [p2]
-        p3 = call(0, p2, "x", descr=strconcatdescr)
+        p3 = call(0, p2, s"x", descr=strconcatdescr)
         jump(p3)
         """
         expected = """
@@ -4035,28 +4055,28 @@
         i5 = int_add(i4, 1)      # will be killed by the backend
         jump(p3)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not', expected)
 
     def test_str_concat_consts(self):
         ops = """
         []
-        p1 = same_as("ab")
-        p2 = same_as("cde")
+        p1 = same_as(s"ab")
+        p2 = same_as(s"cde")
         p3 = call(0, p1, p2, descr=strconcatdescr)
         escape(p3)
         jump()
         """
         expected = """
         []
-        escape("abcde")
+        escape(s"abcde")
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_strunicode_loop(ops, '', expected)
 
     def test_str_slice_1(self):
         ops = """
         [p1, i1, i2]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
         jump(p2, i1, i2)
         """
         expected = """
@@ -4066,12 +4086,12 @@
         copystrcontent(p1, p2, i1, 0, i3)
         jump(p2, i1, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_slice_2(self):
         ops = """
         [p1, i2]
-        p2 = call(0, p1, 0, i2, descr=slicedescr)
+        p2 = call(0, p1, 0, i2, descr=strslicedescr)
         jump(p2, i2)
         """
         expected = """
@@ -4080,13 +4100,13 @@
         copystrcontent(p1, p2, 0, 0, i2)
         jump(p2, i2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_slice_3(self):
         ops = """
         [p1, i1, i2, i3, i4]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
-        p3 = call(0, p2, i3, i4, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
+        p3 = call(0, p2, i3, i4, descr=strslicedescr)
         jump(p3, i1, i2, i3, i4)
         """
         expected = """
@@ -4098,12 +4118,12 @@
         copystrcontent(p1, p3, i6, 0, i5)
         jump(p3, i1, i2, i3, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not, Not', expected)
 
     def test_str_slice_getitem1(self):
         ops = """
         [p1, i1, i2, i3]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
         i4 = strgetitem(p2, i3)
         escape(i4)
         jump(p1, i1, i2, i3)
@@ -4116,7 +4136,7 @@
         escape(i4)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected)
 
     def test_str_slice_plain(self):
         ops = """
@@ -4124,7 +4144,7 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i3)
         strsetitem(p1, 1, i4)
-        p2 = call(0, p1, 1, 2, descr=slicedescr)
+        p2 = call(0, p1, 1, 2, descr=strslicedescr)
         i5 = strgetitem(p2, 0)
         escape(i5)
         jump(i3, i4)
@@ -4134,12 +4154,12 @@
         escape(i4)
         jump(i3, i4)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_slice_concat(self):
         ops = """
         [p1, i1, i2, p2]
-        p3 = call(0, p1, i1, i2, descr=slicedescr)
+        p3 = call(0, p1, i1, i2, descr=strslicedescr)
         p4 = call(0, p3, p2, descr=strconcatdescr)
         jump(p4, i1, i2, p2)
         """
@@ -4155,10 +4175,10 @@
         copystrcontent(p2, p4, 0, i3, i4b)
         jump(p4, i1, i2, p2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected)
 
     # ----------
-    def optimize_loop_extradescrs(self, ops, spectext, optops):
+    def optimize_strunicode_loop_extradescrs(self, ops, spectext, optops):
         from pypy.jit.metainterp.optimizeopt import string
         def my_callinfo_for_oopspec(oopspecindex):
             calldescrtype = type(LLtypeMixin.strequaldescr)
@@ -4173,7 +4193,7 @@
         saved = string.callinfo_for_oopspec
         try:
             string.callinfo_for_oopspec = my_callinfo_for_oopspec
-            self.optimize_loop(ops, spectext, optops)
+            self.optimize_strunicode_loop(ops, spectext, optops)
         finally:
             string.callinfo_for_oopspec = saved
 
@@ -4184,7 +4204,7 @@
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', ops)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', ops)
 
     def test_str_equal_noop2(self):
         ops = """
@@ -4209,12 +4229,13 @@
         escape(i0)
         jump(p1, p2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice1(self):
         ops = """
         [p1, i1, i2, p3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p4, p3, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4226,12 +4247,13 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice2(self):
         ops = """
         [p1, i1, i2, p3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p3, p4, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4243,13 +4265,14 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice3(self):
         ops = """
         [p1, i1, i2, p3]
         guard_nonnull(p3) []
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p3, p4, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4262,13 +4285,14 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice4(self):
         ops = """
         [p1, i1, i2]
-        p3 = call(0, p1, i1, i2, descr=slicedescr)
-        i0 = call(0, p3, "x", descr=strequaldescr)
+        p3 = call(0, p1, i1, i2, descr=strslicedescr)
+        i0 = call(0, p3, s"x", descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2)
         """
@@ -4279,12 +4303,13 @@
         escape(i0)
         jump(p1, i1, i2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice5(self):
         ops = """
         [p1, i1, i2, i3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         p5 = newstr(1)
         strsetitem(p5, 0, i3)
         i0 = call(0, p5, p4, descr=strequaldescr)
@@ -4298,7 +4323,8 @@
         escape(i0)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_none1(self):
         ops = """
@@ -4313,7 +4339,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_none2(self):
         ops = """
@@ -4328,30 +4354,30 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull1(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "hello world", descr=strequaldescr)
+        i0 = call(0, p1, s"hello world", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
         expected = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "hello world", descr=streq_nonnull_descr)
+        i0 = call(0, p1, s"hello world", descr=streq_nonnull_descr)
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull2(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "", descr=strequaldescr)
+        i0 = call(0, p1, s"", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4363,13 +4389,13 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull3(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "x", descr=strequaldescr)
+        i0 = call(0, p1, s"x", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4380,13 +4406,13 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull4(self):
         ops = """
         [p1, p2]
         p4 = call(0, p1, p2, descr=strconcatdescr)
-        i0 = call(0, "hello world", p4, descr=strequaldescr)
+        i0 = call(0, s"hello world", p4, descr=strequaldescr)
         escape(i0)
         jump(p1, p2)
         """
@@ -4401,17 +4427,17 @@
         i5 = strlen(p2)
         i6 = int_add(i4, i5)      # will be killed by the backend
         copystrcontent(p2, p4, 0, i4, i5)
-        i0 = call(0, "hello world", p4, descr=streq_nonnull_descr)
+        i0 = call(0, s"hello world", p4, descr=streq_nonnull_descr)
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected)
 
     def test_str_equal_chars0(self):
         ops = """
         [i1]
         p1 = newstr(0)
-        i0 = call(0, p1, "", descr=strequaldescr)
+        i0 = call(0, p1, s"", descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4420,14 +4446,14 @@
         escape(1)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_chars1(self):
         ops = """
         [i1]
         p1 = newstr(1)
         strsetitem(p1, 0, i1)
-        i0 = call(0, p1, "x", descr=strequaldescr)
+        i0 = call(0, p1, s"x", descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4437,7 +4463,7 @@
         escape(i0)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_chars2(self):
         ops = """
@@ -4445,7 +4471,7 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i1)
         strsetitem(p1, 1, i2)
-        i0 = call(0, p1, "xy", descr=strequaldescr)
+        i0 = call(0, p1, s"xy", descr=strequaldescr)
         escape(i0)
         jump(i1, i2)
         """
@@ -4454,16 +4480,16 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i1)
         strsetitem(p1, 1, i2)
-        i0 = call(0, p1, "xy", descr=streq_lengthok_descr)
+        i0 = call(0, p1, s"xy", descr=streq_lengthok_descr)
         escape(i0)
         jump(i1, i2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected)
 
     def test_str_equal_chars3(self):
         ops = """
         [p1]
-        i0 = call(0, "x", p1, descr=strequaldescr)
+        i0 = call(0, s"x", p1, descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4473,14 +4499,14 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_lengthmismatch1(self):
         ops = """
         [i1]
         p1 = newstr(1)
         strsetitem(p1, 0, i1)
-        i0 = call(0, "xy", p1, descr=strequaldescr)
+        i0 = call(0, s"xy", p1, descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4489,13 +4515,36 @@
         escape(0)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
-    # XXX unicode operations
-    # XXX str2unicode
+    def test_str2unicode_constant(self):
+        ops = """
+        []
+        p0 = call(0, "xy", descr=s2u_descr)      # string -> unicode
+        escape(p0)
+        jump()
+        """
+        expected = """
+        []
+        escape(u"xy")
+        jump()
+        """
+        self.optimize_strunicode_loop_extradescrs(ops, '', expected)
+
+    def test_str2unicode_nonconstant(self):
+        ops = """
+        [p0]
+        p1 = call(0, p0, descr=s2u_descr)      # string -> unicode
+        escape(p1)
+        jump(p1)
+        """
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', ops)
+        # more generally, supporting non-constant but virtual cases is
+        # not obvious, because of the exception UnicodeDecodeError that
+        # can be raised by ll_str2unicode()
 
 
-##class TestOOtype(BaseTestOptimizeOpt, OOtypeMixin):
+##class TestOOtype(OptimizeOptTest, OOtypeMixin):
 
 ##    def test_instanceof(self):
 ##        ops = """

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_resume.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_resume.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_resume.py	Fri Oct 22 23:09:43 2010
@@ -240,6 +240,17 @@
         return FakeBuiltObject(strconcat=[left, right])
     def slice_string(self, str, start, length):
         return FakeBuiltObject(strslice=[str, start, length])
+    def allocate_unicode(self, length):
+        return FakeBuiltObject(unistring=[None]*length)
+    def unicode_setitem(self, unistring, i, fieldnum):
+        value, tag = untag(fieldnum)
+        assert tag == TAGINT
+        assert 0 <= i < len(unistring.unistring)
+        unistring.unistring[i] = value
+    def concat_unicodes(self, left, right):
+        return FakeBuiltObject(uniconcat=[left, right])
+    def slice_unicode(self, str, start, length):
+        return FakeBuiltObject(unislice=[str, start, length])
 
 class FakeBuiltObject(object):
     def __init__(self, **kwds):
@@ -304,6 +315,30 @@
     assert reader.force_all_virtuals() == [
         FakeBuiltObject(strslice=info.fieldnums)]
 
+def test_vuniplaininfo():
+    info = VUniPlainInfo()
+    info.fieldnums = [tag(60, TAGINT)]
+    reader = FakeResumeDataReader()
+    reader._prepare_virtuals([info])
+    assert reader.force_all_virtuals() == [
+        FakeBuiltObject(unistring=[60])]
+
+def test_vuniconcatinfo():
+    info = VUniConcatInfo()
+    info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX)]
+    reader = FakeResumeDataReader()
+    reader._prepare_virtuals([info])
+    assert reader.force_all_virtuals() == [
+        FakeBuiltObject(uniconcat=info.fieldnums)]
+
+def test_vunisliceinfo():
+    info = VUniSliceInfo()
+    info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX), tag(30, TAGBOX)]
+    reader = FakeResumeDataReader()
+    reader._prepare_virtuals([info])
+    assert reader.force_all_virtuals() == [
+        FakeBuiltObject(unislice=info.fieldnums)]
+
 # ____________________________________________________________
 
 

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_string.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_string.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_string.py	Fri Oct 22 23:09:43 2010
@@ -6,14 +6,17 @@
 
 
 class StringTests:
+    _str, _chr = str, chr
+
     def test_eq_residual(self):
+        _str = self._str
         jitdriver = JitDriver(greens = [], reds = ['n', 'i', 's'])
-        global_s = "hello"
+        global_s = _str("hello")
         def f(n, b, s):
             if b:
-                s += "ello"
+                s += _str("ello")
             else:
-                s += "allo"
+                s += _str("allo")
             i = 0
             while n > 0:
                 jitdriver.can_enter_jit(s=s, n=n, i=i)
@@ -21,18 +24,19 @@
                 n -= 1 + (s == global_s)
                 i += 1
             return i
-        res = self.meta_interp(f, [10, True, 'h'], listops=True)
+        res = self.meta_interp(f, [10, True, _str('h')], listops=True)
         assert res == 5
         self.check_loops(**{self.CALL: 1, self.CALL_PURE: 0})
 
     def test_eq_folded(self):
+        _str = self._str
         jitdriver = JitDriver(greens = ['s'], reds = ['n', 'i'])
-        global_s = "hello"
+        global_s = _str("hello")
         def f(n, b, s):
             if b:
-                s += "ello"
+                s += _str("ello")
             else:
-                s += "allo"
+                s += _str("allo")
             i = 0
             while n > 0:
                 jitdriver.can_enter_jit(s=s, n=n, i=i)
@@ -40,31 +44,18 @@
                 n -= 1 + (s == global_s)
                 i += 1
             return i
-        res = self.meta_interp(f, [10, True, 'h'], listops=True)
+        res = self.meta_interp(f, [10, True, _str('h')], listops=True)
         assert res == 5
         self.check_loops(**{self.CALL: 0, self.CALL_PURE: 0})
 
     def test_newstr(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['n', 'm'])
         def f(n, m):
             while True:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                bytecode = 'adlfkj' + chr(n)
-                res = bytecode[n]
-                m -= 1
-                if m < 0:
-                    return ord(res)
-        res = self.meta_interp(f, [6, 10])
-        assert res == 6
-
-    def test_newunicode(self):
-        jitdriver = JitDriver(greens = [], reds = ['n', 'm'])
-        def f(n, m):
-            while True:
-                jitdriver.can_enter_jit(m=m, n=n)
-                jitdriver.jit_merge_point(m=m, n=n)
-                bytecode = u'adlfkj' + unichr(n)
+                bytecode = _str('adlfkj') + _chr(n)
                 res = bytecode[n]
                 m -= 1
                 if m < 0:
@@ -73,95 +64,96 @@
         assert res == 6
 
     def test_char2string_pure(self):
-        for dochr in [chr, ]: #unichr]:
-            jitdriver = JitDriver(greens = [], reds = ['n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            def f(n):
-                while n > 0:
-                    jitdriver.can_enter_jit(n=n)
-                    jitdriver.jit_merge_point(n=n)
-                    s = dochr(n)
-                    if not we_are_jitted():
-                        s += s     # forces to be a string
-                    if n > 100:
-                        escape(s)
-                    n -= 1
-                return 42
-            self.meta_interp(f, [6])
-            self.check_loops(newstr=0, strsetitem=0, strlen=0,
-                             newunicode=0, unicodesetitem=0, unicodelen=0)
+        _str, _chr = self._str, self._chr
+        jitdriver = JitDriver(greens = [], reds = ['n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        def f(n):
+            while n > 0:
+                jitdriver.can_enter_jit(n=n)
+                jitdriver.jit_merge_point(n=n)
+                s = _chr(n)
+                if not we_are_jitted():
+                    s += s     # forces to be a string
+                if n > 100:
+                    escape(s)
+                n -= 1
+            return 42
+        self.meta_interp(f, [6])
+        self.check_loops(newstr=0, strsetitem=0, strlen=0,
+                         newunicode=0, unicodesetitem=0, unicodelen=0)
 
     def test_char2string_escape(self):
-        for dochr in [chr, ]: #unichr]:
-            jitdriver = JitDriver(greens = [], reds = ['n', 'total'])
-            @dont_look_inside
-            def escape(x):
-                return ord(x[0])
-            def f(n):
-                total = 0
-                while n > 0:
-                    jitdriver.can_enter_jit(n=n, total=total)
-                    jitdriver.jit_merge_point(n=n, total=total)
-                    s = dochr(n)
-                    if not we_are_jitted():
-                        s += s    # forces to be a string
-                    total += escape(s)
-                    n -= 1
-                return total
-            res = self.meta_interp(f, [6])
-            assert res == 21
+        _str, _chr = self._str, self._chr
+        jitdriver = JitDriver(greens = [], reds = ['n', 'total'])
+        @dont_look_inside
+        def escape(x):
+            return ord(x[0])
+        def f(n):
+            total = 0
+            while n > 0:
+                jitdriver.can_enter_jit(n=n, total=total)
+                jitdriver.jit_merge_point(n=n, total=total)
+                s = _chr(n)
+                if not we_are_jitted():
+                    s += s    # forces to be a string
+                total += escape(s)
+                n -= 1
+            return total
+        res = self.meta_interp(f, [6])
+        assert res == 21
 
     def test_char2string2char(self):
-        for dochr in [chr, ]: #unichr]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'total'])
-            def f(m):
-                total = 0
-                while m > 0:
-                    jitdriver.can_enter_jit(m=m, total=total)
-                    jitdriver.jit_merge_point(m=m, total=total)
-                    string = dochr(m)
-                    if m > 100:
-                        string += string    # forces to be a string
-                    # read back the character
-                    c = string[0]
-                    total += ord(c)
-                    m -= 1
-                return total
-            res = self.meta_interp(f, [6])
-            assert res == 21
-            self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0,
-                             newunicode=0, unicodegetitem=0, unicodesetitem=0,
-                             unicodelen=0)
+        _str, _chr = self._str, self._chr
+        jitdriver = JitDriver(greens = [], reds = ['m', 'total'])
+        def f(m):
+            total = 0
+            while m > 0:
+                jitdriver.can_enter_jit(m=m, total=total)
+                jitdriver.jit_merge_point(m=m, total=total)
+                string = _chr(m)
+                if m > 100:
+                    string += string    # forces to be a string
+                # read back the character
+                c = string[0]
+                total += ord(c)
+                m -= 1
+            return total
+        res = self.meta_interp(f, [6])
+        assert res == 21
+        self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0,
+                         newunicode=0, unicodegetitem=0, unicodesetitem=0,
+                         unicodelen=0)
 
     def test_strconcat_pure(self):
-        for somestr in ["abc", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            mylist = [somestr+str(i) for i in range(10)]
-            def f(n, m):
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = mylist[n] + mylist[m]
-                    if m > 100:
-                        escape(s)
-                    m -= 1
-                return 42
-            self.meta_interp(f, [6, 7])
-            self.check_loops(newstr=0, strsetitem=0,
-                             newunicode=0, unicodesetitem=0,
-                             call=0, call_pure=0)
+        _str = self._str
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        mylist = [_str("abc") + _str(i) for i in range(10)]
+        def f(n, m):
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = mylist[n] + mylist[m]
+                if m > 100:
+                    escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 7])
+        self.check_loops(newstr=0, strsetitem=0,
+                         newunicode=0, unicodesetitem=0,
+                         call=0, call_pure=0)
 
     def test_strconcat_escape_str_str(self):
+        _str = self._str
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
@@ -171,46 +163,64 @@
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=0, copystrcontent=2,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=0, copystrcontent=2,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=0,
+                             copyunicodecontent=2,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_str_char(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = mylist[n] + chr(m)
+                s = mylist[n] + _chr(m)
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=1,
+                             copyunicodecontent=1,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_char_str(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = chr(n) + mylist[m]
+                s = _chr(n) + mylist[m]
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=1,
+                             copyunicodecontent=1,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_char_char(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
@@ -219,91 +229,132 @@
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = chr(n) + chr(m)
+                s = _chr(n) + _chr(m)
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=2, copystrcontent=0,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=2, copystrcontent=0,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=2,
+                             copyunicodecontent=0,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_str_char_str(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = mylist[n] + chr(n) + mylist[m]
+                s = mylist[n] + _chr(n) + mylist[m]
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=1, copystrcontent=2,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=1, copystrcontent=2,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=1,
+                             copyunicodecontent=2,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_guard_fail(self):
-        for somestr in ["abc", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            mylist = [somestr+str(i) for i in range(12)]
-            def f(n, m):
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = mylist[n] + mylist[m]
-                    if m & 1:
-                        escape(s)
-                    m -= 1
-                return 42
-            self.meta_interp(f, [6, 10])
+        _str = self._str
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        mylist = [_str("abc") + _str(i) for i in range(12)]
+        def f(n, m):
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = mylist[n] + mylist[m]
+                if m & 1:
+                    escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 10])
 
     def test_strslice(self):
-        for somestr in ["abc", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            def f(n, m):
-                assert n >= 0
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = "foobarbazetc"[m:n]
-                    if m <= 5:
-                        escape(s)
-                    m -= 1
-                return 42
-            self.meta_interp(f, [10, 10])
+        _str = self._str
+        longstring = _str("foobarbazetc")
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        def f(n, m):
+            assert n >= 0
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = longstring[m:n]
+                if m <= 5:
+                    escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [10, 10])
 
     def test_streq_char(self):
-        for somestr in ["?abcdefg", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            def f(n, m):
-                assert n >= 0
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = somestr[:m]
-                    escape(s == "?")
-                    m -= 1
-                return 42
-            self.meta_interp(f, [6, 7])
-            self.check_loops(newstr=0, newunicode=0)
-
-
-class TestOOtype(StringTests, OOJitMixin):
-    CALL = "oosend"
-    CALL_PURE = "oosend_pure"
+        _str = self._str
+        longstring = _str("?abcdefg")
+        somechar = _str("?")
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        def f(n, m):
+            assert n >= 0
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = longstring[:m]
+                escape(s == somechar)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 7])
+        self.check_loops(newstr=0, newunicode=0)
+
+
+#class TestOOtype(StringTests, OOJitMixin):
+#    CALL = "oosend"
+#    CALL_PURE = "oosend_pure"
 
 class TestLLtype(StringTests, LLJitMixin):
     CALL = "call"
     CALL_PURE = "call_pure"
+
+class TestLLtypeUnicode(TestLLtype):
+    _str, _chr = unicode, unichr
+
+    def test_str2unicode(self):
+        _str = self._str
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        class Foo:
+            pass
+        @dont_look_inside
+        def escape(x):
+            assert x == _str("6y")
+        def f(n, m):
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                foo = Foo()
+                foo.y = chr(m)
+                foo.y = "y"
+                s = _str(str(n)) + _str(foo.y)
+                escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 7])
+        self.check_loops(call=3,    # str(), _str(), escape()
+                         newunicode=1, unicodegetitem=0,
+                         unicodesetitem=1, copyunicodecontent=1)

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_virtualref.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_virtualref.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_virtualref.py	Fri Oct 22 23:09:43 2010
@@ -93,7 +93,7 @@
         lst = []
         vrefinfo.continue_tracing = lambda vref, virtual: \
                                         lst.append((vref, virtual))
-        resumereader.consume_vref_and_vable(vrefinfo, None)
+        resumereader.consume_vref_and_vable(vrefinfo, None, None)
         del vrefinfo.continue_tracing
         assert len(lst) == 1
         lltype.cast_opaque_ptr(lltype.Ptr(JIT_VIRTUAL_REF),

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/test/test_warmspot.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/test/test_warmspot.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/test/test_warmspot.py	Fri Oct 22 23:09:43 2010
@@ -296,6 +296,69 @@
         assert res == 1
         self.check_loops(int_add=1)   # I get 13 without the loop_header()
 
+    def test_omit_can_enter_jit(self):
+        # Simple test comparing the effects of always giving a can_enter_jit(),
+        # or not giving any.  Mostly equivalent, except that if given, it is
+        # ignored the first time, and so it ends up taking one extra loop to
+        # start JITting.
+        mydriver = JitDriver(greens=[], reds=['m'])
+        #
+        for i2 in range(10):
+            def f2(m):
+                while m > 0:
+                    mydriver.jit_merge_point(m=m)
+                    m -= 1
+            self.meta_interp(f2, [i2])
+            try:
+                self.check_tree_loop_count(1)
+                break
+            except AssertionError:
+                print "f2: no loop generated for i2==%d" % i2
+        else:
+            raise     # re-raise the AssertionError: check_loop_count never 1
+        #
+        for i1 in range(10):
+            def f1(m):
+                while m > 0:
+                    mydriver.can_enter_jit(m=m)
+                    mydriver.jit_merge_point(m=m)
+                    m -= 1
+            self.meta_interp(f1, [i1])
+            try:
+                self.check_tree_loop_count(1)
+                break
+            except AssertionError:
+                print "f1: no loop generated for i1==%d" % i1
+        else:
+            raise     # re-raise the AssertionError: check_loop_count never 1
+        #
+        assert i1 - 1 == i2
+
+    def test_no_loop_at_all(self):
+        mydriver = JitDriver(greens=[], reds=['m'])
+        def f2(m):
+            mydriver.jit_merge_point(m=m)
+            return m - 1
+        def f1(m):
+            while m > 0:
+                m = f2(m)
+        self.meta_interp(f1, [8])
+        # it should generate one "loop" only, which ends in a FINISH
+        # corresponding to the return from f2.
+        self.check_tree_loop_count(1)
+        self.check_loop_count(0)
+
+    def test_simple_loop(self):
+        mydriver = JitDriver(greens=[], reds=['m'])
+        def f1(m):
+            while m > 0:
+                mydriver.jit_merge_point(m=m)
+                m = m - 1
+        self.meta_interp(f1, [8])
+        self.check_loop_count(1)
+        self.check_loops({'int_sub': 1, 'int_gt': 1, 'guard_true': 1,
+                          'jump': 1})
+
 
 class TestLLWarmspot(WarmspotTests, LLJitMixin):
     CPUClass = runner.LLtypeCPU

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/virtualref.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/virtualref.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/virtualref.py	Fri Oct 22 23:09:43 2010
@@ -16,7 +16,8 @@
             ('virtualref_index', lltype.Signed),
             ('forced', rclass.OBJECTPTR))
         self.jit_virtual_ref_vtable = lltype.malloc(rclass.OBJECT_VTABLE,
-                                                    zero=True, flavor='raw')
+                                                    zero=True, flavor='raw',
+                                                    immortal=True)
         self.jit_virtual_ref_vtable.name = rclass.alloc_array_name(
             'jit_virtual_ref')
         # build some constants

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/warmspot.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/warmspot.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/warmspot.py	Fri Oct 22 23:09:43 2010
@@ -110,15 +110,16 @@
             for i in range(len(block.operations)):
                 op = block.operations[i]
                 if (op.opname == 'jit_marker' and
-                    op.args[0].value == marker_name):
+                    op.args[0].value == marker_name and
+                    op.args[1].value.active):   # the jitdriver
                     results.append((graph, block, i))
     return results
 
 def find_can_enter_jit(graphs):
-    results = _find_jit_marker(graphs, 'can_enter_jit')
-    if not results:
-        raise Exception("no can_enter_jit found!")
-    return results
+    return _find_jit_marker(graphs, 'can_enter_jit')
+
+def find_loop_headers(graphs):
+    return _find_jit_marker(graphs, 'loop_header')
 
 def find_jit_merge_points(graphs):
     results = _find_jit_marker(graphs, 'jit_merge_point')
@@ -211,9 +212,9 @@
                 "there are multiple jit_merge_points with the same jitdriver"
 
     def split_graph_and_record_jitdriver(self, graph, block, pos):
-        jd = JitDriverStaticData()
-        jd._jit_merge_point_pos = (graph, block, pos)
         op = block.operations[pos]
+        jd = JitDriverStaticData()
+        jd._jit_merge_point_pos = (graph, op)
         args = op.args[2:]
         s_binding = self.translator.annotator.binding
         jd._portal_args_s = [s_binding(v) for v in args]
@@ -286,10 +287,20 @@
     def make_virtualizable_infos(self):
         vinfos = {}
         for jd in self.jitdrivers_sd:
+            #
+            jd.greenfield_info = None
+            for name in jd.jitdriver.greens:
+                if '.' in name:
+                    from pypy.jit.metainterp.greenfield import GreenFieldInfo
+                    jd.greenfield_info = GreenFieldInfo(self.cpu, jd)
+                    break
+            #
             if not jd.jitdriver.virtualizables:
                 jd.virtualizable_info = None
                 jd.index_of_virtualizable = -1
                 continue
+            else:
+                assert jd.greenfield_info is None, "XXX not supported yet"
             #
             jitdriver = jd.jitdriver
             assert len(jitdriver.virtualizables) == 1    # for now
@@ -457,8 +468,7 @@
             self.make_args_specification(jd)
 
     def make_args_specification(self, jd):
-        graph, block, index = jd._jit_merge_point_pos
-        op = block.operations[index]
+        graph, op = jd._jit_merge_point_pos
         greens_v, reds_v = support.decode_hp_hint_args(op)
         ALLARGS = [v.concretetype for v in (greens_v + reds_v)]
         jd._green_args_spec = [v.concretetype for v in greens_v]
@@ -474,26 +484,37 @@
             [lltype.Signed, llmemory.GCREF], RESTYPE)
 
     def rewrite_can_enter_jits(self):
-        can_enter_jits = find_can_enter_jit(self.translator.graphs)
         sublists = {}
         for jd in self.jitdrivers_sd:
-            sublists[jd.jitdriver] = []
+            sublists[jd.jitdriver] = jd, []
+            jd.no_loop_header = True
+        #
+        loop_headers = find_loop_headers(self.translator.graphs)
+        for graph, block, index in loop_headers:
+            op = block.operations[index]
+            jitdriver = op.args[1].value
+            assert jitdriver in sublists, \
+                   "loop_header with no matching jit_merge_point"
+            jd, sublist = sublists[jitdriver]
+            jd.no_loop_header = False
+        #
+        can_enter_jits = find_can_enter_jit(self.translator.graphs)
         for graph, block, index in can_enter_jits:
             op = block.operations[index]
             jitdriver = op.args[1].value
             assert jitdriver in sublists, \
                    "can_enter_jit with no matching jit_merge_point"
+            jd, sublist = sublists[jitdriver]
             origportalgraph = jd._jit_merge_point_pos[0]
             if graph is not origportalgraph:
-                sublists[jitdriver].append((graph, block, index))
+                sublist.append((graph, block, index))
+                jd.no_loop_header = False
             else:
                 pass   # a 'can_enter_jit' before the 'jit-merge_point', but
                        # originally in the same function: we ignore it here
                        # see e.g. test_jitdriver.test_simple
         for jd in self.jitdrivers_sd:
-            sublist = sublists[jd.jitdriver]
-            assert len(sublist) > 0, \
-                   "found no can_enter_jit for %r" % (jd.jitdriver,)
+            _, sublist = sublists[jd.jitdriver]
             self.rewrite_can_enter_jit(jd, sublist)
 
     def rewrite_can_enter_jit(self, jd, can_enter_jits):
@@ -501,6 +522,19 @@
         FUNCPTR = jd._PTR_JIT_ENTER_FUNCTYPE
         jit_enter_fnptr = self.helper_func(FUNCPTR, jd._maybe_enter_jit_fn)
 
+        if len(can_enter_jits) == 0:
+            # see test_warmspot.test_no_loop_at_all
+            operations = jd.portal_graph.startblock.operations
+            op1 = operations[0]
+            assert (op1.opname == 'jit_marker' and
+                    op1.args[0].value == 'jit_merge_point')
+            op0 = SpaceOperation(
+                'jit_marker',
+                [Constant('can_enter_jit', lltype.Void)] + op1.args[1:],
+                None)
+            operations.insert(0, op0)
+            can_enter_jits = [(jd.portal_graph, jd.portal_graph.startblock, 0)]
+
         for graph, block, index in can_enter_jits:
             if graph is jd._jit_merge_point_pos[0]:
                 continue
@@ -709,8 +743,14 @@
         # ____________________________________________________________
         # Now mutate origportalgraph to end with a call to portal_runner_ptr
         #
-        _, origblock, origindex = jd._jit_merge_point_pos
-        op = origblock.operations[origindex]
+        _, op = jd._jit_merge_point_pos
+        for origblock in origportalgraph.iterblocks():
+            if op in origblock.operations:
+                break
+        else:
+            assert False, "lost the operation %r in the graph %r" % (
+                op, origportalgraph)
+        origindex = origblock.operations.index(op)
         assert op.opname == 'jit_marker'
         assert op.args[0].value == 'jit_merge_point'
         greens_v, reds_v = support.decode_hp_hint_args(op)

Modified: pypy/branch/fast-forward/pypy/jit/metainterp/warmstate.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/metainterp/warmstate.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/metainterp/warmstate.py	Fri Oct 22 23:09:43 2010
@@ -1,5 +1,5 @@
 import sys
-from pypy.rpython.lltypesystem import lltype, llmemory, rstr
+from pypy.rpython.lltypesystem import lltype, llmemory, rstr, rffi
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython.annlowlevel import hlstr, llstr, cast_base_ptr_to_instance
 from pypy.rpython.annlowlevel import cast_object_to_ptr
@@ -24,7 +24,11 @@
     """
     INPUT = lltype.typeOf(x)
     if INPUT is lltype.Signed:
-        return lltype.cast_primitive(TYPE, x)    # XXX missing: Ptr(non-gc)
+        if isinstance(TYPE, lltype.Ptr) and TYPE.TO._gckind == 'raw':
+            # non-gc pointer
+            return rffi.cast(TYPE, x)
+        else:
+            return lltype.cast_primitive(TYPE, x)
     elif INPUT is lltype.Float:
         assert TYPE is lltype.Float
         return x
@@ -172,6 +176,9 @@
             meth(default_value)
 
     def set_param_threshold(self, threshold):
+        if threshold < 0:
+            self.increment_threshold = 0   # never reach the THRESHOLD_LIMIT
+            return
         if threshold < 2:
             threshold = 2
         self.increment_threshold = (self.THRESHOLD_LIMIT // threshold) + 1

Modified: pypy/branch/fast-forward/pypy/jit/tl/pypyjit.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/tl/pypyjit.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/tl/pypyjit.py	Fri Oct 22 23:09:43 2010
@@ -40,6 +40,9 @@
 config.objspace.usemodules.array = True
 config.objspace.usemodules._weakref = False
 config.objspace.usemodules._sre = False
+#
+config.objspace.usemodules._ffi = True
+#
 set_pypy_opt_level(config, level='jit')
 config.objspace.std.withinlineddict = True
 

Modified: pypy/branch/fast-forward/pypy/jit/tl/pypyjit_child.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/tl/pypyjit_child.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/tl/pypyjit_child.py	Fri Oct 22 23:09:43 2010
@@ -16,7 +16,7 @@
     interp.heap.malloc_nonmovable = returns_null     # XXX
 
     from pypy.jit.backend.llgraph.runner import LLtypeCPU
-    LLtypeCPU.supports_floats = False    # for now
+    #LLtypeCPU.supports_floats = False    # for now
     apply_jit(interp, graph, LLtypeCPU)
 
 

Modified: pypy/branch/fast-forward/pypy/jit/tool/loopviewer.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/tool/loopviewer.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/tool/loopviewer.py	Fri Oct 22 23:09:43 2010
@@ -9,7 +9,7 @@
 import optparse
 from pprint import pprint
 from pypy.tool import logparser
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.jit.metainterp.history import ConstInt
 from pypy.rpython.lltypesystem import llmemory, lltype
 

Modified: pypy/branch/fast-forward/pypy/jit/tool/showstats.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/tool/showstats.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/tool/showstats.py	Fri Oct 22 23:09:43 2010
@@ -4,7 +4,7 @@
 import autopath
 import sys, py
 from pypy.tool import logparser
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.jit.metainterp.resoperation import rop
 from pypy.rpython.lltypesystem import lltype, llmemory
 

Modified: pypy/branch/fast-forward/pypy/jit/tool/traceviewer.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/tool/traceviewer.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/tool/traceviewer.py	Fri Oct 22 23:09:43 2010
@@ -253,10 +253,10 @@
 def main(loopfile, use_threshold, view=True):
     countname = py.path.local(loopfile + '.count')
     if countname.check():
-        counts = [re.split('( 20 and use_threshold:
             counts.threshold = l[-20]

Modified: pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py	(original)
+++ pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py	Fri Oct 22 23:09:43 2010
@@ -2,9 +2,11 @@
 from pypy.interpreter.error import OperationError, operationerrfmt
 from pypy.interpreter.gateway import ObjSpace, W_Root, NoneNotWrapped, applevel
 from pypy.interpreter.gateway import interp2app, ObjSpace
-from pypy.interpreter.typedef import TypeDef, make_weakref_descr
+from pypy.interpreter.typedef import TypeDef
 from pypy.interpreter.argument import Arguments
 from pypy.interpreter.baseobjspace import Wrappable
+from pypy.interpreter.typedef import GetSetProperty, descr_get_dict
+from pypy.interpreter.typedef import descr_set_dict
 from pypy.rlib.rarithmetic import r_uint, intmask
 from pypy.rlib.objectmodel import compute_identity_hash
 from pypy.rlib.debug import make_sure_not_resized
@@ -57,6 +59,14 @@
         self.bases_w = bases
         self.w_dict = w_dict
  
+    def instantiate(self, space):
+        cache = space.fromcache(Cache)
+        if self.lookup(space, '__del__') is not None:
+            w_inst = cache.cls_with_del(space, self)
+        else:
+            w_inst = cache.cls_without_del(space, self)
+        return w_inst
+
     def getdict(self):
         return self.w_dict
 
@@ -100,15 +110,15 @@
         return False
 
     @jit.unroll_safe
-    def lookup(self, space, w_attr):
+    def lookup(self, space, attr):
         # returns w_value or interplevel None
-        w_result = space.finditem(self.w_dict, w_attr)
+        w_result = space.finditem_str(self.w_dict, attr)
         if w_result is not None:
             return w_result
         for base in self.bases_w:
             # XXX fix annotation of bases_w to be a list of W_ClassObjects
             assert isinstance(base, W_ClassObject)
-            w_result = base.lookup(space, w_attr)
+            w_result = base.lookup(space, attr)
             if w_result is not None:
                 return w_result
         return None
@@ -122,7 +132,7 @@
                 return space.wrap(self.name)
             elif name == "__bases__":
                 return space.newtuple(self.bases_w)
-        w_value = self.lookup(space, w_attr)
+        w_value = self.lookup(space, name)
         if w_value is None:
             raise operationerrfmt(
                 space.w_AttributeError,
@@ -147,7 +157,7 @@
                 self.setbases(space, w_value)
                 return
             elif name == "__del__":
-                if self.lookup(space, w_attr) is None:
+                if self.lookup(space, name) is None:
                     msg = ("a __del__ method added to an existing class "
                            "will not be called")
                     space.warn(msg, space.w_RuntimeWarning)
@@ -195,13 +205,20 @@
         # NOT_RPYTHON
         return '' % self.name
 
+class Cache:
+    def __init__(self, space):
+        from pypy.interpreter.typedef import _usersubclswithfeature
+        # evil
+        self.cls_without_del = _usersubclswithfeature(
+                space.config, W_InstanceObject, "dict", "weakref")
+        self.cls_with_del = _usersubclswithfeature(
+                space.config, self.cls_without_del, "del")
+
+
 def class_descr_call(space, w_self, __args__):
     self = space.interp_w(W_ClassObject, w_self)
-    if self.lookup(space, space.wrap('__del__')) is not None:
-        w_inst = W_InstanceObjectWithDel(space, self)
-    else:
-        w_inst = W_InstanceObject(space, self)
-    w_init = w_inst.getattr_from_class(space, space.wrap('__init__'))
+    w_inst = self.instantiate(space)
+    w_init = w_inst.getattr_from_class(space, '__init__')
     if w_init is not None:
         w_result = space.call_args(w_init, __args__)
         if not space.is_w(w_result, space.w_None):
@@ -235,7 +252,7 @@
 
 def make_unary_instance_method(name):
     def unaryop(self, space):
-        w_meth = self.getattr(space, space.wrap(name), True)
+        w_meth = self.getattr(space, name, True)
         return space.call_function(w_meth)
     unaryop.func_name = name
     return unaryop
@@ -243,7 +260,7 @@
 def make_binary_returning_notimplemented_instance_method(name):
     def binaryop(self, space, w_other):
         try:
-            w_meth = self.getattr(space, space.wrap(name), False)
+            w_meth = self.getattr(space, name, False)
         except OperationError, e:
             if e.match(space, space.w_AttributeError):
                 return space.w_NotImplemented
@@ -268,7 +285,7 @@
             w_a = self
             w_b = w_other
         if w_a is self:
-            w_meth = self.getattr(space, space.wrap(specialname), False)
+            w_meth = self.getattr(space, specialname, False)
             if w_meth is None:
                 return space.w_NotImplemented
             return space.call_function(w_meth, w_b)
@@ -279,7 +296,7 @@
     def rbinaryop(self, space, w_other):
         w_a, w_b = _coerce_helper(space, self, w_other)
         if w_a is None or w_a is self:
-            w_meth = self.getattr(space, space.wrap(rspecialname), False)
+            w_meth = self.getattr(space, rspecialname, False)
             if w_meth is None:
                 return space.w_NotImplemented
             return space.call_function(w_meth, w_other)
@@ -303,46 +320,34 @@
         raise OperationError(
             space.w_TypeError,
             space.wrap("instance() first arg must be class"))
-    if space.is_w(w_dict, space.w_None):
-        w_dict = None
-    elif not space.is_true(space.isinstance(w_dict, space.w_dict)):
-        raise OperationError(
-            space.w_TypeError,
-            space.wrap("instance() second arg must be dictionary or None"))
-    return W_InstanceObject(space, w_class, w_dict)
+    w_result = w_class.instantiate(space)
+    if not space.is_w(w_dict, space.w_None):
+        w_result.setdict(space, w_dict)
+    return w_result
 
 class W_InstanceObject(Wrappable):
-    def __init__(self, space, w_class, w_dict=None):
-        if w_dict is None:
-            w_dict = space.newdict(instance=True)
+    def __init__(self, space, w_class):
+        # note that user_setup is overridden by the typedef.py machinery
+        self.user_setup(space, space.gettypeobject(self.typedef))
         assert isinstance(w_class, W_ClassObject)
         self.w_class = w_class
-        self.w_dict = w_dict
-        self.space = space
-
-    def getdict(self):
-        return self.w_dict
 
-    def setdict(self, space, w_dict):
-        if (w_dict is None or
-            not space.is_true(space.isinstance(w_dict, space.w_dict))):
-            raise OperationError(
-                space.w_TypeError,
-                space.wrap("__dict__ must be a dictionary object"))
-        self.w_dict = w_dict
+    def user_setup(self, space, w_subtype):
+        self.space = space
 
-    def setclass(self, space, w_class):
+    def set_oldstyle_class(self, space, w_class):
         if w_class is None or not isinstance(w_class, W_ClassObject):
             raise OperationError(
                 space.w_TypeError,
                 space.wrap("__class__ must be set to a class"))
         self.w_class = w_class
 
-    def getattr_from_class(self, space, w_name):
+    def getattr_from_class(self, space, name):
         # Look up w_name in the class dict, and call its __get__.
         # This method ignores the instance dict and the __getattr__.
         # Returns None if not found.
-        w_value = self.w_class.lookup(space, w_name)
+        assert isinstance(name, str)
+        w_value = self.w_class.lookup(space, name)
         if w_value is None:
             return None
         w_descr_get = space.lookup(w_value, '__get__')
@@ -350,19 +355,20 @@
             return w_value
         return space.call_function(w_descr_get, w_value, self, self.w_class)
 
-    def getattr(self, space, w_name, exc=True):
+    def getattr(self, space, name, exc=True):
         # Normal getattr rules: look up w_name in the instance dict,
         # in the class dict, and then via a call to __getatttr__.
-        w_result = space.finditem(self.w_dict, w_name)
+        assert isinstance(name, str)
+        w_result = self.getdictvalue(space, name)
         if w_result is not None:
             return w_result
-        w_result = self.getattr_from_class(space, w_name)
+        w_result = self.getattr_from_class(space, name)
         if w_result is not None:
             return w_result
-        w_meth = self.getattr_from_class(space, space.wrap('__getattr__'))
+        w_meth = self.getattr_from_class(space, '__getattr__')
         if w_meth is not None:
             try:
-                return space.call_function(w_meth, w_name)
+                return space.call_function(w_meth, space.wrap(name))
             except OperationError, e:
                 if not exc and e.match(space, space.w_AttributeError):
                     return None     # eat the AttributeError
@@ -372,7 +378,7 @@
             raise operationerrfmt(
                 space.w_AttributeError,
                 "%s instance has no attribute '%s'",
-                self.w_class.name, space.str_w(w_name))
+                self.w_class.name, name)
         else:
             return None
 
@@ -380,44 +386,46 @@
         name = space.str_w(w_attr)
         if len(name) >= 8 and name[0] == '_':
             if name == "__dict__":
-                return self.w_dict
+                return self.getdict()
             elif name == "__class__":
                 return self.w_class
-        return self.getattr(space, w_attr)
+        return self.getattr(space, name)
 
     def descr_setattr(self, space, w_name, w_value):
         name = unwrap_attr(space, w_name)
-        w_meth = self.getattr_from_class(space, space.wrap('__setattr__'))
+        w_meth = self.getattr_from_class(space, '__setattr__')
         if name and name[0] == "_":
             if name == '__dict__':
                 self.setdict(space, w_value)
                 return
             if name == '__class__':
-                self.setclass(space, w_value)
+                self.set_oldstyle_class(space, w_value)
                 return
             if name == '__del__' and w_meth is None:
-                if (not isinstance(self, W_InstanceObjectWithDel)
-                    and space.finditem(self.w_dict, w_name) is None):
+                cache = space.fromcache(Cache)
+                if (not isinstance(self, cache.cls_with_del)
+                    and self.getdictvalue(space, '__del__') is None):
                     msg = ("a __del__ method added to an instance "
                            "with no __del__ in the class will not be called")
                     space.warn(msg, space.w_RuntimeWarning)
         if w_meth is not None:
             space.call_function(w_meth, w_name, w_value)
         else:
-            self.setdictvalue(space, name, w_value)
+            # bit obscure: appease normalization
+            self.setdictvalue(space, name, w_value, True)
 
     def descr_delattr(self, space, w_name):
         name = unwrap_attr(space, w_name)
         if name and name[0] == "_":
             if name == '__dict__':
                 # use setdict to raise the error
-                self.setdict(space, None)
+                self.setdict(space, space.w_None)
                 return
             elif name == '__class__':
-                # use setclass to raise the error
-                self.setclass(space, None)
+                # use set_oldstyle_class to raise the error
+                self.set_oldstyle_class(space, None)
                 return
-        w_meth = self.getattr_from_class(space, space.wrap('__delattr__'))
+        w_meth = self.getattr_from_class(space, '__delattr__')
         if w_meth is not None:
             space.call_function(w_meth, w_name)
         else:
@@ -428,7 +436,7 @@
                     self.w_class.name, name)
 
     def descr_repr(self, space):
-        w_meth = self.getattr(space, space.wrap('__repr__'), False)
+        w_meth = self.getattr(space, '__repr__', False)
         if w_meth is None:
             w_class = self.w_class
             mod = w_class.get_module_string(space)
@@ -436,19 +444,19 @@
         return space.call_function(w_meth)
 
     def descr_str(self, space):
-        w_meth = self.getattr(space, space.wrap('__str__'), False)
+        w_meth = self.getattr(space, '__str__', False)
         if w_meth is None:
             return self.descr_repr(space)
         return space.call_function(w_meth)
 
     def descr_unicode(self, space):
-        w_meth = self.getattr(space, space.wrap('__unicode__'), False)
+        w_meth = self.getattr(space, '__unicode__', False)
         if w_meth is None:
             return self.descr_str(space)
         return space.call_function(w_meth)
 
     def descr_len(self, space):
-        w_meth = self.getattr(space, space.wrap('__len__'))
+        w_meth = self.getattr(space, '__len__')
         w_result = space.call_function(w_meth)
         if space.is_true(space.isinstance(w_result, space.w_int)):
             if space.is_true(space.lt(w_result, space.wrap(0))):
@@ -461,22 +469,22 @@
             space.wrap("__len__() should return an int"))
 
     def descr_getitem(self, space, w_key):
-        w_meth = self.getattr(space, space.wrap('__getitem__'))
+        w_meth = self.getattr(space, '__getitem__')
         return space.call_function(w_meth, w_key)
 
     def descr_setitem(self, space, w_key, w_value):
-        w_meth = self.getattr(space, space.wrap('__setitem__'))
+        w_meth = self.getattr(space, '__setitem__')
         space.call_function(w_meth, w_key, w_value)
 
     def descr_delitem(self, space, w_key):
-        w_meth = self.getattr(space, space.wrap('__delitem__'))
+        w_meth = self.getattr(space, '__delitem__')
         space.call_function(w_meth, w_key)
 
     def descr_iter(self, space):
-        w_meth = self.getattr(space, space.wrap('__iter__'), False)
+        w_meth = self.getattr(space, '__iter__', False)
         if w_meth is not None:
             return space.call_function(w_meth)
-        w_meth = self.getattr(space, space.wrap('__getitem__'), False)
+        w_meth = self.getattr(space, '__getitem__', False)
         if w_meth is None:
             raise OperationError(
                 space.w_TypeError,
@@ -486,14 +494,14 @@
     # don't see the point
 
     def descr_getslice(self, space, w_i, w_j):
-        w_meth = self.getattr(space, space.wrap('__getslice__'), False)
+        w_meth = self.getattr(space, '__getslice__', False)
         if w_meth is not None:
             return space.call_function(w_meth, w_i, w_j)
         else:
             return space.getitem(self, space.newslice(w_i, w_j, space.w_None))
 
     def descr_setslice(self, space, w_i, w_j, w_sequence):
-        w_meth = self.getattr(space, space.wrap('__setslice__'), False)
+        w_meth = self.getattr(space, '__setslice__', False)
         if w_meth is not None:
             space.call_function(w_meth, w_i, w_j, w_sequence)
         else:
@@ -501,20 +509,20 @@
                           w_sequence)
 
     def descr_delslice(self, space, w_i, w_j):
-        w_meth = self.getattr(space, space.wrap('__delslice__'), False)
+        w_meth = self.getattr(space, '__delslice__', False)
         if w_meth is not None:
             space.call_function(w_meth, w_i, w_j)
         else:
             return space.delitem(self, space.newslice(w_i, w_j, space.w_None))
 
     def descr_call(self, space, __args__):
-        w_meth = self.getattr(space, space.wrap('__call__'))
+        w_meth = self.getattr(space, '__call__')
         return space.call_args(w_meth, __args__)
 
     def descr_nonzero(self, space):
-        w_func = self.getattr(space, space.wrap('__nonzero__'), False)
+        w_func = self.getattr(space, '__nonzero__', False)
         if w_func is None:
-            w_func = self.getattr(space, space.wrap('__len__'), False)
+            w_func = self.getattr(space, '__len__', False)
             if w_func is None:
                 return space.w_True
         w_result = space.call_function(w_func)
@@ -538,7 +546,7 @@
                 not isinstance(w_b, W_InstanceObject)):
                 return space.cmp(w_a, w_b)
         if isinstance(w_a, W_InstanceObject):
-            w_func = w_a.getattr(space, space.wrap('__cmp__'), False)
+            w_func = w_a.getattr(space, '__cmp__', False)
             if w_func is not None:
                 w_res = space.call_function(w_func, w_b)
                 if space.is_w(w_res, space.w_NotImplemented):
@@ -557,7 +565,7 @@
                     return space.wrap(-1)
                 return space.wrap(0)
         if isinstance(w_b, W_InstanceObject):
-            w_func = w_b.getattr(space, space.wrap('__cmp__'), False)
+            w_func = w_b.getattr(space, '__cmp__', False)
             if w_func is not None:
                 w_res = space.call_function(w_func, w_a)
                 if space.is_w(w_res, space.w_NotImplemented):
@@ -578,10 +586,10 @@
         return space.w_NotImplemented
 
     def descr_hash(self, space):
-        w_func = self.getattr(space, space.wrap('__hash__'), False)
+        w_func = self.getattr(space, '__hash__', False)
         if w_func is None:
-            w_eq =  self.getattr(space, space.wrap('__eq__'), False)
-            w_cmp =  self.getattr(space, space.wrap('__cmp__'), False)
+            w_eq =  self.getattr(space, '__eq__', False)
+            w_cmp =  self.getattr(space, '__cmp__', False)
             if w_eq is not None or w_cmp is not None:
                 raise OperationError(space.w_TypeError,
                                      space.wrap("unhashable instance"))
@@ -617,7 +625,7 @@
         return self.descr_int(space)
 
     def descr_index(self, space):
-        w_func = self.getattr(space, space.wrap('__index__'), False)
+        w_func = self.getattr(space, '__index__', False)
         if w_func is not None:
             return space.call_function(w_func)
         raise OperationError(
@@ -625,7 +633,7 @@
             space.wrap("object cannot be interpreted as an index"))
 
     def descr_contains(self, space, w_obj):
-        w_func = self.getattr(space, space.wrap('__contains__'), False)
+        w_func = self.getattr(space, '__contains__', False)
         if w_func is not None:
             return space.wrap(space.is_true(space.call_function(w_func, w_obj)))
         # now do it ourselves
@@ -648,7 +656,7 @@
                 w_a = self
                 w_b = w_other
             if w_a is self:
-                w_func = self.getattr(space, space.wrap('__pow__'), False)
+                w_func = self.getattr(space, '__pow__', False)
                 if w_func is not None:
                     return space.call_function(w_func, w_other)
                 return space.w_NotImplemented
@@ -656,7 +664,7 @@
                 return space.pow(w_a, w_b, space.w_None)
         else:
             # CPython also doesn't try coercion in this case
-            w_func = self.getattr(space, space.wrap('__pow__'), False)
+            w_func = self.getattr(space, '__pow__', False)
             if w_func is not None:
                 return space.call_function(w_func, w_other, w_modulo)
             return space.w_NotImplemented
@@ -668,7 +676,7 @@
                 w_a = self
                 w_b = w_other
             if w_a is self:
-                w_func = self.getattr(space, space.wrap('__rpow__'), False)
+                w_func = self.getattr(space, '__rpow__', False)
                 if w_func is not None:
                     return space.call_function(w_func, w_other)
                 return space.w_NotImplemented
@@ -676,13 +684,13 @@
                 return space.pow(w_b, w_a, space.w_None)
         else:
             # CPython also doesn't try coercion in this case
-            w_func = self.getattr(space, space.wrap('__rpow__'), False)
+            w_func = self.getattr(space, '__rpow__', False)
             if w_func is not None:
                 return space.call_function(w_func, w_other, w_modulo)
             return space.w_NotImplemented
 
     def descr_next(self, space):
-        w_func = self.getattr(space, space.wrap('next'), False)
+        w_func = self.getattr(space, 'next', False)
         if w_func is None:
             raise OperationError(space.w_TypeError,
                                  space.wrap("instance has no next() method"))
@@ -691,10 +699,9 @@
     def descr_del(self, space):
         # Note that this is called from executioncontext.UserDelAction
         # via the space.userdel() method.
-        w_name = space.wrap('__del__')
-        w_func = space.finditem(self.w_dict, w_name)
+        w_func = self.getdictvalue(space, '__del__')
         if w_func is None:
-            w_func = self.getattr_from_class(space, w_name)
+            w_func = self.getattr_from_class(space, '__del__')
         if w_func is not None:
             space.call_function(w_func)
 
@@ -744,6 +751,14 @@
         rmeth,
         unwrap_spec=["self", ObjSpace, W_Root])
 
+
+def descr_del_dict(space, w_inst):
+    # use setdict to raise the error
+    w_inst.setdict(space, space.w_None)
+
+dict_descr = GetSetProperty(descr_get_dict, descr_set_dict, descr_del_dict)
+dict_descr.name = '__dict__'
+
 W_InstanceObject.typedef = TypeDef("instance",
     __new__ = interp2app(descr_instance_new),
     __getattribute__ = interp2app(W_InstanceObject.descr_getattribute,
@@ -797,14 +812,11 @@
                          unwrap_spec=['self', ObjSpace, W_Root, W_Root]),
     next = interp2app(W_InstanceObject.descr_next,
                       unwrap_spec=['self', ObjSpace]),
-    __weakref__ = make_weakref_descr(W_InstanceObject),
     __del__ = interp2app(W_InstanceObject.descr_del,
                          unwrap_spec=['self', ObjSpace]),
     __exit__ = interp2app(W_InstanceObject.descr_exit,
                           unwrap_spec=['self', ObjSpace, W_Root, W_Root, W_Root]),
+    __dict__ = dict_descr,
     **rawdict
 )
-
-class W_InstanceObjectWithDel(W_InstanceObject):
-    def __del__(self):
-        self._enqueue_for_destruction(self.space)
+W_InstanceObject.typedef.acceptable_as_base_class = False

Modified: pypy/branch/fast-forward/pypy/module/__builtin__/test/test_classobj.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/__builtin__/test/test_classobj.py	(original)
+++ pypy/branch/fast-forward/pypy/module/__builtin__/test/test_classobj.py	Fri Oct 22 23:09:43 2010
@@ -955,6 +955,31 @@
         assert x is b
         assert y == 5
 
+    def test_cant_subclass_instance(self):
+        class A:
+            pass
+        try:
+            class B(type(A())):
+                pass
+        except TypeError:
+            pass
+        else:
+            assert 0, "should have raised"
+
+    def test_dict_descriptor(self):
+        import sys
+        if not hasattr(sys, 'pypy_objspaceclass'):
+            skip("on CPython old-style instances don't have a __dict__ descriptor")
+        class A:
+            pass
+        a = A()
+        a.x = 1
+        descr = type(a).__dict__['__dict__']
+        assert descr.__get__(a) == {'x': 1}
+        descr.__set__(a, {'x': 2})
+        assert a.x == 2
+        raises(TypeError, descr.__delete__, a)
+
 
 class AppTestOldStyleSharing(AppTestOldstyle):
     def setup_class(cls):
@@ -993,3 +1018,22 @@
             a = 1
             b = 2
         assert self.is_strdict(A)
+
+class AppTestOldStyleMapDict(AppTestOldstyle):
+    def setup_class(cls):
+        cls.space = gettestobjspace(**{"objspace.std.withmapdict": True})
+        if option.runappdirect:
+            py.test.skip("can only be run on py.py")
+        def has_mapdict(space, w_inst):
+            return space.wrap(w_inst._get_mapdict_map() is not None)
+        cls.w_has_mapdict = cls.space.wrap(gateway.interp2app(has_mapdict))
+
+
+    def test_has_mapdict(self):
+        class A:
+            def __init__(self):
+                self.x = 42
+        a = A()
+        assert a.x == 42
+        assert self.has_mapdict(a)
+

Modified: pypy/branch/fast-forward/pypy/module/_rawffi/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_rawffi/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_rawffi/__init__.py	Fri Oct 22 23:09:43 2010
@@ -1,5 +1,5 @@
 
-""" Low-level interface to libffi
+""" Low-level interface to clibffi
 """
 
 from pypy.interpreter.mixedmodule import MixedModule
@@ -46,12 +46,12 @@
         if hasattr(interp_rawffi, 'check_HRESULT'):
             Module.interpleveldefs['check_HRESULT'] = 'interp_rawffi.check_HRESULT'
 
-        from pypy.rlib import libffi
+        from pypy.rlib import clibffi
         for name in ['FUNCFLAG_STDCALL', 'FUNCFLAG_CDECL', 'FUNCFLAG_PYTHONAPI',
                      'FUNCFLAG_USE_ERRNO', 'FUNCFLAG_USE_LASTERROR',
                      ]:
-            if hasattr(libffi, name):
-                Module.interpleveldefs[name] = "space.wrap(%r)" % getattr(libffi, name)
+            if hasattr(clibffi, name):
+                Module.interpleveldefs[name] = "space.wrap(%r)" % getattr(clibffi, name)
                 
         super(Module, cls).buildloaders()
     buildloaders = classmethod(buildloaders)

Modified: pypy/branch/fast-forward/pypy/module/_rawffi/array.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_rawffi/array.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_rawffi/array.py	Fri Oct 22 23:09:43 2010
@@ -97,7 +97,15 @@
 
 class W_ArrayInstance(W_DataInstance):
     def __init__(self, space, shape, length, address=r_uint(0)):
-        W_DataInstance.__init__(self, space, shape.size * length, address)
+        # Workaround for a strange behavior of libffi: make sure that
+        # we always have at least 8 bytes.  For W_ArrayInstances that are
+        # used as the result value of a function call, ffi_call() writes
+        # 8 bytes into it even if the function's result type asks for less.
+        # This strange behavior is documented.
+        memsize = shape.size * length
+        if memsize < 8:
+            memsize = 8
+        W_DataInstance.__init__(self, space, memsize, address)
         self.length = length
         self.shape = shape
 

Modified: pypy/branch/fast-forward/pypy/module/_rawffi/callback.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_rawffi/callback.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_rawffi/callback.py	Fri Oct 22 23:09:43 2010
@@ -8,8 +8,8 @@
 from pypy.module._rawffi.array import get_elem, push_elem
 from pypy.module._rawffi.interp_rawffi import W_DataInstance, letter2tp, \
      wrap_value, unwrap_value, unwrap_truncate_int
-from pypy.rlib.libffi import USERDATA_P, CallbackFuncPtr, FUNCFLAG_CDECL
-from pypy.rlib.libffi import ffi_type_void
+from pypy.rlib.clibffi import USERDATA_P, CallbackFuncPtr, FUNCFLAG_CDECL
+from pypy.rlib.clibffi import ffi_type_void
 from pypy.module._rawffi.tracker import tracker
 from pypy.interpreter.error import OperationError
 from pypy.interpreter import gateway

Modified: pypy/branch/fast-forward/pypy/module/_rawffi/interp_rawffi.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_rawffi/interp_rawffi.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_rawffi/interp_rawffi.py	Fri Oct 22 23:09:43 2010
@@ -5,7 +5,7 @@
 from pypy.interpreter.gateway import interp2app, NoneNotWrapped
 from pypy.interpreter.typedef import TypeDef, GetSetProperty
 
-from pypy.rlib.libffi import *
+from pypy.rlib.clibffi import *
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.rlib.unroll import unrolling_iterable
 import pypy.rlib.rposix as rposix

Modified: pypy/branch/fast-forward/pypy/module/_rawffi/structure.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_rawffi/structure.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_rawffi/structure.py	Fri Oct 22 23:09:43 2010
@@ -15,7 +15,7 @@
 from pypy.module._rawffi.interp_rawffi import wrap_value, unwrap_value
 from pypy.module._rawffi.interp_rawffi import unpack_shape_with_length
 from pypy.module._rawffi.interp_rawffi import size_alignment
-from pypy.rlib import libffi
+from pypy.rlib import clibffi
 from pypy.rlib.rarithmetic import intmask, r_uint
 
 def unpack_fields(space, w_fields):
@@ -34,7 +34,7 @@
 def round_up(size, alignment):
     return (size + alignment - 1) & -alignment
 
-def size_alignment_pos(fields):
+def size_alignment_pos(fields, is_union=False):
     size = 0
     alignment = 1
     pos = []
@@ -42,16 +42,20 @@
         # fieldtype is a W_Array
         fieldsize = fieldtype.size
         fieldalignment = fieldtype.alignment
-        size = round_up(size, fieldalignment)
         alignment = max(alignment, fieldalignment)
-        pos.append(size)
-        size += intmask(fieldsize)
+        if is_union:
+            pos.append(0)
+            size = max(size, fieldsize)
+        else:
+            size = round_up(size, fieldalignment)
+            pos.append(size)
+            size += intmask(fieldsize)
     size = round_up(size, alignment)
     return size, alignment, pos
 
 
 class W_Structure(W_DataShape):
-    def __init__(self, space, fields, size, alignment):
+    def __init__(self, space, fields, size, alignment, is_union=False):
         name_to_index = {}
         if fields is not None:
             for i in range(len(fields)):
@@ -60,7 +64,7 @@
                     raise operationerrfmt(space.w_ValueError,
                         "duplicate field name %s", name)
                 name_to_index[name] = i
-            size, alignment, pos = size_alignment_pos(fields)
+            size, alignment, pos = size_alignment_pos(fields, is_union)
         else: # opaque case
             fields = []
             pos = []
@@ -104,14 +108,14 @@
     descr_fieldoffset.unwrap_spec = ['self', ObjSpace, str]
 
     # get the corresponding ffi_type
-    ffi_struct = lltype.nullptr(libffi.FFI_STRUCT_P.TO)
+    ffi_struct = lltype.nullptr(clibffi.FFI_STRUCT_P.TO)
 
     def get_basic_ffi_type(self):
         if not self.ffi_struct:
             # Repeated fields are delicate.  Consider for example
             #     struct { int a[5]; }
             # or  struct { struct {int x;} a[5]; }
-            # Seeing no corresponding doc in libffi, let's just repeat
+            # Seeing no corresponding doc in clibffi, let's just repeat
             # the field 5 times...
             fieldtypes = []
             for name, tp in self.fields:
@@ -122,7 +126,7 @@
                 while count + basic_size <= total_size:
                     fieldtypes.append(basic_ffi_type)
                     count += basic_size
-            self.ffi_struct = libffi.make_struct_ffitype_e(self.size,
+            self.ffi_struct = clibffi.make_struct_ffitype_e(self.size,
                                                            self.alignment,
                                                            fieldtypes)
         return self.ffi_struct.ffistruct
@@ -133,15 +137,17 @@
     
 
 
-def descr_new_structure(space, w_type, w_shapeinfo):
+def descr_new_structure(space, w_type, w_shapeinfo, union=0):
+    is_union = bool(union)
     if space.is_true(space.isinstance(w_shapeinfo, space.w_tuple)):
         w_size, w_alignment = space.fixedview(w_shapeinfo, expected_length=2)
         S = W_Structure(space, None, space.int_w(w_size),
-                                     space.int_w(w_alignment))
+                                     space.int_w(w_alignment), is_union)
     else:
         fields = unpack_fields(space, w_shapeinfo)
-        S = W_Structure(space, fields, 0, 0)
+        S = W_Structure(space, fields, 0, 0, is_union)
     return space.wrap(S)
+descr_new_structure.unwrap_spec = [ObjSpace, W_Root, W_Root, int]
 
 W_Structure.typedef = TypeDef(
     'Structure',

Modified: pypy/branch/fast-forward/pypy/module/_rawffi/test/test__rawffi.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_rawffi/test/test__rawffi.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_rawffi/test/test__rawffi.py	Fri Oct 22 23:09:43 2010
@@ -71,7 +71,7 @@
            return s[num];
         }
 
-        char *char_check(char x, char y)
+        const char *char_check(char x, char y)
         {
            if (y == static_str[0])
               return static_str;
@@ -186,13 +186,14 @@
                      sum_x_y
                      give perturb get_s2a check_s2a
                      AAA_first_ordinal_function
+                     ret_un_func
                   """.split()
         eci = ExternalCompilationInfo(export_symbols=symbols)
         return str(platform.compile([c_file], eci, 'x', standalone=False))
     prepare_c_example = staticmethod(prepare_c_example)
     
     def setup_class(cls):
-        from pypy.rlib.libffi import get_libc_name
+        from pypy.rlib.clibffi import get_libc_name
         space = gettestobjspace(usemodules=('_rawffi', 'struct'))
         cls.space = space
         cls.w_lib_name = space.wrap(cls.prepare_c_example())
@@ -948,14 +949,15 @@
         assert a[4] == 't'
 
     def test_union(self):
-        skip("segfaulting")
         import _rawffi
         longsize = _rawffi.sizeof('l')
-        S = _rawffi.Structure((longsize, longsize))
+        S = _rawffi.Structure([('x', 'h'), ('y', 'l')], union=True)
         s = S(autofree=False)
+        s.x = 12345
         lib = _rawffi.CDLL(self.lib_name)
         f = lib.ptr('ret_un_func', [(S, 1)], (S, 1))
         ret = f(s)
+        assert ret.y == 1234500, "ret.y == %d" % (ret.y,)
         s.free()
 
 class AppTestAutoFree:

Modified: pypy/branch/fast-forward/pypy/module/_socket/test/test_sock_app.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_socket/test/test_sock_app.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_socket/test/test_sock_app.py	Fri Oct 22 23:09:43 2010
@@ -254,6 +254,7 @@
     s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
     fd = s.fileno()
     w_obj = rsocket.make_address(c_addr, addrlen).as_object(fd, space)
+    lltype.free(c_addr_ll, flavor='raw')
     assert space.is_true(space.eq(w_obj, space.newtuple([
         space.wrap('lo'),
         space.wrap(socket.ntohs(8)),

Modified: pypy/branch/fast-forward/pypy/module/_sre/interp_sre.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_sre/interp_sre.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_sre/interp_sre.py	Fri Oct 22 23:09:43 2010
@@ -290,10 +290,8 @@
 def SRE_Pattern__new__(space, w_subtype, w_pattern, flags, w_code,
               groups=0, w_groupindex=None, w_indexgroup=None):
     n = space.int_w(space.len(w_code))
-    code = [0] * n
-    for i in range(n):
-        x = space.uint_w(space.getitem(w_code, space.wrap(i)))
-        code[i] = intmask(x)
+    code = [intmask(space.uint_w(space.getitem(w_code, space.wrap(i))))
+            for i in range(n)]
     #
     w_srepat = space.allocate_instance(W_SRE_Pattern, w_subtype)
     srepat = space.interp_w(W_SRE_Pattern, w_srepat)

Modified: pypy/branch/fast-forward/pypy/module/_weakref/interp__weakref.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_weakref/interp__weakref.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_weakref/interp__weakref.py	Fri Oct 22 23:09:43 2010
@@ -7,7 +7,7 @@
 import weakref
 
 
-class WeakrefLifeline(object):
+class WeakrefLifeline(W_Root):
     def __init__(self, space):
         self.space = space       # this is here for W_Root.clear_all_weakrefs()
         self.refs_weak = []

Modified: pypy/branch/fast-forward/pypy/module/_winreg/interp_winreg.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_winreg/interp_winreg.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_winreg/interp_winreg.py	Fri Oct 22 23:09:43 2010
@@ -213,13 +213,10 @@
         subkey = None
     else:
         subkey = space.str_w(w_subkey)
-    dataptr = rffi.str2charp(value)
-    try:
+    with rffi.scoped_str2charp(value) as dataptr:
         ret = rwinreg.RegSetValue(hkey, subkey, rwinreg.REG_SZ, dataptr, len(value))
-    finally:
-        rffi.free_charp(dataptr)
-    if ret != 0:
-        raiseWindowsError(space, ret, 'RegSetValue')
+        if ret != 0:
+            raiseWindowsError(space, ret, 'RegSetValue')
 SetValue.unwrap_spec = [ObjSpace, W_Root, W_Root, int, str]
 
 def QueryValue(space, w_hkey, w_subkey):
@@ -238,23 +235,15 @@
         subkey = None
     else:
         subkey = space.str_w(w_subkey)
-    bufsize_p = lltype.malloc(rwin32.PLONG.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwin32.PLONG.TO, 1) as bufsize_p:
         ret = rwinreg.RegQueryValue(hkey, subkey, None, bufsize_p)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegQueryValue')
-        buf = lltype.malloc(rffi.CCHARP.TO, bufsize_p[0], flavor='raw')
-        try:
+        with lltype.scoped_alloc(rffi.CCHARP.TO, bufsize_p[0]) as buf:
             ret = rwinreg.RegQueryValue(hkey, subkey, buf, bufsize_p)
             if ret != 0:
                 raiseWindowsError(space, ret, 'RegQueryValue')
             return space.wrap(rffi.charp2strn(buf, bufsize_p[0] - 1))
-        finally:
-            lltype.free(buf, flavor='raw')
-    finally:
-        lltype.free(bufsize_p, flavor='raw')
-    if ret != 0:
-        raiseWindowsError(space, ret, 'RegQueryValue')
 QueryValue.unwrap_spec = [ObjSpace, W_Root, W_Root]
 
 def convert_to_regdata(space, w_value, typ):
@@ -413,16 +402,14 @@
 value_name is a string indicating the value to query"""
     hkey = hkey_w(w_hkey, space)
     null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
-    retDataSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retDataSize:
         ret = rwinreg.RegQueryValueEx(hkey, subkey, null_dword, null_dword,
                                       None, retDataSize)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegQueryValueEx')
-        databuf = lltype.malloc(rffi.CCHARP.TO, retDataSize[0], flavor='raw')
-        try:
-            retType = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-            try:
+
+        with lltype.scoped_alloc(rffi.CCHARP.TO, retDataSize[0]) as databuf:
+            with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retType:
 
                 ret = rwinreg.RegQueryValueEx(hkey, subkey, null_dword,
                                               retType, databuf, retDataSize)
@@ -433,12 +420,6 @@
                                          retDataSize[0], retType[0]),
                     space.wrap(retType[0]),
                     ])
-            finally:
-                lltype.free(retType, flavor='raw')
-        finally:
-            lltype.free(databuf, flavor='raw')
-    finally:
-        lltype.free(retDataSize, flavor='raw')
 
 QueryValueEx.unwrap_spec = [ObjSpace, W_Root, str]
 
@@ -455,14 +436,11 @@
 The return value is the handle of the opened key.
 If the function fails, an exception is raised."""
     hkey = hkey_w(w_hkey, space)
-    rethkey = lltype.malloc(rwinreg.PHKEY.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwinreg.PHKEY.TO, 1) as rethkey:
         ret = rwinreg.RegCreateKey(hkey, subkey, rethkey)
         if ret != 0:
             raiseWindowsError(space, ret, 'CreateKey')
         return space.wrap(W_HKEY(rethkey[0]))
-    finally:
-        lltype.free(rethkey, flavor='raw')
 CreateKey.unwrap_spec = [ObjSpace, W_Root, str]
 
 def DeleteKey(space, w_hkey, subkey):
@@ -505,14 +483,11 @@
 The result is a new handle to the specified key
 If the function fails, an EnvironmentError exception is raised."""
     hkey = hkey_w(w_hkey, space)
-    rethkey = lltype.malloc(rwinreg.PHKEY.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwinreg.PHKEY.TO, 1) as rethkey:
         ret = rwinreg.RegOpenKeyEx(hkey, subkey, res, sam, rethkey)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegOpenKeyEx')
         return space.wrap(W_HKEY(rethkey[0]))
-    finally:
-        lltype.free(rethkey, flavor='raw')
 OpenKey.unwrap_spec = [ObjSpace, W_Root, str, int, rffi.r_uint]
 
 def EnumValue(space, w_hkey, index):
@@ -532,10 +507,8 @@
     hkey = hkey_w(w_hkey, space)
     null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
 
-    retValueSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-    try:
-        retDataSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-        try:
+    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retValueSize:
+        with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retDataSize:
             ret = rwinreg.RegQueryInfoKey(
                 hkey, None, null_dword, null_dword,
                 null_dword, null_dword, null_dword,
@@ -547,14 +520,9 @@
             retValueSize[0] += 1
             retDataSize[0] += 1
 
-            valuebuf = lltype.malloc(rffi.CCHARP.TO, retValueSize[0],
-                                     flavor='raw')
-            try:
-                databuf = lltype.malloc(rffi.CCHARP.TO, retDataSize[0],
-                                        flavor='raw')
-                try:
-                    retType = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-                    try:
+            with lltype.scoped_alloc(rffi.CCHARP.TO, retValueSize[0]) as valuebuf:
+                with lltype.scoped_alloc(rffi.CCHARP.TO, retDataSize[0]) as databuf:
+                    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retType:
                         ret = rwinreg.RegEnumValue(
                             hkey, index, valuebuf, retValueSize,
                             null_dword, retType, databuf, retDataSize)
@@ -567,16 +535,6 @@
                                                  retDataSize[0], retType[0]),
                             space.wrap(retType[0]),
                             ])
-                    finally:
-                        lltype.free(retType, flavor='raw')
-                finally:
-                    lltype.free(databuf, flavor='raw')
-            finally:
-                lltype.free(valuebuf, flavor='raw')
-        finally:
-            lltype.free(retDataSize, flavor='raw')
-    finally:
-        lltype.free(retValueSize, flavor='raw')
 
 EnumValue.unwrap_spec = [ObjSpace, W_Root, int]
 
@@ -593,10 +551,8 @@
     null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
 
     # max key name length is 255
-    buf = lltype.malloc(rffi.CCHARP.TO, 256, flavor='raw')
-    try:
-        retValueSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-        try:
+    with lltype.scoped_alloc(rffi.CCHARP.TO, 256) as buf:
+        with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retValueSize:
             retValueSize[0] = r_uint(256) # includes NULL terminator
             ret = rwinreg.RegEnumKeyEx(hkey, index, buf, retValueSize,
                                        null_dword, None, null_dword,
@@ -604,10 +560,6 @@
             if ret != 0:
                 raiseWindowsError(space, ret, 'RegEnumKeyEx')
             return space.wrap(rffi.charp2str(buf))
-        finally:
-            lltype.free(retValueSize, flavor='raw')
-    finally:
-        lltype.free(buf, flavor='raw')
 
 EnumKey.unwrap_spec = [ObjSpace, W_Root, int]
 
@@ -622,12 +574,9 @@
 A long integer that identifies when the key was last modified (if available)
  as 100's of nanoseconds since Jan 1, 1600."""
     hkey = hkey_w(w_hkey, space)
-    nSubKeys = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-    try:
-        nValues = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-        try:
-            ft = lltype.malloc(rwin32.PFILETIME.TO, 1, flavor='raw')
-            try:
+    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as nSubKeys:
+        with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as nValues:
+            with lltype.scoped_alloc(rwin32.PFILETIME.TO, 1) as ft:
                 null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
                 ret = rwinreg.RegQueryInfoKey(
                     hkey, None, null_dword, null_dword,
@@ -641,12 +590,6 @@
                 return space.newtuple([space.wrap(nSubKeys[0]),
                                        space.wrap(nValues[0]),
                                        space.wrap(l)])
-            finally:
-                lltype.free(ft, flavor='raw')
-        finally:
-            lltype.free(nValues, flavor='raw')
-    finally:
-        lltype.free(nSubKeys, flavor='raw')
 QueryInfoKey.unwrap_spec = [ObjSpace, W_Root]
 
 def str_or_None_w(space, w_obj):
@@ -667,12 +610,9 @@
 If the function fails, an EnvironmentError exception is raised."""
     machine = str_or_None_w(space, w_machine)
     hkey = hkey_w(w_hkey, space)
-    rethkey = lltype.malloc(rwinreg.PHKEY.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwinreg.PHKEY.TO, 1) as rethkey:
         ret = rwinreg.RegConnectRegistry(machine, hkey, rethkey)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegConnectRegistry')
         return space.wrap(W_HKEY(rethkey[0]))
-    finally:
-        lltype.free(rethkey, flavor='raw')
 ConnectRegistry.unwrap_spec = [ObjSpace, W_Root, W_Root]

Modified: pypy/branch/fast-forward/pypy/module/array/interp_array.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/array/interp_array.py	(original)
+++ pypy/branch/fast-forward/pypy/module/array/interp_array.py	Fri Oct 22 23:09:43 2010
@@ -193,32 +193,30 @@
                           mytype.bytes
                     raise OperationError(space.w_OverflowError,
                                          space.wrap(msg))
-            elif mytype.unwrap == 'str_w' or mytype.unwrap == 'unicode_w':
+                return rffi.cast(mytype.itemtype, item)
+            if mytype.unwrap == 'str_w' or mytype.unwrap == 'unicode_w':
                 if len(item) != 1:
                     msg = 'array item must be char'
                     raise OperationError(space.w_TypeError, space.wrap(msg))
                 item = item[0]
-
+                return rffi.cast(mytype.itemtype, item)
+            #
+            # "regular" case: it fits in an rpython integer (lltype.Signed)
+            result = rffi.cast(mytype.itemtype, item)
             if mytype.canoverflow:
-                msg = None
-                if mytype.signed:
-                    if item < -1 << (mytype.bytes * 8 - 1):
+                if rffi.cast(lltype.Signed, result) != item:
+                    # overflow.  build the correct message
+                    if item < 0:
                         msg = ('signed %d-byte integer is less than minimum' %
                                mytype.bytes)
-                    elif item > (1 << (mytype.bytes * 8 - 1)) - 1:
+                    else:
                         msg = ('signed %d-byte integer is greater than maximum'
                                % mytype.bytes)
-                else:
-                    if item < 0:
-                        msg = ('unsigned %d-byte integer is less than minimum'
-                               % mytype.bytes)
-                    elif item > (1 << (mytype.bytes * 8)) - 1:
-                        msg = ('unsigned %d-byte integer is greater'
-                               ' than maximum' % mytype.bytes)
-                if msg is not None:
+                    if not mytype.signed:
+                        msg = 'un' + msg      # 'signed' => 'unsigned'
                     raise OperationError(space.w_OverflowError,
                                          space.wrap(msg))
-            return rffi.cast(mytype.itemtype, item)
+            return result
 
         def __del__(self):
             self.setlen(0)

Modified: pypy/branch/fast-forward/pypy/module/bz2/interp_bz2.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/bz2/interp_bz2.py	(original)
+++ pypy/branch/fast-forward/pypy/module/bz2/interp_bz2.py	Fri Oct 22 23:09:43 2010
@@ -225,6 +225,11 @@
         if self.current_size > 0:
             rffi.keep_buffer_alive_until_here(self.raw_buf, self.gc_buf)
 
+    def __enter__(self):
+        return self
+    def __exit__(self, *args):
+        self.free()
+
 # ____________________________________________________________
 #
 # Make the BZ2File type by internally inheriting from W_File.
@@ -531,33 +536,30 @@
         if not self.running:
             raise OperationError(self.space.w_ValueError,
                 self.space.wrap("this object was already flushed"))
-        
-        out = OutBuffer(self.bzs)
+
         in_bufsize = datasize
-        in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-        for i in range(datasize):
-            in_buf[i] = data[i]
 
-        try:
-        
-            self.bzs.c_next_in = in_buf
-            rffi.setintfield(self.bzs, 'c_avail_in', in_bufsize)
+        with OutBuffer(self.bzs) as out:
+            with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
 
-            while True:
-                bzerror = BZ2_bzCompress(self.bzs, BZ_RUN)
-                if bzerror != BZ_RUN_OK:
-                    _catch_bz2_error(self.space, bzerror)
+                for i in range(datasize):
+                    in_buf[i] = data[i]
 
-                if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
-                    break
-                elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
-                    out.prepare_next_chunk()
+                self.bzs.c_next_in = in_buf
+                rffi.setintfield(self.bzs, 'c_avail_in', in_bufsize)
 
-            res = out.make_result_string()
-            return self.space.wrap(res)
-        finally:
-            lltype.free(in_buf, flavor='raw')
-            out.free()
+                while True:
+                    bzerror = BZ2_bzCompress(self.bzs, BZ_RUN)
+                    if bzerror != BZ_RUN_OK:
+                        _catch_bz2_error(self.space, bzerror)
+
+                    if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
+                        break
+                    elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
+
+                res = out.make_result_string()
+                return self.space.wrap(res)
 
     compress.unwrap_spec = ['self', 'bufferstr']
     
@@ -566,9 +568,8 @@
             raise OperationError(self.space.w_ValueError,
                 self.space.wrap("this object was already flushed"))
         self.running = False
-        
-        out = OutBuffer(self.bzs)
-        try:
+
+        with OutBuffer(self.bzs) as out:
             while True:
                 bzerror = BZ2_bzCompress(self.bzs, BZ_FINISH)
                 if bzerror == BZ_STREAM_END:
@@ -581,8 +582,6 @@
 
             res = out.make_result_string()
             return self.space.wrap(res)
-        finally:
-            out.free()
     flush.unwrap_spec = ['self']
 
 W_BZ2Compressor.typedef = TypeDef("BZ2Compressor",
@@ -641,38 +640,37 @@
         if not self.running:
             raise OperationError(self.space.w_EOFError,
                 self.space.wrap("end of stream was already found"))
-        
+
         in_bufsize = len(data)
-        in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-        for i in range(in_bufsize):
-            in_buf[i] = data[i]
 
-        out = OutBuffer(self.bzs)
-        try:
+        with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
+            for i in range(in_bufsize):
+                in_buf[i] = data[i]
             self.bzs.c_next_in = in_buf
             rffi.setintfield(self.bzs, 'c_avail_in', in_bufsize)
 
-            while True:
-                bzerror = BZ2_bzDecompress(self.bzs)
-                if bzerror == BZ_STREAM_END:
-                    if rffi.getintfield(self.bzs, 'c_avail_in') != 0:
-                        unused = [self.bzs.c_next_in[i] for i in range(rffi.getintfield(self.bzs, 'c_avail_in'))]
-                        self.unused_data = "".join(unused)
-                    self.running = False
-                    break
-                if bzerror != BZ_OK:
-                    _catch_bz2_error(self.space, bzerror)
-
-                if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
-                    break
-                elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
-                    out.prepare_next_chunk()
+            with OutBuffer(self.bzs) as out:
+                while True:
+                    bzerror = BZ2_bzDecompress(self.bzs)
+                    if bzerror == BZ_STREAM_END:
+                        if rffi.getintfield(self.bzs, 'c_avail_in') != 0:
+                            unused = [self.bzs.c_next_in[i]
+                                      for i in range(
+                                          rffi.getintfield(self.bzs,
+                                                           'c_avail_in'))]
+                            self.unused_data = "".join(unused)
+                        self.running = False
+                        break
+                    if bzerror != BZ_OK:
+                        _catch_bz2_error(self.space, bzerror)
+
+                    if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
+                        break
+                    elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
 
-            res = out.make_result_string()
-            return self.space.wrap(res)
-        finally:
-            lltype.free(in_buf, flavor='raw')
-            out.free()
+                res = out.make_result_string()
+                return self.space.wrap(res)
 
     decompress.unwrap_spec = ['self', 'bufferstr']
 
@@ -695,43 +693,39 @@
     if compresslevel < 1 or compresslevel > 9:
         raise OperationError(space.w_ValueError,
             space.wrap("compresslevel must be between 1 and 9"))
-            
-    bzs = lltype.malloc(bz_stream.TO, flavor='raw', zero=True)
-    in_bufsize = len(data)
-    # conforming to bz2 manual, this is large enough to fit compressed
-    # data in one shot. We will check it later anyway.
-    out = OutBuffer(bzs, in_bufsize + (in_bufsize / 100 + 1) + 600)
-
-    in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-    for i in range(in_bufsize):
-        in_buf[i] = data[i]
-
-    try:
-        bzs.c_next_in = in_buf
-        rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
 
-        bzerror = BZ2_bzCompressInit(bzs, compresslevel, 0, 0)
-        if bzerror != BZ_OK:
-            _catch_bz2_error(space, bzerror)
+    with lltype.scoped_alloc(bz_stream.TO, zero=True) as bzs:
+        in_bufsize = len(data)
+
+        with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
+            for i in range(in_bufsize):
+                in_buf[i] = data[i]
+            bzs.c_next_in = in_buf
+            rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
+
+            # conforming to bz2 manual, this is large enough to fit compressed
+            # data in one shot. We will check it later anyway.
+            with OutBuffer(bzs,
+                           in_bufsize + (in_bufsize / 100 + 1) + 600) as out:
+
+                bzerror = BZ2_bzCompressInit(bzs, compresslevel, 0, 0)
+                if bzerror != BZ_OK:
+                    _catch_bz2_error(space, bzerror)
 
-        while True:
-            bzerror = BZ2_bzCompress(bzs, BZ_FINISH)
-            if bzerror == BZ_STREAM_END:
-                break
-            elif bzerror != BZ_FINISH_OK:
+                while True:
+                    bzerror = BZ2_bzCompress(bzs, BZ_FINISH)
+                    if bzerror == BZ_STREAM_END:
+                        break
+                    elif bzerror != BZ_FINISH_OK:
+                        BZ2_bzCompressEnd(bzs)
+                        _catch_bz2_error(space, bzerror)
+
+                    if rffi.getintfield(bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
+
+                res = out.make_result_string()
                 BZ2_bzCompressEnd(bzs)
-                _catch_bz2_error(space, bzerror)
-            
-            if rffi.getintfield(bzs, 'c_avail_out') == 0:
-                out.prepare_next_chunk()
-
-        res = out.make_result_string()
-        BZ2_bzCompressEnd(bzs)
-        return space.wrap(res)
-    finally:
-        lltype.free(bzs, flavor='raw')
-        lltype.free(in_buf, flavor='raw')
-        out.free()
+                return space.wrap(res)
 compress.unwrap_spec = [ObjSpace, 'bufferstr', int]
 
 def decompress(space, data):
@@ -744,40 +738,34 @@
     if in_bufsize == 0:
         return space.wrap("")
 
-    bzs = lltype.malloc(bz_stream.TO, flavor='raw', zero=True)
-    in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-    for i in range(in_bufsize):
-        in_buf[i] = data[i]
-
-    out = OutBuffer(bzs)
-    try:
-        bzs.c_next_in = in_buf
-        rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
-    
-        bzerror = BZ2_bzDecompressInit(bzs, 0, 0)
-        if bzerror != BZ_OK:
-            _catch_bz2_error(space, bzerror)
-        
-        while True:
-            bzerror = BZ2_bzDecompress(bzs)
-            if bzerror == BZ_STREAM_END:
-                break
-            if bzerror != BZ_OK:
-                BZ2_bzDecompressEnd(bzs)
-                _catch_bz2_error(space, bzerror)
-        
-            if rffi.getintfield(bzs, 'c_avail_in') == 0:
+    with lltype.scoped_alloc(bz_stream.TO, zero=True) as bzs:
+        with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
+            for i in range(in_bufsize):
+                in_buf[i] = data[i]
+            bzs.c_next_in = in_buf
+            rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
+
+            with OutBuffer(bzs) as out:
+                bzerror = BZ2_bzDecompressInit(bzs, 0, 0)
+                if bzerror != BZ_OK:
+                    _catch_bz2_error(space, bzerror)
+
+                while True:
+                    bzerror = BZ2_bzDecompress(bzs)
+                    if bzerror == BZ_STREAM_END:
+                        break
+                    if bzerror != BZ_OK:
+                        BZ2_bzDecompressEnd(bzs)
+                    _catch_bz2_error(space, bzerror)
+
+                    if rffi.getintfield(bzs, 'c_avail_in') == 0:
+                        BZ2_bzDecompressEnd(bzs)
+                        raise OperationError(space.w_ValueError, space.wrap(
+                            "couldn't find end of stream"))
+                    elif rffi.getintfield(bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
+
+                res = out.make_result_string()
                 BZ2_bzDecompressEnd(bzs)
-                raise OperationError(space.w_ValueError,
-                                     space.wrap("couldn't find end of stream"))
-            elif rffi.getintfield(bzs, 'c_avail_out') == 0:
-                out.prepare_next_chunk()
-
-        res = out.make_result_string()
-        BZ2_bzDecompressEnd(bzs)
-        return space.wrap(res)
-    finally:
-        lltype.free(bzs, flavor='raw')
-        lltype.free(in_buf, flavor='raw')
-        out.free()
+                return space.wrap(res)
 decompress.unwrap_spec = [ObjSpace, 'bufferstr']

Modified: pypy/branch/fast-forward/pypy/module/cpyext/classobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/cpyext/classobject.py	(original)
+++ pypy/branch/fast-forward/pypy/module/cpyext/classobject.py	Fri Oct 22 23:09:43 2010
@@ -15,16 +15,20 @@
     class is the class of new object.  The dict parameter will be used as the
     object's __dict__; if NULL, a new dictionary will be created for the
     instance."""
-    if not PyClass_Check(space, w_class):
+    if not isinstance(w_class, W_ClassObject):
         return PyErr_BadInternalCall(space)
-    return W_InstanceObject(space, w_class, w_dict)
+    w_result = w_class.instantiate(space)
+    if w_dict is not None:
+        w_result.setdict(space, w_dict)
+    return w_result
 
 @cpython_api([PyObject, PyObject], PyObject, error=CANNOT_FAIL)
 def _PyInstance_Lookup(space, w_instance, w_name):
+    name = space.str_w(w_name)
     assert isinstance(w_instance, W_InstanceObject)
-    w_result = space.finditem(w_instance.w_dict, w_name)
+    w_result = w_instance.getdictvalue(space, name)
     if w_result is not None:
         return w_result
-    return w_instance.w_class.lookup(space, w_name)
+    return w_instance.w_class.lookup(space, name)
 
 

Modified: pypy/branch/fast-forward/pypy/module/cpyext/test/test_cpyext.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/cpyext/test/test_cpyext.py	(original)
+++ pypy/branch/fast-forward/pypy/module/cpyext/test/test_cpyext.py	Fri Oct 22 23:09:43 2010
@@ -16,6 +16,7 @@
 from pypy.module.cpyext.pyobject import Py_DecRef, InvalidPointerException
 from pypy.translator.goal import autopath
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 
 @api.cpython_api([], api.PyObject)
 def PyPy_Crash1(space):
@@ -78,7 +79,6 @@
         self.frozen_refcounts[w_obj] = obj.c_ob_refcnt
     #state.print_refcounts()
     self.frozen_ll2callocations = set(ll2ctypes.ALLOCATED.values())
-    lltype.start_tracking_allocations()
 
 class LeakCheckingTest(object):
     def check_and_print_leaks(self):
@@ -126,17 +126,8 @@
         for w_obj in lost_objects_w:
             print >>sys.stderr, "Lost object %r" % (w_obj, )
             leaking = True
-        for llvalue in set(ll2ctypes.ALLOCATED.values()) - self.frozen_ll2callocations:
-            if getattr(llvalue, "_traceback", None): # this means that the allocation should be tracked
-                leaking = True
-                print >>sys.stderr, "Did not deallocate %r (ll2ctypes)" % (llvalue, )
-                print >>sys.stderr, "\t" + "\n\t".join(llvalue._traceback.splitlines())
-        for llvalue in lltype.ALLOCATED.keys():
-            leaking = True
-            print >>sys.stderr, "Did not deallocate %r (llvalue)" % (llvalue, )
-            print >>sys.stderr, "\t" + "\n\t".join(llvalue._traceback.splitlines())
-
-        lltype.stop_tracking_allocations()
+        # the actual low-level leak checking is done by pypy.tool.leakfinder,
+        # enabled automatically by pypy.conftest.
         return leaking
 
 class AppTestCpythonExtensionBase(LeakCheckingTest):

Modified: pypy/branch/fast-forward/pypy/module/cpyext/test/test_unicodeobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/cpyext/test/test_unicodeobject.py	(original)
+++ pypy/branch/fast-forward/pypy/module/cpyext/test/test_unicodeobject.py	Fri Oct 22 23:09:43 2010
@@ -177,13 +177,14 @@
             encoded_charp = rffi.str2charp(encoded)
             strict_charp = rffi.str2charp("strict")
             if endian is not None:
-                pendian = lltype.malloc(rffi.INTP.TO, 1, flavor='raw')
                 if endian < 0:
-                    pendian[0] = -1
+                    value = -1
                 elif endian > 0:
-                    pendian[0] = 1
+                    value = 1
                 else:
-                    pendian[0] = 0
+                    value = 0
+                pendian = lltype.malloc(rffi.INTP.TO, 1, flavor='raw')
+                pendian[0] = rffi.cast(rffi.INT, value)
             else:
                 pendian = None
 

Modified: pypy/branch/fast-forward/pypy/module/gc/referents.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/gc/referents.py	(original)
+++ pypy/branch/fast-forward/pypy/module/gc/referents.py	Fri Oct 22 23:09:43 2010
@@ -15,8 +15,16 @@
 
 def try_cast_gcref_to_w_root(gcref):
     w_obj = rgc.try_cast_gcref_to_instance(W_Root, gcref)
-    if not we_are_translated() and not hasattr(w_obj, 'typedef'):
-        w_obj = None
+    # Ignore the instances of W_Root that are not really valid as Python
+    # objects.  There is e.g. WeakrefLifeline in module/_weakref that
+    # inherits from W_Root for internal reasons.  Such instances don't
+    # have a typedef at all (or have a null typedef after translation).
+    if not we_are_translated():
+        if not hasattr(w_obj, 'typedef'):
+            return None
+    else:
+        if w_obj is None or not w_obj.typedef:
+            return None
     return w_obj
 
 def wrap(space, gcref):

Modified: pypy/branch/fast-forward/pypy/module/imp/test/test_import.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/imp/test/test_import.py	(original)
+++ pypy/branch/fast-forward/pypy/module/imp/test/test_import.py	Fri Oct 22 23:09:43 2010
@@ -792,6 +792,7 @@
     extrapath = udir.ensure("pythonpath", dir=1) 
     extrapath.join("urllib.py").write("print 42\n")
     old = os.environ.get('PYTHONPATH', None)
+    oldlang = os.environ.pop('LANG', None)
     try: 
         os.environ['PYTHONPATH'] = str(extrapath)
         output = py.process.cmdexec('''"%s" "%s" -c "import urllib"''' % 
@@ -800,6 +801,8 @@
     finally: 
         if old: 
             os.environ['PYTHONPATH'] = old 
+        if oldlang:
+            os.environ['LANG'] = oldlang
 
 class AppTestImportHooks(object):
     def test_meta_path(self):

Modified: pypy/branch/fast-forward/pypy/module/pypyjit/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/pypyjit/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/pypyjit/__init__.py	Fri Oct 22 23:09:43 2010
@@ -6,6 +6,7 @@
 
     interpleveldefs = {
         'set_param':    'interp_jit.set_param',
+        'residual_call': 'interp_jit.residual_call',
     }
 
     def setup_after_space_initialization(self):

Modified: pypy/branch/fast-forward/pypy/module/pypyjit/interp_jit.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/pypyjit/interp_jit.py	(original)
+++ pypy/branch/fast-forward/pypy/module/pypyjit/interp_jit.py	Fri Oct 22 23:09:43 2010
@@ -5,10 +5,10 @@
 
 from pypy.tool.pairtype import extendabletype
 from pypy.rlib.rarithmetic import r_uint, intmask
-from pypy.rlib.jit import JitDriver, hint, we_are_jitted
+from pypy.rlib.jit import JitDriver, hint, we_are_jitted, dont_look_inside
 import pypy.interpreter.pyopcode   # for side-effects
 from pypy.interpreter.error import OperationError, operationerrfmt
-from pypy.interpreter.gateway import ObjSpace, Arguments
+from pypy.interpreter.gateway import ObjSpace, Arguments, W_Root
 from pypy.interpreter.pycode import PyCode, CO_GENERATOR
 from pypy.interpreter.pyframe import PyFrame
 from pypy.interpreter.pyopcode import ExitFrame
@@ -131,3 +131,10 @@
                                   "no JIT parameter '%s'", key)
 
 set_param.unwrap_spec = [ObjSpace, Arguments]
+
+ at dont_look_inside
+def residual_call(space, w_callable, args):
+    '''For testing.  Invokes callable(...), but without letting
+    the JIT follow the call.'''
+    return space.call_args(w_callable, args)
+residual_call.unwrap_spec = [ObjSpace, W_Root, Arguments]

Modified: pypy/branch/fast-forward/pypy/module/pypyjit/policy.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/pypyjit/policy.py	(original)
+++ pypy/branch/fast-forward/pypy/module/pypyjit/policy.py	Fri Oct 22 23:09:43 2010
@@ -11,7 +11,7 @@
         if '.' in modname:
             modname, _ = modname.split('.', 1)
         if modname in ['pypyjit', 'signal', 'micronumpy', 'math', 'exceptions',
-                       'imp', 'sys', 'array']:
+                       'imp', 'sys', 'array', '_ffi']:
             return True
         return False
 

Modified: pypy/branch/fast-forward/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/branch/fast-forward/pypy/module/pypyjit/test/test_pypy_c.py	Fri Oct 22 23:09:43 2010
@@ -79,8 +79,11 @@
 
 
 class PyPyCJITTests(object):
-    def run_source(self, source, expected_max_ops, *testcases):
+    def run_source(self, source, expected_max_ops, *testcases, **kwds):
         assert isinstance(expected_max_ops, int)
+        threshold = kwds.pop('threshold', 3)
+        if kwds:
+            raise TypeError, 'Unsupported keyword arguments: %s' % kwds.keys()
         source = py.code.Source(source)
         filepath = self.tmpdir.join('case%d.py' % self.counter)
         logfilepath = filepath.new(ext='.log')
@@ -92,7 +95,7 @@
             import sys
             try: # make the file runnable by CPython
                 import pypyjit
-                pypyjit.set_param(threshold=3)
+                pypyjit.set_param(threshold=%d)
             except ImportError:
                 pass
 
@@ -102,7 +105,7 @@
                 print >> sys.stderr, 'got:', repr(result)
                 assert result == expected
                 assert type(result) is type(expected)
-        """)
+        """ % threshold)
         for testcase in testcases * 2:
             print >> f, "check(%r, %r)" % testcase
         print >> f, "print 'OK :-)'"
@@ -116,6 +119,8 @@
         result = child_stdout.read()
         child_stdout.close()
         assert result
+        if result.strip().startswith('SKIP:'):
+            py.test.skip(result.strip())
         assert result.splitlines()[-1].strip() == 'OK :-)'
         self.parse_loops(logfilepath)
         self.print_loops()
@@ -123,9 +128,10 @@
         if self.total_ops > expected_max_ops:
             assert 0, "too many operations: got %d, expected maximum %d" % (
                 self.total_ops, expected_max_ops)
+        return result
 
     def parse_loops(self, opslogfile):
-        from pypy.jit.metainterp.test.oparser import parse
+        from pypy.jit.tool.oparser import parse
         from pypy.tool import logparser
         assert opslogfile.check()
         log = logparser.parse_log_file(str(opslogfile))
@@ -272,7 +278,7 @@
         assert len(ops) == 2
         assert not ops[0].get_opnames("call")
         assert not ops[0].get_opnames("new")
-        assert len(ops[0].get_opnames("guard")) <= 7
+        assert len(ops[0].get_opnames("guard")) <= 3     # we get 2 withmapdict
         assert not ops[1] # second LOOKUP_METHOD folded away
 
         ops = self.get_by_bytecode("CALL_METHOD")
@@ -283,7 +289,7 @@
             else:
                 assert not bytecode.get_opnames("call")
             assert not bytecode.get_opnames("new")
-            assert len(bytecode.get_opnames("guard")) <= 9
+            assert len(bytecode.get_opnames("guard")) <= 6
         assert len(ops[1]) < len(ops[0])
 
         ops = self.get_by_bytecode("LOAD_ATTR")
@@ -317,8 +323,8 @@
         assert len(ops) == 2
         assert not ops[0].get_opnames("call")
         assert not ops[0].get_opnames("new")
-        assert len(ops[0].get_opnames("guard")) <= 7
-        assert len(ops[0].get_opnames("getfield")) < 6
+        assert len(ops[0].get_opnames("guard")) <= 3    # we get 2 withmapdict
+        assert len(ops[0].get_opnames("getfield")) <= 5 # we get <5 withmapdict
         assert not ops[1] # second LOOKUP_METHOD folded away
 
     def test_default_and_kw(self):
@@ -382,7 +388,7 @@
                     a.x = 2
                     i = i + a.x
                 return i
-        ''', 67,
+        ''', 69,
                    ([20], 20),
                    ([31], 32))
 
@@ -390,7 +396,7 @@
                 self.get_by_bytecode("CALL_FUNCTION"))
         assert not callA.get_opnames("call")
         assert not callA.get_opnames("new")
-        assert len(callA.get_opnames("guard")) <= 8
+        assert len(callA.get_opnames("guard")) <= 2
         assert not callisinstance1.get_opnames("call")
         assert not callisinstance1.get_opnames("new")
         assert len(callisinstance1.get_opnames("guard")) <= 2
@@ -554,17 +560,13 @@
 
     def test_blockstack_virtualizable(self):
         self.run_source('''
-        def g(k):
-            s = 0
-            for i in range(k, k+2):
-                s += 1
-            return s
+        from pypyjit import residual_call
 
         def main():
             i = 0
             while i < 100:
                 try:
-                    g(i)
+                    residual_call(len, [])
                 except:
                     pass
                 i += 1
@@ -606,16 +608,17 @@
         #     call that can raise is not exchanged into getarrayitem_gc
 
     def test_overflow_checking(self):
+        startvalue = sys.maxint - 2147483647
         self.run_source('''
         def main():
             def f(a,b):
                 if a < 0: return -1
                 return a-b
-            total = 0
+            total = %d
             for i in range(100000):
                 total += f(i, 5)
             return total
-        ''', 170, ([], 4999450000L))
+        ''' % startvalue, 170, ([], startvalue + 4999450000L))
 
     def test_boolrewrite_invers(self):
         for a, b, res, ops in (('2000', '2000', 20001000, 51),
@@ -742,6 +745,8 @@
                     '''%(op1, float(a)/4.0, float(b)/4.0, op2), 109, ([], res))
 
     def test_boolrewrite_ptr(self):
+        # XXX this test is way too imprecise in what it is actually testing
+        # it should count the number of guards instead
         compares = ('a == b', 'b == a', 'a != b', 'b != a', 'a == c', 'c != b')
         for e1 in compares:
             for e2 in compares:
@@ -765,7 +770,7 @@
                 print
                 print 'Test:', e1, e2, n, res
                 self.run_source('''
-                class tst:
+                class tst(object):
                     pass
                 def main():
                     a = tst()
@@ -781,24 +786,6 @@
                     return sa
                 '''%(e1, e2), n, ([], res))
 
-    def test_boolrewrite_ptr_single(self):
-        self.run_source('''
-            class tst:
-                pass
-            def main():
-                a = tst()
-                b = tst()
-                c = tst()
-                sa = 0
-                for i in range(1000):
-                    if a == b: sa += 1
-                    else: sa += 2
-                    if a != b: sa += 10000
-                    else: sa += 20000
-                    if i > 750: a = b
-                return sa
-            ''', 215, ([], 12481752))
-
     def test_array_sum(self):
         for tc, maxops in zip('bhilBHILfd', (38,) * 6 + (40, 40, 41, 38)):
             res = 19352859
@@ -847,7 +834,12 @@
             ''', 65, ([], 122880))
 
     def test_array_intimg(self):
-        for tc, maxops in zip('ilILd', (67, 67, 69, 69, 61)):
+        # XXX this test is way too imprecise in what it is actually testing
+        # it should count the number of guards instead
+        for tc, maxops in zip('ilILd', (67, 67, 70, 70, 61)):
+            print
+            print '='*65
+            print '='*20, 'running test for tc=%r' % (tc,), '='*20
             res = 73574560
             if tc in 'IL':
                 res = long(res)
@@ -1130,6 +1122,44 @@
             return sa
         ''', 88, ([], 1997001))
 
+    def test__ffi_call(self):
+        from pypy.rlib.test.test_libffi import get_libm_name
+        libm_name = get_libm_name(sys.platform)
+        out = self.run_source('''
+        def main():
+            try:
+                from _ffi import CDLL, types
+            except ImportError:
+                sys.stdout.write('SKIP: cannot import _ffi')
+                return 0
+
+            libm = CDLL('%(libm_name)s')
+            pow = libm.getfunc('pow', [types.double, types.double],
+                               types.double)
+            print pow.getaddr()
+            i = 0
+            res = 0
+            while i < 2000:
+                res += pow(2, 3)
+                i += 1
+            return res
+        ''' % locals(),
+                              76, ([], 8.0*2000), threshold=1000)
+        pow_addr = int(out.splitlines()[0])
+        ops = self.get_by_bytecode('CALL_FUNCTION')
+        assert len(ops) == 2 # we get two loops, because of specialization
+        call_function = ops[0]
+        last_ops = [op.getopname() for op in call_function[-5:]]
+        assert last_ops == ['force_token',
+                            'setfield_gc',
+                            'call_may_force',
+                            'guard_not_forced',
+                            'guard_no_exception']
+        call = call_function[-3]
+        assert call.getarg(0).value == pow_addr
+        assert call.getarg(1).value == 2.0
+        assert call.getarg(2).value == 3.0
+
     # test_circular
 
 class AppTestJIT(PyPyCJITTests):
@@ -1155,6 +1185,17 @@
         cls.pypy_c = option.pypy_c
 
 
+def test_interface_residual_call():
+    space = gettestobjspace(usemodules=['pypyjit'])
+    space.appexec([], """():
+        import pypyjit
+        def f(*args, **kwds):
+            return (args, kwds)
+        res = pypyjit.residual_call(f, 4, x=6)
+        assert res == ((4,), {'x': 6})
+    """)
+
+
 def has_info(pypy_c, option):
     g = os.popen('"%s" --info' % pypy_c, 'r')
     lines = g.readlines()

Modified: pypy/branch/fast-forward/pypy/module/signal/interp_signal.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/signal/interp_signal.py	(original)
+++ pypy/branch/fast-forward/pypy/module/signal/interp_signal.py	Fri Oct 22 23:09:43 2010
@@ -1,7 +1,6 @@
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import W_Root, ObjSpace
 from pypy.interpreter.executioncontext import AsyncAction, AbstractActionFlag
-from pypy.rlib.rarithmetic import LONG_BIT, intmask
 import signal as cpy_signal
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.translator.tool.cbuild import ExternalCompilationInfo
@@ -64,8 +63,8 @@
 class CheckSignalAction(AsyncAction):
     """An action that is automatically invoked when a signal is received."""
 
-    # The C-level signal handler sets the highest bit of pypysig_occurred:
-    bitmask = intmask(1 << (LONG_BIT-1))
+    # The C-level signal handler sets the bit 30 of pypysig_occurred:
+    bitmask = 1 << 30
 
     def __init__(self, space):
         AsyncAction.__init__(self, space)

Modified: pypy/branch/fast-forward/pypy/module/sys/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/sys/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/sys/__init__.py	Fri Oct 22 23:09:43 2010
@@ -7,13 +7,15 @@
     """Sys Builtin Module. """
     def __init__(self, space, w_name):
         """NOT_RPYTHON""" # because parent __init__ isn't
+        if space.config.translating:
+            del self.__class__.interpleveldefs['pypy_getudir']
         super(Module, self).__init__(space, w_name) 
         self.checkinterval = 100
         self.recursionlimit = 100
         self.w_default_encoder = None
         self.defaultencoding = "ascii"
         self.filesystemencoding = None
-        
+
     interpleveldefs = {
         '__name__'              : '(space.wrap("sys"))', 
         '__doc__'               : '(space.wrap("PyPy sys module"))', 
@@ -39,7 +41,7 @@
         'py3kwarning'           : 'space.w_False',
         'warnoptions'           : 'state.get(space).w_warnoptions', 
         'builtin_module_names'  : 'state.w_None',
-        'pypy_getudir'          : 'state.pypy_getudir', 
+        'pypy_getudir'          : 'state.pypy_getudir',    # not translated
         'pypy_initial_path'     : 'state.pypy_initial_path',
 
         '_getframe'             : 'vm._getframe', 

Modified: pypy/branch/fast-forward/pypy/module/sys/state.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/sys/state.py	(original)
+++ pypy/branch/fast-forward/pypy/module/sys/state.py	Fri Oct 22 23:09:43 2010
@@ -95,15 +95,8 @@
 def getio(space):
     return space.fromcache(IOState)
 
-def _pypy_getudir(space):
-    """NOT_RPYTHON"""
+def pypy_getudir(space):
+    """NOT_RPYTHON
+    (should be removed from interpleveldefs before translation)"""
     from pypy.tool.udir import udir
     return space.wrap(str(udir))
-_pypy_getudir._annspecialcase_ = "override:ignore"
-
-# we need the indirection because this function will live in a dictionary with other 
-# RPYTHON functions and share call sites with them. Better it not be a special-case
-# directly. 
-def pypy_getudir(space):
-    return _pypy_getudir(space)
-

Modified: pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c
==============================================================================
--- pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c	(original)
+++ pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c	Fri Oct 22 23:09:43 2010
@@ -533,3 +533,7 @@
 	return inp;
 }
 
+int my_unused_function(void)
+{
+    return 42;
+}

Modified: pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py	(original)
+++ pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py	Fri Oct 22 23:09:43 2010
@@ -148,3 +148,7 @@
         # but it segfaults for some reason.
         if sys.platform == 'win32':
             assert f() == 0x12345678
+
+    def test_restype(self):
+        foo = lib.my_unused_function
+        assert foo.restype is c_int     # by default

Modified: pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py	(original)
+++ pypy/branch/fast-forward/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py	Fri Oct 22 23:09:43 2010
@@ -10,7 +10,7 @@
     """
     def test_array_of_pointers(self):
         # tests array item assignements & pointer.contents = ...
-        A = POINTER(c_int) * 24
+        A = POINTER(c_long) * 24
         a = A()
         l = c_long(2)
         p = pointer(l)

Modified: pypy/branch/fast-forward/pypy/module/thread/ll_thread.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/thread/ll_thread.py	(original)
+++ pypy/branch/fast-forward/pypy/module/thread/ll_thread.py	Fri Oct 22 23:09:43 2010
@@ -111,7 +111,7 @@
             c_thread_releaselock(self._lock)
 
     def __del__(self):
-        lltype.free(self._lock, flavor='raw')
+        lltype.free(self._lock, flavor='raw', track_allocation=False)
 
 # ____________________________________________________________
 #
@@ -128,10 +128,13 @@
 null_ll_lock = lltype.nullptr(TLOCKP.TO)
 
 def allocate_ll_lock():
-    ll_lock = lltype.malloc(TLOCKP.TO, flavor='raw')
+    # track_allocation=False here; be careful to lltype.free() it.  The
+    # reason it is set to False is that we get it from all app-level
+    # lock objects, as well as from the GIL, which exists at shutdown.
+    ll_lock = lltype.malloc(TLOCKP.TO, flavor='raw', track_allocation=False)
     res = c_thread_lock_init(ll_lock)
     if res == -1:
-        lltype.free(ll_lock, flavor='raw')
+        lltype.free(ll_lock, flavor='raw', track_allocation=False)
         raise error("out of resources")
     return ll_lock
 

Modified: pypy/branch/fast-forward/pypy/objspace/flow/flowcontext.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/flow/flowcontext.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/flow/flowcontext.py	Fri Oct 22 23:09:43 2010
@@ -398,6 +398,20 @@
                     break
 
 class FlowSpaceFrame(pyframe.CPythonFrame):
+
+    def SETUP_WITH(self, offsettoend, next_instr):
+        # A simpler version than the 'real' 2.7 one:
+        # directly call manager.__enter__(), don't use special lookup functions
+        # which don't make sense on the RPython type system.
+        from pypy.interpreter.pyopcode import WithBlock
+        w_manager = self.peekvalue()
+        w_exit = self.space.getattr(w_manager, self.space.wrap("__exit__"))
+        self.settopvalue(w_exit)
+        w_result = self.space.call_method(w_manager, "__enter__")
+        block = WithBlock(self, next_instr + offsettoend)
+        self.append_block(block)
+        self.pushvalue(w_result)
+
     # XXX Unimplemented 2.7 opcodes ----------------
 
     # Set literals, set comprehensions
@@ -413,9 +427,6 @@
     def MAP_ADD(self, oparg, next_instr):
         raise NotImplementedError("MAP_ADD")
 
-    # `with` statement
-
-    
     def make_arguments(self, nargs):
         return ArgumentsForTranslation(self.space, self.peekvalues(nargs))
     def argument_factory(self, *args):
@@ -427,3 +438,12 @@
             raise operr
         return pyframe.PyFrame.handle_operation_error(self, ec, operr,
                                                       *args, **kwds)
+
+    def call_contextmanager_exit_function(self, w_func, w_typ, w_val, w_tb):
+        if w_typ is not self.space.w_None:
+            # The annotator won't allow to merge exception types with None.
+            # Replace it with an object which will break translation when used
+            # (except maybe with 'exc_typ is None')
+            w_typ = self.space.wrap(self.space)
+        return self.space.call_function(w_func, w_typ, w_val, w_tb)
+

Modified: pypy/branch/fast-forward/pypy/objspace/flow/test/test_objspace.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/flow/test/test_objspace.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/flow/test/test_objspace.py	Fri Oct 22 23:09:43 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 import new
 import py
 from pypy.objspace.flow.model import Constant, Block, Link, Variable, traverse
@@ -828,6 +829,25 @@
         simplify_graph(graph)
         assert self.all_operations(graph) == {'getitem': 1}
 
+    def test_context_manager(self):
+        def f(c, x):
+            with x:
+                pass
+        graph = self.codetest(f)
+        # 2 method calls: x.__enter__() and x.__exit__(None, None, None)
+        assert self.all_operations(graph) == {'getattr': 2,
+                                              'simple_call': 2}
+        #
+        def g(): pass
+        def f(c, x):
+            with x:
+                g()
+        graph = self.codetest(f)
+        assert self.all_operations(graph) == {
+            'getattr': 2,     # __enter__ and __exit__
+            'simple_call': 4, # __enter__, g and 2 possible calls to __exit__
+            'is_true': 1}     # check the result of __exit__()
+
     def monkey_patch_code(self, code, stacksize, flags, codestring, names, varnames):
         c = code
         return new.code(c.co_argcount, c.co_nlocals, stacksize, flags,

Modified: pypy/branch/fast-forward/pypy/objspace/std/celldict.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/celldict.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/celldict.py	Fri Oct 22 23:09:43 2010
@@ -45,7 +45,7 @@
         if space.is_w(space.type(w_key), space.w_str):
             self.impl_setitem_str(self.space.str_w(w_key), w_value)
         else:
-            self._as_rdict().setitem(w_key, w_value)
+            self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
     def impl_setitem_str(self, name, w_value, shadows_type=True):
         self.getcell(name, True).w_value = w_value
@@ -66,7 +66,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
         
     def impl_length(self):
         # inefficient, but do we care?
@@ -85,7 +85,7 @@
         elif _is_sane_hash(space, w_lookup_type):
             return None
         else:
-            return self._as_rdict().getitem(w_lookup)
+            return self._as_rdict().impl_fallback_getitem(w_lookup)
 
     def impl_getitem_str(self, lookup):
         res = self.getcell(lookup, False)

Modified: pypy/branch/fast-forward/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/dictmultiobject.py	Fri Oct 22 23:09:43 2010
@@ -102,17 +102,17 @@
         else:
             return None
 
-    # _________________________________________________________________ 
+    # _________________________________________________________________
     # implementation methods
     def impl_getitem(self, w_key):
         #return w_value or None
         raise NotImplementedError("abstract base class")
 
-    def impl_getitem_str(self, w_key):
+    def impl_getitem_str(self, key):
         #return w_value or None
         raise NotImplementedError("abstract base class")
 
-    def impl_setitem_str(self,  key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value, shadows_type=True):
         raise NotImplementedError("abstract base class")
 
     def impl_setitem(self,  w_key, w_value):
@@ -120,7 +120,7 @@
 
     def impl_delitem(self, w_key):
         raise NotImplementedError("abstract base class")
- 
+
     def impl_length(self):
         raise NotImplementedError("abstract base class")
 
@@ -310,7 +310,7 @@
         if space.is_w(space.type(w_key), space.w_str):
             self.impl_setitem_str(self.space.str_w(w_key), w_value)
         else:
-            self._as_rdict().setitem(w_key, w_value)
+            self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
     def impl_setitem_str(self, key, w_value, shadows_type=True):
         self.content[key] = w_value
@@ -324,7 +324,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
         
     def impl_length(self):
         return len(self.content)
@@ -344,7 +344,7 @@
         elif _is_sane_hash(space, w_lookup_type):
             return None
         else:
-            return self._as_rdict().getitem(w_key)
+            return self._as_rdict().impl_fallback_getitem(w_key)
 
     def impl_iter(self):
         return StrIteratorImplementation(self.space, self)
@@ -414,7 +414,7 @@
             StrDictImplementation.impl_setitem_str(
                 self, self.space.str_w(w_key), w_value, False)
         else:
-            self._as_rdict().setitem(w_key, w_value)
+            self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
     def impl_shadows_anything(self):
         return (self._shadows_anything or 
@@ -446,7 +446,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
 
     def impl_get_builtin_indexed(self, i):
         return self.shadowed[i]

Modified: pypy/branch/fast-forward/pypy/objspace/std/fake.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/fake.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/fake.py	Fri Oct 22 23:09:43 2010
@@ -21,7 +21,6 @@
     #debug_print("faking obj %s" % x)
     ft = fake_type(type(x))
     return ft(space, x)
-fake_object._annspecialcase_ = "override:fake_object"
 
 import sys
 
@@ -47,7 +46,6 @@
         w_exc = space.wrap(exc)
         w_value = space.wrap(value)
     raise OperationError, OperationError(w_exc, w_value), tb
-wrap_exception._annspecialcase_ = "override:ignore"
 
 def fake_type(cpy_type):
     assert type(cpy_type) is type

Modified: pypy/branch/fast-forward/pypy/objspace/std/model.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/model.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/model.py	Fri Oct 22 23:09:43 2010
@@ -18,6 +18,7 @@
     "withsmallint"   : ["smallintobject.W_SmallIntObject"],
     "withstrslice"   : ["strsliceobject.W_StringSliceObject"],
     "withstrjoin"    : ["strjoinobject.W_StringJoinObject"],
+    "withstrbuf"     : ["strbufobject.W_StringBufferObject"],
     "withrope"       : ["ropeobject.W_RopeObject",
                         "ropeobject.W_RopeIterObject"],
     "withropeunicode": ["ropeunicodeobject.W_RopeUnicodeObject",
@@ -77,6 +78,7 @@
         from pypy.objspace.std import ropeunicodeobject
         from pypy.objspace.std import strsliceobject
         from pypy.objspace.std import strjoinobject
+        from pypy.objspace.std import strbufobject
         from pypy.objspace.std import typeobject
         from pypy.objspace.std import sliceobject
         from pypy.objspace.std import longobject
@@ -228,6 +230,13 @@
                 (unicodeobject.W_UnicodeObject,
                                        strjoinobject.delegate_join2unicode)
                 ]
+        elif config.objspace.std.withstrbuf:
+            self.typeorder[strbufobject.W_StringBufferObject] += [
+                (stringobject.W_StringObject,
+                                       strbufobject.delegate_buf2str),
+                (unicodeobject.W_UnicodeObject,
+                                       strbufobject.delegate_buf2unicode)
+                ]
         if config.objspace.std.withrangelist:
             self.typeorder[rangeobject.W_RangeListObject] += [
                 (listobject.W_ListObject,

Modified: pypy/branch/fast-forward/pypy/objspace/std/objspace.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/objspace.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/objspace.py	Fri Oct 22 23:09:43 2010
@@ -23,6 +23,7 @@
 from pypy.objspace.std.listobject import W_ListObject
 from pypy.objspace.std.longobject import W_LongObject
 from pypy.objspace.std.noneobject import W_NoneObject
+from pypy.objspace.std.objectobject import W_ObjectObject
 from pypy.objspace.std.ropeobject import W_RopeObject
 from pypy.objspace.std.iterobject import W_SeqIterObject
 from pypy.objspace.std.setobject import W_SetObject, W_FrozensetObject
@@ -242,7 +243,6 @@
             w_result = getattr(self, 'w_' + x.__name__)
             return w_result
         return None
-    wrap_exception_cls._annspecialcase_ = "override:wrap_exception_cls"
 
     def unwrap(self, w_obj):
         if isinstance(w_obj, Wrappable):
@@ -318,9 +318,14 @@
             w_subtype = w_type.check_user_subclass(w_subtype)
             if cls.typedef.applevel_subclasses_base is not None:
                 cls = cls.typedef.applevel_subclasses_base
-            subcls = get_unique_interplevel_subclass(
-                    self.config, cls, w_subtype.hasdict, w_subtype.nslots != 0,
-                    w_subtype.needsdel, w_subtype.weakrefable)
+            if (self.config.objspace.std.withmapdict and cls is W_ObjectObject
+                    and not w_subtype.needsdel):
+                from pypy.objspace.std.mapdict import get_subclass_of_correct_size
+                subcls = get_subclass_of_correct_size(self, cls, w_subtype)
+            else:
+                subcls = get_unique_interplevel_subclass(
+                        self.config, cls, w_subtype.hasdict, w_subtype.nslots != 0,
+                        w_subtype.needsdel, w_subtype.weakrefable)
             instance = instantiate(subcls)
             assert isinstance(instance, cls)
             instance.user_setup(self, w_subtype)

Modified: pypy/branch/fast-forward/pypy/objspace/std/sharingdict.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/sharingdict.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/sharingdict.py	Fri Oct 22 23:09:43 2010
@@ -71,7 +71,7 @@
         elif _is_sane_hash(space, w_lookup_type):
             return None
         else:
-            return self._as_rdict().getitem(w_lookup)
+            return self._as_rdict().impl_fallback_getitem(w_lookup)
 
     def impl_getitem_str(self, lookup):
         i = self.structure.lookup_position(lookup)
@@ -84,7 +84,7 @@
         if space.is_w(space.type(w_key), space.w_str):
             self.impl_setitem_str(self.space.str_w(w_key), w_value)
         else:
-            self._as_rdict().setitem(w_key, w_value)
+            self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
     @unroll_safe
     def impl_setitem_str(self, key, w_value, shadows_type=True):
@@ -132,7 +132,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
         
     def impl_length(self):
         return self.structure.length

Modified: pypy/branch/fast-forward/pypy/objspace/std/stringobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/stringobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/stringobject.py	Fri Oct 22 23:09:43 2010
@@ -14,7 +14,7 @@
 from pypy.rlib.rstring import StringBuilder, string_repeat
 from pypy.interpreter.buffer import StringBuffer
 
-from pypy.objspace.std.stringtype import sliced, joined, wrapstr, wrapchar, \
+from pypy.objspace.std.stringtype import sliced, wrapstr, wrapchar, \
      stringendswith, stringstartswith, joined2
 
 from pypy.objspace.std.formatting import mod_format

Modified: pypy/branch/fast-forward/pypy/objspace/std/stringtype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/stringtype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/stringtype.py	Fri Oct 22 23:09:43 2010
@@ -55,19 +55,14 @@
             return W_StringSliceObject(s, start, stop)
     return wrapstr(space, s[start:stop])
 
-def joined(space, strlist):
-    assert not space.config.objspace.std.withrope
-    if space.config.objspace.std.withstrjoin:
-        from pypy.objspace.std.strjoinobject import W_StringJoinObject
-        return W_StringJoinObject(strlist)
-    else:
-        return wrapstr(space, "".join(strlist))
-
 def joined2(space, str1, str2):
     assert not space.config.objspace.std.withrope
     if space.config.objspace.std.withstrjoin:
         from pypy.objspace.std.strjoinobject import W_StringJoinObject
         return W_StringJoinObject([str1, str2])
+    elif space.config.objspace.std.withstrbuf:
+        from pypy.objspace.std.strbufobject import joined2
+        return joined2(str1, str2)
     else:
         return wrapstr(space, str1 + str2)
 

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_dictmultiobject.py	Fri Oct 22 23:09:43 2010
@@ -602,6 +602,15 @@
                 classofinstance=classofinstance,
                 from_strdict_shared=from_strdict_shared)
 
+    def finditem_str(self, w_dict, s):
+        return w_dict.getitem_str(s) # assume it's a multidict
+
+    def setitem_str(self, w_dict, s, w_value):
+        return w_dict.setitem_str(s, w_value) # assume it's a multidict
+
+    def delitem(self, w_dict, w_s):
+        return w_dict.delitem(w_s) # assume it's a multidict
+
     def allocate_instance(self, cls, type):
         return object.__new__(cls)
 
@@ -611,7 +620,7 @@
     w_StopIteration = StopIteration
     w_None = None
     StringObjectCls = FakeString
-    w_dict = None
+    w_dict = W_DictMultiObject
     iter = iter
     fixedview = list
     listview  = list
@@ -687,6 +696,14 @@
         assert self.impl.length() == 0
         self.check_not_devolved()
 
+    def test_clear(self):
+        self.fill_impl()
+        assert self.impl.length() == 2
+        self.impl.clear()
+        assert self.impl.length() == 0
+        self.check_not_devolved()
+
+
     def test_keys(self):
         self.fill_impl()
         keys = self.impl.keys()

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_shadowtracking.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_shadowtracking.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_shadowtracking.py	Fri Oct 22 23:09:43 2010
@@ -3,7 +3,8 @@
 
 class TestShadowTracking(object):
     def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withshadowtracking": True})
+        cls.space = gettestobjspace(**{"objspace.std.withshadowtracking": True,
+                                       "objspace.std.withmapdict": False})
 
     def test_simple_shadowing(self):
         space = self.space

Modified: pypy/branch/fast-forward/pypy/objspace/std/typeobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/typeobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/typeobject.py	Fri Oct 22 23:09:43 2010
@@ -76,7 +76,9 @@
                           'weakrefable',
                           'hasdict',
                           'nslots',
-                          'instancetypedef']
+                          'instancetypedef',
+                          'terminator',
+                          ]
 
     # for config.objspace.std.getattributeshortcut
     # (False is a conservative default, fixed during real usage)
@@ -118,6 +120,12 @@
                 # dict_w of any of the types in the mro changes, or if the mro
                 # itself changes
                 w_self._version_tag = VersionTag()
+        if space.config.objspace.std.withmapdict:
+            from pypy.objspace.std.mapdict import DictTerminator, NoDictTerminator
+            if w_self.hasdict:
+                w_self.terminator = DictTerminator(space, w_self)
+            else:
+                w_self.terminator = NoDictTerminator(space, w_self)
 
     def mutated(w_self):
         space = w_self.space

Modified: pypy/branch/fast-forward/pypy/rlib/_rsocket_rffi.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/_rsocket_rffi.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/_rsocket_rffi.py	Fri Oct 22 23:09:43 2010
@@ -332,10 +332,11 @@
                                ('sll_hatype', rffi.INT),
                                ('sll_addr', rffi.CFixedArray(rffi.CHAR, 8)),
                                ('sll_halen', rffi.INT)],
-                              )
+                              ifdef='AF_PACKET')
 
     CConfig.ifreq = platform.Struct('struct ifreq', [('ifr_ifindex', rffi.INT),
-                                 ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))])
+                                 ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))],
+                                    ifdef='AF_PACKET')
 
 if _WIN32:
     CConfig.WSAEVENT = platform.SimpleType('WSAEVENT', rffi.VOIDP)
@@ -546,8 +547,9 @@
     socketpair_t = rffi.CArray(socketfd_type)
     socketpair = external('socketpair', [rffi.INT, rffi.INT, rffi.INT,
                           lltype.Ptr(socketpair_t)], rffi.INT)
-    ioctl = external('ioctl', [socketfd_type, rffi.INT, lltype.Ptr(ifreq)],
-                     rffi.INT)
+    if ifreq is not None:
+        ioctl = external('ioctl', [socketfd_type, rffi.INT, lltype.Ptr(ifreq)],
+                         rffi.INT)
 
 if _WIN32:
     ioctlsocket = external('ioctlsocket',

Modified: pypy/branch/fast-forward/pypy/rlib/jit.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/jit.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/jit.py	Fri Oct 22 23:09:43 2010
@@ -77,6 +77,12 @@
         return result
     return decorator
 
+def oopspec(spec):
+    def decorator(func):
+        func.oopspec = spec
+        return func
+    return decorator
+
 class Entry(ExtRegistryEntry):
     _about_ = hint
 
@@ -139,6 +145,24 @@
         return hop.inputconst(lltype.Signed, _we_are_jitted)
 
 
+def jit_debug(string, arg1=-sys.maxint-1, arg2=-sys.maxint-1,
+                      arg3=-sys.maxint-1, arg4=-sys.maxint-1):
+    """When JITted, cause an extra operation DEBUG_MERGE_POINT to appear in
+    the graphs.  Should not be left after debugging."""
+    keepalive_until_here(string) # otherwise the whole function call is removed
+jit_debug.oopspec = 'jit.debug(string, arg1, arg2, arg3, arg4)'
+
+def assert_green(value):
+    """Very strong assert: checks that 'value' is a green
+    (a JIT compile-time constant)."""
+    keepalive_until_here(value)
+assert_green._annspecialcase_ = 'specialize:argtype(0)'
+assert_green.oopspec = 'jit.assert_green(value)'
+
+class AssertGreenFailed(Exception):
+    pass
+
+
 ##def force_virtualizable(virtualizable):
 ##    pass
 
@@ -250,8 +274,9 @@
     several independent JITting interpreters in it.
     """
 
+    active = True          # if set to False, this JitDriver is ignored
     virtualizables = []
-    
+
     def __init__(self, greens=None, reds=None, virtualizables=None,
                  get_jitcell_at=None, set_jitcell_at=None,
                  get_printable_location=None, confirm_enter_jit=None,
@@ -266,7 +291,8 @@
             self.virtualizables = virtualizables
         for v in self.virtualizables:
             assert v in self.reds
-        self._alllivevars = dict.fromkeys(self.greens + self.reds)
+        self._alllivevars = dict.fromkeys(
+            [name for name in self.greens + self.reds if '.' not in name])
         self._make_extregistryentries()
         self.get_jitcell_at = get_jitcell_at
         self.set_jitcell_at = set_jitcell_at
@@ -355,10 +381,16 @@
 
     def compute_result_annotation(self, **kwds_s):
         from pypy.annotation import model as annmodel
+
+        if self.instance.__name__ == 'jit_merge_point':
+            if not self.annotate_hooks(**kwds_s):
+                return None      # wrong order, try again later
+
         driver = self.instance.im_self
         keys = kwds_s.keys()
         keys.sort()
-        expected = ['s_' + name for name in driver.greens + driver.reds]
+        expected = ['s_' + name for name in driver.greens + driver.reds
+                                if '.' not in name]
         expected.sort()
         if keys != expected:
             raise JitHintError("%s expects the following keyword "
@@ -382,30 +414,35 @@
                                    key[2:])
             cache[key] = s_value
 
-        if self.instance.__name__ == 'jit_merge_point':
-            self.annotate_hooks(**kwds_s)
-            
         return annmodel.s_None
 
     def annotate_hooks(self, **kwds_s):
         driver = self.instance.im_self
         s_jitcell = self.bookkeeper.valueoftype(BaseJitCell)
-        self.annotate_hook(driver.get_jitcell_at, driver.greens, **kwds_s)
-        self.annotate_hook(driver.set_jitcell_at, driver.greens, [s_jitcell],
-                           **kwds_s)
-        self.annotate_hook(driver.get_printable_location, driver.greens, **kwds_s)
+        h = self.annotate_hook
+        return (h(driver.get_jitcell_at, driver.greens, **kwds_s)
+            and h(driver.set_jitcell_at, driver.greens, [s_jitcell], **kwds_s)
+            and h(driver.get_printable_location, driver.greens, **kwds_s))
 
     def annotate_hook(self, func, variables, args_s=[], **kwds_s):
         if func is None:
-            return
+            return True
         bk = self.bookkeeper
         s_func = bk.immutablevalue(func)
         uniquekey = 'jitdriver.%s' % func.func_name
         args_s = args_s[:]
         for name in variables:
-            s_arg = kwds_s['s_' + name]
+            if '.' not in name:
+                s_arg = kwds_s['s_' + name]
+            else:
+                objname, fieldname = name.split('.')
+                s_instance = kwds_s['s_' + objname]
+                s_arg = s_instance.classdef.about_attribute(fieldname)
+                if s_arg is None:
+                    return False     # wrong order, try again later
             args_s.append(s_arg)
         bk.emulate_pbc_call(uniquekey, s_func, args_s)
+        return True
 
     def specialize_call(self, hop, **kwds_i):
         # XXX to be complete, this could also check that the concretetype
@@ -416,9 +453,42 @@
         greens_v = []
         reds_v = []
         for name in driver.greens:
-            i = kwds_i['i_' + name]
-            r_green = hop.args_r[i]
-            v_green = hop.inputarg(r_green, arg=i)
+            if '.' not in name:
+                i = kwds_i['i_' + name]
+                r_green = hop.args_r[i]
+                v_green = hop.inputarg(r_green, arg=i)
+            else:
+                if hop.rtyper.type_system.name == 'ootypesystem':
+                    py.test.skip("lltype only")
+                objname, fieldname = name.split('.')   # see test_green_field
+                assert objname in driver.reds
+                i = kwds_i['i_' + objname]
+                s_red = hop.args_s[i]
+                r_red = hop.args_r[i]
+                while True:
+                    try:
+                        mangled_name, r_field = r_red._get_field(fieldname)
+                        break
+                    except KeyError:
+                        pass
+                    assert r_red.rbase is not None, (
+                        "field %r not found in %r" % (name,
+                                                      r_red.lowleveltype.TO))
+                    r_red = r_red.rbase
+                GTYPE = r_red.lowleveltype.TO
+                assert GTYPE._immutable_field(mangled_name), (
+                    "field %r must be declared as immutable" % name)
+                if not hasattr(driver, 'll_greenfields'):
+                    driver.ll_greenfields = {}
+                driver.ll_greenfields[name] = GTYPE, mangled_name
+                #
+                v_red = hop.inputarg(r_red, arg=i)
+                c_llname = hop.inputconst(lltype.Void, mangled_name)
+                v_green = hop.genop('getfield', [v_red, c_llname],
+                                    resulttype = r_field)
+                s_green = s_red.classdef.about_attribute(fieldname)
+                assert s_green is not None
+                hop.rtyper.annotator.setbinding(v_green, s_green)
             greens_v.append(v_green)
         for name in driver.reds:
             i = kwds_i['i_' + name]

Modified: pypy/branch/fast-forward/pypy/rlib/rgc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rgc.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rgc.py	Fri Oct 22 23:09:43 2010
@@ -170,7 +170,14 @@
         return hop.genop('gc_set_max_heap_size', [v_nbytes],
                          resulttype=lltype.Void)
 
-def can_move(p):    # NB. must not be called with NULL pointers
+def can_move(p):
+    """Check if the GC object 'p' is at an address that can move.
+    Must not be called with None.  With non-moving GCs, it is always False.
+    With some moving GCs like the SemiSpace GC, it is always True.
+    With other moving GCs like the MiniMark GC, it can be True for some
+    time, then False for the same object, when we are sure that it won't
+    move any more.
+    """
     return True
 
 class CanMoveEntry(ExtRegistryEntry):

Modified: pypy/branch/fast-forward/pypy/rlib/rmmap.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rmmap.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rmmap.py	Fri Oct 22 23:09:43 2010
@@ -50,7 +50,7 @@
     constant_names = ['MAP_SHARED', 'MAP_PRIVATE',
                       'PROT_READ', 'PROT_WRITE',
                       'MS_SYNC']
-    opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS',
+    opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS', 'MAP_NORESERVE',
                           'PROT_EXEC',
                           'MAP_DENYWRITE', 'MAP_EXECUTABLE']
     for name in constant_names:

Modified: pypy/branch/fast-forward/pypy/rlib/rsre/rsre_char.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rsre/rsre_char.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rsre/rsre_char.py	Fri Oct 22 23:09:43 2010
@@ -4,6 +4,7 @@
 import sys
 from pypy.rlib.rlocale import tolower, isalnum
 from pypy.rlib.unroll import unrolling_iterable
+from pypy.rlib import jit
 
 # Note: the unicode parts of this module require you to call
 # rsre_char.set_unicode_db() first, to select one of the modules
@@ -43,6 +44,7 @@
 # XXX can we import those safely from sre_constants?
 SRE_INFO_PREFIX = 1
 SRE_INFO_LITERAL = 2
+SRE_INFO_CHARSET = 4
 SRE_FLAG_LOCALE = 4 # honour system locale
 SRE_FLAG_UNICODE = 32 # use unicode locale
 OPCODE_INFO = 17
@@ -64,33 +66,27 @@
 
 #### Category helpers
 
-ascii_char_info = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 6, 2,
-2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0,
-0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 25, 25, 25, 25, 25, 25, 25, 25,
-25, 25, 0, 0, 0, 0, 0, 0, 0, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
-24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 0, 0,
-0, 0, 16, 0, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
-24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 0, 0, 0, 0, 0 ]
-
+is_a_word = [(chr(i).isalnum() or chr(i) == '_') for i in range(256)]
 linebreak = ord("\n")
 underline = ord("_")
 
 def is_digit(code):
-    return code < 128 and (ascii_char_info[code] & 1 != 0)
+    return code <= 57 and code >= 48
 
 def is_uni_digit(code):
     assert unicodedb is not None
     return unicodedb.isdecimal(code)
 
 def is_space(code):
-    return code < 128 and (ascii_char_info[code] & 2 != 0)
+    return code == 32 or (code <= 13 and code >= 9)
 
 def is_uni_space(code):
     assert unicodedb is not None
     return unicodedb.isspace(code)
 
 def is_word(code):
-    return code < 128 and (ascii_char_info[code] & 16 != 0)
+    assert code >= 0
+    return code < 256 and is_a_word[code]
 
 def is_uni_word(code):
     assert unicodedb is not None
@@ -142,6 +138,7 @@
 SET_OK = -1
 SET_NOT_OK = -2
 
+ at jit.unroll_safe
 def check_charset(pattern, ppos, char_code):
     """Checks whether a character matches set of arbitrary length.
     The set starts at pattern[ppos]."""

Modified: pypy/branch/fast-forward/pypy/rlib/rsre/rsre_core.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rsre/rsre_core.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rsre/rsre_core.py	Fri Oct 22 23:09:43 2010
@@ -1,9 +1,11 @@
 import sys
-from pypy.rlib.debug import check_nonneg
+from pypy.rlib.debug import check_nonneg, make_sure_not_modified
 from pypy.rlib.unroll import unrolling_iterable
 from pypy.rlib.rsre import rsre_char
 from pypy.tool.sourcetools import func_with_new_name
 from pypy.rlib.objectmodel import we_are_translated
+from pypy.rlib import jit
+from pypy.rlib.rsre.rsre_jit import install_jitdriver, install_jitdriver_spec
 
 
 OPCODE_FAILURE            = 0
@@ -56,16 +58,19 @@
     _seen_specname[specname] = True
     # Install a copy of the function under the name '_spec_funcname' in each
     # concrete subclass
+    specialized_methods = []
     for prefix, concreteclass in [('str', StrMatchContext),
                                   ('uni', UnicodeMatchContext)]:
         newfunc = func_with_new_name(func, prefix + specname)
         assert not hasattr(concreteclass, specname)
         setattr(concreteclass, specname, newfunc)
+        specialized_methods.append(newfunc)
     # Return a dispatcher function, specialized on the exact type of 'ctx'
     def dispatch(ctx, *args):
         return getattr(ctx, specname)(*args)
     dispatch._annspecialcase_ = 'specialize:argtype(0)'
-    return dispatch
+    dispatch._specialized_methods_ = specialized_methods
+    return func_with_new_name(dispatch, specname)
 
 # ____________________________________________________________
 
@@ -75,6 +80,7 @@
 
 class AbstractMatchContext(object):
     """Abstract base class"""
+    _immutable_fields_ = ['pattern[*]', 'flags', 'end']
     match_start = 0
     match_end = 0
     match_marks = None
@@ -164,6 +170,8 @@
     def __init__(self, pattern, string, match_start, end, flags):
         AbstractMatchContext.__init__(self, pattern, match_start, end, flags)
         self._string = string
+        if not we_are_translated() and isinstance(string, unicode):
+            self.flags |= rsre_char.SRE_FLAG_UNICODE   # for rsre_re.py
 
     def str(self, index):
         check_nonneg(index)
@@ -238,8 +246,9 @@
         self.start_ptr = ptr
         self.start_marks = marks
 
+    @jit.unroll_safe
     def find_first_result(self, ctx):
-        ppos = self.ppos
+        ppos = jit.hint(self.ppos, promote=True)
         while ctx.pat(ppos):
             result = sre_match(ctx, ppos + 1, self.start_ptr, self.start_marks)
             ppos += ctx.pat(ppos)
@@ -250,6 +259,10 @@
     find_next_result = find_first_result
 
 class RepeatOneMatchResult(MatchResult):
+    install_jitdriver('RepeatOne',
+                      greens=['nextppos', 'ctx.pattern'],
+                      reds=['ptr', 'self', 'ctx'],
+                      debugprint=(1, 0))   # indices in 'greens'
 
     def __init__(self, nextppos, minptr, ptr, marks):
         self.nextppos = nextppos
@@ -259,8 +272,11 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
+        nextppos = self.nextppos
         while ptr >= self.minptr:
-            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
+            ctx.jitdriver_RepeatOne.jit_merge_point(
+                self=self, ptr=ptr, ctx=ctx, nextppos=nextppos)
+            result = sre_match(ctx, nextppos, ptr, self.start_marks)
             ptr -= 1
             if result is not None:
                 self.subresult = result
@@ -270,6 +286,10 @@
 
 
 class MinRepeatOneMatchResult(MatchResult):
+    install_jitdriver('MinRepeatOne',
+                      greens=['nextppos', 'ppos3', 'ctx.pattern'],
+                      reds=['ptr', 'self', 'ctx'],
+                      debugprint=(2, 0))   # indices in 'greens'
 
     def __init__(self, nextppos, ppos3, maxptr, ptr, marks):
         self.nextppos = nextppos
@@ -280,29 +300,32 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
+        nextppos = self.nextppos
+        ppos3 = self.ppos3
         while ptr <= self.maxptr:
-            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
+            ctx.jitdriver_MinRepeatOne.jit_merge_point(
+                self=self, ptr=ptr, ctx=ctx, nextppos=nextppos, ppos3=ppos3)
+            result = sre_match(ctx, nextppos, ptr, self.start_marks)
             if result is not None:
                 self.subresult = result
                 self.start_ptr = ptr
                 return self
-            if not self.next_char_ok(ctx, ptr):
+            if not self.next_char_ok(ctx, ptr, ppos3):
                 break
             ptr += 1
 
     def find_next_result(self, ctx):
         ptr = self.start_ptr
-        if not self.next_char_ok(ctx, ptr):
+        if not self.next_char_ok(ctx, ptr, self.ppos3):
             return
         self.start_ptr = ptr + 1
         return self.find_first_result(ctx)
 
-    def next_char_ok(self, ctx, ptr):
+    def next_char_ok(self, ctx, ptr, ppos):
         if ptr == ctx.end:
             return False
-        ppos = self.ppos3
         op = ctx.pat(ppos)
-        for op1, (checkerfn, _) in unroll_char_checker:
+        for op1, checkerfn in unroll_char_checker:
             if op1 == op:
                 return checkerfn(ctx, ptr, ppos)
         raise Error("next_char_ok[%d]" % op)
@@ -325,41 +348,34 @@
         self.next = next     # chained list
 
 class MaxUntilMatchResult(AbstractUntilMatchResult):
+    install_jitdriver('MaxUntil',
+                      greens=['ppos', 'tailppos', 'match_more', 'ctx.pattern'],
+                      reds=['ptr', 'marks', 'self', 'ctx'],
+                      debugprint=(3, 0, 2))
 
     def find_first_result(self, ctx):
-        enum = sre_match(ctx, self.ppos + 3, self.cur_ptr, self.cur_marks)
-        return self.search_next(ctx, enum, resume=False)
+        return self.search_next(ctx, match_more=True)
 
     def find_next_result(self, ctx):
-        return self.search_next(ctx, None, resume=True)
+        return self.search_next(ctx, match_more=False)
 
-    def search_next(self, ctx, enum, resume):
+    def search_next(self, ctx, match_more):
         ppos = self.ppos
-        min = ctx.pat(ppos+1)
-        max = ctx.pat(ppos+2)
+        tailppos = self.tailppos
         ptr = self.cur_ptr
         marks = self.cur_marks
         while True:
-            while True:
-                if (enum is not None and
-                    (ptr != ctx.match_end or self.num_pending < min)):
-                    #               ^^^^^^^^^^ zero-width match protection
-                    # matched one more 'item'.  record it and continue.
-                    self.pending = Pending(ptr, marks, enum, self.pending)
-                    self.num_pending += 1
-                    ptr = ctx.match_end
-                    marks = ctx.match_marks
-                    break
-                # 'item' no longer matches.
-                if not resume and self.num_pending >= min:
-                    # try to match 'tail' if we have enough 'item'
-                    result = sre_match(ctx, self.tailppos, ptr, marks)
-                    if result is not None:
-                        self.subresult = result
-                        self.cur_ptr = ptr
-                        self.cur_marks = marks
-                        return self
-                resume = False
+            ctx.jitdriver_MaxUntil.jit_merge_point(
+                ppos=ppos, tailppos=tailppos, match_more=match_more,
+                ptr=ptr, marks=marks, self=self, ctx=ctx)
+            if match_more:
+                max = ctx.pat(ppos+2)
+                if max == 65535 or self.num_pending < max:
+                    # try to match one more 'item'
+                    enum = sre_match(ctx, ppos + 3, ptr, marks)
+                else:
+                    enum = None    # 'max' reached, no more matches
+            else:
                 p = self.pending
                 if p is None:
                     return
@@ -369,11 +385,27 @@
                 marks = p.marks
                 enum = p.enum.move_to_next_result(ctx)
             #
-            if max == 65535 or self.num_pending < max:
-                # try to match one more 'item'
-                enum = sre_match(ctx, ppos + 3, ptr, marks)
+            min = ctx.pat(ppos+1)
+            if (enum is not None and
+                (ptr != ctx.match_end or self.num_pending < min)):
+                #               ^^^^^^^^^^ zero-width match protection
+                # matched one more 'item'.  record it and continue.
+                self.pending = Pending(ptr, marks, enum, self.pending)
+                self.num_pending += 1
+                ptr = ctx.match_end
+                marks = ctx.match_marks
+                match_more = True
             else:
-                enum = None    # 'max' reached, no more matches
+                # 'item' no longer matches.
+                if self.num_pending >= min:
+                    # try to match 'tail' if we have enough 'item'
+                    result = sre_match(ctx, tailppos, ptr, marks)
+                    if result is not None:
+                        self.subresult = result
+                        self.cur_ptr = ptr
+                        self.cur_marks = marks
+                        return self
+                match_more = False
 
 class MinUntilMatchResult(AbstractUntilMatchResult):
 
@@ -384,6 +416,7 @@
         return self.search_next(ctx, resume=True)
 
     def search_next(self, ctx, resume):
+        # XXX missing jit support here
         ppos = self.ppos
         min = ctx.pat(ppos+1)
         max = ctx.pat(ppos+2)
@@ -429,6 +462,7 @@
 # ____________________________________________________________
 
 @specializectx
+ at jit.unroll_safe
 def sre_match(ctx, ppos, ptr, marks):
     """Returns either None or a MatchResult object.  Usually we only need
     the first result, but there is the case of REPEAT...UNTIL where we
@@ -437,6 +471,13 @@
     while True:
         op = ctx.pat(ppos)
         ppos += 1
+        make_sure_not_modified(ctx.pattern)
+
+        #jit.jit_debug("sre_match", op, ppos, ptr)
+        #
+        # When using the JIT, calls to sre_match() must always have a constant
+        # (green) argument for 'ppos'.  If not, the following assert fails.
+        jit.assert_green(op)
 
         if op == OPCODE_FAILURE:
             return
@@ -712,13 +753,23 @@
 @specializectx
 def find_repetition_end(ctx, ppos, ptr, maxcount):
     end = ctx.end
-    # adjust end
-    if maxcount != 65535:
+    if maxcount <= 1:
+        if maxcount == 1 and ptr < end:
+            # Relatively common case: maxcount == 1.  If we are not at the
+            # end of the string, it's done by a single direct check.
+            op = ctx.pat(ppos)
+            for op1, checkerfn in unroll_char_checker:
+                if op1 == op:
+                    if checkerfn(ctx, ptr, ppos):
+                        return ptr + 1
+        return ptr
+    elif maxcount != 65535:
+        # adjust end
         end1 = ptr + maxcount
         if end1 <= end:
             end = end1
     op = ctx.pat(ppos)
-    for op1, (_, fre) in unroll_char_checker:
+    for op1, fre in unroll_fre_checker:
         if op1 == op:
             return fre(ctx, ptr, end, ppos)
     raise Error("rsre.find_repetition_end[%d]" % op)
@@ -751,23 +802,60 @@
     if checkerfn == match_ANY_ALL:
         def fre(ctx, ptr, end, ppos):
             return end
+    elif checkerfn == match_IN:
+        install_jitdriver_spec('MatchIn', 
+                               greens=['ppos', 'ctx.pattern'],
+                               reds=['ptr', 'end', 'ctx'],
+                               debugprint=(1, 0))
+        @specializectx
+        def fre(ctx, ptr, end, ppos):
+            while True:
+                ctx.jitdriver_MatchIn.jit_merge_point(ctx=ctx, ptr=ptr,
+                                                      end=end, ppos=ppos)
+                if ptr < end and checkerfn(ctx, ptr, ppos):
+                    ptr += 1
+                else:
+                    return ptr
+    elif checkerfn == match_IN_IGNORE:
+        install_jitdriver_spec('MatchInIgnore', 
+                               greens=['ppos', 'ctx.pattern'],
+                               reds=['ptr', 'end', 'ctx'],
+                               debugprint=(1, 0))
+        @specializectx
+        def fre(ctx, ptr, end, ppos):
+            while True:
+                ctx.jitdriver_MatchInIgnore.jit_merge_point(ctx=ctx, ptr=ptr,
+                                                            end=end, ppos=ppos)
+                if ptr < end and checkerfn(ctx, ptr, ppos):
+                    ptr += 1
+                else:
+                    return ptr
     else:
+        # in the other cases, the fre() function is not JITted at all
+        # and is present as a residual call.
+        @specializectx
         def fre(ctx, ptr, end, ppos):
             while ptr < end and checkerfn(ctx, ptr, ppos):
                 ptr += 1
             return ptr
-    return checkerfn, fre
+    fre = func_with_new_name(fre, 'fre_' + checkerfn.__name__)
+    return fre
+
+unroll_char_checker = [
+    (OPCODE_ANY,                match_ANY),
+    (OPCODE_ANY_ALL,            match_ANY_ALL),
+    (OPCODE_IN,                 match_IN),
+    (OPCODE_IN_IGNORE,          match_IN_IGNORE),
+    (OPCODE_LITERAL,            match_LITERAL),
+    (OPCODE_LITERAL_IGNORE,     match_LITERAL_IGNORE),
+    (OPCODE_NOT_LITERAL,        match_NOT_LITERAL),
+    (OPCODE_NOT_LITERAL_IGNORE, match_NOT_LITERAL_IGNORE),
+    ]
+unroll_fre_checker = [(_op, _make_fre(_fn))
+                      for (_op, _fn) in unroll_char_checker]
 
-unroll_char_checker = unrolling_iterable([
-    (OPCODE_ANY,                _make_fre(match_ANY)),
-    (OPCODE_ANY_ALL,            _make_fre(match_ANY_ALL)),
-    (OPCODE_IN,                 _make_fre(match_IN)),
-    (OPCODE_IN_IGNORE,          _make_fre(match_IN_IGNORE)),
-    (OPCODE_LITERAL,            _make_fre(match_LITERAL)),
-    (OPCODE_LITERAL_IGNORE,     _make_fre(match_LITERAL_IGNORE)),
-    (OPCODE_NOT_LITERAL,        _make_fre(match_NOT_LITERAL)),
-    (OPCODE_NOT_LITERAL_IGNORE, _make_fre(match_NOT_LITERAL_IGNORE)),
-    ])
+unroll_char_checker = unrolling_iterable(unroll_char_checker)
+unroll_fre_checker  = unrolling_iterable(unroll_fre_checker)
 
 ##### At dispatch
 
@@ -873,74 +961,139 @@
     else:
         return None
 
+install_jitdriver('Match',
+                  greens=['ctx.pattern'], reds=['ctx'],
+                  debugprint=(0,))
+
 def match_context(ctx):
     ctx.original_pos = ctx.match_start
     if ctx.end < ctx.match_start:
         return False
+    ctx.jitdriver_Match.jit_merge_point(ctx=ctx)
     return sre_match(ctx, 0, ctx.match_start, None) is not None
 
 def search_context(ctx):
     ctx.original_pos = ctx.match_start
     if ctx.end < ctx.match_start:
         return False
-    if ctx.pat(0) == OPCODE_INFO:
-        if ctx.pat(2) & rsre_char.SRE_INFO_PREFIX and ctx.pat(5) > 1:
-            return fast_search(ctx)
-    return regular_search(ctx)
+    base = 0
+    charset = False
+    if ctx.pat(base) == OPCODE_INFO:
+        flags = ctx.pat(2)
+        if flags & rsre_char.SRE_INFO_PREFIX:
+            if ctx.pat(5) > 1:
+                return fast_search(ctx)
+        else:
+            charset = (flags & rsre_char.SRE_INFO_CHARSET)
+        base += 1 + ctx.pat(1)
+    if ctx.pat(base) == OPCODE_LITERAL:
+        return literal_search(ctx, base)
+    if charset:
+        return charset_search(ctx, base)
+    return regular_search(ctx, base)
+
+install_jitdriver('RegularSearch',
+                  greens=['base', 'ctx.pattern'],
+                  reds=['start', 'ctx'],
+                  debugprint=(1, 0))
 
-def regular_search(ctx):
+def regular_search(ctx, base):
     start = ctx.match_start
     while start <= ctx.end:
-        if sre_match(ctx, 0, start, None) is not None:
+        ctx.jitdriver_RegularSearch.jit_merge_point(ctx=ctx, start=start,
+                                                    base=base)
+        if sre_match(ctx, base, start, None) is not None:
             ctx.match_start = start
             return True
         start += 1
     return False
 
+install_jitdriver_spec("LiteralSearch",
+                       greens=['base', 'character', 'ctx.pattern'],
+                       reds=['start', 'ctx'],
+                       debugprint=(2, 0, 1))
+ at specializectx
+def literal_search(ctx, base):
+    # pattern starts with a literal character.  this is used
+    # for short prefixes, and if fast search is disabled
+    character = ctx.pat(base + 1)
+    base += 2
+    start = ctx.match_start
+    while start < ctx.end:
+        ctx.jitdriver_LiteralSearch.jit_merge_point(ctx=ctx, start=start,
+                                          base=base, character=character)
+        if ctx.str(start) == character:
+            if sre_match(ctx, base, start + 1, None) is not None:
+                ctx.match_start = start
+                return True
+        start += 1
+    return False
+
+install_jitdriver_spec("CharsetSearch",
+                       greens=['base', 'ctx.pattern'],
+                       reds=['start', 'ctx'],
+                       debugprint=(1, 0))
+ at specializectx
+def charset_search(ctx, base):
+    # pattern starts with a character from a known set
+    start = ctx.match_start
+    while start < ctx.end:
+        ctx.jitdriver_CharsetSearch.jit_merge_point(ctx=ctx, start=start,
+                                                    base=base)
+        if rsre_char.check_charset(ctx.pattern, 5, ctx.str(start)):
+            if sre_match(ctx, base, start, None) is not None:
+                ctx.match_start = start
+                return True
+        start += 1
+    return False
+
+install_jitdriver_spec('FastSearch',
+                       greens=['i', 'prefix_len', 'ctx.pattern'],
+                       reds=['string_position', 'ctx'],
+                       debugprint=(2, 0))
 @specializectx
 def fast_search(ctx):
     # skips forward in a string as fast as possible using information from
     # an optimization info block
     #  <1=skip> <2=flags> <3=min> <4=...>
     #        <5=length> <6=skip> <7=prefix data> 
-    flags = ctx.pat(2)
+    string_position = ctx.match_start
+    if string_position >= ctx.end:
+        return False
     prefix_len = ctx.pat(5)
     assert prefix_len >= 0
-    prefix_skip = ctx.pat(6)
-    assert prefix_skip >= 0
-    overlap_offset = 7 + prefix_len - 1
-    assert overlap_offset >= 0
-    pattern_offset = ctx.pat(1) + 1
-    ppos_start = pattern_offset + 2 * prefix_skip
-    assert ppos_start >= 0
     i = 0
-    string_position = ctx.match_start
-    end = ctx.end
-    while string_position < end:
-        while True:
-            char_ord = ctx.str(string_position)
-            if char_ord != ctx.pat(7 + i):
-                if i == 0:
-                    break
-                else:
-                    i = ctx.pat(overlap_offset + i)
-            else:
-                i += 1
-                if i == prefix_len:
-                    # found a potential match
-                    start = string_position + 1 - prefix_len
-                    assert start >= 0
-                    ptr = start + prefix_skip
-                    if flags & rsre_char.SRE_INFO_LITERAL:
-                        # matched all of pure literal pattern
-                        ctx.match_start = start
-                        ctx.match_end = ptr
-                        ctx.match_marks = None
-                        return True
-                    if sre_match(ctx, ppos_start, ptr, None) is not None:
-                        ctx.match_start = start
-                        return True
-                    i = ctx.pat(overlap_offset + i)
-                break
+    while True:
+        ctx.jitdriver_FastSearch.jit_merge_point(ctx=ctx,
+                string_position=string_position, i=i, prefix_len=prefix_len)
+        char_ord = ctx.str(string_position)
+        if char_ord != ctx.pat(7 + i):
+            if i > 0:
+                overlap_offset = prefix_len + (7 - 1)
+                i = ctx.pat(overlap_offset + i)
+                continue
+        else:
+            i += 1
+            if i == prefix_len:
+                # found a potential match
+                start = string_position + 1 - prefix_len
+                assert start >= 0
+                prefix_skip = ctx.pat(6)
+                ptr = start + prefix_skip
+                #flags = ctx.pat(2)
+                #if flags & rsre_char.SRE_INFO_LITERAL:
+                #    # matched all of pure literal pattern
+                #    ctx.match_start = start
+                #    ctx.match_end = ptr
+                #    ctx.match_marks = None
+                #    return True
+                pattern_offset = ctx.pat(1) + 1
+                ppos_start = pattern_offset + 2 * prefix_skip
+                if sre_match(ctx, ppos_start, ptr, None) is not None:
+                    ctx.match_start = start
+                    return True
+                overlap_offset = prefix_len + (7 - 1)
+                i = ctx.pat(overlap_offset + i)
         string_position += 1
-    return False
+        if string_position >= ctx.end:
+            return False

Modified: pypy/branch/fast-forward/pypy/rlib/rsre/test/test_match.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rsre/test/test_match.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rsre/test/test_match.py	Fri Oct 22 23:09:43 2010
@@ -1,24 +1,49 @@
-import _sre, re, sre_compile
-from pypy.rlib.rsre import rsre_core
+import re
+from pypy.rlib.rsre import rsre_core, rsre_char
 
 
-def get_code(regexp, flags=0, allargs=False):
-    class GotIt(Exception):
-        pass
-    def my_compile(pattern, flags, code, *args):
-        print code
-        raise GotIt(code, flags, args)
-    saved = _sre.compile
-    try:
-        _sre.compile = my_compile
-        try:
-            sre_compile.compile(regexp, flags)
-        except GotIt, e:
-            pass
+def get_hacked_sre_compile(my_compile):
+    """Return a copy of the sre_compile module for which the _sre
+    module is a custom module that has _sre.compile == my_compile
+    and CODESIZE == rsre_char.CODESIZE.
+    """
+    import sre_compile, __builtin__, new
+    sre_hacked = new.module("_sre_hacked")
+    sre_hacked.compile = my_compile
+    sre_hacked.MAGIC = sre_compile.MAGIC
+    sre_hacked.CODESIZE = rsre_char.CODESIZE
+    sre_hacked.getlower = rsre_char.getlower
+    def my_import(name, *args):
+        if name == '_sre':
+            return sre_hacked
         else:
-            raise ValueError("did not reach _sre.compile()!")
+            return default_import(name, *args)
+    src = sre_compile.__file__
+    if src.lower().endswith('.pyc') or src.lower().endswith('.pyo'):
+        src = src[:-1]
+    mod = new.module("sre_compile_hacked")
+    default_import = __import__
+    try:
+        __builtin__.__import__ = my_import
+        execfile(src, mod.__dict__)
     finally:
-        _sre.compile = saved
+        __builtin__.__import__ = default_import
+    return mod
+
+class GotIt(Exception):
+    pass
+def my_compile(pattern, flags, code, *args):
+    print code
+    raise GotIt(code, flags, args)
+sre_compile_hacked = get_hacked_sre_compile(my_compile)
+
+def get_code(regexp, flags=0, allargs=False):
+    try:
+        sre_compile_hacked.compile(regexp, flags)
+    except GotIt, e:
+        pass
+    else:
+        raise ValueError("did not reach _sre.compile()!")
     if allargs:
         return e.args
     else:

Modified: pypy/branch/fast-forward/pypy/rlib/rstring.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rstring.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rstring.py	Fri Oct 22 23:09:43 2010
@@ -69,6 +69,9 @@
     def build(self):
         return self.tp("").join(self.l)
 
+    def getlength(self):
+        return len(self.build())
+
 class StringBuilder(AbstractStringBuilder):
     tp = str
 
@@ -122,9 +125,12 @@
         assert isinstance(s_times, SomeInteger)
         return s_None
 
+    def method_getlength(self):
+        return SomeInteger(nonneg=True)
+
     def method_build(self):
         return SomeString()
-    
+
     def rtyper_makerepr(self, rtyper):
         return rtyper.type_system.rbuilder.stringbuilder_repr
 
@@ -144,6 +150,9 @@
         assert isinstance(s_times, SomeInteger)
         return s_None
 
+    def method_getlength(self):
+        return SomeInteger(nonneg=True)
+
     def method_build(self):
         return SomeUnicodeString()
     

Modified: pypy/branch/fast-forward/pypy/rlib/test/test_jit.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/test/test_jit.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/test/test_jit.py	Fri Oct 22 23:09:43 2010
@@ -1,10 +1,17 @@
 import py
+from pypy.conftest import option
 from pypy.rlib.jit import hint, we_are_jitted, JitDriver, purefunction_promote
-from pypy.rlib.jit import JitHintError
+from pypy.rlib.jit import JitHintError, oopspec
 from pypy.translator.translator import TranslationContext, graphof
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.rpython.lltypesystem import lltype
 
+def test_oopspec():
+    @oopspec('foobar')
+    def fn():
+        pass
+    assert fn.oopspec == 'foobar'
+    
 class BaseTestJIT(BaseRtypingTest):
     def test_hint(self):
         def f():
@@ -104,6 +111,26 @@
             return n
         py.test.raises(JitHintError, self.gengraph, fn, [int])
 
+    def test_green_field(self):
+        def get_printable_location(xfoo):
+            return str(ord(xfoo))   # xfoo must be annotated as a character
+        myjitdriver = JitDriver(greens=['x.foo'], reds=['n', 'x'],
+                                get_printable_location=get_printable_location)
+        class A(object):
+            _immutable_fields_ = ['foo']
+        def fn(n):
+            x = A()
+            x.foo = chr(n)
+            while n > 0:
+                myjitdriver.can_enter_jit(x=x, n=n)
+                myjitdriver.jit_merge_point(x=x, n=n)
+                n -= 1
+            return n
+        t = self.gengraph(fn, [int])[0]
+        if option.view:
+            t.view()
+        # assert did not raise
+
 
 class TestJITLLtype(BaseTestJIT, LLRtypeMixin):
     pass

Modified: pypy/branch/fast-forward/pypy/rlib/test/test_rdynload.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/test/test_rdynload.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/test/test_rdynload.py	Fri Oct 22 23:09:43 2010
@@ -1,15 +1,22 @@
 from pypy.rlib.rdynload import *
-from pypy.rlib.libffi import get_libc_name
+from pypy.rlib.clibffi import get_libc_name
 from pypy.rpython.lltypesystem import rffi, lltype
 import py
 
 class TestDLOperations:
     def test_dlopen(self):
-        py.test.raises(DLOpenError, "dlopen(rffi.str2charp('xxxxxxxxxxxx'))")
-        assert dlopen(rffi.str2charp(get_libc_name()))
+        s = rffi.str2charp('xxxxxxxxxxxx')
+        py.test.raises(DLOpenError, "dlopen(s)")
+        rffi.free_charp(s)
+        #
+        s = rffi.str2charp(get_libc_name())
+        assert dlopen(s)
+        rffi.free_charp(s)
 
     def test_dlsym(self):
-        lib = dlopen(rffi.str2charp(get_libc_name()))
+        s = rffi.str2charp(get_libc_name())
+        lib = dlopen(s)
+        rffi.free_charp(s)
         handle = rffi.cast(lltype.Ptr(lltype.FuncType([lltype.Signed],
                            lltype.Signed)), dlsym(lib, 'abs'))
         assert 1 == handle(1)

Modified: pypy/branch/fast-forward/pypy/rlib/test/test_rsocket.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/test/test_rsocket.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/test/test_rsocket.py	Fri Oct 22 23:09:43 2010
@@ -437,3 +437,31 @@
             foo = self.serv.accept()
         py.test.raises(SocketError, raise_error)
 
+def _test_cond_include(cond):
+    # Test that _rsocket_rffi is importable even on platforms where
+    # AF_PACKET or AF_NETLINK is not defined.
+    import re
+    from pypy.rlib import _rsocket_rffi
+    srcfile = _rsocket_rffi.__file__
+    if srcfile.lower().endswith('c') or srcfile.lower().endswith('o'):
+        srcfile = srcfile[:-1]      # .pyc => .py
+    assert srcfile.lower().endswith('.py')
+    sourcelines = open(srcfile, 'rb').read().splitlines()
+    found = False
+    for i, line in enumerate(sourcelines):
+        line2 = re.sub(r"(\s*COND_HEADER\s*=)",
+                      r"\1'#undef %s\\n'+" % cond,
+                      line)
+        if line2 != line:
+            found = True
+            sourcelines[i] = line2
+    assert found
+    d = {}
+    sourcelines.append('')
+    exec '\n'.join(sourcelines) in d
+
+def test_no_AF_PACKET():
+    _test_cond_include('AF_PACKET')
+
+def test_no_AF_NETLINK():
+    _test_cond_include('AF_NETLINK')

Modified: pypy/branch/fast-forward/pypy/rlib/test/test_rstring.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/test/test_rstring.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/test/test_rstring.py	Fri Oct 22 23:09:43 2010
@@ -29,6 +29,7 @@
     s = StringBuilder()
     s.append("a")
     s.append("abc")
+    assert s.getlength() == len('aabc')
     s.append("a")
     s.append_slice("abc", 1, 2)
     s.append_multiple_char('d', 4)
@@ -39,6 +40,7 @@
     s.append(u'a')
     s.append(u'abc')
     s.append_slice(u'abcdef', 1, 2)
+    assert s.getlength() == len('aabcb')
     s.append_multiple_char(u'd', 4)
     assert s.build() == 'aabcbdddd'
     assert isinstance(s.build(), unicode)

Modified: pypy/branch/fast-forward/pypy/rlib/test/test_rzlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/test/test_rzlib.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/test/test_rzlib.py	Fri Oct 22 23:09:43 2010
@@ -189,6 +189,8 @@
     assert unused3 == len('more_garbage')
     assert data3 == ''
 
+    rzlib.deflateEnd(stream)
+
 
 def test_decompress_max_length():
     """
@@ -205,6 +207,8 @@
     assert finished2 is True
     assert unused2 == 0
 
+    rzlib.deflateEnd(stream)
+
 
 def test_cornercases():
     """
@@ -215,6 +219,7 @@
     bytes += rzlib.compress(stream, "")
     bytes += rzlib.compress(stream, "", rzlib.Z_FINISH)
     assert zlib.decompress(bytes) == ""
+    rzlib.deflateEnd(stream)
 
     stream = rzlib.inflateInit()
     data, finished, unused = rzlib.decompress(stream, "")
@@ -228,3 +233,4 @@
         assert finished is False
         assert unused > 0
         buf = buf[-unused:]
+    rzlib.deflateEnd(stream)

Modified: pypy/branch/fast-forward/pypy/rpython/llinterp.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/llinterp.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/llinterp.py	Fri Oct 22 23:09:43 2010
@@ -48,8 +48,7 @@
 
     current_interpreter = None
 
-    def __init__(self, typer, tracing=True, exc_data_ptr=None,
-                 malloc_check=True):
+    def __init__(self, typer, tracing=True, exc_data_ptr=None):
         self.bindings = {}
         self.typer = typer
         # 'heap' is module or object that provides malloc, etc for lltype ops
@@ -57,9 +56,7 @@
         self.exc_data_ptr = exc_data_ptr
         self.frame_stack = []
         self.tracer = None
-        self.malloc_check = malloc_check
         self.frame_class = LLFrame
-        self.mallocs = {}
         if tracing:
             self.tracer = Tracer()
 
@@ -163,24 +160,6 @@
             return self.exc_data_ptr
         return None
 
-    def remember_malloc(self, ptr, llframe):
-        # err....
-        self.mallocs[ptr._obj] = llframe
-
-    def remember_free(self, ptr):
-        try:
-            del self.mallocs[ptr._obj]
-        except KeyError:
-            self._rehash_mallocs()
-            del self.mallocs[ptr._obj]
-
-    def _rehash_mallocs(self):
-        # rehashing is needed because some objects' hash may change
-        # when being turned to 
-        items = self.mallocs.items()
-        self.mallocs = {}
-        self.mallocs.update(items)
-
     def _store_exception(self, exc):
         raise PleaseOverwriteStoreException("You just invoked ll2ctypes callback without overwriting _store_exception on llinterpreter")
 
@@ -726,23 +705,23 @@
     def op_malloc(self, obj, flags):
         flavor = flags['flavor']
         zero = flags.get('zero', False)
+        track_allocation = flags.get('track_allocation', True)
         if flavor == "stack":
             result = self.heap.malloc(obj, zero=zero, flavor='raw')
             self.alloca_objects.append(result)
             return result
-        ptr = self.heap.malloc(obj, zero=zero, flavor=flavor)
-        if flavor == 'raw' and self.llinterpreter.malloc_check:
-            self.llinterpreter.remember_malloc(ptr, self)
+        ptr = self.heap.malloc(obj, zero=zero, flavor=flavor,
+                               track_allocation=track_allocation)
         return ptr
 
     def op_malloc_varsize(self, obj, flags, size):
         flavor = flags['flavor']
         zero = flags.get('zero', False)
+        track_allocation = flags.get('track_allocation', True)
         assert flavor in ('gc', 'raw')
         try:
-            ptr = self.heap.malloc(obj, size, zero=zero, flavor=flavor)
-            if flavor == 'raw' and self.llinterpreter.malloc_check:
-                self.llinterpreter.remember_malloc(ptr, self)
+            ptr = self.heap.malloc(obj, size, zero=zero, flavor=flavor,
+                                   track_allocation=track_allocation)
             return ptr
         except MemoryError:
             self.make_llexception()
@@ -759,11 +738,10 @@
         zero = flags.get('zero', False)
         return self.heap.malloc_nonmovable(TYPE, size, zero=zero)
 
-    def op_free(self, obj, flavor):
-        assert isinstance(flavor, str)
-        if flavor == 'raw' and self.llinterpreter.malloc_check:
-            self.llinterpreter.remember_free(obj)
-        self.heap.free(obj, flavor=flavor)
+    def op_free(self, obj, flags):
+        assert flags['flavor'] == 'raw'
+        track_allocation = flags.get('track_allocation', True)
+        self.heap.free(obj, flavor='raw', track_allocation=track_allocation)
 
     def op_shrink_array(self, obj, smallersize):
         return self.heap.shrink_array(obj, smallersize)
@@ -1037,6 +1015,13 @@
     def op_stack_malloc(self, size): # mmh
         raise NotImplementedError("backend only")
 
+    def op_track_alloc_start(self, addr):
+        # we don't do tracking at this level
+        checkadr(addr)
+
+    def op_track_alloc_stop(self, addr):
+        checkadr(addr)
+
     # ____________________________________________________________
     # Overflow-detecting variants
 

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/ll2ctypes.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/ll2ctypes.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/ll2ctypes.py	Fri Oct 22 23:09:43 2010
@@ -72,7 +72,7 @@
                 PIECESIZE = 0x08000000
         PIECES = 10
         m = rmmap.mmap(-1, PIECES * PIECESIZE,
-                       rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS,
+                       rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS|rmmap.MAP_NORESERVE,
                        rmmap.PROT_READ|rmmap.PROT_WRITE)
         m.close = lambda : None    # leak instead of giving a spurious
                                    # error at CPython's shutdown
@@ -827,6 +827,8 @@
         except (ValueError, OverflowError):
             for tc in 'HIL':
                 if array(tc).itemsize == array('u').itemsize:
+                    import struct
+                    cobj &= 256 ** struct.calcsize(tc) - 1
                     llobj = array('u', array(tc, (cobj,)).tostring())[0]
                     break
             else:

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/llarena.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/llarena.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/llarena.py	Fri Oct 22 23:09:43 2010
@@ -440,7 +440,7 @@
                                    [rffi.INT],
                                    rffi.INT,
                                    sandboxsafe=True, _nowrapper=True)
-    _dev_zero = rffi.str2charp('/dev/zero')   # prebuilt
+    _dev_zero = rffi.str2charp_immortal('/dev/zero')   # prebuilt
 
     def clear_large_memory_chunk(baseaddr, size):
         # on some Unixy platforms, reading from /dev/zero is the fastest way

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/llmemory.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/llmemory.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/llmemory.py	Fri Oct 22 23:09:43 2010
@@ -105,11 +105,13 @@
         if (isinstance(self.TYPE, lltype.ContainerType)
             and self.TYPE._gckind == 'gc'):
             assert self.repeat == 1
-            p = lltype.malloc(self.TYPE, flavor='raw', zero=zero)
+            p = lltype.malloc(self.TYPE, flavor='raw', zero=zero,
+                              track_allocation=False)
             return cast_ptr_to_adr(p)
         else:
             T = lltype.FixedSizeArray(self.TYPE, self.repeat)
-            p = lltype.malloc(T, flavor='raw', zero=zero)
+            p = lltype.malloc(T, flavor='raw', zero=zero,
+                              track_allocation=False)
             array_adr = cast_ptr_to_adr(p)
             return array_adr + ArrayItemsOffset(T)
 
@@ -288,7 +290,8 @@
             count = 0
         p = lltype.malloc(parenttype or self.TYPE, count,
                           immortal = self.TYPE._gckind == 'raw',
-                          zero = zero)
+                          zero = zero,
+                          track_allocation = False)
         return cast_ptr_to_adr(p)
 
     def raw_memcopy(self, srcadr, dstadr):

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/lloperation.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/lloperation.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/lloperation.py	Fri Oct 22 23:09:43 2010
@@ -405,6 +405,8 @@
     'raw_load':             LLOp(sideeffects=False),
     'raw_store':            LLOp(),
     'stack_malloc':         LLOp(), # mmh
+    'track_alloc_start':    LLOp(),
+    'track_alloc_stop':     LLOp(),
     'adr_add':              LLOp(canfold=True),
     'adr_sub':              LLOp(canfold=True),
     'adr_delta':            LLOp(canfold=True),

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/lltype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/lltype.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/lltype.py	Fri Oct 22 23:09:43 2010
@@ -1,7 +1,3 @@
-import StringIO
-import traceback
-import sys
-
 import py
 from pypy.rlib.rarithmetic import (r_int, r_uint, intmask, r_singlefloat,
                                    r_ulonglong, r_longlong, r_longfloat,
@@ -10,25 +6,13 @@
 from pypy.tool.uid import Hashable
 from pypy.tool.tls import tlsobject
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 from types import NoneType
 from sys import maxint
 import weakref
 
 TLS = tlsobject()
 
-# Track allocations to detect memory leaks
-# Don't track 'gc' and immortal mallocs
-TRACK_ALLOCATIONS = False
-ALLOCATED = identity_dict()
-
-def start_tracking_allocations():
-    global TRACK_ALLOCATIONS
-    TRACK_ALLOCATIONS = True
-    ALLOCATED.clear()
-
-def stop_tracking_allocations():
-    global TRACK_ALLOCATIONS
-    TRACK_ALLOCATIONS = False
 
 class _uninitialized(object):
     def __init__(self, TYPE):
@@ -801,6 +785,8 @@
             return llmemory.cast_adr_to_ptr(value, TGT)
     elif TGT == llmemory.Address and isinstance(ORIG, Ptr):
         return llmemory.cast_ptr_to_adr(value)
+    elif TGT == Signed and isinstance(ORIG, Ptr) and ORIG.TO._gckind == 'raw':
+        return llmemory.cast_adr_to_int(llmemory.cast_ptr_to_adr(value), 'symbolic')
     raise TypeError("don't know how to cast from %r to %r" % (ORIG, TGT))
 
 
@@ -1387,41 +1373,21 @@
     __slots__ = ('_TYPE',
                  '_parent_type', '_parent_index', '_keepparent',
                  '_wrparent',
-                 '__weakref__', '_traceback',
-                 '__storage')
+                 '__weakref__',
+                 '_storage')
 
-    def __init__(self, TYPE, track_allocation=None):
+    def __init__(self, TYPE):
         self._wrparent = None
         self._TYPE = TYPE
         self._storage = True    # means "use default storage", as opposed to:
                                 #    None            - container was freed
                                 #     - using ctypes
                                 #                      (see ll2ctypes.py)
-        if track_allocation is not False and TRACK_ALLOCATIONS:
-            self._traceback = self._get_traceback()
-            ALLOCATED[self] = None
-        else:
-            self._traceback = None
-
-    def _get_traceback(self):
-        frame = sys._getframe().f_back.f_back.f_back.f_back
-        sio = StringIO.StringIO()
-        traceback.print_stack(frame, file=sio)
-        return sio.getvalue()
 
     def _free(self):
         self._check()   # no double-frees
         self._storage = None
 
-    def _storage_get(self):
-        return self.__storage
-
-    def _storage_set(self, value):
-        self.__storage = value
-        if value is not True and self in ALLOCATED:
-            del ALLOCATED[self]
-    _storage = property(_storage_get, _storage_set)
-
     def _was_freed(self):
         if self._storage is None:
             return True
@@ -1500,12 +1466,12 @@
 
     __slots__ = ('_hash_cache_', '_compilation_info')
 
-    def __new__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None, track_allocation=None):
+    def __new__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None):
         my_variety = _struct_variety(TYPE._names)
         return object.__new__(my_variety)
 
-    def __init__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None, track_allocation=None):
-        _parentable.__init__(self, TYPE, track_allocation)
+    def __init__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None):
+        _parentable.__init__(self, TYPE)
         if n is not None and TYPE._arrayfld is None:
             raise TypeError("%r is not variable-sized" % (TYPE,))
         if n is None and TYPE._arrayfld is not None:
@@ -1513,8 +1479,7 @@
         first, FIRSTTYPE = TYPE._first_struct()
         for fld, typ in TYPE._flds.items():
             if fld == TYPE._arrayfld:
-                value = _array(typ, n, initialization=initialization, parent=self, parentindex=fld,
-                               track_allocation=track_allocation)
+                value = _array(typ, n, initialization=initialization, parent=self, parentindex=fld)
             else:
                 value = typ._allocate(initialization=initialization, parent=self, parentindex=fld)
             setattr(self, fld, value)
@@ -1575,12 +1540,12 @@
 
     __slots__ = ('items',)
 
-    def __init__(self, TYPE, n, initialization=None, parent=None, parentindex=None, track_allocation=None):
+    def __init__(self, TYPE, n, initialization=None, parent=None, parentindex=None):
         if not isinstance(n, int):
             raise TypeError, "array length must be an int"
         if n < 0:
             raise ValueError, "negative array length"
-        _parentable.__init__(self, TYPE, track_allocation)
+        _parentable.__init__(self, TYPE)
         try:
             myrange = range(n)
         except OverflowError:
@@ -1647,7 +1612,7 @@
     _cache = weakref.WeakKeyDictionary()  # parentarray -> {subarrays}
 
     def __init__(self, TYPE, parent, baseoffset_or_fieldname):
-        _parentable.__init__(self, TYPE, track_allocation=False)
+        _parentable.__init__(self, TYPE)
         self._setparentstructure(parent, baseoffset_or_fieldname)
         # Keep the parent array alive, we share the same allocation.
         # Don't do it if we are inside a GC object, though -- it's someone
@@ -1655,6 +1620,13 @@
         if typeOf(top_container(parent))._gckind == 'raw':
             self._keepparent = parent
 
+    def __str__(self):
+        parent = self._wrparent()
+        if parent is None:
+            return '_subarray at %s in already freed' % (self._parent_index,)
+        return '_subarray at %r in %s' % (self._parent_index,
+                                          parent._TYPE)
+
     def __repr__(self):
         parent = self._wrparent()
         if parent is None:
@@ -1868,8 +1840,9 @@
         return id(self.value)
 
 
-def malloc(T, n=None, flavor='gc', immortal=False, zero=False):
-    assert flavor != 'cpy'
+def malloc(T, n=None, flavor='gc', immortal=False, zero=False,
+           track_allocation=True):
+    assert flavor in ('gc', 'raw')
     if zero or immortal:
         initialization = 'example'
     elif flavor == 'raw':
@@ -1877,9 +1850,9 @@
     else:
         initialization = 'malloc'
     if isinstance(T, Struct):
-        o = _struct(T, n, initialization=initialization, track_allocation=flavor == "raw" and not immortal)
+        o = _struct(T, n, initialization=initialization)
     elif isinstance(T, Array):
-        o = _array(T, n, initialization=initialization, track_allocation=flavor == "raw" and not immortal)
+        o = _array(T, n, initialization=initialization)
     elif isinstance(T, OpaqueType):
         assert n is None
         o = _opaque(T, initialization=initialization)
@@ -1887,17 +1860,50 @@
         raise TypeError, "malloc for Structs and Arrays only"
     if T._gckind != 'gc' and not immortal and flavor.startswith('gc'):
         raise TypeError, "gc flavor malloc of a non-GC non-immortal structure"
+    if flavor == "raw" and not immortal and track_allocation:
+        leakfinder.remember_malloc(o, framedepth=2)
     solid = immortal or not flavor.startswith('gc') # immortal or non-gc case
     return _ptr(Ptr(T), o, solid)
 
-def free(p, flavor):
+def free(p, flavor, track_allocation=True):
     if flavor.startswith('gc'):
         raise TypeError, "gc flavor free"
     T = typeOf(p)
     if not isinstance(T, Ptr) or p._togckind() != 'raw':
         raise TypeError, "free(): only for pointers to non-gc containers"
+    if track_allocation:
+        leakfinder.remember_free(p._obj0)
     p._obj0._free()
 
+def _make_scoped_allocator(T):
+    class ScopedAlloc:
+        def __init__(self, n=None, zero=False):
+            if n is None:
+                self.buf = malloc(T, flavor='raw', zero=zero)
+            else:
+                self.buf = malloc(T, n, flavor='raw', zero=zero)
+
+        def __enter__(self):
+            return self.buf
+
+        def __exit__(self, *args):
+            free(self.buf, flavor='raw')
+
+    ScopedAlloc.__name__ = 'ScopedAlloc_%s' % (T,)
+    return ScopedAlloc
+_make_scoped_allocator._annspecialcase_ = 'specialize:memo'
+
+def scoped_alloc(T, n=None, zero=False):
+    """Returns a context manager which handles allocation and
+    deallocation of temporary memory. Use it in a with statement::
+
+        with scoped_alloc(Array(Signed), 1) as array:
+            ...use array...
+        ...it's freed now.
+    """
+    return _make_scoped_allocator(T)(n=n, zero=zero)
+scoped_alloc._annspecialcase_ = 'specialize:arg(0)'
+
 def functionptr(TYPE, name, **attrs):
     if not isinstance(TYPE, FuncType):
         raise TypeError, "functionptr() for FuncTypes only"

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/rbuilder.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/rbuilder.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/rbuilder.py	Fri Oct 22 23:09:43 2010
@@ -100,6 +100,10 @@
         ll_builder.used = used
 
     @staticmethod
+    def ll_getlength(ll_builder):
+        return ll_builder.used
+
+    @staticmethod
     def ll_build(ll_builder):
         final_size = ll_builder.used
         assert final_size >= 0

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/rclass.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/rclass.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/rclass.py	Fri Oct 22 23:09:43 2010
@@ -329,16 +329,33 @@
             fields['__class__'] = 'typeptr', get_type_repr(self.rtyper)
         else:
             # instance attributes
-            if llfields is None:
-                llfields = []
             attrs = self.classdef.attrs.items()
             attrs.sort()
+            myllfields = []
             for name, attrdef in attrs:
                 if not attrdef.readonly:
                     r = self.rtyper.getrepr(attrdef.s_value)
                     mangled_name = 'inst_' + name
                     fields[name] = mangled_name, r
-                    llfields.append((mangled_name, r.lowleveltype))
+                    myllfields.append((mangled_name, r.lowleveltype))
+
+            # Sort the instance attributes by decreasing "likely size",
+            # as reported by rffi.sizeof(), to minimize padding holes in C.
+            # Fields of the same size are sorted by name (by attrs.sort()
+            # above) just to minimize randomness.
+            def keysize((_, T)):
+                if T is lltype.Void:
+                    return None
+                from pypy.rpython.lltypesystem.rffi import sizeof
+                try:
+                    return -sizeof(T)
+                except StandardError:
+                    return None
+            myllfields.sort(key = keysize)
+            if llfields is None:
+                llfields = myllfields
+            else:
+                llfields = llfields + myllfields
 
             self.rbase = getinstancerepr(self.rtyper, self.classdef.basedef,
                                          self.gcflavor)
@@ -403,7 +420,7 @@
         return cast_pointer(self.lowleveltype, result)
 
     def create_instance(self):
-        return malloc(self.object_type, flavor=self.gcflavor)
+        return malloc(self.object_type, flavor=self.gcflavor, immortal=True)
 
     def initialize_prebuilt_data(self, value, classdef, result):
         if self.classdef is not None:

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/rffi.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/rffi.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/rffi.py	Fri Oct 22 23:09:43 2010
@@ -609,6 +609,15 @@
         return array
     str2charp._annenforceargs_ = [strtype]
 
+    def str2charp_immortal(s):
+        "NOT_RPYTHON"
+        array = lltype.malloc(TYPEP.TO, len(s) + 1, flavor='raw',
+                              immortal=True)
+        for i in range(len(s)):
+            array[i] = s[i]
+        array[len(s)] = lastchar
+        return array
+
     def free_charp(cp):
         lltype.free(cp, flavor='raw')
 
@@ -646,10 +655,18 @@
         """
         Either free a non-moving buffer or keep the original storage alive.
         """
-        if rgc.can_move(data):
+        # We cannot rely on rgc.can_move(data) here, because its result
+        # might have changed since get_nonmovingbuffer().  Instead we check
+        # if 'buf' points inside 'data'.  This is only possible if we
+        # followed the 2nd case in get_nonmovingbuffer(); in the first case,
+        # 'buf' points to its own raw-malloced memory.
+        data = llstrtype(data)
+        data_start = cast_ptr_to_adr(data) + \
+            offsetof(STRTYPE, 'chars') + itemoffsetof(STRTYPE.chars, 0)
+        followed_2nd_path = (buf == cast(TYPEP, data_start))
+        keepalive_until_here(data)
+        if not followed_2nd_path:
             lltype.free(buf, flavor='raw')
-        else:
-            keepalive_until_here(data)
 
     # int -> (char*, str)
     def alloc_buffer(count):
@@ -719,19 +736,19 @@
         l = [cp[i] for i in range(size)]
         return emptystr.join(l)
 
-    return (str2charp, free_charp, charp2str,
+    return (str2charp, str2charp_immortal, free_charp, charp2str,
             get_nonmovingbuffer, free_nonmovingbuffer,
             alloc_buffer, str_from_buffer, keep_buffer_alive_until_here,
             charp2strn, charpsize2str,
             )
 
-(str2charp, free_charp, charp2str,
+(str2charp, str2charp_immortal, free_charp, charp2str,
  get_nonmovingbuffer, free_nonmovingbuffer,
  alloc_buffer, str_from_buffer, keep_buffer_alive_until_here,
  charp2strn, charpsize2str,
  ) = make_string_mappings(str)
 
-(unicode2wcharp, free_wcharp, wcharp2unicode,
+(unicode2wcharp, unicode2wcharp_immortal, free_wcharp, wcharp2unicode,
  get_nonmoving_unicodebuffer, free_nonmoving_unicodebuffer,
  alloc_unicodebuffer, unicode_from_buffer, keep_unicodebuffer_alive_until_here,
  wcharp2unicoden, wcharpsize2unicode,
@@ -918,3 +935,11 @@
     """
     return cast(lltype.Signed, getattr(pdst, fieldname))
 getintfield._annspecialcase_ = 'specialize:ll_and_arg(1)'
+
+class scoped_str2charp:
+    def __init__(self, value):
+        self.buf = str2charp(value)
+    def __enter__(self):
+        return self.buf
+    def __exit__(self, *args):
+        free_charp(self.buf)

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_ll2ctypes.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_ll2ctypes.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_ll2ctypes.py	Fri Oct 22 23:09:43 2010
@@ -765,6 +765,7 @@
         assert abs(float(b[1]) - 1.1) < 1E-6
         assert isinstance(b[2], rffi.r_singlefloat)
         assert abs(float(b[2]) - 2.2) < 1E-6
+        lltype.free(a, flavor='raw')
 
     def test_different_signatures(self):
         if sys.platform=='win32':
@@ -879,6 +880,7 @@
         qsort(rffi.cast(rffi.VOIDP, a), 5, rffi.sizeof(rffi.INT), compare)
         for i in range(5):
             assert a[i] == i + 1
+        lltype.free(a, flavor='raw')
 
     def test_array_type_bug(self):
         A = lltype.Array(lltype.Signed)

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_llmemory.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_llmemory.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_llmemory.py	Fri Oct 22 23:09:43 2010
@@ -324,12 +324,14 @@
     p_t = lltype.malloc(T)
     assert p_t.s == lltype.nullptr(S)
     # raw malloc does not
-    p_raw_t = lltype.malloc(T, flavor="raw")
-    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.s")
+    U = lltype.Struct("U", ('x', lltype.Signed))
+    p_raw_t = lltype.malloc(U, flavor="raw")
+    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.x")
+    lltype.free(p_raw_t, flavor="raw")
     # this sort of raw_malloc too
-    p_raw_t = cast_adr_to_ptr(raw_malloc(sizeof(T)), lltype.Ptr(T))
-    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.s")
-    
+    p_raw_t = cast_adr_to_ptr(raw_malloc(sizeof(U)), lltype.Ptr(U))
+    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.x")
+
 
 def test_raw_malloc_signed_bunch():
     adr = raw_malloc(sizeof(lltype.Signed) * 50)
@@ -601,7 +603,8 @@
     a = lltype.malloc(A, flavor='raw')
     src = cast_ptr_to_adr(a) + itemoffsetof(A, 0)
     raw_memclear(src, sizeof(lltype.Signed) * 0)
-    
+    lltype.free(a, flavor="raw")
+
 def test_nonneg():
     S1 = lltype.GcStruct('S1', ('x', lltype.Float))
     A1 = lltype.GcArray(lltype.Float)

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_lltype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_lltype.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_lltype.py	Fri Oct 22 23:09:43 2010
@@ -1,7 +1,9 @@
+from __future__ import with_statement
 import py
 from pypy.rpython.lltypesystem.lltype import *
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 
 def isweak(p, T):
     try:
@@ -804,22 +806,20 @@
 
 
 class TestTrackAllocation:
-    def setup_method(self, func):
-        start_tracking_allocations()
-
-    def teardown_method(self, func):
-        assert not lltype.ALLOCATED, "Memory was not correctly freed"
-        stop_tracking_allocations()
+    def test_automatic_tracking(self):
+        # calls to start_tracking_allocations/stop_tracking_allocations
+        # should occur automatically from pypy/conftest.py.  Check that.
+        assert leakfinder.TRACK_ALLOCATIONS
 
     def test_track_allocation(self):
         """A malloc'd buffer fills the ALLOCATED dictionary"""
-        assert lltype.TRACK_ALLOCATIONS
-        assert not lltype.ALLOCATED
+        assert leakfinder.TRACK_ALLOCATIONS
+        assert not leakfinder.ALLOCATED
         buf = malloc(Array(Signed), 1, flavor="raw")
-        assert len(lltype.ALLOCATED) == 1
-        assert lltype.ALLOCATED.keys() == [buf._obj]
+        assert len(leakfinder.ALLOCATED) == 1
+        assert leakfinder.ALLOCATED.keys() == [buf._obj]
         free(buf, flavor="raw")
-        assert not lltype.ALLOCATED
+        assert not leakfinder.ALLOCATED
 
     def test_str_from_buffer(self):
         """gc-managed memory does not need to be freed"""
@@ -828,16 +828,28 @@
         for i in range(size): raw_buf[i] = 'a'
         rstr = rffi.str_from_buffer(raw_buf, gc_buf, size, size)
         rffi.keep_buffer_alive_until_here(raw_buf, gc_buf)
-        assert not lltype.ALLOCATED
+        assert not leakfinder.ALLOCATED
 
     def test_leak_traceback(self):
         """Test info stored for allocated items"""
         buf = malloc(Array(Signed), 1, flavor="raw")
-        traceback = lltype.ALLOCATED.keys()[0]._traceback
+        traceback = leakfinder.ALLOCATED.values()[0]
         lines = traceback.splitlines()
         assert 'malloc(' in lines[-1] and 'flavor="raw")' in lines[-1]
 
-        # XXX The traceback should not be too long
+        # The traceback should not be too long
         print traceback
 
         free(buf, flavor="raw")
+
+    def test_no_tracking(self):
+        p1 = malloc(Array(Signed), 1, flavor='raw', track_allocation=False)
+        p2 = malloc(Array(Signed), 1, flavor='raw', track_allocation=False)
+        free(p2, flavor='raw', track_allocation=False)
+        # p1 is not freed
+
+    def test_scoped_allocator(self):
+        with scoped_alloc(Array(Signed), 1) as array:
+            array[0] = -42
+            x = array[0]
+        assert x == -42

Modified: pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_rffi.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_rffi.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/lltypesystem/test/test_rffi.py	Fri Oct 22 23:09:43 2010
@@ -9,7 +9,7 @@
 from pypy.rpython.lltypesystem.rstr import STR
 from pypy.rpython.lltypesystem import lltype
 from pypy.tool.udir import udir
-from pypy.rpython.test.test_llinterp import interpret, MallocMismatch
+from pypy.rpython.test.test_llinterp import interpret
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.annotation.annrpython import RPythonAnnotator
 from pypy.rpython.rtyper import RPythonTyper
@@ -787,3 +787,10 @@
     mixann.getgraph(f2, [], s_None)
     mixann.finish()
 
+def test_force_cast_unichar():
+    x = cast(lltype.UniChar, -1)
+    assert isinstance(x, unicode)
+    if sys.maxunicode == 65535:
+        assert cast(LONG, x) == 65535
+    else:
+        assert cast(LONG, cast(INT, x)) == -1

Modified: pypy/branch/fast-forward/pypy/rpython/memory/gc/minimarkpage.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/memory/gc/minimarkpage.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/memory/gc/minimarkpage.py	Fri Oct 22 23:09:43 2010
@@ -100,11 +100,14 @@
         # allocation of the given size.
         length = small_request_threshold / WORD + 1
         self.page_for_size = lltype.malloc(rffi.CArray(PAGE_PTR), length,
-                                           flavor='raw', zero=True)
+                                           flavor='raw', zero=True,
+                                           immortal=True)
         self.full_page_for_size = lltype.malloc(rffi.CArray(PAGE_PTR), length,
-                                                flavor='raw', zero=True)
+                                                flavor='raw', zero=True,
+                                                immortal=True)
         self.nblocks_for_size = lltype.malloc(rffi.CArray(lltype.Signed),
-                                              length, flavor='raw')
+                                              length, flavor='raw',
+                                              immortal=True)
         self.hdrsize = llmemory.raw_malloc_usage(llmemory.sizeof(PAGE_HEADER))
         assert page_size > self.hdrsize
         self.nblocks_for_size[0] = 0    # unused
@@ -114,11 +117,13 @@
         self.max_pages_per_arena = arena_size // page_size
         self.arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR),
                                           self.max_pages_per_arena,
-                                          flavor='raw', zero=True)
+                                          flavor='raw', zero=True,
+                                          immortal=True)
         # this is used in mass_free() only
         self.old_arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR),
                                               self.max_pages_per_arena,
-                                              flavor='raw', zero=True)
+                                              flavor='raw', zero=True,
+                                              immortal=True)
         #
         # the arena currently consumed; it must have at least one page
         # available, or be NULL.  The arena object that we point to is
@@ -281,7 +286,7 @@
         npages = (arena_end - firstpage) // self.page_size
         #
         # Allocate an ARENA object and initialize it
-        arena = lltype.malloc(ARENA, flavor='raw')
+        arena = lltype.malloc(ARENA, flavor='raw', track_allocation=False)
         arena.base = arena_base
         arena.nfreepages = 0        # they are all uninitialized pages
         arena.totalpages = npages
@@ -332,7 +337,7 @@
                     #
                     # The whole arena is empty.  Free it.
                     llarena.arena_free(arena.base)
-                    lltype.free(arena, flavor='raw')
+                    lltype.free(arena, flavor='raw', track_allocation=False)
                     #
                 else:
                     # Insert 'arena' in the correct arenas_lists[n]

Modified: pypy/branch/fast-forward/pypy/rpython/memory/gctransform/asmgcroot.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/memory/gctransform/asmgcroot.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/memory/gctransform/asmgcroot.py	Fri Oct 22 23:09:43 2010
@@ -75,7 +75,8 @@
             key = (TYPE, num)
             if key not in sradict:
                 CONTAINER = lltype.FixedSizeArray(TYPE, 1)
-                p = lltype.malloc(CONTAINER, flavor='raw', zero=True)
+                p = lltype.malloc(CONTAINER, flavor='raw', zero=True,
+                                  immortal=True)
                 sradict[key] = Constant(p, lltype.Ptr(CONTAINER))
             sra.append(sradict[key])
         #

Modified: pypy/branch/fast-forward/pypy/rpython/memory/gctransform/transform.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/memory/gctransform/transform.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/memory/gctransform/transform.py	Fri Oct 22 23:09:43 2010
@@ -430,7 +430,8 @@
         return self.parenttransformer.gct_malloc_varsize(hop)
     
     def gct_free(self, hop):
-        flavor = hop.spaceop.args[1].value
+        flags = hop.spaceop.args[1].value
+        flavor = flags['flavor']
         assert flavor == 'raw'
         return self.parenttransformer.gct_free(hop)
 
@@ -532,6 +533,8 @@
                           resulttype=llmemory.Address)
         if flags.get('zero'):
             hop.genop("raw_memclear", [v_raw, c_size])
+        if flags.get('track_allocation', True):
+            hop.genop("track_alloc_start", [v_raw])
         return v_raw
 
     def gct_fv_stack_malloc(self, hop, flags, TYPE, c_size):
@@ -602,15 +605,20 @@
                                [self.raw_malloc_varsize_ptr, v_length,
                                 c_const_size, c_item_size, c_offset_to_length],
                                resulttype=llmemory.Address)
+        if flags.get('track_allocation', True):
+            hop.genop("track_alloc_start", [v_raw])
         return v_raw
 
     def gct_free(self, hop):
         op = hop.spaceop
-        flavor = op.args[1].value
+        flags = op.args[1].value
+        flavor = flags['flavor']
         v = op.args[0]
         assert flavor != 'cpy', "cannot free CPython objects directly"
         if flavor == 'raw':
             v = hop.genop("cast_ptr_to_adr", [v], resulttype=llmemory.Address)
+            if flags.get('track_allocation', True):
+                hop.genop("track_alloc_stop", [v])
             hop.genop('raw_free', [v])
         else:
             assert False, "%s has no support for free with flavor %r" % (self, flavor)           

Modified: pypy/branch/fast-forward/pypy/rpython/memory/gcwrapper.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/memory/gcwrapper.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/memory/gcwrapper.py	Fri Oct 22 23:09:43 2010
@@ -42,7 +42,8 @@
     #
     # Interface for the llinterp
     #
-    def malloc(self, TYPE, n=None, flavor='gc', zero=False):
+    def malloc(self, TYPE, n=None, flavor='gc', zero=False,
+               track_allocation=True):
         if flavor == 'gc':
             typeid = self.get_type_id(TYPE)
             addr = self.gc.malloc(typeid, n, zero=zero)
@@ -51,7 +52,8 @@
                 gctypelayout.zero_gc_pointers(result)
             return result
         else:
-            return lltype.malloc(TYPE, n, flavor=flavor, zero=zero)
+            return lltype.malloc(TYPE, n, flavor=flavor, zero=zero,
+                                 track_allocation=track_allocation)
 
     def malloc_nonmovable(self, TYPE, n=None, zero=False):
         typeid = self.get_type_id(TYPE)
@@ -69,9 +71,10 @@
             return self.gc.shrink_array(addr, smallersize)
         return False
 
-    def free(self, TYPE, flavor='gc'):
+    def free(self, TYPE, flavor='gc', track_allocation=True):
         assert flavor != 'gc'
-        return lltype.free(TYPE, flavor=flavor)
+        return lltype.free(TYPE, flavor=flavor,
+                           track_allocation=track_allocation)
 
     def setfield(self, obj, fieldname, fieldvalue):
         STRUCT = lltype.typeOf(obj).TO

Modified: pypy/branch/fast-forward/pypy/rpython/memory/support.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/memory/support.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/memory/support.py	Fri Oct 22 23:09:43 2010
@@ -30,7 +30,8 @@
                 # we zero-initialize the chunks to make the translation
                 # backends happy, but we don't need to do it at run-time.
                 zero = not we_are_translated()
-                return lltype.malloc(CHUNK, flavor="raw", zero=zero)
+                return lltype.malloc(CHUNK, flavor="raw", zero=zero,
+                                     track_allocation=False)
                 
             result = self.free_list
             self.free_list = result.next
@@ -44,7 +45,7 @@
                 # Don't cache the old chunks but free them immediately.
                 # Helps debugging, and avoids that old chunks full of
                 # addresses left behind by a test end up in genc...
-                lltype.free(chunk, flavor="raw")
+                lltype.free(chunk, flavor="raw", track_allocation=False)
 
     unused_chunks = FreeList()
     cache[chunk_size] = unused_chunks, null_chunk

Modified: pypy/branch/fast-forward/pypy/rpython/memory/test/test_gc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/memory/test/test_gc.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/memory/test/test_gc.py	Fri Oct 22 23:09:43 2010
@@ -368,6 +368,14 @@
         res = self.interpret(f, [4, 42])
         assert res == 12
 
+    def test_print_leak(self):
+        def f(n):
+            for i in range(n):
+                print i
+            return 42
+        res = self.interpret(f, [10])
+        assert res == 42
+
     def test_weakref_across_minor_collection(self):
         import weakref
         class A:

Modified: pypy/branch/fast-forward/pypy/rpython/module/ll_time.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/module/ll_time.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/module/ll_time.py	Fri Oct 22 23:09:43 2010
@@ -108,7 +108,7 @@
 
                 errcode = -1
                 if self.GETTIMEOFDAY_NO_TZ:
-                    errcode = g_gettimeofday(t)
+                    errcode = c_gettimeofday(t)
                 else:
                     errcode = c_gettimeofday(t, void)
 

Modified: pypy/branch/fast-forward/pypy/rpython/rbuilder.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/rbuilder.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/rbuilder.py	Fri Oct 22 23:09:43 2010
@@ -36,8 +36,12 @@
         hop.exception_cannot_occur()
         return hop.gendirectcall(self.ll_append_multiple_char, *vlist)
 
+    def rtype_method_getlength(self, hop):
+        vlist = hop.inputargs(self)
+        hop.exception_cannot_occur()
+        return hop.gendirectcall(self.ll_getlength, *vlist)
+
     def rtype_method_build(self, hop):
         vlist = hop.inputargs(self)
         hop.exception_cannot_occur()
         return hop.gendirectcall(self.ll_build, *vlist)
-

Modified: pypy/branch/fast-forward/pypy/rpython/rbuiltin.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/rbuiltin.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/rbuiltin.py	Fri Oct 22 23:09:43 2010
@@ -345,17 +345,22 @@
 BUILTIN_TYPER[object.__init__] = rtype_object__init__
 # annotation of low-level types
 
-def rtype_malloc(hop, i_flavor=None, i_zero=None):
+def rtype_malloc(hop, i_flavor=None, i_zero=None, i_track_allocation=None):
     assert hop.args_s[0].is_constant()
     vlist = [hop.inputarg(lltype.Void, arg=0)]
     opname = 'malloc'
-    v_flavor, v_zero = parse_kwds(hop, (i_flavor, lltype.Void), (i_zero, None))
+    v_flavor, v_zero, v_track_allocation = parse_kwds(hop,
+        (i_flavor, lltype.Void),
+        (i_zero, None),
+        (i_track_allocation, None))
 
     flags = {'flavor': 'gc'}
     if v_flavor is not None:
         flags['flavor'] = v_flavor.value
     if i_zero is not None:
         flags['zero'] = v_zero.value
+    if i_track_allocation is not None:
+        flags['track_allocation'] = v_track_allocation.value
     vlist.append(hop.inputconst(lltype.Void, flags))
         
     if hop.nb_args == 2:
@@ -366,10 +371,19 @@
     hop.exception_is_here()
     return hop.genop(opname, vlist, resulttype = hop.r_result.lowleveltype)
 
-def rtype_free(hop, i_flavor):
-    assert i_flavor == 1
+def rtype_free(hop, i_flavor, i_track_allocation=None):
+    vlist = [hop.inputarg(hop.args_r[0], arg=0)]
+    v_flavor, v_track_allocation = parse_kwds(hop,
+        (i_flavor, lltype.Void),
+        (i_track_allocation, None))
+    #
+    assert v_flavor is not None and v_flavor.value == 'raw'
+    flags = {'flavor': 'raw'}
+    if i_track_allocation is not None:
+        flags['track_allocation'] = v_track_allocation.value
+    vlist.append(hop.inputconst(lltype.Void, flags))
+    #
     hop.exception_cannot_occur()
-    vlist = hop.inputargs(hop.args_r[0], lltype.Void)
     hop.genop('free', vlist)
 
 def rtype_const_result(hop):
@@ -584,8 +598,9 @@
     vinst, = hop.inputargs(hop.args_r[0])
     flavor = hop.args_r[0].gcflavor
     assert flavor != 'gc'
-    cflavor = hop.inputconst(lltype.Void, flavor)
-    return hop.genop('free', [vinst, cflavor])
+    flags = {'flavor': flavor}
+    cflags = hop.inputconst(lltype.Void, flags)
+    return hop.genop('free', [vinst, cflags])
     
 BUILTIN_TYPER[objectmodel.free_non_gc_object] = rtype_free_non_gc_object
 

Modified: pypy/branch/fast-forward/pypy/rpython/rpbc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/rpbc.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/rpbc.py	Fri Oct 22 23:09:43 2010
@@ -256,6 +256,8 @@
     def convert_const(self, value):
         if isinstance(value, types.MethodType) and value.im_self is None:
             value = value.im_func   # unbound method -> bare function
+        elif isinstance(value, staticmethod):
+            value = value.__get__(42) # hackish, get the function wrapped by staticmethod
         if self.lowleveltype is Void:
             return None
         if value is None:

Modified: pypy/branch/fast-forward/pypy/rpython/rtyper.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/rtyper.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/rtyper.py	Fri Oct 22 23:09:43 2010
@@ -421,7 +421,7 @@
             assert noexclink.exitcase is None
             if pos == "removed":
                 # the exception cannot actually occur at all.
-                # See for example rspecialcase.rtype_call_specialcase().
+                # This is set by calling exception_cannot_occur().
                 # We just remove all exception links.
                 block.exitswitch = None
                 block.exits = block.exits[:1]
@@ -1019,7 +1019,7 @@
 from pypy.rpython import rint, rbool, rfloat
 from pypy.rpython import rrange
 from pypy.rpython import rstr, rdict, rlist
-from pypy.rpython import rclass, rbuiltin, rpbc, rspecialcase
+from pypy.rpython import rclass, rbuiltin, rpbc
 from pypy.rpython import rexternalobj
 from pypy.rpython import rptr
 from pypy.rpython import rgeneric

Modified: pypy/branch/fast-forward/pypy/rpython/test/test_llinterp.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/test/test_llinterp.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/test/test_llinterp.py	Fri Oct 22 23:09:43 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 import py
 import sys
 from pypy.rpython.lltypesystem.lltype import typeOf, pyobjectptr, Ptr,\
@@ -5,6 +6,7 @@
 from pypy.rpython.lltypesystem.lloperation import llop
 from pypy.rpython.llinterp import LLInterpreter, LLException, log
 from pypy.rpython.rmodel import inputconst
+from pypy.rpython.annlowlevel import hlstr
 from pypy.translator.translator import TranslationContext, graphof
 from pypy.rpython.rint import signed_repr
 from pypy.rpython.lltypesystem import rstr, lltype
@@ -12,13 +14,11 @@
 from pypy.annotation.model import lltype_to_annotation
 from pypy.rlib.rarithmetic import r_uint, ovfcheck
 from pypy.rpython.ootypesystem import ootype
+from pypy.tool import leakfinder
 from pypy import conftest
 
 # switch on logging of interp to show more info on failing tests
 
-class MallocMismatch(Exception):
-    pass
-
 def setup_module(mod):
     mod.logstate = py.log._getstate()
     py.log.setconsumer("llinterp", py.log.STDOUT)
@@ -72,7 +72,7 @@
 
 def get_interpreter(func, values, view='auto', viewbefore='auto', policy=None,
                     someobjects=False, type_system="lltype", backendopt=False,
-                    config=None, malloc_check=True, **extraconfigopts):
+                    config=None, **extraconfigopts):
     extra_key = [(key, value) for key, value in extraconfigopts.iteritems()]
     extra_key.sort()
     extra_key = tuple(extra_key)
@@ -97,7 +97,7 @@
                                    viewbefore, policy, type_system=type_system,
                                    backendopt=backendopt, config=config,
                                    **extraconfigopts)
-        interp = LLInterpreter(typer, malloc_check=malloc_check)
+        interp = LLInterpreter(typer)
         _tcache[key] = (t, interp, graph)
         # keep the cache small 
         _lastinterpreted.append(key) 
@@ -115,10 +115,17 @@
     interp, graph = get_interpreter(func, values, view, viewbefore, policy,
                                     someobjects, type_system=type_system,
                                     backendopt=backendopt, config=config,
-                                    malloc_check=malloc_check, **kwargs)
-    result = interp.eval_graph(graph, values)
-    if malloc_check and interp.mallocs:
-        raise MallocMismatch(interp.mallocs)
+                                    **kwargs)
+    if not malloc_check:
+        result = interp.eval_graph(graph, values)
+    else:
+        prev = leakfinder.start_tracking_allocations()
+        try:
+            result = interp.eval_graph(graph, values)
+        finally:
+            leaks = leakfinder.stop_tracking_allocations(False, prev)
+        if leaks:
+            raise leakfinder.MallocMismatch(leaks)
     return result
 
 def interpret_raises(exc, func, values, view='auto', viewbefore='auto',
@@ -418,6 +425,7 @@
             assert result
 
 def test_stack_malloc():
+    py.test.skip("stack-flavored mallocs no longer supported")
     class A(object):
         pass
     def f():
@@ -430,6 +438,7 @@
     assert result == 1
 
 def test_invalid_stack_access():
+    py.test.skip("stack-flavored mallocs no longer supported")
     class A(object):
         pass
     globala = A()
@@ -605,7 +614,7 @@
         if x:
             free(t, flavor='raw')
     interpret(f, [1])
-    py.test.raises(MallocMismatch, "interpret(f, [0])")
+    py.test.raises(leakfinder.MallocMismatch, "interpret(f, [0])")
     
     def f():
         t1 = malloc(T, flavor='raw')
@@ -615,3 +624,37 @@
 
     interpret(f, [])
 
+def test_context_manager():
+    state = []
+    class C:
+        def __enter__(self):
+            state.append('acquire')
+            return self
+        def __exit__(self, *args):
+            if args[1] is not None:
+                state.append('raised')
+            state.append('release')
+    def f():
+        try:
+            with C() as c:
+                state.append('use')
+                raise ValueError
+        except ValueError:
+            pass
+        return ', '.join(state)
+    res = interpret(f, [])
+    assert hlstr(res) == 'acquire, use, raised, release'
+
+
+def test_scoped_allocator():
+    from pypy.rpython.lltypesystem.lltype import scoped_alloc, Array, Signed
+    T = Array(Signed)
+    
+    def f():
+        x = 0
+        with scoped_alloc(T, 1) as array:
+            array[0] = -42
+            x = array[0]
+        assert x == -42
+
+    res = interpret(f, [])

Modified: pypy/branch/fast-forward/pypy/rpython/test/test_nongc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/test/test_nongc.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/test/test_nongc.py	Fri Oct 22 23:09:43 2010
@@ -79,7 +79,7 @@
     py.test.raises(TypeError,rtyper.specialize) # results in an invalid cast
 
 def test_isinstance():
-    class A:
+    class A(object):
         _alloc_flavor_ = "raw"
     class B(A):
         pass
@@ -95,7 +95,24 @@
             o = B()
         else:
             o = C()
-        return 100*isinstance(o, A)+10*isinstance(o, B)+1*isinstance(o ,C)
+        res = 100*isinstance(o, A) + 10*isinstance(o, B) + 1*isinstance(o, C)
+        if i == 0:
+            pass
+        elif i == 1:
+            assert isinstance(o, A)
+            free_non_gc_object(o)
+        elif i == 2:
+            assert isinstance(o, B)
+            free_non_gc_object(o)
+        else:
+            assert isinstance(o, C)
+            free_non_gc_object(o)
+        return res
+
+    assert f(1) == 100
+    assert f(2) == 110
+    assert f(3) == 111
+    assert f(0) == 0
 
     a = RPythonAnnotator()
     #does not raise:
@@ -131,10 +148,14 @@
             d = b
         elif i == 2:
             e = c
-        return (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
+        res =  (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
                 0x0008*(a is e) | 0x0010*(b is c) | 0x0020*(b is d) |
                 0x0040*(b is e) | 0x0080*(c is d) | 0x0100*(c is e) |
                 0x0200*(d is e))
+        free_non_gc_object(a)
+        free_non_gc_object(b)
+        free_non_gc_object(c)
+        return res
     a = RPythonAnnotator()
     #does not raise:
     s = a.build_types(f, [int])
@@ -169,10 +190,13 @@
             d = b
         elif i == 2:
             e = c
-        return (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
+        res =  (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
                 0x0008*(a is e) | 0x0010*(b is c) | 0x0020*(b is d) |
                 0x0040*(b is e) | 0x0080*(c is d) | 0x0100*(c is e) |
                 0x0200*(d is e))
+        free_non_gc_object(a)
+        free_non_gc_object(b)
+        return res
     a = RPythonAnnotator()
     #does not raise:
     s = a.build_types(f, [int])

Modified: pypy/branch/fast-forward/pypy/rpython/test/test_rbuilder.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/test/test_rbuilder.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/test/test_rbuilder.py	Fri Oct 22 23:09:43 2010
@@ -1,4 +1,4 @@
-
+import py
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.rpython.lltypesystem.rbuilder import *
 from pypy.rpython.annlowlevel import llstr, hlstr
@@ -55,8 +55,29 @@
         assert res == 'aabcabcdefbuuuu'
         assert isinstance(res, unicode)
 
+    def test_string_getlength(self):
+        def func():
+            s = StringBuilder()
+            s.append("a")
+            s.append("abc")
+            return s.getlength()
+        res = self.interpret(func, [])
+        assert res == 4
+
+    def test_unicode_getlength(self):
+        def func():
+            s = UnicodeBuilder()
+            s.append(u"a")
+            s.append(u"abc")
+            return s.getlength()
+        res = self.interpret(func, [])
+        assert res == 4
+
 class TestLLtype(BaseTestStringBuilder, LLRtypeMixin):
     pass
 
 class TestOOtype(BaseTestStringBuilder, OORtypeMixin):
-    pass
+    def test_string_getlength(self):
+        py.test.skip("getlength(): not implemented on ootype")
+    def test_unicode_getlength(self):
+        py.test.skip("getlength(): not implemented on ootype")

Modified: pypy/branch/fast-forward/pypy/rpython/test/test_rclass.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/test/test_rclass.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/test/test_rclass.py	Fri Oct 22 23:09:43 2010
@@ -3,7 +3,7 @@
 from pypy.translator.translator import TranslationContext, graphof
 from pypy.rpython.lltypesystem.lltype import *
 from pypy.rpython.ootypesystem import ootype
-from pypy.rlib.rarithmetic import intmask
+from pypy.rlib.rarithmetic import intmask, r_longlong
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.objspace.flow.model import summary
 
@@ -319,6 +319,17 @@
         res = self.interpret(f, [])
         assert res == 42
 
+    def test_staticmethod2(self):
+        class A(object):
+            f = staticmethod(lambda x, y: x*y)
+        class B(A):
+            f = staticmethod(lambda x, y: x+y)
+        def f():
+            b = B()
+            return b.f(6, 7)
+        res = self.interpret(f, [])
+        assert res == 13
+
     def test_is(self):
         class A: pass
         class B(A): pass
@@ -1001,6 +1012,40 @@
         res = self.interpret(f, [5])
         assert res == 0
 
+    def test_order_of_fields(self):
+        class A(object):
+            pass
+        def f(n):
+            a = A()
+            a.as_int = n
+            a.as_char = chr(n)
+            a.as_unichar = unichr(n)
+            a.as_double = n + 0.5
+            a.as_bool = bool(n)
+            a.as_void = None
+            a.as_longlong = r_longlong(n)
+            a.as_reference = A()
+            return a
+
+        res = self.interpret(f, [5])
+        names = list(typeOf(res).TO._names)
+        i = names.index('inst_as_int')
+        c = names.index('inst_as_char')
+        u = names.index('inst_as_unichar')
+        d = names.index('inst_as_double')
+        b = names.index('inst_as_bool')
+        v = names.index('inst_as_void')
+        l = names.index('inst_as_longlong')
+        r = names.index('inst_as_reference')
+        assert v == 1      # void fields are first
+        assert sorted([c, b]) == [7, 8]
+        if sys.maxint == 2147483647:
+            assert sorted([u, i, r]) == [4, 5, 6]        # 32-bit types
+            assert sorted([d, l]) == [2, 3]              # 64-bit types
+        else:
+            assert sorted([u]) == [6]                    # 32-bit types
+            assert sorted([i, r, d, l]) == [2, 3, 4, 5]  # 64-bit types
+
 
 class TestOOtype(BaseTestRclass, OORtypeMixin):
 

Modified: pypy/branch/fast-forward/pypy/rpython/test/test_rptr.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rpython/test/test_rptr.py	(original)
+++ pypy/branch/fast-forward/pypy/rpython/test/test_rptr.py	Fri Oct 22 23:09:43 2010
@@ -212,10 +212,31 @@
 
     S = Struct('S', ('x', Signed))
     def fn(n):
-        p = malloc(S, flavor='whatever')
+        p = malloc(S, flavor='raw')
         p.x = n
         result = p.x
-        free(p, flavor='whatever')
+        free(p, flavor='raw')
+        return n
+
+    res = interpret(fn, [23])
+    assert res == 23
+
+    S = Struct('S', ('x', Signed))
+    def fn(n):
+        p = malloc(S, flavor='raw', track_allocation=False)
+        p.x = n
+        result = p.x
+        return n
+
+    res = interpret(fn, [23])
+    assert res == 23
+
+    S = Struct('S', ('x', Signed))
+    def fn(n):
+        p = malloc(S, flavor='raw', track_allocation=False)
+        p.x = n
+        result = p.x
+        free(p, flavor='raw', track_allocation=False)
         return n
 
     res = interpret(fn, [23])

Modified: pypy/branch/fast-forward/pypy/translator/c/funcgen.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/funcgen.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/funcgen.py	Fri Oct 22 23:09:43 2010
@@ -427,7 +427,7 @@
         r = self.expr(op.result)
         return 'OP_CALL_ARGS((%s), %s);' % (', '.join(args), r)
 
-    def generic_call(self, FUNC, fnexpr, args_v, v_result):
+    def generic_call(self, FUNC, fnexpr, args_v, v_result, targets=None):
         args = []
         assert len(args_v) == len(FUNC.TO.ARGS)
         for v, ARGTYPE in zip(args_v, FUNC.TO.ARGS):
@@ -444,17 +444,26 @@
             # skip assignment of 'void' return value
             r = self.expr(v_result)
             line = '%s = %s' % (r, line)
+        if targets:
+            for graph in targets:
+                if getattr(graph, 'inhibit_tail_call', False):
+                    line += '\nPYPY_INHIBIT_TAIL_CALL();'
+                    break
         return line
 
     def OP_DIRECT_CALL(self, op):
         fn = op.args[0]
+        try:
+            targets = [fn.value._obj.graph]
+        except AttributeError:
+            targets = None
         return self.generic_call(fn.concretetype, self.expr(fn),
-                                 op.args[1:], op.result)
+                                 op.args[1:], op.result, targets)
 
     def OP_INDIRECT_CALL(self, op):
         fn = op.args[0]
         return self.generic_call(fn.concretetype, self.expr(fn),
-                                 op.args[1:-1], op.result)
+                                 op.args[1:-1], op.result, op.args[-1].value)
 
     def OP_ADR_CALL(self, op):
         ARGTYPES = [v.concretetype for v in op.args[1:]]

Modified: pypy/branch/fast-forward/pypy/translator/c/gcc/instruction.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/gcc/instruction.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/gcc/instruction.py	Fri Oct 22 23:09:43 2010
@@ -82,6 +82,11 @@
     def all_sources_of(self, localvar):
         return [localvar]
 
+class InsnCondJump(Insn):     # only for debugging; not used internally
+    _args_ = ['label']
+    def __init__(self, label):
+        self.label = label
+
 class Label(Insn):
     _args_ = ['label', 'lineno']
     def __init__(self, label, lineno):
@@ -170,9 +175,12 @@
         self.delta = -7     # use an odd value as marker
 
 class InsnStop(Insn):
-    pass
+    _args_ = ['reason']
+    def __init__(self, reason='?'):
+        self.reason = reason
 
 class InsnRet(InsnStop):
+    _args_ = []
     framesize = 0
     def __init__(self, registers):
         self.registers = registers

Modified: pypy/branch/fast-forward/pypy/translator/c/gcc/test/elf/track5.s
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/gcc/test/elf/track5.s	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/gcc/test/elf/track5.s	Fri Oct 22 23:09:43 2010
@@ -44,7 +44,7 @@
 	addl	%eax, %ebx
 	jmp	.L1221
 .L1227:
-	call	RPyAbort
+	;;call	RPyAbort
 	cmpl	12(%esi), %ebx
 	jb	.L1229
 	addl	$20, %esp

Modified: pypy/branch/fast-forward/pypy/translator/c/gcc/trackgcroot.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/gcc/trackgcroot.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/gcc/trackgcroot.py	Fri Oct 22 23:09:43 2010
@@ -6,7 +6,7 @@
 from pypy.translator.c.gcc.instruction import InsnFunctionStart, InsnStop
 from pypy.translator.c.gcc.instruction import InsnSetLocal, InsnCopyLocal
 from pypy.translator.c.gcc.instruction import InsnPrologue, InsnEpilogue
-from pypy.translator.c.gcc.instruction import InsnGCROOT
+from pypy.translator.c.gcc.instruction import InsnGCROOT, InsnCondJump
 from pypy.translator.c.gcc.instruction import InsnStackAdjust
 from pypy.translator.c.gcc.instruction import InsnCannotFollowEsp
 from pypy.translator.c.gcc.instruction import LocalVar, somenewvalue
@@ -46,6 +46,7 @@
         self.findlabels()
         self.parse_instructions()
         try:
+            self.trim_unreachable_instructions()
             self.find_noncollecting_calls()
             if not self.list_collecting_call_insns():
                 return []
@@ -122,19 +123,36 @@
                 assert label not in self.labels, "duplicate label: %s" % label
                 self.labels[label] = Label(label, lineno)
 
+    def trim_unreachable_instructions(self):
+        reached = set([self.insns[0]])
+        prevlen = 0
+        while len(reached) > prevlen:
+            prevlen = len(reached)
+            for insn in self.insns:
+                if insn not in reached:
+                    for previnsn in insn.previous_insns:
+                        if previnsn in reached:
+                            # this instruction is reachable too
+                            reached.add(insn)
+                            break
+        # now kill all unreachable instructions
+        i = 0
+        while i < len(self.insns):
+            if self.insns[i] in reached:
+                i += 1
+            else:
+                del self.insns[i]
+
     def find_noncollecting_calls(self):
-        cannot_collect = self.CANNOT_COLLECT.copy()
+        cannot_collect = {}
         for line in self.lines:
             match = self.r_gcnocollect_marker.search(line)
             if match:
                 name = match.group(1)
                 cannot_collect[name] = True
         #
-        if self.format in ('darwin', 'mingw32', 'msvc'):
-            self.cannot_collect = dict.fromkeys(
-                ['_' + name for name in cannot_collect])
-        else:
-            self.cannot_collect = cannot_collect
+        self.cannot_collect = dict.fromkeys(
+            [self.function_names_prefix + name for name in cannot_collect])
 
     def append_instruction(self, insn):
         # Add the instruction to the list, and link it to the previous one.
@@ -410,7 +428,8 @@
         return result
     # ____________________________________________________________
 
-    CANNOT_COLLECT = {    # some of the most used functions that cannot collect
+    BASE_FUNCTIONS_NOT_RETURNING = {
+        'abort': None,
         'pypy_debug_catch_fatal_exception': None,
         'RPyAbort': None,
         'RPyAssertFailed': None,
@@ -644,7 +663,7 @@
                 if label != '0':
                     self.register_jump_to(label)
                 tablelin += 1
-            return InsnStop()
+            return InsnStop("jump table")
         if self.r_unaryinsn_star.match(line):
             # that looks like an indirect tail-call.
             # tail-calls are equivalent to RET for us
@@ -658,7 +677,7 @@
             assert not target.startswith('.')
             # tail-calls are equivalent to RET for us
             return InsnRet(self.CALLEE_SAVE_REGISTERS)
-        return InsnStop()
+        return InsnStop("jump")
     
     def register_jump_to(self, label):
         if not isinstance(self.insns[-1], InsnStop):
@@ -682,7 +701,7 @@
         else:
             label = match.group(1)
         self.register_jump_to(label)
-        return []
+        return [InsnCondJump(label)]
 
     visit_jmpl = visit_jmp
     visit_je = conditional_jump
@@ -754,7 +773,7 @@
                     target, = sources
 
         if target in self.FUNCTIONS_NOT_RETURNING:
-            return [InsnStop(), InsnCannotFollowEsp()]
+            return [InsnStop(target)]
         if self.format == 'mingw32' and target == '__alloca':
             # in functions with large stack requirements, windows
             # needs a call to _alloca(), to turn reserved pages
@@ -885,7 +904,7 @@
             # statically known pointer to a register
 
             # %eax -> %rax
-            new_line = re.sub(r"%e(ax|bx|cx|dx|di|si)$", r"%r\1", line)
+            new_line = re.sub(r"%e(ax|bx|cx|dx|di|si|bp)$", r"%r\1", line)
             # %r10d -> %r10
             new_line = re.sub(r"%r(\d+)d$", r"%r\1", new_line)
             return func(self, new_line)
@@ -951,6 +970,7 @@
 
 class ElfFunctionGcRootTracker32(FunctionGcRootTracker32):
     format = 'elf'
+    function_names_prefix = ''
 
     ESP     = '%esp'
     EBP     = '%ebp'
@@ -984,13 +1004,14 @@
     r_bottom_marker = re.compile(r"\t/[*] GC_STACK_BOTTOM [*]/")
 
     FUNCTIONS_NOT_RETURNING = {
-        'abort': None,
         '_exit': None,
         '__assert_fail': None,
         '___assert_rtn': None,
         'L___assert_rtn$stub': None,
         'L___eprintf$stub': None,
         }
+    for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING:
+        FUNCTIONS_NOT_RETURNING[_name] = None
 
     def __init__(self, lines, filetag=0):
         match = self.r_functionstart.match(lines[0])
@@ -1010,6 +1031,8 @@
 
 class ElfFunctionGcRootTracker64(FunctionGcRootTracker64):
     format = 'elf64'
+    function_names_prefix = ''
+
     ESP = '%rsp'
     EBP = '%rbp'
     EAX = '%rax'
@@ -1042,13 +1065,14 @@
     r_bottom_marker = re.compile(r"\t/[*] GC_STACK_BOTTOM [*]/")
 
     FUNCTIONS_NOT_RETURNING = {
-        'abort': None,
         '_exit': None,
         '__assert_fail': None,
         '___assert_rtn': None,
         'L___assert_rtn$stub': None,
         'L___eprintf$stub': None,
         }
+    for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING:
+        FUNCTIONS_NOT_RETURNING[_name] = None
 
     def __init__(self, lines, filetag=0):
         match = self.r_functionstart.match(lines[0])
@@ -1068,6 +1092,7 @@
 
 class DarwinFunctionGcRootTracker(ElfFunctionGcRootTracker32):
     format = 'darwin'
+    function_names_prefix = '_'
 
     r_functionstart = re.compile(r"_(\w+):\s*$")
     OFFSET_LABELS   = 0
@@ -1079,15 +1104,19 @@
 
 class Mingw32FunctionGcRootTracker(DarwinFunctionGcRootTracker):
     format = 'mingw32'
+    function_names_prefix = '_'
 
     FUNCTIONS_NOT_RETURNING = {
-        '_abort': None,
         '_exit': None,
         '__assert': None,
         }
+    for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING:
+        FUNCTIONS_NOT_RETURNING['_' + _name] = None
 
 class MsvcFunctionGcRootTracker(FunctionGcRootTracker32):
     format = 'msvc'
+    function_names_prefix = '_'
+
     ESP = 'esp'
     EBP = 'ebp'
     EAX = 'eax'
@@ -1127,7 +1156,6 @@
     r_bottom_marker = re.compile(r"; .+\tpypy_asm_stack_bottom\(\);")
 
     FUNCTIONS_NOT_RETURNING = {
-        '_abort': None,
         '__exit': None,
         '__assert': None,
         '__wassert': None,
@@ -1136,6 +1164,8 @@
         'DWORD PTR __imp__abort': None,
         'DWORD PTR __imp___wassert': None,
         }
+    for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING:
+        FUNCTIONS_NOT_RETURNING['_' + _name] = None
 
     @classmethod
     def init_regexp(cls):
@@ -1543,9 +1573,7 @@
         assert self.seen_main
 
         def _globalname(name, disp=""):
-            if self.format in ('darwin', 'mingw32', 'msvc'):
-                name = '_' + name
-            return name
+            return tracker_cls.function_names_prefix + name
 
         def _variant(**kwargs):
             txt = kwargs[self.format]

Modified: pypy/branch/fast-forward/pypy/translator/c/src/g_include.h
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/src/g_include.h	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/src/g_include.h	Fri Oct 22 23:09:43 2010
@@ -53,6 +53,7 @@
 #  include "src/rtyper.h"
 #  include "src/debug_print.h"
 #  include "src/debug_traceback.h"
+#  include "src/debug_alloc.h"
 #ifndef AVR
 #  include "src/ll_os.h"
 #  include "src/ll_strtod.h"

Modified: pypy/branch/fast-forward/pypy/translator/c/src/main.h
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/src/main.h	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/src/main.h	Fri Oct 22 23:09:43 2010
@@ -53,10 +53,16 @@
     }
 
     exitcode = STANDALONE_ENTRY_POINT(list);
+
+#ifdef RPY_ASSERT
+    pypy_debug_alloc_results();
+#endif
+
     if (RPyExceptionOccurred()) {
         /* print the RPython traceback */
         pypy_debug_catch_fatal_exception();
     }
+
     return exitcode;
 
  memory_out:

Modified: pypy/branch/fast-forward/pypy/translator/c/src/signals.h
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/src/signals.h	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/src/signals.h	Fri Oct 22 23:09:43 2010
@@ -6,20 +6,6 @@
 
 #include 
 
-#ifndef LONG_MAX
-#if SIZEOF_LONG == 4
-#define LONG_MAX 0X7FFFFFFFL
-#elif SIZEOF_LONG == 8
-#define LONG_MAX 0X7FFFFFFFFFFFFFFFL
-#else
-#error "could not set LONG_MAX in pyport.h"
-#endif
-#endif
-
-#ifndef LONG_MIN
-#define LONG_MIN (-LONG_MAX-1)
-#endif
-
 #include 
 
 #ifdef MS_WINDOWS
@@ -28,10 +14,6 @@
 
 #include 
 
-#ifndef SIG_ERR
-#define SIG_ERR ((PyOS_sighandler_t)(-1))
-#endif
-
 #if defined(PYOS_OS2) && !defined(PYCC_GCC)
 #define NSIG 12
 #include 
@@ -65,11 +47,11 @@
 /* utility to poll for signals that arrived */
 int pypysig_poll(void);   /* => signum or -1 */
 
-/* When a signal is received, the high bit of pypysig_occurred is set.
-   After all signals are processed by pypysig_poll(), the high bit is
+/* When a signal is received, the bit 30 of pypysig_occurred is set.
+   After all signals are processed by pypysig_poll(), the bit 30 is
    cleared again.  The variable is exposed and RPython code is free to
    use the other bits in any way. */
-#define PENDING_SIGNAL_BIT   (LONG_MIN)   /* high bit */
+#define PENDING_SIGNAL_BIT   (1 << 30)
 /* This is a struct for the JIT. See interp_signal.py. */
 struct pypysig_long_struct {
     long value;

Modified: pypy/branch/fast-forward/pypy/translator/c/src/stack.h
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/src/stack.h	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/src/stack.h	Fri Oct 22 23:09:43 2010
@@ -33,6 +33,12 @@
 		&& LL_stack_too_big_slowpath());
 }
 
+#ifdef __GNUC__
+#  define PYPY_INHIBIT_TAIL_CALL()   asm("/* inhibit_tail_call */")
+#else
+#  define PYPY_INHIBIT_TAIL_CALL()   /* add hints for other compilers here */
+#endif
+
 
 #ifndef PYPY_NOT_MAIN_FILE
 #include 

Modified: pypy/branch/fast-forward/pypy/translator/c/test/test_genc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/test/test_genc.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/test/test_genc.py	Fri Oct 22 23:09:43 2010
@@ -426,6 +426,7 @@
     if py.test.config.option.view:
         t.view()
     assert ' BarStruct ' in t.driver.cbuilder.c_source_filename.read()
+    free(foo, flavor="raw")
 
 def test_recursive_llhelper():
     from pypy.rpython.annlowlevel import llhelper
@@ -473,7 +474,27 @@
         return f(s)
     a_f = A(f, "f")
     a_g = A(g, "g")
-    t = lltype.malloc(STRUCT, flavor="raw")
+    t = lltype.malloc(STRUCT, flavor="raw", immortal=True)
     t.bar = llhelper(FTPTR, a_f.make_func())
     fn = compile(chooser, [bool])
     assert fn(True)
+
+def test_inhibit_tail_call():
+    from pypy.rpython.lltypesystem import lltype
+    def foobar_fn(n):
+        return 42
+    foobar_fn._dont_inline_ = True
+    def main(n):
+        return foobar_fn(n)
+    #
+    t = Translation(main, [int], backend="c")
+    t.rtype()
+    t.context._graphof(foobar_fn).inhibit_tail_call = True
+    t.source_c()
+    lines = t.driver.cbuilder.c_source_filename.readlines()
+    for i, line in enumerate(lines):
+        if '= pypy_g_foobar_fn' in line:
+            break
+    else:
+        assert 0, "the call was not found in the C source"
+    assert 'PYPY_INHIBIT_TAIL_CALL();' in lines[i+1]

Modified: pypy/branch/fast-forward/pypy/translator/c/test/test_lltyped.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/test/test_lltyped.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/test/test_lltyped.py	Fri Oct 22 23:09:43 2010
@@ -401,6 +401,7 @@
             for i in range(n):
                 p = malloc(S, flavor='raw', zero=True)
                 if p.x != 0 or p.y != 0:
+                    free(p, flavor='raw')
                     return -1
                 p.x = i
                 p.y = i
@@ -418,14 +419,16 @@
         def f(n):
             for length in range(n-1, -1, -1):
                 p = malloc(S, length, flavor='raw', zero=True)
-                if p.x != 0:
-                    return -1
-                p.x = n
-                for j in range(length):
-                    if p.y[j] != 0:
-                        return -3
-                    p.y[j] = n^j
-                free(p, flavor='raw')
+                try:
+                    if p.x != 0:
+                        return -1
+                    p.x = n
+                    for j in range(length):
+                        if p.y[j] != 0:
+                            return -3
+                        p.y[j] = n^j
+                finally:
+                    free(p, flavor='raw')
             return 42
 
         fn = self.getcompiled(f, [int])
@@ -655,7 +658,7 @@
     def test_prebuilt_ll2ctypes_array(self):
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         A = rffi.CArray(Char)
-        a = malloc(A, 6, flavor='raw')
+        a = malloc(A, 6, flavor='raw', immortal=True)
         a[0] = 'a'
         a[1] = 'b'
         a[2] = 'c'
@@ -676,7 +679,7 @@
     def test_ll2ctypes_array_from_c(self):
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         A = rffi.CArray(Char)
-        a = malloc(A, 6, flavor='raw')
+        a = malloc(A, 6, flavor='raw', immortal=True)
         a[0] = 'a'
         a[1] = 'b'
         a[2] = 'c'

Modified: pypy/branch/fast-forward/pypy/translator/c/test/test_newgc.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/test/test_newgc.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/test/test_newgc.py	Fri Oct 22 23:09:43 2010
@@ -624,13 +624,13 @@
         os.unlink(self.filename)
 
     def define_callback_with_collect(cls):
-        from pypy.rlib.libffi import ffi_type_pointer, cast_type_to_ffitype,\
+        from pypy.rlib.clibffi import ffi_type_pointer, cast_type_to_ffitype,\
              CDLL, ffi_type_void, CallbackFuncPtr, ffi_type_sint
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         import gc
         ffi_size_t = cast_type_to_ffitype(rffi.SIZE_T)
 
-        from pypy.rlib.libffi import get_libc_name
+        from pypy.rlib.clibffi import get_libc_name
 
         def callback(ll_args, ll_res, stuff):
             gc.collect()

Modified: pypy/branch/fast-forward/pypy/translator/c/test/test_standalone.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/test/test_standalone.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/test/test_standalone.py	Fri Oct 22 23:09:43 2010
@@ -16,11 +16,16 @@
 class StandaloneTests(object):
     config = None
 
-    def compile(self, entry_point, debug=True, shared=False):
+    def compile(self, entry_point, debug=True, shared=False,
+                stackcheck=False):
         t = TranslationContext(self.config)
         t.buildannotator().build_types(entry_point, [s_list_of_strings])
         t.buildrtyper().specialize()
 
+        if stackcheck:
+            from pypy.translator.transform import insert_ll_stackcheck
+            insert_ll_stackcheck(t)
+
         t.config.translation.shared = shared
 
         cbuilder = CStandaloneBuilder(t, entry_point, t.config)
@@ -630,6 +635,22 @@
             else:
                 os.environ['CC'] = old_cc
 
+    def test_inhibit_tail_call(self):
+        # the point is to check that the f()->f() recursion stops
+        from pypy.rlib.rstackovf import StackOverflow
+        def f(n):
+            if n <= 0:
+                return 42
+            return f(n+1)
+        def entry_point(argv):
+            try:
+                return f(1)
+            except StackOverflow:
+                print 'hi!'
+                return 0
+        t, cbuilder = self.compile(entry_point, stackcheck=True)
+        out = cbuilder.cmdexec("")
+        assert out.strip() == "hi!"
 
 class TestMaemo(TestStandalone):
     def setup_class(cls):

Modified: pypy/branch/fast-forward/pypy/translator/c/test/test_typed.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/test/test_typed.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/test/test_typed.py	Fri Oct 22 23:09:43 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 import autopath
 import sys
 import math
@@ -823,3 +824,41 @@
             while int(x + frac) >= -sys.maxint-1:
                 x -= 1
             assert f(x + frac) == -666
+
+    def test_context_manager(self):
+        state = []
+        class C:
+            def __init__(self, name):
+                self.name = name
+            def __enter__(self):
+                state.append('acquire')
+                return self
+            def __exit__(self, typ, value, tb):
+                if typ is not None:
+                    if value is None:
+                        raise RuntimeError('test failed')
+                    state.append('raised')
+                else:
+                    if value is not None:
+                        raise RuntimeError('test failed')
+                state.append('release')
+
+        def func(n):
+            del state[:]
+            try:
+                with C('hello') as c:
+                    state.append(c.name)
+                    if n == 1:
+                        raise ValueError
+                    elif n == 2:
+                        raise TypeError
+            except (ValueError, TypeError):
+                pass
+            return ', '.join(state)
+        f = self.getcompiled(func, [int])
+        res = f(0)
+        assert res == 'acquire, hello, release'
+        res = f(1)
+        assert res == 'acquire, hello, raised, release'
+        res = f(2)
+        assert res == 'acquire, hello, raised, release'

Modified: pypy/branch/fast-forward/pypy/translator/goal/ann_override.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/goal/ann_override.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/goal/ann_override.py	Fri Oct 22 23:09:43 2010
@@ -27,21 +27,6 @@
         pol.pypytypes = {}
         pol.single_space = single_space
 
-    #def override__wrap_exception_cls(pol, space, x):
-    #    import pypy.objspace.std.typeobject as typeobject
-    #    clsdef = getbookkeeper().getuniqueclassdef(typeobject.W_TypeObject)
-    #    return annmodel.SomeInstance(clsdef, can_be_None=True)
-    #
-    #def override__fake_object(pol, space, x):
-    #    from pypy.interpreter import typedef
-    #    clsdef = getbookkeeper().getuniqueclassdef(typedef.W_Root)
-    #    return annmodel.SomeInstance(clsdef)    
-    #
-    #def override__cpy_compile(pol, self, source, filename, mode, flags):
-    #    from pypy.interpreter import pycode
-    #    clsdef = getbookkeeper().getuniqueclassdef(pycode.PyCode)
-    #    return annmodel.SomeInstance(clsdef)    
-
     def specialize__wrap(pol,  funcdesc, args_s):
         from pypy.interpreter.baseobjspace import Wrappable
         from pypy.annotation.classdef import ClassDef

Modified: pypy/branch/fast-forward/pypy/translator/goal/app_main.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/goal/app_main.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/goal/app_main.py	Fri Oct 22 23:09:43 2010
@@ -306,7 +306,7 @@
             break
         elif arg == '-u':
             options["unbuffered"] = True
-        elif arg == '-O':
+        elif arg == '-O' or arg == '-OO':
             pass
         elif arg == '--version' or arg == '-V':
             print "Python", sys.version
@@ -451,9 +451,12 @@
                 python_startup = os.getenv('PYTHONSTARTUP')
                 if python_startup:
                     try:
-                        startup = open(python_startup).read()
-                    except IOError:
-                        pass
+                        f = open(python_startup)
+                        startup = f.read()
+                        f.close()
+                    except IOError, e:
+                        print >> sys.stderr, "Could not open PYTHONSTARTUP"
+                        print >> sys.stderr, "IOError:", e
                     else:
                         def run_it():
                             co_python_startup = compile(startup,

Modified: pypy/branch/fast-forward/pypy/translator/jvm/test/test_class.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/jvm/test/test_class.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/jvm/test/test_class.py	Fri Oct 22 23:09:43 2010
@@ -1,6 +1,6 @@
 import py
 from pypy.translator.jvm.test.runtest import JvmTest
-from pypy.translator.oosupport.test_template.class_ import BaseTestClass, BaseTestSpecialcase
+from pypy.translator.oosupport.test_template.class_ import BaseTestClass
 
 class TestJvmClass(JvmTest, BaseTestClass):    
     def test_overridden_classattr_as_defaults(self):
@@ -26,6 +26,3 @@
             
     def test_specialize_methods(self):
         py.test.skip('ABSTRACT METHOD FIX: RE-TEST AFTER MERGE')
-
-class TestJvmSpecialCase(JvmTest, BaseTestSpecialcase):
-    pass

Modified: pypy/branch/fast-forward/pypy/translator/oosupport/test_template/class_.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/oosupport/test_template/class_.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/oosupport/test_template/class_.py	Fri Oct 22 23:09:43 2010
@@ -1,6 +1,5 @@
 import py
 from pypy.rpython.test import test_rclass
-from pypy.rpython.test.test_rspecialcase import BaseTestRspecialcase
 
 class BaseTestClass(test_rclass.TestOOtype):
     def test_abstract_method(self):
@@ -66,6 +65,3 @@
 
     def test_cast_object_mix_null(self):
         py.test.skip('cannot return ootype.NULL from translated functions')
-
-class BaseTestSpecialcase(BaseTestRspecialcase):
-    pass

Modified: pypy/branch/fast-forward/pypy/translator/platform/darwin.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/platform/darwin.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/platform/darwin.py	Fri Oct 22 23:09:43 2010
@@ -14,7 +14,10 @@
     
     def __init__(self, cc=None):
         if cc is None:
-            cc = 'gcc'
+            try:
+                cc = os.environ['CC']
+            except KeyError:
+                cc = 'gcc'
         self.cc = cc
 
     def _args_for_shared(self, args):

Modified: pypy/branch/fast-forward/pypy/translator/transform.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/transform.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/transform.py	Fri Oct 22 23:09:43 2010
@@ -221,15 +221,19 @@
     stack_check_ptr_const = Constant(stack_check_ptr, lltype.typeOf(stack_check_ptr))
     edges = set()
     insert_in = set()
+    block2graph = {}
     for caller in translator.graphs:
         for block, callee in find_calls_from(translator, caller):
             if getattr(getattr(callee, 'func', None),
                        'insert_stack_check_here', False):
                 insert_in.add(callee.startblock)
+                block2graph[callee.startblock] = callee
                 continue
             if block is not caller.startblock:
                 edges.add((caller.startblock, block))
+                block2graph[caller.startblock] = caller
             edges.add((block, callee.startblock))
+            block2graph[block] = caller
 
     edgelist = [Edge(block1, block2) for (block1, block2) in edges]
     edgedict = make_edge_dict(edgelist)
@@ -241,6 +245,10 @@
         v.concretetype = lltype.Void
         unwind_op = SpaceOperation('direct_call', [stack_check_ptr_const], v)
         block.operations.insert(0, unwind_op)
+        # prevents cycles of tail calls from occurring -- such cycles would
+        # not consume any stack, so would turn into potentially infinite loops
+        graph = block2graph[block]
+        graph.inhibit_tail_call = True
     return len(insert_in)
 
 


From afa at codespeak.net  Fri Oct 22 23:54:13 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 22 Oct 2010 23:54:13 +0200 (CEST)
Subject: [pypy-svn] r78230 - in pypy/branch/fast-forward/pypy: interpreter
	module/__builtin__ rlib
Message-ID: <20101022215413.6F9D8282BDC@codespeak.net>

Author: afa
Date: Fri Oct 22 23:54:08 2010
New Revision: 78230

Modified:
   pypy/branch/fast-forward/pypy/interpreter/pyopcode.py
   pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py
   pypy/branch/fast-forward/pypy/rlib/clibffi.py
Log:
Fixes after the merge


Modified: pypy/branch/fast-forward/pypy/interpreter/pyopcode.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyopcode.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyopcode.py	Fri Oct 22 23:54:08 2010
@@ -933,7 +933,6 @@
             raise NotImplementedError("WITH_CLEANUP for CPython <= 2.4")
 
         unroller = self.space.interpclass_w(w_unroller)
-        w_exit = self.popvalue()
         is_app_exc = (unroller is not None and
                       isinstance(unroller, SApplicationException))
         if is_app_exc:
@@ -1375,7 +1374,8 @@
 block_classes = {'SETUP_LOOP': LoopBlock,
                  'SETUP_EXCEPT': ExceptBlock,
                  'SETUP_FINALLY': FinallyBlock,
-                 'SETUP_WITH': WithBlock}
+                 'SETUP_WITH': WithBlock,
+                 }
 
 ### helpers written at the application-level ###
 # Some of these functions are expected to be generally useful if other

Modified: pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py	(original)
+++ pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py	Fri Oct 22 23:54:08 2010
@@ -2,7 +2,7 @@
 from pypy.interpreter.error import OperationError, operationerrfmt
 from pypy.interpreter.gateway import ObjSpace, W_Root, NoneNotWrapped, applevel
 from pypy.interpreter.gateway import interp2app, ObjSpace
-from pypy.interpreter.typedef import TypeDef
+from pypy.interpreter.typedef import TypeDef, make_weakref_descr
 from pypy.interpreter.argument import Arguments
 from pypy.interpreter.baseobjspace import Wrappable
 from pypy.interpreter.typedef import GetSetProperty, descr_get_dict
@@ -706,7 +706,7 @@
             space.call_function(w_func)
 
     def descr_exit(self, space, w_type, w_value, w_tb):
-        w_func = self.getattr(space, space.wrap('__exit__'), False)
+        w_func = self.getattr(space, '__exit__', False)
         if w_func is not None:
             return space.call_function(w_func, w_type, w_value, w_tb)
 

Modified: pypy/branch/fast-forward/pypy/rlib/clibffi.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/clibffi.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/clibffi.py	Fri Oct 22 23:54:08 2010
@@ -148,7 +148,7 @@
               # ffi_type_slong and ffi_type_ulong are omitted because
               # their meaning changes too much from one libffi version to
               # another.  DON'T USE THEM!  use cast_type_to_ffitype().
-              'float', 'pointer', 'void',
+              'float', 'longdouble', 'pointer', 'void',
               # by size
               'sint8', 'uint8', 'sint16', 'uint16', 'sint32', 'uint32',
               'sint64', 'uint64']
@@ -171,14 +171,16 @@
 
 def _signed_type_for(TYPE):
     sz = rffi.sizeof(TYPE)
-    if sz == 2:   return ffi_type_sint16
+    if sz == 1:   return ffi_type_sint8
+    elif sz == 2: return ffi_type_sint16
     elif sz == 4: return ffi_type_sint32
     elif sz == 8: return ffi_type_sint64
     else: raise ValueError("unsupported type size for %r" % (TYPE,))
 
 def _unsigned_type_for(TYPE):
     sz = rffi.sizeof(TYPE)
-    if sz == 2:   return ffi_type_uint16
+    if sz == 1:   return ffi_type_uint8
+    elif sz == 2: return ffi_type_uint16
     elif sz == 4: return ffi_type_uint32
     elif sz == 8: return ffi_type_uint64
     else: raise ValueError("unsupported type size for %r" % (TYPE,))
@@ -186,6 +188,7 @@
 TYPE_MAP = {
     rffi.DOUBLE : ffi_type_double,
     rffi.FLOAT  : ffi_type_float,
+    rffi.LONGDOUBLE : ffi_type_longdouble,
     rffi.UCHAR  : ffi_type_uchar,
     rffi.CHAR   : ffi_type_schar,
     rffi.SHORT  : ffi_type_sshort,
@@ -200,6 +203,7 @@
     rffi.LONGLONG  : _signed_type_for(rffi.LONGLONG),
     lltype.Void    : ffi_type_void,
     lltype.UniChar : _unsigned_type_for(lltype.UniChar),
+    lltype.Bool    : _unsigned_type_for(lltype.Bool),
     }
 
 def external(name, args, result, **kwds):


From afa at codespeak.net  Sat Oct 23 00:08:36 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sat, 23 Oct 2010 00:08:36 +0200 (CEST)
Subject: [pypy-svn] r78231 - pypy/branch/fast-forward/pypy/module/__builtin__
Message-ID: <20101022220836.0E553282BE8@codespeak.net>

Author: afa
Date: Sat Oct 23 00:08:34 2010
New Revision: 78231

Modified:
   pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py
Log:
More fixes


Modified: pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py	(original)
+++ pypy/branch/fast-forward/pypy/module/__builtin__/interp_classobj.py	Sat Oct 23 00:08:34 2010
@@ -604,7 +604,7 @@
         return w_ret
 
     def descr_int(self, space):
-        w_func = self.getattr(space, space.wrap('__int__'), False)
+        w_func = self.getattr(space, '__int__', False)
         if w_func is not None:
             return space.call_function(w_func)
 
@@ -619,7 +619,7 @@
                 space.wrap("__trunc__ returned non-Integral"))
 
     def descr_long(self, space):
-        w_func = self.getattr(space, space.wrap('__long__'), False)
+        w_func = self.getattr(space, '__long__', False)
         if w_func is not None:
             return space.call_function(w_func)
         return self.descr_int(space)


From afa at codespeak.net  Sat Oct 23 11:09:52 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sat, 23 Oct 2010 11:09:52 +0200 (CEST)
Subject: [pypy-svn] r78232 - pypy/branch/fast-forward/pypy/rlib
Message-ID: <20101023090952.C762F282B9C@codespeak.net>

Author: afa
Date: Sat Oct 23 11:09:50 2010
New Revision: 78232

Modified:
   pypy/branch/fast-forward/pypy/rlib/clibffi.py
Log:
Fix after merge


Modified: pypy/branch/fast-forward/pypy/rlib/clibffi.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/clibffi.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/clibffi.py	Sat Oct 23 11:09:50 2010
@@ -376,6 +376,8 @@
 FUNCFLAG_STDCALL   = 0
 FUNCFLAG_CDECL     = 1  # for WINAPI calls
 FUNCFLAG_PYTHONAPI = 4
+FUNCFLAG_USE_ERRNO = 8
+FUNCFLAG_USE_LASTERROR = 16
 
 class AbstractFuncPtr(object):
     ll_cif = lltype.nullptr(FFI_CIFP.TO)


From afa at codespeak.net  Sat Oct 23 16:59:32 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sat, 23 Oct 2010 16:59:32 +0200 (CEST)
Subject: [pypy-svn] r78233 -
	pypy/branch/fast-forward/lib-python/modified-2.7.0/test
Message-ID: <20101023145932.13AC7282B9C@codespeak.net>

Author: afa
Date: Sat Oct 23 16:59:30 2010
New Revision: 78233

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_dumbdbm.py
      - copied, changed from r78232, pypy/branch/fast-forward/lib-python/2.7.0/test/test_dumbdbm.py
Log:
Close the file in dumbdbm test


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_dumbdbm.py (from r78232, pypy/branch/fast-forward/lib-python/2.7.0/test/test_dumbdbm.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_dumbdbm.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_dumbdbm.py	Sat Oct 23 16:59:30 2010
@@ -107,9 +107,11 @@
         f.close()
 
         # Mangle the file by adding \r before each newline
-        data = open(_fname + '.dir').read()
+        with open(_fname + '.dir') as f:
+            data = f.read()
         data = data.replace('\n', '\r\n')
-        open(_fname + '.dir', 'wb').write(data)
+        with open(_fname + '.dir', 'wb') as f:
+            f.write(data)
 
         f = dumbdbm.open(_fname)
         self.assertEqual(f['1'], 'hello')


From afa at codespeak.net  Sat Oct 23 17:37:33 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sat, 23 Oct 2010 17:37:33 +0200 (CEST)
Subject: [pypy-svn] r78234 - in
	pypy/branch/fast-forward/pypy/interpreter/pyparser: . test
Message-ID: <20101023153733.B4BDE36E0C3@codespeak.net>

Author: afa
Date: Sat Oct 23 17:37:29 2010
New Revision: 78234

Modified:
   pypy/branch/fast-forward/pypy/interpreter/pyparser/pyparse.py
   pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py
Log:
while compiling, decoding errors should be turned into SyntaxErrors


Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pyparse.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/pyparse.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pyparse.py	Sat Oct 23 17:37:29 2010
@@ -125,6 +125,11 @@
                     if space.is_w(e.w_type, space.w_LookupError):
                         raise error.SyntaxError("Unknown encoding: %s" % enc,
                                                 filename=compile_info.filename)
+                    # Transform unicode errors into SyntaxError
+                    if space.is_w(e.w_type, space.w_UnicodeDecodeError):
+                        e.normalize_exception(space)
+                        w_message = space.str(e.get_w_value(space))
+                        raise error.SyntaxError(space.str_w(w_message))
                     raise
 
         flags = compile_info.flags

Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py	Sat Oct 23 17:37:29 2010
@@ -56,6 +56,10 @@
         input = "# coding: not-here"
         exc = py.test.raises(SyntaxError, self.parse, input).value
         assert exc.msg == "Unknown encoding: not-here"
+        input = u"# coding: ascii\n\xe2".encode('utf-8')
+        exc = py.test.raises(SyntaxError, self.parse, input).value
+        assert exc.msg == ("'ascii' codec can't decode byte 0xc3 "
+                           "in position 16: ordinal not in range(128)")
 
     def test_syntax_error(self):
         parse = self.parse


From agaynor at codespeak.net  Sat Oct 23 20:05:13 2010
From: agaynor at codespeak.net (agaynor at codespeak.net)
Date: Sat, 23 Oct 2010 20:05:13 +0200 (CEST)
Subject: [pypy-svn] r78235 - pypy/branch/fast-forward/pypy/doc/config
Message-ID: <20101023180513.163B6282B9C@codespeak.net>

Author: agaynor
Date: Sat Oct 23 20:05:10 2010
New Revision: 78235

Added:
   pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._hashlib.txt
Log:
Document the _hashlib module.


Added: pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._hashlib.txt
==============================================================================
--- (empty file)
+++ pypy/branch/fast-forward/pypy/doc/config/objspace.usemodules._hashlib.txt	Sat Oct 23 20:05:10 2010
@@ -0,0 +1,2 @@
+Use the '_hashlib' module.
+Used by the 'hashlib' standard lib module, and indirectly by the various cryptographic libs. This module is expected to be working and is included by default.


From afa at codespeak.net  Sat Oct 23 22:15:54 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sat, 23 Oct 2010 22:15:54 +0200 (CEST)
Subject: [pypy-svn] r78236 - in pypy/branch/fast-forward/pypy: objspace/std
	objspace/std/test rlib rlib/test
Message-ID: <20101023201554.3CE2A36E0CD@codespeak.net>

Author: afa
Date: Sat Oct 23 22:15:50 2010
New Revision: 78236

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/longtype.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_longobject.py
   pypy/branch/fast-forward/pypy/rlib/rbigint.py
   pypy/branch/fast-forward/pypy/rlib/test/test_rbigint.py
Log:
Add long.bit_length()


Modified: pypy/branch/fast-forward/pypy/objspace/std/longtype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/longtype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/longtype.py	Sat Oct 23 22:15:50 2010
@@ -91,6 +91,13 @@
 def descr_get_imag(space, w_obj):
     return space.newlong(0)
 
+def bit_length(space, w_obj):
+    try:
+        return space.wrap(w_obj.num.bit_length())
+    except OverflowError:
+        raise OperationError(space.w_OverflowError,
+                             space.wrap("too many digits in integer"))
+
 # ____________________________________________________________
 
 long_typedef = StdTypeDef("long",
@@ -106,5 +113,6 @@
     denominator = typedef.GetSetProperty(descr_get_denominator),
     real = typedef.GetSetProperty(descr_get_real),
     imag = typedef.GetSetProperty(descr_get_imag),
+    bit_length = gateway.interp2app(bit_length),
 )
 long_typedef.registermethods(globals())

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_longobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_longobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_longobject.py	Sat Oct 23 22:15:50 2010
@@ -269,3 +269,7 @@
             pass
         
         assert type(L(7).conjugate()) is long
+
+    def test_bit_length(self):
+        assert 8L.bit_length() == 4
+        assert (-1<<40).bit_length() == 41

Modified: pypy/branch/fast-forward/pypy/rlib/rbigint.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rbigint.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rbigint.py	Sat Oct 23 22:15:50 2010
@@ -558,6 +558,23 @@
         if self._numdigits() == 1 and self.digits[0] == 0:
             self.sign = 0
 
+    def bit_length(self):
+        i = self._numdigits()
+        if i == 1 and self.digits[0] == 0:
+            return 0
+        msd = self.digits[i - 1]
+        msd_bits = 0
+        if msd >= 32:
+            msd_bits += 6
+            msd >>= 6
+        msd_bits += [
+            0, 1, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4,
+            5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5
+            ][msd]
+        # yes, this can overflow: a huge number which fits 3 gigabytes of
+        # memory has around 24 gigabits!
+        bits = ovfcheck((i-1) * SHIFT + msd_bits)
+        return bits
 
     def __repr__(self):
         return "" % (self.digits, self.sign, self.str())

Modified: pypy/branch/fast-forward/pypy/rlib/test/test_rbigint.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/test/test_rbigint.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/test/test_rbigint.py	Sat Oct 23 22:15:50 2010
@@ -325,6 +325,16 @@
         b = rbigint.fromlong(-1<<3000)
         assert a.mul(b).tolong() == (-1<<10000)*(-1<<3000)
 
+    def test_bit_length(self):
+        assert rbigint.fromlong(0).bit_length() == 0
+        assert rbigint.fromlong(1).bit_length() == 1
+        assert rbigint.fromlong(2).bit_length() == 2
+        assert rbigint.fromlong(3).bit_length() == 2
+        assert rbigint.fromlong(4).bit_length() == 3
+        assert rbigint.fromlong(-3).bit_length() == 2
+        assert rbigint.fromlong(-4).bit_length() == 3
+        assert rbigint.fromlong(1<<40).bit_length() == 41
+
 class TestInternalFunctions(object):
     def test__inplace_divrem1(self):
         # signs are not handled in the helpers!


From afa at codespeak.net  Sat Oct 23 22:32:29 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sat, 23 Oct 2010 22:32:29 +0200 (CEST)
Subject: [pypy-svn] r78237 - pypy/branch/fast-forward/lib-python
Message-ID: <20101023203229.86909282B9C@codespeak.net>

Author: afa
Date: Sat Oct 23 22:32:28 2010
New Revision: 78237

Modified:
   pypy/branch/fast-forward/lib-python/TODO
Log:
More TODO items


Modified: pypy/branch/fast-forward/lib-python/TODO
==============================================================================
--- pypy/branch/fast-forward/lib-python/TODO	(original)
+++ pypy/branch/fast-forward/lib-python/TODO	Sat Oct 23 22:32:28 2010
@@ -28,6 +28,10 @@
 
 - missing functions in itertools: combinations, product, compress...
 
+- in test_os.py, fix posix.setregid(-1, -1), posix.setreuid(-1, -1).  This
+  proably requires to use the git_t typedef instead of rffi.INT.
+
+
 Medium tasks
 ------------
 
@@ -36,6 +40,8 @@
 
 - missing builtin: memoryview
 
+- signal.set_wakeup_fd()
+
 Longer tasks
 ------------
 


From antocuni at codespeak.net  Sun Oct 24 12:05:30 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Sun, 24 Oct 2010 12:05:30 +0200 (CEST)
Subject: [pypy-svn] r78238 - pypy/extradoc/talk
Message-ID: <20101024100530.28289282B9C@codespeak.net>

Author: antocuni
Date: Sun Oct 24 12:05:27 2010
New Revision: 78238

Modified:
   pypy/extradoc/talk/bibtex.bib
Log:
update


Modified: pypy/extradoc/talk/bibtex.bib
==============================================================================
--- pypy/extradoc/talk/bibtex.bib	(original)
+++ pypy/extradoc/talk/bibtex.bib	Sun Oct 24 12:05:27 2010
@@ -25,7 +25,8 @@
     title = {High performance implementation of Python for CLI/.NET with JIT compiler generation for dynamic languages},
     school = {{DISI}, Universit\'a di Genova},
     author = {Antonio Cuni},
-    year = {2010}
+    year = {2010},
+	note = {Technical Report {DISI-TH-2010-05}}
 },
 
 @inproceedings{ancona_rpython:dls_2007,


From arigo at codespeak.net  Sun Oct 24 13:23:58 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 24 Oct 2010 13:23:58 +0200 (CEST)
Subject: [pypy-svn] r78239 - pypy/trunk/pypy/rlib/rsre
Message-ID: <20101024112358.41B20282B9C@codespeak.net>

Author: arigo
Date: Sun Oct 24 13:23:56 2010
New Revision: 78239

Modified:
   pypy/trunk/pypy/rlib/rsre/rsre_jit.py
Log:
Reactivate JITing of regexps.  It seems to give a speed-up at
least on spambayes.


Modified: pypy/trunk/pypy/rlib/rsre/rsre_jit.py
==============================================================================
--- pypy/trunk/pypy/rlib/rsre/rsre_jit.py	(original)
+++ pypy/trunk/pypy/rlib/rsre/rsre_jit.py	Sun Oct 24 13:23:56 2010
@@ -2,7 +2,7 @@
 
 
 class RSreJitDriver(JitDriver):
-    active = False        # XXX temporary?
+    active = True
 
     def __init__(self, name, debugprint, **kwds):
         JitDriver.__init__(self, **kwds)


From arigo at codespeak.net  Sun Oct 24 13:27:26 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 24 Oct 2010 13:27:26 +0200 (CEST)
Subject: [pypy-svn] r78240 - pypy/trunk/pypy/jit/codewriter/test
Message-ID: <20101024112726.9FB7536E0C3@codespeak.net>

Author: arigo
Date: Sun Oct 24 13:27:25 2010
New Revision: 78240

Modified:
   pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py
Log:
Fix the test.


Modified: pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/test/test_jtransform.py	Sun Oct 24 13:27:25 2010
@@ -691,6 +691,7 @@
     class FakeJitDriverSD:
         index = 42
         class jitdriver:
+            active = True
             greens = ['green1', 'green2', 'voidgreen3']
             reds = ['red1', 'red2', 'voidred3']
     jd = FakeJitDriverSD()


From arigo at codespeak.net  Sun Oct 24 13:32:45 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 24 Oct 2010 13:32:45 +0200 (CEST)
Subject: [pypy-svn] r78241 - pypy/trunk/pypy/jit/tool
Message-ID: <20101024113245.8C2E1282B9C@codespeak.net>

Author: arigo
Date: Sun Oct 24 13:32:44 2010
New Revision: 78241

Added:
   pypy/trunk/pypy/jit/tool/pypytrace.vim
Log:
A vim syntax highlighing too, just because.


Added: pypy/trunk/pypy/jit/tool/pypytrace.vim
==============================================================================
--- (empty file)
+++ pypy/trunk/pypy/jit/tool/pypytrace.vim	Sun Oct 24 13:32:44 2010
@@ -0,0 +1,30 @@
+" Language   : PyPy JIT traces
+" Maintainer : Armin Rigo
+
+if exists("b:current_syntax")
+ finish
+endif
+
+syn case ignore
+
+syn match pypyNumber      '\<[0-9.]\+\>'
+syn match pypyConstPtr    '\'
+syn region pypyDescr      start=/descr=/ contains=pypyDescrField
+syn match pypyDescrField  '[.]\w\+ ' contained
+syn match pypyOpNameStart '^' nextgroup=pypyOpName
+syn match pypyOpNameEqual ' = ' nextgroup=pypyOpName
+syn match pypyOpName      '\l\l\w\+' contained
+syn match pypyFailArgs    '[[].*[]]'
+syn match pypyLoopArgs    '^[[].*'
+syn match pypyLoopStart   '^#.*'
+syn match pypyDebugMergePoint  '^debug_merge_point(.\+)'
+
+hi def link pypyLoopStart   Structure
+"hi def link pypyLoopArgs    PreProc
+hi def link pypyFailArgs    String
+hi def link pypyOpName      Statement
+hi def link pypyDebugMergePoint  Comment
+hi def link pypyConstPtr    Constant
+hi def link pypyNumber      Number
+hi def link pypyDescr       String
+hi def link pypyDescrField  Label


From arigo at codespeak.net  Sun Oct 24 13:33:43 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 24 Oct 2010 13:33:43 +0200 (CEST)
Subject: [pypy-svn] r78242 - pypy/branch/leak-finder-more
Message-ID: <20101024113343.D2BB0282B9C@codespeak.net>

Author: arigo
Date: Sun Oct 24 13:33:42 2010
New Revision: 78242

Added:
   pypy/branch/leak-finder-more/
      - copied from r78241, pypy/trunk/
Log:
Enable leakfinder also in app-level tests.



From arigo at codespeak.net  Sun Oct 24 13:35:08 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 24 Oct 2010 13:35:08 +0200 (CEST)
Subject: [pypy-svn] r78243 - pypy/branch/leak-finder-more/pypy
Message-ID: <20101024113508.7D78336E0C3@codespeak.net>

Author: arigo
Date: Sun Oct 24 13:35:07 2010
New Revision: 78243

Modified:
   pypy/branch/leak-finder-more/pypy/conftest.py
Log:
Do it.


Modified: pypy/branch/leak-finder-more/pypy/conftest.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/conftest.py	(original)
+++ pypy/branch/leak-finder-more/pypy/conftest.py	Sun Oct 24 13:35:07 2010
@@ -327,6 +327,18 @@
 class PyPyTestFunction(py.test.collect.Function):
     # All PyPy test items catch and display OperationErrors specially.
     #
+    def runtest(self):
+        leakfinder.start_tracking_allocations()
+        try:
+            self._runtest()
+        finally:
+            if leakfinder.TRACK_ALLOCATIONS:
+                leaks = leakfinder.stop_tracking_allocations(False)
+            else:
+                leaks = None   # stop_tracking_allocations() already called
+        if leaks:        # check for leaks, but only if the test passed so far
+            raise leakfinder.MallocMismatch(leaks)
+
     def execute_appex(self, space, target, *args):
         try:
             target(*args)
@@ -353,16 +365,9 @@
     def _keywords(self):
         return super(IntTestFunction, self)._keywords() + ['interplevel']
 
-    def runtest(self):
+    def _runtest(self):
         try:
-            leakfinder.start_tracking_allocations()
-            try:
-                super(IntTestFunction, self).runtest()
-            finally:
-                if leakfinder.TRACK_ALLOCATIONS:
-                    leaks = leakfinder.stop_tracking_allocations(False)
-                else:
-                    leaks = None   # stop_tracking_allocations() already called
+            super(IntTestFunction, self).runtest()
         except OperationError, e:
             check_keyboard_interrupt(e)
             raise
@@ -381,8 +386,6 @@
                 _pygame_imported = True
                 assert option.view, ("should not invoke Pygame "
                                      "if conftest.option.view is False")
-        if leaks:        # check for leaks, but only if the test passed so far
-            raise leakfinder.MallocMismatch(leaks)
 
 class AppTestFunction(PyPyTestFunction):
     def _prunetraceback(self, traceback):
@@ -395,7 +398,7 @@
     def _keywords(self):
         return ['applevel'] + super(AppTestFunction, self)._keywords()
 
-    def runtest(self):
+    def _runtest(self):
         target = self.obj
         if option.runappdirect:
             return target()
@@ -427,7 +430,7 @@
                     space.setattr(w_instance, space.wrap(name[2:]), 
                                   getattr(instance, name)) 
 
-    def runtest(self):
+    def _runtest(self):
         target = self.obj
         if option.runappdirect:
             return target()


From arigo at codespeak.net  Sun Oct 24 14:20:52 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 24 Oct 2010 14:20:52 +0200 (CEST)
Subject: [pypy-svn] r78244 - pypy/branch/leak-finder-more/pypy
Message-ID: <20101024122052.AD55A282B9C@codespeak.net>

Author: arigo
Date: Sun Oct 24 14:20:50 2010
New Revision: 78244

Modified:
   pypy/branch/leak-finder-more/pypy/conftest.py
Log:
Bah.


Modified: pypy/branch/leak-finder-more/pypy/conftest.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/conftest.py	(original)
+++ pypy/branch/leak-finder-more/pypy/conftest.py	Sun Oct 24 14:20:50 2010
@@ -328,16 +328,29 @@
     # All PyPy test items catch and display OperationErrors specially.
     #
     def runtest(self):
-        leakfinder.start_tracking_allocations()
+        self.runtest_open()
         try:
-            self._runtest()
+            self.runtest_perform()
         finally:
-            if leakfinder.TRACK_ALLOCATIONS:
-                leaks = leakfinder.stop_tracking_allocations(False)
-            else:
-                leaks = None   # stop_tracking_allocations() already called
-        if leaks:        # check for leaks, but only if the test passed so far
-            raise leakfinder.MallocMismatch(leaks)
+            self.runtest_close()
+        self.runtest_finish()
+
+    def runtest_open(self):
+        leakfinder.start_tracking_allocations()
+
+    def runtest_perform(self):
+        super(PyPyTestFunction, self).runtest()
+
+    def runtest_close(self):
+        if leakfinder.TRACK_ALLOCATIONS:
+            self._pypytest_leaks = leakfinder.stop_tracking_allocations(False)
+        else:            # stop_tracking_allocations() already called
+            self._pypytest_leaks = None
+
+    def runtest_finish(self):
+        # check for leaks, but only if the test passed so far
+        if self._pypytest_leaks:
+            raise leakfinder.MallocMismatch(self._pypytest_leaks)
 
     def execute_appex(self, space, target, *args):
         try:
@@ -365,9 +378,9 @@
     def _keywords(self):
         return super(IntTestFunction, self)._keywords() + ['interplevel']
 
-    def _runtest(self):
+    def runtest_perform(self):
         try:
-            super(IntTestFunction, self).runtest()
+            super(IntTestFunction, self).runtest_perform()
         except OperationError, e:
             check_keyboard_interrupt(e)
             raise
@@ -380,12 +393,15 @@
                         py.test.skip('%s: %s' % (e.__class__.__name__, e))
                 cls = cls.__bases__[0]
             raise
+
+    def runtest_finish(self):
         if 'pygame' in sys.modules:
             global _pygame_imported
             if not _pygame_imported:
                 _pygame_imported = True
                 assert option.view, ("should not invoke Pygame "
                                      "if conftest.option.view is False")
+        super(IntTestFunction, self).runtest_finish()
 
 class AppTestFunction(PyPyTestFunction):
     def _prunetraceback(self, traceback):
@@ -398,7 +414,7 @@
     def _keywords(self):
         return ['applevel'] + super(AppTestFunction, self)._keywords()
 
-    def _runtest(self):
+    def runtest_perform(self):
         target = self.obj
         if option.runappdirect:
             return target()
@@ -430,7 +446,7 @@
                     space.setattr(w_instance, space.wrap(name[2:]), 
                                   getattr(instance, name)) 
 
-    def _runtest(self):
+    def runtest_perform(self):
         target = self.obj
         if option.runappdirect:
             return target()


From arigo at codespeak.net  Sun Oct 24 14:21:52 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 24 Oct 2010 14:21:52 +0200 (CEST)
Subject: [pypy-svn] r78245 - pypy/trunk/pypy/annotation/test
Message-ID: <20101024122152.8AE48282B9C@codespeak.net>

Author: arigo
Date: Sun Oct 24 14:21:51 2010
New Revision: 78245

Modified:
   pypy/trunk/pypy/annotation/test/test_annrpython.py
Log:
Let it run on python 2.5.


Modified: pypy/trunk/pypy/annotation/test/test_annrpython.py
==============================================================================
--- pypy/trunk/pypy/annotation/test/test_annrpython.py	(original)
+++ pypy/trunk/pypy/annotation/test/test_annrpython.py	Sun Oct 24 14:21:51 2010
@@ -1,4 +1,4 @@
-
+from __future__ import with_statement
 import autopath
 import py.test
 import sys


From arigo at codespeak.net  Sun Oct 24 14:27:23 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 24 Oct 2010 14:27:23 +0200 (CEST)
Subject: [pypy-svn] r78246 - in pypy/trunk/pypy/module/_ffi: . test
Message-ID: <20101024122723.ACFB3282B9C@codespeak.net>

Author: arigo
Date: Sun Oct 24 14:27:22 2010
New Revision: 78246

Modified:
   pypy/trunk/pypy/module/_ffi/   (props changed)
   pypy/trunk/pypy/module/_ffi/test/   (props changed)
Log:
fixeol



From arigo at codespeak.net  Sun Oct 24 14:35:45 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 24 Oct 2010 14:35:45 +0200 (CEST)
Subject: [pypy-svn] r78247 - in pypy/trunk/pypy/module: _winreg bz2
Message-ID: <20101024123545.BB30F36E0C3@codespeak.net>

Author: arigo
Date: Sun Oct 24 14:35:44 2010
New Revision: 78247

Modified:
   pypy/trunk/pypy/module/_winreg/interp_winreg.py
   pypy/trunk/pypy/module/bz2/interp_bz2.py
Log:
More with_statements.


Modified: pypy/trunk/pypy/module/_winreg/interp_winreg.py
==============================================================================
--- pypy/trunk/pypy/module/_winreg/interp_winreg.py	(original)
+++ pypy/trunk/pypy/module/_winreg/interp_winreg.py	Sun Oct 24 14:35:44 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 from pypy.interpreter.baseobjspace import Wrappable
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.gateway import interp2app

Modified: pypy/trunk/pypy/module/bz2/interp_bz2.py
==============================================================================
--- pypy/trunk/pypy/module/bz2/interp_bz2.py	(original)
+++ pypy/trunk/pypy/module/bz2/interp_bz2.py	Sun Oct 24 14:35:44 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 from pypy.rpython.tool import rffi_platform as platform
 from pypy.rpython.lltypesystem import rffi
 from pypy.rpython.lltypesystem import lltype


From david at codespeak.net  Sun Oct 24 14:37:18 2010
From: david at codespeak.net (david at codespeak.net)
Date: Sun, 24 Oct 2010 14:37:18 +0200 (CEST)
Subject: [pypy-svn] r78248 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . test
Message-ID: <20101024123718.671FE282B9C@codespeak.net>

Author: david
Date: Sun Oct 24 14:37:16 2010
New Revision: 78248

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py
Log:
Refactor guards and finish res op to leave using common code.
Start implementig bridges based on refactored code.


Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Sun Oct 24 14:37:16 2010
@@ -1,11 +1,14 @@
-from pypy.jit.backend.arm.codebuilder import ARMv7Builder
-from pypy.jit.backend.arm import registers as r
 from pypy.jit.backend.arm import conditions as c
-from pypy.jit.backend.llsupport.regalloc import compute_vars_longevity
+from pypy.jit.backend.arm import registers as r
+from pypy.jit.backend.arm.arch import WORD, FUNC_ALIGN
+from pypy.jit.backend.arm.codebuilder import ARMv7Builder, ARMv7InMemoryBuilder
 from pypy.jit.backend.arm.regalloc import ARMRegisterManager
+from pypy.jit.backend.llsupport.regalloc import compute_vars_longevity
+from pypy.jit.metainterp.history import ConstInt, Box, BasicFailDescr
 from pypy.jit.metainterp.resoperation import rop
-from pypy.jit.metainterp.history import ConstInt, Box
-from pypy.rpython.lltypesystem import lltype
+from pypy.rlib import rgc
+from pypy.rpython.annlowlevel import llhelper
+from pypy.rpython.lltypesystem import lltype, rffi, llmemory
 # XXX Move to llsupport
 from pypy.jit.backend.x86.support import values_array
 
@@ -17,15 +20,132 @@
         self.cpu = cpu
         self.input_arg_boxes_int = values_array(lltype.Signed, failargs_limit) # merge with fail_boxes_int later
         self.fail_boxes_int = values_array(lltype.Signed, failargs_limit)
+        self._debug_asm = True
+
+        self._gen_exit_path()
+        self.align()
+        self.mc._start_addr = self.mc.curraddr()
+
+
+    def setup_failure_recovery(self):
+
+        @rgc.no_collect
+        def failure_recovery_func(mem_loc, stackloc):
+            """mem_loc is a structure in memory describing where the values for
+            the failargs are stored. stacklock is the address of the stack
+            section where the registers were saved."""
+            enc = rffi.cast(rffi.CCHARP, mem_loc)
+            stack = rffi.cast(rffi.CCHARP, stackloc)
+            return self.decode_registers_and_descr(enc, stack)
+
+        self.failure_recovery_func = failure_recovery_func
+
+    @rgc.no_collect
+    def decode_registers_and_descr(self, enc, stack):
+        """Decode locations encoded in memory at enc and write the values to
+        the failboxes.
+        Registers are saved on the stack
+        XXX Rest to follow"""
+        i = -1
+        while(True):
+            i += 1
+            r = enc[i]
+            if r == '\xFF':
+                break
+            reg = ord(enc[i])
+            self.fail_boxes_int.setitem(i, self.decode32(stack, reg*WORD))
+        assert enc[i] == '\xFF'
+        descr = self.decode32(enc, i+1)
+        return descr
+
+    def decode32(self, mem, index):
+        highval = ord(mem[index+3])
+        if highval >= 128:
+            highval -= 256
+        return (ord(mem[index])
+                | ord(mem[index+1]) << 8
+                | ord(mem[index+2]) << 16
+                | highval << 24)
+
+    def encode32(self, mem, i, n):
+        mem[i] = chr(n & 0xFF)
+        mem[i+1] = chr((n >> 8) & 0xFF)
+        mem[i+2] = chr((n >> 16) & 0xFF)
+        mem[i+3] = chr((n >> 24) & 0xFF)
+
+    def _gen_exit_path(self):
+        self.setup_failure_recovery()
+        functype = lltype.Ptr(lltype.FuncType([lltype.Signed, lltype.Signed], lltype.Signed))
+        decode_registers_addr = llhelper(functype, self.failure_recovery_func)
+        self.mc.PUSH(range(12))     # registers r0 .. r11
+        self.mc.MOV_rr(r.r0, r.lr)  # move mem block address, to r0 to pass as
+                                    # parameter to next procedure call
+        self.mc.MOV_rr(r.r1, r.sp)  # pass the current stack pointer as second param
+        self.mc.gen_load_int(r.r2, rffi.cast(lltype.Signed, decode_registers_addr))
+        self.mc.gen_load_int(r.lr, self.mc.curraddr()+self.mc.size_of_gen_load_int+WORD)
+        self.mc.MOV_rr(r.pc, r.r2)
+        self.mc.MOV_rr(r.ip, r.r0)
+        self.mc.LDM(r.sp, range(12), w=1) # XXX Replace with POP instr. someday
+
+        self.mc.MOV_rr(r.r0, r.ip)
+
+        self.gen_func_epilog()
+
+    def _gen_path_to_exit_path(self, op, args, regalloc, fcond=c.AL):
+        box = Box()
+        reg = regalloc.try_allocate_reg(box)
+        # XXX free this memory
+        mem = lltype.malloc(rffi.CArray(lltype.Char), len(args)+5, flavor='raw')
+        for i in range(len(args)):
+            curreg = regalloc.try_allocate_reg(args[i])
+            mem[i] = chr(curreg)
+        i = len(args)
+        mem[i] = chr(0xFF)
+        memaddr = rffi.cast(lltype.Signed, mem)
+
+
+        n = self.cpu.get_fail_descr_number(op.getdescr())
+        self.encode32(mem, i+1, n)
+        self.mc.gen_load_int(r.lr, memaddr, cond=fcond)
+        self.mc.gen_load_int(reg, self.mc.baseaddr(), cond=fcond)
+        self.mc.MOV_rr(r.pc, reg, cond=fcond)
+
+        op.getdescr()._arm_guard_reg = reg
+        return memaddr
+
+    def align(self):
+        while(self.mc.curraddr() % FUNC_ALIGN != 0):
+            self.mc.writechar(chr(0))
+
+    def gen_func_epilog(self,cond=c.AL):
+        self.mc.LDM(r.sp, r.callee_restored_registers, cond=cond, w=1)
+
+    def gen_func_prolog(self):
+        self.mc.PUSH(r.callee_saved_registers)
+
+    def gen_bootstrap_code(self, inputargs, regalloc):
+        for i in range(len(inputargs)):
+            reg = regalloc.try_allocate_reg(inputargs[i])
+            addr = self.fail_boxes_int.get_addr_for_num(i)
+            self.mc.gen_load_int(reg, addr)
+            self.mc.LDR_ri(reg, reg)
+
+    def gen_bridge_bootstrap_code(self, inputargs, regalloc):
+        for i in range(len(inputargs)):
+            reg = regalloc.try_allocate_reg(inputargs[i])
+            addr = self.fail_boxes_int.get_addr_for_num(i)
+            self.mc.gen_load_int(reg, addr)
+            self.mc.LDR_ri(reg, reg)
 
+    # cpu interface
     def assemble_loop(self, inputargs, operations, looptoken):
-        assert len(inputargs) == 1
         longevity = compute_vars_longevity(inputargs, operations)
         regalloc = ARMRegisterManager(longevity, assembler=self.mc)
+        loop_start=self.mc.curraddr()
         self.gen_func_prolog()
         self.gen_bootstrap_code(inputargs, regalloc)
         loop_head=self.mc.curraddr()
-        looptoken._arm_bootstrap_code = self.mc.baseaddr()
+        looptoken._arm_bootstrap_code = loop_start
         looptoken._arm_loop_code = loop_head
         looptoken._temp_inputargs = inputargs#XXX remove
         fcond=c.AL
@@ -33,12 +153,42 @@
             opnum = op.getopnum()
             fcond = self.operations[opnum](self, op, regalloc, fcond)
         self.gen_func_epilog()
-        f = open('loop.asm', 'wb')
-        for i in range(self.mc._pos):
-            f.write(self.mc._data[i])
-        f.close()
+        if self._debug_asm:
+            self._dump_trace('loop.asm')
         print 'Done assembling'
 
+    def assemble_bridge(self, faildescr, inputargs, operations):
+        # XXX need to restore args here
+        longevity = compute_vars_longevity(inputargs, operations)
+        regalloc = ARMRegisterManager(longevity, assembler=self.mc)
+        bridge_head = self.mc.curraddr()
+        self.gen_bridge_bootstrap_code(inputargs, regalloc)
+        fcond = c.AL
+        for op in operations:
+            opnum = op.getopnum()
+            fcond = self.operations[opnum](self, op, regalloc, fcond)
+        self.gen_func_epilog()
+        print 'Done building bridges'
+        self.patch_trace(faildescr, bridge_head)
+        print 'Done patching trace'
+        if self._debug_asm:
+            self._dump_trace('bridge.asm')
+
+
+    def _dump_trace(self, name):
+        self.mc._dump_trace(name)
+
+    def patch_trace(self, faildescr, bridge_addr):
+        # XXX make sure there is enough space at patch target
+        fcond = faildescr._arm_guard_cond
+        b = ARMv7InMemoryBuilder(faildescr._arm_guard_code, faildescr._arm_guard_code+100)
+        reg = faildescr._arm_guard_reg
+        b.gen_load_int(reg, bridge_addr, fcond)
+        b.MOV_rr(r.pc, reg, cond=fcond)
+
+
+    # Resoperations
+
     def emit_op_jump(self, op, regalloc, fcond):
         tmp = Box()
         tmpreg = regalloc.try_allocate_reg(tmp)
@@ -49,13 +199,13 @@
             # XXX only if every value is in a register
             self.mc.MOV_rr(inpreg, reg)
         loop_code = op.getdescr()._arm_loop_code
-        self.gen_load_int(tmpreg, loop_code)
+        self.mc.gen_load_int(tmpreg, loop_code)
         self.mc.MOV_rr(r.pc, tmpreg)
         regalloc.possibly_free_var(tmpreg)
         return fcond
 
     def emit_op_finish(self, op, regalloc, fcond):
-        self.gen_write_back(op, op.getarglist(), regalloc, fcond)
+        self._gen_path_to_exit_path(op, op.getarglist(), regalloc, fcond)
         return fcond
 
     def emit_op_int_le(self, op, regalloc, fcond):
@@ -74,46 +224,14 @@
 
     def emit_op_guard_true(self, op, regalloc, fcond):
         assert fcond == c.GT
-        self.gen_write_back(op, op.getfailargs(), regalloc, fcond)
-        self.gen_func_epilog(cond=fcond)
+        descr = op.getdescr()
+        assert isinstance(descr, BasicFailDescr)
+        memaddr = self._gen_path_to_exit_path(op, op.getfailargs(), regalloc, fcond)
+        descr._failure_recovery_code = memaddr
+        descr._arm_guard_code = self.mc.curraddr()
+        descr._arm_guard_cond = fcond
         return c.AL
 
-    def gen_write_back(self, op, args, regalloc, fcond):
-        temp = Box()
-        temp_reg = regalloc.try_allocate_reg(temp)
-        for i in range(len(args)):
-            reg = regalloc.try_allocate_reg(args[i])
-            addr = self.fail_boxes_int.get_addr_for_num(i)
-            self.gen_load_int(temp_reg, addr, cond=fcond)
-            self.mc.STR_ri(reg, temp_reg, cond=fcond)
-
-        regalloc.possibly_free_var(temp_reg)
-        n = self.cpu.get_fail_descr_number(op.getdescr())
-        self.mc.MOV_ri(r.r0, n, cond=fcond)
-
-    def gen_func_epilog(self,cond=c.AL):
-        self.mc.LDM(r.sp, r.callee_restored_registers, cond=cond)
-
-    def gen_func_prolog(self):
-        self.mc.PUSH(r.callee_saved_registers)
-
-    def gen_bootstrap_code(self, inputargs, regalloc):
-        for i in range(len(inputargs)):
-            reg = regalloc.try_allocate_reg(inputargs[i])
-            addr = self.input_arg_boxes_int.get_addr_for_num(i)
-            self.gen_load_int(reg, addr)
-            self.mc.LDR_ri(reg, reg)
-
-    def gen_load_int(self, reg, value, cond=c.AL):
-        assert reg != r.ip, 'ip is used to load int'
-        self.mc.MOV_ri(reg, (value & 0xFF), cond=cond)
-
-        for offset in range(8, 25, 8):
-            self.mc.MOV_ri(r.ip, (value >> offset) & 0xFF, cond=cond)
-            self.mc.ORR_rr(reg, reg, r.ip, offset, cond=cond)
-
-
-
 def make_operation_list():
     def notimplemented(self, op, regalloc, fcond):
         raise NotImplementedError

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Sun Oct 24 14:37:16 2010
@@ -1,22 +1,29 @@
-import conditions as cond
-import registers as reg
-from pypy.rlib.rmmap import alloc
-from pypy.rpython.lltypesystem import lltype, rffi
+from pypy.jit.backend.arm import conditions as cond
+from pypy.jit.backend.arm import registers as reg
+from pypy.jit.backend.arm.arch import WORD
 from pypy.jit.backend.arm.instruction_builder import define_instructions
 
-class ARMv7Builder(object):
+from pypy.rlib.rmmap import alloc, PTR
+from pypy.rpython.lltypesystem import lltype, rffi
 
-    def __init__(self):
-        self._data = alloc(1024)
+class AbstractARMv7Builder(object):
+    def _init(self, data, map_size):
+        self._data = data
+        self._size = map_size
         self._pos = 0
 
+    def _dump_trace(self, name):
+        f = open('output/%s' % name, 'wb')
+        for i in range(self._pos):
+            f.write(self._data[i])
+        f.close()
+
     def PUSH(self, regs, cond=cond.AL):
         assert reg.sp not in regs
         instr = self._encode_reg_list(cond << 28 | 0x92D << 16, regs)
         self.write32(instr)
 
-    def LDM(self, rn, regs, cond=cond.AL):
-        w = 0
+    def LDM(self, rn, regs, w=0, cond=cond.AL):
         instr = cond << 28 | 0x89 << 20 | w << 21 | (rn & 0xFF) << 16
         instr = self._encode_reg_list(instr, regs)
         self.write32(instr)
@@ -58,4 +65,30 @@
     def curraddr(self):
         return self.baseaddr() + self._pos
 
-define_instructions(ARMv7Builder)
+    size_of_gen_load_int = 7 * WORD
+    def gen_load_int(self, r, value, cond=cond.AL):
+        assert r != reg.ip, 'ip is used to load int'
+        self.MOV_ri(r, (value & 0xFF), cond=cond)
+
+        for offset in range(8, 25, 8):
+            t = (value >> offset) & 0xFF
+            #if t == 0:
+            #    continue
+            self.MOV_ri(reg.ip, t, cond=cond)
+            self.ORR_rr(r, r, reg.ip, offset, cond=cond)
+
+class ARMv7InMemoryBuilder(AbstractARMv7Builder):
+    def __init__(self, start, end):
+        map_size = end - start
+        data = rffi.cast(PTR, start)
+        self._init(data, map_size)
+
+class ARMv7Builder(AbstractARMv7Builder):
+
+    def __init__(self):
+        map_size = 1024
+        data = alloc(1024)
+        self._pos = 0
+        self._init(data, map_size)
+
+define_instructions(AbstractARMv7Builder)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	Sun Oct 24 14:37:16 2010
@@ -56,3 +56,18 @@
 supervisor_and_coproc = {
     'MCR': {'op1': 0x20, 'op': 1, 'rn':0, 'coproc':0},
 }
+
+block_data = {
+    'STMDA': {'op': 0x0},
+    'LDMDA': {'op': 0x1},
+    'STMIA': {'op': 0x8},
+    'LDMDB': {'op': 0x11},
+    'STMIB': {'op': 0x18},
+    'LDMIB': {'op': 0x19},
+    'STM':   {'op': 0x4},
+    'LDM':   {'op': 0x5},
+}
+branch = {
+    'B':     {'op': 0x20},
+    'BL':    {'op': 0x30},
+}

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py	Sun Oct 24 14:37:16 2010
@@ -18,6 +18,9 @@
     def compile_loop(self, inputargs, operations, looptoken):
         self.assembler.assemble_loop(inputargs, operations, looptoken)
 
+    def compile_bridge(self, faildescr, inputargs, operations):
+        self.assembler.assemble_bridge(faildescr, inputargs, operations)
+
     def set_future_value_int(self, index, intvalue):
         self.assembler.input_arg_boxes_int.setitem(index, intvalue)
 

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py	Sun Oct 24 14:37:16 2010
@@ -10,7 +10,7 @@
 
 def run_asm(asm):
     BOOTSTRAP_TP = lltype.FuncType([], lltype.Signed)
-    addr = asm.mc.baseaddr()
+    addr = asm.mc._start_addr
     assert addr % 8 == 0
     func = rffi.cast(lltype.Ptr(BOOTSTRAP_TP), addr)
     return func()

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py	Sun Oct 24 14:37:16 2010
@@ -1,9 +1,14 @@
-from pypy.jit.backend.arm import registers as r
 from pypy.jit.backend.arm import conditions as c
+from pypy.jit.backend.arm import registers as r
+from pypy.jit.backend.arm.arch import WORD
 from pypy.jit.backend.arm.assembler import AssemblerARM
+from pypy.jit.backend.arm.codebuilder import ARMv7InMemoryBuilder
 from pypy.jit.backend.arm.test.support import skip_unless_arm, run_asm
 from pypy.jit.metainterp.resoperation import rop
 
+from pypy.rpython.annlowlevel import llhelper
+from pypy.rpython.lltypesystem import lltype, rffi, llmemory
+
 skip_unless_arm()
 
 class TestRunningAssembler():
@@ -16,19 +21,19 @@
 
     def test_load_small_int_to_reg(self):
         self.a.gen_func_prolog()
-        self.a.gen_load_int(r.r0, 123)
+        self.a.mc.gen_load_int(r.r0, 123)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 123
 
     def test_load_medium_int_to_reg(self):
         self.a.gen_func_prolog()
-        self.a.gen_load_int(r.r0, 0xBBD7)
+        self.a.mc.gen_load_int(r.r0, 0xBBD7)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 48087
 
     def test_load_int_to_reg(self):
         self.a.gen_func_prolog()
-        self.a.gen_load_int(r.r0, 0xFFFFFF85)
+        self.a.mc.gen_load_int(r.r0, 0xFFFFFF85)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == -123
 
@@ -43,14 +48,14 @@
 
     def test_sub(self):
         self.a.gen_func_prolog()
-        self.a.gen_load_int(r.r1, 123456)
+        self.a.mc.gen_load_int(r.r1, 123456)
         self.a.mc.SUB_ri(r.r0, r.r1, 123)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 123333
 
     def test_cmp(self):
         self.a.gen_func_prolog()
-        self.a.gen_load_int(r.r1, 22)
+        self.a.mc.gen_load_int(r.r1, 22)
         self.a.mc.CMP(r.r1, 123)
         self.a.mc.MOV_ri(r.r0, 1, c.LE)
         self.a.mc.MOV_ri(r.r0, 0, c.GT)
@@ -59,7 +64,7 @@
 
     def test_int_le_false(self):
         self.a.gen_func_prolog()
-        self.a.gen_load_int(r.r1, 2222)
+        self.a.mc.gen_load_int(r.r1, 2222)
         self.a.mc.CMP(r.r1, 123)
         self.a.mc.MOV_ri(r.r0, 1, c.LE)
         self.a.mc.MOV_ri(r.r0, 0, c.GT)
@@ -73,7 +78,7 @@
         self.a.mc.CMP(r.r1, 0) # z=0, z=1
         self.a.mc.MOV_ri(r.r1, 0, cond=c.NE)
         self.a.mc.MOV_ri(r.r1, 7, cond=c.EQ)
-        self.a.gen_load_int(r.r4, loop_head, cond=c.NE)
+        self.a.mc.gen_load_int(r.r4, loop_head, cond=c.NE)
         self.a.mc.MOV_rr(r.pc, r.r4, cond=c.NE)
         self.a.mc.MOV_rr(r.r0, r.r1)
         self.a.gen_func_epilog()
@@ -85,10 +90,25 @@
         loop_head = self.a.mc.curraddr()
         self.a.mc.ADD_ri(r.r1, r.r1, 1)
         self.a.mc.CMP(r.r1, 9)
-        self.a.gen_load_int(r.r4, loop_head, cond=c.NE)
+        self.a.mc.gen_load_int(r.r4, loop_head, cond=c.NE)
         self.a.mc.MOV_rr(r.pc, r.r4, cond=c.NE)
         self.a.mc.MOV_rr(r.r0, r.r1)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 9
 
 
+    def test_call_python_func(self):
+        functype = lltype.Ptr(lltype.FuncType([lltype.Signed], lltype.Signed))
+        call_addr = rffi.cast(lltype.Signed, llhelper(functype, callme))
+        self.a.gen_func_prolog()
+        self.a.mc.MOV_ri(r.r0, 123)
+        self.a.mc.gen_load_int(r.r1, call_addr)
+        self.a.mc.gen_load_int(r.lr, self.a.mc.curraddr()+self.a.mc.size_of_gen_load_int+WORD)
+        self.a.mc.MOV_rr(r.pc, r.r1)
+        self.a.gen_func_epilog()
+        assert run_asm(self.a) == 133
+
+def callme(inp):
+    i = inp + 10
+    return i
+


From david at codespeak.net  Sun Oct 24 14:47:01 2010
From: david at codespeak.net (david at codespeak.net)
Date: Sun, 24 Oct 2010 14:47:01 +0200 (CEST)
Subject: [pypy-svn] r78249 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101024124701.09BE836E0C3@codespeak.net>

Author: david
Date: Sun Oct 24 14:47:00 2010
New Revision: 78249

Added:
   pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py
Log:
Arch constants

Added: pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py
==============================================================================
--- (empty file)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py	Sun Oct 24 14:47:00 2010
@@ -0,0 +1,2 @@
+FUNC_ALIGN=8
+WORD=4


From antocuni at codespeak.net  Sun Oct 24 17:54:50 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Sun, 24 Oct 2010 17:54:50 +0200 (CEST)
Subject: [pypy-svn] r78250 - pypy/extradoc/talk
Message-ID: <20101024155450.EB96F282B9C@codespeak.net>

Author: antocuni
Date: Sun Oct 24 17:54:45 2010
New Revision: 78250

Modified:
   pypy/extradoc/talk/bibtex.bib
Log:
s/\t/'    '


Modified: pypy/extradoc/talk/bibtex.bib
==============================================================================
--- pypy/extradoc/talk/bibtex.bib	(original)
+++ pypy/extradoc/talk/bibtex.bib	Sun Oct 24 17:54:45 2010
@@ -26,7 +26,7 @@
     school = {{DISI}, Universit\'a di Genova},
     author = {Antonio Cuni},
     year = {2010},
-	note = {Technical Report {DISI-TH-2010-05}}
+    note = {Technical Report {DISI-TH-2010-05}}
 },
 
 @inproceedings{ancona_rpython:dls_2007,


From afa at codespeak.net  Sun Oct 24 20:07:08 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sun, 24 Oct 2010 20:07:08 +0200 (CEST)
Subject: [pypy-svn] r78251 - in pypy/branch/fast-forward/pypy:
	annotation/test jit/codewriter/test jit/tool module/_ffi
	module/_ffi/test module/_winreg module/bz2 rlib/rsre
Message-ID: <20101024180708.AFCAC282B9C@codespeak.net>

Author: afa
Date: Sun Oct 24 20:07:06 2010
New Revision: 78251

Added:
   pypy/branch/fast-forward/pypy/jit/tool/pypytrace.vim
      - copied unchanged from r78250, pypy/trunk/pypy/jit/tool/pypytrace.vim
Modified:
   pypy/branch/fast-forward/pypy/annotation/test/test_annrpython.py
   pypy/branch/fast-forward/pypy/jit/codewriter/test/test_jtransform.py
   pypy/branch/fast-forward/pypy/module/_ffi/   (props changed)
   pypy/branch/fast-forward/pypy/module/_ffi/test/   (props changed)
   pypy/branch/fast-forward/pypy/module/_winreg/interp_winreg.py
   pypy/branch/fast-forward/pypy/module/bz2/interp_bz2.py
   pypy/branch/fast-forward/pypy/rlib/rsre/rsre_jit.py
Log:
Merge from trunk
78228:78250


Modified: pypy/branch/fast-forward/pypy/annotation/test/test_annrpython.py
==============================================================================
--- pypy/branch/fast-forward/pypy/annotation/test/test_annrpython.py	(original)
+++ pypy/branch/fast-forward/pypy/annotation/test/test_annrpython.py	Sun Oct 24 20:07:06 2010
@@ -1,4 +1,4 @@
-
+from __future__ import with_statement
 import autopath
 import py.test
 import sys

Modified: pypy/branch/fast-forward/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/branch/fast-forward/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/branch/fast-forward/pypy/jit/codewriter/test/test_jtransform.py	Sun Oct 24 20:07:06 2010
@@ -691,6 +691,7 @@
     class FakeJitDriverSD:
         index = 42
         class jitdriver:
+            active = True
             greens = ['green1', 'green2', 'voidgreen3']
             reds = ['red1', 'red2', 'voidred3']
     jd = FakeJitDriverSD()

Modified: pypy/branch/fast-forward/pypy/module/_winreg/interp_winreg.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_winreg/interp_winreg.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_winreg/interp_winreg.py	Sun Oct 24 20:07:06 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 from pypy.interpreter.baseobjspace import Wrappable
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.gateway import interp2app

Modified: pypy/branch/fast-forward/pypy/module/bz2/interp_bz2.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/bz2/interp_bz2.py	(original)
+++ pypy/branch/fast-forward/pypy/module/bz2/interp_bz2.py	Sun Oct 24 20:07:06 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 from pypy.rpython.tool import rffi_platform as platform
 from pypy.rpython.lltypesystem import rffi
 from pypy.rpython.lltypesystem import lltype

Modified: pypy/branch/fast-forward/pypy/rlib/rsre/rsre_jit.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rsre/rsre_jit.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rsre/rsre_jit.py	Sun Oct 24 20:07:06 2010
@@ -2,7 +2,7 @@
 
 
 class RSreJitDriver(JitDriver):
-    active = False        # XXX temporary?
+    active = True
 
     def __init__(self, name, debugprint, **kwds):
         JitDriver.__init__(self, **kwds)


From cfbolz at codespeak.net  Mon Oct 25 11:06:04 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Mon, 25 Oct 2010 11:06:04 +0200 (CEST)
Subject: [pypy-svn] r78252 - pypy/extradoc/sprintinfo/ddorf2010
Message-ID: <20101025090604.B333F282C04@codespeak.net>

Author: cfbolz
Date: Mon Oct 25 11:06:01 2010
New Revision: 78252

Added:
   pypy/extradoc/sprintinfo/ddorf2010/planning.txt   (contents, props changed)
Log:
(all) planning for today


Added: pypy/extradoc/sprintinfo/ddorf2010/planning.txt
==============================================================================
--- (empty file)
+++ pypy/extradoc/sprintinfo/ddorf2010/planning.txt	Mon Oct 25 11:06:01 2010
@@ -0,0 +1,27 @@
+people present:
+    Anto
+    David
+    Carl Friedrich
+    Ronny
+    Armin
+    Lukas
+
+
+topics:
+
+- get mmap to work on Beagleboard (David, Armin)
+- bridges in the ARM backend (David, Armin)
+- hg conversion (Ronny, Anto)
+- memory benchmarks (Carl Friedrich, Lukas)
+- buildbot improvements, locks, email on failing tests
+- run pypy-c-nojit in the benchmarks
+- make it possible to run benchmark on branches
+- understand problems of mapdict
+- rewrite ctypes to use _ffi
+- can classes with non-type metaclasses use typeversion?
+
+
+scheduled discussions:
+
+- hg migration planning
+- understand h?kan's branch


From david at codespeak.net  Mon Oct 25 13:46:24 2010
From: david at codespeak.net (david at codespeak.net)
Date: Mon, 25 Oct 2010 13:46:24 +0200 (CEST)
Subject: [pypy-svn] r78253 -
	pypy/branch/arm-backend/pypy/rpython/lltypesystem
Message-ID: <20101025114624.33CE4282C04@codespeak.net>

Author: david
Date: Mon Oct 25 13:46:22 2010
New Revision: 78253

Modified:
   pypy/branch/arm-backend/pypy/rpython/lltypesystem/ll2ctypes.py
Log:
(arigo, david) Hack for mmap to use mmap64

Modified: pypy/branch/arm-backend/pypy/rpython/lltypesystem/ll2ctypes.py
==============================================================================
--- pypy/branch/arm-backend/pypy/rpython/lltypesystem/ll2ctypes.py	(original)
+++ pypy/branch/arm-backend/pypy/rpython/lltypesystem/ll2ctypes.py	Mon Oct 25 13:46:22 2010
@@ -528,7 +528,7 @@
                 return _items
             _items.append(nextitem)
             i += 1
-        
+
     items = property(getitems)
 
 class _array_of_known_length(_array_of_unknown_length):
@@ -892,7 +892,7 @@
             return clibname+'.dll'
         else:
             return ctypes.util.find_library('c')
-        
+
     libc_name = get_libc_name()     # Make sure the name is determined during import, not at runtime
     # XXX is this always correct???
     standard_c_lib = ctypes.CDLL(get_libc_name(), **load_library_kwargs)
@@ -938,9 +938,12 @@
             return lib[elem]
         except AttributeError:
             pass
-    
+
     old_eci = funcptr._obj.compilation_info
     funcname = funcptr._obj._name
+    #XXX Fix this, hack for ARM
+    if funcname == 'mmap':
+        funcname = 'mmap64'
     if hasattr(old_eci, '_with_ctypes'):
         eci = old_eci._with_ctypes
     else:


From afa at codespeak.net  Mon Oct 25 14:08:12 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 25 Oct 2010 14:08:12 +0200 (CEST)
Subject: [pypy-svn] r78254 - pypy/branch/fast-forward/pypy/objspace/std
Message-ID: <20101025120812.19CFA282C06@codespeak.net>

Author: afa
Date: Mon Oct 25 14:08:11 2010
New Revision: 78254

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/longtype.py
Log:
Fix translation


Modified: pypy/branch/fast-forward/pypy/objspace/std/longtype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/longtype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/longtype.py	Mon Oct 25 14:08:11 2010
@@ -92,8 +92,9 @@
     return space.newlong(0)
 
 def bit_length(space, w_obj):
+    bigint = space.bigint_w(w_obj)
     try:
-        return space.wrap(w_obj.num.bit_length())
+        return space.wrap(bigint.bit_length())
     except OverflowError:
         raise OperationError(space.w_OverflowError,
                              space.wrap("too many digits in integer"))


From afa at codespeak.net  Mon Oct 25 14:13:51 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 25 Oct 2010 14:13:51 +0200 (CEST)
Subject: [pypy-svn] r78256 - in pypy/branch/fast-forward/pypy/objspace/flow:
	. test
Message-ID: <20101025121351.8551B282C06@codespeak.net>

Author: afa
Date: Mon Oct 25 14:13:50 2010
New Revision: 78256

Modified:
   pypy/branch/fast-forward/pypy/objspace/flow/flowcontext.py
   pypy/branch/fast-forward/pypy/objspace/flow/test/test_objspace.py
Log:
Another R to RPython: __exit__ must not return True to swallow exceptions.
This fixes constructs like::

    with c:
        res = fn()
    return res

otherwise the flow space finds a path where no exception is raised,
but 'res' is not initialized.


Modified: pypy/branch/fast-forward/pypy/objspace/flow/flowcontext.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/flow/flowcontext.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/flow/flowcontext.py	Mon Oct 25 14:13:50 2010
@@ -445,5 +445,8 @@
             # Replace it with an object which will break translation when used
             # (except maybe with 'exc_typ is None')
             w_typ = self.space.wrap(self.space)
-        return self.space.call_function(w_func, w_typ, w_val, w_tb)
+        self.space.call_function(w_func, w_typ, w_val, w_tb)
+        # Return None so that the flow space statically knows that we didn't
+        # swallow the exception
+        return self.space.w_None
 

Modified: pypy/branch/fast-forward/pypy/objspace/flow/test/test_objspace.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/flow/test/test_objspace.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/flow/test/test_objspace.py	Mon Oct 25 14:13:50 2010
@@ -841,12 +841,13 @@
         def g(): pass
         def f(c, x):
             with x:
-                g()
+                res = g()
+            return res
         graph = self.codetest(f)
         assert self.all_operations(graph) == {
             'getattr': 2,     # __enter__ and __exit__
             'simple_call': 4, # __enter__, g and 2 possible calls to __exit__
-            'is_true': 1}     # check the result of __exit__()
+            }
 
     def monkey_patch_code(self, code, stacksize, flags, codestring, names, varnames):
         c = code


From afa at codespeak.net  Mon Oct 25 14:17:00 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 25 Oct 2010 14:17:00 +0200 (CEST)
Subject: [pypy-svn] r78257 - in pypy/branch/fast-forward/pypy/module: _io
	_io/test thread
Message-ID: <20101025121700.A88CB282C06@codespeak.net>

Author: afa
Date: Mon Oct 25 14:16:59 2010
New Revision: 78257

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
   pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
   pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
   pypy/branch/fast-forward/pypy/module/thread/os_lock.py
Log:
Implement BufferedReader.read()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Mon Oct 25 14:16:59 2010
@@ -1,10 +1,12 @@
+from __future__ import with_statement
 from pypy.interpreter.typedef import (
     TypeDef, generic_new_descr)
 from pypy.interpreter.gateway import interp2app, unwrap_spec
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.error import OperationError
 from pypy.rpython.lltypesystem import lltype, rffi
-from pypy.module._io.interp_iobase import W_IOBase
+from pypy.rlib.rstring import StringBuilder
+from pypy.module._io.interp_iobase import W_IOBase, convert_size
 from pypy.module._io.interp_io import DEFAULT_BUFFER_SIZE
 from pypy.module.thread.os_lock import Lock
 
@@ -14,6 +16,9 @@
         self.buffer = lltype.nullptr(rffi.CCHARP.TO)
         self.lock = None
 
+        self.readable = False
+        self.writable = False
+
     def _init(self, space):
         if self.buffer_size <= 0:
             raise OperationError(space.w_ValueError, space.wrap(
@@ -44,9 +49,24 @@
         self.abs_pos = pos
         return pos
 
+    def _readahead(self):
+        if self.readable and self.read_end != -1:
+            return self.read_end - self.pos
+        return 0
+
+    def _unsupportedoperation(self, space, message):
+        w_exc = space.getattr(space.getbuiltinmodule('_io'),
+                              space.wrap('UnsupportedOperation'))
+        raise OperationError(w_exc, space.wrap(message))
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def read_w(self, space, w_size=None):
+        self._unsupportedoperation(space, "read")
+
 W_BufferedIOBase.typedef = TypeDef(
     '_BufferedIOBase', W_IOBase.typedef,
     __new__ = generic_new_descr(W_BufferedIOBase),
+    read = interp2app(W_BufferedIOBase.read_w),
     )
 
 class W_BufferedReader(W_BufferedIOBase):
@@ -63,7 +83,6 @@
         self.raw = raw
         self.buffer_size = buffer_size
         self.readable = True
-        self.writable = False
 
         self._init(space)
         self._reset_buf()
@@ -71,10 +90,108 @@
     def _reset_buf(self):
         self.read_end = -1
 
+    def _closed(self, space):
+        return self.raw._closed(space)
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def read_w(self, space, w_size=None):
+        self._check_closed(space, "read of closed file")
+        size = convert_size(space, w_size)
+
+        if size < 0:
+            # read until the end of stream
+            with self.lock:
+                res = self._read_all(space)
+        else:
+            res = self._read_fast(size)
+            if res is None:
+                with self.lock:
+                    res = self._read_generic(space, size)
+        return space.wrap(res)
+
+    def _read_all(self, space):
+        builder = StringBuilder()
+        # First copy what we have in the current buffer
+        current_size = self._readahead()
+        data = None
+        if current_size:
+            data = rffi.charpsize2str(rffi.ptradd(self.buffer, self.pos),
+                                      current_size)
+            builder.append(data)
+        self._reset_buf()
+        # We're going past the buffer's bounds, flush it
+        if self.writable:
+            self._writer_flush_unlocked(restore_pos=True)
+
+        while True:
+            # Read until EOF or until read() would block
+            w_data = space.call_method(self.raw, "read")
+            if space.is_w(w_data, space.w_None):
+                break
+            data = space.str_w(w_data)
+            size = len(data)
+            if size == 0:
+                break
+            builder.append(data)
+            current_size += size
+            if self.abs_pos != -1:
+                self.abs_pos += size
+        return builder.build()
+
+    def _read_generic(self, space, n):
+        """Generic read function: read from the stream until enough bytes are
+           read, or until an EOF occurs or until read() would block."""
+        current_size = self._readahead()
+        if n <= current_size:
+            return self._read_fast(n)
+
+        builder = StringBuilder(n)
+        remaining = n
+        written = 0
+        data = None
+        if current_size:
+            data = rffi.charpsize2str(rffi.ptradd(self.buffer, self.pos),
+                                      current_size)
+            builder.append(data)
+        self._reset_buf()
+
+        # XXX potential bug in CPython? The following is not enabled.
+        # We're going past the buffer's bounds, flush it
+        ## if self.writable:
+        ##     self._writer_flush_unlocked(restore_pos=True)
+
+        while remaining > 0:
+            # Read until EOF or until read() would block
+            w_data = space.call_method(self.raw, "read", space.wrap(remaining))
+            if space.is_w(w_data, space.w_None):
+                break
+            data = space.str_w(w_data)
+            size = len(data)
+            if size == 0:
+                break
+            builder.append(data)
+            current_size += size
+            remaining -= size
+            if self.abs_pos != -1:
+                self.abs_pos += size
+        return builder.build()
+
+    def _read_fast(self, n):
+        """Read n bytes from the buffer if it can, otherwise return None.
+           This function is simple enough that it can run unlocked."""
+        current_size = self._readahead()
+        if n <= current_size:
+            res = rffi.charpsize2str(rffi.ptradd(self.buffer, self.pos), n)
+            self.pos += n
+            return res
+        return None
+
 W_BufferedReader.typedef = TypeDef(
     'BufferedReader', W_BufferedIOBase.typedef,
     __new__ = generic_new_descr(W_BufferedReader),
     __init__  = interp2app(W_BufferedReader.descr_init),
+
+    read = interp2app(W_BufferedReader.read_w),
     )
 
 class W_BufferedWriter(W_BufferedIOBase):

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_fileio.py	Mon Oct 25 14:16:59 2010
@@ -1,4 +1,3 @@
-from pypy.module._io.interp_iobase import W_RawIOBase
 from pypy.interpreter.typedef import (
     TypeDef, interp_attrproperty, interp_attrproperty_w, GetSetProperty,
     make_weakref_descr)
@@ -9,6 +8,7 @@
 from pypy.rlib.rstring import StringBuilder
 from os import O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_TRUNC
 import sys, os, stat, errno
+from pypy.module._io.interp_iobase import W_RawIOBase, convert_size
 
 def interp_member_w(name, cls, doc=None):
     "NOT_RPYTHON: initialization-time only"
@@ -93,12 +93,6 @@
 
     return readable, writable, append, flags
 
-def convert_size(space, w_size):
-    if space.is_w(w_size, space.w_None):
-        return -1
-    else:
-        return space.int_w(w_size)
-
 SMALLCHUNK = 8 * 1024
 BIGCHUNK = 512 * 1024
 
@@ -211,10 +205,14 @@
     def descr_get_mode(space, self):
         return space.wrap(self._mode())
 
-    def _check_closed(self, space):
+    def _closed(self, space):
+        return self.fd < 0
+
+    def _check_closed(self, space, message=None):
+        if message is None:
+            message = "I/O operation on closed file"
         if self.fd < 0:
-            raise OperationError(space.w_ValueError, space.wrap(
-                "I/O operation on closed file"))
+            raise OperationError(space.w_ValueError, space.wrap(message))
 
     def _close(self, space):
         if self.fd < 0:

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	Mon Oct 25 14:16:59 2010
@@ -47,11 +47,12 @@
         # attribute as returned by whatever subclass.
         return self.__IOBase_closed
 
-    def _check_closed(self, space):
+    def _check_closed(self, space, message=None):
+        if message is None:
+            message = "I/O operation on closed file"
         if self._closed(space):
             raise OperationError(
-                space.w_ValueError,
-                space.wrap("I/O operation on closed file"))
+                space.w_ValueError, space.wrap(message))
 
     def closed_get_w(space, self):
         return space.newbool(self.__IOBase_closed)

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_stringio.py	Mon Oct 25 14:16:59 2010
@@ -9,7 +9,7 @@
         self.buf = []
         self.pos = 0
 
-    def _check_closed(self, space):
+    def _check_closed(self, space, message=None):
         pass
     def _check_initialized(self):
         pass

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Mon Oct 25 14:16:59 2010
@@ -8,7 +8,15 @@
         tmpfile.write("a\nb\nc", mode='wb')
         cls.w_tmpfile = cls.space.wrap(str(tmpfile))
 
-    def test_simple(self):
+    def test_simple_read(self):
         import _io
         raw = _io.FileIO(self.tmpfile)
-        _io.BufferedReader(raw)
+        f = _io.BufferedReader(raw)
+        assert f.read() == "a\nb\nc"
+        f.close()
+        #
+        raw.seek(0)
+        f = _io.BufferedReader(raw)
+        r = f.read(4)
+        assert r == "a\nb\n"
+        f.close()

Modified: pypy/branch/fast-forward/pypy/module/thread/os_lock.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/thread/os_lock.py	(original)
+++ pypy/branch/fast-forward/pypy/module/thread/os_lock.py	Mon Oct 25 14:16:59 2010
@@ -30,6 +30,7 @@
     "A wrappable box around an interp-level lock object."
 
     def __init__(self, space):
+        self.space = space
         try:
             self.lock = thread.allocate_lock()
         except thread.error:
@@ -70,6 +71,13 @@
     def descr__exit__(self, space, __args__):
         self.descr_lock_release(space)
 
+    def __enter__(self):
+        self.descr_lock_acquire(self.space)
+        return self
+
+    def __exit__(self, *args):
+        self.descr_lock_release(self.space)
+
 descr_acquire = interp2app(Lock.descr_lock_acquire,
                            unwrap_spec=['self', ObjSpace, int])
 descr_release = interp2app(Lock.descr_lock_release,


From arigo at codespeak.net  Mon Oct 25 14:42:03 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 25 Oct 2010 14:42:03 +0200 (CEST)
Subject: [pypy-svn] r78258 -
	pypy/branch/leak-finder-more/pypy/module/_rawffi/test
Message-ID: <20101025124203.22A5736E363@codespeak.net>

Author: arigo
Date: Mon Oct 25 14:42:01 2010
New Revision: 78258

Modified:
   pypy/branch/leak-finder-more/pypy/module/_rawffi/test/test_nested.py
Log:
Fix.


Modified: pypy/branch/leak-finder-more/pypy/module/_rawffi/test/test_nested.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/module/_rawffi/test/test_nested.py	(original)
+++ pypy/branch/leak-finder-more/pypy/module/_rawffi/test/test_nested.py	Mon Oct 25 14:42:01 2010
@@ -107,7 +107,6 @@
         assert S.fieldoffset('x') == 0
         assert S.fieldoffset('ar') == A5alignment
         s = S()
-        s = S()
         s.x = 'G'
         raises(TypeError, 's.ar')
         assert s.fieldaddress('ar') == s.buffer + S.fieldoffset('ar')


From arigo at codespeak.net  Mon Oct 25 14:43:04 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 25 Oct 2010 14:43:04 +0200 (CEST)
Subject: [pypy-svn] r78259 - pypy/branch/leak-finder-more/pypy/rlib
Message-ID: <20101025124304.40688282C06@codespeak.net>

Author: arigo
Date: Mon Oct 25 14:43:02 2010
New Revision: 78259

Modified:
   pypy/branch/leak-finder-more/pypy/rlib/clibffi.py
Log:
Another 'track_allocation=False'.


Modified: pypy/branch/leak-finder-more/pypy/rlib/clibffi.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/rlib/clibffi.py	(original)
+++ pypy/branch/leak-finder-more/pypy/rlib/clibffi.py	Mon Oct 25 14:43:02 2010
@@ -432,7 +432,8 @@
                  flags=FUNCFLAG_CDECL):
         AbstractFuncPtr.__init__(self, "callback", argtypes, restype, flags)
         self.ll_closure = closureHeap.alloc()
-        self.ll_userdata = lltype.malloc(USERDATA_P.TO, flavor='raw')
+        self.ll_userdata = lltype.malloc(USERDATA_P.TO, flavor='raw',
+                                         track_allocation=False)
         self.ll_userdata.callback = rffi.llhelper(CALLBACK_TP, func)
         self.ll_userdata.addarg = additional_arg
         res = c_ffi_prep_closure(self.ll_closure, self.ll_cif,
@@ -447,7 +448,7 @@
             closureHeap.free(self.ll_closure)
             self.ll_closure = lltype.nullptr(FFI_CLOSUREP.TO)
         if self.ll_userdata:
-            lltype.free(self.ll_userdata, flavor='raw')
+            lltype.free(self.ll_userdata, flavor='raw', track_allocation=False)
             self.ll_userdata = lltype.nullptr(USERDATA_P.TO)
 
 class RawFuncPtr(AbstractFuncPtr):


From arigo at codespeak.net  Mon Oct 25 14:44:43 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 25 Oct 2010 14:44:43 +0200 (CEST)
Subject: [pypy-svn] r78260 -
	pypy/branch/leak-finder-more/pypy/module/_rawffi/test
Message-ID: <20101025124443.7512F36E363@codespeak.net>

Author: arigo
Date: Mon Oct 25 14:44:41 2010
New Revision: 78260

Modified:
   pypy/branch/leak-finder-more/pypy/module/_rawffi/test/test__rawffi.py
Log:
Fix.


Modified: pypy/branch/leak-finder-more/pypy/module/_rawffi/test/test__rawffi.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/module/_rawffi/test/test__rawffi.py	(original)
+++ pypy/branch/leak-finder-more/pypy/module/_rawffi/test/test__rawffi.py	Mon Oct 25 14:44:41 2010
@@ -296,6 +296,7 @@
         assert _rawffi.charp2string(res[0]) is None
         arg1.free()
         arg2.free()
+        a.free()
 
     def test_raw_callable(self):
         import _rawffi


From arigo at codespeak.net  Mon Oct 25 14:47:33 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Mon, 25 Oct 2010 14:47:33 +0200 (CEST)
Subject: [pypy-svn] r78261 - pypy/branch/leak-finder-more/pypy/module/rctime
Message-ID: <20101025124733.7E788282C06@codespeak.net>

Author: arigo
Date: Mon Oct 25 14:47:31 2010
New Revision: 78261

Modified:
   pypy/branch/leak-finder-more/pypy/module/rctime/interp_time.py
Log:
Fix.


Modified: pypy/branch/leak-finder-more/pypy/module/rctime/interp_time.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/module/rctime/interp_time.py	(original)
+++ pypy/branch/leak-finder-more/pypy/module/rctime/interp_time.py	Mon Oct 25 14:47:31 2010
@@ -69,7 +69,7 @@
 CLOCKS_PER_SEC = cConfig.CLOCKS_PER_SEC
 clock_t = cConfig.clock_t
 tm = cConfig.tm
-glob_buf = lltype.malloc(tm, flavor='raw', zero=True)
+glob_buf = lltype.malloc(tm, flavor='raw', zero=True, immortal=True)
 
 if cConfig.has_gettimeofday:
     c_gettimeofday = external('gettimeofday', [rffi.VOIDP, rffi.VOIDP], rffi.INT)


From david at codespeak.net  Mon Oct 25 16:41:21 2010
From: david at codespeak.net (david at codespeak.net)
Date: Mon, 25 Oct 2010 16:41:21 +0200 (CEST)
Subject: [pypy-svn] r78264 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101025144121.75D43282BE3@codespeak.net>

Author: david
Date: Mon Oct 25 16:41:19 2010
New Revision: 78264

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
Log:
(david, arigo) Add breakpoint instruction

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Mon Oct 25 16:41:19 2010
@@ -37,6 +37,9 @@
         else:
             raise NotImplentedError
 
+    def BKPT(self, cond=cond.AL):
+        self.write32(cond << 28 | 0x1200070)
+
     def _encode_reg_list(self, instr, regs):
         for reg in regs:
             instr |= 0x1 << reg
@@ -87,7 +90,7 @@
 
     def __init__(self):
         map_size = 1024
-        data = alloc(1024)
+        data = alloc(map_size)
         self._pos = 0
         self._init(data, map_size)
 


From david at codespeak.net  Mon Oct 25 16:45:59 2010
From: david at codespeak.net (david at codespeak.net)
Date: Mon, 25 Oct 2010 16:45:59 +0200 (CEST)
Subject: [pypy-svn] r78267 -
	pypy/branch/arm-backend/pypy/jit/backend/arm/tools
Message-ID: <20101025144559.82B2A282BE3@codespeak.net>

Author: david
Date: Mon Oct 25 16:45:58 2010
New Revision: 78267

Added:
   pypy/branch/arm-backend/pypy/jit/backend/arm/tools/
   pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py   (contents, props changed)
Log:
Wrapper for calling objdump on dumped memory

Added: pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py
==============================================================================
--- (empty file)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py	Mon Oct 25 16:45:58 2010
@@ -0,0 +1,5 @@
+#!/usr/bin/env python
+import os
+import sys
+
+os.system('objdump -D --architecture=arm --target=binary %s' % sys.argv[1])


From david at codespeak.net  Mon Oct 25 16:47:23 2010
From: david at codespeak.net (david at codespeak.net)
Date: Mon, 25 Oct 2010 16:47:23 +0200 (CEST)
Subject: [pypy-svn] r78268 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101025144723.0BCB9282BE3@codespeak.net>

Author: david
Date: Mon Oct 25 16:47:22 2010
New Revision: 78268

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py
Log:
(david, arigo) Encode and decode args and registers for jumps and bridges

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Mon Oct 25 16:47:22 2010
@@ -123,19 +123,15 @@
     def gen_func_prolog(self):
         self.mc.PUSH(r.callee_saved_registers)
 
-    def gen_bootstrap_code(self, inputargs, regalloc):
-        for i in range(len(inputargs)):
-            reg = regalloc.try_allocate_reg(inputargs[i])
-            addr = self.fail_boxes_int.get_addr_for_num(i)
-            self.mc.gen_load_int(reg, addr)
-            self.mc.LDR_ri(reg, reg)
-
-    def gen_bridge_bootstrap_code(self, inputargs, regalloc):
+    def gen_bootstrap_code(self, inputargs, regalloc, looptoken):
+        regs = []
         for i in range(len(inputargs)):
             reg = regalloc.try_allocate_reg(inputargs[i])
             addr = self.fail_boxes_int.get_addr_for_num(i)
             self.mc.gen_load_int(reg, addr)
             self.mc.LDR_ri(reg, reg)
+            regs.append(reg)
+        looptoken._arm_arglocs = regs
 
     # cpu interface
     def assemble_loop(self, inputargs, operations, looptoken):
@@ -143,7 +139,7 @@
         regalloc = ARMRegisterManager(longevity, assembler=self.mc)
         loop_start=self.mc.curraddr()
         self.gen_func_prolog()
-        self.gen_bootstrap_code(inputargs, regalloc)
+        self.gen_bootstrap_code(inputargs, regalloc, looptoken)
         loop_head=self.mc.curraddr()
         looptoken._arm_bootstrap_code = loop_start
         looptoken._arm_loop_code = loop_head
@@ -158,11 +154,13 @@
         print 'Done assembling'
 
     def assemble_bridge(self, faildescr, inputargs, operations):
-        # XXX need to restore args here
+        enc = rffi.cast(rffi.CCHARP, faildescr._failure_recovery_code)
         longevity = compute_vars_longevity(inputargs, operations)
         regalloc = ARMRegisterManager(longevity, assembler=self.mc)
+
+        regalloc.update_bindings(enc, inputargs)
         bridge_head = self.mc.curraddr()
-        self.gen_bridge_bootstrap_code(inputargs, regalloc)
+
         fcond = c.AL
         for op in operations:
             opnum = op.getopnum()
@@ -188,14 +186,13 @@
 
 
     # Resoperations
-
     def emit_op_jump(self, op, regalloc, fcond):
         tmp = Box()
         tmpreg = regalloc.try_allocate_reg(tmp)
-        inputargs = op.getdescr()._temp_inputargs
+        registers = op.getdescr()._arm_arglocs
         for i in range(op.numargs()):
             reg = regalloc.try_allocate_reg(op.getarg(i))
-            inpreg = regalloc.try_allocate_reg(inputargs[i])
+            inpreg = registers[i]
             # XXX only if every value is in a register
             self.mc.MOV_rr(inpreg, reg)
         loop_code = op.getdescr()._arm_loop_code
@@ -226,9 +223,9 @@
         assert fcond == c.GT
         descr = op.getdescr()
         assert isinstance(descr, BasicFailDescr)
+        descr._arm_guard_code = self.mc.curraddr()
         memaddr = self._gen_path_to_exit_path(op, op.getfailargs(), regalloc, fcond)
         descr._failure_recovery_code = memaddr
-        descr._arm_guard_code = self.mc.curraddr()
         descr._arm_guard_cond = fcond
         return c.AL
 

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py	Mon Oct 25 16:47:22 2010
@@ -11,6 +11,10 @@
     def __init__(self, longevity, frame_manager=None, assembler=None):
         RegisterManager.__init__(self, longevity, frame_manager, assembler)
 
+    def update_bindings(self, enc, inputargs):
+        for i in range(len(inputargs)):
+           self.try_allocate_reg(inputargs[i], ord(enc[i]))
+
 class ARMFrameManager(FrameManager):
     @staticmethod
     def frame_pos(loc, type):


From fijall at gmail.com  Mon Oct 25 16:50:50 2010
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 25 Oct 2010 16:50:50 +0200
Subject: [pypy-svn] r78267 -
	pypy/branch/arm-backend/pypy/jit/backend/arm/tools
In-Reply-To: <20101025144559.82B2A282BE3@codespeak.net>
References: <20101025144559.82B2A282BE3@codespeak.net>
Message-ID: 

Hey.

Can we make this directory 'tool' instead of 'tools' like every other
tool directory?

Cheers,
fijal

On Mon, Oct 25, 2010 at 4:45 PM,   wrote:
> Author: david
> Date: Mon Oct 25 16:45:58 2010
> New Revision: 78267
>
> Added:
> ? pypy/branch/arm-backend/pypy/jit/backend/arm/tools/
> ? pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py ? (contents, props changed)
> Log:
> Wrapper for calling objdump on dumped memory
>
> Added: pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py
> ==============================================================================
> --- (empty file)
> +++ pypy/branch/arm-backend/pypy/jit/backend/arm/tools/objdump.py ? ? ? Mon Oct 25 16:45:58 2010
> @@ -0,0 +1,5 @@
> +#!/usr/bin/env python
> +import os
> +import sys
> +
> +os.system('objdump -D --architecture=arm --target=binary %s' % sys.argv[1])
> _______________________________________________
> pypy-svn mailing list
> pypy-svn at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-svn
>


From david at codespeak.net  Mon Oct 25 16:54:03 2010
From: david at codespeak.net (david at codespeak.net)
Date: Mon, 25 Oct 2010 16:54:03 +0200 (CEST)
Subject: [pypy-svn] r78269 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: tool tools
Message-ID: <20101025145403.6F926282C06@codespeak.net>

Author: david
Date: Mon Oct 25 16:54:02 2010
New Revision: 78269

Added:
   pypy/branch/arm-backend/pypy/jit/backend/arm/tool/
      - copied from r78267, pypy/branch/arm-backend/pypy/jit/backend/arm/tools/
Removed:
   pypy/branch/arm-backend/pypy/jit/backend/arm/tools/
Log:
tools -> tool


From david at codespeak.net  Mon Oct 25 17:22:48 2010
From: david at codespeak.net (david at codespeak.net)
Date: Mon, 25 Oct 2010 17:22:48 +0200 (CEST)
Subject: [pypy-svn] r78270 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101025152248.4012B282C06@codespeak.net>

Author: david
Date: Mon Oct 25 17:22:46 2010
New Revision: 78270

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py
Log:
Unify input arg and fail boxes

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Mon Oct 25 17:22:46 2010
@@ -18,7 +18,6 @@
     def __init__(self, cpu, failargs_limit=1000):
         self.mc = ARMv7Builder()
         self.cpu = cpu
-        self.input_arg_boxes_int = values_array(lltype.Signed, failargs_limit) # merge with fail_boxes_int later
         self.fail_boxes_int = values_array(lltype.Signed, failargs_limit)
         self._debug_asm = True
 

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py	Mon Oct 25 17:22:46 2010
@@ -22,7 +22,7 @@
         self.assembler.assemble_bridge(faildescr, inputargs, operations)
 
     def set_future_value_int(self, index, intvalue):
-        self.assembler.input_arg_boxes_int.setitem(index, intvalue)
+        self.assembler.fail_boxes_int.setitem(index, intvalue)
 
     def get_latest_value_int(self, index):
         return self.assembler.fail_boxes_int.getitem(index)


From afa at codespeak.net  Mon Oct 25 17:28:24 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 25 Oct 2010 17:28:24 +0200 (CEST)
Subject: [pypy-svn] r78271 - in pypy/branch/fast-forward/pypy: module/_io
	module/_io/test rlib
Message-ID: <20101025152824.553C5282C06@codespeak.net>

Author: afa
Date: Mon Oct 25 17:28:21 2010
New Revision: 78271

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
   pypy/branch/fast-forward/pypy/rlib/streamio.py
Log:
Add BufferedReader.seek(),
and fix buffer management.


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Mon Oct 25 17:28:21 2010
@@ -3,17 +3,23 @@
     TypeDef, generic_new_descr)
 from pypy.interpreter.gateway import interp2app, unwrap_spec
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
-from pypy.interpreter.error import OperationError
+from pypy.interpreter.error import OperationError, operationerrfmt
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.rlib.rstring import StringBuilder
+from pypy.rlib.rarithmetic import r_longlong
 from pypy.module._io.interp_iobase import W_IOBase, convert_size
 from pypy.module._io.interp_io import DEFAULT_BUFFER_SIZE
 from pypy.module.thread.os_lock import Lock
 
+class BlockingIOError(Exception):
+    pass
+
 class W_BufferedIOBase(W_IOBase):
     def __init__(self, space):
         W_IOBase.__init__(self, space)
         self.buffer = lltype.nullptr(rffi.CCHARP.TO)
+        self.pos = 0     # Current logical position in the buffer
+        self.raw_pos = 0 # Position of the raw stream in the buffer.
         self.lock = None
 
         self.readable = False
@@ -54,6 +60,13 @@
             return self.read_end - self.pos
         return 0
 
+    def _raw_offset(self):
+        if self.raw_pos and (
+            (self.readable and self.read_end != -1) or
+            (self.writable and self.write_end != -1)):
+            return self.raw_pos - self.pos
+        return 0
+
     def _unsupportedoperation(self, space, message):
         w_exc = space.getattr(space.getbuiltinmodule('_io'),
                               space.wrap('UnsupportedOperation'))
@@ -63,10 +76,55 @@
     def read_w(self, space, w_size=None):
         self._unsupportedoperation(space, "read")
 
+    @unwrap_spec('self', ObjSpace, r_longlong, int)
+    def seek_w(self, space, pos, whence=0):
+        if whence not in (0, 1, 2):
+            raise operationerrfmt(space.w_ValueError,
+                "whence must be between 0 and 2, not %d", whence)
+        self._check_closed(space, "seek of closed file")
+        if whence != 2 and self.readable:
+            # Check if seeking leaves us inside the current buffer, so as to
+            # return quickly if possible. Also, we needn't take the lock in
+            # this fast path.
+            current = self._raw_tell(space)
+            available = self._readahead()
+            if available > 0:
+                if whence == 0:
+                    offset = pos - (current - self._raw_offset())
+                else:
+                    offset = pos
+                if -self.pos <= offset <= available:
+                    self.pos += offset
+                    return space.wrap(current - available + offset)
+
+        # Fallback: invoke raw seek() method and clear buffer
+        with self.lock:
+            if self.writable:
+                self._writer_flush_unlocked(restore_pos=False)
+                self._writer_reset_buf()
+
+            if whence == 1:
+                pos -= self._raw_offset()
+            n = self._raw_seek(space, pos, whence)
+            self.raw_pos = -1
+            if self.readable:
+                self._reader_reset_buf()
+            return space.wrap(n)
+
+    def _raw_seek(self, space, pos, whence):
+        w_pos = space.call_method(self.raw, "seek",
+                                  space.wrap(pos), space.wrap(whence))
+        pos = space.r_longlong_w(w_pos)
+        if pos < 0:
+            raise OperationError(space.w_IOError, space.wrap(
+                "Raw stream returned invalid position"))
+        return pos
+
 W_BufferedIOBase.typedef = TypeDef(
     '_BufferedIOBase', W_IOBase.typedef,
     __new__ = generic_new_descr(W_BufferedIOBase),
     read = interp2app(W_BufferedIOBase.read_w),
+    seek = interp2app(W_BufferedIOBase.seek_w),
     )
 
 class W_BufferedReader(W_BufferedIOBase):
@@ -85,9 +143,9 @@
         self.readable = True
 
         self._init(space)
-        self._reset_buf()
+        self._reader_reset_buf()
 
-    def _reset_buf(self):
+    def _reader_reset_buf(self):
         self.read_end = -1
 
     def _closed(self, space):
@@ -110,6 +168,7 @@
         return space.wrap(res)
 
     def _read_all(self, space):
+        "Read all the file, don't update the cache"
         builder = StringBuilder()
         # First copy what we have in the current buffer
         current_size = self._readahead()
@@ -118,7 +177,7 @@
             data = rffi.charpsize2str(rffi.ptradd(self.buffer, self.pos),
                                       current_size)
             builder.append(data)
-        self._reset_buf()
+        self._reader_reset_buf()
         # We're going past the buffer's bounds, flush it
         if self.writable:
             self._writer_flush_unlocked(restore_pos=True)
@@ -138,6 +197,28 @@
                 self.abs_pos += size
         return builder.build()
 
+    def _raw_read(self, space, n):
+        w_data = space.call_method(self.raw, "read", space.wrap(n))
+        if space.is_w(w_data, space.w_None):
+            raise BlockingIOError()
+        data = space.str_w(w_data)
+        if self.abs_pos != -1:
+            self.abs_pos += len(data)
+        return data
+
+    def _fill_buffer(self, space):
+        start = self.read_end
+        if start == -1:
+            start = 0
+        length = self.buffer_size - start
+        data = self._raw_read(space, length)
+        size = len(data)
+        if size > 0:
+            for i in range(start, start + size):
+                self.buffer[i] = data[i]
+            self.read_end = self.raw_pos = start + size
+        return size
+
     def _read_generic(self, space, n):
         """Generic read function: read from the stream until enough bytes are
            read, or until an EOF occurs or until read() would block."""
@@ -153,27 +234,56 @@
             data = rffi.charpsize2str(rffi.ptradd(self.buffer, self.pos),
                                       current_size)
             builder.append(data)
-        self._reset_buf()
+        self._reader_reset_buf()
 
         # XXX potential bug in CPython? The following is not enabled.
         # We're going past the buffer's bounds, flush it
         ## if self.writable:
         ##     self._writer_flush_unlocked(restore_pos=True)
 
+        # Read whole blocks, and don't buffer them
         while remaining > 0:
-            # Read until EOF or until read() would block
-            w_data = space.call_method(self.raw, "read", space.wrap(remaining))
-            if space.is_w(w_data, space.w_None):
+            r = self.buffer_size * (remaining // self.buffer_size)
+            if r == 0:
                 break
-            data = space.str_w(w_data)
+            try:
+                data = self._raw_read(space, r)
+            except BlockingIOError:
+                if written == 0:
+                    return None
+                data = ""
             size = len(data)
             if size == 0:
-                break
-            builder.append(data)
-            current_size += size
+                return builder.build()
             remaining -= size
-            if self.abs_pos != -1:
-                self.abs_pos += size
+            written += size
+
+        self.pos = 0
+        self.raw_pos = 0
+        self.read_end = 0
+
+        while remaining > 0 and self.read_end < self.buffer_size:
+            # Read until EOF or until read() would block
+            try:
+                size = self._fill_buffer(space)
+            except BlockingIOError:
+                if written == 0:
+                    return None
+                break
+
+            if remaining > 0:
+                if size > remaining:
+                    size = remaining
+                # XXX inefficient
+                l = []
+                for i in range(self.pos,self.pos + size):
+                    l.append(self.buffer[i])
+                data = ''.join(l)
+                builder.append(data)
+
+                written += size
+                self.pos += size
+                remaining -= size
         return builder.build()
 
     def _read_fast(self, n):

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Mon Oct 25 17:28:21 2010
@@ -20,3 +20,14 @@
         r = f.read(4)
         assert r == "a\nb\n"
         f.close()
+
+    def test_seek(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile)
+        f = _io.BufferedReader(raw)
+        assert f.read() == "a\nb\nc"
+        f.seek(0)
+        assert f.read() == "a\nb\nc"
+        f.seek(-2, 2)
+        assert f.read() == "\nc"
+        f.close()

Modified: pypy/branch/fast-forward/pypy/rlib/streamio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/streamio.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/streamio.py	Mon Oct 25 17:28:21 2010
@@ -200,6 +200,7 @@
                 raise WindowsError(rwin32.GetLastError(),
                                    "Could not truncate file")
         finally:
+            # we restore the file pointer position in any case
             os.lseek(fd, curpos, 0)
 
 


From david at codespeak.net  Mon Oct 25 17:41:25 2010
From: david at codespeak.net (david at codespeak.net)
Date: Mon, 25 Oct 2010 17:41:25 +0200 (CEST)
Subject: [pypy-svn] r78272 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101025154125.3F4C8282C08@codespeak.net>

Author: david
Date: Mon Oct 25 17:41:23 2010
New Revision: 78272

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py
Log:
(david, arigo) handle holes in bridges

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Mon Oct 25 17:41:23 2010
@@ -49,6 +49,8 @@
         while(True):
             i += 1
             r = enc[i]
+            if r == '\xFE':
+                continue
             if r == '\xFF':
                 break
             reg = ord(enc[i])
@@ -96,8 +98,12 @@
         # XXX free this memory
         mem = lltype.malloc(rffi.CArray(lltype.Char), len(args)+5, flavor='raw')
         for i in range(len(args)):
-            curreg = regalloc.try_allocate_reg(args[i])
-            mem[i] = chr(curreg)
+            if args[i]:
+                curreg = regalloc.try_allocate_reg(args[i])
+                mem[i] = chr(curreg)
+            else:
+                mem[i] = '\xFE'
+
         i = len(args)
         mem[i] = chr(0xFF)
         memaddr = rffi.cast(lltype.Signed, mem)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py	Mon Oct 25 17:41:23 2010
@@ -12,8 +12,12 @@
         RegisterManager.__init__(self, longevity, frame_manager, assembler)
 
     def update_bindings(self, enc, inputargs):
+        j = 0
         for i in range(len(inputargs)):
-           self.try_allocate_reg(inputargs[i], ord(enc[i]))
+            while enc[j] == '\xFE':
+                j += 1
+            self.try_allocate_reg(inputargs[i], ord(enc[j]))
+            j += 1
 
 class ARMFrameManager(FrameManager):
     @staticmethod


From david at codespeak.net  Mon Oct 25 17:47:26 2010
From: david at codespeak.net (david at codespeak.net)
Date: Mon, 25 Oct 2010 17:47:26 +0200 (CEST)
Subject: [pypy-svn] r78273 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101025154726.33D0A282C08@codespeak.net>

Author: david
Date: Mon Oct 25 17:47:24 2010
New Revision: 78273

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py
Log:
(david, arigo) implement get_latest_value_count

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Mon Oct 25 17:47:24 2010
@@ -57,6 +57,7 @@
             self.fail_boxes_int.setitem(i, self.decode32(stack, reg*WORD))
         assert enc[i] == '\xFF'
         descr = self.decode32(enc, i+1)
+        self.fail_boxes_count = i
         return descr
 
     def decode32(self, mem, index):

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/runner.py	Mon Oct 25 17:47:24 2010
@@ -27,6 +27,13 @@
     def get_latest_value_int(self, index):
         return self.assembler.fail_boxes_int.getitem(index)
 
+    def get_latest_value_count(self):
+        return self.assembler.fail_boxes_count
+
+    def clear_latest_values(self, count):
+        # XXX TODO
+        pass
+
     def execute_token(self, executable_token):
         addr = executable_token._arm_bootstrap_code
         assert addr % 8 == 0


From afa at codespeak.net  Mon Oct 25 18:21:01 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 25 Oct 2010 18:21:01 +0200 (CEST)
Subject: [pypy-svn] r78274 - pypy/branch/leak-finder-more/pypy/module/cpyext
Message-ID: <20101025162101.14732282BE7@codespeak.net>

Author: afa
Date: Mon Oct 25 18:20:59 2010
New Revision: 78274

Modified:
   pypy/branch/leak-finder-more/pypy/module/cpyext/api.py
Log:
Fix memory leak when dlopen() fails.


Modified: pypy/branch/leak-finder-more/pypy/module/cpyext/api.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/module/cpyext/api.py	(original)
+++ pypy/branch/leak-finder-more/pypy/module/cpyext/api.py	Mon Oct 25 18:20:59 2010
@@ -908,8 +908,10 @@
         from pypy.rlib import rdynload
         try:
             ll_libname = rffi.str2charp(path)
-            dll = rdynload.dlopen(ll_libname)
-            lltype.free(ll_libname, flavor='raw')
+            try:
+                dll = rdynload.dlopen(ll_libname)
+            finally:
+                lltype.free(ll_libname, flavor='raw')
         except rdynload.DLOpenError, e:
             raise operationerrfmt(
                 space.w_ImportError,


From afa at codespeak.net  Mon Oct 25 18:22:29 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 25 Oct 2010 18:22:29 +0200 (CEST)
Subject: [pypy-svn] r78275 - pypy/branch/leak-finder-more/pypy/module/cpyext
Message-ID: <20101025162229.8630B282BE7@codespeak.net>

Author: afa
Date: Mon Oct 25 18:22:28 2010
New Revision: 78275

Modified:
   pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py
Log:
This function is a @specialize.memo() and will be called only once.
Mark allocations as immortal.


Modified: pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py	(original)
+++ pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py	Mon Oct 25 18:22:28 2010
@@ -193,10 +193,12 @@
     if state.new_method_def:
         return state.new_method_def
     from pypy.module.cpyext.modsupport import PyMethodDef
-    ptr = lltype.malloc(PyMethodDef, flavor="raw", zero=True)
-    ptr.c_ml_name = rffi.str2charp("__new__")
+    ptr = lltype.malloc(PyMethodDef, flavor="raw", zero=True,
+                        immortal=True)
+    ptr.c_ml_name = rffi.str2charp_immortal("__new__")
     rffi.setintfield(ptr, 'c_ml_flags', METH_VARARGS | METH_KEYWORDS)
-    ptr.c_ml_doc = rffi.str2charp("T.__new__(S, ...) -> a new object with type S, a subtype of T")
+    ptr.c_ml_doc = rffi.str2charp_immortal(
+        "T.__new__(S, ...) -> a new object with type S, a subtype of T")
     state.new_method_def = ptr
     return ptr
 


From cfbolz at codespeak.net  Mon Oct 25 18:46:34 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Mon, 25 Oct 2010 18:46:34 +0200 (CEST)
Subject: [pypy-svn] r78276 - in pypy/trunk/pypy/objspace/std: . test
Message-ID: <20101025164634.CA26736E393@codespeak.net>

Author: cfbolz
Date: Mon Oct 25 18:46:32 2010
New Revision: 78276

Modified:
   pypy/trunk/pypy/objspace/std/test/test_typeobject.py
   pypy/trunk/pypy/objspace/std/typeobject.py
Log:
(cfbolz, arigo): you learn something new every day. in cpython you can add slots
that exist as methods as well and it "works": the slot is mostly ignored.


Modified: pypy/trunk/pypy/objspace/std/test/test_typeobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_typeobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_typeobject.py	Mon Oct 25 18:46:32 2010
@@ -1085,4 +1085,12 @@
 
         assert b == 1
 
-        
+    def test_slots_with_method_in_class(self):
+        # this works in cpython...
+        class A(object):
+            __slots__ = ["f"]
+            def f(self, x):
+                return x + 1
+        a = A()
+        assert a.f(1) == 2
+

Modified: pypy/trunk/pypy/objspace/std/typeobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/typeobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/typeobject.py	Mon Oct 25 18:46:32 2010
@@ -561,9 +561,11 @@
     slot_name = _mangle(slot_name, w_self.name)
     # Force interning of slot names.
     slot_name = space.str_w(space.new_interned_str(slot_name))
-    member = Member(w_self.nslots, slot_name, w_self)
-    w_self.dict_w[slot_name] = space.wrap(member)
-    w_self.nslots += 1
+    if slot_name not in w_self.dict_w:
+        # in cpython it is ignored less, but we probably don't care
+        member = Member(w_self.nslots, slot_name, w_self)
+        w_self.dict_w[slot_name] = space.wrap(member)
+        w_self.nslots += 1
 
 def create_dict_slot(w_self):
     if not w_self.hasdict:


From hakanardo at codespeak.net  Mon Oct 25 21:39:00 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Mon, 25 Oct 2010 21:39:00 +0200 (CEST)
Subject: [pypy-svn] r78278 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101025193900.BD000282BDA@codespeak.net>

Author: hakanardo
Date: Mon Oct 25 21:38:58 2010
New Revision: 78278

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py
Log:
Some tests ajusted to reflect the new optimization

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	Mon Oct 25 21:38:58 2010
@@ -98,7 +98,7 @@
             return res
         res = self.meta_interp(f, [4, -1])
         assert res == 145
-        self.check_loops(int_add = 1)
+        self.check_loops(int_add = 1, everywhere=True)
 
     def test_oosend_base(self):
         myjitdriver = JitDriver(greens = [], reds = ['x', 'y', 'w'])

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py	Mon Oct 25 21:38:58 2010
@@ -316,10 +316,9 @@
 
         self.check_tree_loop_count(2)      # the loop and the entry path
         # we get:
-        #    ENTER             - compile the new loop
-        #    ENTER             - compile the entry bridge
+        #    ENTER             - compile the new loop and entry bridge
         #    ENTER             - compile the leaving path
-        self.check_enter_count(3)
+        self.check_enter_count(2)
 
 class VirtualMiscTests:
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py	Mon Oct 25 21:38:58 2010
@@ -197,7 +197,7 @@
             return xy.inst_x
         res = self.meta_interp(f, [20])
         assert res == 134
-        self.check_loops(getfield_gc=1, setfield_gc=1)
+        self.check_loops(getfield_gc=1, setfield_gc=1, everywhere=True)
 
     # ------------------------------
 
@@ -1124,7 +1124,7 @@
  
          res = self.meta_interp(f, [10])
          assert res == 55
-         self.check_loops(new_with_vtable=0, ptr_eq=1)
+         self.check_loops(new_with_vtable=0, ptr_eq=1, everywhere=True)
 
     def test_virtual_child_frame_with_arrays(self):
         myjitdriver = JitDriver(greens = [], reds = ['frame'],


From agaynor at codespeak.net  Mon Oct 25 22:56:22 2010
From: agaynor at codespeak.net (agaynor at codespeak.net)
Date: Mon, 25 Oct 2010 22:56:22 +0200 (CEST)
Subject: [pypy-svn] r78279 - in pypy/trunk/pypy: interpreter objspace/test
Message-ID: <20101025205622.8F29B282BDA@codespeak.net>

Author: agaynor
Date: Mon Oct 25 22:56:18 2010
New Revision: 78279

Modified:
   pypy/trunk/pypy/interpreter/function.py
   pypy/trunk/pypy/objspace/test/test_descroperation.py
Log:
Fixed instantiating subclasses of classmethod.  This was found via Django, thanks to Amaury for help with the patch.


Modified: pypy/trunk/pypy/interpreter/function.py
==============================================================================
--- pypy/trunk/pypy/interpreter/function.py	(original)
+++ pypy/trunk/pypy/interpreter/function.py	Mon Oct 25 22:56:18 2010
@@ -589,12 +589,14 @@
             w_klass = space.type(w_obj)
         return space.wrap(Method(space, self.w_function, w_klass, space.w_None))
 
-    def descr_classmethod__new__(space, w_type, w_function):
+    def descr_classmethod__new__(space, w_subtype, w_function):
         if not space.is_true(space.callable(w_function)):
             typename = space.type(w_function).getname(space, '?')
             raise operationerrfmt(space.w_TypeError,
                                   "'%s' object is not callable", typename)
-        return space.wrap(ClassMethod(w_function))
+        instance = space.allocate_instance(ClassMethod, w_subtype)
+        instance.__init__(w_function)
+        return space.wrap(instance)
 
 class FunctionWithFixedCode(Function):
     can_change_code = False

Modified: pypy/trunk/pypy/objspace/test/test_descroperation.py
==============================================================================
--- pypy/trunk/pypy/objspace/test/test_descroperation.py	(original)
+++ pypy/trunk/pypy/objspace/test/test_descroperation.py	Mon Oct 25 22:56:18 2010
@@ -400,7 +400,7 @@
         class D(object):
             def __init__(self, a):
                 self.a = a
-        
+
         assert A(1) == B(1)
         assert B(1) == A(1)
         assert A(1) == C(1)
@@ -457,6 +457,20 @@
         else:
             assert False, "did not raise"
 
+    def test_attribute_error(self):
+        class classmethodonly(classmethod):
+            def __get__(self, instance, type):
+                if instance is not None:
+                    raise AttributeError("Must be called on a class, not an instance.")
+                return super(classmethodonly, self).__get__(instance, type)
+
+        class A(object):
+            @classmethodonly
+            def a(cls):
+                return 3
+
+        raises(AttributeError, lambda: A().a)
+
 
 class AppTestWithBuiltinShortcut(AppTest_Descroperation):
     OPTIONS = {'objspace.std.builtinshortcut': True}


From afa at codespeak.net  Mon Oct 25 23:32:35 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Mon, 25 Oct 2010 23:32:35 +0200 (CEST)
Subject: [pypy-svn] r78280 - in pypy/branch/leak-finder-more/pypy:
	module/cpyext rpython/lltypesystem
Message-ID: <20101025213235.81F6F282C08@codespeak.net>

Author: afa
Date: Mon Oct 25 23:32:32 2010
New Revision: 78280

Modified:
   pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py
   pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/llarena.py
   pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/lltype.py
   pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/rffi.py
Log:
lltype.render_immortal() can mark an allocation as immortal
after the call to lltype.malloc()
Use it to kill rffi.str2charp_immortal


Modified: pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py	(original)
+++ pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py	Mon Oct 25 23:32:32 2010
@@ -195,10 +195,12 @@
     from pypy.module.cpyext.modsupport import PyMethodDef
     ptr = lltype.malloc(PyMethodDef, flavor="raw", zero=True,
                         immortal=True)
-    ptr.c_ml_name = rffi.str2charp_immortal("__new__")
+    ptr.c_ml_name = rffi.str2charp("__new__")
+    lltype.render_immortal(ptr.c_ml_name)
     rffi.setintfield(ptr, 'c_ml_flags', METH_VARARGS | METH_KEYWORDS)
-    ptr.c_ml_doc = rffi.str2charp_immortal(
+    ptr.c_ml_doc = rffi.str2charp(
         "T.__new__(S, ...) -> a new object with type S, a subtype of T")
+    lltype.render_immortal(ptr.c_ml_doc)
     state.new_method_def = ptr
     return ptr
 

Modified: pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/llarena.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/llarena.py	(original)
+++ pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/llarena.py	Mon Oct 25 23:32:32 2010
@@ -440,7 +440,8 @@
                                    [rffi.INT],
                                    rffi.INT,
                                    sandboxsafe=True, _nowrapper=True)
-    _dev_zero = rffi.str2charp_immortal('/dev/zero')   # prebuilt
+    _dev_zero = rffi.str2charp('/dev/zero')   # prebuilt
+    lltype.render_immortal(_dev_zero)
 
     def clear_large_memory_chunk(baseaddr, size):
         # on some Unixy platforms, reading from /dev/zero is the fastest way

Modified: pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/lltype.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/lltype.py	(original)
+++ pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/lltype.py	Mon Oct 25 23:32:32 2010
@@ -1868,6 +1868,13 @@
         leakfinder.remember_free(p._obj0)
     p._obj0._free()
 
+def render_immortal(p, track_allocation=True):
+    T = typeOf(p)
+    if not isinstance(T, Ptr) or p._togckind() != 'raw':
+        raise TypeError, "free(): only for pointers to non-gc containers"
+    if track_allocation:
+        leakfinder.remember_free(p._obj0)
+
 def _make_scoped_allocator(T):
     class ScopedAlloc:
         def __init__(self, n=None, zero=False):

Modified: pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/rffi.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/rffi.py	(original)
+++ pypy/branch/leak-finder-more/pypy/rpython/lltypesystem/rffi.py	Mon Oct 25 23:32:32 2010
@@ -607,15 +607,6 @@
         return array
     str2charp._annenforceargs_ = [strtype]
 
-    def str2charp_immortal(s):
-        "NOT_RPYTHON"
-        array = lltype.malloc(TYPEP.TO, len(s) + 1, flavor='raw',
-                              immortal=True)
-        for i in range(len(s)):
-            array[i] = s[i]
-        array[len(s)] = lastchar
-        return array
-
     def free_charp(cp):
         lltype.free(cp, flavor='raw')
 
@@ -734,19 +725,19 @@
         l = [cp[i] for i in range(size)]
         return emptystr.join(l)
 
-    return (str2charp, str2charp_immortal, free_charp, charp2str,
+    return (str2charp, free_charp, charp2str,
             get_nonmovingbuffer, free_nonmovingbuffer,
             alloc_buffer, str_from_buffer, keep_buffer_alive_until_here,
             charp2strn, charpsize2str,
             )
 
-(str2charp, str2charp_immortal, free_charp, charp2str,
+(str2charp, free_charp, charp2str,
  get_nonmovingbuffer, free_nonmovingbuffer,
  alloc_buffer, str_from_buffer, keep_buffer_alive_until_here,
  charp2strn, charpsize2str,
  ) = make_string_mappings(str)
 
-(unicode2wcharp, unicode2wcharp_immortal, free_wcharp, wcharp2unicode,
+(unicode2wcharp, free_wcharp, wcharp2unicode,
  get_nonmoving_unicodebuffer, free_nonmoving_unicodebuffer,
  alloc_unicodebuffer, unicode_from_buffer, keep_unicodebuffer_alive_until_here,
  wcharp2unicoden, wcharpsize2unicode,


From afa at codespeak.net  Tue Oct 26 09:30:03 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 09:30:03 +0200 (CEST)
Subject: [pypy-svn] r78281 - in pypy/branch/leak-finder-more/pypy:
	annotation module/cpyext
Message-ID: <20101026073003.A6009282C14@codespeak.net>

Author: afa
Date: Tue Oct 26 09:29:56 2010
New Revision: 78281

Modified:
   pypy/branch/leak-finder-more/pypy/annotation/builtin.py
   pypy/branch/leak-finder-more/pypy/module/cpyext/cdatetime.py
   pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py
Log:
Reduce the number of reported leaks in the cpyext module,
and try to fix translation


Modified: pypy/branch/leak-finder-more/pypy/annotation/builtin.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/annotation/builtin.py	(original)
+++ pypy/branch/leak-finder-more/pypy/annotation/builtin.py	Tue Oct 26 09:29:56 2010
@@ -453,6 +453,9 @@
     #p = lltype.malloc(T, flavor=s_flavor.const)
     #lltype.free(p, flavor=s_flavor.const)
 
+def render_immortal(s_p, s_track_allocation=None):
+    assert s_track_allocation is None or s_track_allocation.is_constant()
+
 def typeOf(s_val):
     lltype = annotation_to_lltype(s_val, info="in typeOf(): ")
     return immutablevalue(lltype)
@@ -520,6 +523,7 @@
 
 BUILTIN_ANALYZERS[lltype.malloc] = malloc
 BUILTIN_ANALYZERS[lltype.free] = free
+BUILTIN_ANALYZERS[lltype.render_immortal] = render_immortal
 BUILTIN_ANALYZERS[lltype.typeOf] = typeOf
 BUILTIN_ANALYZERS[lltype.cast_primitive] = cast_primitive
 BUILTIN_ANALYZERS[lltype.nullptr] = nullptr

Modified: pypy/branch/leak-finder-more/pypy/module/cpyext/cdatetime.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/module/cpyext/cdatetime.py	(original)
+++ pypy/branch/leak-finder-more/pypy/module/cpyext/cdatetime.py	Tue Oct 26 09:29:56 2010
@@ -4,7 +4,7 @@
 from pypy.module.cpyext.api import (
     cpython_api, CANNOT_FAIL, cpython_struct, PyObjectFields)
 from pypy.module.cpyext.import_ import PyImport_Import
-from pypy.module.cpyext.typeobject import PyTypeObjectPtr
+from pypy.module.cpyext.typeobject import PyTypeObjectPtr, render_immortal
 from pypy.module.cpyext.state import State
 from pypy.interpreter.error import OperationError
 from pypy.tool.sourcetools import func_renamer
@@ -22,25 +22,34 @@
 @cpython_api([], lltype.Ptr(PyDateTime_CAPI),
              error=lltype.nullptr(PyDateTime_CAPI))
 def _PyDateTime_Import(space):
-    datetimeAPI = lltype.malloc(PyDateTime_CAPI, flavor='raw')
+    datetimeAPI = lltype.malloc(PyDateTime_CAPI, flavor='raw',
+                                track_allocation=False)
 
     if not we_are_translated():
         datetimeAPI_dealloc(space)
         space.fromcache(State).datetimeAPI = datetimeAPI
 
     w_datetime = PyImport_Import(space, space.wrap("datetime"))
+
     w_type = space.getattr(w_datetime, space.wrap("date"))
     datetimeAPI.c_DateType = rffi.cast(
         PyTypeObjectPtr, make_ref(space, w_type))
+    render_immortal(datetimeAPI.c_DateType, w_type)
+
     w_type = space.getattr(w_datetime, space.wrap("datetime"))
     datetimeAPI.c_DateTimeType = rffi.cast(
         PyTypeObjectPtr, make_ref(space, w_type))
+    render_immortal(datetimeAPI.c_DateTimeType, w_type)
+
     w_type = space.getattr(w_datetime, space.wrap("time"))
     datetimeAPI.c_TimeType = rffi.cast(
         PyTypeObjectPtr, make_ref(space, w_type))
+    render_immortal(datetimeAPI.c_TimeType, w_type)
+
     w_type = space.getattr(w_datetime, space.wrap("timedelta"))
     datetimeAPI.c_DeltaType = rffi.cast(
         PyTypeObjectPtr, make_ref(space, w_type))
+    render_immortal(datetimeAPI.c_DeltaType, w_type)
 
     return datetimeAPI
 

Modified: pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py	(original)
+++ pypy/branch/leak-finder-more/pypy/module/cpyext/typeobject.py	Tue Oct 26 09:29:56 2010
@@ -538,12 +538,23 @@
     w_obj.ready()
 
     finish_type_2(space, py_type, w_obj)
+    render_immortal(py_type, w_obj)
 
     state = space.fromcache(RefcountState)
     state.non_heaptypes_w.append(w_obj)
 
     return w_obj
 
+def render_immortal(py_type, w_obj):
+    lltype.render_immortal(py_type.c_tp_bases)
+    lltype.render_immortal(py_type.c_tp_mro)
+
+    assert isinstance(w_obj, W_TypeObject)
+    if w_obj.is_cpytype():
+        lltype.render_immortal(py_type.c_tp_dict)
+    else:
+        lltype.render_immortal(py_type.c_tp_name)
+
 def finish_type_1(space, pto):
     """
     Sets up tp_bases, necessary before creating the interpreter type.


From cfbolz at codespeak.net  Tue Oct 26 10:27:09 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Tue, 26 Oct 2010 10:27:09 +0200 (CEST)
Subject: [pypy-svn] r78282 - pypy/extradoc/sprintinfo/ddorf2010
Message-ID: <20101026082709.35A47282C16@codespeak.net>

Author: cfbolz
Date: Tue Oct 26 10:27:05 2010
New Revision: 78282

Modified:
   pypy/extradoc/sprintinfo/ddorf2010/planning.txt
Log:
(all) planning for today


Modified: pypy/extradoc/sprintinfo/ddorf2010/planning.txt
==============================================================================
--- pypy/extradoc/sprintinfo/ddorf2010/planning.txt	(original)
+++ pypy/extradoc/sprintinfo/ddorf2010/planning.txt	Tue Oct 26 10:27:05 2010
@@ -9,14 +9,15 @@
 
 topics:
 
-- get mmap to work on Beagleboard (David, Armin)
-- bridges in the ARM backend (David, Armin)
-- hg conversion (Ronny, Anto)
-- memory benchmarks (Carl Friedrich, Lukas)
+- get mmap to work on Beagleboard DONE
+- bridges in the ARM backend SORT OF DONE
+- add spilling and location support (David, Armin around)
+- hg conversion WE HAVE A PLAN (Anto, Ronny)
+- memory benchmarks IN PROGRESS (Carl Friedrich, Lukas when he shows up)
 - buildbot improvements, locks, email on failing tests
 - run pypy-c-nojit in the benchmarks
 - make it possible to run benchmark on branches
-- understand problems of mapdict
+- understand problems of mapdict (Carl Friedrich, Armin)
 - rewrite ctypes to use _ffi
 - can classes with non-type metaclasses use typeversion?
 
@@ -25,3 +26,4 @@
 
 - hg migration planning
 - understand h?kan's branch
+- JIT introduction (today after lunch)


From afa at codespeak.net  Tue Oct 26 11:19:06 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 11:19:06 +0200 (CEST)
Subject: [pypy-svn] r78283 - in pypy/branch/leak-finder-more/pypy: rpython
	translator/c/test
Message-ID: <20101026091906.F1FB8282C18@codespeak.net>

Author: afa
Date: Tue Oct 26 11:19:05 2010
New Revision: 78283

Modified:
   pypy/branch/leak-finder-more/pypy/rpython/rbuiltin.py
   pypy/branch/leak-finder-more/pypy/translator/c/test/test_lltyped.py
Log:
when translated, lltype.render_immortal() generates a call to the "track_alloc_stop" operation.


Modified: pypy/branch/leak-finder-more/pypy/rpython/rbuiltin.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/rpython/rbuiltin.py	(original)
+++ pypy/branch/leak-finder-more/pypy/rpython/rbuiltin.py	Tue Oct 26 11:19:05 2010
@@ -386,6 +386,14 @@
     hop.exception_cannot_occur()
     hop.genop('free', vlist)
 
+def rtype_render_immortal(hop, i_track_allocation=None):
+    vlist = [hop.inputarg(hop.args_r[0], arg=0)]
+    v_track_allocation = parse_kwds(hop,
+        (i_track_allocation, None))
+    hop.exception_cannot_occur()
+    if i_track_allocation is None or v_track_allocation.value:
+        hop.genop('track_alloc_stop', vlist)
+
 def rtype_const_result(hop):
     hop.exception_cannot_occur()
     return hop.inputconst(hop.r_result.lowleveltype, hop.s_result.const)
@@ -523,6 +531,7 @@
 
 BUILTIN_TYPER[lltype.malloc] = rtype_malloc
 BUILTIN_TYPER[lltype.free] = rtype_free
+BUILTIN_TYPER[lltype.render_immortal] = rtype_render_immortal
 BUILTIN_TYPER[lltype.cast_primitive] = rtype_cast_primitive
 BUILTIN_TYPER[lltype.cast_pointer] = rtype_cast_pointer
 BUILTIN_TYPER[lltype.cast_opaque_ptr] = rtype_cast_opaque_ptr

Modified: pypy/branch/leak-finder-more/pypy/translator/c/test/test_lltyped.py
==============================================================================
--- pypy/branch/leak-finder-more/pypy/translator/c/test/test_lltyped.py	(original)
+++ pypy/branch/leak-finder-more/pypy/translator/c/test/test_lltyped.py	Tue Oct 26 11:19:05 2010
@@ -841,3 +841,17 @@
         assert res == -98765432
         res = fn(1)
         assert res == -9999999
+
+    def test_render_immortal(self):
+        A = FixedSizeArray(Signed, 1)
+        a1 = malloc(A, flavor='raw')
+        render_immortal(a1)
+        a1[0] = 42
+        def llf():
+            a2 = malloc(A, flavor='raw')
+            render_immortal(a2)
+            a2[0] = 3
+            return a1[0] + a2[0]
+        fn = self.getcompiled(llf)
+        assert fn() == 45
+


From david at codespeak.net  Tue Oct 26 11:36:43 2010
From: david at codespeak.net (david at codespeak.net)
Date: Tue, 26 Oct 2010 11:36:43 +0200 (CEST)
Subject: [pypy-svn] r78284 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . test
Message-ID: <20101026093643.BAAEF282C18@codespeak.net>

Author: david
Date: Tue Oct 26 11:36:42 2010
New Revision: 78284

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
Log:
Introduce locations for registers and replace register numbers with the locations


Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Tue Oct 26 11:36:42 2010
@@ -80,16 +80,16 @@
         functype = lltype.Ptr(lltype.FuncType([lltype.Signed, lltype.Signed], lltype.Signed))
         decode_registers_addr = llhelper(functype, self.failure_recovery_func)
         self.mc.PUSH(range(12))     # registers r0 .. r11
-        self.mc.MOV_rr(r.r0, r.lr)  # move mem block address, to r0 to pass as
+        self.mc.MOV_rr(r.r0.value, r.lr.value)  # move mem block address, to r0 to pass as
                                     # parameter to next procedure call
-        self.mc.MOV_rr(r.r1, r.sp)  # pass the current stack pointer as second param
-        self.mc.gen_load_int(r.r2, rffi.cast(lltype.Signed, decode_registers_addr))
-        self.mc.gen_load_int(r.lr, self.mc.curraddr()+self.mc.size_of_gen_load_int+WORD)
-        self.mc.MOV_rr(r.pc, r.r2)
-        self.mc.MOV_rr(r.ip, r.r0)
-        self.mc.LDM(r.sp, range(12), w=1) # XXX Replace with POP instr. someday
+        self.mc.MOV_rr(r.r1.value, r.sp.value)  # pass the current stack pointer as second param
+        self.mc.gen_load_int(r.r2.value, rffi.cast(lltype.Signed, decode_registers_addr))
+        self.mc.gen_load_int(r.lr.value, self.mc.curraddr()+self.mc.size_of_gen_load_int+WORD)
+        self.mc.MOV_rr(r.pc.value, r.r2.value)
+        self.mc.MOV_rr(r.ip.value, r.r0.value)
+        self.mc.LDM(r.sp.value, range(12), w=1) # XXX Replace with POP instr. someday
 
-        self.mc.MOV_rr(r.r0, r.ip)
+        self.mc.MOV_rr(r.r0.value, r.ip.value)
 
         self.gen_func_epilog()
 
@@ -101,7 +101,7 @@
         for i in range(len(args)):
             if args[i]:
                 curreg = regalloc.try_allocate_reg(args[i])
-                mem[i] = chr(curreg)
+                mem[i] = chr(curreg.value)
             else:
                 mem[i] = '\xFE'
 
@@ -112,11 +112,14 @@
 
         n = self.cpu.get_fail_descr_number(op.getdescr())
         self.encode32(mem, i+1, n)
-        self.mc.gen_load_int(r.lr, memaddr, cond=fcond)
-        self.mc.gen_load_int(reg, self.mc.baseaddr(), cond=fcond)
-        self.mc.MOV_rr(r.pc, reg, cond=fcond)
-
-        op.getdescr()._arm_guard_reg = reg
+        self.mc.gen_load_int(r.lr.value, memaddr, cond=fcond)
+        self.mc.gen_load_int(reg.value, self.mc.baseaddr(), cond=fcond)
+        self.mc.MOV_rr(r.pc.value, reg.value, cond=fcond)
+
+        # This register is used for patching when assembling a bridge
+        # guards going to be patched are allways conditional
+        if fcond != c.AL:
+            op.getdescr()._arm_guard_reg = reg
         return memaddr
 
     def align(self):
@@ -124,18 +127,18 @@
             self.mc.writechar(chr(0))
 
     def gen_func_epilog(self,cond=c.AL):
-        self.mc.LDM(r.sp, r.callee_restored_registers, cond=cond, w=1)
+        self.mc.LDM(r.sp.value, [reg.value for reg in r.callee_restored_registers], cond=cond, w=1)
 
     def gen_func_prolog(self):
-        self.mc.PUSH(r.callee_saved_registers)
+        self.mc.PUSH([reg.value for reg in r.callee_saved_registers])
 
     def gen_bootstrap_code(self, inputargs, regalloc, looptoken):
         regs = []
         for i in range(len(inputargs)):
             reg = regalloc.try_allocate_reg(inputargs[i])
             addr = self.fail_boxes_int.get_addr_for_num(i)
-            self.mc.gen_load_int(reg, addr)
-            self.mc.LDR_ri(reg, reg)
+            self.mc.gen_load_int(reg.value, addr)
+            self.mc.LDR_ri(reg.value, reg.value)
             regs.append(reg)
         looptoken._arm_arglocs = regs
 
@@ -187,8 +190,8 @@
         fcond = faildescr._arm_guard_cond
         b = ARMv7InMemoryBuilder(faildescr._arm_guard_code, faildescr._arm_guard_code+100)
         reg = faildescr._arm_guard_reg
-        b.gen_load_int(reg, bridge_addr, fcond)
-        b.MOV_rr(r.pc, reg, cond=fcond)
+        b.gen_load_int(reg.value, bridge_addr, fcond)
+        b.MOV_rr(r.pc.value, reg.value, cond=fcond)
 
 
     # Resoperations
@@ -200,10 +203,10 @@
             reg = regalloc.try_allocate_reg(op.getarg(i))
             inpreg = registers[i]
             # XXX only if every value is in a register
-            self.mc.MOV_rr(inpreg, reg)
+            self.mc.MOV_rr(inpreg.value, reg.value)
         loop_code = op.getdescr()._arm_loop_code
-        self.mc.gen_load_int(tmpreg, loop_code)
-        self.mc.MOV_rr(r.pc, tmpreg)
+        self.mc.gen_load_int(tmpreg.value, loop_code)
+        self.mc.MOV_rr(r.pc.value, tmpreg.value)
         regalloc.possibly_free_var(tmpreg)
         return fcond
 
@@ -214,14 +217,14 @@
     def emit_op_int_le(self, op, regalloc, fcond):
         reg = regalloc.try_allocate_reg(op.getarg(0))
         assert isinstance(op.getarg(1), ConstInt)
-        self.mc.CMP(reg, op.getarg(1).getint())
+        self.mc.CMP(reg.value, op.getarg(1).getint())
         return c.GT
 
     def emit_op_int_add(self, op, regalloc, fcond):
         reg = regalloc.try_allocate_reg(op.getarg(0))
         res = regalloc.try_allocate_reg(op.result)
         assert isinstance(op.getarg(1), ConstInt)
-        self.mc.ADD_ri(res, reg, op.getarg(1).getint())
+        self.mc.ADD_ri(res.value, reg.value, op.getarg(1).getint())
         regalloc.possibly_free_vars_for_op(op)
         return fcond
 

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Tue Oct 26 11:36:42 2010
@@ -70,15 +70,18 @@
 
     size_of_gen_load_int = 7 * WORD
     def gen_load_int(self, r, value, cond=cond.AL):
-        assert r != reg.ip, 'ip is used to load int'
+        """r is the register number, value is the value to be loaded to the
+        register"""
+        assert r != reg.ip.value, 'ip is used to load int'
         self.MOV_ri(r, (value & 0xFF), cond=cond)
+        ip = reg.ip.value
 
         for offset in range(8, 25, 8):
             t = (value >> offset) & 0xFF
             #if t == 0:
             #    continue
-            self.MOV_ri(reg.ip, t, cond=cond)
-            self.ORR_rr(r, r, reg.ip, offset, cond=cond)
+            self.MOV_ri(ip, t, cond=cond)
+            self.ORR_rr(r, r, ip, offset, cond=cond)
 
 class ARMv7InMemoryBuilder(AbstractARMv7Builder):
     def __init__(self, start, end):

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py	Tue Oct 26 11:36:42 2010
@@ -5,7 +5,7 @@
 class ARMRegisterManager(RegisterManager):
     all_regs              = r.all_regs
     box_types             = None       # or a list of acceptable types
-    no_lower_byte_regs    = r.all_regs
+    no_lower_byte_regs    = all_regs
     save_around_call_regs = all_regs
 
     def __init__(self, longevity, frame_manager=None, assembler=None):
@@ -16,7 +16,7 @@
         for i in range(len(inputargs)):
             while enc[j] == '\xFE':
                 j += 1
-            self.try_allocate_reg(inputargs[i], ord(enc[j]))
+            self.try_allocate_reg(inputargs[i], r.all_regs[ord(enc[j])])
             j += 1
 
 class ARMFrameManager(FrameManager):

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/registers.py	Tue Oct 26 11:36:42 2010
@@ -1,12 +1,17 @@
-r0, r1, r2, r3, r4, r5, r6, r7, r8, r9, r10, r11, r12, r13, r14, r15 = range(16)
+from pypy.jit.backend.arm.locations import RegisterLocation
+
+registers = [RegisterLocation(i) for i in range(16)]
+r0, r1, r2, r3, r4, r5, r6, r7, r8, r9, r10, r11, r12, r13, r14, r15 = registers
+
 # aliases for registers
-fp = 11
-ip = 12
-sp = 13
-lr = 14
-pc = 15
+fp = r11
+ip = r12
+sp = r13
+lr = r14
+pc = r15
+
+all_regs = registers[:12]
 
-all_regs = range(12)
 callee_resp = [r4, r5, r6, r7, r8, r9, r10, r11]
 callee_saved_registers = callee_resp+[lr]
 callee_restored_registers = callee_resp+[pc]

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py	Tue Oct 26 11:36:42 2010
@@ -21,78 +21,78 @@
 
     def test_load_small_int_to_reg(self):
         self.a.gen_func_prolog()
-        self.a.mc.gen_load_int(r.r0, 123)
+        self.a.mc.gen_load_int(r.r0.value, 123)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 123
 
     def test_load_medium_int_to_reg(self):
         self.a.gen_func_prolog()
-        self.a.mc.gen_load_int(r.r0, 0xBBD7)
+        self.a.mc.gen_load_int(r.r0.value, 0xBBD7)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 48087
 
     def test_load_int_to_reg(self):
         self.a.gen_func_prolog()
-        self.a.mc.gen_load_int(r.r0, 0xFFFFFF85)
+        self.a.mc.gen_load_int(r.r0.value, 0xFFFFFF85)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == -123
 
 
     def test_or(self):
         self.a.gen_func_prolog()
-        self.a.mc.MOV_ri(r.r1, 8)
-        self.a.mc.MOV_ri(r.r2, 8)
-        self.a.mc.ORR_rr(r.r0, r.r1, r.r2, 4)
+        self.a.mc.MOV_ri(r.r1.value, 8)
+        self.a.mc.MOV_ri(r.r2.value, 8)
+        self.a.mc.ORR_rr(r.r0.value, r.r1.value, r.r2.value, 4)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 0x88
 
     def test_sub(self):
         self.a.gen_func_prolog()
-        self.a.mc.gen_load_int(r.r1, 123456)
-        self.a.mc.SUB_ri(r.r0, r.r1, 123)
+        self.a.mc.gen_load_int(r.r1.value, 123456)
+        self.a.mc.SUB_ri(r.r0.value, r.r1.value, 123)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 123333
 
     def test_cmp(self):
         self.a.gen_func_prolog()
-        self.a.mc.gen_load_int(r.r1, 22)
-        self.a.mc.CMP(r.r1, 123)
-        self.a.mc.MOV_ri(r.r0, 1, c.LE)
-        self.a.mc.MOV_ri(r.r0, 0, c.GT)
+        self.a.mc.gen_load_int(r.r1.value, 22)
+        self.a.mc.CMP(r.r1.value, 123)
+        self.a.mc.MOV_ri(r.r0.value, 1, c.LE)
+        self.a.mc.MOV_ri(r.r0.value, 0, c.GT)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 1
 
     def test_int_le_false(self):
         self.a.gen_func_prolog()
-        self.a.mc.gen_load_int(r.r1, 2222)
-        self.a.mc.CMP(r.r1, 123)
-        self.a.mc.MOV_ri(r.r0, 1, c.LE)
-        self.a.mc.MOV_ri(r.r0, 0, c.GT)
+        self.a.mc.gen_load_int(r.r1.value, 2222)
+        self.a.mc.CMP(r.r1.value, 123)
+        self.a.mc.MOV_ri(r.r0.value, 1, c.LE)
+        self.a.mc.MOV_ri(r.r0.value, 0, c.GT)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 0
 
     def test_simple_jump(self):
         self.a.gen_func_prolog()
-        self.a.mc.MOV_ri(r.r1, 1)
+        self.a.mc.MOV_ri(r.r1.value, 1)
         loop_head = self.a.mc.curraddr()
-        self.a.mc.CMP(r.r1, 0) # z=0, z=1
-        self.a.mc.MOV_ri(r.r1, 0, cond=c.NE)
-        self.a.mc.MOV_ri(r.r1, 7, cond=c.EQ)
-        self.a.mc.gen_load_int(r.r4, loop_head, cond=c.NE)
-        self.a.mc.MOV_rr(r.pc, r.r4, cond=c.NE)
-        self.a.mc.MOV_rr(r.r0, r.r1)
+        self.a.mc.CMP(r.r1.value, 0) # z=0, z=1
+        self.a.mc.MOV_ri(r.r1.value, 0, cond=c.NE)
+        self.a.mc.MOV_ri(r.r1.value, 7, cond=c.EQ)
+        self.a.mc.gen_load_int(r.r4.value, loop_head, cond=c.NE)
+        self.a.mc.MOV_rr(r.pc.value, r.r4.value, cond=c.NE)
+        self.a.mc.MOV_rr(r.r0.value, r.r1.value)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 7
 
     def test_jump(self):
         self.a.gen_func_prolog()
-        self.a.mc.MOV_ri(r.r1, 1)
+        self.a.mc.MOV_ri(r.r1.value, 1)
         loop_head = self.a.mc.curraddr()
-        self.a.mc.ADD_ri(r.r1, r.r1, 1)
-        self.a.mc.CMP(r.r1, 9)
-        self.a.mc.gen_load_int(r.r4, loop_head, cond=c.NE)
-        self.a.mc.MOV_rr(r.pc, r.r4, cond=c.NE)
-        self.a.mc.MOV_rr(r.r0, r.r1)
+        self.a.mc.ADD_ri(r.r1.value, r.r1.value, 1)
+        self.a.mc.CMP(r.r1.value, 9)
+        self.a.mc.gen_load_int(r.r4.value, loop_head, cond=c.NE)
+        self.a.mc.MOV_rr(r.pc.value, r.r4.value, cond=c.NE)
+        self.a.mc.MOV_rr(r.r0.value, r.r1.value)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 9
 
@@ -101,10 +101,10 @@
         functype = lltype.Ptr(lltype.FuncType([lltype.Signed], lltype.Signed))
         call_addr = rffi.cast(lltype.Signed, llhelper(functype, callme))
         self.a.gen_func_prolog()
-        self.a.mc.MOV_ri(r.r0, 123)
-        self.a.mc.gen_load_int(r.r1, call_addr)
-        self.a.mc.gen_load_int(r.lr, self.a.mc.curraddr()+self.a.mc.size_of_gen_load_int+WORD)
-        self.a.mc.MOV_rr(r.pc, r.r1)
+        self.a.mc.MOV_ri(r.r0.value, 123)
+        self.a.mc.gen_load_int(r.r1.value, call_addr)
+        self.a.mc.gen_load_int(r.lr.value, self.a.mc.curraddr()+self.a.mc.size_of_gen_load_int+WORD)
+        self.a.mc.MOV_rr(r.pc.value, r.r1.value)
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 133
 

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	Tue Oct 26 11:36:42 2010
@@ -25,97 +25,97 @@
         self.cb = CodeBuilder()
 
     def test_ldr(self):
-        self.cb.LDR_ri(r.r0, r.r1)
+        self.cb.LDR_ri(r.r0.value, r.r1.value)
         self.assert_equal('LDR r0, [r1]')
 
     def test_ldr_neg(self):
-        self.cb.LDR_ri(r.r3, r.fp, -16)
+        self.cb.LDR_ri(r.r3.value, r.fp.value, -16)
         self.assert_equal('LDR r3, [fp, #-16]')
 
     def test_add_ri(self):
-        self.cb.ADD_ri(r.r0, r.r1, 1)
+        self.cb.ADD_ri(r.r0.value, r.r1.value, 1)
         self.assert_equal('ADD r0, r1, #1')
 
     def test_mov_rr(self):
-        self.cb.MOV_rr(r.r7, r.r12)
+        self.cb.MOV_rr(r.r7.value, r.r12.value)
         self.assert_equal('MOV r7, r12')
 
     def test_mov_ri(self):
-        self.cb.MOV_ri(r.r9, 123)
+        self.cb.MOV_ri(r.r9.value, 123)
         self.assert_equal('MOV r9, #123')
 
     def test_mov_ri2(self):
-        self.cb.MOV_ri(r.r9, 255)
+        self.cb.MOV_ri(r.r9.value, 255)
         self.assert_equal('MOV r9, #255')
 
     def test_mov_ri_max(self):
         py.test.skip('Check the actual largest thing')
-        self.cb.MOV_ri(r.r9, 0xFFF)
+        self.cb.MOV_ri(r.r9.value, 0xFFF)
         self.assert_equal('MOV r9, #4095')
 
     def test_str_ri(self):
-        self.cb.STR_ri(r.r9, r.r14)
+        self.cb.STR_ri(r.r9.value, r.r14.value)
         self.assert_equal('STR r9, [r14]')
 
     def test_str_ri_offset(self):
-        self.cb.STR_ri(r.r9, r.r14, 23)
+        self.cb.STR_ri(r.r9.value, r.r14.value, 23)
         self.assert_equal('STR r9, [r14, #23]')
 
     def test_str_ri_offset(self):
-        self.cb.STR_ri(r.r9, r.r14, -20)
+        self.cb.STR_ri(r.r9.value, r.r14.value, -20)
         self.assert_equal('STR r9, [r14, #-20]')
 
     def test_asr_ri(self):
-        self.cb.ASR_ri(r.r7, r.r5, 24)
+        self.cb.ASR_ri(r.r7.value, r.r5.value, 24)
         self.assert_equal('ASR r7, r5, #24')
 
     def test_orr_rr_no_shift(self):
-        self.cb.ORR_rr(r.r0, r.r7,r.r12)
+        self.cb.ORR_rr(r.r0.value, r.r7.value,r.r12.value)
         self.assert_equal('ORR r0, r7, r12')
 
     def test_orr_rr_lsl_8(self):
-        self.cb.ORR_rr(r.r0, r.r7,r.r12, 8)
+        self.cb.ORR_rr(r.r0.value, r.r7.value,r.r12.value, 8)
         self.assert_equal('ORR r0, r7, r12, lsl #8')
 
     def test_push_one_reg(self):
-        self.cb.PUSH([r.r1])
+        self.cb.PUSH([r.r1.value])
         self.assert_equal('PUSH {r1}')
 
     def test_push_multiple(self):
-        self.cb.PUSH([r.r1, r.r3, r.r6, r.r8, r.pc])
+        self.cb.PUSH([reg.value for reg in [r.r1, r.r3, r.r6, r.r8, r.pc]])
         self.assert_equal('PUSH {r1, r3, r6, r8, pc}')
 
     def test_push_multiple2(self):
-        self.cb.PUSH([r.fp, r.ip, r.lr, r.pc])
+        self.cb.PUSH([reg.value for reg in [r.fp, r.ip, r.lr, r.pc]])
         self.assert_equal('PUSH {fp, ip, lr, pc}')
 
     def test_ldm_one_reg(self):
-        self.cb.LDM(r.sp, [r.fp])
+        self.cb.LDM(r.sp.value, [r.fp.value])
         self.assert_equal('LDM sp, {fp}')
 
     def test_ldm_multiple_reg(self):
-        self.cb.LDM(r.sp, [r.fp, r.ip, r.lr])
+        self.cb.LDM(r.sp.value, [reg.value for reg in [r.fp, r.ip, r.lr]])
         self.assert_equal('LDM sp, {fp, ip, lr}')
 
     def test_ldm_multiple_reg2(self):
-        self.cb.LDM(r.sp, [r.fp, r.sp, r.pc])
+        self.cb.LDM(r.sp.value, [reg.value for reg in [r.fp, r.sp, r.pc]])
         self.assert_equal("LDM sp, {fp, sp, pc}")
 
     def test_sub_ri(self):
-        self.cb.SUB_ri(r.r2, r.r4, 123)
+        self.cb.SUB_ri(r.r2.value, r.r4.value, 123)
         self.assert_equal('SUB r2, r4, #123')
 
     def test_sub_ri2(self):
         py.test.skip('XXX check the actual largest value')
-        self.cb.SUB_ri(r.r3, r.r7, 0xFFF)
+        self.cb.SUB_ri(r.r3.value, r.r7.value, 0xFFF)
         self.assert_equal('SUB r3, r7, #4095')
 
     def test_cmp_ri(self):
-        self.cb.CMP(r.r3, 123)
+        self.cb.CMP(r.r3.value, 123)
         self.assert_equal('CMP r3, #123')
 
     def test_mcr(self):
-        self.cb.MCR(15, 0, r.r1, 7, 10,0)
+        self.cb.MCR(15, 0, r.r1.value, 7, 10,0)
 
         self.assert_equal('MCR P15, 0, r1, c7, c10, 0')
 
@@ -149,31 +149,31 @@
     if table['result'] and table['base']:
         def f(self):
             func = getattr(self.cb, name)
-            func(r.r3, r.r7, 23)
+            func(r.r3.value, r.r7.value, 23)
             self.assert_equal('%s r3, r7, #23' % name[:name.index('_')])
     elif not table['base']:
         def f(self):
             func = getattr(self.cb, name)
-            func(r.r3, 23)
+            func(r.r3.value, 23)
             self.assert_equal('%s r3, #23' % name[:name.index('_')])
     else:
         def f(self):
             func = getattr(self.cb, name)
-            func(r.r3, 23)
+            func(r.r3.value, 23)
             self.assert_equal('%s r3, #23' % name[:name.index('_')])
     return f
 
 def gen_test_imm_func(name, table):
     def f(self):
         func = getattr(self.cb, name)
-        func(r.r3, r.r7, 23)
+        func(r.r3.value, r.r7.value, 23)
         self.assert_equal('%s r3, [r7, #23]' % name[:name.index('_')])
     return f
 
 def gen_test_reg_func(name, table):
     def f(self):
         func = getattr(self.cb, name)
-        func(r.r3, r.r7, r.r12)
+        func(r.r3.value, r.r7.value, r.r12.value)
         self.assert_equal('%s r3, [r7, r12]' % name[:name.index('_')])
     return f
 
@@ -181,18 +181,18 @@
     if name[-2:] == 'ri':
         def f(self):
             func = getattr(self.cb, name)
-            func(r.r3, r.r7, 12)
+            func(r.r3.value, r.r7.value, 12)
             self.assert_equal('%s r3, r7, #12' % name[:name.index('_')])
 
     elif table['base'] and table['result']:
         def f(self):
             func = getattr(self.cb, name)
-            func(r.r3, r.r7, r.r12)
+            func(r.r3.value, r.r7.value, r.r12.value)
             self.assert_equal('%s r3, r7, r12' % name[:name.index('_')])
     else:
         def f(self):
             func = getattr(self.cb, name)
-            func(r.r3, r.r7)
+            func(r.r3.value, r.r7.value)
             self.assert_equal('%s r3, r7' % name[:name.index('_')])
 
     return f


From arigo at codespeak.net  Tue Oct 26 11:38:21 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 11:38:21 +0200 (CEST)
Subject: [pypy-svn] r78285 - pypy/branch/set-object-cleanup
Message-ID: <20101026093821.C9E4A282C18@codespeak.net>

Author: arigo
Date: Tue Oct 26 11:38:20 2010
New Revision: 78285

Added:
   pypy/branch/set-object-cleanup/
      - copied from r78284, pypy/trunk/
Log:
A branch in which to clean up a bit setobject.py.



From arigo at codespeak.net  Tue Oct 26 11:39:01 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 11:39:01 +0200 (CEST)
Subject: [pypy-svn] r78286 - in pypy/trunk/pypy/tool/release: . test
Message-ID: <20101026093901.CFD1E282C18@codespeak.net>

Author: arigo
Date: Tue Oct 26 11:39:00 2010
New Revision: 78286

Modified:
   pypy/trunk/pypy/tool/release/   (props changed)
   pypy/trunk/pypy/tool/release/__init__.py   (props changed)
   pypy/trunk/pypy/tool/release/force-builds.py   (props changed)
   pypy/trunk/pypy/tool/release/make_release.py   (props changed)
   pypy/trunk/pypy/tool/release/test/   (props changed)
   pypy/trunk/pypy/tool/release/test/__init__.py   (props changed)
   pypy/trunk/pypy/tool/release/test/test_make_release.py   (props changed)
Log:
fixeol



From fijal at codespeak.net  Tue Oct 26 13:21:05 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Tue, 26 Oct 2010 13:21:05 +0200 (CEST)
Subject: [pypy-svn] r78287 - pypy/trunk/pypy/config
Message-ID: <20101026112105.48208282C18@codespeak.net>

Author: fijal
Date: Tue Oct 26 13:21:03 2010
New Revision: 78287

Modified:
   pypy/trunk/pypy/config/pypyoption.py
Log:
_ffi module also depends on libffi available on the system


Modified: pypy/trunk/pypy/config/pypyoption.py
==============================================================================
--- pypy/trunk/pypy/config/pypyoption.py	(original)
+++ pypy/trunk/pypy/config/pypyoption.py	Tue Oct 26 13:21:03 2010
@@ -76,6 +76,7 @@
     # no _rawffi if importing pypy.rlib.clibffi raises ImportError
     # or CompilationError
     "_rawffi"   : ["pypy.rlib.clibffi"],
+    "_ffi"      : ["pypy.rlib.clibffi"],
 
     "zlib"      : ["pypy.rlib.rzlib"],
     "bz2"       : ["pypy.module.bz2.interp_bz2"],


From arigo at codespeak.net  Tue Oct 26 13:41:10 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 13:41:10 +0200 (CEST)
Subject: [pypy-svn] r78288 - in
	pypy/branch/set-object-cleanup/pypy/objspace/std: . test
Message-ID: <20101026114110.AAEC9282C18@codespeak.net>

Author: arigo
Date: Tue Oct 26 13:41:08 2010
New Revision: 78288

Modified:
   pypy/branch/set-object-cleanup/pypy/objspace/std/frozensettype.py
   pypy/branch/set-object-cleanup/pypy/objspace/std/setobject.py
   pypy/branch/set-object-cleanup/pypy/objspace/std/settype.py
   pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_setobject.py
Log:
Rewrite a bit sets.  Tests pass, whatever that means.


Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/frozensettype.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/frozensettype.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/frozensettype.py	Tue Oct 26 13:41:08 2010
@@ -39,13 +39,13 @@
 def descr__frozenset__new__(space, w_frozensettype,
                             w_iterable=gateway.NoneNotWrapped):
     from pypy.objspace.std.setobject import W_FrozensetObject
-    from pypy.objspace.std.setobject import _is_frozenset_exact
+    from pypy.objspace.std.setobject import make_setdata_from_w_iterable
     if (space.is_w(w_frozensettype, space.w_frozenset) and
-        _is_frozenset_exact(w_iterable)):
+        w_iterable is not None and type(w_iterable) is W_FrozensetObject):
         return w_iterable
     w_obj = space.allocate_instance(W_FrozensetObject, w_frozensettype)
-    W_FrozensetObject.__init__(w_obj, space, None)
-
+    data = make_setdata_from_w_iterable(space, w_iterable)
+    W_FrozensetObject.__init__(w_obj, space, data)
     return w_obj
 
 frozenset_typedef = StdTypeDef("frozenset",

Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/setobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/setobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/setobject.py	Tue Oct 26 13:41:08 2010
@@ -21,11 +21,10 @@
         return False
 
 
-    def __init__(w_self, space, setdata=None):
-        if setdata is None:
-            w_self.setdata = r_dict(space.eq_w, space.hash_w)
-        else:
-            w_self.setdata = setdata.copy()
+    def __init__(w_self, space, setdata):
+        """Initialize the set by taking ownership of 'setdata'."""
+        assert setdata is not None
+        w_self.setdata = setdata
 
     def __repr__(w_self):
         """representation for debugging purposes"""
@@ -33,6 +32,7 @@
         return "<%s(%s)>" % (w_self.__class__.__name__, ', '.join(reprlist))
 
     def _newobj(w_self, space, rdict_w=None):
+        """Make a new set or frozenset by taking ownership of 'rdict_w'."""
         #return space.call(space.type(w_self),W_SetIterObject(rdict_w))
         objtype = type(w_self)
         if objtype is W_SetObject:
@@ -55,10 +55,7 @@
 
 class W_FrozensetObject(W_BaseSetObject):
     from pypy.objspace.std.frozensettype import frozenset_typedef as typedef
-
-    def __init__(w_self, space, setdata):
-        W_BaseSetObject.__init__(w_self, space, setdata)
-        w_self.hash = -1
+    hash = 0
 
 registerimplementation(W_BaseSetObject)
 registerimplementation(W_SetObject)
@@ -109,8 +106,14 @@
 
 # some helper functions
 
+def newset(space):
+    return r_dict(space.eq_w, space.hash_w)
+
 def make_setdata_from_w_iterable(space, w_iterable=None):
-    data = r_dict(space.eq_w, space.hash_w)
+    """Return a new r_dict with the content of w_iterable."""
+    if isinstance(w_iterable, W_BaseSetObject):
+        return w_iterable.setdata.copy()
+    data = newset(space)
     if w_iterable is not None:
         for w_item in space.listview(w_iterable):
             data[w_item] = None
@@ -119,7 +122,7 @@
 def _initialize_set(space, w_obj, w_iterable=None):
     w_obj.setdata.clear()
     if w_iterable is not None:
-        w_obj.setdata.update(make_setdata_from_w_iterable(space, w_iterable))
+        w_obj.setdata = make_setdata_from_w_iterable(space, w_iterable)
 
 def _convert_set_to_frozenset(space, w_obj):
     if space.is_true(space.isinstance(w_obj, space.w_set)):
@@ -130,12 +133,6 @@
 
 # helper functions for set operation on dicts
 
-def _is_frozenset_exact(w_obj):
-    if (w_obj is not None) and (type(w_obj) is W_FrozensetObject):
-        return True
-    else:
-        return False
-
 def _is_eq(ld, rd):
     if len(ld) != len(rd):
         return False
@@ -144,66 +141,41 @@
             return False
     return True
 
-def _union_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    ld.update(rdict)
-    return ld, rdict
-
-def _difference_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    del_list_w = []
+def _difference_dict(space, ld, rd):
+    result = newset(space)
     for w_key in ld:
-        if w_key in rdict:
-            del_list_w.append(w_key)
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    return ld, rdict
-
-def _intersection_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    del_list_w = []
-    for w_key in ld:
-        if w_key not in rdict:
-            del_list_w.append(w_key)
-
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    return ld, rdict
-
+        if w_key not in rd:
+            result[w_key] = None
+    return result
 
-def _symmetric_difference_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
+def _difference_dict_update(space, ld, rd):
+    if ld is rd:
+        ld.clear()     # for the case 'a.difference_update(a)'
     else:
-        ld = ldict.copy()
-    del_list_w = []
-    add_list_w = []
+        for w_key in rd:
+            try:
+                del ld[w_key]
+            except KeyError:
+                pass
+
+def _intersection_dict(space, ld, rd):
+    result = newset(space)
+    if len(ld) > len(rd):
+        ld, rd = rd, ld     # loop over the smaller dict
     for w_key in ld:
-        if w_key in rdict:
-            del_list_w.append(w_key)
+        if w_key in rd:
+            result[w_key] = None
+    return result
 
-    for w_key in rdict:
+def _symmetric_difference_dict(space, ld, rd):
+    result = newset(space)
+    for w_key in ld:
+        if w_key not in rd:
+            result[w_key] = None
+    for w_key in rd:
         if w_key not in ld:
-            add_list_w.append(w_key)
-
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    for w_key in add_list_w:
-        ld[w_key] = None
-
-    return ld, rdict
+            result[w_key] = None
+    return result
 
 def _issubset_dict(ldict, rdict):
     if len(ldict) > len(rdict):
@@ -220,14 +192,13 @@
 def set_update__Set_BaseSet(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _union_dict(ld, rd, True)
-    return space.w_None
+    ld.update(rd)
 
 def set_update__Set_ANY(space, w_left, w_other):
     """Update a set with the union of itself and another."""
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _union_dict(ld, rd, True)
-    return space.w_None
+    ld = w_left.setdata
+    for w_item in space.listview(w_other):
+        ld[w_item] = None
 
 def inplace_or__Set_Set(space, w_left, w_other):
     set_update__Set_BaseSet(space, w_left, w_other)
@@ -241,25 +212,23 @@
     This has no effect if the element is already present.
     """
     w_left.setdata[w_other] = None
-    return space.w_None
 
 def set_copy__Set(space, w_set):
-    return w_set._newobj(space,w_set.setdata)
+    return w_set._newobj(space, w_set.setdata.copy())
 
 def frozenset_copy__Frozenset(space, w_left):
-    if _is_frozenset_exact(w_left):
+    if type(w_left) is W_FrozensetObject:
         return w_left
     else:
-        return set_copy__Set(space,w_left)
+        return set_copy__Set(space, w_left)
 
 def set_clear__Set(space, w_left):
     w_left.setdata.clear()
-    return space.w_None
 
 def set_difference__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _difference_dict(ld, rd, False)
+    new_ld = _difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 set_difference__Set_Frozenset = set_difference__Set_Set
@@ -272,7 +241,7 @@
 
 def set_difference__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _difference_dict(ld, rd, False)
+    new_ld = _difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 frozenset_difference__Frozenset_ANY = set_difference__Set_ANY
@@ -281,15 +250,17 @@
 def set_difference_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _difference_dict(ld, rd, True)
-    return space.w_None
+    _difference_dict_update(space, ld, rd)
 
 set_difference_update__Set_Frozenset = set_difference_update__Set_Set
 
 def set_difference_update__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _difference_dict(ld, rd, True)
-    return space.w_None
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        try:
+            del ld[w_key]
+        except KeyError:
+            pass
 
 def inplace_sub__Set_Set(space, w_left, w_other):
     set_difference_update__Set_Set(space, w_left, w_other)
@@ -467,7 +438,7 @@
 
 def hash__Frozenset(space, w_set):
     multi = r_uint(1822399083) + r_uint(1822399083) + 1
-    if w_set.hash != -1:
+    if w_set.hash != 0:
         return space.wrap(w_set.hash)
     hash = 1927868237
     hash *= (len(w_set.setdata) + 1)
@@ -476,7 +447,7 @@
         value = ((h ^ (h << 16) ^ 89869747)  * multi)
         hash = intmask(hash ^ value)
     hash = hash * 69069 + 907133923
-    if hash == -1:
+    if hash == 0:
         hash = 590923713
     hash = intmask(hash)
     w_set.hash = hash
@@ -484,29 +455,31 @@
     return space.wrap(hash)
 
 def set_pop__Set(space, w_left):
-    if len(w_left.setdata) == 0:
+    for w_key in w_left.setdata:
+        break
+    else:
         raise OperationError(space.w_KeyError,
                                 space.wrap('pop from an empty set'))
-    w_keys = w_left.setdata.keys()
-    w_value = w_keys[0]
-    del w_left.setdata[w_value]
-
-    return w_value
+    del w_left.setdata[w_key]
+    return w_key
 
 def set_intersection__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _intersection_dict(ld, rd, False)
-    return w_left._newobj(space,new_ld)
+    new_ld = _intersection_dict(space, ld, rd)
+    return w_left._newobj(space, new_ld)
 
 set_intersection__Set_Frozenset = set_intersection__Set_Set
 set_intersection__Frozenset_Frozenset = set_intersection__Set_Set
 set_intersection__Frozenset_Set = set_intersection__Set_Set
 
 def set_intersection__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _intersection_dict(ld, rd, False)
-    return w_left._newobj(space,new_ld)
+    result = newset(space)
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        if w_key in ld:
+            result[w_key] = None
+    return w_left._newobj(space, result)
 
 frozenset_intersection__Frozenset_ANY = set_intersection__Set_ANY
 
@@ -518,15 +491,18 @@
 def set_intersection_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _intersection_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _intersection_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 set_intersection_update__Set_Frozenset = set_intersection_update__Set_Set
 
 def set_intersection_update__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _intersection_dict(ld, rd, True)
-    return space.w_None
+    result = newset(space)
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        if w_key in ld:
+            result[w_key] = None
+    w_left.setdata = result
 
 def inplace_and__Set_Set(space, w_left, w_other):
     set_intersection_update__Set_Set(space, w_left, w_other)
@@ -537,7 +513,7 @@
 def set_symmetric_difference__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _symmetric_difference_dict(ld, rd, False)
+    new_ld = _symmetric_difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 set_symmetric_difference__Set_Frozenset = set_symmetric_difference__Set_Set
@@ -553,7 +529,7 @@
 
 def set_symmetric_difference__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _symmetric_difference_dict(ld, rd, False)
+    new_ld = _symmetric_difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 frozenset_symmetric_difference__Frozenset_ANY = \
@@ -562,16 +538,16 @@
 def set_symmetric_difference_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _symmetric_difference_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _symmetric_difference_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 set_symmetric_difference_update__Set_Frozenset = \
                                     set_symmetric_difference_update__Set_Set
 
 def set_symmetric_difference_update__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _symmetric_difference_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _symmetric_difference_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 def inplace_xor__Set_Set(space, w_left, w_other):
     set_symmetric_difference_update__Set_Set(space, w_left, w_other)
@@ -582,8 +558,9 @@
 def set_union__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _union_dict(ld, rd, False)
-    return w_left._newobj(space, new_ld)
+    result = ld.copy()
+    result.update(rd)
+    return w_left._newobj(space, result)
 
 set_union__Set_Frozenset = set_union__Set_Set
 set_union__Frozenset_Set = set_union__Set_Set
@@ -595,9 +572,11 @@
 
 
 def set_union__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _union_dict(ld, rd, False)
-    return w_left._newobj(space, new_ld)
+    ld = w_left.setdata()
+    result = ld.copy()
+    for w_key in space.viewlist(w_other):
+        result[w_key] = None
+    return w_left._newobj(space, result)
 
 frozenset_union__Frozenset_ANY = set_union__Set_ANY
 
@@ -629,15 +608,6 @@
             init_defaults)
     _initialize_set(space, w_set, w_iterable)
 
-def init__Frozenset(space, w_set, __args__):
-    w_iterable, = __args__.parse_obj(
-            None, 'set',
-            init_signature,
-            init_defaults)
-    if w_set.hash == -1:
-        _initialize_set(space, w_set, w_iterable)
-        hash__Frozenset(space, w_set)
-
 app = gateway.applevel("""
     def setrepr(currently_in_repr, s):
         'The app-level part of repr().'

Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/settype.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/settype.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/settype.py	Tue Oct 26 13:41:08 2010
@@ -66,9 +66,9 @@
 register_all(vars(), globals())
 
 def descr__new__(space, w_settype, __args__):
-    from pypy.objspace.std.setobject import W_SetObject
+    from pypy.objspace.std.setobject import W_SetObject, newset
     w_obj = space.allocate_instance(W_SetObject, w_settype)
-    W_SetObject.__init__(w_obj, space, None)
+    W_SetObject.__init__(w_obj, space, newset(space))
     return w_obj
 
 set_typedef = StdTypeDef("set",

Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_setobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_setobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_setobject.py	Tue Oct 26 13:41:08 2010
@@ -10,6 +10,7 @@
 import py.test
 from pypy.objspace.std.setobject import W_SetObject, W_FrozensetObject
 from pypy.objspace.std.setobject import _initialize_set
+from pypy.objspace.std.setobject import newset, make_setdata_from_w_iterable
 from pypy.objspace.std.setobject import set_intersection__Set_Set
 from pypy.objspace.std.setobject import set_intersection__Set_ANY
 from pypy.objspace.std.setobject import eq__Set_Set
@@ -28,12 +29,12 @@
         self.false = self.space.w_False
 
     def test_and(self):
-        s = W_SetObject(self.space, None)
+        s = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, s, self.word)
-        t0 = W_SetObject(self.space, None)
+        t0 = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, t0, self.otherword)
-        t1 = W_FrozensetObject(self.space, None)
-        _initialize_set(self.space, t1, self.otherword)
+        t1 = W_FrozensetObject(self.space,
+                make_setdata_from_w_iterable(self.space, self.otherword))
         r0 = set_intersection__Set_Set(self.space, s, t0)
         r1 = set_intersection__Set_Set(self.space, s, t1)
         assert eq__Set_Set(self.space, r0, r1) == self.true
@@ -41,9 +42,9 @@
         assert eq__Set_Set(self.space, r0, sr) == self.true
 
     def test_compare(self):
-        s = W_SetObject(self.space, None)
+        s = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, s, self.word)
-        t = W_SetObject(self.space, None)
+        t = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, t, self.word)
         assert self.space.eq_w(s,t)
         u = self.space.wrap(set('simsalabim'))


From arigo at codespeak.net  Tue Oct 26 13:45:02 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 13:45:02 +0200 (CEST)
Subject: [pypy-svn] r78289 - in
	pypy/branch/set-object-cleanup/pypy/objspace/std: . test
Message-ID: <20101026114502.8C17B282C18@codespeak.net>

Author: arigo
Date: Tue Oct 26 13:45:01 2010
New Revision: 78289

Modified:
   pypy/branch/set-object-cleanup/pypy/objspace/std/setobject.py
   pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_setobject.py
Log:
Test and fixes.


Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/setobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/setobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/setobject.py	Tue Oct 26 13:45:01 2010
@@ -572,9 +572,9 @@
 
 
 def set_union__Set_ANY(space, w_left, w_other):
-    ld = w_left.setdata()
+    ld = w_left.setdata
     result = ld.copy()
-    for w_key in space.viewlist(w_other):
+    for w_key in space.listview(w_other):
         result[w_key] = None
     return w_left._newobj(space, result)
 

Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_setobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_setobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_setobject.py	Tue Oct 26 13:45:01 2010
@@ -57,6 +57,11 @@
         b = a | set('abc')
         assert type(b) is subset
 
+    def test_union(self):
+        a = set([4, 5])
+        b = a.union([5, 7])
+        assert sorted(b) == [4, 5, 7]
+
     def test_compare(self):
         raises(TypeError, cmp, set('abc'), set('abd'))
         assert set('abc') != 'abc'


From arigo at codespeak.net  Tue Oct 26 13:58:09 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 13:58:09 +0200 (CEST)
Subject: [pypy-svn] r78290 - pypy/branch/set-object-cleanup/pypy/interpreter
Message-ID: <20101026115809.A136A282C18@codespeak.net>

Author: arigo
Date: Tue Oct 26 13:58:08 2010
New Revision: 78290

Modified:
   pypy/branch/set-object-cleanup/pypy/interpreter/baseobjspace.py
Log:
Fix the docstring.


Modified: pypy/branch/set-object-cleanup/pypy/interpreter/baseobjspace.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/interpreter/baseobjspace.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/interpreter/baseobjspace.py	Tue Oct 26 13:58:08 2010
@@ -739,7 +739,7 @@
 
     def unpackiterable(self, w_iterable, expected_length=-1):
         """Unpack an iterable object into a real (interpreter-level) list.
-        Raise a real (subclass of) ValueError if the length is wrong."""
+        Raise an OperationError(w_ValueError) if the length is wrong."""
         w_iterator = self.iter(w_iterable)
         items = []
         while True:


From arigo at codespeak.net  Tue Oct 26 14:12:15 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 14:12:15 +0200 (CEST)
Subject: [pypy-svn] r78291 - pypy/branch/set-object-cleanup/pypy/objspace/std
Message-ID: <20101026121215.207885080C@codespeak.net>

Author: arigo
Date: Tue Oct 26 14:12:13 2010
New Revision: 78291

Modified:
   pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
Log:
A faster implementation of dict.fromkeys().


Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	Tue Oct 26 14:12:13 2010
@@ -141,18 +141,17 @@
 
 @gateway.unwrap_spec(ObjSpace, W_Root, W_Root, W_Root)
 def descr_fromkeys(space, w_type, w_keys, w_fill=None):
+    from pypy.objspace.std.dictmultiobject import W_DictMultiObject
     if w_fill is None:
         w_fill = space.w_None
-    w_dict = space.call_function(w_type)
-    w_iter = space.iter(w_keys)
-    while True:
-        try:
-            w_key = space.next(w_iter)
-        except OperationError, e:
-            if not e.match(space, space.w_StopIteration):
-                raise
-            break
-        space.setitem(w_dict, w_key, w_fill)
+    if w_type is space.w_dict:
+        w_dict = W_DictMultiObject.allocate_and_init_instance(space, w_type)
+        for w_key in space.listview(w_keys):
+            w_dict.setitem(w_key, w_fill)
+    else:
+        w_dict = space.call_function(w_type)
+        for w_key in space.listview(w_keys):
+            space.setitem(w_dict, w_key, w_fill)
     return w_dict
 
 


From arigo at codespeak.net  Tue Oct 26 14:15:50 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 14:15:50 +0200 (CEST)
Subject: [pypy-svn] r78292 - pypy/branch/set-object-cleanup/pypy/objspace/std
Message-ID: <20101026121550.09FC8282C18@codespeak.net>

Author: arigo
Date: Tue Oct 26 14:15:49 2010
New Revision: 78292

Modified:
   pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
Log:
Kill code.


Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	Tue Oct 26 14:15:49 2010
@@ -90,51 +90,17 @@
         dict.__delitem__(d, k)
         return k, v
 
-    def get(d, k, v=None):
-        if k in d:
-            return dict.__getitem__(d, k)
-        else:
-            return v
-
     def setdefault(d, k, v=None):
         if k in d:
             return dict.__getitem__(d, k)
         else:
             dict.__setitem__(d, k, v)
             return v
-
-    def pop(d, k, defaults):     # XXX defaults is actually *defaults
-        if len(defaults) > 1:
-            raise TypeError, "pop expected at most 2 arguments, got %d" % (
-                1 + len(defaults))
-        try:
-            v = dict.__getitem__(d, k)
-            dict.__delitem__(d, k)
-        except KeyError, e:
-            if defaults:
-                return defaults[0]
-            else:
-                raise e
-        return v
-
-    def iteritems(d):
-        return iter(dict.items(d))
-
-    def iterkeys(d):
-        return iter(dict.keys(d))
-
-    def itervalues(d):
-        return iter(dict.values(d))
 ''', filename=__file__)
 
 dict_update__ANY             = app.interphook("update")
 dict_popitem__ANY            = app.interphook("popitem")
-dict_get__ANY_ANY_ANY        = app.interphook("get")
 dict_setdefault__ANY_ANY_ANY = app.interphook("setdefault")
-dict_pop__ANY_ANY            = app.interphook("pop")
-dict_iteritems__ANY          = app.interphook("iteritems")
-dict_iterkeys__ANY           = app.interphook("iterkeys")
-dict_itervalues__ANY         = app.interphook("itervalues")
 update1                      = app.interphook("update1")
 
 register_all(vars(), globals())


From arigo at codespeak.net  Tue Oct 26 14:27:58 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 14:27:58 +0200 (CEST)
Subject: [pypy-svn] r78293 - pypy/branch/set-object-cleanup/pypy/objspace/std
Message-ID: <20101026122758.E0599282C18@codespeak.net>

Author: arigo
Date: Tue Oct 26 14:27:57 2010
New Revision: 78293

Modified:
   pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
   pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
Log:
Kill dict.update() at app-level, and implement it as just
a call to dict.__init__().


Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	Tue Oct 26 14:27:57 2010
@@ -645,7 +645,7 @@
     w_src, w_kwds = __args__.parse_obj(
             None, 'dict',
             init_signature, # signature
-            init_defaults)                           # default argument
+            init_defaults)  # default argument
     if w_src is None:
         pass
     elif space.findattr(w_src, space.wrap("keys")) is None:
@@ -665,6 +665,9 @@
         from pypy.objspace.std.dicttype import update1
         update1(space, w_dict, w_kwds)
 
+def dict_update__DictMulti(space, w_dict, __args__):
+    init__DictMulti(space, w_dict, __args__)
+
 def getitem__DictMulti_ANY(space, w_dict, w_key):
     w_value = w_dict.getitem(w_key)
     if w_value is not None:

Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	Tue Oct 26 14:27:57 2010
@@ -72,15 +72,6 @@
             for k,v in o:
                 dict.__setitem__(d, k, v)
 
-    def update(d, *args, **kwargs):
-        len_args = len(args)
-        if len_args == 1:
-            update1(d, args[0])
-        elif len_args > 1:
-            raise TypeError("update takes at most 1 (non-keyword) argument")
-        if kwargs:
-            update1(d, kwargs)
-
     def popitem(d):
         for k in dict.iterkeys(d):
             break
@@ -98,7 +89,6 @@
             return v
 ''', filename=__file__)
 
-dict_update__ANY             = app.interphook("update")
 dict_popitem__ANY            = app.interphook("popitem")
 dict_setdefault__ANY_ANY_ANY = app.interphook("setdefault")
 update1                      = app.interphook("update1")


From arigo at codespeak.net  Tue Oct 26 14:35:06 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 14:35:06 +0200 (CEST)
Subject: [pypy-svn] r78294 - pypy/branch/set-object-cleanup/pypy/objspace/std
Message-ID: <20101026123506.66045282C18@codespeak.net>

Author: arigo
Date: Tue Oct 26 14:35:04 2010
New Revision: 78294

Modified:
   pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
Log:
Fix the name that appears in error messages.


Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	Tue Oct 26 14:35:04 2010
@@ -641,9 +641,9 @@
 init_signature = Signature(['seq_or_map'], None, 'kwargs')
 init_defaults = [None]
 
-def init__DictMulti(space, w_dict, __args__):
+def init_or_update(space, w_dict, __args__, funcname):
     w_src, w_kwds = __args__.parse_obj(
-            None, 'dict',
+            None, funcname,
             init_signature, # signature
             init_defaults)  # default argument
     if w_src is None:
@@ -665,8 +665,11 @@
         from pypy.objspace.std.dicttype import update1
         update1(space, w_dict, w_kwds)
 
+def init__DictMulti(space, w_dict, __args__):
+    init_or_update(space, w_dict, __args__, 'dict')
+
 def dict_update__DictMulti(space, w_dict, __args__):
-    init__DictMulti(space, w_dict, __args__)
+    init_or_update(space, w_dict, __args__, 'dict.update')
 
 def getitem__DictMulti_ANY(space, w_dict, w_key):
     w_value = w_dict.getitem(w_key)


From arigo at codespeak.net  Tue Oct 26 14:52:14 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 14:52:14 +0200 (CEST)
Subject: [pypy-svn] r78295 - in
	pypy/branch/set-object-cleanup/pypy/objspace/std: . test
Message-ID: <20101026125214.041475080C@codespeak.net>

Author: arigo
Date: Tue Oct 26 14:52:13 2010
New Revision: 78295

Modified:
   pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
   pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
   pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_dictmultiobject.py
Log:
Move update1() to interp-level.


Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	Tue Oct 26 14:52:13 2010
@@ -641,28 +641,42 @@
 init_signature = Signature(['seq_or_map'], None, 'kwargs')
 init_defaults = [None]
 
+def update1(space, w_dict, w_data):
+    if space.findattr(w_data, space.wrap("keys")) is None:
+        # no 'keys' method, so we assume it is a sequence of pairs
+        for w_pair in space.listview(w_data):
+            pair = space.fixedview(w_pair)
+            if len(pair) != 2:
+                raise OperationError(space.w_ValueError,
+                             space.wrap("sequence of pairs expected"))
+            w_key, w_value = pair
+            w_dict.setitem(w_key, w_value)
+    else:
+        if isinstance(w_data, W_DictMultiObject):    # optimization case only
+            update1_dict_dict(space, w_dict, w_data)
+        else:
+            # general case -- "for k in o.keys(): dict.__setitem__(d, k, o[k])"
+            w_keys = space.call_method(w_data, "keys")
+            for w_key in space.listview(w_keys):
+                w_value = space.getitem(w_data, w_key)
+                w_dict.setitem(w_key, w_value)
+
+def update1_dict_dict(space, w_dict, w_data):
+    iterator = w_data.iter()
+    while 1:
+        w_key, w_value = iterator.next()
+        if w_key is None:
+            break
+        w_dict.setitem(w_key, w_value)
+
 def init_or_update(space, w_dict, __args__, funcname):
     w_src, w_kwds = __args__.parse_obj(
             None, funcname,
             init_signature, # signature
             init_defaults)  # default argument
-    if w_src is None:
-        pass
-    elif space.findattr(w_src, space.wrap("keys")) is None:
-        list_of_w_pairs = space.listview(w_src)
-        for w_pair in list_of_w_pairs:
-            pair = space.fixedview(w_pair)
-            if len(pair)!=2:
-                raise OperationError(space.w_ValueError,
-                             space.wrap("dict() takes a sequence of pairs"))
-            w_k, w_v = pair
-            w_dict.setitem(w_k, w_v)
-    else:
-        if space.is_true(w_src):
-            from pypy.objspace.std.dicttype import update1
-            update1(space, w_dict, w_src)
+    if w_src is not None:
+        update1(space, w_dict, w_src)
     if space.is_true(w_kwds):
-        from pypy.objspace.std.dicttype import update1
         update1(space, w_dict, w_kwds)
 
 def init__DictMulti(space, w_dict, __args__):
@@ -764,9 +778,8 @@
     return w_res
 
 def dict_copy__DictMulti(space, w_self):
-    from pypy.objspace.std.dicttype import update1
     w_new = W_DictMultiObject.allocate_and_init_instance(space)
-    update1(space, w_new, w_self)
+    update1_dict_dict(space, w_new, w_self)
     return w_new
 
 def dict_items__DictMulti(space, w_self):

Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	Tue Oct 26 14:52:13 2010
@@ -64,14 +64,6 @@
     # CPython does not call it when doing builtin operations.  The
     # same for other operations.
 
-    def update1(d, o):
-        if hasattr(o, 'keys'):
-            for k in o.keys():
-                dict.__setitem__(d, k, o[k])
-        else:
-            for k,v in o:
-                dict.__setitem__(d, k, v)
-
     def popitem(d):
         for k in dict.iterkeys(d):
             break
@@ -91,7 +83,6 @@
 
 dict_popitem__ANY            = app.interphook("popitem")
 dict_setdefault__ANY_ANY_ANY = app.interphook("setdefault")
-update1                      = app.interphook("update1")
 
 register_all(vars(), globals())
 

Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_dictmultiobject.py	Tue Oct 26 14:52:13 2010
@@ -446,7 +446,9 @@
         d1 = {}
         d2 = D(a='foo')
         d1.update(d2)
-        assert d1['a'] == 42 # fails on CPython, d1['a'] == 'foo'
+        assert d1['a'] == 'foo'
+        # a bit of an obscure case: now (from r78295) we get the same result
+        # as CPython does
 
     def test_index_keyerror_unpacking(self):
         d = {}


From arigo at codespeak.net  Tue Oct 26 14:54:24 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 14:54:24 +0200 (CEST)
Subject: [pypy-svn] r78296 - pypy/branch/set-object-cleanup/pypy/objspace/std
Message-ID: <20101026125424.5DF74282C1A@codespeak.net>

Author: arigo
Date: Tue Oct 26 14:54:22 2010
New Revision: 78296

Modified:
   pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
   pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
Log:
Move dict.setdefault() to interp-level.  For now it may still require two lookups
instead of one.


Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	Tue Oct 26 14:54:22 2010
@@ -810,6 +810,15 @@
     else:
         return w_default
 
+def dict_setdefault__DictMulti_ANY_ANY(space, w_dict, w_key, w_default):
+    # XXX should be more efficient, with only one dict lookup
+    w_value = w_dict.getitem(w_key)
+    if w_value is not None:
+        return w_value
+    else:
+        w_dict.setitem(w_key, w_default)
+        return w_default
+
 def dict_pop__DictMulti_ANY(space, w_dict, w_key, w_defaults):
     defaults = space.listview(w_defaults)
     len_defaults = len(defaults)

Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	Tue Oct 26 14:54:22 2010
@@ -72,17 +72,9 @@
         v = dict.__getitem__(d, k)
         dict.__delitem__(d, k)
         return k, v
-
-    def setdefault(d, k, v=None):
-        if k in d:
-            return dict.__getitem__(d, k)
-        else:
-            dict.__setitem__(d, k, v)
-            return v
 ''', filename=__file__)
 
 dict_popitem__ANY            = app.interphook("popitem")
-dict_setdefault__ANY_ANY_ANY = app.interphook("setdefault")
 
 register_all(vars(), globals())
 


From arigo at codespeak.net  Tue Oct 26 14:57:44 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 14:57:44 +0200 (CEST)
Subject: [pypy-svn] r78297 - pypy/branch/set-object-cleanup/pypy/objspace/std
Message-ID: <20101026125744.5543E282C1A@codespeak.net>

Author: arigo
Date: Tue Oct 26 14:57:42 2010
New Revision: 78297

Modified:
   pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
   pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
Log:
Interp-level implementation of dict.popitem().


Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	Tue Oct 26 14:57:42 2010
@@ -836,6 +836,16 @@
         w_dict.delitem(w_key)
         return w_item
 
+def dict_popitem__DictMulti(space, w_dict):
+    # XXX should somehow use the same trick as CPython: saving the index
+    # of the last popped item in the hash table, so that the next call to
+    # popitem() can be more efficient, instead of always starting from the
+    # beginning of the hash table.
+    iterator = w_dict.iter()
+    w_key, w_value = iterator.next()
+    w_dict.delitem(w_key)
+    return space.newtuple([w_key, w_value])
+
 app = gateway.applevel('''
     def dictrepr(currently_in_repr, d):
         # Now we only handle one implementation of dicts, this one.

Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dicttype.py	Tue Oct 26 14:57:42 2010
@@ -53,29 +53,6 @@
 # This can return when multimethods have been fixed
 #dict_str        = StdObjSpace.str
 
-# default application-level implementations for some operations
-# most of these (notably not popitem and update*) are overwritten
-# in dictmultiobject
-# gateway is imported in the stdtypedef module
-app = gateway.applevel('''
-
-    # in the following functions we use dict.__setitem__ instead of
-    # d[k]=...  because when a subclass of dict override __setitem__,
-    # CPython does not call it when doing builtin operations.  The
-    # same for other operations.
-
-    def popitem(d):
-        for k in dict.iterkeys(d):
-            break
-        else:
-            raise KeyError("popitem(): dictionary is empty")
-        v = dict.__getitem__(d, k)
-        dict.__delitem__(d, k)
-        return k, v
-''', filename=__file__)
-
-dict_popitem__ANY            = app.interphook("popitem")
-
 register_all(vars(), globals())
 
 @gateway.unwrap_spec(ObjSpace, W_Root, W_Root, W_Root)


From arigo at codespeak.net  Tue Oct 26 15:30:43 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 15:30:43 +0200 (CEST)
Subject: [pypy-svn] r78298 - in
	pypy/branch/set-object-cleanup/pypy/objspace/std: . test
Message-ID: <20101026133043.D28065080E@codespeak.net>

Author: arigo
Date: Tue Oct 26 15:30:42 2010
New Revision: 78298

Modified:
   pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
   pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_dictmultiobject.py
Log:
Test and fix.


Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/dictmultiobject.py	Tue Oct 26 15:30:42 2010
@@ -843,6 +843,9 @@
     # beginning of the hash table.
     iterator = w_dict.iter()
     w_key, w_value = iterator.next()
+    if w_key is None:
+        raise OperationError(space.w_KeyError,
+                             space.wrap("popitem(): dictionary is empty"))
     w_dict.delitem(w_key)
     return space.newtuple([w_key, w_value])
 

Modified: pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/branch/set-object-cleanup/pypy/objspace/std/test/test_dictmultiobject.py	Tue Oct 26 15:30:42 2010
@@ -238,6 +238,7 @@
         it1 = d.popitem()
         assert len(d) == 0
         assert (it!=it1) and (it1==(1,2) or it1==(3,4))
+        raises(KeyError, d.popitem)
     
     def test_setdefault(self):
         d = {1:2, 3:4}


From david at codespeak.net  Tue Oct 26 16:35:11 2010
From: david at codespeak.net (david at codespeak.net)
Date: Tue, 26 Oct 2010 16:35:11 +0200 (CEST)
Subject: [pypy-svn] r78299 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101026143511.1A5A0282C18@codespeak.net>

Author: david
Date: Tue Oct 26 16:35:09 2010
New Revision: 78299

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
Log:
Add support for constant int values from args

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Tue Oct 26 16:35:09 2010
@@ -1,10 +1,11 @@
 from pypy.jit.backend.arm import conditions as c
+from pypy.jit.backend.arm import locations
 from pypy.jit.backend.arm import registers as r
 from pypy.jit.backend.arm.arch import WORD, FUNC_ALIGN
 from pypy.jit.backend.arm.codebuilder import ARMv7Builder, ARMv7InMemoryBuilder
 from pypy.jit.backend.arm.regalloc import ARMRegisterManager
 from pypy.jit.backend.llsupport.regalloc import compute_vars_longevity
-from pypy.jit.metainterp.history import ConstInt, Box, BasicFailDescr
+from pypy.jit.metainterp.history import ConstInt, BoxInt, Box, BasicFailDescr
 from pypy.jit.metainterp.resoperation import rop
 from pypy.rlib import rgc
 from pypy.rpython.annlowlevel import llhelper
@@ -46,6 +47,7 @@
         Registers are saved on the stack
         XXX Rest to follow"""
         i = -1
+        fail_index = 0
         while(True):
             i += 1
             r = enc[i]
@@ -53,11 +55,20 @@
                 continue
             if r == '\xFF':
                 break
-            reg = ord(enc[i])
-            self.fail_boxes_int.setitem(i, self.decode32(stack, reg*WORD))
+            if r == '\xFD':
+                # imm value
+                value = self.decode32(enc, i+1)
+                i += 4
+            else:
+                reg = ord(enc[i])
+                value = self.decode32(stack, reg*WORD)
+
+            self.fail_boxes_int.setitem(fail_index, value)
+            fail_index += 1
+
         assert enc[i] == '\xFF'
         descr = self.decode32(enc, i+1)
-        self.fail_boxes_count = i
+        self.fail_boxes_count = fail_index
         return descr
 
     def decode32(self, mem, index):
@@ -97,21 +108,31 @@
         box = Box()
         reg = regalloc.try_allocate_reg(box)
         # XXX free this memory
-        mem = lltype.malloc(rffi.CArray(lltype.Char), len(args)+5, flavor='raw')
-        for i in range(len(args)):
+        mem = lltype.malloc(rffi.CArray(lltype.Char), (len(args)+5)*4, flavor='raw')
+        i = 0
+        j = 0
+        while(i < len(args)):
             if args[i]:
-                curreg = regalloc.try_allocate_reg(args[i])
-                mem[i] = chr(curreg.value)
+                if not isinstance(args[i], ConstInt):
+                    curreg = regalloc.try_allocate_reg(args[i])
+                    mem[j] = chr(curreg.value)
+                    j+=1
+                else:
+                    mem[j] = '\xFD'
+                    j+=1
+                    self.encode32(mem, j, args[i].getint())
+                    j+=4
             else:
-                mem[i] = '\xFE'
+                mem[j] = '\xFE'
+                j+=1
+            i+=1
 
-        i = len(args)
-        mem[i] = chr(0xFF)
+        mem[j] = chr(0xFF)
         memaddr = rffi.cast(lltype.Signed, mem)
 
 
         n = self.cpu.get_fail_descr_number(op.getdescr())
-        self.encode32(mem, i+1, n)
+        self.encode32(mem, j+1, n)
         self.mc.gen_load_int(r.lr.value, memaddr, cond=fcond)
         self.mc.gen_load_int(reg.value, self.mc.baseaddr(), cond=fcond)
         self.mc.MOV_rr(r.pc.value, reg.value, cond=fcond)
@@ -146,13 +167,13 @@
     def assemble_loop(self, inputargs, operations, looptoken):
         longevity = compute_vars_longevity(inputargs, operations)
         regalloc = ARMRegisterManager(longevity, assembler=self.mc)
+        self.align()
         loop_start=self.mc.curraddr()
         self.gen_func_prolog()
         self.gen_bootstrap_code(inputargs, regalloc, looptoken)
         loop_head=self.mc.curraddr()
         looptoken._arm_bootstrap_code = loop_start
         looptoken._arm_loop_code = loop_head
-        looptoken._temp_inputargs = inputargs#XXX remove
         fcond=c.AL
         for op in operations:
             opnum = op.getopnum()
@@ -221,10 +242,17 @@
         return c.GT
 
     def emit_op_int_add(self, op, regalloc, fcond):
-        reg = regalloc.try_allocate_reg(op.getarg(0))
+        if isinstance(op.getarg(0), BoxInt) and isinstance(op.getarg(1), BoxInt):
+            explode
         res = regalloc.try_allocate_reg(op.result)
-        assert isinstance(op.getarg(1), ConstInt)
-        self.mc.ADD_ri(res.value, reg.value, op.getarg(1).getint())
+        if isinstance(op.getarg(1), ConstInt):
+            reg = regalloc.try_allocate_reg(op.getarg(0))
+            arg1 = op.getarg(1)
+        else:
+            reg = regalloc.try_allocate_reg(op.getarg(1))
+            arg1 = op.getarg(0)
+
+        self.mc.ADD_ri(res.value, reg.value, arg1.getint())
         regalloc.possibly_free_vars_for_op(op)
         return fcond
 


From afa at codespeak.net  Tue Oct 26 16:49:50 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 16:49:50 +0200 (CEST)
Subject: [pypy-svn] r78300 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101026144950.47A34282C18@codespeak.net>

Author: afa
Date: Tue Oct 26 16:49:43 2010
New Revision: 78300

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Log:
Implement BufferedWriter: write(), flush().


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Tue Oct 26 16:49:43 2010
@@ -8,7 +8,7 @@
 from pypy.rlib.rstring import StringBuilder
 from pypy.rlib.rarithmetic import r_longlong
 from pypy.module._io.interp_iobase import W_IOBase, convert_size
-from pypy.module._io.interp_io import DEFAULT_BUFFER_SIZE
+from pypy.module._io.interp_io import DEFAULT_BUFFER_SIZE, W_BlockingIOError
 from pypy.module.thread.os_lock import Lock
 
 class BlockingIOError(Exception):
@@ -17,14 +17,53 @@
 class W_BufferedIOBase(W_IOBase):
     def __init__(self, space):
         W_IOBase.__init__(self, space)
+
         self.buffer = lltype.nullptr(rffi.CCHARP.TO)
-        self.pos = 0     # Current logical position in the buffer
-        self.raw_pos = 0 # Position of the raw stream in the buffer.
+        self.pos = 0        # Current logical position in the buffer
+        self.raw_pos = 0    # Position of the raw stream in the buffer.
+
+        self.read_end = -1  # Just after the last buffered byte in the buffer,
+                            # or -1 if the buffer isn't ready for reading
+
+        self.write_pos = 0  # Just after the last byte actually written
+        self.write_end = -1 # Just after the last byte waiting to be written,
+                            # or -1 if the buffer isn't ready for writing.
+
         self.lock = None
 
         self.readable = False
         self.writable = False
 
+    def _unsupportedoperation(self, space, message):
+        w_exc = space.getattr(space.getbuiltinmodule('_io'),
+                              space.wrap('UnsupportedOperation'))
+        raise OperationError(w_exc, space.wrap(message))
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def read_w(self, space, w_size=None):
+        self._unsupportedoperation(space, "read")
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def write_w(self, space, w_size=None):
+        self._unsupportedoperation(space, "write")
+
+    def _reader_reset_buf(self):
+        self.read_end = -1
+
+    def _writer_reset_buf(self):
+        self.write_pos = 0
+        self.write_end = -1
+
+W_BufferedIOBase.typedef = TypeDef(
+    '_BufferedIOBase', W_IOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedIOBase),
+    read = interp2app(W_BufferedIOBase.read_w),
+    write = interp2app(W_BufferedIOBase.write_w),
+    )
+
+class BufferedMixin:
+    _mixin_ = True
+
     def _init(self, space):
         if self.buffer_size <= 0:
             raise OperationError(space.w_ValueError, space.wrap(
@@ -61,21 +100,12 @@
         return 0
 
     def _raw_offset(self):
-        if self.raw_pos and (
+        if self.raw_pos >= 0 and (
             (self.readable and self.read_end != -1) or
             (self.writable and self.write_end != -1)):
             return self.raw_pos - self.pos
         return 0
 
-    def _unsupportedoperation(self, space, message):
-        w_exc = space.getattr(space.getbuiltinmodule('_io'),
-                              space.wrap('UnsupportedOperation'))
-        raise OperationError(w_exc, space.wrap(message))
-
-    @unwrap_spec('self', ObjSpace, W_Root)
-    def read_w(self, space, w_size=None):
-        self._unsupportedoperation(space, "read")
-
     @unwrap_spec('self', ObjSpace, r_longlong, int)
     def seek_w(self, space, pos, whence=0):
         if whence not in (0, 1, 2):
@@ -100,7 +130,7 @@
         # Fallback: invoke raw seek() method and clear buffer
         with self.lock:
             if self.writable:
-                self._writer_flush_unlocked(restore_pos=False)
+                self._writer_flush_unlocked(space)
                 self._writer_reset_buf()
 
             if whence == 1:
@@ -120,14 +150,80 @@
                 "Raw stream returned invalid position"))
         return pos
 
-W_BufferedIOBase.typedef = TypeDef(
-    '_BufferedIOBase', W_IOBase.typedef,
-    __new__ = generic_new_descr(W_BufferedIOBase),
-    read = interp2app(W_BufferedIOBase.read_w),
-    seek = interp2app(W_BufferedIOBase.seek_w),
-    )
+    def _closed(self, space):
+        return self.raw._closed(space)
+
+    @unwrap_spec('self', ObjSpace)
+    def close_w(self, space):
+        with self.lock:
+            if self._closed(space):
+                return
+        space.call_method(self, "flush")
+        with self.lock:
+            space.call_method(self.raw, "close")
+
+    @unwrap_spec('self', ObjSpace)
+    def flush_w(self, space):
+        return space.call_method(self.raw, "flush")
+
+    def _writer_flush_unlocked(self, space, restore_pos=False):
+        if self.write_end == -1 or self.write_pos == self.write_end:
+            return
+        # First, rewind
+        rewind = self._raw_offset() + (self.pos - self.write_pos)
+        if rewind != 0:
+            self._raw_seek(space, -rewind, 1)
+            self.raw_pos -= rewind
 
-class W_BufferedReader(W_BufferedIOBase):
+        written = 0
+        while self.write_pos < self.write_end:
+            try:
+                n = self._raw_write(space, self.write_pos, self.write_end)
+            except OperationError, e:
+                if not e.match(space, space.gettypeobject(
+                    W_BlockingIOError.typedef)):
+                    raise
+                self.write_pos += e.written
+                self.raw_pos = self.write_pos
+                written += e.written
+                # re-raise the error
+                e.written = written
+                raise
+            self.write_pos += n
+            self.raw_pos = self.write_pos
+            written += n
+            # Partial writes can return successfully when interrupted by a
+            # signal (see write(2)).  We must run signal handlers before
+            # blocking another time, possibly indefinitely.
+            # XXX PyErr_CheckSignals()
+
+        if restore_pos:
+            forward = rewind - written
+            if forward:
+                self._raw_seek(space, forward, 1)
+                self.raw_pos += forward
+
+        self._writer_reset_buf()
+
+    def _write(self, space, data):
+        w_data = space.wrap(data)
+        w_written = space.call_method(self.raw, "write", w_data)
+        written = space.getindex_w(w_written, space.w_IOError)
+        if not 0 <= written <= len(data):
+            raise OperationError(space.w_IOError, space.wrap(
+                "raw write() returned invalid length"))
+        if self.abs_pos != -1:
+            self.abs_pos += written
+        return written
+
+    def _raw_write(self, space, start, end):
+        # XXX inefficient
+        l = []
+        for i in range(start, end):
+            l.append(self.buffer[i])
+        return self._write(space, ''.join(l))
+
+class W_BufferedReader(W_BufferedIOBase, BufferedMixin):
     def __init__(self, space):
         W_BufferedIOBase.__init__(self, space)
         self.ok = False
@@ -145,12 +241,6 @@
         self._init(space)
         self._reader_reset_buf()
 
-    def _reader_reset_buf(self):
-        self.read_end = -1
-
-    def _closed(self, space):
-        return self.raw._closed(space)
-
     @unwrap_spec('self', ObjSpace, W_Root)
     def read_w(self, space, w_size=None):
         self._check_closed(space, "read of closed file")
@@ -180,7 +270,7 @@
         self._reader_reset_buf()
         # We're going past the buffer's bounds, flush it
         if self.writable:
-            self._writer_flush_unlocked(restore_pos=True)
+            self._writer_flush_unlocked(space, restore_pos=True)
 
         while True:
             # Read until EOF or until read() would block
@@ -239,7 +329,7 @@
         # XXX potential bug in CPython? The following is not enabled.
         # We're going past the buffer's bounds, flush it
         ## if self.writable:
-        ##     self._writer_flush_unlocked(restore_pos=True)
+        ##     self._writer_flush_unlocked(space, restore_pos=True)
 
         # Read whole blocks, and don't buffer them
         while remaining > 0:
@@ -302,13 +392,159 @@
     __init__  = interp2app(W_BufferedReader.descr_init),
 
     read = interp2app(W_BufferedReader.read_w),
+
+    # from the mixin class
+    seek = interp2app(W_BufferedReader.seek_w),
+    close = interp2app(W_BufferedReader.close_w),
+    flush = interp2app(W_BufferedReader.flush_w),
     )
 
-class W_BufferedWriter(W_BufferedIOBase):
-    pass
+class W_BufferedWriter(W_BufferedIOBase, BufferedMixin):
+    @unwrap_spec('self', ObjSpace, W_Root, int)
+    def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE):
+        raw = space.interp_w(W_IOBase, w_raw)
+        raw.check_writable_w(space)
+
+        self.raw = raw
+        self.buffer_size = buffer_size
+        self.writable = True
+
+        self._init(space)
+        self._writer_reset_buf()
+
+    def _adjust_position(self, new_pos):
+        self.pos = new_pos
+        if self.readable and self.read_end != -1 and self.read_end < new_pos:
+            self.read_end = self.pos
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def write_w(self, space, w_data):
+        self._check_closed(space, "write to closed file")
+        data = space.str_w(w_data)
+        size = len(data)
+
+        with self.lock:
+
+            if (not (self.readable and self.read_end == -1) and
+                not (self.writable and self.write_end == -1)):
+                self.pos = 0
+                self.raw_pos = 0
+            available = self.buffer_size - self.pos
+            # Fast path: the data to write can be fully buffered
+            if size <= available:
+                for i in range(size):
+                    self.buffer[self.pos + i] = data[i]
+                if self.write_end == -1:
+                    self.write_pos = self.pos
+                self._adjust_position(self.pos + size)
+                if self.pos > self.write_end:
+                    self.write_end = self.pos
+                return space.wrap(size)
+
+            # First write the current buffer
+            try:
+                self._writer_flush_unlocked(space)
+            except OperationError, e:
+                if not e.match(space, space.gettypeobject(
+                    W_BlockingIOError.typedef)):
+                    raise
+                if self.readable:
+                    self._reader_reset_buf()
+                # Make some place by shifting the buffer
+                for i in range(self.write_pos, self.write_end):
+                    self.buffer[i - self.write_pos] = self.buffer[i]
+                self.write_end -= self.write_pos
+                self.raw_pos -= self.write_pos
+                self.pos -= self.write_pos
+                self.write_pos = 0
+                available = self.buffer_size - self.write_end
+                if size <= available:
+                    # Everything can be buffered
+                    for i in range(size):
+                        self.buffer[self.write_end + i] = data[i]
+                    self.write_end += size
+                    return space.wrap(size)
+                # Buffer as much as possible
+                for i in range(available):
+                    self.buffer[self.write_end + i] = data[i]
+                    self.write_end += available
+                # Raise previous exception
+                e.written = available
+                raise
+
+            # Adjust the raw stream position if it is away from the logical
+            # stream position. This happens if the read buffer has been filled
+            # but not modified (and therefore _bufferedwriter_flush_unlocked()
+            # didn't rewind the raw stream by itself).
+            offset = self._raw_offset()
+            if offset:
+                self._raw_seek(space, -offset, 1)
+                self.raw_pos -= offset
+
+            # Then write buf itself. At this point the buffer has been emptied
+            remaining = size
+            written = 0
+            while remaining > self.buffer_size:
+                try:
+                    n = self._write(space, data[written:size - written])
+                except OperationError, e:
+                    if not e.match(space, space.gettypeobject(
+                        W_BlockingIOError.typedef)):
+                        raise
+                    written += e.written
+                    remaining -= e.written
+                    if remaining > self.buffer_size:
+                        # Can't buffer everything, still buffer as much as
+                        # possible
+                        for i in range(self.buffer_size):
+                            self.buffer[i] = data[written + i]
+                        self.raw_pos = 0
+                        self._adjust_position(self.buffer_size)
+                        self.write_end = self.buffer_size
+                        e.written = written + self.buffer_size
+                        raise
+                    break
+                written += n
+                remaining -= n
+                # Partial writes can return successfully when interrupted by a
+                # signal (see write(2)).  We must run signal handlers before
+                # blocking another time, possibly indefinitely.
+                # XXX PyErr_CheckSignals()
+
+            if self.readable:
+                self._reader_reset_buf()
+            if remaining > 0:
+                for i in range(remaining):
+                    self.buffer[i] = data[written + i]
+                written += remaining
+            self.write_pos = 0
+            self.write_end = remaining
+            self._adjust_position(remaining)
+            self.raw_pos = 0
+        return space.wrap(written)
+
+    @unwrap_spec('self', ObjSpace)
+    def flush_w(self, space):
+        self._check_closed(space, "flush of closed file")
+        with self.lock:
+            self._writer_flush_unlocked(space)
+            if self.readable:
+                # Rewind the raw stream so that its position corresponds to
+                # the current logical position.
+                self._raw_seek(space, -self._raw_offset(), 1)
+                self._reader_reset_buf()
+
 W_BufferedWriter.typedef = TypeDef(
     'BufferedWriter', W_BufferedIOBase.typedef,
     __new__ = generic_new_descr(W_BufferedWriter),
+    __init__  = interp2app(W_BufferedWriter.descr_init),
+
+    write = interp2app(W_BufferedWriter.write_w),
+    flush = interp2app(W_BufferedWriter.flush_w),
+
+    # from the mixin class
+    seek = interp2app(W_BufferedWriter.seek_w),
+    close = interp2app(W_BufferedWriter.close_w),
     )
 
 class W_BufferedRWPair(W_BufferedIOBase):

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Tue Oct 26 16:49:43 2010
@@ -1,7 +1,8 @@
 from pypy.conftest import gettestobjspace
+from pypy.interpreter.gateway import interp2app
 from pypy.tool.udir import udir
 
-class AppTestBufferedIO:
+class AppTestBufferedReader:
     def setup_class(cls):
         cls.space = gettestobjspace(usemodules=['_io'])
         tmpfile = udir.join('tmpfile')
@@ -15,7 +16,7 @@
         assert f.read() == "a\nb\nc"
         f.close()
         #
-        raw.seek(0)
+        raw = _io.FileIO(self.tmpfile)
         f = _io.BufferedReader(raw)
         r = f.read(4)
         assert r == "a\nb\n"
@@ -31,3 +32,28 @@
         f.seek(-2, 2)
         assert f.read() == "\nc"
         f.close()
+
+class AppTestBufferedWriter:
+    def setup_class(cls):
+        cls.space = gettestobjspace(usemodules=['_io'])
+        tmpfile = udir.join('tmpfile')
+        cls.w_tmpfile = cls.space.wrap(str(tmpfile))
+        def readfile(space):
+            return space.wrap(tmpfile.read())
+        cls.w_readfile = cls.space.wrap(interp2app(readfile))
+
+    def test_write(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile, 'w')
+        f = _io.BufferedWriter(raw)
+        f.write("abcd")
+        f.close()
+        assert self.readfile() == "abcd"
+
+    def test_largewrite(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile, 'w')
+        f = _io.BufferedWriter(raw)
+        f.write("abcd" * 5000)
+        f.close()
+        assert self.readfile() == "abcd" * 5000


From arigo at codespeak.net  Tue Oct 26 17:05:51 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 17:05:51 +0200 (CEST)
Subject: [pypy-svn] r78301 - in pypy/trunk/pypy: interpreter objspace/std
	objspace/std/test
Message-ID: <20101026150551.B0CB1282C18@codespeak.net>

Author: arigo
Date: Tue Oct 26 17:05:48 2010
New Revision: 78301

Modified:
   pypy/trunk/pypy/interpreter/baseobjspace.py
   pypy/trunk/pypy/objspace/std/dictmultiobject.py
   pypy/trunk/pypy/objspace/std/dicttype.py
   pypy/trunk/pypy/objspace/std/frozensettype.py
   pypy/trunk/pypy/objspace/std/setobject.py
   pypy/trunk/pypy/objspace/std/settype.py
   pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py
   pypy/trunk/pypy/objspace/std/test/test_setobject.py
Log:
Merge branch/set-object-cleanup.

General clean-up of setobject.py, reducing the number of copies
of the whole data that we need.

In dicttype.py, remove some very old app-level helpers and implement them
at interp-level.


Modified: pypy/trunk/pypy/interpreter/baseobjspace.py
==============================================================================
--- pypy/trunk/pypy/interpreter/baseobjspace.py	(original)
+++ pypy/trunk/pypy/interpreter/baseobjspace.py	Tue Oct 26 17:05:48 2010
@@ -739,7 +739,7 @@
 
     def unpackiterable(self, w_iterable, expected_length=-1):
         """Unpack an iterable object into a real (interpreter-level) list.
-        Raise a real (subclass of) ValueError if the length is wrong."""
+        Raise an OperationError(w_ValueError) if the length is wrong."""
         w_iterator = self.iter(w_iterable)
         items = []
         while True:

Modified: pypy/trunk/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/dictmultiobject.py	Tue Oct 26 17:05:48 2010
@@ -641,30 +641,50 @@
 init_signature = Signature(['seq_or_map'], None, 'kwargs')
 init_defaults = [None]
 
-def init__DictMulti(space, w_dict, __args__):
-    w_src, w_kwds = __args__.parse_obj(
-            None, 'dict',
-            init_signature, # signature
-            init_defaults)                           # default argument
-    if w_src is None:
-        pass
-    elif space.findattr(w_src, space.wrap("keys")) is None:
-        list_of_w_pairs = space.listview(w_src)
-        for w_pair in list_of_w_pairs:
+def update1(space, w_dict, w_data):
+    if space.findattr(w_data, space.wrap("keys")) is None:
+        # no 'keys' method, so we assume it is a sequence of pairs
+        for w_pair in space.listview(w_data):
             pair = space.fixedview(w_pair)
-            if len(pair)!=2:
+            if len(pair) != 2:
                 raise OperationError(space.w_ValueError,
-                             space.wrap("dict() takes a sequence of pairs"))
-            w_k, w_v = pair
-            w_dict.setitem(w_k, w_v)
+                             space.wrap("sequence of pairs expected"))
+            w_key, w_value = pair
+            w_dict.setitem(w_key, w_value)
     else:
-        if space.is_true(w_src):
-            from pypy.objspace.std.dicttype import update1
-            update1(space, w_dict, w_src)
+        if isinstance(w_data, W_DictMultiObject):    # optimization case only
+            update1_dict_dict(space, w_dict, w_data)
+        else:
+            # general case -- "for k in o.keys(): dict.__setitem__(d, k, o[k])"
+            w_keys = space.call_method(w_data, "keys")
+            for w_key in space.listview(w_keys):
+                w_value = space.getitem(w_data, w_key)
+                w_dict.setitem(w_key, w_value)
+
+def update1_dict_dict(space, w_dict, w_data):
+    iterator = w_data.iter()
+    while 1:
+        w_key, w_value = iterator.next()
+        if w_key is None:
+            break
+        w_dict.setitem(w_key, w_value)
+
+def init_or_update(space, w_dict, __args__, funcname):
+    w_src, w_kwds = __args__.parse_obj(
+            None, funcname,
+            init_signature, # signature
+            init_defaults)  # default argument
+    if w_src is not None:
+        update1(space, w_dict, w_src)
     if space.is_true(w_kwds):
-        from pypy.objspace.std.dicttype import update1
         update1(space, w_dict, w_kwds)
 
+def init__DictMulti(space, w_dict, __args__):
+    init_or_update(space, w_dict, __args__, 'dict')
+
+def dict_update__DictMulti(space, w_dict, __args__):
+    init_or_update(space, w_dict, __args__, 'dict.update')
+
 def getitem__DictMulti_ANY(space, w_dict, w_key):
     w_value = w_dict.getitem(w_key)
     if w_value is not None:
@@ -758,9 +778,8 @@
     return w_res
 
 def dict_copy__DictMulti(space, w_self):
-    from pypy.objspace.std.dicttype import update1
     w_new = W_DictMultiObject.allocate_and_init_instance(space)
-    update1(space, w_new, w_self)
+    update1_dict_dict(space, w_new, w_self)
     return w_new
 
 def dict_items__DictMulti(space, w_self):
@@ -791,6 +810,15 @@
     else:
         return w_default
 
+def dict_setdefault__DictMulti_ANY_ANY(space, w_dict, w_key, w_default):
+    # XXX should be more efficient, with only one dict lookup
+    w_value = w_dict.getitem(w_key)
+    if w_value is not None:
+        return w_value
+    else:
+        w_dict.setitem(w_key, w_default)
+        return w_default
+
 def dict_pop__DictMulti_ANY(space, w_dict, w_key, w_defaults):
     defaults = space.listview(w_defaults)
     len_defaults = len(defaults)
@@ -808,6 +836,19 @@
         w_dict.delitem(w_key)
         return w_item
 
+def dict_popitem__DictMulti(space, w_dict):
+    # XXX should somehow use the same trick as CPython: saving the index
+    # of the last popped item in the hash table, so that the next call to
+    # popitem() can be more efficient, instead of always starting from the
+    # beginning of the hash table.
+    iterator = w_dict.iter()
+    w_key, w_value = iterator.next()
+    if w_key is None:
+        raise OperationError(space.w_KeyError,
+                             space.wrap("popitem(): dictionary is empty"))
+    w_dict.delitem(w_key)
+    return space.newtuple([w_key, w_value])
+
 app = gateway.applevel('''
     def dictrepr(currently_in_repr, d):
         # Now we only handle one implementation of dicts, this one.

Modified: pypy/trunk/pypy/objspace/std/dicttype.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/dicttype.py	(original)
+++ pypy/trunk/pypy/objspace/std/dicttype.py	Tue Oct 26 17:05:48 2010
@@ -53,106 +53,21 @@
 # This can return when multimethods have been fixed
 #dict_str        = StdObjSpace.str
 
-# default application-level implementations for some operations
-# most of these (notably not popitem and update*) are overwritten
-# in dictmultiobject
-# gateway is imported in the stdtypedef module
-app = gateway.applevel('''
-
-    # in the following functions we use dict.__setitem__ instead of
-    # d[k]=...  because when a subclass of dict override __setitem__,
-    # CPython does not call it when doing builtin operations.  The
-    # same for other operations.
-
-    def update1(d, o):
-        if hasattr(o, 'keys'):
-            for k in o.keys():
-                dict.__setitem__(d, k, o[k])
-        else:
-            for k,v in o:
-                dict.__setitem__(d, k, v)
-
-    def update(d, *args, **kwargs):
-        len_args = len(args)
-        if len_args == 1:
-            update1(d, args[0])
-        elif len_args > 1:
-            raise TypeError("update takes at most 1 (non-keyword) argument")
-        if kwargs:
-            update1(d, kwargs)
-
-    def popitem(d):
-        for k in dict.iterkeys(d):
-            break
-        else:
-            raise KeyError("popitem(): dictionary is empty")
-        v = dict.__getitem__(d, k)
-        dict.__delitem__(d, k)
-        return k, v
-
-    def get(d, k, v=None):
-        if k in d:
-            return dict.__getitem__(d, k)
-        else:
-            return v
-
-    def setdefault(d, k, v=None):
-        if k in d:
-            return dict.__getitem__(d, k)
-        else:
-            dict.__setitem__(d, k, v)
-            return v
-
-    def pop(d, k, defaults):     # XXX defaults is actually *defaults
-        if len(defaults) > 1:
-            raise TypeError, "pop expected at most 2 arguments, got %d" % (
-                1 + len(defaults))
-        try:
-            v = dict.__getitem__(d, k)
-            dict.__delitem__(d, k)
-        except KeyError, e:
-            if defaults:
-                return defaults[0]
-            else:
-                raise e
-        return v
-
-    def iteritems(d):
-        return iter(dict.items(d))
-
-    def iterkeys(d):
-        return iter(dict.keys(d))
-
-    def itervalues(d):
-        return iter(dict.values(d))
-''', filename=__file__)
-
-dict_update__ANY             = app.interphook("update")
-dict_popitem__ANY            = app.interphook("popitem")
-dict_get__ANY_ANY_ANY        = app.interphook("get")
-dict_setdefault__ANY_ANY_ANY = app.interphook("setdefault")
-dict_pop__ANY_ANY            = app.interphook("pop")
-dict_iteritems__ANY          = app.interphook("iteritems")
-dict_iterkeys__ANY           = app.interphook("iterkeys")
-dict_itervalues__ANY         = app.interphook("itervalues")
-update1                      = app.interphook("update1")
-
 register_all(vars(), globals())
 
 @gateway.unwrap_spec(ObjSpace, W_Root, W_Root, W_Root)
 def descr_fromkeys(space, w_type, w_keys, w_fill=None):
+    from pypy.objspace.std.dictmultiobject import W_DictMultiObject
     if w_fill is None:
         w_fill = space.w_None
-    w_dict = space.call_function(w_type)
-    w_iter = space.iter(w_keys)
-    while True:
-        try:
-            w_key = space.next(w_iter)
-        except OperationError, e:
-            if not e.match(space, space.w_StopIteration):
-                raise
-            break
-        space.setitem(w_dict, w_key, w_fill)
+    if w_type is space.w_dict:
+        w_dict = W_DictMultiObject.allocate_and_init_instance(space, w_type)
+        for w_key in space.listview(w_keys):
+            w_dict.setitem(w_key, w_fill)
+    else:
+        w_dict = space.call_function(w_type)
+        for w_key in space.listview(w_keys):
+            space.setitem(w_dict, w_key, w_fill)
     return w_dict
 
 

Modified: pypy/trunk/pypy/objspace/std/frozensettype.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/frozensettype.py	(original)
+++ pypy/trunk/pypy/objspace/std/frozensettype.py	Tue Oct 26 17:05:48 2010
@@ -39,13 +39,13 @@
 def descr__frozenset__new__(space, w_frozensettype,
                             w_iterable=gateway.NoneNotWrapped):
     from pypy.objspace.std.setobject import W_FrozensetObject
-    from pypy.objspace.std.setobject import _is_frozenset_exact
+    from pypy.objspace.std.setobject import make_setdata_from_w_iterable
     if (space.is_w(w_frozensettype, space.w_frozenset) and
-        _is_frozenset_exact(w_iterable)):
+        w_iterable is not None and type(w_iterable) is W_FrozensetObject):
         return w_iterable
     w_obj = space.allocate_instance(W_FrozensetObject, w_frozensettype)
-    W_FrozensetObject.__init__(w_obj, space, None)
-
+    data = make_setdata_from_w_iterable(space, w_iterable)
+    W_FrozensetObject.__init__(w_obj, space, data)
     return w_obj
 
 frozenset_typedef = StdTypeDef("frozenset",

Modified: pypy/trunk/pypy/objspace/std/setobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/setobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/setobject.py	Tue Oct 26 17:05:48 2010
@@ -21,11 +21,10 @@
         return False
 
 
-    def __init__(w_self, space, setdata=None):
-        if setdata is None:
-            w_self.setdata = r_dict(space.eq_w, space.hash_w)
-        else:
-            w_self.setdata = setdata.copy()
+    def __init__(w_self, space, setdata):
+        """Initialize the set by taking ownership of 'setdata'."""
+        assert setdata is not None
+        w_self.setdata = setdata
 
     def __repr__(w_self):
         """representation for debugging purposes"""
@@ -33,6 +32,7 @@
         return "<%s(%s)>" % (w_self.__class__.__name__, ', '.join(reprlist))
 
     def _newobj(w_self, space, rdict_w=None):
+        """Make a new set or frozenset by taking ownership of 'rdict_w'."""
         #return space.call(space.type(w_self),W_SetIterObject(rdict_w))
         objtype = type(w_self)
         if objtype is W_SetObject:
@@ -55,10 +55,7 @@
 
 class W_FrozensetObject(W_BaseSetObject):
     from pypy.objspace.std.frozensettype import frozenset_typedef as typedef
-
-    def __init__(w_self, space, setdata):
-        W_BaseSetObject.__init__(w_self, space, setdata)
-        w_self.hash = -1
+    hash = 0
 
 registerimplementation(W_BaseSetObject)
 registerimplementation(W_SetObject)
@@ -109,8 +106,14 @@
 
 # some helper functions
 
+def newset(space):
+    return r_dict(space.eq_w, space.hash_w)
+
 def make_setdata_from_w_iterable(space, w_iterable=None):
-    data = r_dict(space.eq_w, space.hash_w)
+    """Return a new r_dict with the content of w_iterable."""
+    if isinstance(w_iterable, W_BaseSetObject):
+        return w_iterable.setdata.copy()
+    data = newset(space)
     if w_iterable is not None:
         for w_item in space.listview(w_iterable):
             data[w_item] = None
@@ -119,7 +122,7 @@
 def _initialize_set(space, w_obj, w_iterable=None):
     w_obj.setdata.clear()
     if w_iterable is not None:
-        w_obj.setdata.update(make_setdata_from_w_iterable(space, w_iterable))
+        w_obj.setdata = make_setdata_from_w_iterable(space, w_iterable)
 
 def _convert_set_to_frozenset(space, w_obj):
     if space.is_true(space.isinstance(w_obj, space.w_set)):
@@ -130,12 +133,6 @@
 
 # helper functions for set operation on dicts
 
-def _is_frozenset_exact(w_obj):
-    if (w_obj is not None) and (type(w_obj) is W_FrozensetObject):
-        return True
-    else:
-        return False
-
 def _is_eq(ld, rd):
     if len(ld) != len(rd):
         return False
@@ -144,66 +141,41 @@
             return False
     return True
 
-def _union_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    ld.update(rdict)
-    return ld, rdict
-
-def _difference_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    del_list_w = []
+def _difference_dict(space, ld, rd):
+    result = newset(space)
     for w_key in ld:
-        if w_key in rdict:
-            del_list_w.append(w_key)
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    return ld, rdict
-
-def _intersection_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    del_list_w = []
-    for w_key in ld:
-        if w_key not in rdict:
-            del_list_w.append(w_key)
-
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    return ld, rdict
-
+        if w_key not in rd:
+            result[w_key] = None
+    return result
 
-def _symmetric_difference_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
+def _difference_dict_update(space, ld, rd):
+    if ld is rd:
+        ld.clear()     # for the case 'a.difference_update(a)'
     else:
-        ld = ldict.copy()
-    del_list_w = []
-    add_list_w = []
+        for w_key in rd:
+            try:
+                del ld[w_key]
+            except KeyError:
+                pass
+
+def _intersection_dict(space, ld, rd):
+    result = newset(space)
+    if len(ld) > len(rd):
+        ld, rd = rd, ld     # loop over the smaller dict
     for w_key in ld:
-        if w_key in rdict:
-            del_list_w.append(w_key)
+        if w_key in rd:
+            result[w_key] = None
+    return result
 
-    for w_key in rdict:
+def _symmetric_difference_dict(space, ld, rd):
+    result = newset(space)
+    for w_key in ld:
+        if w_key not in rd:
+            result[w_key] = None
+    for w_key in rd:
         if w_key not in ld:
-            add_list_w.append(w_key)
-
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    for w_key in add_list_w:
-        ld[w_key] = None
-
-    return ld, rdict
+            result[w_key] = None
+    return result
 
 def _issubset_dict(ldict, rdict):
     if len(ldict) > len(rdict):
@@ -220,14 +192,13 @@
 def set_update__Set_BaseSet(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _union_dict(ld, rd, True)
-    return space.w_None
+    ld.update(rd)
 
 def set_update__Set_ANY(space, w_left, w_other):
     """Update a set with the union of itself and another."""
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _union_dict(ld, rd, True)
-    return space.w_None
+    ld = w_left.setdata
+    for w_item in space.listview(w_other):
+        ld[w_item] = None
 
 def inplace_or__Set_Set(space, w_left, w_other):
     set_update__Set_BaseSet(space, w_left, w_other)
@@ -241,25 +212,23 @@
     This has no effect if the element is already present.
     """
     w_left.setdata[w_other] = None
-    return space.w_None
 
 def set_copy__Set(space, w_set):
-    return w_set._newobj(space,w_set.setdata)
+    return w_set._newobj(space, w_set.setdata.copy())
 
 def frozenset_copy__Frozenset(space, w_left):
-    if _is_frozenset_exact(w_left):
+    if type(w_left) is W_FrozensetObject:
         return w_left
     else:
-        return set_copy__Set(space,w_left)
+        return set_copy__Set(space, w_left)
 
 def set_clear__Set(space, w_left):
     w_left.setdata.clear()
-    return space.w_None
 
 def set_difference__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _difference_dict(ld, rd, False)
+    new_ld = _difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 set_difference__Set_Frozenset = set_difference__Set_Set
@@ -272,7 +241,7 @@
 
 def set_difference__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _difference_dict(ld, rd, False)
+    new_ld = _difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 frozenset_difference__Frozenset_ANY = set_difference__Set_ANY
@@ -281,15 +250,17 @@
 def set_difference_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _difference_dict(ld, rd, True)
-    return space.w_None
+    _difference_dict_update(space, ld, rd)
 
 set_difference_update__Set_Frozenset = set_difference_update__Set_Set
 
 def set_difference_update__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _difference_dict(ld, rd, True)
-    return space.w_None
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        try:
+            del ld[w_key]
+        except KeyError:
+            pass
 
 def inplace_sub__Set_Set(space, w_left, w_other):
     set_difference_update__Set_Set(space, w_left, w_other)
@@ -467,7 +438,7 @@
 
 def hash__Frozenset(space, w_set):
     multi = r_uint(1822399083) + r_uint(1822399083) + 1
-    if w_set.hash != -1:
+    if w_set.hash != 0:
         return space.wrap(w_set.hash)
     hash = 1927868237
     hash *= (len(w_set.setdata) + 1)
@@ -476,7 +447,7 @@
         value = ((h ^ (h << 16) ^ 89869747)  * multi)
         hash = intmask(hash ^ value)
     hash = hash * 69069 + 907133923
-    if hash == -1:
+    if hash == 0:
         hash = 590923713
     hash = intmask(hash)
     w_set.hash = hash
@@ -484,29 +455,31 @@
     return space.wrap(hash)
 
 def set_pop__Set(space, w_left):
-    if len(w_left.setdata) == 0:
+    for w_key in w_left.setdata:
+        break
+    else:
         raise OperationError(space.w_KeyError,
                                 space.wrap('pop from an empty set'))
-    w_keys = w_left.setdata.keys()
-    w_value = w_keys[0]
-    del w_left.setdata[w_value]
-
-    return w_value
+    del w_left.setdata[w_key]
+    return w_key
 
 def set_intersection__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _intersection_dict(ld, rd, False)
-    return w_left._newobj(space,new_ld)
+    new_ld = _intersection_dict(space, ld, rd)
+    return w_left._newobj(space, new_ld)
 
 set_intersection__Set_Frozenset = set_intersection__Set_Set
 set_intersection__Frozenset_Frozenset = set_intersection__Set_Set
 set_intersection__Frozenset_Set = set_intersection__Set_Set
 
 def set_intersection__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _intersection_dict(ld, rd, False)
-    return w_left._newobj(space,new_ld)
+    result = newset(space)
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        if w_key in ld:
+            result[w_key] = None
+    return w_left._newobj(space, result)
 
 frozenset_intersection__Frozenset_ANY = set_intersection__Set_ANY
 
@@ -518,15 +491,18 @@
 def set_intersection_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _intersection_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _intersection_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 set_intersection_update__Set_Frozenset = set_intersection_update__Set_Set
 
 def set_intersection_update__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _intersection_dict(ld, rd, True)
-    return space.w_None
+    result = newset(space)
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        if w_key in ld:
+            result[w_key] = None
+    w_left.setdata = result
 
 def inplace_and__Set_Set(space, w_left, w_other):
     set_intersection_update__Set_Set(space, w_left, w_other)
@@ -537,7 +513,7 @@
 def set_symmetric_difference__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _symmetric_difference_dict(ld, rd, False)
+    new_ld = _symmetric_difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 set_symmetric_difference__Set_Frozenset = set_symmetric_difference__Set_Set
@@ -553,7 +529,7 @@
 
 def set_symmetric_difference__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _symmetric_difference_dict(ld, rd, False)
+    new_ld = _symmetric_difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 frozenset_symmetric_difference__Frozenset_ANY = \
@@ -562,16 +538,16 @@
 def set_symmetric_difference_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _symmetric_difference_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _symmetric_difference_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 set_symmetric_difference_update__Set_Frozenset = \
                                     set_symmetric_difference_update__Set_Set
 
 def set_symmetric_difference_update__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _symmetric_difference_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _symmetric_difference_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 def inplace_xor__Set_Set(space, w_left, w_other):
     set_symmetric_difference_update__Set_Set(space, w_left, w_other)
@@ -582,8 +558,9 @@
 def set_union__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _union_dict(ld, rd, False)
-    return w_left._newobj(space, new_ld)
+    result = ld.copy()
+    result.update(rd)
+    return w_left._newobj(space, result)
 
 set_union__Set_Frozenset = set_union__Set_Set
 set_union__Frozenset_Set = set_union__Set_Set
@@ -595,9 +572,11 @@
 
 
 def set_union__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _union_dict(ld, rd, False)
-    return w_left._newobj(space, new_ld)
+    ld = w_left.setdata
+    result = ld.copy()
+    for w_key in space.listview(w_other):
+        result[w_key] = None
+    return w_left._newobj(space, result)
 
 frozenset_union__Frozenset_ANY = set_union__Set_ANY
 
@@ -629,15 +608,6 @@
             init_defaults)
     _initialize_set(space, w_set, w_iterable)
 
-def init__Frozenset(space, w_set, __args__):
-    w_iterable, = __args__.parse_obj(
-            None, 'set',
-            init_signature,
-            init_defaults)
-    if w_set.hash == -1:
-        _initialize_set(space, w_set, w_iterable)
-        hash__Frozenset(space, w_set)
-
 app = gateway.applevel("""
     def setrepr(currently_in_repr, s):
         'The app-level part of repr().'

Modified: pypy/trunk/pypy/objspace/std/settype.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/settype.py	(original)
+++ pypy/trunk/pypy/objspace/std/settype.py	Tue Oct 26 17:05:48 2010
@@ -66,9 +66,9 @@
 register_all(vars(), globals())
 
 def descr__new__(space, w_settype, __args__):
-    from pypy.objspace.std.setobject import W_SetObject
+    from pypy.objspace.std.setobject import W_SetObject, newset
     w_obj = space.allocate_instance(W_SetObject, w_settype)
-    W_SetObject.__init__(w_obj, space, None)
+    W_SetObject.__init__(w_obj, space, newset(space))
     return w_obj
 
 set_typedef = StdTypeDef("set",

Modified: pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py	Tue Oct 26 17:05:48 2010
@@ -238,6 +238,7 @@
         it1 = d.popitem()
         assert len(d) == 0
         assert (it!=it1) and (it1==(1,2) or it1==(3,4))
+        raises(KeyError, d.popitem)
     
     def test_setdefault(self):
         d = {1:2, 3:4}
@@ -446,7 +447,9 @@
         d1 = {}
         d2 = D(a='foo')
         d1.update(d2)
-        assert d1['a'] == 42 # fails on CPython, d1['a'] == 'foo'
+        assert d1['a'] == 'foo'
+        # a bit of an obscure case: now (from r78295) we get the same result
+        # as CPython does
 
     def test_index_keyerror_unpacking(self):
         d = {}

Modified: pypy/trunk/pypy/objspace/std/test/test_setobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_setobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_setobject.py	Tue Oct 26 17:05:48 2010
@@ -10,6 +10,7 @@
 import py.test
 from pypy.objspace.std.setobject import W_SetObject, W_FrozensetObject
 from pypy.objspace.std.setobject import _initialize_set
+from pypy.objspace.std.setobject import newset, make_setdata_from_w_iterable
 from pypy.objspace.std.setobject import set_intersection__Set_Set
 from pypy.objspace.std.setobject import set_intersection__Set_ANY
 from pypy.objspace.std.setobject import eq__Set_Set
@@ -28,12 +29,12 @@
         self.false = self.space.w_False
 
     def test_and(self):
-        s = W_SetObject(self.space, None)
+        s = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, s, self.word)
-        t0 = W_SetObject(self.space, None)
+        t0 = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, t0, self.otherword)
-        t1 = W_FrozensetObject(self.space, None)
-        _initialize_set(self.space, t1, self.otherword)
+        t1 = W_FrozensetObject(self.space,
+                make_setdata_from_w_iterable(self.space, self.otherword))
         r0 = set_intersection__Set_Set(self.space, s, t0)
         r1 = set_intersection__Set_Set(self.space, s, t1)
         assert eq__Set_Set(self.space, r0, r1) == self.true
@@ -41,9 +42,9 @@
         assert eq__Set_Set(self.space, r0, sr) == self.true
 
     def test_compare(self):
-        s = W_SetObject(self.space, None)
+        s = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, s, self.word)
-        t = W_SetObject(self.space, None)
+        t = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, t, self.word)
         assert self.space.eq_w(s,t)
         u = self.space.wrap(set('simsalabim'))
@@ -56,6 +57,11 @@
         b = a | set('abc')
         assert type(b) is subset
 
+    def test_union(self):
+        a = set([4, 5])
+        b = a.union([5, 7])
+        assert sorted(b) == [4, 5, 7]
+
     def test_compare(self):
         raises(TypeError, cmp, set('abc'), set('abd'))
         assert set('abc') != 'abc'


From arigo at codespeak.net  Tue Oct 26 17:06:10 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 17:06:10 +0200 (CEST)
Subject: [pypy-svn] r78302 - pypy/branch/set-object-cleanup
Message-ID: <20101026150610.5D9E6282C1A@codespeak.net>

Author: arigo
Date: Tue Oct 26 17:06:09 2010
New Revision: 78302

Removed:
   pypy/branch/set-object-cleanup/
Log:
Remove merged branch.



From david at codespeak.net  Tue Oct 26 17:12:27 2010
From: david at codespeak.net (david at codespeak.net)
Date: Tue, 26 Oct 2010 17:12:27 +0200 (CEST)
Subject: [pypy-svn] r78303 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101026151227.68AF5282C18@codespeak.net>

Author: david
Date: Tue Oct 26 17:12:25 2010
New Revision: 78303

Added:
   pypy/branch/arm-backend/pypy/jit/backend/arm/locations.py
Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
Log:
Implemente guard_false, int_sub, int_eq

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Tue Oct 26 17:12:25 2010
@@ -47,9 +47,10 @@
         Registers are saved on the stack
         XXX Rest to follow"""
         i = -1
-        fail_index = 0
+        fail_index = -1
         while(True):
             i += 1
+            fail_index += 1
             r = enc[i]
             if r == '\xFE':
                 continue
@@ -64,7 +65,6 @@
                 value = self.decode32(stack, reg*WORD)
 
             self.fail_boxes_int.setitem(fail_index, value)
-            fail_index += 1
 
         assert enc[i] == '\xFF'
         descr = self.decode32(enc, i+1)
@@ -116,16 +116,15 @@
                 if not isinstance(args[i], ConstInt):
                     curreg = regalloc.try_allocate_reg(args[i])
                     mem[j] = chr(curreg.value)
-                    j+=1
+                    j += 1
                 else:
                     mem[j] = '\xFD'
-                    j+=1
-                    self.encode32(mem, j, args[i].getint())
-                    j+=4
+                    self.encode32(mem, j+1, args[i].getint())
+                    j += 5
             else:
                 mem[j] = '\xFE'
-                j+=1
-            i+=1
+                j += 1
+            i += 1
 
         mem[j] = chr(0xFF)
         memaddr = rffi.cast(lltype.Signed, mem)
@@ -241,31 +240,70 @@
         self.mc.CMP(reg.value, op.getarg(1).getint())
         return c.GT
 
+    def emit_op_int_eq(self, op, regalloc, fcond):
+        reg = regalloc.try_allocate_reg(op.getarg(0))
+        assert isinstance(op.getarg(1), ConstInt)
+        self.mc.CMP(reg.value, op.getarg(1).getint())
+        return c.EQ
+
     def emit_op_int_add(self, op, regalloc, fcond):
-        if isinstance(op.getarg(0), BoxInt) and isinstance(op.getarg(1), BoxInt):
-            explode
+        # assuming only one argument is constant
         res = regalloc.try_allocate_reg(op.result)
         if isinstance(op.getarg(1), ConstInt):
             reg = regalloc.try_allocate_reg(op.getarg(0))
             arg1 = op.getarg(1)
+            self.mc.ADD_ri(res.value, reg.value, arg1.getint())
+        elif isinstance(op.getarg(0), ConstInt):
+            reg = regalloc.try_allocate_reg(op.getarg(1))
+            arg1 = op.getarg(0)
+            self.mc.ADD_ri(res.value, reg.value, arg1.getint())
         else:
+            r1 = regalloc.try_allocate_reg(op.getarg(0))
+            r2 = regalloc.try_allocate_reg(op.getarg(1))
+            self.mc.ADD_rr(res.value, r1.value, r2.value)
+
+        regalloc.possibly_free_vars_for_op(op)
+        return fcond
+
+    def emit_op_int_sub(self, op, regalloc, fcond):
+        # assuming only one argument is constant
+        res = regalloc.try_allocate_reg(op.result)
+        if isinstance(op.getarg(1), ConstInt):
+            reg = regalloc.try_allocate_reg(op.getarg(0))
+            arg1 = op.getarg(1)
+            self.mc.SUB_ri(res.value, reg.value, arg1.getint())
+        elif isinstance(op.getarg(0), ConstInt):
             reg = regalloc.try_allocate_reg(op.getarg(1))
             arg1 = op.getarg(0)
+            self.mc.SUB_ri(res.value, reg.value, arg1.getint())
+        else:
+            r1 = regalloc.try_allocate_reg(op.getarg(0))
+            r2 = regalloc.try_allocate_reg(op.getarg(1))
+            self.mc.SUB_rr(res.value, r1.value, r2.value)
 
-        self.mc.ADD_ri(res.value, reg.value, arg1.getint())
         regalloc.possibly_free_vars_for_op(op)
         return fcond
 
-    def emit_op_guard_true(self, op, regalloc, fcond):
-        assert fcond == c.GT
+    # Guards
+    def _emit_guard(self, op, regalloc, fcond):
         descr = op.getdescr()
         assert isinstance(descr, BasicFailDescr)
         descr._arm_guard_code = self.mc.curraddr()
         memaddr = self._gen_path_to_exit_path(op, op.getfailargs(), regalloc, fcond)
         descr._failure_recovery_code = memaddr
         descr._arm_guard_cond = fcond
+
+    def emit_op_guard_true(self, op, regalloc, fcond):
+        assert fcond == c.GT
+        self._emit_guard(op, regalloc, fcond)
         return c.AL
 
+    def emit_op_guard_false(self, op, regalloc, fcond):
+        assert fcond == c.EQ
+        self._emit_guard(op, regalloc, fcond)
+        return c.AL
+
+
 def make_operation_list():
     def notimplemented(self, op, regalloc, fcond):
         raise NotImplementedError

Added: pypy/branch/arm-backend/pypy/jit/backend/arm/locations.py
==============================================================================
--- (empty file)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/locations.py	Tue Oct 26 17:12:25 2010
@@ -0,0 +1,7 @@
+class AssemblerLocation(object):
+    pass
+
+class RegisterLocation(AssemblerLocation):
+
+    def __init__(self, value):
+        self.value = value


From afa at codespeak.net  Tue Oct 26 17:57:45 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 17:57:45 +0200 (CEST)
Subject: [pypy-svn] r78307 - pypy/branch/fast-forward/pypy/module/_io
Message-ID: <20101026155745.73CF5282C18@codespeak.net>

Author: afa
Date: Tue Oct 26 17:57:42 2010
New Revision: 78307

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
Log:
Fix translation


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Tue Oct 26 17:57:42 2010
@@ -486,7 +486,7 @@
             written = 0
             while remaining > self.buffer_size:
                 try:
-                    n = self._write(space, data[written:size - written])
+                    n = self._write(space, data[written:])
                 except OperationError, e:
                     if not e.match(space, space.gettypeobject(
                         W_BlockingIOError.typedef)):


From cfbolz at codespeak.net  Tue Oct 26 18:12:17 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Tue, 26 Oct 2010 18:12:17 +0200 (CEST)
Subject: [pypy-svn] r78308 - in pypy/trunk/pypy/objspace/std: . test
Message-ID: <20101026161217.8F001282C18@codespeak.net>

Author: cfbolz
Date: Tue Oct 26 18:12:15 2010
New Revision: 78308

Modified:
   pypy/trunk/pypy/objspace/std/test/test_mapdict.py
   pypy/trunk/pypy/objspace/std/test/test_shadowtracking.py
   pypy/trunk/pypy/objspace/std/test/test_versionedtype.py
   pypy/trunk/pypy/objspace/std/typeobject.py
Log:
(arigo, cfbolz): after some discussion we decided that a custom metaclass
doesn't interfere with versiontag.


Modified: pypy/trunk/pypy/objspace/std/test/test_mapdict.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_mapdict.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_mapdict.py	Tue Oct 26 18:12:15 2010
@@ -753,6 +753,25 @@
             return a.x
         #
         res = self.check(f, 'x')
+        assert res == (1, 0, 0)
+        res = self.check(f, 'x')
+        assert res == (0, 1, 0)
+        res = self.check(f, 'x')
+        assert res == (0, 1, 0)
+
+    def test_old_style_base(self):
+        class B:
+            pass
+        class C(object):
+            pass
+        class A(C, B):
+            pass
+        a = A()
+        a.x = 42
+        def f():
+            return a.x
+        #
+        res = self.check(f, 'x')
         assert res == (0, 0, 1)
         res = self.check(f, 'x')
         assert res == (0, 0, 1)

Modified: pypy/trunk/pypy/objspace/std/test/test_shadowtracking.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_shadowtracking.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_shadowtracking.py	Tue Oct 26 18:12:15 2010
@@ -140,10 +140,11 @@
 
     def test_class_that_cannot_be_cached(self):
         import __pypy__
-        class metatype(type):
+        class X:
             pass
-        class A(object):
-            __metaclass__ = metatype
+        class Y(object):
+            pass
+        class A(Y, X):
             def f(self):
                 return 42
 
@@ -241,3 +242,21 @@
             foo = 3
         D.__bases__ = (C, F)
         assert e.foo == 3
+
+    def test_custom_metaclass(self):
+        import __pypy__
+        class MetaA(type):
+            def __getattribute__(self, x):
+                return 1
+        def f(self):
+            return 42
+        A = type.__new__(MetaA, "A", (), {"f": f})
+        l = [type.__getattribute__(A, "__new__")(A)] * 10
+        __pypy__.reset_method_cache_counter()
+        for i, a in enumerate(l):
+            assert a.f() == 42
+        cache_counter = __pypy__.method_cache_counter("f")
+        assert cache_counter[0] >= 5
+        assert cache_counter[1] >= 1 # should be (27, 3)
+        assert sum(cache_counter) == 10
+

Modified: pypy/trunk/pypy/objspace/std/test/test_versionedtype.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_versionedtype.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_versionedtype.py	Tue Oct 26 18:12:15 2010
@@ -12,10 +12,12 @@
                 def f(self): pass
             class B(A):
                 pass
-            class metatype(type):
+            class X:
+                pass
+            class Y(object):
+                pass
+            class C(Y, X):
                 pass
-            class C(object):
-                __metaclass__ = metatype
             return A, B, C
         """)
         return space.unpackiterable(w_types)

Modified: pypy/trunk/pypy/objspace/std/typeobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/typeobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/typeobject.py	Tue Oct 26 18:12:15 2010
@@ -104,14 +104,12 @@
 
         if overridetypedef is not None:
             setup_builtin_type(w_self)
-            custom_metaclass = False
         else:
             setup_user_defined_type(w_self)
-            custom_metaclass = not space.is_w(space.type(w_self), space.w_type)
         w_self.w_same_layout_as = get_parent_layout(w_self)
 
         if space.config.objspace.std.withtypeversion:
-            if custom_metaclass or not is_mro_purely_of_types(w_self.mro_w):
+            if not is_mro_purely_of_types(w_self.mro_w):
                 pass
             else:
                 # the _version_tag should change, whenever the content of


From cfbolz at codespeak.net  Tue Oct 26 18:13:35 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Tue, 26 Oct 2010 18:13:35 +0200 (CEST)
Subject: [pypy-svn] r78309 - pypy/trunk/pypy/objspace/std
Message-ID: <20101026161335.223CB282C18@codespeak.net>

Author: cfbolz
Date: Tue Oct 26 18:13:33 2010
New Revision: 78309

Modified:
   pypy/trunk/pypy/objspace/std/typeobject.py
Log:
(arigo, cfbolz): essential optimization: don't intern the slot name if the slot
will be discarded.


Modified: pypy/trunk/pypy/objspace/std/typeobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/typeobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/typeobject.py	Tue Oct 26 18:13:33 2010
@@ -557,9 +557,9 @@
                              space.wrap('__slots__ must be identifiers'))
     # create member
     slot_name = _mangle(slot_name, w_self.name)
-    # Force interning of slot names.
-    slot_name = space.str_w(space.new_interned_str(slot_name))
     if slot_name not in w_self.dict_w:
+        # Force interning of slot names.
+        slot_name = space.str_w(space.new_interned_str(slot_name))
         # in cpython it is ignored less, but we probably don't care
         member = Member(w_self.nslots, slot_name, w_self)
         w_self.dict_w[slot_name] = space.wrap(member)


From arigo at codespeak.net  Tue Oct 26 19:06:52 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Tue, 26 Oct 2010 19:06:52 +0200 (CEST)
Subject: [pypy-svn] r78311 - pypy/branch/bisect-module
Message-ID: <20101026170652.EE57D282C18@codespeak.net>

Author: arigo
Date: Tue Oct 26 19:06:51 2010
New Revision: 78311

Added:
   pypy/branch/bisect-module/
      - copied from r78310, pypy/trunk/
Log:
Write the _bisect module at interp-level.



From afa at codespeak.net  Tue Oct 26 19:30:11 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 19:30:11 +0200 (CEST)
Subject: [pypy-svn] r78313 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101026173011.B8F81282C18@codespeak.net>

Author: afa
Date: Tue Oct 26 19:30:09 2010
New Revision: 78313

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Log:
Fix properties of incompletely initialized Buffered* objects.
This is especially important to avoid segfaults in __del__...


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Tue Oct 26 19:30:09 2010
@@ -1,6 +1,6 @@
 from __future__ import with_statement
 from pypy.interpreter.typedef import (
-    TypeDef, generic_new_descr)
+    TypeDef, GetSetProperty, generic_new_descr)
 from pypy.interpreter.gateway import interp2app, unwrap_spec
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.error import OperationError, operationerrfmt
@@ -11,12 +11,15 @@
 from pypy.module._io.interp_io import DEFAULT_BUFFER_SIZE, W_BlockingIOError
 from pypy.module.thread.os_lock import Lock
 
+STATE_ZERO, STATE_OK, STATE_DETACHED = range(3)
+
 class BlockingIOError(Exception):
     pass
 
 class W_BufferedIOBase(W_IOBase):
     def __init__(self, space):
         W_IOBase.__init__(self, space)
+        self.state = STATE_ZERO
 
         self.buffer = lltype.nullptr(rffi.CCHARP.TO)
         self.pos = 0        # Current logical position in the buffer
@@ -84,6 +87,18 @@
         except OperationError:
             pass
 
+    def _check_init(self, space):
+        if self.state == STATE_ZERO:
+            raise OperationError(space.w_ValueError, space.wrap(
+                "I/O operation on uninitialized object"))
+        elif self.state == STATE_DETACHED:
+            raise OperationError(space.w_ValueError, space.wrap(
+                "raw stream has been detached"))
+
+    def _check_closed(self, space, message=None):
+        self._check_init(space)
+        W_IOBase._check_closed(self, space, message)
+
     def _raw_tell(self, space):
         w_pos = space.call_method(self.raw, "tell")
         pos = space.r_longlong_w(w_pos)
@@ -94,6 +109,10 @@
         self.abs_pos = pos
         return pos
 
+    def closed_get_w(space, self):
+        self._check_init(space)
+        return space.getattr(self.raw, space.wrap("closed"))
+
     def _readahead(self):
         if self.readable and self.read_end != -1:
             return self.read_end - self.pos
@@ -155,6 +174,7 @@
 
     @unwrap_spec('self', ObjSpace)
     def close_w(self, space):
+        self._check_init(space)
         with self.lock:
             if self._closed(space):
                 return
@@ -164,6 +184,7 @@
 
     @unwrap_spec('self', ObjSpace)
     def flush_w(self, space):
+        self._check_init(space)
         return space.call_method(self.raw, "flush")
 
     def _writer_flush_unlocked(self, space, restore_pos=False):
@@ -223,7 +244,7 @@
             l.append(self.buffer[i])
         return self._write(space, ''.join(l))
 
-class W_BufferedReader(W_BufferedIOBase, BufferedMixin):
+class W_BufferedReader(BufferedMixin, W_BufferedIOBase):
     def __init__(self, space):
         W_BufferedIOBase.__init__(self, space)
         self.ok = False
@@ -231,6 +252,7 @@
 
     @unwrap_spec('self', ObjSpace, W_Root, int)
     def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE):
+        self.state = STATE_ZERO
         raw = space.interp_w(W_IOBase, w_raw)
         raw.check_readable_w(space)
 
@@ -240,6 +262,7 @@
 
         self._init(space)
         self._reader_reset_buf()
+        self.state = STATE_OK
 
     @unwrap_spec('self', ObjSpace, W_Root)
     def read_w(self, space, w_size=None):
@@ -397,11 +420,13 @@
     seek = interp2app(W_BufferedReader.seek_w),
     close = interp2app(W_BufferedReader.close_w),
     flush = interp2app(W_BufferedReader.flush_w),
+    closed = GetSetProperty(W_BufferedReader.closed_get_w),
     )
 
-class W_BufferedWriter(W_BufferedIOBase, BufferedMixin):
+class W_BufferedWriter(BufferedMixin, W_BufferedIOBase):
     @unwrap_spec('self', ObjSpace, W_Root, int)
     def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE):
+        self.state = STATE_ZERO
         raw = space.interp_w(W_IOBase, w_raw)
         raw.check_writable_w(space)
 
@@ -411,6 +436,7 @@
 
         self._init(space)
         self._writer_reset_buf()
+        self.state = STATE_OK
 
     def _adjust_position(self, new_pos):
         self.pos = new_pos
@@ -545,6 +571,7 @@
     # from the mixin class
     seek = interp2app(W_BufferedWriter.seek_w),
     close = interp2app(W_BufferedWriter.close_w),
+    closed = GetSetProperty(W_BufferedWriter.closed_get_w),
     )
 
 class W_BufferedRWPair(W_BufferedIOBase):

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Tue Oct 26 19:30:09 2010
@@ -1,4 +1,4 @@
-from pypy.conftest import gettestobjspace
+from pypy.conftest import gettestobjspace, option
 from pypy.interpreter.gateway import interp2app
 from pypy.tool.udir import udir
 
@@ -38,9 +38,12 @@
         cls.space = gettestobjspace(usemodules=['_io'])
         tmpfile = udir.join('tmpfile')
         cls.w_tmpfile = cls.space.wrap(str(tmpfile))
-        def readfile(space):
-            return space.wrap(tmpfile.read())
-        cls.w_readfile = cls.space.wrap(interp2app(readfile))
+        if option.runappdirect:
+            cls.w_readfile = tmpfile.read
+        else:
+            def readfile(space):
+                return space.wrap(tmpfile.read())
+            cls.w_readfile = cls.space.wrap(interp2app(readfile))
 
     def test_write(self):
         import _io
@@ -57,3 +60,12 @@
         f.write("abcd" * 5000)
         f.close()
         assert self.readfile() == "abcd" * 5000
+
+    def test_incomplete(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile)
+        b = _io.BufferedWriter.__new__(_io.BufferedWriter)
+        raises(IOError, b.__init__, raw) # because file is not writable
+        raises(ValueError, getattr, b, 'closed')
+        raises(ValueError, b.flush)
+        raises(ValueError, b.close)


From afa at codespeak.net  Tue Oct 26 19:56:02 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 19:56:02 +0200 (CEST)
Subject: [pypy-svn] r78314 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101026175602.E77EF282C18@codespeak.net>

Author: afa
Date: Tue Oct 26 19:56:01 2010
New Revision: 78314

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Log:
Test and fix


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Tue Oct 26 19:56:01 2010
@@ -451,8 +451,8 @@
 
         with self.lock:
 
-            if (not (self.readable and self.read_end == -1) and
-                not (self.writable and self.write_end == -1)):
+            if (not (self.readable and self.read_end != -1) and
+                not (self.writable and self.write_end != -1)):
                 self.pos = 0
                 self.raw_pos = 0
             available = self.buffer_size - self.pos

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Tue Oct 26 19:56:01 2010
@@ -69,3 +69,33 @@
         raises(ValueError, getattr, b, 'closed')
         raises(ValueError, b.flush)
         raises(ValueError, b.close)
+
+    def test_check_several_writes(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile, 'w')
+        b = _io.BufferedWriter(raw, 13)
+
+        for i in range(4):
+            assert b.write('x' * 10) == 10
+        b.flush()
+        assert self.readfile() == 'x' * 40
+
+    def test_destructor(self):
+        import _io
+
+        record = []
+        class MyIO(_io.BufferedWriter):
+            def __del__(self):
+                record.append(1)
+                super(MyIO, self).__del__()
+            def close(self):
+                record.append(2)
+                super(MyIO, self).close()
+            def flush(self):
+                record.append(3)
+                super(MyIO, self).flush()
+        raw = _io.FileIO(self.tmpfile, 'w')
+        MyIO(raw)
+        import gc; gc.collect()
+        assert record == [1, 2, 3]
+


From hakanardo at codespeak.net  Tue Oct 26 19:57:58 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Tue, 26 Oct 2010 19:57:58 +0200 (CEST)
Subject: [pypy-svn] r78315 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101026175758.E42D8282C18@codespeak.net>

Author: hakanardo
Date: Tue Oct 26 19:57:57 2010
New Revision: 78315

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py
Log:
virtuals with circular refs

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	Tue Oct 26 19:57:57 2010
@@ -47,9 +47,14 @@
     def get_key_box(self):
         return self.box
 
-    # FIXME: Reise get_args_for_fail instead?
-    def get_forced_boxes(self):
-        return [self.force_box()]
+    # FIXME: Reuse get_args_for_fail instead?
+    def get_forced_boxes(self, already_seen):
+        key = self.get_key_box()
+        if key in already_seen:
+            return []
+        box = self.force_box()
+        already_seen.append(self.get_key_box())
+        return [box]
 
     def get_args_for_fail(self, modifier):
         pass

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Tue Oct 26 19:57:57 2010
@@ -52,7 +52,7 @@
         
         inputargs = []
         for arg in jump_args:
-            for a in self.getvalue(arg).get_forced_boxes():
+            for a in self.getvalue(arg).get_forced_boxes([]):
                 if not isinstance(a, Const):
                     inputargs.append(a)
 
@@ -75,7 +75,7 @@
                 args = []
                 #for arg in newop.getarglist():
                 for arg in [argmap[a] for a in inputargs]:
-                    args.extend(self.getvalue(arg).get_forced_boxes())
+                    args.extend(self.getvalue(arg).get_forced_boxes([]))
                 newop.initarglist(args + inputargs[len(args):])
 
             #print 'P: ', str(newop)
@@ -104,7 +104,7 @@
                             jmp = self.optimizer.newoperations[-1]
                             if jmp.getopnum() == rop.JUMP:
                                 newval = self.getvalue(argmap[a])
-                                newarg = newval.get_forced_boxes()
+                                newarg = newval.get_forced_boxes([])
                                 jmp.initarglist(jmp.getarglist() + newarg)
 
         return inputargs

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Tue Oct 26 19:57:57 2010
@@ -121,13 +121,17 @@
                 fieldvalue = self._fields[ofs]
                 fieldvalue.get_args_for_fail(modifier)
 
-    # FIXME: circular references
-    def get_forced_boxes(self):
+    def get_forced_boxes(self, already_seen):
+        key = self.get_key_box()
+        if key in already_seen:
+            return []
+        already_seen.append(key)
         if self.box is None:
             lst = self._get_field_descr_list()
             fieldboxes = []
             for ofs in lst:
-                fieldboxes.extend(self._fields[ofs].get_forced_boxes())
+                boxes = self._fields[ofs].get_forced_boxes(already_seen)
+                fieldboxes.extend(boxes)
             return fieldboxes
         else:
             return [self.box]
@@ -204,11 +208,15 @@
     def _make_virtual(self, modifier):
         return modifier.make_varray(self.arraydescr)
 
-    def get_forced_boxes(self):
+    def get_forced_boxes(self, already_seen):
+        key = self.get_key_box()
+        if key in already_seen:
+            return []
+        already_seen.append(key)
         if self.box is None:
             boxes = []
             for itemvalue in self._items:
-                boxes.extend(itemvalue.get_forced_boxes())
+                boxes.extend(itemvalue.get_forced_boxes(already_seen))
             return boxes
         else:
             return [self.box]

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py	Tue Oct 26 19:57:57 2010
@@ -34,6 +34,32 @@
         self.check_loops(new=0, new_with_vtable=0,
                                 getfield_gc=0, setfield_gc=0)
 
+    def test_virtualized_circular1(self):
+        class MyNode():
+            pass
+        myjitdriver = JitDriver(greens = [], reds = ['n', 'node'])
+        def f(n):
+            node = MyNode()
+            node.value = 0
+            node.extra = 0
+            node.ref = node
+            while n > 0:
+                myjitdriver.can_enter_jit(n=n, node=node)
+                myjitdriver.jit_merge_point(n=n, node=node)
+                next = MyNode()
+                next.value = node.ref.value + n
+                next.extra = node.ref.extra + 1
+                next.ref = next
+                node = next
+                n -= 1
+            return node.value * node.extra
+        assert f(10) == 55 * 10
+        res = self.meta_interp(f, [10])
+        assert res == 55 * 10
+        self.check_loop_count(1)
+        self.check_loops(new=0, new_with_vtable=0,
+                                getfield_gc=0, setfield_gc=0)
+
     def test_virtualized_float(self):
         myjitdriver = JitDriver(greens = [], reds = ['n', 'node'])
         def f(n):


From afa at codespeak.net  Tue Oct 26 20:47:02 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 20:47:02 +0200 (CEST)
Subject: [pypy-svn] r78316 - in pypy/branch/fast-forward/pypy: config
	interpreter objspace/std objspace/std/test objspace/test
	tool/release tool/release/test
Message-ID: <20101026184702.640E8282C18@codespeak.net>

Author: afa
Date: Tue Oct 26 20:46:59 2010
New Revision: 78316

Modified:
   pypy/branch/fast-forward/pypy/config/pypyoption.py
   pypy/branch/fast-forward/pypy/interpreter/baseobjspace.py
   pypy/branch/fast-forward/pypy/interpreter/function.py
   pypy/branch/fast-forward/pypy/objspace/std/dictmultiobject.py
   pypy/branch/fast-forward/pypy/objspace/std/dicttype.py
   pypy/branch/fast-forward/pypy/objspace/std/frozensettype.py
   pypy/branch/fast-forward/pypy/objspace/std/setobject.py
   pypy/branch/fast-forward/pypy/objspace/std/settype.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_dictmultiobject.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_mapdict.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_setobject.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_shadowtracking.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_versionedtype.py
   pypy/branch/fast-forward/pypy/objspace/std/typeobject.py
   pypy/branch/fast-forward/pypy/objspace/test/test_descroperation.py
   pypy/branch/fast-forward/pypy/tool/release/   (props changed)
   pypy/branch/fast-forward/pypy/tool/release/__init__.py   (props changed)
   pypy/branch/fast-forward/pypy/tool/release/force-builds.py   (props changed)
   pypy/branch/fast-forward/pypy/tool/release/make_release.py   (props changed)
   pypy/branch/fast-forward/pypy/tool/release/test/   (props changed)
   pypy/branch/fast-forward/pypy/tool/release/test/__init__.py   (props changed)
   pypy/branch/fast-forward/pypy/tool/release/test/test_make_release.py   (props changed)
Log:
Merge from trunk: -r78251:78315


Modified: pypy/branch/fast-forward/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/fast-forward/pypy/config/pypyoption.py	(original)
+++ pypy/branch/fast-forward/pypy/config/pypyoption.py	Tue Oct 26 20:46:59 2010
@@ -76,6 +76,7 @@
     # no _rawffi if importing pypy.rlib.clibffi raises ImportError
     # or CompilationError
     "_rawffi"   : ["pypy.rlib.clibffi"],
+    "_ffi"      : ["pypy.rlib.clibffi"],
 
     "zlib"      : ["pypy.rlib.rzlib"],
     "bz2"       : ["pypy.module.bz2.interp_bz2"],

Modified: pypy/branch/fast-forward/pypy/interpreter/baseobjspace.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/baseobjspace.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/baseobjspace.py	Tue Oct 26 20:46:59 2010
@@ -739,7 +739,7 @@
 
     def unpackiterable(self, w_iterable, expected_length=-1):
         """Unpack an iterable object into a real (interpreter-level) list.
-        Raise a real (subclass of) ValueError if the length is wrong."""
+        Raise an OperationError(w_ValueError) if the length is wrong."""
         w_iterator = self.iter(w_iterable)
         items = []
         while True:

Modified: pypy/branch/fast-forward/pypy/interpreter/function.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/function.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/function.py	Tue Oct 26 20:46:59 2010
@@ -589,12 +589,14 @@
             w_klass = space.type(w_obj)
         return space.wrap(Method(space, self.w_function, w_klass, space.w_None))
 
-    def descr_classmethod__new__(space, w_type, w_function):
+    def descr_classmethod__new__(space, w_subtype, w_function):
         if not space.is_true(space.callable(w_function)):
             typename = space.type(w_function).getname(space, '?')
             raise operationerrfmt(space.w_TypeError,
                                   "'%s' object is not callable", typename)
-        return space.wrap(ClassMethod(w_function))
+        instance = space.allocate_instance(ClassMethod, w_subtype)
+        instance.__init__(w_function)
+        return space.wrap(instance)
 
 class FunctionWithFixedCode(Function):
     can_change_code = False

Modified: pypy/branch/fast-forward/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/dictmultiobject.py	Tue Oct 26 20:46:59 2010
@@ -641,30 +641,50 @@
 init_signature = Signature(['seq_or_map'], None, 'kwargs')
 init_defaults = [None]
 
-def init__DictMulti(space, w_dict, __args__):
-    w_src, w_kwds = __args__.parse_obj(
-            None, 'dict',
-            init_signature, # signature
-            init_defaults)                           # default argument
-    if w_src is None:
-        pass
-    elif space.findattr(w_src, space.wrap("keys")) is None:
-        list_of_w_pairs = space.listview(w_src)
-        for w_pair in list_of_w_pairs:
+def update1(space, w_dict, w_data):
+    if space.findattr(w_data, space.wrap("keys")) is None:
+        # no 'keys' method, so we assume it is a sequence of pairs
+        for w_pair in space.listview(w_data):
             pair = space.fixedview(w_pair)
-            if len(pair)!=2:
+            if len(pair) != 2:
                 raise OperationError(space.w_ValueError,
-                             space.wrap("dict() takes a sequence of pairs"))
-            w_k, w_v = pair
-            w_dict.setitem(w_k, w_v)
+                             space.wrap("sequence of pairs expected"))
+            w_key, w_value = pair
+            w_dict.setitem(w_key, w_value)
     else:
-        if space.is_true(w_src):
-            from pypy.objspace.std.dicttype import update1
-            update1(space, w_dict, w_src)
+        if isinstance(w_data, W_DictMultiObject):    # optimization case only
+            update1_dict_dict(space, w_dict, w_data)
+        else:
+            # general case -- "for k in o.keys(): dict.__setitem__(d, k, o[k])"
+            w_keys = space.call_method(w_data, "keys")
+            for w_key in space.listview(w_keys):
+                w_value = space.getitem(w_data, w_key)
+                w_dict.setitem(w_key, w_value)
+
+def update1_dict_dict(space, w_dict, w_data):
+    iterator = w_data.iter()
+    while 1:
+        w_key, w_value = iterator.next()
+        if w_key is None:
+            break
+        w_dict.setitem(w_key, w_value)
+
+def init_or_update(space, w_dict, __args__, funcname):
+    w_src, w_kwds = __args__.parse_obj(
+            None, funcname,
+            init_signature, # signature
+            init_defaults)  # default argument
+    if w_src is not None:
+        update1(space, w_dict, w_src)
     if space.is_true(w_kwds):
-        from pypy.objspace.std.dicttype import update1
         update1(space, w_dict, w_kwds)
 
+def init__DictMulti(space, w_dict, __args__):
+    init_or_update(space, w_dict, __args__, 'dict')
+
+def dict_update__DictMulti(space, w_dict, __args__):
+    init_or_update(space, w_dict, __args__, 'dict.update')
+
 def getitem__DictMulti_ANY(space, w_dict, w_key):
     w_value = w_dict.getitem(w_key)
     if w_value is not None:
@@ -758,9 +778,8 @@
     return w_res
 
 def dict_copy__DictMulti(space, w_self):
-    from pypy.objspace.std.dicttype import update1
     w_new = W_DictMultiObject.allocate_and_init_instance(space)
-    update1(space, w_new, w_self)
+    update1_dict_dict(space, w_new, w_self)
     return w_new
 
 def dict_items__DictMulti(space, w_self):
@@ -791,6 +810,15 @@
     else:
         return w_default
 
+def dict_setdefault__DictMulti_ANY_ANY(space, w_dict, w_key, w_default):
+    # XXX should be more efficient, with only one dict lookup
+    w_value = w_dict.getitem(w_key)
+    if w_value is not None:
+        return w_value
+    else:
+        w_dict.setitem(w_key, w_default)
+        return w_default
+
 def dict_pop__DictMulti_ANY(space, w_dict, w_key, w_defaults):
     defaults = space.listview(w_defaults)
     len_defaults = len(defaults)
@@ -808,6 +836,19 @@
         w_dict.delitem(w_key)
         return w_item
 
+def dict_popitem__DictMulti(space, w_dict):
+    # XXX should somehow use the same trick as CPython: saving the index
+    # of the last popped item in the hash table, so that the next call to
+    # popitem() can be more efficient, instead of always starting from the
+    # beginning of the hash table.
+    iterator = w_dict.iter()
+    w_key, w_value = iterator.next()
+    if w_key is None:
+        raise OperationError(space.w_KeyError,
+                             space.wrap("popitem(): dictionary is empty"))
+    w_dict.delitem(w_key)
+    return space.newtuple([w_key, w_value])
+
 app = gateway.applevel('''
     def dictrepr(currently_in_repr, d):
         # Now we only handle one implementation of dicts, this one.

Modified: pypy/branch/fast-forward/pypy/objspace/std/dicttype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/dicttype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/dicttype.py	Tue Oct 26 20:46:59 2010
@@ -53,106 +53,21 @@
 # This can return when multimethods have been fixed
 #dict_str        = StdObjSpace.str
 
-# default application-level implementations for some operations
-# most of these (notably not popitem and update*) are overwritten
-# in dictmultiobject
-# gateway is imported in the stdtypedef module
-app = gateway.applevel('''
-
-    # in the following functions we use dict.__setitem__ instead of
-    # d[k]=...  because when a subclass of dict override __setitem__,
-    # CPython does not call it when doing builtin operations.  The
-    # same for other operations.
-
-    def update1(d, o):
-        if hasattr(o, 'keys'):
-            for k in o.keys():
-                dict.__setitem__(d, k, o[k])
-        else:
-            for k,v in o:
-                dict.__setitem__(d, k, v)
-
-    def update(d, *args, **kwargs):
-        len_args = len(args)
-        if len_args == 1:
-            update1(d, args[0])
-        elif len_args > 1:
-            raise TypeError("update takes at most 1 (non-keyword) argument")
-        if kwargs:
-            update1(d, kwargs)
-
-    def popitem(d):
-        for k in dict.iterkeys(d):
-            break
-        else:
-            raise KeyError("popitem(): dictionary is empty")
-        v = dict.__getitem__(d, k)
-        dict.__delitem__(d, k)
-        return k, v
-
-    def get(d, k, v=None):
-        if k in d:
-            return dict.__getitem__(d, k)
-        else:
-            return v
-
-    def setdefault(d, k, v=None):
-        if k in d:
-            return dict.__getitem__(d, k)
-        else:
-            dict.__setitem__(d, k, v)
-            return v
-
-    def pop(d, k, defaults):     # XXX defaults is actually *defaults
-        if len(defaults) > 1:
-            raise TypeError, "pop expected at most 2 arguments, got %d" % (
-                1 + len(defaults))
-        try:
-            v = dict.__getitem__(d, k)
-            dict.__delitem__(d, k)
-        except KeyError, e:
-            if defaults:
-                return defaults[0]
-            else:
-                raise e
-        return v
-
-    def iteritems(d):
-        return iter(dict.items(d))
-
-    def iterkeys(d):
-        return iter(dict.keys(d))
-
-    def itervalues(d):
-        return iter(dict.values(d))
-''', filename=__file__)
-
-dict_update__ANY             = app.interphook("update")
-dict_popitem__ANY            = app.interphook("popitem")
-dict_get__ANY_ANY_ANY        = app.interphook("get")
-dict_setdefault__ANY_ANY_ANY = app.interphook("setdefault")
-dict_pop__ANY_ANY            = app.interphook("pop")
-dict_iteritems__ANY          = app.interphook("iteritems")
-dict_iterkeys__ANY           = app.interphook("iterkeys")
-dict_itervalues__ANY         = app.interphook("itervalues")
-update1                      = app.interphook("update1")
-
 register_all(vars(), globals())
 
 @gateway.unwrap_spec(ObjSpace, W_Root, W_Root, W_Root)
 def descr_fromkeys(space, w_type, w_keys, w_fill=None):
+    from pypy.objspace.std.dictmultiobject import W_DictMultiObject
     if w_fill is None:
         w_fill = space.w_None
-    w_dict = space.call_function(w_type)
-    w_iter = space.iter(w_keys)
-    while True:
-        try:
-            w_key = space.next(w_iter)
-        except OperationError, e:
-            if not e.match(space, space.w_StopIteration):
-                raise
-            break
-        space.setitem(w_dict, w_key, w_fill)
+    if w_type is space.w_dict:
+        w_dict = W_DictMultiObject.allocate_and_init_instance(space, w_type)
+        for w_key in space.listview(w_keys):
+            w_dict.setitem(w_key, w_fill)
+    else:
+        w_dict = space.call_function(w_type)
+        for w_key in space.listview(w_keys):
+            space.setitem(w_dict, w_key, w_fill)
     return w_dict
 
 

Modified: pypy/branch/fast-forward/pypy/objspace/std/frozensettype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/frozensettype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/frozensettype.py	Tue Oct 26 20:46:59 2010
@@ -39,13 +39,13 @@
 def descr__frozenset__new__(space, w_frozensettype,
                             w_iterable=gateway.NoneNotWrapped):
     from pypy.objspace.std.setobject import W_FrozensetObject
-    from pypy.objspace.std.setobject import _is_frozenset_exact
+    from pypy.objspace.std.setobject import make_setdata_from_w_iterable
     if (space.is_w(w_frozensettype, space.w_frozenset) and
-        _is_frozenset_exact(w_iterable)):
+        w_iterable is not None and type(w_iterable) is W_FrozensetObject):
         return w_iterable
     w_obj = space.allocate_instance(W_FrozensetObject, w_frozensettype)
-    W_FrozensetObject.__init__(w_obj, space, None)
-
+    data = make_setdata_from_w_iterable(space, w_iterable)
+    W_FrozensetObject.__init__(w_obj, space, data)
     return w_obj
 
 frozenset_typedef = StdTypeDef("frozenset",

Modified: pypy/branch/fast-forward/pypy/objspace/std/setobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/setobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/setobject.py	Tue Oct 26 20:46:59 2010
@@ -21,11 +21,10 @@
         return False
 
 
-    def __init__(w_self, space, setdata=None):
-        if setdata is None:
-            w_self.setdata = r_dict(space.eq_w, space.hash_w)
-        else:
-            w_self.setdata = setdata.copy()
+    def __init__(w_self, space, setdata):
+        """Initialize the set by taking ownership of 'setdata'."""
+        assert setdata is not None
+        w_self.setdata = setdata
 
     def __repr__(w_self):
         """representation for debugging purposes"""
@@ -33,6 +32,7 @@
         return "<%s(%s)>" % (w_self.__class__.__name__, ', '.join(reprlist))
 
     def _newobj(w_self, space, rdict_w=None):
+        """Make a new set or frozenset by taking ownership of 'rdict_w'."""
         #return space.call(space.type(w_self),W_SetIterObject(rdict_w))
         objtype = type(w_self)
         if objtype is W_SetObject:
@@ -55,10 +55,7 @@
 
 class W_FrozensetObject(W_BaseSetObject):
     from pypy.objspace.std.frozensettype import frozenset_typedef as typedef
-
-    def __init__(w_self, space, setdata):
-        W_BaseSetObject.__init__(w_self, space, setdata)
-        w_self.hash = -1
+    hash = 0
 
 registerimplementation(W_BaseSetObject)
 registerimplementation(W_SetObject)
@@ -109,8 +106,14 @@
 
 # some helper functions
 
+def newset(space):
+    return r_dict(space.eq_w, space.hash_w)
+
 def make_setdata_from_w_iterable(space, w_iterable=None):
-    data = r_dict(space.eq_w, space.hash_w)
+    """Return a new r_dict with the content of w_iterable."""
+    if isinstance(w_iterable, W_BaseSetObject):
+        return w_iterable.setdata.copy()
+    data = newset(space)
     if w_iterable is not None:
         for w_item in space.listview(w_iterable):
             data[w_item] = None
@@ -119,7 +122,7 @@
 def _initialize_set(space, w_obj, w_iterable=None):
     w_obj.setdata.clear()
     if w_iterable is not None:
-        w_obj.setdata.update(make_setdata_from_w_iterable(space, w_iterable))
+        w_obj.setdata = make_setdata_from_w_iterable(space, w_iterable)
 
 def _convert_set_to_frozenset(space, w_obj):
     if space.is_true(space.isinstance(w_obj, space.w_set)):
@@ -130,12 +133,6 @@
 
 # helper functions for set operation on dicts
 
-def _is_frozenset_exact(w_obj):
-    if (w_obj is not None) and (type(w_obj) is W_FrozensetObject):
-        return True
-    else:
-        return False
-
 def _is_eq(ld, rd):
     if len(ld) != len(rd):
         return False
@@ -144,66 +141,41 @@
             return False
     return True
 
-def _union_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    ld.update(rdict)
-    return ld, rdict
-
-def _difference_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    del_list_w = []
+def _difference_dict(space, ld, rd):
+    result = newset(space)
     for w_key in ld:
-        if w_key in rdict:
-            del_list_w.append(w_key)
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    return ld, rdict
-
-def _intersection_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    del_list_w = []
-    for w_key in ld:
-        if w_key not in rdict:
-            del_list_w.append(w_key)
-
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    return ld, rdict
-
+        if w_key not in rd:
+            result[w_key] = None
+    return result
 
-def _symmetric_difference_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
+def _difference_dict_update(space, ld, rd):
+    if ld is rd:
+        ld.clear()     # for the case 'a.difference_update(a)'
     else:
-        ld = ldict.copy()
-    del_list_w = []
-    add_list_w = []
+        for w_key in rd:
+            try:
+                del ld[w_key]
+            except KeyError:
+                pass
+
+def _intersection_dict(space, ld, rd):
+    result = newset(space)
+    if len(ld) > len(rd):
+        ld, rd = rd, ld     # loop over the smaller dict
     for w_key in ld:
-        if w_key in rdict:
-            del_list_w.append(w_key)
+        if w_key in rd:
+            result[w_key] = None
+    return result
 
-    for w_key in rdict:
+def _symmetric_difference_dict(space, ld, rd):
+    result = newset(space)
+    for w_key in ld:
+        if w_key not in rd:
+            result[w_key] = None
+    for w_key in rd:
         if w_key not in ld:
-            add_list_w.append(w_key)
-
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    for w_key in add_list_w:
-        ld[w_key] = None
-
-    return ld, rdict
+            result[w_key] = None
+    return result
 
 def _issubset_dict(ldict, rdict):
     if len(ldict) > len(rdict):
@@ -220,14 +192,13 @@
 def set_update__Set_BaseSet(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _union_dict(ld, rd, True)
-    return space.w_None
+    ld.update(rd)
 
 def set_update__Set_ANY(space, w_left, w_other):
     """Update a set with the union of itself and another."""
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _union_dict(ld, rd, True)
-    return space.w_None
+    ld = w_left.setdata
+    for w_item in space.listview(w_other):
+        ld[w_item] = None
 
 def inplace_or__Set_Set(space, w_left, w_other):
     set_update__Set_BaseSet(space, w_left, w_other)
@@ -241,25 +212,23 @@
     This has no effect if the element is already present.
     """
     w_left.setdata[w_other] = None
-    return space.w_None
 
 def set_copy__Set(space, w_set):
-    return w_set._newobj(space,w_set.setdata)
+    return w_set._newobj(space, w_set.setdata.copy())
 
 def frozenset_copy__Frozenset(space, w_left):
-    if _is_frozenset_exact(w_left):
+    if type(w_left) is W_FrozensetObject:
         return w_left
     else:
-        return set_copy__Set(space,w_left)
+        return set_copy__Set(space, w_left)
 
 def set_clear__Set(space, w_left):
     w_left.setdata.clear()
-    return space.w_None
 
 def set_difference__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _difference_dict(ld, rd, False)
+    new_ld = _difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 set_difference__Set_Frozenset = set_difference__Set_Set
@@ -272,7 +241,7 @@
 
 def set_difference__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _difference_dict(ld, rd, False)
+    new_ld = _difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 frozenset_difference__Frozenset_ANY = set_difference__Set_ANY
@@ -281,15 +250,17 @@
 def set_difference_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _difference_dict(ld, rd, True)
-    return space.w_None
+    _difference_dict_update(space, ld, rd)
 
 set_difference_update__Set_Frozenset = set_difference_update__Set_Set
 
 def set_difference_update__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _difference_dict(ld, rd, True)
-    return space.w_None
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        try:
+            del ld[w_key]
+        except KeyError:
+            pass
 
 def inplace_sub__Set_Set(space, w_left, w_other):
     set_difference_update__Set_Set(space, w_left, w_other)
@@ -467,7 +438,7 @@
 
 def hash__Frozenset(space, w_set):
     multi = r_uint(1822399083) + r_uint(1822399083) + 1
-    if w_set.hash != -1:
+    if w_set.hash != 0:
         return space.wrap(w_set.hash)
     hash = 1927868237
     hash *= (len(w_set.setdata) + 1)
@@ -476,7 +447,7 @@
         value = ((h ^ (h << 16) ^ 89869747)  * multi)
         hash = intmask(hash ^ value)
     hash = hash * 69069 + 907133923
-    if hash == -1:
+    if hash == 0:
         hash = 590923713
     hash = intmask(hash)
     w_set.hash = hash
@@ -484,29 +455,31 @@
     return space.wrap(hash)
 
 def set_pop__Set(space, w_left):
-    if len(w_left.setdata) == 0:
+    for w_key in w_left.setdata:
+        break
+    else:
         raise OperationError(space.w_KeyError,
                                 space.wrap('pop from an empty set'))
-    w_keys = w_left.setdata.keys()
-    w_value = w_keys[0]
-    del w_left.setdata[w_value]
-
-    return w_value
+    del w_left.setdata[w_key]
+    return w_key
 
 def set_intersection__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _intersection_dict(ld, rd, False)
-    return w_left._newobj(space,new_ld)
+    new_ld = _intersection_dict(space, ld, rd)
+    return w_left._newobj(space, new_ld)
 
 set_intersection__Set_Frozenset = set_intersection__Set_Set
 set_intersection__Frozenset_Frozenset = set_intersection__Set_Set
 set_intersection__Frozenset_Set = set_intersection__Set_Set
 
 def set_intersection__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _intersection_dict(ld, rd, False)
-    return w_left._newobj(space,new_ld)
+    result = newset(space)
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        if w_key in ld:
+            result[w_key] = None
+    return w_left._newobj(space, result)
 
 frozenset_intersection__Frozenset_ANY = set_intersection__Set_ANY
 
@@ -518,15 +491,18 @@
 def set_intersection_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _intersection_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _intersection_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 set_intersection_update__Set_Frozenset = set_intersection_update__Set_Set
 
 def set_intersection_update__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _intersection_dict(ld, rd, True)
-    return space.w_None
+    result = newset(space)
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        if w_key in ld:
+            result[w_key] = None
+    w_left.setdata = result
 
 def inplace_and__Set_Set(space, w_left, w_other):
     set_intersection_update__Set_Set(space, w_left, w_other)
@@ -537,7 +513,7 @@
 def set_symmetric_difference__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _symmetric_difference_dict(ld, rd, False)
+    new_ld = _symmetric_difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 set_symmetric_difference__Set_Frozenset = set_symmetric_difference__Set_Set
@@ -553,7 +529,7 @@
 
 def set_symmetric_difference__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _symmetric_difference_dict(ld, rd, False)
+    new_ld = _symmetric_difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 frozenset_symmetric_difference__Frozenset_ANY = \
@@ -562,16 +538,16 @@
 def set_symmetric_difference_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _symmetric_difference_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _symmetric_difference_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 set_symmetric_difference_update__Set_Frozenset = \
                                     set_symmetric_difference_update__Set_Set
 
 def set_symmetric_difference_update__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _symmetric_difference_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _symmetric_difference_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 def inplace_xor__Set_Set(space, w_left, w_other):
     set_symmetric_difference_update__Set_Set(space, w_left, w_other)
@@ -582,8 +558,9 @@
 def set_union__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _union_dict(ld, rd, False)
-    return w_left._newobj(space, new_ld)
+    result = ld.copy()
+    result.update(rd)
+    return w_left._newobj(space, result)
 
 set_union__Set_Frozenset = set_union__Set_Set
 set_union__Frozenset_Set = set_union__Set_Set
@@ -595,9 +572,11 @@
 
 
 def set_union__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _union_dict(ld, rd, False)
-    return w_left._newobj(space, new_ld)
+    ld = w_left.setdata
+    result = ld.copy()
+    for w_key in space.listview(w_other):
+        result[w_key] = None
+    return w_left._newobj(space, result)
 
 frozenset_union__Frozenset_ANY = set_union__Set_ANY
 
@@ -629,15 +608,6 @@
             init_defaults)
     _initialize_set(space, w_set, w_iterable)
 
-def init__Frozenset(space, w_set, __args__):
-    w_iterable, = __args__.parse_obj(
-            None, 'set',
-            init_signature,
-            init_defaults)
-    if w_set.hash == -1:
-        _initialize_set(space, w_set, w_iterable)
-        hash__Frozenset(space, w_set)
-
 app = gateway.applevel("""
     def setrepr(currently_in_repr, s):
         'The app-level part of repr().'

Modified: pypy/branch/fast-forward/pypy/objspace/std/settype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/settype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/settype.py	Tue Oct 26 20:46:59 2010
@@ -66,9 +66,9 @@
 register_all(vars(), globals())
 
 def descr__new__(space, w_settype, __args__):
-    from pypy.objspace.std.setobject import W_SetObject
+    from pypy.objspace.std.setobject import W_SetObject, newset
     w_obj = space.allocate_instance(W_SetObject, w_settype)
-    W_SetObject.__init__(w_obj, space, None)
+    W_SetObject.__init__(w_obj, space, newset(space))
     return w_obj
 
 set_typedef = StdTypeDef("set",

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_dictmultiobject.py	Tue Oct 26 20:46:59 2010
@@ -238,6 +238,7 @@
         it1 = d.popitem()
         assert len(d) == 0
         assert (it!=it1) and (it1==(1,2) or it1==(3,4))
+        raises(KeyError, d.popitem)
     
     def test_setdefault(self):
         d = {1:2, 3:4}
@@ -446,7 +447,9 @@
         d1 = {}
         d2 = D(a='foo')
         d1.update(d2)
-        assert d1['a'] == 42 # fails on CPython, d1['a'] == 'foo'
+        assert d1['a'] == 'foo'
+        # a bit of an obscure case: now (from r78295) we get the same result
+        # as CPython does
 
     def test_index_keyerror_unpacking(self):
         d = {}

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_mapdict.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_mapdict.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_mapdict.py	Tue Oct 26 20:46:59 2010
@@ -753,6 +753,25 @@
             return a.x
         #
         res = self.check(f, 'x')
+        assert res == (1, 0, 0)
+        res = self.check(f, 'x')
+        assert res == (0, 1, 0)
+        res = self.check(f, 'x')
+        assert res == (0, 1, 0)
+
+    def test_old_style_base(self):
+        class B:
+            pass
+        class C(object):
+            pass
+        class A(C, B):
+            pass
+        a = A()
+        a.x = 42
+        def f():
+            return a.x
+        #
+        res = self.check(f, 'x')
         assert res == (0, 0, 1)
         res = self.check(f, 'x')
         assert res == (0, 0, 1)

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_setobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_setobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_setobject.py	Tue Oct 26 20:46:59 2010
@@ -10,6 +10,7 @@
 import py.test
 from pypy.objspace.std.setobject import W_SetObject, W_FrozensetObject
 from pypy.objspace.std.setobject import _initialize_set
+from pypy.objspace.std.setobject import newset, make_setdata_from_w_iterable
 from pypy.objspace.std.setobject import set_intersection__Set_Set
 from pypy.objspace.std.setobject import set_intersection__Set_ANY
 from pypy.objspace.std.setobject import eq__Set_Set
@@ -28,12 +29,12 @@
         self.false = self.space.w_False
 
     def test_and(self):
-        s = W_SetObject(self.space, None)
+        s = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, s, self.word)
-        t0 = W_SetObject(self.space, None)
+        t0 = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, t0, self.otherword)
-        t1 = W_FrozensetObject(self.space, None)
-        _initialize_set(self.space, t1, self.otherword)
+        t1 = W_FrozensetObject(self.space,
+                make_setdata_from_w_iterable(self.space, self.otherword))
         r0 = set_intersection__Set_Set(self.space, s, t0)
         r1 = set_intersection__Set_Set(self.space, s, t1)
         assert eq__Set_Set(self.space, r0, r1) == self.true
@@ -41,9 +42,9 @@
         assert eq__Set_Set(self.space, r0, sr) == self.true
 
     def test_compare(self):
-        s = W_SetObject(self.space, None)
+        s = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, s, self.word)
-        t = W_SetObject(self.space, None)
+        t = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, t, self.word)
         assert self.space.eq_w(s,t)
         u = self.space.wrap(set('simsalabim'))
@@ -56,6 +57,11 @@
         b = a | set('abc')
         assert type(b) is subset
 
+    def test_union(self):
+        a = set([4, 5])
+        b = a.union([5, 7])
+        assert sorted(b) == [4, 5, 7]
+
     def test_compare(self):
         raises(TypeError, cmp, set('abc'), set('abd'))
         assert set('abc') != 'abc'

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_shadowtracking.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_shadowtracking.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_shadowtracking.py	Tue Oct 26 20:46:59 2010
@@ -140,10 +140,11 @@
 
     def test_class_that_cannot_be_cached(self):
         import __pypy__
-        class metatype(type):
+        class X:
             pass
-        class A(object):
-            __metaclass__ = metatype
+        class Y(object):
+            pass
+        class A(Y, X):
             def f(self):
                 return 42
 
@@ -241,3 +242,21 @@
             foo = 3
         D.__bases__ = (C, F)
         assert e.foo == 3
+
+    def test_custom_metaclass(self):
+        import __pypy__
+        class MetaA(type):
+            def __getattribute__(self, x):
+                return 1
+        def f(self):
+            return 42
+        A = type.__new__(MetaA, "A", (), {"f": f})
+        l = [type.__getattribute__(A, "__new__")(A)] * 10
+        __pypy__.reset_method_cache_counter()
+        for i, a in enumerate(l):
+            assert a.f() == 42
+        cache_counter = __pypy__.method_cache_counter("f")
+        assert cache_counter[0] >= 5
+        assert cache_counter[1] >= 1 # should be (27, 3)
+        assert sum(cache_counter) == 10
+

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_typeobject.py	Tue Oct 26 20:46:59 2010
@@ -1109,4 +1109,12 @@
 
         assert b == 1
 
-        
+    def test_slots_with_method_in_class(self):
+        # this works in cpython...
+        class A(object):
+            __slots__ = ["f"]
+            def f(self, x):
+                return x + 1
+        a = A()
+        assert a.f(1) == 2
+

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_versionedtype.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_versionedtype.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_versionedtype.py	Tue Oct 26 20:46:59 2010
@@ -12,10 +12,12 @@
                 def f(self): pass
             class B(A):
                 pass
-            class metatype(type):
+            class X:
+                pass
+            class Y(object):
+                pass
+            class C(Y, X):
                 pass
-            class C(object):
-                __metaclass__ = metatype
             return A, B, C
         """)
         return space.unpackiterable(w_types)

Modified: pypy/branch/fast-forward/pypy/objspace/std/typeobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/typeobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/typeobject.py	Tue Oct 26 20:46:59 2010
@@ -106,14 +106,12 @@
 
         if overridetypedef is not None:
             setup_builtin_type(w_self)
-            custom_metaclass = False
         else:
             setup_user_defined_type(w_self)
-            custom_metaclass = not space.is_w(space.type(w_self), space.w_type)
         w_self.w_same_layout_as = get_parent_layout(w_self)
 
         if space.config.objspace.std.withtypeversion:
-            if custom_metaclass or not is_mro_purely_of_types(w_self.mro_w):
+            if not is_mro_purely_of_types(w_self.mro_w):
                 pass
             else:
                 # the _version_tag should change, whenever the content of
@@ -581,11 +579,13 @@
                              space.wrap('__slots__ must be identifiers'))
     # create member
     slot_name = _mangle(slot_name, w_self.name)
-    # Force interning of slot names.
-    slot_name = space.str_w(space.new_interned_str(slot_name))
-    member = Member(w_self.nslots, slot_name, w_self)
-    w_self.dict_w[slot_name] = space.wrap(member)
-    w_self.nslots += 1
+    if slot_name not in w_self.dict_w:
+        # Force interning of slot names.
+        slot_name = space.str_w(space.new_interned_str(slot_name))
+        # in cpython it is ignored less, but we probably don't care
+        member = Member(w_self.nslots, slot_name, w_self)
+        w_self.dict_w[slot_name] = space.wrap(member)
+        w_self.nslots += 1
 
 def create_dict_slot(w_self):
     if not w_self.hasdict:

Modified: pypy/branch/fast-forward/pypy/objspace/test/test_descroperation.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/test/test_descroperation.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/test/test_descroperation.py	Tue Oct 26 20:46:59 2010
@@ -416,7 +416,7 @@
         class D(object):
             def __init__(self, a):
                 self.a = a
-        
+
         assert A(1) == B(1)
         assert B(1) == A(1)
         assert A(1) == C(1)
@@ -479,6 +479,20 @@
                 return self
         raises(TypeError, iter, x())
 
+    def test_attribute_error(self):
+        class classmethodonly(classmethod):
+            def __get__(self, instance, type):
+                if instance is not None:
+                    raise AttributeError("Must be called on a class, not an instance.")
+                return super(classmethodonly, self).__get__(instance, type)
+
+        class A(object):
+            @classmethodonly
+            def a(cls):
+                return 3
+
+        raises(AttributeError, lambda: A().a)
+
     def test_isinstance_and_issubclass(self):
         class Meta(type):
             def __instancecheck__(cls, instance):


From afa at codespeak.net  Tue Oct 26 21:36:36 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 21:36:36 +0200 (CEST)
Subject: [pypy-svn] r78317 -
	pypy/branch/fast-forward/lib-python/modified-2.7.0/test
Message-ID: <20101026193636.96711282C18@codespeak.net>

Author: afa
Date: Tue Oct 26 21:36:34 2010
New Revision: 78317

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_compile.py
      - copied, changed from r78315, pypy/branch/fast-forward/lib-python/2.7.0/test/test_compile.py
Log:
Skip an implementation specific test


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_compile.py (from r78315, pypy/branch/fast-forward/lib-python/2.7.0/test/test_compile.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_compile.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_compile.py	Tue Oct 26 21:36:34 2010
@@ -3,6 +3,7 @@
 import _ast
 from test import test_support
 import textwrap
+from test.test_support import check_impl_detail
 
 class TestSpecifics(unittest.TestCase):
 
@@ -90,12 +91,13 @@
         self.assertEqual(m.results, ('z', g))
         exec 'z = locals()' in g, m
         self.assertEqual(m.results, ('z', m))
-        try:
-            exec 'z = b' in m
-        except TypeError:
-            pass
-        else:
-            self.fail('Did not validate globals as a real dict')
+        if check_impl_detail():
+            try:
+                exec 'z = b' in m
+            except TypeError:
+                pass
+            else:
+                self.fail('Did not validate globals as a real dict')
 
         class A:
             "Non-mapping"


From hakanardo at codespeak.net  Tue Oct 26 21:58:22 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Tue, 26 Oct 2010 21:58:22 +0200 (CEST)
Subject: [pypy-svn] r78319 -
	pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test
Message-ID: <20101026195822.BA144282C18@codespeak.net>

Author: hakanardo
Date: Tue Oct 26 21:58:21 2010
New Revision: 78319

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_jump.py
Log:
failing test

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_jump.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_jump.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_jump.py	Tue Oct 26 21:58:21 2010
@@ -142,3 +142,22 @@
                              ('push', s12),
                              ('mov', ebx, s12),
                              ('pop', ebx)]
+def test_crash():
+    assembler = MockAssembler()
+    dstvalues = [-56, -64, -72, -80, -88, 15, 14, 15, 13, 12, 15, -96, 
+                 -104, -112, 10, 9, 8, 7, 6, 3, -120, 2, 1]
+    srcvalues = [-56, -64, -72, -80, -88, -224, -216, -224, -248, -232,
+                 -224, -96, -104, -112, -184, -192, -264, 2, -312, -344,
+                 10, 8, 14]
+    print len(dstvalues), len(srcvalues)
+    class MyLocation(AssemblerLocation):
+        def __init__(self, value):
+            self.value = value
+        def location_code(self):
+            return 'b'
+        
+    src = [MyLocation(i) for i in srcvalues]
+    dst = [MyLocation(i) for i in dstvalues]
+        
+    remap_frame_layout(assembler, src, dst, '?')
+    assert assembler.ops == []


From afa at codespeak.net  Tue Oct 26 22:09:49 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 22:09:49 +0200 (CEST)
Subject: [pypy-svn] r78320 - in
	pypy/branch/fast-forward/pypy/interpreter/pyparser: . test
Message-ID: <20101026200949.C694E282C18@codespeak.net>

Author: afa
Date: Tue Oct 26 22:09:48 2010
New Revision: 78320

Modified:
   pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py
   pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py
   pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py
Log:
Allow \r as a newline;
also tokens that go to the end of the line
should stop on either \r or \n.

This also fixes a cpython test in test_compile.py


Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/genpytokenize.py	Tue Oct 26 22:09:48 2010
@@ -17,12 +17,17 @@
 def makePyPseudoDFA ():
     import string
     states = []
+    def makeEOL():
+        return group(states,
+                     newArcPair(states, "\n"),
+                     chain(states,
+                           newArcPair(states, "\r"),
+                           maybe(states, newArcPair(states, "\n"))))
     # ____________________________________________________________
     def makeLineCont ():
         return chain(states,
                      newArcPair(states, "\\"),
-                     maybe(states, newArcPair(states, "\r")),
-                     newArcPair(states, "\n"))
+                     makeEOL())
     # ____________________________________________________________
     # Ignore stuff
     def makeWhitespace ():
@@ -130,9 +135,7 @@
                      newArcPair(states, "~"))
     bracket = groupStr(states, "[](){}")
     special = group(states,
-                    chain(states,
-                          maybe(states, newArcPair(states, "\r")),
-                          newArcPair(states, "\n")),
+                    makeEOL(),
                     groupStr(states, "@:;.,`"))
     funny = group(states, operator, bracket, special)
     # ____________________________________________________________
@@ -146,13 +149,13 @@
                           makeStrPrefix(),
                           newArcPair(states, "'"),
                           any(states,
-                              notGroupStr(states, "\n'\\")),
+                              notGroupStr(states, "\r\n'\\")),
                           any(states,
                               chain(states,
                                     newArcPair(states, "\\"),
                                     newArcPair(states, DEFAULT),
                                     any(states,
-                                        notGroupStr(states, "\n'\\")))),
+                                        notGroupStr(states, "\r\n'\\")))),
                           group(states,
                                 newArcPair(states, "'"),
                                 makeLineCont())),
@@ -160,13 +163,13 @@
                           makeStrPrefix(),
                           newArcPair(states, '"'),
                           any(states,
-                              notGroupStr(states, '\n"\\')),
+                              notGroupStr(states, '\r\n"\\')),
                           any(states,
                               chain(states,
                                     newArcPair(states, "\\"),
                                     newArcPair(states, DEFAULT),
                                     any(states,
-                                        notGroupStr(states, '\n"\\')))),
+                                        notGroupStr(states, '\r\n"\\')))),
                           group(states,
                                 newArcPair(states, '"'),
                                 makeLineCont())))

Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenize.py	Tue Oct 26 22:09:48 2010
@@ -24,7 +24,7 @@
 # Automatically generated DFA's
 
 accepts = [True, True, True, True, True, True, True, True,
-           True, True, False, True, True, True, False, False,
+           True, True, False, True, True, True, True, False,
            False, False, True, True, True, True, True, False,
            True, False, True, False, True, False, False,
            True, False, False, False, False, True, False,
@@ -143,9 +143,11 @@
     # 14
     {'\n': 13},
     # 15
-    {automata.DEFAULT: 30, '\n': 27, "'": 28, '\\': 29},
+    {automata.DEFAULT: 30, '\n': 27,
+     '\r': 27, "'": 28, '\\': 29},
     # 16
-    {automata.DEFAULT: 33, '\n': 27, '"': 31, '\\': 32},
+    {automata.DEFAULT: 33, '\n': 27,
+     '\r': 27, '"': 31, '\\': 32},
     # 17
     {'\n': 13, '\r': 14},
     # 18
@@ -195,13 +197,15 @@
     # 29
     {automata.DEFAULT: 37, '\n': 13, '\r': 14},
     # 30
-    {automata.DEFAULT: 30, '\n': 27, "'": 13, '\\': 29},
+    {automata.DEFAULT: 30, '\n': 27,
+     '\r': 27, "'": 13, '\\': 29},
     # 31
     {'"': 13},
     # 32
     {automata.DEFAULT: 38, '\n': 13, '\r': 14},
     # 33
-    {automata.DEFAULT: 33, '\n': 27, '"': 13, '\\': 32},
+    {automata.DEFAULT: 33, '\n': 27,
+     '\r': 27, '"': 13, '\\': 32},
     # 34
     {'+': 39, '-': 39, '0': 40, '1': 40,
      '2': 40, '3': 40, '4': 40, '5': 40,
@@ -215,9 +219,11 @@
      '4': 36, '5': 36, '6': 36, '7': 36,
      '8': 36, '9': 36, 'J': 13, 'j': 13},
     # 37
-    {automata.DEFAULT: 37, '\n': 27, "'": 13, '\\': 29},
+    {automata.DEFAULT: 37, '\n': 27,
+     '\r': 27, "'": 13, '\\': 29},
     # 38
-    {automata.DEFAULT: 38, '\n': 27, '"': 13, '\\': 32},
+    {automata.DEFAULT: 38, '\n': 27,
+     '\r': 27, '"': 13, '\\': 32},
     # 39
     {'0': 40, '1': 40, '2': 40, '3': 40,
      '4': 40, '5': 40, '6': 40, '7': 40,

Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py	Tue Oct 26 22:09:48 2010
@@ -96,6 +96,9 @@
         exc = py.test.raises(IndentationError, parse, input).value
         assert exc.msg == "unindent does not match any outer indentation level"
 
+    def test_mac_newline(self):
+        self.parse("this_is\ra_mac\rfile")
+
     def test_mode(self):
         assert self.parse("x = 43*54").type == syms.file_input
         tree = self.parse("43**54", "eval")


From afa at codespeak.net  Tue Oct 26 22:42:32 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 22:42:32 +0200 (CEST)
Subject: [pypy-svn] r78321 - in
	pypy/branch/fast-forward/pypy/interpreter/astcompiler: . test
Message-ID: <20101026204232.A98125080C@codespeak.net>

Author: afa
Date: Tue Oct 26 22:42:30 2010
New Revision: 78321

Modified:
   pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py
   pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py
Log:
"import None as x" is not a SyntaxError
(None is not an assignment target here)


Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/astbuilder.py	Tue Oct 26 22:42:30 2010
@@ -196,7 +196,8 @@
                 if len(import_name.children) == 1:
                     import_name = import_name.children[0]
                     continue
-                alias = self.alias_for_import_name(import_name.children[0])
+                alias = self.alias_for_import_name(import_name.children[0],
+                                                   store=False)
                 asname_node = import_name.children[2]
                 alias.asname = asname_node.value
                 self.check_forbidden_name(alias.asname, asname_node)

Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/test/test_astbuilder.py	Tue Oct 26 22:42:30 2010
@@ -795,6 +795,10 @@
                 input = template % (name,)
                 exc = py.test.raises(SyntaxError, self.get_ast, input).value
                 assert exc.msg == "cannot assign to %s" % (name,)
+        # This is ok.
+        self.get_ast("from None import x")
+        self.get_ast("from x import None as y")
+        self.get_ast("import None as x")
 
     def test_lambda(self):
         lam = self.get_first_expr("lambda x: expr")


From afa at codespeak.net  Tue Oct 26 23:03:07 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 23:03:07 +0200 (CEST)
Subject: [pypy-svn] r78323 - in pypy/branch/fast-forward/pypy: interpreter
	module/__builtin__/test
Message-ID: <20101026210307.07552282C18@codespeak.net>

Author: afa
Date: Tue Oct 26 23:03:05 2010
New Revision: 78323

Modified:
   pypy/branch/fast-forward/pypy/interpreter/pycompiler.py
   pypy/branch/fast-forward/pypy/module/__builtin__/test/test_builtin.py
Log:
Check node type when compiling an AST tree


Modified: pypy/branch/fast-forward/pypy/interpreter/pycompiler.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pycompiler.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pycompiler.py	Tue Oct 26 23:03:05 2010
@@ -6,7 +6,7 @@
 from pypy.interpreter import pycode
 from pypy.interpreter.pyparser import future, pyparse, error as parseerror
 from pypy.interpreter.astcompiler import (astbuilder, codegen, consts, misc,
-                                          optimize)
+                                          optimize, ast)
 from pypy.interpreter.error import OperationError
 
 
@@ -107,6 +107,18 @@
         self.compiler_flags = self.future_flags.allowed_flags
 
     def compile_ast(self, node, filename, mode, flags):
+        if mode == 'eval':
+            check = isinstance(node, ast.Expression)
+        elif mode == 'exec':
+            check = isinstance(node, ast.Module)
+        elif mode == 'input':
+            check = isinstance(node, ast.Interactive)
+        else:
+            check = True
+        if not check:
+            raise OperationError(self.space.w_TypeError, self.space.wrap(
+                "invalid node type"))
+
         future_pos = misc.parse_future(node)
         info = pyparse.CompileInfo(filename, mode, flags, future_pos)
         return self._compile_ast(node, info)

Modified: pypy/branch/fast-forward/pypy/module/__builtin__/test/test_builtin.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/__builtin__/test/test_builtin.py	(original)
+++ pypy/branch/fast-forward/pypy/module/__builtin__/test/test_builtin.py	Tue Oct 26 23:03:05 2010
@@ -453,7 +453,15 @@
     def test_unicode_encoding_compile(self):
         code = u"# -*- coding: utf-8 -*-\npass\n"
         raises(SyntaxError, compile, code, "tmp", "exec")
-            
+
+    def test_recompile_ast(self):
+        import _ast
+        # raise exception when node type doesn't match with compile mode
+        co1 = compile('print 1', '', 'exec', _ast.PyCF_ONLY_AST)
+        raises(TypeError, compile, co1, '', 'eval')
+        co2 = compile('1+1', '', 'eval', _ast.PyCF_ONLY_AST)
+        compile(co2, '', 'eval')
+
     def test_isinstance(self):
         assert isinstance(5, int)
         assert isinstance(5, object)


From afa at codespeak.net  Tue Oct 26 23:43:58 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 23:43:58 +0200 (CEST)
Subject: [pypy-svn] r78326 - pypy/branch/fast-forward/lib_pypy
Message-ID: <20101026214358.464BF5080C@codespeak.net>

Author: afa
Date: Tue Oct 26 23:43:56 2010
New Revision: 78326

Modified:
   pypy/branch/fast-forward/lib_pypy/datetime.py
Log:
Fix tests in test_datetime.py


Modified: pypy/branch/fast-forward/lib_pypy/datetime.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/datetime.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/datetime.py	Tue Oct 26 23:43:56 2010
@@ -217,6 +217,11 @@
                                 # strftime is going to have at this: escape %
                                 Zreplace = s.replace('%', '%%')
                     newformat.append(Zreplace)
+                elif ch == 'f':
+                    if isinstance(object, (time, datetime)):
+                        newformat.append('%06d' % object.microsecond)
+                    else:
+                        newformat.append('000000')
                 else:
                     push('%')
                     push(ch)
@@ -575,6 +580,10 @@
     microseconds = property(lambda self: self.__microseconds,
                             doc="microseconds")
 
+    def total_seconds(self):
+        return ((self.days * 86400 + self.seconds) * 10**6
+                + self.microseconds) / 1e6
+
     def __add__(self, other):
         if isinstance(other, timedelta):
             # for CPython compatibility, we cannot use
@@ -803,6 +812,14 @@
 
     __str__ = isoformat
 
+    def __format__(self, format):
+        if not isinstance(format, (str, unicode)):
+            raise ValueError("__format__ excepts str or unicode, not %s" %
+                             format.__class__.__name__)
+        if not format:
+            return str(self)
+        return self.strftime(format)
+
     # Read-only field accessors
     year = property(lambda self: self.__year,
                     doc="year (%d-%d)" % (MINYEAR, MAXYEAR))
@@ -1238,6 +1255,14 @@
 
     __str__ = isoformat
 
+    def __format__(self, format):
+        if not isinstance(format, (str, unicode)):
+            raise ValueError("__format__ excepts str or unicode, not %s" %
+                             format.__class__.__name__)
+        if not format:
+            return str(self)
+        return self.strftime(format)
+
     def strftime(self, fmt):
         """Format using strftime().  The date part of the timestamp passed
         to underlying strftime should not be used.
@@ -1587,7 +1612,11 @@
     @classmethod
     def strptime(cls, date_string, format):
         'string, format -> new datetime parsed from a string (like time.strptime()).'
-        return cls(*_time.strptime(date_string, format)[0:6])
+        # _strptime._strptime returns a two-element tuple.  The first
+        # element is a time.struct_time object.  The second is the
+        # microseconds (which are not defined for time.struct_time).
+        struct, micros = _time.strptime(date_string, format)
+        return cls(*(struct[0:6] + (micros,)))
 
     def utcoffset(self):
         """Return the timezone offset in minutes east of UTC (negative west of


From afa at codespeak.net  Tue Oct 26 23:48:13 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Tue, 26 Oct 2010 23:48:13 +0200 (CEST)
Subject: [pypy-svn] r78327 -
	pypy/branch/fast-forward/pypy/interpreter/pyparser
Message-ID: <20101026214813.0ED79282C18@codespeak.net>

Author: afa
Date: Tue Oct 26 23:48:12 2010
New Revision: 78327

Modified:
   pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenizer.py
Log:
Use CPython error messages


Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenizer.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenizer.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/pytokenizer.py	Tue Oct 26 23:48:12 2010
@@ -92,8 +92,9 @@
 
         if contstr:
             if not line:
-                raise TokenError("EOF while scanning triple-quoted string",
-                                 line, lnum-1, 0, token_list)
+                raise TokenError(
+                    "EOF while scanning triple-quoted string literal",
+                    line, lnum-1, 0, token_list)
             endmatch = endDFA.recognize(line)
             if endmatch >= 0:
                 pos = end = endmatch
@@ -229,7 +230,7 @@
                 if start < 0:
                     start = pos
                 if start

Author: afa
Date: Wed Oct 27 00:58:52 2010
New Revision: 78330

Modified:
   pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py
   pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py
Log:
Implement zipimporter._get_filename()


Modified: pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py	(original)
+++ pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py	Wed Oct 27 00:58:52 2010
@@ -303,6 +303,16 @@
             "Cannot find source for %s in %s", filename, self.name)
     get_source.unwrap_spec = ['self', ObjSpace, str]
 
+    def get_filename(self, space, fullname):
+        filename = self.mangle(fullname)
+        for _, is_package, ext in ENUMERATE_EXTS:
+            if self.have_modulefile(space, filename + ext):
+                return space.wrap(self.filename + os.path.sep +
+                                  self.corr_zname(filename + ext))
+        raise operationerrfmt(self.w_ZipImportError,
+            "Cannot find module %s in %s", filename, self.name)
+    get_filename.unwrap_spec = ['self', ObjSpace, str]
+
     def is_package(self, space, fullname):
         filename = self.mangle(fullname)
         for _, is_package, ext in ENUMERATE_EXTS:
@@ -373,6 +383,7 @@
     get_data    = interp2app(W_ZipImporter.get_data),
     get_code    = interp2app(W_ZipImporter.get_code),
     get_source  = interp2app(W_ZipImporter.get_source),
+    _get_filename = interp2app(W_ZipImporter.get_filename),
     is_package  = interp2app(W_ZipImporter.is_package),
     load_module = interp2app(W_ZipImporter.load_module),
     archive     = GetSetProperty(W_ZipImporter.getarchive),

Modified: pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py	(original)
+++ pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py	Wed Oct 27 00:58:52 2010
@@ -269,6 +269,8 @@
         assert z.get_code('xx')
         assert z.get_source('xx') == "5"
         assert z.archive == self.zipfile
+        mod = z.load_module('xx')
+        assert z._get_filename('xx') == mod.__file__
 
     def test_archive(self):
         """


From hakanardo at codespeak.net  Wed Oct 27 08:17:49 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Wed, 27 Oct 2010 08:17:49 +0200 (CEST)
Subject: [pypy-svn] r78336 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101027061749.01CA5282C18@codespeak.net>

Author: hakanardo
Date: Wed Oct 27 08:17:48 2010
New Revision: 78336

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py
Log:
Dont duplicate inputargs

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Wed Oct 27 08:17:48 2010
@@ -51,8 +51,9 @@
         self.snapshot_map ={None: None}
         
         inputargs = []
+        seen = []
         for arg in jump_args:
-            for a in self.getvalue(arg).get_forced_boxes([]):
+            for a in self.getvalue(arg).get_forced_boxes(seen):
                 if not isinstance(a, Const):
                     inputargs.append(a)
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py	Wed Oct 27 08:17:48 2010
@@ -34,6 +34,28 @@
         self.check_loops(new=0, new_with_vtable=0,
                                 getfield_gc=0, setfield_gc=0)
 
+    def test_virtualized2(self):
+        myjitdriver = JitDriver(greens = [], reds = ['n', 'node1', 'node2'])
+        def f(n):
+            node1 = self._new()
+            node1.value = 0
+            node2 = self._new()
+            node2.value = 0
+            while n > 0:
+                myjitdriver.can_enter_jit(n=n, node1=node1, node2=node2)
+                myjitdriver.jit_merge_point(n=n, node1=node1, node2=node2)
+                next1 = self._new()
+                next1.value = node1.value + n + node2.value
+                next2 = self._new()
+                next2.value = next1.value
+                node1 = next1
+                node2 = next2
+                n -= 1
+            return node1.value * node2.value
+        assert f(10) == self.meta_interp(f, [10])
+        self.check_loops(new=0, new_with_vtable=0,
+                         getfield_gc=0, setfield_gc=0)
+
     def test_virtualized_circular1(self):
         class MyNode():
             pass


From afa at codespeak.net  Wed Oct 27 08:35:28 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 27 Oct 2010 08:35:28 +0200 (CEST)
Subject: [pypy-svn] r78337 - in
	pypy/branch/fast-forward/pypy/module/zipimport: . test
Message-ID: <20101027063528.77E85282C18@codespeak.net>

Author: afa
Date: Wed Oct 27 08:35:26 2010
New Revision: 78337

Modified:
   pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py
   pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py
Log:
Fix zipimporter in the case of sub-subdirectories


Modified: pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py	(original)
+++ pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py	Wed Oct 27 08:35:26 2010
@@ -204,7 +204,6 @@
     def have_modulefile(self, space, filename):
         if ZIPSEP != os.path.sep:
             filename = filename.replace(os.path.sep, ZIPSEP)
-        w = space.wrap
         try:
             self.dir[filename]
             return True
@@ -212,18 +211,20 @@
             return False
 
     def find_module(self, space, fullname, w_path=None):
-        filename = self.mangle(fullname)
+        filename = self.make_filename(fullname)
         for _, _, ext in ENUMERATE_EXTS:
             if self.have_modulefile(space, filename + ext):
                 return space.wrap(self)
     find_module.unwrap_spec = ['self', ObjSpace, str, W_Root]
 
-    def mangle(self, name):
-        return name.replace('.', ZIPSEP)
+    def make_filename(self, fullname):
+        i = fullname.rfind('.')
+        subname = fullname[i+1:]
+        return self.prefix + subname
 
     def load_module(self, space, fullname):
         w = space.wrap
-        filename = self.mangle(fullname)
+        filename = self.make_filename(fullname)
         last_exc = None
         for compiled, is_package, ext in ENUMERATE_EXTS:
             fname = filename + ext
@@ -237,7 +238,8 @@
                 pass
             else:
                 if is_package:
-                    pkgpath = self.name + os.path.sep + filename
+                    pkgpath = (self.name + os.path.sep +
+                               self.corr_zname(filename))
                 else:
                     pkgpath = None
                 try:
@@ -272,7 +274,7 @@
     get_data.unwrap_spec = ['self', ObjSpace, str]
 
     def get_code(self, space, fullname):
-        filename = self.mangle(fullname)
+        filename = self.make_filename(fullname)
         w = space.wrap
         for compiled, _, ext in ENUMERATE_EXTS:
             if self.have_modulefile(space, filename + ext):
@@ -288,7 +290,7 @@
     get_code.unwrap_spec = ['self', ObjSpace, str]
 
     def get_source(self, space, fullname):
-        filename = self.mangle(fullname)
+        filename = self.make_filename(fullname)
         found = False
         for compiled, _, ext in ENUMERATE_EXTS:
             fname = filename + ext
@@ -304,7 +306,7 @@
     get_source.unwrap_spec = ['self', ObjSpace, str]
 
     def get_filename(self, space, fullname):
-        filename = self.mangle(fullname)
+        filename = self.make_filename(fullname)
         for _, is_package, ext in ENUMERATE_EXTS:
             if self.have_modulefile(space, filename + ext):
                 return space.wrap(self.filename + os.path.sep +
@@ -314,7 +316,7 @@
     get_filename.unwrap_spec = ['self', ObjSpace, str]
 
     def is_package(self, space, fullname):
-        filename = self.mangle(fullname)
+        filename = self.make_filename(fullname)
         for _, is_package, ext in ENUMERATE_EXTS:
             if self.have_modulefile(space, filename + ext):
                 return space.wrap(is_package)

Modified: pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py	(original)
+++ pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py	Wed Oct 27 08:35:26 2010
@@ -292,6 +292,16 @@
         assert archive == self.zipfile
         assert realprefix == prefix
 
+    def test_subdirectory_importer(self):
+        import os
+        import zipimport
+        self.writefile(
+            self, os.sep.join(("directory", "package", "__init__.py")), "")
+        z = zipimport.zipimporter(self.zipfile + "/directory")
+        mod = z.load_module("package")
+        assert z.is_package("package")
+        assert z._get_filename("package") == mod.__file__
+
     def test_zip_directory_cache(self):
         """ Check full dictionary interface
         """


From afa at codespeak.net  Wed Oct 27 09:14:28 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 27 Oct 2010 09:14:28 +0200 (CEST)
Subject: [pypy-svn] r78338 - pypy/branch/fast-forward/pypy/module/zipimport
Message-ID: <20101027071428.C97D0282C18@codespeak.net>

Author: afa
Date: Wed Oct 27 09:14:27 2010
New Revision: 78338

Modified:
   pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py
Log:
Fix translation


Modified: pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py	(original)
+++ pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py	Wed Oct 27 09:14:27 2010
@@ -218,8 +218,9 @@
     find_module.unwrap_spec = ['self', ObjSpace, str, W_Root]
 
     def make_filename(self, fullname):
-        i = fullname.rfind('.')
-        subname = fullname[i+1:]
+        startpos = fullname.rfind('.') + 1 # 0 when not found
+        assert startpos >= 0
+        subname = fullname[start:]
         return self.prefix + subname
 
     def load_module(self, space, fullname):


From afa at codespeak.net  Wed Oct 27 09:50:42 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 27 Oct 2010 09:50:42 +0200 (CEST)
Subject: [pypy-svn] r78339 - pypy/branch/fast-forward/pypy/module/zipimport
Message-ID: <20101027075042.45509282C18@codespeak.net>

Author: afa
Date: Wed Oct 27 09:50:40 2010
New Revision: 78339

Modified:
   pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py
Log:
Stupid typo


Modified: pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py	(original)
+++ pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py	Wed Oct 27 09:50:40 2010
@@ -220,7 +220,7 @@
     def make_filename(self, fullname):
         startpos = fullname.rfind('.') + 1 # 0 when not found
         assert startpos >= 0
-        subname = fullname[start:]
+        subname = fullname[startpos:]
         return self.prefix + subname
 
     def load_module(self, space, fullname):


From david at codespeak.net  Wed Oct 27 10:36:38 2010
From: david at codespeak.net (david at codespeak.net)
Date: Wed, 27 Oct 2010 10:36:38 +0200 (CEST)
Subject: [pypy-svn] r78340 - pypy/extradoc/sprintinfo/ddorf2010
Message-ID: <20101027083638.BD39A282C1A@codespeak.net>

Author: david
Date: Wed Oct 27 10:36:34 2010
New Revision: 78340

Modified:
   pypy/extradoc/sprintinfo/ddorf2010/planning.txt
Log:
(All) planning for today

Modified: pypy/extradoc/sprintinfo/ddorf2010/planning.txt
==============================================================================
--- pypy/extradoc/sprintinfo/ddorf2010/planning.txt	(original)
+++ pypy/extradoc/sprintinfo/ddorf2010/planning.txt	Wed Oct 27 10:36:34 2010
@@ -11,19 +11,22 @@
 
 - get mmap to work on Beagleboard DONE
 - bridges in the ARM backend SORT OF DONE
-- add spilling and location support (David, Armin around)
-- hg conversion WE HAVE A PLAN (Anto, Ronny)
-- memory benchmarks IN PROGRESS (Carl Friedrich, Lukas when he shows up)
+- add spilling and location support IN PROGRESS (David, Armin around)
+- ARM int trace operations (David)
+- hg conversion WE HAVE MORE OF A PLAN (Anto, Ronny)
+- memory benchmarks DONE
+- refine heap analysis CONFUSING (Carl Friedrich, Lukas)
+- (Re)write _bisect module (Armin)
 - buildbot improvements, locks, email on failing tests
 - run pypy-c-nojit in the benchmarks
 - make it possible to run benchmark on branches
 - understand problems of mapdict (Carl Friedrich, Armin)
 - rewrite ctypes to use _ffi
-- can classes with non-type metaclasses use typeversion?
+- can classes with non-type metaclasses use typeversion? DONE
 
 
 scheduled discussions:
 
 - hg migration planning
 - understand h?kan's branch
-- JIT introduction (today after lunch)
+- JIT introduction (still today after lunch)


From arigo at codespeak.net  Wed Oct 27 11:44:30 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 11:44:30 +0200 (CEST)
Subject: [pypy-svn] r78343 -
	pypy/branch/cleanup-dict-impl/pypy/objspace/std/test
Message-ID: <20101027094430.2F324282C1A@codespeak.net>

Author: arigo
Date: Wed Oct 27 11:44:26 2010
New Revision: 78343

Added:
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_shadowtracking.py
      - copied unchanged from r78139, pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_shadowtracking.py
Log:
(cfbolz, arigo)
Restore test_shadowtracking because it contains tests of the
method cache.



From arigo at codespeak.net  Wed Oct 27 11:47:06 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 11:47:06 +0200 (CEST)
Subject: [pypy-svn] r78344 -
	pypy/branch/cleanup-dict-impl/pypy/objspace/std/test
Message-ID: <20101027094706.D8E1C282C1A@codespeak.net>

Author: arigo
Date: Wed Oct 27 11:47:05 2010
New Revision: 78344

Added:
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_methodcache.py
      - copied, changed from r78343, pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_shadowtracking.py
Removed:
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_shadowtracking.py
Log:
(cfbolz, arigo)
Rename test_shadowtracking to test_methodcache, and drop the tests
about shadow tracking.


Copied: pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_methodcache.py (from r78343, pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_shadowtracking.py)
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_shadowtracking.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_methodcache.py	Wed Oct 27 11:47:05 2010
@@ -1,119 +1,8 @@
 from pypy.conftest import gettestobjspace
 from pypy.objspace.std.test.test_typeobject import AppTestTypeObject
 
-class TestShadowTracking(object):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withshadowtracking": True,
-                                       "objspace.std.withmapdict": False})
-
-    def test_simple_shadowing(self):
-        space = self.space
-        w_inst = space.appexec([], """():
-            class A(object):
-                def f(self):
-                    return 42
-            a = A()
-            return a
-        """)
-        assert not w_inst.getdict().shadows_anything()
-        space.appexec([w_inst], """(a):
-            a.g = "foo"
-        """)
-        assert not w_inst.getdict().shadows_anything()
-        space.appexec([w_inst], """(a):
-            a.f = "foo"
-        """)
-        assert w_inst.getdict().shadows_anything()
-
-    def test_shadowing_via__dict__(self):
-        space = self.space
-        w_inst = space.appexec([], """():
-            class A(object):
-                def f(self):
-                    return 42
-            a = A()
-            return a
-        """)
-        assert not w_inst.getdict().shadows_anything()
-        space.appexec([w_inst], """(a):
-            a.__dict__["g"] = "foo"
-        """)
-        assert not w_inst.getdict().shadows_anything()
-        space.appexec([w_inst], """(a):
-            a.__dict__["f"] = "foo"
-        """)
-        assert w_inst.getdict().shadows_anything()
-
-    def test_changing__dict__(self):
-        space = self.space
-        w_inst = space.appexec([], """():
-            class A(object):
-                def f(self):
-                    return 42
-            a = A()
-            return a
-        """)
-        assert not w_inst.getdict().shadows_anything()
-        space.appexec([w_inst], """(a):
-            a.__dict__ = {}
-        """)
-        assert w_inst.getdict().shadows_anything()
-
-    def test_changing__class__(self):
-        space = self.space
-        w_inst = space.appexec([], """():
-            class A(object):
-                def f(self):
-                    return 42
-            a = A()
-            return a
-        """)
-        assert not w_inst.getdict().shadows_anything()
-        space.appexec([w_inst], """(a):
-            class B(object):
-                def g(self):
-                    return 42
-            a.__class__ = B
-        """)
-        assert w_inst.getdict().shadows_anything()
-
-    def test_changing_the_type(self):
-        space = self.space
-        w_inst = space.appexec([], """():
-            class A(object):
-                pass
-            a = A()
-            a.x = 72
-            return a
-        """)
-        assert not w_inst.getdict().shadows_anything()
-        w_x = space.appexec([w_inst], """(a):
-            a.__class__.x = 42
-            return a.x
-        """)
-        assert space.unwrap(w_x) == 72
-        assert w_inst.getdict().shadows_anything()
-
-class AppTestShadowTracking(object):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withshadowtracking": True})
-
-    def test_shadowtracking_does_not_blow_up(self):
-        class A(object):
-            def f(self):
-                return 42
-        a = A()
-        assert a.f() == 42
-        a.f = lambda : 43
-        assert a.f() == 43
-
-class AppTestTypeWithMethodCache(AppTestTypeObject):
-
-    def setup_class(cls):
-        cls.space = gettestobjspace(
-            **{"objspace.std.withmethodcachecounter" : True})
 
-class AppTestMethodCaching(AppTestShadowTracking):
+class AppTestMethodCaching(AppTestTypeObject):
     def setup_class(cls):
         cls.space = gettestobjspace(
             **{"objspace.std.withmethodcachecounter": True})


From afa at codespeak.net  Wed Oct 27 11:54:10 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 27 Oct 2010 11:54:10 +0200 (CEST)
Subject: [pypy-svn] r78345 - pypy/branch/fast-forward/pypy/module/_ast/test
Message-ID: <20101027095410.E391B282C1A@codespeak.net>

Author: afa
Date: Wed Oct 27 11:54:09 2010
New Revision: 78345

Modified:
   pypy/branch/fast-forward/pypy/module/_ast/test/test_ast.py
Log:
Fix test, which also failed with -A.


Modified: pypy/branch/fast-forward/pypy/module/_ast/test/test_ast.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_ast/test/test_ast.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_ast/test/test_ast.py	Wed Oct 27 11:54:09 2010
@@ -43,7 +43,7 @@
         for op in mod.body.ops:
             assert isinstance(op, ast.Lt)
         mod.body.ops[0] = ast.Gt()
-        co = compile(mod, "", "exec")
+        co = compile(mod, "", "eval")
         assert not eval(co)
 
     def test_string(self):


From arigo at codespeak.net  Wed Oct 27 12:58:43 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 12:58:43 +0200 (CEST)
Subject: [pypy-svn] r78346 - pypy/branch/cleanup-dict-impl/pypy/objspace/std
Message-ID: <20101027105843.006B2282C1C@codespeak.net>

Author: arigo
Date: Wed Oct 27 12:58:41 2010
New Revision: 78346

Modified:
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py
Log:
(cfbolz, arigo) ObjectMixin is not used there actually.


Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py	Wed Oct 27 12:58:41 2010
@@ -418,7 +418,7 @@
     rangen = unroll.unrolling_iterable(range(n))
     nmin1 = n - 1
     rangenmin1 = unroll.unrolling_iterable(range(nmin1))
-    class subcls(ObjectMixin, BaseMapdictObject, supercls):
+    class subcls(BaseMapdictObject, supercls):
         def _init_empty(self, map):
             from pypy.rlib.debug import make_sure_not_resized
             for i in rangen:


From arigo at codespeak.net  Wed Oct 27 13:04:13 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 13:04:13 +0200 (CEST)
Subject: [pypy-svn] r78347 - pypy/branch/cleanup-dict-impl/pypy/objspace/std
Message-ID: <20101027110413.1FDBC282C1C@codespeak.net>

Author: arigo
Date: Wed Oct 27 13:04:11 2010
New Revision: 78347

Modified:
   pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py
Log:
(cfbolz, arigo)
Using the JIT, we actually don't really want to have many different
RPython-level classes, because promoting their class gives many
different paths in the assembler code.  So for now we fix the number
at exactly 5.


Modified: pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/objspace/std/mapdict.py	Wed Oct 27 13:04:11 2010
@@ -388,6 +388,8 @@
     assert space.config.objspace.std.withmapdict
     map = w_type.terminator
     classes = memo_get_subclass_of_correct_size(space, cls)
+    if SUBCLASSES_MIN_FIELDS == SUBCLASSES_MAX_FIELDS:
+        return classes[0]
     size = map.size_estimate()
     debug.check_nonneg(size)
     if size < len(classes):
@@ -396,7 +398,8 @@
         return classes[len(classes)-1]
 get_subclass_of_correct_size._annspecialcase_ = "specialize:arg(1)"
 
-NUM_SUBCLASSES = 10 # XXX tweak this number
+SUBCLASSES_MIN_FIELDS = 5 # XXX tweak these numbers
+SUBCLASSES_MAX_FIELDS = 5
 
 def memo_get_subclass_of_correct_size(space, supercls):
     key = space, supercls
@@ -405,9 +408,12 @@
     except KeyError:
         assert not hasattr(supercls, "__del__")
         result = []
-        for i in range(1, NUM_SUBCLASSES+1):
+        for i in range(SUBCLASSES_MIN_FIELDS, SUBCLASSES_MAX_FIELDS+1):
             result.append(_make_subclass_size_n(supercls, i))
-        result.insert(0, result[0])    # for 0, use the same class as for 1
+        for i in range(SUBCLASSES_MIN_FIELDS):
+            result.insert(0, result[0])
+        if SUBCLASSES_MIN_FIELDS == SUBCLASSES_MAX_FIELDS:
+            assert len(set(result)) == 1
         _subclass_cache[key] = result
         return result
 memo_get_subclass_of_correct_size._annspecialcase_ = "specialize:memo"


From david at codespeak.net  Wed Oct 27 13:18:30 2010
From: david at codespeak.net (david at codespeak.net)
Date: Wed, 27 Oct 2010 13:18:30 +0200 (CEST)
Subject: [pypy-svn] r78348 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . test
Message-ID: <20101027111830.7E8F0282C1C@codespeak.net>

Author: david
Date: Wed Oct 27 13:18:28 2010
New Revision: 78348

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
Log:
Extend int_sub and int_add instruction encoding

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Wed Oct 27 13:18:28 2010
@@ -178,6 +178,7 @@
             opnum = op.getopnum()
             fcond = self.operations[opnum](self, op, regalloc, fcond)
         self.gen_func_epilog()
+        print inputargs, operations
         if self._debug_asm:
             self._dump_trace('loop.asm')
         print 'Done assembling'
@@ -249,14 +250,18 @@
     def emit_op_int_add(self, op, regalloc, fcond):
         # assuming only one argument is constant
         res = regalloc.try_allocate_reg(op.result)
-        if isinstance(op.getarg(1), ConstInt):
-            reg = regalloc.try_allocate_reg(op.getarg(0))
-            arg1 = op.getarg(1)
-            self.mc.ADD_ri(res.value, reg.value, arg1.getint())
-        elif isinstance(op.getarg(0), ConstInt):
-            reg = regalloc.try_allocate_reg(op.getarg(1))
-            arg1 = op.getarg(0)
-            self.mc.ADD_ri(res.value, reg.value, arg1.getint())
+        if isinstance(op.getarg(0), ConstInt) or isinstance(op.getarg(1), ConstInt):
+            if isinstance(op.getarg(1), ConstInt):
+                reg = regalloc.try_allocate_reg(op.getarg(0))
+                arg1 = op.getarg(1)
+            elif isinstance(op.getarg(0), ConstInt):
+                reg = regalloc.try_allocate_reg(op.getarg(1))
+                arg1 = op.getarg(0)
+            value = arg1.getint()
+            if value < 0:
+                self.mc.SUB_ri(res.value, reg.value, -1 * value)
+            else:
+                self.mc.ADD_ri(res.value, reg.value, value)
         else:
             r1 = regalloc.try_allocate_reg(op.getarg(0))
             r2 = regalloc.try_allocate_reg(op.getarg(1))
@@ -268,14 +273,23 @@
     def emit_op_int_sub(self, op, regalloc, fcond):
         # assuming only one argument is constant
         res = regalloc.try_allocate_reg(op.result)
-        if isinstance(op.getarg(1), ConstInt):
-            reg = regalloc.try_allocate_reg(op.getarg(0))
-            arg1 = op.getarg(1)
-            self.mc.SUB_ri(res.value, reg.value, arg1.getint())
-        elif isinstance(op.getarg(0), ConstInt):
-            reg = regalloc.try_allocate_reg(op.getarg(1))
-            arg1 = op.getarg(0)
-            self.mc.SUB_ri(res.value, reg.value, arg1.getint())
+        if isinstance(op.getarg(0), ConstInt) or isinstance(op.getarg(1), ConstInt):
+            if isinstance(op.getarg(1), ConstInt):
+                reg = regalloc.try_allocate_reg(op.getarg(0))
+                value = op.getarg(1).getint()
+                if value < 0:
+                    self.mc.ADD_ri(res.value, reg.value, -1 * value)
+                else:
+                    self.mc.SUB_ri(res.value, reg.value, value)
+            elif isinstance(op.getarg(0), ConstInt):
+                reg = regalloc.try_allocate_reg(op.getarg(1))
+                value = op.getarg(0).getint()
+                if value < 0:
+                    self.mc.ADD_ri(res.value, reg.value, -1 * value)
+                    self.mc.MVN_rr(res.value, res.value)
+                else:
+                    # reverse substract ftw
+                    self.mc.RSB_ri(res.value, reg.value, value)
         else:
             r1 = regalloc.try_allocate_reg(op.getarg(0))
             r2 = regalloc.try_allocate_reg(op.getarg(1))

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	Wed Oct 27 13:18:28 2010
@@ -35,6 +35,8 @@
         | (table['op'] & 0x1F) << 20)
     if table['result'] and table['base']:
         def imm_func(self, rd, rn, imm=0, cond=cond.AL, s=0):
+            if imm < 0:
+                raise ValueError
             # XXX check condition on rn
             self.write32(n
                 | cond << 28

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	Wed Oct 27 13:18:28 2010
@@ -16,10 +16,12 @@
 
     def hexdump(self):
         return ''.join(self.buffer)
+
 class ASMTest(object):
     def assert_equal(self, asm):
         assert self.cb.hexdump() == assemble(asm)
 
+
 class TestInstrCodeBuilder(ASMTest):
     def setup_method(self, ffuu_method):
         self.cb = CodeBuilder()
@@ -151,6 +153,7 @@
             func = getattr(self.cb, name)
             func(r.r3.value, r.r7.value, 23)
             self.assert_equal('%s r3, r7, #23' % name[:name.index('_')])
+            py.test.raises(ValueError, 'func(r.r3.value, r.r7.value, -12)')
     elif not table['base']:
         def f(self):
             func = getattr(self.cb, name)


From david at codespeak.net  Wed Oct 27 14:59:27 2010
From: david at codespeak.net (david at codespeak.net)
Date: Wed, 27 Oct 2010 14:59:27 +0200 (CEST)
Subject: [pypy-svn] r78352 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101027125927.12C51282C1C@codespeak.net>

Author: david
Date: Wed Oct 27 14:59:25 2010
New Revision: 78352

Added:
   pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
Log:
move resop implementations to mixins


Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Wed Oct 27 14:59:25 2010
@@ -10,11 +10,14 @@
 from pypy.rlib import rgc
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import lltype, rffi, llmemory
+from pypy.jit.backend.arm.opassembler import (GuardOpAssembler,
+                                                IntOpAsslember,
+                                                OpAssembler)
 # XXX Move to llsupport
 from pypy.jit.backend.x86.support import values_array
 
 
-class AssemblerARM(object):
+class AssemblerARM(GuardOpAssembler, IntOpAsslember, OpAssembler):
 
     def __init__(self, cpu, failargs_limit=1000):
         self.mc = ARMv7Builder()
@@ -215,112 +218,9 @@
         b.MOV_rr(r.pc.value, reg.value, cond=fcond)
 
 
-    # Resoperations
-    def emit_op_jump(self, op, regalloc, fcond):
-        tmp = Box()
-        tmpreg = regalloc.try_allocate_reg(tmp)
-        registers = op.getdescr()._arm_arglocs
-        for i in range(op.numargs()):
-            reg = regalloc.try_allocate_reg(op.getarg(i))
-            inpreg = registers[i]
-            # XXX only if every value is in a register
-            self.mc.MOV_rr(inpreg.value, reg.value)
-        loop_code = op.getdescr()._arm_loop_code
-        self.mc.gen_load_int(tmpreg.value, loop_code)
-        self.mc.MOV_rr(r.pc.value, tmpreg.value)
-        regalloc.possibly_free_var(tmpreg)
-        return fcond
-
-    def emit_op_finish(self, op, regalloc, fcond):
-        self._gen_path_to_exit_path(op, op.getarglist(), regalloc, fcond)
-        return fcond
-
-    def emit_op_int_le(self, op, regalloc, fcond):
-        reg = regalloc.try_allocate_reg(op.getarg(0))
-        assert isinstance(op.getarg(1), ConstInt)
-        self.mc.CMP(reg.value, op.getarg(1).getint())
-        return c.GT
-
-    def emit_op_int_eq(self, op, regalloc, fcond):
-        reg = regalloc.try_allocate_reg(op.getarg(0))
-        assert isinstance(op.getarg(1), ConstInt)
-        self.mc.CMP(reg.value, op.getarg(1).getint())
-        return c.EQ
-
-    def emit_op_int_add(self, op, regalloc, fcond):
-        # assuming only one argument is constant
-        res = regalloc.try_allocate_reg(op.result)
-        if isinstance(op.getarg(0), ConstInt) or isinstance(op.getarg(1), ConstInt):
-            if isinstance(op.getarg(1), ConstInt):
-                reg = regalloc.try_allocate_reg(op.getarg(0))
-                arg1 = op.getarg(1)
-            elif isinstance(op.getarg(0), ConstInt):
-                reg = regalloc.try_allocate_reg(op.getarg(1))
-                arg1 = op.getarg(0)
-            value = arg1.getint()
-            if value < 0:
-                self.mc.SUB_ri(res.value, reg.value, -1 * value)
-            else:
-                self.mc.ADD_ri(res.value, reg.value, value)
-        else:
-            r1 = regalloc.try_allocate_reg(op.getarg(0))
-            r2 = regalloc.try_allocate_reg(op.getarg(1))
-            self.mc.ADD_rr(res.value, r1.value, r2.value)
-
-        regalloc.possibly_free_vars_for_op(op)
-        return fcond
-
-    def emit_op_int_sub(self, op, regalloc, fcond):
-        # assuming only one argument is constant
-        res = regalloc.try_allocate_reg(op.result)
-        if isinstance(op.getarg(0), ConstInt) or isinstance(op.getarg(1), ConstInt):
-            if isinstance(op.getarg(1), ConstInt):
-                reg = regalloc.try_allocate_reg(op.getarg(0))
-                value = op.getarg(1).getint()
-                if value < 0:
-                    self.mc.ADD_ri(res.value, reg.value, -1 * value)
-                else:
-                    self.mc.SUB_ri(res.value, reg.value, value)
-            elif isinstance(op.getarg(0), ConstInt):
-                reg = regalloc.try_allocate_reg(op.getarg(1))
-                value = op.getarg(0).getint()
-                if value < 0:
-                    self.mc.ADD_ri(res.value, reg.value, -1 * value)
-                    self.mc.MVN_rr(res.value, res.value)
-                else:
-                    # reverse substract ftw
-                    self.mc.RSB_ri(res.value, reg.value, value)
-        else:
-            r1 = regalloc.try_allocate_reg(op.getarg(0))
-            r2 = regalloc.try_allocate_reg(op.getarg(1))
-            self.mc.SUB_rr(res.value, r1.value, r2.value)
-
-        regalloc.possibly_free_vars_for_op(op)
-        return fcond
-
-    # Guards
-    def _emit_guard(self, op, regalloc, fcond):
-        descr = op.getdescr()
-        assert isinstance(descr, BasicFailDescr)
-        descr._arm_guard_code = self.mc.curraddr()
-        memaddr = self._gen_path_to_exit_path(op, op.getfailargs(), regalloc, fcond)
-        descr._failure_recovery_code = memaddr
-        descr._arm_guard_cond = fcond
-
-    def emit_op_guard_true(self, op, regalloc, fcond):
-        assert fcond == c.GT
-        self._emit_guard(op, regalloc, fcond)
-        return c.AL
-
-    def emit_op_guard_false(self, op, regalloc, fcond):
-        assert fcond == c.EQ
-        self._emit_guard(op, regalloc, fcond)
-        return c.AL
-
-
 def make_operation_list():
     def notimplemented(self, op, regalloc, fcond):
-        raise NotImplementedError
+        raise NotImplementedError, op
 
     operations = [None] * (rop._LAST+1)
     for key, value in rop.__dict__.items():

Added: pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
==============================================================================
--- (empty file)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	Wed Oct 27 14:59:25 2010
@@ -0,0 +1,126 @@
+from pypy.jit.backend.arm import conditions as c
+from pypy.jit.backend.arm import locations
+from pypy.jit.backend.arm import registers as r
+from pypy.jit.backend.arm.arch import WORD, FUNC_ALIGN
+from pypy.jit.backend.arm.codebuilder import ARMv7Builder, ARMv7InMemoryBuilder
+from pypy.jit.backend.arm.regalloc import ARMRegisterManager
+from pypy.jit.backend.llsupport.regalloc import compute_vars_longevity
+from pypy.jit.metainterp.history import ConstInt, BoxInt, Box, BasicFailDescr
+from pypy.jit.metainterp.resoperation import rop
+from pypy.rlib import rgc
+from pypy.rpython.annlowlevel import llhelper
+from pypy.rpython.lltypesystem import lltype, rffi, llmemory
+
+class IntOpAsslember(object):
+    _mixin_ = True
+
+    # XXX support constants larger than imm
+
+    def emit_op_int_add(self, op, regalloc, fcond):
+        # assuming only one argument is constant
+        res = regalloc.try_allocate_reg(op.result)
+        if isinstance(op.getarg(0), ConstInt) or isinstance(op.getarg(1), ConstInt):
+            if isinstance(op.getarg(1), ConstInt):
+                reg = regalloc.try_allocate_reg(op.getarg(0))
+                arg1 = op.getarg(1)
+            elif isinstance(op.getarg(0), ConstInt):
+                reg = regalloc.try_allocate_reg(op.getarg(1))
+                arg1 = op.getarg(0)
+            value = arg1.getint()
+            if value < 0:
+                self.mc.SUB_ri(res.value, reg.value, -1 * value)
+            else:
+                self.mc.ADD_ri(res.value, reg.value, value)
+        else:
+            r1 = regalloc.try_allocate_reg(op.getarg(0))
+            r2 = regalloc.try_allocate_reg(op.getarg(1))
+            self.mc.ADD_rr(res.value, r1.value, r2.value)
+
+        regalloc.possibly_free_vars_for_op(op)
+        return fcond
+
+    def emit_op_int_sub(self, op, regalloc, fcond):
+        # assuming only one argument is constant
+        res = regalloc.try_allocate_reg(op.result)
+        if isinstance(op.getarg(0), ConstInt) or isinstance(op.getarg(1), ConstInt):
+            if isinstance(op.getarg(1), ConstInt):
+                reg = regalloc.try_allocate_reg(op.getarg(0))
+                value = op.getarg(1).getint()
+                if value < 0:
+                    self.mc.ADD_ri(res.value, reg.value, -1 * value)
+                else:
+                    self.mc.SUB_ri(res.value, reg.value, value)
+            elif isinstance(op.getarg(0), ConstInt):
+                reg = regalloc.try_allocate_reg(op.getarg(1))
+                value = op.getarg(0).getint()
+                if value < 0:
+                    self.mc.ADD_ri(res.value, reg.value, -1 * value)
+                    self.mc.MVN_rr(res.value, res.value)
+                else:
+                    # reverse substract ftw
+                    self.mc.RSB_ri(res.value, reg.value, value)
+        else:
+            r1 = regalloc.try_allocate_reg(op.getarg(0))
+            r2 = regalloc.try_allocate_reg(op.getarg(1))
+            self.mc.SUB_rr(res.value, r1.value, r2.value)
+
+        regalloc.possibly_free_vars_for_op(op)
+        return fcond
+
+    def emit_op_int_mul(self, op, regalloc, fcond):
+        import pdb; pdb.set_trace()
+
+class GuardOpAssembler(object):
+    _mixin_ = True
+
+    def _emit_guard(self, op, regalloc, fcond):
+        descr = op.getdescr()
+        assert isinstance(descr, BasicFailDescr)
+        descr._arm_guard_code = self.mc.curraddr()
+        memaddr = self._gen_path_to_exit_path(op, op.getfailargs(), regalloc, fcond)
+        descr._failure_recovery_code = memaddr
+        descr._arm_guard_cond = fcond
+
+    def emit_op_guard_true(self, op, regalloc, fcond):
+        assert fcond == c.GT
+        self._emit_guard(op, regalloc, fcond)
+        return c.AL
+
+    def emit_op_guard_false(self, op, regalloc, fcond):
+        assert fcond == c.EQ
+        self._emit_guard(op, regalloc, fcond)
+        return c.AL
+
+class OpAssembler(object):
+    _mixin_ = True
+
+    def emit_op_jump(self, op, regalloc, fcond):
+        tmp = Box()
+        tmpreg = regalloc.try_allocate_reg(tmp)
+        registers = op.getdescr()._arm_arglocs
+        for i in range(op.numargs()):
+            reg = regalloc.try_allocate_reg(op.getarg(i))
+            inpreg = registers[i]
+            # XXX only if every value is in a register
+            self.mc.MOV_rr(inpreg.value, reg.value)
+        loop_code = op.getdescr()._arm_loop_code
+        self.mc.gen_load_int(tmpreg.value, loop_code)
+        self.mc.MOV_rr(r.pc.value, tmpreg.value)
+        regalloc.possibly_free_var(tmpreg)
+        return fcond
+
+    def emit_op_finish(self, op, regalloc, fcond):
+        self._gen_path_to_exit_path(op, op.getarglist(), regalloc, fcond)
+        return fcond
+
+    def emit_op_int_le(self, op, regalloc, fcond):
+        reg = regalloc.try_allocate_reg(op.getarg(0))
+        assert isinstance(op.getarg(1), ConstInt)
+        self.mc.CMP(reg.value, op.getarg(1).getint())
+        return c.GT
+
+    def emit_op_int_eq(self, op, regalloc, fcond):
+        reg = regalloc.try_allocate_reg(op.getarg(0))
+        assert isinstance(op.getarg(1), ConstInt)
+        self.mc.CMP(reg.value, op.getarg(1).getint())
+        return c.EQ


From arigo at codespeak.net  Wed Oct 27 15:15:11 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 15:15:11 +0200 (CEST)
Subject: [pypy-svn] r78353 - in pypy/trunk/pypy: module/gc module/gc/test
	objspace/std
Message-ID: <20101027131511.4EE55282C1C@codespeak.net>

Author: arigo
Date: Wed Oct 27 15:15:09 2010
New Revision: 78353

Modified:
   pypy/trunk/pypy/module/gc/interp_gc.py
   pypy/trunk/pypy/module/gc/test/test_gc.py
   pypy/trunk/pypy/objspace/std/typeobject.py
Log:
(amaury on irc, arigo)
When doing an explicit gc.collect(), also clear the method cache.
This is the easiest way to fix the "semi-leaks" shown by the new
test in test_gc.  We could also make weak pointers in the cache,
but so far it would be messy and seriously slower.


Modified: pypy/trunk/pypy/module/gc/interp_gc.py
==============================================================================
--- pypy/trunk/pypy/module/gc/interp_gc.py	(original)
+++ pypy/trunk/pypy/module/gc/interp_gc.py	Wed Oct 27 15:15:09 2010
@@ -5,6 +5,11 @@
 
 def collect(space):
     "Run a full collection."
+    # First clear the method cache.  See test_gc for an example of why.
+    if space.config.objspace.std.withmethodcache:
+        from pypy.objspace.std.typeobject import MethodCache
+        cache = space.fromcache(MethodCache)
+        cache.clear()
     rgc.collect()
     return space.wrap(0)
     

Modified: pypy/trunk/pypy/module/gc/test/test_gc.py
==============================================================================
--- pypy/trunk/pypy/module/gc/test/test_gc.py	(original)
+++ pypy/trunk/pypy/module/gc/test/test_gc.py	Wed Oct 27 15:15:09 2010
@@ -103,3 +103,22 @@
         import gc
         gc.dump_heap_stats(self.fname)
 
+
+class AppTestGcMethodCache(object):
+    def setup_class(cls):
+        cls.space = gettestobjspace(**{"objspace.std.withmethodcache": True})
+
+    def test_clear_method_cache(self):
+        import gc, weakref
+        rlist = []
+        def f():
+            class C(object):
+                def f(self):
+                    pass
+            C().f()    # Fill the method cache
+            rlist.append(weakref.ref(C))
+        for i in range(5):
+            f()
+        gc.collect()    # the classes C should all go away here
+        for r in rlist:
+            assert r() is None

Modified: pypy/trunk/pypy/objspace/std/typeobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/typeobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/typeobject.py	Wed Oct 27 15:15:09 2010
@@ -60,6 +60,15 @@
             self.hits = {}
             self.misses = {}
 
+    def clear(self):
+        None_None = (None, None)
+        for i in range(len(self.versions)):
+            self.versions[i] = None
+        for i in range(len(self.names)):
+            self.names[i] = None
+        for i in range(len(self.lookup_where)):
+            self.lookup_where[i] = None_None
+
 
 class W_TypeObject(W_Object):
     from pypy.objspace.std.typetype import type_typedef as typedef


From arigo at codespeak.net  Wed Oct 27 15:20:42 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 15:20:42 +0200 (CEST)
Subject: [pypy-svn] r78354 - in pypy/branch/cleanup-dict-impl/pypy/doc: .
	config
Message-ID: <20101027132042.C0AB9282C1C@codespeak.net>

Author: arigo
Date: Wed Oct 27 15:20:41 2010
New Revision: 78354

Removed:
   pypy/branch/cleanup-dict-impl/pypy/doc/config/objspace.std.withinlineddict.txt
   pypy/branch/cleanup-dict-impl/pypy/doc/config/objspace.std.withsharingdict.txt
Modified:
   pypy/branch/cleanup-dict-impl/pypy/doc/interpreter-optimizations.txt
Log:
Fixes to docs.


Modified: pypy/branch/cleanup-dict-impl/pypy/doc/interpreter-optimizations.txt
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/doc/interpreter-optimizations.txt	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/doc/interpreter-optimizations.txt	Wed Oct 27 15:20:41 2010
@@ -153,8 +153,8 @@
 dicts:
 the representation of the instance dict contains only a list of values.
 
-You can enable this feature with the :config:`objspace.std.withsharingdict`
-option.
+A more advanced version of sharing dicts, called *map dicts,* is available
+with the :config:`objspace.std.withmapdict` option.
 
 Builtin-Shadowing
 +++++++++++++++++
@@ -219,8 +219,7 @@
 shadowing the class attribute. If we know that there is no shadowing (since
 instance dict tells us that) we can save this lookup on the instance dictionary.
 
-You can enable this feature with the :config:`objspace.std.withshadowtracking`
-option.
+*This was deprecated and is no longer available.*
 
 
 Method Caching


From david at codespeak.net  Wed Oct 27 15:29:10 2010
From: david at codespeak.net (david at codespeak.net)
Date: Wed, 27 Oct 2010 15:29:10 +0200 (CEST)
Subject: [pypy-svn] r78355 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . test
Message-ID: <20101027132910.3F78D282C1C@codespeak.net>

Author: david
Date: Wed Oct 27 15:29:08 2010
New Revision: 78355

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
Log:
Encode multiplication operations

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	Wed Oct 27 15:29:08 2010
@@ -107,6 +107,38 @@
                     | (crm & 0xF))
     return f
 
+def define_multiply_instructions(name, table):
+    n = (table['op'] & 0xF) << 20 | 0x9 << 4
+    if 'acc' in table and table['acc']:
+        def f(self, rd, rn, rm, ra, cond=cond.AL, s=0):
+            self.write32(n
+                        | cond << 28
+                        | (s & 0x1)
+                        | (rd & 0xF) << 16
+                        | (ra & 0xF) << 12
+                        | (rm & 0xF) << 8
+                        | (rn & 0xF))
+    elif 'long' in table and table['long']:
+       def f(self, rdhi, rdlo, rn, rm, cond=cond.AL, s=0):
+            assert rdhi != rdlo
+            self.write32(n
+                        | cond << 28
+                        | (s & 0x1)
+                        | (rdhi & 0xF) << 16
+                        | (rdlo & 0xF) << 12
+                        | (rm & 0xF) << 8
+                        | (rn & 0xF))
+    else:
+        def f(self, rd, rn, rm, cond=cond.AL, s=0):
+            self.write32(n
+                        | cond << 28
+                        | (s & 0x1)
+                        | (rd & 0xF) << 16
+                        | (rm & 0xF) << 8
+                        | (rn & 0xF))
+
+    return f
+
 def imm_operation(rt, rn, imm):
     return ((rn & 0xFF) << 16
     | (rt & 0xFF) << 12
@@ -126,14 +158,12 @@
         setattr(target, key, f)
 
 def define_instructions(target):
-    for key, val in instructions.load_store.iteritems():
-        define_instruction(define_load_store_func, key, val, target)
-
-    for key, val in instructions.data_proc.iteritems():
-        define_instruction(define_data_proc, key, val, target)
-
-    for key, val in instructions.data_proc_imm.iteritems():
-        define_instruction(define_data_proc_imm, key, val, target)
-
-    for key, val in instructions.supervisor_and_coproc.iteritems():
-        define_instruction(define_supervisor_and_coproc, key, val, target)
+    i_g_map = [(instructions.load_store, define_load_store_func),
+                (instructions.data_proc, define_data_proc),
+                (instructions.data_proc_imm, define_data_proc_imm),
+                (instructions.supervisor_and_coproc, define_supervisor_and_coproc),
+                (instructions.multiply, define_multiply_instructions)]
+
+    for inss, gen in i_g_map:
+        for key, val in inss.iteritems():
+            define_instruction(gen, key, val, target)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	Wed Oct 27 15:29:08 2010
@@ -71,3 +71,14 @@
     'B':     {'op': 0x20},
     'BL':    {'op': 0x30},
 }
+
+multiply = {
+    'MUL':   {'op':0x0},
+    'MLA':   {'op':0x2, 'acc': True},
+    'UMAAL': {'op':0x4, 'long': True},
+    'MLS':   {'op':0x6, 'acc': True},
+    'UMULL': {'op':0x8, 'long': True},
+    'UMLAL': {'op':0xA, 'long': True},
+    'SMULL': {'op':0xC, 'long': True},
+    'SMLAL': {'op':0xE, 'long': True},
+}

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	Wed Oct 27 15:29:08 2010
@@ -141,6 +141,8 @@
     for key, value, in instructions.data_proc_imm.iteritems():
         build_test(gen_test_data_proc_imm_func, key, value, test_name)
 
+    for key, value, in instructions.multiply.iteritems():
+        build_test(gen_test_mul_func, key, value, test_name)
 # XXX refactor this functions
 
 def build_test(builder, key, value, test_name):
@@ -180,6 +182,24 @@
         self.assert_equal('%s r3, [r7, r12]' % name[:name.index('_')])
     return f
 
+def gen_test_mul_func(name, table):
+    if 'acc' in table and table['acc']:
+        def f(self):
+            func = getattr(self.cb, name)
+            func(r.r3.value, r.r7.value, r.r12.value, r.r13.value)
+            self.assert_equal('%s r3, r7, r12, r13' % name)
+    elif 'long' in table and table['long']:
+        def f(self):
+            func = getattr(self.cb, name)
+            func(r.r3.value, r.r13.value, r.r7.value, r.r12.value)
+            self.assert_equal('%s r13, r3, r7, r12' % name)
+    else:
+        def f(self):
+            func = getattr(self.cb, name)
+            func(r.r3.value, r.r7.value, r.r12.value)
+            self.assert_equal('%s r3, r7, r12' % name)
+    return f
+
 def gen_test_data_reg_func(name, table):
     if name[-2:] == 'ri':
         def f(self):


From arigo at codespeak.net  Wed Oct 27 15:29:51 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 15:29:51 +0200 (CEST)
Subject: [pypy-svn] r78356 - pypy/build/bot2/pypybuildbot
Message-ID: <20101027132951.C28F0282C1C@codespeak.net>

Author: arigo
Date: Wed Oct 27 15:29:50 2010
New Revision: 78356

Modified:
   pypy/build/bot2/pypybuildbot/master.py
Log:
Add "macmini-mvt" to the list of buildslaves.
No nightly scheduled run so far.


Modified: pypy/build/bot2/pypybuildbot/master.py
==============================================================================
--- pypy/build/bot2/pypybuildbot/master.py	(original)
+++ pypy/build/bot2/pypybuildbot/master.py	Wed Oct 27 15:29:50 2010
@@ -138,7 +138,7 @@
     )
 
 pypyJITTranslatedTestFactoryOSX = pypybuilds.Translated(
-    platform='osx',
+    platform='osx',    # works also for "OS/X 64"
     translationArgs=jit_translation_args,
     targetArgs=[],
     lib_python=True,
@@ -178,7 +178,7 @@
 JITLINUX32 = "pypy-c-jit-linux-x86-32"
 JITLINUX64 = "pypy-c-jit-linux-x86-64"
 OJITLINUX32 = "pypy-c-Ojit-no-jit-linux-x86-32"
-JITMACOSX32 = "pypy-c-jit-macosx-x86-32"
+JITMACOSX64 = "pypy-c-jit-macosx-x86-64"
 JITWIN32 = "pypy-c-jit-win-x86-32"
 
 JITONLYLINUX32 = "jitonly-own-linux-x86-32"
@@ -288,9 +288,9 @@
                    'factory': pypyJITTranslatedTestFactory64,
                    'category': 'jit64',
                   },
-                  {"name" : JITMACOSX32,
-                   "slavenames": ["minime"],
-                   'builddir' : JITMACOSX32,
+                  {"name" : JITMACOSX64,
+                   "slavenames": ["macmini-mvt"],
+                   'builddir' : JITMACOSX64,
                    'factory' : pypyJITTranslatedTestFactoryOSX,
                    'category' : 'jit',
                    },


From arigo at codespeak.net  Wed Oct 27 15:37:29 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 15:37:29 +0200 (CEST)
Subject: [pypy-svn] r78357 - pypy/trunk/pypy/jit/backend
Message-ID: <20101027133729.AA9975080C@codespeak.net>

Author: arigo
Date: Wed Oct 27 15:37:28 2010
New Revision: 78357

Modified:
   pypy/trunk/pypy/jit/backend/detect_cpu.py
Log:
Detect 32-vs-64-bits processors by relying systematically
on the value of sys.maxint.


Modified: pypy/trunk/pypy/jit/backend/detect_cpu.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/detect_cpu.py	(original)
+++ pypy/trunk/pypy/jit/backend/detect_cpu.py	Wed Oct 27 15:37:28 2010
@@ -23,11 +23,6 @@
         mach = os.popen('uname -m', 'r').read().strip()
         if not mach:
             raise ProcessorAutodetectError, "cannot run 'uname -m'"
-    if mach == 'x86_64':
-        if sys.maxint == 2147483647:
-            mach = 'x86'     # it's a 64-bit processor but in 32-bits mode, maybe
-        else:
-            assert sys.maxint == 2 ** 63 - 1
     try:
         return {'i386': 'x86',
                 'i486': 'x86',
@@ -36,17 +31,21 @@
                 'i86pc': 'x86',    # Solaris/Intel
                 'x86':   'x86',    # Apple
                 'Power Macintosh': 'ppc',
-                'x86_64': 'x86_64', 
+                'x86_64': 'x86', 
                 }[mach]
     except KeyError:
         raise ProcessorAutodetectError, "unsupported processor '%s'" % mach
 
 def autodetect():
     model = autodetect_main_model()
-    if model == 'x86':
-        from pypy.jit.backend.x86.detect_sse2 import detect_sse2
-        if not detect_sse2():
-            model = 'x86-without-sse2'
+    if sys.maxint == 2**63-1:
+        model += '_64'
+    else:
+        assert sys.maxint == 2**31-1
+        if model == 'x86':
+            from pypy.jit.backend.x86.detect_sse2 import detect_sse2
+            if not detect_sse2():
+                model = 'x86-without-sse2'
     return model
 
 def getcpuclassname(backend_name="auto"):


From arigo at codespeak.net  Wed Oct 27 16:03:52 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 16:03:52 +0200 (CEST)
Subject: [pypy-svn] r78358 - pypy/branch/cleanup-dict-impl/pypy/doc/config
Message-ID: <20101027140352.C99B5282C1C@codespeak.net>

Author: arigo
Date: Wed Oct 27 16:03:50 2010
New Revision: 78358

Modified:
   pypy/branch/cleanup-dict-impl/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt
Log:
Kill dead link.


Modified: pypy/branch/cleanup-dict-impl/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt	(original)
+++ pypy/branch/cleanup-dict-impl/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt	Wed Oct 27 16:03:50 2010
@@ -5,8 +5,6 @@
 case.  So far, this only works for calls with no keyword, no ``*arg``
 and no ``**arg`` but it would be easy to extend.
 
-Gives the best results combined with :config:`objspace.std.withshadowtracking`.
-
 For more information, see the section in `Standard Interpreter Optimizations`_.
 
 .. _`Standard Interpreter Optimizations`: ../interpreter-optimizations.html#lookup-method-call-method


From arigo at codespeak.net  Wed Oct 27 16:20:29 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 16:20:29 +0200 (CEST)
Subject: [pypy-svn] r78359 - in pypy/trunk/pypy: config config/test doc
	doc/config interpreter jit/codewriter jit/tl
	module/__builtin__ module/__builtin__/test
	module/pypyjit/test objspace objspace/std objspace/std/test
Message-ID: <20101027142029.C28495080C@codespeak.net>

Author: arigo
Date: Wed Oct 27 16:20:26 2010
New Revision: 78359

Added:
   pypy/trunk/pypy/objspace/std/test/test_methodcache.py
      - copied, changed from r78358, pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_methodcache.py
Removed:
   pypy/trunk/pypy/doc/config/objspace.std.withinlineddict.txt
   pypy/trunk/pypy/doc/config/objspace.std.withshadowtracking.txt
   pypy/trunk/pypy/doc/config/objspace.std.withsharingdict.txt
   pypy/trunk/pypy/objspace/std/inlinedict.py
   pypy/trunk/pypy/objspace/std/sharingdict.py
   pypy/trunk/pypy/objspace/std/test/test_inlinedict.py
   pypy/trunk/pypy/objspace/std/test/test_shadowtracking.py
   pypy/trunk/pypy/objspace/std/test/test_sharingdict.py
Modified:
   pypy/trunk/pypy/config/pypyoption.py
   pypy/trunk/pypy/config/test/test_pypyoption.py
   pypy/trunk/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt
   pypy/trunk/pypy/doc/interpreter-optimizations.txt
   pypy/trunk/pypy/interpreter/baseobjspace.py
   pypy/trunk/pypy/interpreter/typedef.py
   pypy/trunk/pypy/jit/codewriter/jtransform.py
   pypy/trunk/pypy/jit/tl/pypyjit.py
   pypy/trunk/pypy/module/__builtin__/interp_classobj.py
   pypy/trunk/pypy/module/__builtin__/test/test_classobj.py
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
   pypy/trunk/pypy/objspace/descroperation.py
   pypy/trunk/pypy/objspace/std/callmethod.py
   pypy/trunk/pypy/objspace/std/celldict.py
   pypy/trunk/pypy/objspace/std/dictmultiobject.py
   pypy/trunk/pypy/objspace/std/mapdict.py
   pypy/trunk/pypy/objspace/std/objspace.py
   pypy/trunk/pypy/objspace/std/proxyobject.py
   pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py
Log:
(cfbolz, arigo)
Merge branch/cleanup-dict-impl: get rid of the inlinedict and sharingdict
and use mapdict instead -- now by default with the JIT, including a fix
that stops mapdict from creating tons of versions of the code.


Modified: pypy/trunk/pypy/config/pypyoption.py
==============================================================================
--- pypy/trunk/pypy/config/pypyoption.py	(original)
+++ pypy/trunk/pypy/config/pypyoption.py	Wed Oct 27 16:20:26 2010
@@ -229,27 +229,15 @@
                    requires=[("objspace.opcodes.CALL_LIKELY_BUILTIN", False),
                              ("objspace.honor__builtins__", False)]),
 
-        BoolOption("withsharingdict",
-                   "use dictionaries that share the keys part",
-                   default=False),
-
         BoolOption("withdictmeasurement",
                    "create huge files with masses of information "
                    "about dictionaries",
                    default=False),
 
-        BoolOption("withinlineddict",
-                   "make instances more compact by revoming a level of indirection",
-                   default=False,
-                   requires=[("objspace.std.withshadowtracking", False)]),
-
         BoolOption("withmapdict",
                    "make instances really small but slow without the JIT",
                    default=False,
-                   requires=[("objspace.std.withshadowtracking", False),
-                             ("objspace.std.withinlineddict", False),
-                             ("objspace.std.withsharingdict", False),
-                             ("objspace.std.getattributeshortcut", True),
+                   requires=[("objspace.std.getattributeshortcut", True),
                              ("objspace.std.withtypeversion", True),
                        ]),
 
@@ -266,12 +254,6 @@
                    # weakrefs needed, because of get_subclasses()
                    requires=[("translation.rweakref", True)]),
 
-        BoolOption("withshadowtracking",
-                   "track whether an instance attribute shadows a type"
-                   " attribute",
-                   default=False,
-                   requires=[("objspace.std.withtypeversion", True),
-                             ("translation.rweakref", True)]),
         BoolOption("withmethodcache",
                    "try to cache method lookups",
                    default=False,
@@ -343,9 +325,6 @@
         config.objspace.std.suggest(optimized_list_getitem=True)
         config.objspace.std.suggest(getattributeshortcut=True)
         config.objspace.std.suggest(newshortcut=True)        
-        if type_system != 'ootype':
-            config.objspace.std.suggest(withsharingdict=True)
-        config.objspace.std.suggest(withinlineddict=True)
 
     # extra costly optimizations only go in level 3
     if level == '3':
@@ -374,7 +353,7 @@
     # extra optimizations with the JIT
     if level == 'jit':
         config.objspace.std.suggest(withcelldict=True)
-        #config.objspace.std.suggest(withmapdict=True)
+        config.objspace.std.suggest(withmapdict=True)
 
 
 def enable_allworkingmodules(config):

Modified: pypy/trunk/pypy/config/test/test_pypyoption.py
==============================================================================
--- pypy/trunk/pypy/config/test/test_pypyoption.py	(original)
+++ pypy/trunk/pypy/config/test/test_pypyoption.py	Wed Oct 27 16:20:26 2010
@@ -47,7 +47,7 @@
 def test_set_pypy_opt_level():
     conf = get_pypy_config()
     set_pypy_opt_level(conf, '2')
-    assert conf.objspace.std.withsharingdict
+    assert conf.objspace.std.newshortcut
     conf = get_pypy_config()
     set_pypy_opt_level(conf, '0')
     assert not conf.objspace.std.newshortcut
@@ -59,7 +59,6 @@
 
     assert not conf.objspace.std.withtypeversion
     assert not conf.objspace.std.withmethodcache
-    assert not conf.objspace.std.withshadowtracking
 
 def test_check_documentation():
     def check_file_exists(fn):

Modified: pypy/trunk/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt
==============================================================================
--- pypy/trunk/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt	(original)
+++ pypy/trunk/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt	Wed Oct 27 16:20:26 2010
@@ -5,8 +5,6 @@
 case.  So far, this only works for calls with no keyword, no ``*arg``
 and no ``**arg`` but it would be easy to extend.
 
-Gives the best results combined with :config:`objspace.std.withshadowtracking`.
-
 For more information, see the section in `Standard Interpreter Optimizations`_.
 
 .. _`Standard Interpreter Optimizations`: ../interpreter-optimizations.html#lookup-method-call-method

Modified: pypy/trunk/pypy/doc/interpreter-optimizations.txt
==============================================================================
--- pypy/trunk/pypy/doc/interpreter-optimizations.txt	(original)
+++ pypy/trunk/pypy/doc/interpreter-optimizations.txt	Wed Oct 27 16:20:26 2010
@@ -153,8 +153,8 @@
 dicts:
 the representation of the instance dict contains only a list of values.
 
-You can enable this feature with the :config:`objspace.std.withsharingdict`
-option.
+A more advanced version of sharing dicts, called *map dicts,* is available
+with the :config:`objspace.std.withmapdict` option.
 
 Builtin-Shadowing
 +++++++++++++++++
@@ -219,8 +219,7 @@
 shadowing the class attribute. If we know that there is no shadowing (since
 instance dict tells us that) we can save this lookup on the instance dictionary.
 
-You can enable this feature with the :config:`objspace.std.withshadowtracking`
-option.
+*This was deprecated and is no longer available.*
 
 
 Method Caching

Modified: pypy/trunk/pypy/interpreter/baseobjspace.py
==============================================================================
--- pypy/trunk/pypy/interpreter/baseobjspace.py	(original)
+++ pypy/trunk/pypy/interpreter/baseobjspace.py	Wed Oct 27 16:20:26 2010
@@ -36,13 +36,10 @@
             return space.finditem_str(w_dict, attr)
         return None
 
-    def getdictvalue_attr_is_in_class(self, space, attr):
-        return self.getdictvalue(space, attr)
-
-    def setdictvalue(self, space, attr, w_value, shadows_type=True):
+    def setdictvalue(self, space, attr, w_value):
         w_dict = self.getdict()
         if w_dict is not None:
-            space.setitem_str(w_dict, attr, w_value, shadows_type)
+            space.setitem_str(w_dict, attr, w_value)
             return True
         return False
 
@@ -657,7 +654,7 @@
         """shortcut for space.int_w(space.hash(w_obj))"""
         return self.int_w(self.hash(w_obj))
 
-    def setitem_str(self, w_obj, key, w_value, shadows_type=True):
+    def setitem_str(self, w_obj, key, w_value):
         return self.setitem(w_obj, self.wrap(key), w_value)
 
     def finditem_str(self, w_obj, key):

Modified: pypy/trunk/pypy/interpreter/typedef.py
==============================================================================
--- pypy/trunk/pypy/interpreter/typedef.py	(original)
+++ pypy/trunk/pypy/interpreter/typedef.py	Wed Oct 27 16:20:26 2010
@@ -262,16 +262,7 @@
                 return self.slots_w[index]
         add(Proto)
 
-    wantdict = "dict" in features
-    if wantdict and config.objspace.std.withinlineddict:
-        from pypy.objspace.std.objectobject import W_ObjectObject
-        from pypy.objspace.std.inlinedict import make_mixin
-        if supercls is W_ObjectObject:
-            Mixin = make_mixin(config)
-            add(Mixin)
-            wantdict = False
-
-    if wantdict:
+    if "dict" in features:
         base_user_setup = supercls.user_setup.im_func
         if "user_setup" in body:
             base_user_setup = body["user_setup"]
@@ -290,15 +281,6 @@
             def setclass(self, space, w_subtype):
                 # only used by descr_set___class__
                 self.w__class__ = w_subtype
-                if space.config.objspace.std.withshadowtracking:
-                    self.w__dict__.set_shadows_anything()
-
-            def getdictvalue_attr_is_in_class(self, space, name):
-                w_dict = self.w__dict__
-                if space.config.objspace.std.withshadowtracking:
-                    if not w_dict.shadows_anything():
-                        return None
-                return space.finditem_str(w_dict, name)
 
         add(Proto)
 

Modified: pypy/trunk/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/jtransform.py	Wed Oct 27 16:20:26 2010
@@ -172,6 +172,7 @@
 
     def rewrite_op_same_as(self, op): pass
     def rewrite_op_cast_pointer(self, op): pass
+    def rewrite_op_cast_opaque_ptr(self, op): pass   # rlib.rerased
     def rewrite_op_cast_primitive(self, op): pass
     def rewrite_op_cast_bool_to_int(self, op): pass
     def rewrite_op_cast_bool_to_uint(self, op): pass

Modified: pypy/trunk/pypy/jit/tl/pypyjit.py
==============================================================================
--- pypy/trunk/pypy/jit/tl/pypyjit.py	(original)
+++ pypy/trunk/pypy/jit/tl/pypyjit.py	Wed Oct 27 16:20:26 2010
@@ -44,7 +44,6 @@
 config.objspace.usemodules._ffi = True
 #
 set_pypy_opt_level(config, level='jit')
-config.objspace.std.withinlineddict = True
 
 if BACKEND == 'c':
     config.objspace.std.multimethods = 'mrd'

Modified: pypy/trunk/pypy/module/__builtin__/interp_classobj.py
==============================================================================
--- pypy/trunk/pypy/module/__builtin__/interp_classobj.py	(original)
+++ pypy/trunk/pypy/module/__builtin__/interp_classobj.py	Wed Oct 27 16:20:26 2010
@@ -410,8 +410,7 @@
         if w_meth is not None:
             space.call_function(w_meth, w_name, w_value)
         else:
-            # bit obscure: appease normalization
-            self.setdictvalue(space, name, w_value, True)
+            self.setdictvalue(space, name, w_value)
 
     def descr_delattr(self, space, w_name):
         name = unwrap_attr(space, w_name)

Modified: pypy/trunk/pypy/module/__builtin__/test/test_classobj.py
==============================================================================
--- pypy/trunk/pypy/module/__builtin__/test/test_classobj.py	(original)
+++ pypy/trunk/pypy/module/__builtin__/test/test_classobj.py	Wed Oct 27 16:20:26 2010
@@ -954,28 +954,7 @@
         raises(TypeError, descr.__delete__, a)
 
 
-class AppTestOldStyleSharing(AppTestOldstyle):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withsharingdict": True})
-        if option.runappdirect:
-            py.test.skip("can only be run on py.py")
-        def is_sharing(space, w_inst):
-            from pypy.objspace.std.sharingdict import SharedDictImplementation
-            w_d = w_inst.getdict()
-            return space.wrap(isinstance(w_d, SharedDictImplementation) and w_d.r_dict_content is None)
-        cls.w_is_sharing = cls.space.wrap(gateway.interp2app(is_sharing))
-
-
-    def test_real_sharing(self):
-        class A:
-            def __init__(self):
-                self.x = 42
-        A1, A2, A3 = A(), A(), A()
-        assert self.is_sharing(A3)
-        assert self.is_sharing(A2)
-        assert self.is_sharing(A1)
-
-class AppTestOldStyleModDict(object):
+class AppTestOldStyleClassStrDict(object):
     def setup_class(cls):
         if option.runappdirect:
             py.test.skip("can only be run on py.py")

Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Wed Oct 27 16:20:26 2010
@@ -136,9 +136,10 @@
         assert opslogfile.check()
         log = logparser.parse_log_file(str(opslogfile))
         parts = logparser.extract_category(log, 'jit-log-opt-')
+        self.rawloops = [part for part in parts
+                         if not from_entry_bridge(part, parts)]
         # skip entry bridges, they can contain random things
-        self.loops = [parse(part, no_namespace=True) for part in parts
-                          if not from_entry_bridge(part, parts)]
+        self.loops = [parse(part, no_namespace=True) for part in self.rawloops]
         self.sliced_loops = [] # contains all bytecodes of all loops
         self.total_ops = 0
         for loop in self.loops:
@@ -162,12 +163,11 @@
         return [ops for ops in self.sliced_loops if ops.bytecode == name]
 
     def print_loops(self):
-        for loop in self.loops:
+        for rawloop in self.rawloops:
             print
             print '@' * 79
             print
-            for op in loop.operations:
-                print op
+            print rawloop.rstrip()
         print
         print '@' * 79
 
@@ -278,7 +278,7 @@
         assert len(ops) == 2
         assert not ops[0].get_opnames("call")
         assert not ops[0].get_opnames("new")
-        assert len(ops[0].get_opnames("guard")) <= 3     # we get 2 withmapdict
+        assert len(ops[0].get_opnames("guard")) <= 2
         assert not ops[1] # second LOOKUP_METHOD folded away
 
         ops = self.get_by_bytecode("CALL_METHOD")
@@ -323,8 +323,8 @@
         assert len(ops) == 2
         assert not ops[0].get_opnames("call")
         assert not ops[0].get_opnames("new")
-        assert len(ops[0].get_opnames("guard")) <= 3    # we get 2 withmapdict
-        assert len(ops[0].get_opnames("getfield")) <= 5 # we get <5 withmapdict
+        assert len(ops[0].get_opnames("guard")) <= 2
+        assert len(ops[0].get_opnames("getfield")) <= 4
         assert not ops[1] # second LOOKUP_METHOD folded away
 
     def test_default_and_kw(self):

Modified: pypy/trunk/pypy/objspace/descroperation.py
==============================================================================
--- pypy/trunk/pypy/objspace/descroperation.py	(original)
+++ pypy/trunk/pypy/objspace/descroperation.py	Wed Oct 27 16:20:26 2010
@@ -64,9 +64,7 @@
                     w_type = space.type(w_obj)
                     return space.get_and_call_function(w_get, w_descr, w_obj,
                                                        w_type)
-            w_value = w_obj.getdictvalue_attr_is_in_class(space, name)
-        else:
-            w_value = w_obj.getdictvalue(space, name)
+        w_value = w_obj.getdictvalue(space, name)
         if w_value is not None:
             return w_value
         if w_descr is not None:
@@ -76,13 +74,11 @@
     def descr__setattr__(space, w_obj, w_name, w_value):
         name = space.str_w(w_name)
         w_descr = space.lookup(w_obj, name)
-        shadows_type = False
         if w_descr is not None:
             if space.is_data_descr(w_descr):
                 space.set(w_descr, w_obj, w_value)
                 return
-            shadows_type = True
-        if w_obj.setdictvalue(space, name, w_value, shadows_type):
+        if w_obj.setdictvalue(space, name, w_value):
             return
         raiseattrerror(space, w_obj, name, w_descr)
 

Modified: pypy/trunk/pypy/objspace/std/callmethod.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/callmethod.py	(original)
+++ pypy/trunk/pypy/objspace/std/callmethod.py	Wed Oct 27 16:20:26 2010
@@ -44,7 +44,7 @@
         else:
             typ = type(w_descr)
             if typ is function.Function or typ is function.FunctionWithFixedCode:
-                w_value = w_obj.getdictvalue_attr_is_in_class(space, name)
+                w_value = w_obj.getdictvalue(space, name)
                 if w_value is None:
                     # fast method path: a function object in the class,
                     # nothing in the instance
@@ -103,7 +103,7 @@
         w_descr = space.lookup(w_obj, methname)
         typ = type(w_descr)
         if typ is function.Function or typ is function.FunctionWithFixedCode:
-            w_value = w_obj.getdictvalue_attr_is_in_class(space, methname)
+            w_value = w_obj.getdictvalue(space, methname)
             if w_value is None:
                 # fast method path: a function object in the class,
                 # nothing in the instance

Modified: pypy/trunk/pypy/objspace/std/celldict.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/celldict.py	(original)
+++ pypy/trunk/pypy/objspace/std/celldict.py	Wed Oct 27 16:20:26 2010
@@ -47,7 +47,7 @@
         else:
             self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
-    def impl_setitem_str(self, name, w_value, shadows_type=True):
+    def impl_setitem_str(self, name, w_value):
         self.getcell(name, True).w_value = w_value
 
     def impl_delitem(self, w_key):

Modified: pypy/trunk/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/dictmultiobject.py	Wed Oct 27 16:20:26 2010
@@ -49,14 +49,6 @@
         elif space.config.objspace.std.withdictmeasurement:
             assert w_type is None
             return MeasuringDictImplementation(space)
-        elif space.config.objspace.std.withsharingdict and instance:
-            from pypy.objspace.std.sharingdict import SharedDictImplementation
-            assert w_type is None
-            return SharedDictImplementation(space)
-        elif (space.config.objspace.std.withshadowtracking and instance and
-                classofinstance is not None):
-            assert w_type is None
-            return ShadowDetectingDictImplementation(space, classofinstance)
         elif instance or strdict or module:
             assert w_type is None
             return StrDictImplementation(space)
@@ -112,7 +104,7 @@
         #return w_value or None
         raise NotImplementedError("abstract base class")
 
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         raise NotImplementedError("abstract base class")
 
     def impl_setitem(self,  w_key, w_value):
@@ -165,20 +157,13 @@
         key = OPTIMIZED_BUILTINS[i]
         return self.impl_getitem_str(key)
 
-    # this method will only be seen whan a certain config option is used
-    def impl_shadows_anything(self):
-        return True
-
-    def impl_set_shadows_anything(self):
-        pass
-
     # _________________________________________________________________
     # fallback implementation methods
 
     def impl_fallback_setitem(self, w_key, w_value):
         self.r_dict_content[w_key] = w_value
 
-    def impl_fallback_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_fallback_setitem_str(self, key, w_value):
         return self.impl_fallback_setitem(self.space.wrap(key), w_value)
 
     def impl_fallback_delitem(self, w_key):
@@ -211,18 +196,12 @@
         key = OPTIMIZED_BUILTINS[i]
         return self.impl_fallback_getitem_str(key)
 
-    def impl_fallback_shadows_anything(self):
-        return True
-
-    def impl_fallback_set_shadows_anything(self):
-        pass
-
 
 implementation_methods = [
     ("getitem", 1),
     ("getitem_str", 1),
     ("length", 0),
-    ("setitem_str", 3),
+    ("setitem_str", 2),
     ("setitem", 2),
     ("delitem", 1),
     ("iter", 0),
@@ -231,8 +210,6 @@
     ("keys", 0),
     ("clear", 0),
     ("get_builtin_indexed", 1),
-    ("shadows_anything", 0),
-    ("set_shadows_anything", 0),
 ]
 
 
@@ -312,7 +289,7 @@
         else:
             self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         self.content[key] = w_value
 
     def impl_delitem(self, w_key):
@@ -388,47 +365,12 @@
             return None, None
 
 
-class ShadowDetectingDictImplementation(StrDictImplementation):
-    def __init__(self, space, w_type):
-        StrDictImplementation.__init__(self, space)
-        self.w_type = w_type
-        self.original_version_tag = w_type.version_tag()
-        if self.original_version_tag is None:
-            self._shadows_anything = True
-        else:
-            self._shadows_anything = False
-
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
-        if shadows_type:
-            self._shadows_anything = True
-        StrDictImplementation.impl_setitem_str(
-            self, key, w_value, shadows_type)
-
-    def impl_setitem(self, w_key, w_value):
-        space = self.space
-        if space.is_w(space.type(w_key), space.w_str):
-            if not self._shadows_anything:
-                w_obj = self.w_type.lookup(space.str_w(w_key))
-                if w_obj is not None:
-                    self._shadows_anything = True
-            StrDictImplementation.impl_setitem_str(
-                self, self.space.str_w(w_key), w_value, False)
-        else:
-            self._as_rdict().impl_fallback_setitem(w_key, w_value)
-
-    def impl_shadows_anything(self):
-        return (self._shadows_anything or 
-                self.w_type.version_tag() is not self.original_version_tag)
-
-    def impl_set_shadows_anything(self):
-        self._shadows_anything = True
-
 class WaryDictImplementation(StrDictImplementation):
     def __init__(self, space):
         StrDictImplementation.__init__(self, space)
         self.shadowed = [None] * len(BUILTIN_TO_INDEX)
 
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         i = BUILTIN_TO_INDEX.get(key, -1)
         if i != -1:
             self.shadowed[i] = w_value
@@ -558,7 +500,7 @@
         self.info.writes += 1
         self.content[w_key] = w_value
         self.info.maxcontents = max(self.info.maxcontents, len(self.content))
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         self.info.setitem_strs += 1
         self.impl_setitem(self.space.wrap(key), w_value)
     def impl_delitem(self, w_key):

Modified: pypy/trunk/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/mapdict.py	(original)
+++ pypy/trunk/pypy/objspace/std/mapdict.py	Wed Oct 27 16:20:26 2010
@@ -1,4 +1,4 @@
-from pypy.rlib import jit, objectmodel
+from pypy.rlib import jit, objectmodel, debug
 
 from pypy.interpreter.baseobjspace import W_Root
 from pypy.objspace.std.dictmultiobject import W_DictMultiObject
@@ -10,6 +10,9 @@
 # attribute shapes
 
 NUM_DIGITS = 4
+NUM_DIGITS_POW2 = 1 << NUM_DIGITS
+# note: we use "x * NUM_DIGITS_POW2" instead of "x << NUM_DIGITS" because
+# we want to propagate knowledge that the result cannot be negative
 
 class AbstractAttribute(object):
     _immutable_fields_ = ['w_cls']
@@ -69,8 +72,10 @@
         attr = self._get_new_attr(selector[0], selector[1])
         oldattr = obj._get_mapdict_map()
         if not jit.we_are_jitted():
-            oldattr._size_estimate += attr.size_estimate() - oldattr.size_estimate()
-            assert oldattr.size_estimate() >= oldattr.length()
+            size_est = (oldattr._size_estimate + attr.size_estimate()
+                                               - oldattr.size_estimate())
+            assert size_est >= (oldattr.length() * NUM_DIGITS_POW2)
+            oldattr._size_estimate = size_est
         if attr.length() > obj._mapdict_storage_length():
             # note that attr.size_estimate() is always at least attr.length()
             new_storage = [None] * attr.size_estimate()
@@ -188,7 +193,7 @@
         self.selector = selector
         self.position = back.length()
         self.back = back
-        self._size_estimate = self.length() << NUM_DIGITS
+        self._size_estimate = self.length() * NUM_DIGITS_POW2
 
     def _copy_attr(self, obj, new_obj):
         w_value = self.read(obj, self.selector)
@@ -291,7 +296,7 @@
     def getdictvalue(self, space, attrname):
         return self._get_mapdict_map().read(self, (attrname, DICT))
 
-    def setdictvalue(self, space, attrname, w_value, shadows_type=True):
+    def setdictvalue(self, space, attrname, w_value):
         return self._get_mapdict_map().write(self, (attrname, DICT), w_value)
 
     def deldictvalue(self, space, w_name):
@@ -383,16 +388,18 @@
     assert space.config.objspace.std.withmapdict
     map = w_type.terminator
     classes = memo_get_subclass_of_correct_size(space, cls)
+    if SUBCLASSES_MIN_FIELDS == SUBCLASSES_MAX_FIELDS:
+        return classes[0]
     size = map.size_estimate()
-    if not size:
-        size = 1
-    try:
-        return classes[size - 1]
-    except IndexError:
-        return classes[-1]
+    debug.check_nonneg(size)
+    if size < len(classes):
+        return classes[size]
+    else:
+        return classes[len(classes)-1]
 get_subclass_of_correct_size._annspecialcase_ = "specialize:arg(1)"
 
-NUM_SUBCLASSES = 10 # XXX tweak this number
+SUBCLASSES_MIN_FIELDS = 5 # XXX tweak these numbers
+SUBCLASSES_MAX_FIELDS = 5
 
 def memo_get_subclass_of_correct_size(space, supercls):
     key = space, supercls
@@ -401,8 +408,12 @@
     except KeyError:
         assert not hasattr(supercls, "__del__")
         result = []
-        for i in range(1, NUM_SUBCLASSES+1):
+        for i in range(SUBCLASSES_MIN_FIELDS, SUBCLASSES_MAX_FIELDS+1):
             result.append(_make_subclass_size_n(supercls, i))
+        for i in range(SUBCLASSES_MIN_FIELDS):
+            result.insert(0, result[0])
+        if SUBCLASSES_MIN_FIELDS == SUBCLASSES_MAX_FIELDS:
+            assert len(set(result)) == 1
         _subclass_cache[key] = result
         return result
 memo_get_subclass_of_correct_size._annspecialcase_ = "specialize:memo"
@@ -413,7 +424,7 @@
     rangen = unroll.unrolling_iterable(range(n))
     nmin1 = n - 1
     rangenmin1 = unroll.unrolling_iterable(range(nmin1))
-    class subcls(ObjectMixin, BaseMapdictObject, supercls):
+    class subcls(BaseMapdictObject, supercls):
         def _init_empty(self, map):
             from pypy.rlib.debug import make_sure_not_resized
             for i in rangen:
@@ -506,8 +517,8 @@
     def impl_getitem_str(self, key):
         return self.w_obj.getdictvalue(self.space, key)
 
-    def impl_setitem_str(self,  key, w_value, shadows_type=True):
-        flag = self.w_obj.setdictvalue(self.space, key, w_value, shadows_type)
+    def impl_setitem_str(self,  key, w_value):
+        flag = self.w_obj.setdictvalue(self.space, key, w_value)
         assert flag
 
     def impl_setitem(self,  w_key, w_value):
@@ -588,8 +599,6 @@
 # ____________________________________________________________
 # Magic caching
 
-# XXX we also would like getdictvalue_attr_is_in_class() above
-
 class CacheEntry(object):
     map = None
     version_tag = None

Modified: pypy/trunk/pypy/objspace/std/objspace.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/objspace.py	(original)
+++ pypy/trunk/pypy/objspace/std/objspace.py	Wed Oct 27 16:20:26 2010
@@ -437,7 +437,7 @@
             if is_data:
                 w_get = self.lookup(w_descr, "__get__")
             if w_get is None:
-                w_value = w_obj.getdictvalue_attr_is_in_class(self, name)
+                w_value = w_obj.getdictvalue(self, name)
                 if w_value is not None:
                     return w_value
                 if not is_data:
@@ -489,14 +489,12 @@
             return w_obj.getitem(w_key)
         return ObjSpace.finditem(self, w_obj, w_key)
 
-    def setitem_str(self, w_obj, key, w_value, shadows_type=True):
+    def setitem_str(self, w_obj, key, w_value):
         """ Same as setitem, but takes string instead of any wrapped object
-
-        XXX what shadows_type means???
         """
         if (isinstance(w_obj, W_DictMultiObject) and
                 not w_obj.user_overridden_class):
-            w_obj.setitem_str(key, w_value, shadows_type)
+            w_obj.setitem_str(key, w_value)
         else:
             self.setitem(w_obj, self.wrap(key), w_value)
 

Modified: pypy/trunk/pypy/objspace/std/proxyobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/proxyobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/proxyobject.py	Wed Oct 27 16:20:26 2010
@@ -43,7 +43,7 @@
                     raise
                 return None
         
-        def setdictvalue(self, space, attr, w_value, shadows_type=True):
+        def setdictvalue(self, space, attr, w_value):
             try:
                 space.call_function(self.w_controller, space.wrap('__setattr__'),
                    space.wrap(attr), w_value)

Modified: pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py	Wed Oct 27 16:20:26 2010
@@ -4,7 +4,6 @@
      StrDictImplementation
 
 from pypy.objspace.std.celldict import ModuleDictImplementation
-from pypy.objspace.std.sharingdict import SharedDictImplementation
 from pypy.conftest import gettestobjspace
 
 
@@ -513,32 +512,6 @@
         assert getattr(a, s) == 42
 
 
-class TestW_DictSharing(TestW_DictObject):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withsharingdict": True})
-
-class AppTest_DictSharing(AppTest_DictObject):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withsharingdict": True})
-
-    def test_values_does_not_share(self):
-        class A(object):
-            pass
-        a = A()
-        a.abc = 12
-        l = a.__dict__.values()
-        assert l == [12]
-        l[0] = 24
-        assert a.abc == 12
-
-    def test_items(self):
-        class A(object):
-            pass
-        a = A()
-        a.abc = 12
-        a.__dict__.items() == [("abc", 12)]
-
-
 class AppTestModuleDict(object):
     def setup_class(cls):
         cls.space = gettestobjspace(**{"objspace.std.withcelldict": True})
@@ -632,10 +605,8 @@
     class objspace:
         class std:
             withdictmeasurement = False
-            withsharingdict = False
             withsmalldicts = False
             withcelldict = False
-            withshadowtracking = False
         class opcodes:
             CALL_LIKELY_BUILTIN = False
 
@@ -770,9 +741,6 @@
     string = "int"
     string2 = "isinstance"
 
-class TestSharedDictImplementation(BaseTestRDictImplementation):
-    ImplementionClass = SharedDictImplementation
-
 
 class BaseTestDevolvedDictImplementation(BaseTestRDictImplementation):
     def fill_impl(self):
@@ -794,8 +762,6 @@
     string = "int"
     string2 = "isinstance"
 
-class TestDevolvedSharedDictImplementation(BaseTestDevolvedDictImplementation):
-    ImplementionClass = SharedDictImplementation
 
 def test_module_uses_strdict():
     fakespace = FakeSpace()

Copied: pypy/trunk/pypy/objspace/std/test/test_methodcache.py (from r78358, pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_methodcache.py)
==============================================================================
--- pypy/branch/cleanup-dict-impl/pypy/objspace/std/test/test_methodcache.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_methodcache.py	Wed Oct 27 16:20:26 2010
@@ -29,10 +29,11 @@
 
     def test_class_that_cannot_be_cached(self):
         import __pypy__
-        class metatype(type):
+        class X:
             pass
-        class A(object):
-            __metaclass__ = metatype
+        class Y(object):
+            pass
+        class A(Y, X):
             def f(self):
                 return 42
 
@@ -130,3 +131,20 @@
             foo = 3
         D.__bases__ = (C, F)
         assert e.foo == 3
+
+    def test_custom_metaclass(self):
+        import __pypy__
+        class MetaA(type):
+            def __getattribute__(self, x):
+                return 1
+        def f(self):
+            return 42
+        A = type.__new__(MetaA, "A", (), {"f": f})
+        l = [type.__getattribute__(A, "__new__")(A)] * 10
+        __pypy__.reset_method_cache_counter()
+        for i, a in enumerate(l):
+            assert a.f() == 42
+        cache_counter = __pypy__.method_cache_counter("f")
+        assert cache_counter[0] >= 5
+        assert cache_counter[1] >= 1 # should be (27, 3)
+        assert sum(cache_counter) == 10


From arigo at codespeak.net  Wed Oct 27 16:21:46 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 16:21:46 +0200 (CEST)
Subject: [pypy-svn] r78361 - pypy/branch/cleanup-dict-impl
Message-ID: <20101027142146.91E14282C1C@codespeak.net>

Author: arigo
Date: Wed Oct 27 16:21:45 2010
New Revision: 78361

Removed:
   pypy/branch/cleanup-dict-impl/
Log:
(cfbolz, arigo)
Kill merged branch.



From arigo at codespeak.net  Wed Oct 27 16:22:41 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 16:22:41 +0200 (CEST)
Subject: [pypy-svn] r78362 - pypy/branch/mapdict-without-jit
Message-ID: <20101027142241.7E87C282C1C@codespeak.net>

Author: arigo
Date: Wed Oct 27 16:22:40 2010
New Revision: 78362

Added:
   pypy/branch/mapdict-without-jit/
      - copied from r78361, pypy/trunk/
Log:
(cfbolz, arigo)
A branch where we speed up mapdict if you don't use the JIT.



From arigo at codespeak.net  Wed Oct 27 16:47:56 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 16:47:56 +0200 (CEST)
Subject: [pypy-svn] r78364 - in
	pypy/branch/mapdict-without-jit/pypy/objspace/std: . test
Message-ID: <20101027144756.A15C4282C1C@codespeak.net>

Author: arigo
Date: Wed Oct 27 16:47:54 2010
New Revision: 78364

Modified:
   pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
   pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_mapdict.py
Log:
(cfbolz, arigo)
By tiny steps that all make sense, in the end we reach something
completely ridiculous.


Modified: pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	Wed Oct 27 16:47:54 2010
@@ -15,13 +15,14 @@
 # we want to propagate knowledge that the result cannot be negative
 
 class AbstractAttribute(object):
-    _immutable_fields_ = ['w_cls']
+    _immutable_fields_ = ['terminator']
     cache_attrs = None
     _size_estimate = 0
 
-    def __init__(self, space, w_cls):
+    def __init__(self, space, terminator):
         self.space = space
-        self.w_cls = w_cls
+        assert isinstance(terminator, Terminator)
+        self.terminator = terminator
 
     def read(self, obj, selector):
         raise NotImplementedError("abstract base class")
@@ -42,7 +43,7 @@
         raise NotImplementedError("abstract base class")
 
     def get_terminator(self):
-        raise NotImplementedError("abstract base class")
+        return self.terminator
 
     def set_terminator(self, obj, terminator):
         raise NotImplementedError("abstract base class")
@@ -95,10 +96,15 @@
         raise NotImplementedError("abstract base class")
 
     def __repr__(self):
-        return "<%s w_cls=%s>" % (self.__class__.__name__, self.w_cls)
+        return "<%s>" % (self.__class__.__name__,)
 
 
 class Terminator(AbstractAttribute):
+    _immutable_fields_ = ['w_cls']
+
+    def __init__(self, space, w_cls):
+        AbstractAttribute.__init__(self, space, self)
+        self.w_cls = w_cls
 
     def read(self, obj, selector):
         return None
@@ -116,9 +122,6 @@
     def length(self):
         return 0
 
-    def get_terminator(self):
-        return self
-
     def set_terminator(self, obj, terminator):
         result = Object()
         result.space = self.space
@@ -128,6 +131,9 @@
     def remove_dict_entries(self, obj):
         return self.copy(obj)
 
+    def __repr__(self):
+        return "<%s w_cls=%s>" % (self.__class__.__name__, self.w_cls)
+
 class DictTerminator(Terminator):
     _immutable_fields_ = ['devolved_dict_terminator']
     def __init__(self, space, w_cls):
@@ -189,7 +195,7 @@
 class PlainAttribute(AbstractAttribute):
     _immutable_fields_ = ['selector', 'position', 'back']
     def __init__(self, selector, back):
-        AbstractAttribute.__init__(self, back.space, back.w_cls)
+        AbstractAttribute.__init__(self, back.space, back.terminator)
         self.selector = selector
         self.position = back.length()
         self.back = back
@@ -232,9 +238,6 @@
     def length(self):
         return self.position + 1
 
-    def get_terminator(self):
-        return self.back.get_terminator()
-
     def set_terminator(self, obj, terminator):
         new_obj = self.back.set_terminator(obj, terminator)
         self._copy_attr(obj, new_obj)
@@ -328,7 +331,7 @@
         assert flag
 
     def getclass(self, space):
-        return self._get_mapdict_map().w_cls
+        return self._get_mapdict_map().terminator.w_cls
 
     def setclass(self, space, w_cls):
         new_obj = self._get_mapdict_map().set_terminator(self, w_cls.terminator)
@@ -609,7 +612,7 @@
 INVALID_CACHE_ENTRY = CacheEntry()
 INVALID_CACHE_ENTRY.map = objectmodel.instantiate(AbstractAttribute)
                              # different from any real map ^^^
-INVALID_CACHE_ENTRY.map.w_cls = None
+INVALID_CACHE_ENTRY.map.terminator = None
 
 def init_mapdict_cache(pycode):
     num_entries = len(pycode.co_names_w)
@@ -621,7 +624,7 @@
     entry = pycode._mapdict_caches[nameindex]
     map = w_obj._get_mapdict_map()
     if map is entry.map:
-        version_tag = map.w_cls.version_tag()
+        version_tag = map.terminator.w_cls.version_tag()
         if version_tag is entry.version_tag:
             # everything matches, it's incredibly fast
             if pycode.space.config.objspace.std.withmethodcachecounter:
@@ -634,7 +637,7 @@
     space = pycode.space
     w_name = pycode.co_names_w[nameindex]
     if map is not None:
-        w_type = map.w_cls
+        w_type = map.terminator.w_cls
         w_descr = w_type.getattribute_if_not_from_object()
         if w_descr is not None:
             return space._handle_getattribute(w_descr, w_obj, w_name)

Modified: pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_mapdict.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_mapdict.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_mapdict.py	Wed Oct 27 16:47:54 2010
@@ -30,7 +30,8 @@
                         PlainAttribute(("a", DICT),
                                        Terminator(space, w_cls)))
     assert aa.space is space
-    assert aa.w_cls is w_cls
+    assert aa.terminator.w_cls is w_cls
+    assert aa.get_terminator() is aa.terminator
 
     obj = Object()
     obj.map, obj.storage = aa, [10, 20]


From arigo at codespeak.net  Wed Oct 27 16:51:11 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 16:51:11 +0200 (CEST)
Subject: [pypy-svn] r78365 -
	pypy/branch/mapdict-without-jit/pypy/objspace/std
Message-ID: <20101027145111.9ADA736C230@codespeak.net>

Author: arigo
Date: Wed Oct 27 16:51:10 2010
New Revision: 78365

Modified:
   pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
Log:
(cfbolz, arigo)
Make read() use index().


Modified: pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	Wed Oct 27 16:51:10 2010
@@ -25,7 +25,10 @@
         self.terminator = terminator
 
     def read(self, obj, selector):
-        raise NotImplementedError("abstract base class")
+        index = self.index(selector)
+        if index < 0:
+            return self.terminator._read_terminator(obj, selector)
+        return obj._mapdict_read_storage(index)
 
     def write(self, obj, selector, w_value):
         raise NotImplementedError("abstract base class")
@@ -106,7 +109,7 @@
         AbstractAttribute.__init__(self, space, self)
         self.w_cls = w_cls
 
-    def read(self, obj, selector):
+    def _read_terminator(self, obj, selector):
         return None
 
     def write(self, obj, selector, w_value):
@@ -155,12 +158,12 @@
 
 
 class DevolvedDictTerminator(Terminator):
-    def read(self, obj, selector):
+    def _read_terminator(self, obj, selector):
         if selector[1] == DICT:
             w_dict = obj.getdict()
             space = self.space
             return space.finditem_str(w_dict, selector[0])
-        return Terminator.read(self, obj, selector)
+        return Terminator._read_terminator(self, obj, selector)
 
     def write(self, obj, selector, w_value):
         if selector[1] == DICT:
@@ -205,11 +208,6 @@
         w_value = self.read(obj, self.selector)
         new_obj._get_mapdict_map().add_attr(new_obj, self.selector, w_value)
 
-    def read(self, obj, selector):
-        if selector == self.selector:
-            return obj._mapdict_read_storage(self.position)
-        return self.back.read(obj, selector)
-
     def write(self, obj, selector, w_value):
         if selector == self.selector:
             obj._mapdict_write_storage(self.position, w_value)


From arigo at codespeak.net  Wed Oct 27 16:53:52 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 16:53:52 +0200 (CEST)
Subject: [pypy-svn] r78366 -
	pypy/branch/mapdict-without-jit/pypy/objspace/std
Message-ID: <20101027145352.CF27A36C230@codespeak.net>

Author: arigo
Date: Wed Oct 27 16:53:50 2010
New Revision: 78366

Modified:
   pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
Log:
(cfbolz, arigo)
Replace write() with a call to index().


Modified: pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	Wed Oct 27 16:53:50 2010
@@ -31,7 +31,11 @@
         return obj._mapdict_read_storage(index)
 
     def write(self, obj, selector, w_value):
-        raise NotImplementedError("abstract base class")
+        index = self.index(selector)
+        if index < 0:
+            return self.terminator._write_terminator(obj, selector, w_value)
+        obj._mapdict_write_storage(index, w_value)
+        return True
 
     def delete(self, obj, selector):
         return None
@@ -112,7 +116,7 @@
     def _read_terminator(self, obj, selector):
         return None
 
-    def write(self, obj, selector, w_value):
+    def _write_terminator(self, obj, selector, w_value):
         obj._get_mapdict_map().add_attr(obj, selector, w_value)
         return True
 
@@ -151,10 +155,10 @@
 
 
 class NoDictTerminator(Terminator):
-    def write(self, obj, selector, w_value):
+    def _write_terminator(self, obj, selector, w_value):
         if selector[1] == DICT:
             return False
-        return Terminator.write(self, obj, selector, w_value)
+        return Terminator._write_terminator(self, obj, selector, w_value)
 
 
 class DevolvedDictTerminator(Terminator):
@@ -165,13 +169,13 @@
             return space.finditem_str(w_dict, selector[0])
         return Terminator._read_terminator(self, obj, selector)
 
-    def write(self, obj, selector, w_value):
+    def _write_terminator(self, obj, selector, w_value):
         if selector[1] == DICT:
             w_dict = obj.getdict()
             space = self.space
             space.setitem_str(w_dict, selector[0], w_value)
             return True
-        return Terminator.write(self, obj, selector, w_value)
+        return Terminator._write_terminator(self, obj, selector, w_value)
 
     def delete(self, obj, selector):
         from pypy.interpreter.error import OperationError
@@ -208,12 +212,6 @@
         w_value = self.read(obj, self.selector)
         new_obj._get_mapdict_map().add_attr(new_obj, self.selector, w_value)
 
-    def write(self, obj, selector, w_value):
-        if selector == self.selector:
-            obj._mapdict_write_storage(self.position, w_value)
-            return True
-        return self.back.write(obj, selector, w_value)
-
     def delete(self, obj, selector):
         if selector == self.selector:
             # ok, attribute is deleted


From fijal at codespeak.net  Wed Oct 27 17:17:34 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Wed, 27 Oct 2010 17:17:34 +0200 (CEST)
Subject: [pypy-svn] r78368 - pypy/extradoc/talk/pycon2011
Message-ID: <20101027151734.41AB25080C@codespeak.net>

Author: fijal
Date: Wed Oct 27 17:17:31 2010
New Revision: 78368

Added:
   pypy/extradoc/talk/pycon2011/
   pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
      - copied unchanged from r78367, user/agaynor/pycon-2011/pypy-optimizations.txt
Log:
(agaynor, fijal) create dir and import draft from agaynor's home


From arigo at codespeak.net  Wed Oct 27 17:47:57 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 17:47:57 +0200 (CEST)
Subject: [pypy-svn] r78370 - in pypy/branch/mapdict-without-jit/pypy:
	module/__pypy__ module/gc module/gc/test objspace/std
	objspace/std/test
Message-ID: <20101027154757.EBDFC282C1D@codespeak.net>

Author: arigo
Date: Wed Oct 27 17:47:56 2010
New Revision: 78370

Modified:
   pypy/branch/mapdict-without-jit/pypy/module/__pypy__/__init__.py
   pypy/branch/mapdict-without-jit/pypy/module/__pypy__/interp_magic.py
   pypy/branch/mapdict-without-jit/pypy/module/gc/interp_gc.py
   pypy/branch/mapdict-without-jit/pypy/module/gc/test/test_gc.py
   pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
   pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_dictmultiobject.py
   pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_mapdict.py
Log:
(cfbolz, arigo)
Add the IndexCache to mapdict, similar to MethodCache.


Modified: pypy/branch/mapdict-without-jit/pypy/module/__pypy__/__init__.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/module/__pypy__/__init__.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/module/__pypy__/__init__.py	Wed Oct 27 17:47:56 2010
@@ -23,5 +23,8 @@
                                  'interp_magic.method_cache_counter')
             self.extra_interpdef('reset_method_cache_counter',
                                  'interp_magic.reset_method_cache_counter')
+            if self.space.config.objspace.std.withmapdict:
+                self.extra_interpdef('mapdict_cache_counter',
+                                     'interp_magic.mapdict_cache_counter')
         PYC_MAGIC = get_pyc_magic(self.space)
         self.extra_interpdef('PYC_MAGIC', 'space.wrap(%d)' % PYC_MAGIC)

Modified: pypy/branch/mapdict-without-jit/pypy/module/__pypy__/interp_magic.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/module/__pypy__/interp_magic.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/module/__pypy__/interp_magic.py	Wed Oct 27 17:47:56 2010
@@ -2,6 +2,7 @@
 from pypy.interpreter.gateway import ObjSpace
 from pypy.rlib.objectmodel import we_are_translated
 from pypy.objspace.std.typeobject import MethodCache
+from pypy.objspace.std.mapdict import IndexCache
 
 def internal_repr(space, w_object):
     return space.wrap('%r' % (w_object,))
@@ -36,4 +37,17 @@
     cache = space.fromcache(MethodCache)
     cache.misses = {}
     cache.hits = {}
-
+    if space.config.objspace.std.withmapdict:
+        cache = space.fromcache(IndexCache)
+        cache.misses = {}
+        cache.hits = {}
+
+def mapdict_cache_counter(space, name):
+    """Return a tuple (index_cache_hits, index_cache_misses) for lookups
+    in the mapdict cache with the given attribute name."""
+    assert space.config.objspace.std.withmethodcachecounter
+    assert space.config.objspace.std.withmapdict
+    cache = space.fromcache(IndexCache)
+    return space.newtuple([space.newint(cache.hits.get(name, 0)),
+                           space.newint(cache.misses.get(name, 0))])
+mapdict_cache_counter.unwrap_spec = [ObjSpace, str]

Modified: pypy/branch/mapdict-without-jit/pypy/module/gc/interp_gc.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/module/gc/interp_gc.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/module/gc/interp_gc.py	Wed Oct 27 17:47:56 2010
@@ -10,6 +10,10 @@
         from pypy.objspace.std.typeobject import MethodCache
         cache = space.fromcache(MethodCache)
         cache.clear()
+        if space.config.objspace.std.withmapdict:
+            from pypy.objspace.std.mapdict import IndexCache
+            cache = space.fromcache(IndexCache)
+            cache.clear()
     rgc.collect()
     return space.wrap(0)
     

Modified: pypy/branch/mapdict-without-jit/pypy/module/gc/test/test_gc.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/module/gc/test/test_gc.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/module/gc/test/test_gc.py	Wed Oct 27 17:47:56 2010
@@ -122,3 +122,8 @@
         gc.collect()    # the classes C should all go away here
         for r in rlist:
             assert r() is None
+
+class AppTestGcMapDictIndexCache(AppTestGcMethodCache):
+    def setup_class(cls):
+        cls.space = gettestobjspace(**{"objspace.std.withmethodcache": True,
+                                       "objspace.std.withmapdict": True})

Modified: pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	Wed Oct 27 17:47:56 2010
@@ -1,4 +1,5 @@
 from pypy.rlib import jit, objectmodel, debug
+from pypy.rlib.rarithmetic import intmask, r_uint
 
 from pypy.interpreter.baseobjspace import W_Root
 from pypy.objspace.std.dictmultiobject import W_DictMultiObject
@@ -40,7 +41,44 @@
     def delete(self, obj, selector):
         return None
 
+    @jit.purefunction
     def index(self, selector):
+        if self.space.config.objspace.std.withmethodcache:
+            return self._index_cache(selector)
+        else:
+            return self._index(selector)
+
+    def _index_cache(self, selector):
+        space = self.space
+        cache = space.fromcache(IndexCache)
+        SHIFT2 = r_uint.BITS - space.config.objspace.std.methodcachesizeexp
+        SHIFT1 = SHIFT2 - 5
+        attrs_as_int = objectmodel.current_object_addr_as_int(self)
+        # ^^^Note: see comment in typeobject.py for
+        # _pure_lookup_where_with_method_cache()
+        hash_selector = objectmodel.compute_hash(selector)
+        product = intmask(attrs_as_int * hash_selector)
+        index_hash = (r_uint(product) ^ (r_uint(product) << SHIFT1)) >> SHIFT2
+        # ^^^Note2: same comment too
+        cached_attr = cache.attrs[index_hash]
+        if cached_attr is self:
+            cached_selector = cache.selectors[index_hash]
+            if cached_selector == selector:
+                index = cache.indices[index_hash]
+                if space.config.objspace.std.withmethodcachecounter:
+                    name = selector[0]
+                    cache.hits[name] = cache.hits.get(name, 0) + 1
+                return index
+        index = self._index(selector)
+        cache.attrs[index_hash] = self
+        cache.selectors[index_hash] = selector
+        cache.indices[index_hash] = index
+        if space.config.objspace.std.withmethodcachecounter:
+            name = selector[0]
+            cache.misses[name] = cache.misses.get(name, 0) + 1
+        return index
+
+    def _index(self, selector):
         return -1
 
     def copy(self, obj):
@@ -221,10 +259,10 @@
             self._copy_attr(obj, new_obj)
         return new_obj
 
-    def index(self, selector):
+    def _index(self, selector):
         if selector == self.selector:
             return self.position
-        return self.back.index(selector)
+        return self.back._index(selector)
 
     def copy(self, obj):
         new_obj = self.back.copy(obj)
@@ -267,6 +305,23 @@
     # RPython reasons
     w_obj._set_mapdict_storage_and_map(new_obj.storage, new_obj.map)
 
+class IndexCache(object):
+    def __init__(self, space):
+        assert space.config.objspace.std.withmethodcache
+        SIZE = 1 << space.config.objspace.std.methodcachesizeexp
+        self.attrs = [None] * SIZE
+        self.selectors = [None] * SIZE
+        self.indices = [0] * SIZE
+        if space.config.objspace.std.withmethodcachecounter:
+            self.hits = {}
+            self.misses = {}
+
+    def clear(self):
+        for i in range(len(self.attrs)):
+            self.attrs[i] = None
+        for i in range(len(self.selectors)):
+            self.selectors[i] = None
+
 # ____________________________________________________________
 # object implementation
 

Modified: pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_dictmultiobject.py	Wed Oct 27 17:47:56 2010
@@ -607,6 +607,7 @@
             withdictmeasurement = False
             withsmalldicts = False
             withcelldict = False
+            withmethodcache = False
         class opcodes:
             CALL_LIKELY_BUILTIN = False
 

Modified: pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_mapdict.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_mapdict.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/objspace/std/test/test_mapdict.py	Wed Oct 27 17:47:56 2010
@@ -24,7 +24,6 @@
         hasdict = False
 
 def test_plain_attribute():
-    space = " "
     w_cls = "class"
     aa = PlainAttribute(("b", DICT),
                         PlainAttribute(("a", DICT),
@@ -778,3 +777,52 @@
         assert res == (0, 0, 1)
         res = self.check(f, 'x')
         assert res == (0, 0, 1)
+
+class AppTestCaching(AppTestWithMapDict):
+    def setup_class(cls):
+        cls.space = gettestobjspace(
+            **{"objspace.std.withmethodcachecounter": True,
+               "objspace.std.withmapdict": True,
+               "objspace.opcodes.CALL_METHOD": True})
+
+    def test_mix_classes(self):
+        import __pypy__
+        class A(object):
+            def f(self):
+                return 42
+        class B(object):
+            def f(self):
+                return 43
+        class C(object):
+            def f(self):
+                return 44
+        l = [A(), B(), C()] * 10
+        __pypy__.reset_method_cache_counter()
+        # 'exec' to make sure that a.f() is compiled with CALL_METHOD
+        exec """for i, a in enumerate(l):
+                    assert a.f() == 42 + i % 3
+"""
+        cache_counter = __pypy__.mapdict_cache_counter("f")
+        assert cache_counter[0] >= 15
+        assert cache_counter[1] >= 3 # should be (27, 3)
+        assert sum(cache_counter) == 30
+
+    def test_mix_classes_attribute(self):
+        import __pypy__
+        class A(object):
+            def __init__(self):
+                self.x = 42
+        class B(object):
+            def __init__(self):
+                self.x = 43
+        class C(object):
+            def __init__(self):
+                self.x = 44
+        l = [A(), B(), C()] * 10
+        __pypy__.reset_method_cache_counter()
+        for i, a in enumerate(l):
+            assert a.x == 42 + i % 3
+        cache_counter = __pypy__.mapdict_cache_counter("x")
+        assert cache_counter[0] >= 15
+        assert cache_counter[1] >= 3 # should be (27, 3)
+        assert sum(cache_counter) == 30


From arigo at codespeak.net  Wed Oct 27 18:05:51 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 18:05:51 +0200 (CEST)
Subject: [pypy-svn] r78372 -
	pypy/branch/mapdict-without-jit/pypy/objspace/std
Message-ID: <20101027160551.26FCA5080C@codespeak.net>

Author: arigo
Date: Wed Oct 27 18:05:48 2010
New Revision: 78372

Modified:
   pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
Log:
Translation fix

Modified: pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	Wed Oct 27 18:05:48 2010
@@ -310,7 +310,8 @@
         assert space.config.objspace.std.withmethodcache
         SIZE = 1 << space.config.objspace.std.methodcachesizeexp
         self.attrs = [None] * SIZE
-        self.selectors = [None] * SIZE
+        self._empty_selector = (None, INVALID)
+        self.selectors = [self._empty_selector] * SIZE
         self.indices = [0] * SIZE
         if space.config.objspace.std.withmethodcachecounter:
             self.hits = {}
@@ -320,7 +321,7 @@
         for i in range(len(self.attrs)):
             self.attrs[i] = None
         for i in range(len(self.selectors)):
-            self.selectors[i] = None
+            self.selectors[i] = self._empty_selector
 
 # ____________________________________________________________
 # object implementation


From fijal at codespeak.net  Wed Oct 27 19:18:01 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Wed, 27 Oct 2010 19:18:01 +0200 (CEST)
Subject: [pypy-svn] r78373 - pypy/extradoc/talk/pycon2011
Message-ID: <20101027171801.E0F23282C1D@codespeak.net>

Author: fijal
Date: Wed Oct 27 19:17:59 2010
New Revision: 78373

Modified:
   pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
Log:
(agaynor, fijal) improve

Modified: pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
==============================================================================
--- pypy/extradoc/talk/pycon2011/pypy-optimizations.txt	(original)
+++ pypy/extradoc/talk/pycon2011/pypy-optimizations.txt	Wed Oct 27 19:17:59 2010
@@ -7,10 +7,37 @@
 Classification: Discuss in depth
 Abstract:
 
-This talk is 3 parts, what's hard to optimize about Python, how PyPy is organized, and what optimizations our JIT can do.
+The talk would contain a list of features that are hard to optimize
+in Python, why and how we're fixing it in PyPy. This also might be
+used as a rough guide what features in Python are optimizable right
+now and what are not.
+
+The list of mentioned features is not exhaustive, however we will try to
+focus at least on the following:
+
+* Dynamic language - In Python code we have no known types, like a
+  static language.  Even operations like "a + b" can do anything,
+  unless we know more about the code.
+
+* Frame introspection - Frame objects need to be allocated for every
+  function call, and all local variables are stored on the frame, and
+  must be accessible from further down the call stack.
 
+* Object model - All user defined Python objects have a dictionary which
+  stores their attributes, as does every type.  When Python does an
+  attribute lookup this requires 2 dictionary lookups.
+
+  In PyPy we use an approach similar to one used by v8 with hidden classes
+  (except more pypy specific) called shared dictionaries.
+
+* FFI calls - calling C from Python is costly and hard to optimize.
+  In PyPy we decided to go via ctypes, this part will explain how are
+  we going to handle ctypes calls to be fast.
+
+This talk is 3 parts, what's hard to optimize about Python, how PyPy is organized, and what optimizations our JIT can do.
+ 
 I. Why Python is hard (8 minutes)
-    A) Dynamic language
+    A) Dynamic language - No known types, even operations such as "+" can
     B) Frame introspection.
     C) Object model.
     D) CPython


From afa at codespeak.net  Wed Oct 27 19:53:31 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 27 Oct 2010 19:53:31 +0200 (CEST)
Subject: [pypy-svn] r78374 - in pypy/branch/fast-forward/pypy/objspace/std:
	. test
Message-ID: <20101027175331.42054282BE0@codespeak.net>

Author: afa
Date: Wed Oct 27 19:53:29 2010
New Revision: 78374

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
Log:
Slice assignment cannot change the size in RPython.
unfortunately, this is only true when translated...


Modified: pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/bytearrayobject.py	Wed Oct 27 19:53:29 2010
@@ -397,9 +397,31 @@
     if step != 1:
         raise OperationError(space.w_NotImplementedError,
                              space.wrap("fixme: only step=1 for the moment"))
+    _setitem_helper(w_bytearray, start, stop, slicelength,
+                    space.str_w(w_other))
+
+def _setitem_helper(w_bytearray, start, stop, slicelength, data):
     assert start >= 0
     assert stop >= 0
-    w_bytearray.data[start:stop] = [c for c in space.str_w(w_other)]
+    step = 1
+    len2 = len(data)
+    delta = slicelength - len2
+    if delta < 0:
+        delta = -delta
+        newsize = len(w_bytearray.data) + delta
+        w_bytearray.data += ['\0'] * delta
+        lim = start + len2
+        i = newsize - 1
+        while i >= lim:
+            w_bytearray.data[i] = w_bytearray.data[i-delta]
+            i -= 1
+    elif start >= 0:
+        del w_bytearray.data[start:start+delta]
+    else:
+        assert delta == 0
+    for i in range(len2):
+        w_bytearray.data[start] = data[i]
+        start += step
 
 # __________________________________________________________
 # Buffer interface

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_bytes.py	Wed Oct 27 19:53:29 2010
@@ -175,6 +175,8 @@
         assert b == 'ABC...defghi'
         b[3:6] = '()'
         assert b == 'ABC()defghi'
+        b[6:6] = '<<'
+        assert b == 'ABC()d<

Author: afa
Date: Wed Oct 27 19:54:43 2010
New Revision: 78375

Added:
   pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py   (contents, props changed)
Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py
Log:
Start to implement _io.BytesIO


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Wed Oct 27 19:54:43 2010
@@ -17,6 +17,29 @@
     pass
 
 class W_BufferedIOBase(W_IOBase):
+    def _unsupportedoperation(self, space, message):
+        w_exc = space.getattr(space.getbuiltinmodule('_io'),
+                              space.wrap('UnsupportedOperation'))
+        raise OperationError(w_exc, space.wrap(message))
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def read_w(self, space, w_size=None):
+        self._unsupportedoperation(space, "read")
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def write_w(self, space, w_data):
+        self._unsupportedoperation(space, "write")
+
+W_BufferedIOBase.typedef = TypeDef(
+    '_BufferedIOBase', W_IOBase.typedef,
+    __new__ = generic_new_descr(W_BufferedIOBase),
+    read = interp2app(W_BufferedIOBase.read_w),
+    write = interp2app(W_BufferedIOBase.write_w),
+    )
+
+class BufferedMixin:
+    _mixin_ = True
+
     def __init__(self, space):
         W_IOBase.__init__(self, space)
         self.state = STATE_ZERO
@@ -37,19 +60,6 @@
         self.readable = False
         self.writable = False
 
-    def _unsupportedoperation(self, space, message):
-        w_exc = space.getattr(space.getbuiltinmodule('_io'),
-                              space.wrap('UnsupportedOperation'))
-        raise OperationError(w_exc, space.wrap(message))
-
-    @unwrap_spec('self', ObjSpace, W_Root)
-    def read_w(self, space, w_size=None):
-        self._unsupportedoperation(space, "read")
-
-    @unwrap_spec('self', ObjSpace, W_Root)
-    def write_w(self, space, w_size=None):
-        self._unsupportedoperation(space, "write")
-
     def _reader_reset_buf(self):
         self.read_end = -1
 
@@ -57,16 +67,6 @@
         self.write_pos = 0
         self.write_end = -1
 
-W_BufferedIOBase.typedef = TypeDef(
-    '_BufferedIOBase', W_IOBase.typedef,
-    __new__ = generic_new_descr(W_BufferedIOBase),
-    read = interp2app(W_BufferedIOBase.read_w),
-    write = interp2app(W_BufferedIOBase.write_w),
-    )
-
-class BufferedMixin:
-    _mixin_ = True
-
     def _init(self, space):
         if self.buffer_size <= 0:
             raise OperationError(space.w_ValueError, space.wrap(
@@ -245,11 +245,6 @@
         return self._write(space, ''.join(l))
 
 class W_BufferedReader(BufferedMixin, W_BufferedIOBase):
-    def __init__(self, space):
-        W_BufferedIOBase.__init__(self, space)
-        self.ok = False
-        self.detached = False
-
     @unwrap_spec('self', ObjSpace, W_Root, int)
     def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE):
         self.state = STATE_ZERO

Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py	Wed Oct 27 19:54:43 2010
@@ -1,11 +1,97 @@
-from pypy.module._io.interp_bufferedio import W_BufferedIOBase
 from pypy.interpreter.typedef import (
     TypeDef, generic_new_descr)
+from pypy.interpreter.gateway import interp2app, unwrap_spec
+from pypy.interpreter.baseobjspace import ObjSpace, W_Root
+from pypy.module._io.interp_bufferedio import W_BufferedIOBase
+from pypy.module._io.interp_iobase import convert_size
+
+def buffer2string(buffer, start, end):
+    from pypy.rlib.rstring import StringBuilder
+    builder = StringBuilder(end - start)
+    for i in range(start, end):
+        builder.append(buffer[i])
+    return builder.build()
 
 class W_BytesIO(W_BufferedIOBase):
-    pass
+    def __init__(self, space):
+        W_BufferedIOBase.__init__(self, space)
+        self.pos = 0
+        self.string_size = 0
+        self.buf = []
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def descr_init(self, space, w_initvalue=None):
+        # In case __init__ is called multiple times
+        self.string_size = 0
+        self.pos = 0
+
+        if not space.is_w(w_initvalue, space.w_None):
+            self.write_w(space, w_initvalue)
+            self.pos = 0
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def read_w(self, space, w_size=None):
+        self._check_closed(space)
+        size = convert_size(space, w_size)
+
+        # adjust invalid sizes
+        n = self.string_size - self.pos
+        if not 0 <= n <= size:
+            size = n
+            if size < 0:
+                size = 0
+
+        output = buffer2string(self.buf, self.pos, self.pos + size)
+        self.pos += size
+        return space.wrap(output)
+
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def write_w(self, space, w_data):
+        self._check_closed(space)
+        buf = space.buffer_w(w_data)
+        length = buf.getlength()
+        if length == 0:
+            return
+
+        if self.pos + length > len(self.buf):
+            self.buf.extend(['\0'] * (self.pos + length - len(self.buf)))
+
+        if self.pos > self.string_size:
+            # In case of overseek, pad with null bytes the buffer region
+            # between the end of stream and the current position.
+            #
+            # 0   lo      string_size                           hi
+            # |   |<---used--->|<----------available----------->|
+            # |   |            <--to pad-->|<---to write--->    |
+            # 0   buf                   position
+            for i in range(self.string_size, self.pos):
+                self.buf[i] = '\0'
+
+        # Copy the data to the internal buffer, overwriting some of the
+        # existing data if self->pos < self->string_size.
+        for i in range(length):
+            self.buf[self.pos + i] = buf.getitem(i)
+        self.pos += length
+
+        # Set the new length of the internal string if it has changed
+        if self.string_size < self.pos:
+            self.string_size = self.pos
+
+        return space.wrap(length)
+
+    @unwrap_spec('self', ObjSpace)
+    def getvalue_w(self, space):
+        self._check_closed(space)
+        return space.wrap(buffer2string(self.buf, 0, self.string_size))
+
+
 W_BytesIO.typedef = TypeDef(
     'BytesIO', W_BufferedIOBase.typedef,
     __new__ = generic_new_descr(W_BytesIO),
+    __init__  = interp2app(W_BytesIO.descr_init),
+
+    read = interp2app(W_BytesIO.read_w),
+    write = interp2app(W_BytesIO.write_w),
+    getvalue = interp2app(W_BytesIO.getvalue_w),
     )
 

Added: pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py
==============================================================================
--- (empty file)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py	Wed Oct 27 19:54:43 2010
@@ -0,0 +1,25 @@
+from pypy.conftest import gettestobjspace
+
+class AppTestBytesIO:
+    def setup_class(cls):
+        cls.space = gettestobjspace(usemodules=['_io'])
+
+    def test_write(self):
+        import _io
+        f = _io.BytesIO()
+        assert f.write("hello") == 5
+        import gc; gc.collect()
+        assert f.getvalue() == "hello"
+        f.close()
+
+    def test_read(self):
+        import _io
+        f = _io.BytesIO("hello")
+        assert f.read() == "hello"
+        import gc; gc.collect()
+        f.close()
+
+    ## def test_seek(self):
+    ##     import _io
+    ##     f = _io.BytesIO("hello")
+    ##     assert f.seek(-1, 2) == 4


From afa at codespeak.net  Wed Oct 27 19:56:19 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 27 Oct 2010 19:56:19 +0200 (CEST)
Subject: [pypy-svn] r78376 - pypy/branch/fast-forward/pypy/translator/c/src
Message-ID: <20101027175619.9DA945080C@codespeak.net>

Author: afa
Date: Wed Oct 27 19:56:18 2010
New Revision: 78376

Modified:
   pypy/branch/fast-forward/pypy/translator/c/src/debug_alloc.h
Log:
Fix compilation on Windows, when debug_alloc is enabled


Modified: pypy/branch/fast-forward/pypy/translator/c/src/debug_alloc.h
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/c/src/debug_alloc.h	(original)
+++ pypy/branch/fast-forward/pypy/translator/c/src/debug_alloc.h	Wed Oct 27 19:56:18 2010
@@ -62,8 +62,8 @@
     count++;
   if (count > 0)
     {
-      fprintf(stderr, "debug_alloc.h: %ld mallocs left", count);
       char *env = getenv("PYPY_ALLOC");
+      fprintf(stderr, "debug_alloc.h: %ld mallocs left", count);
       if (env && *env)
         {
           fprintf(stderr, " (most recent first):\n");


From arigo at codespeak.net  Wed Oct 27 19:57:18 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Wed, 27 Oct 2010 19:57:18 +0200 (CEST)
Subject: [pypy-svn] r78377 - pypy/trunk/pypy/objspace/std
Message-ID: <20101027175718.7EE4E282BE0@codespeak.net>

Author: arigo
Date: Wed Oct 27 19:57:16 2010
New Revision: 78377

Modified:
   pypy/trunk/pypy/objspace/std/listobject.py
Log:
Document this assert (thanks amaury).


Modified: pypy/trunk/pypy/objspace/std/listobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/listobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/listobject.py	Wed Oct 27 19:57:16 2010
@@ -283,7 +283,7 @@
         elif start >= 0:
             del items[start:start+delta]
         else:
-            assert delta==0
+            assert delta==0   # start<0 is only possible with slicelength==0
     elif len2 != slicelength:  # No resize for extended slices
         raise operationerrfmt(space.w_ValueError, "attempt to "
               "assign sequence of size %d to extended slice of size %d",


From afa at codespeak.net  Wed Oct 27 23:20:51 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Wed, 27 Oct 2010 23:20:51 +0200 (CEST)
Subject: [pypy-svn] r78380 -
	pypy/branch/fast-forward/pypy/translator/goal/test2
Message-ID: <20101027212051.6E52A5080C@codespeak.net>

Author: afa
Date: Wed Oct 27 23:20:49 2010
New Revision: 78380

Modified:
   pypy/branch/fast-forward/pypy/translator/goal/test2/test_app_main.py
Log:
Fix test when PYTHONPATH is not set


Modified: pypy/branch/fast-forward/pypy/translator/goal/test2/test_app_main.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/goal/test2/test_app_main.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/goal/test2/test_app_main.py	Wed Oct 27 23:20:49 2010
@@ -489,7 +489,8 @@
                 yield
             finally:
                 old_cwd.chdir()
-                os.putenv('PYTHONPATH', old_pythonpath)
+                if old_pythonpath is not None:
+                    os.putenv('PYTHONPATH', old_pythonpath)
         
         tmpdir.join('site.py').write('print "SHOULD NOT RUN"')
         runme_py = tmpdir.join('runme.py')


From afa at codespeak.net  Thu Oct 28 00:10:10 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 00:10:10 +0200 (CEST)
Subject: [pypy-svn] r78381 - in
	pypy/branch/fast-forward/pypy/translator/goal: . test2
Message-ID: <20101027221010.AD52B36C1FD@codespeak.net>

Author: afa
Date: Thu Oct 28 00:10:04 2010
New Revision: 78381

Modified:
   pypy/branch/fast-forward/pypy/translator/goal/app_main.py
   pypy/branch/fast-forward/pypy/translator/goal/test2/test_app_main.py
Log:
The command::
   python -i -c "import sys; sys.exit(1)"
- must display the SystemExit traceback
- must enter interactive mode


Modified: pypy/branch/fast-forward/pypy/translator/goal/app_main.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/goal/app_main.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/goal/app_main.py	Thu Oct 28 00:10:04 2010
@@ -21,6 +21,25 @@
 
 originalexcepthook = sys.__excepthook__
 
+def handle_sys_exit(e):
+    # exit if we catch a w_SystemExit
+    exitcode = e.code
+    if exitcode is None:
+        exitcode = 0
+    else:
+        try:
+            exitcode = int(exitcode)
+        except:
+            # not an integer: print it to stderr
+            try:
+                stderr = sys.stderr
+            except AttributeError:
+                pass   # too bad
+            else:
+                print >> stderr, exitcode
+            exitcode = 1
+    raise SystemExit(exitcode)
+
 def run_toplevel(f, *fargs, **fkwds):
     """Calls f() and handles all OperationErrors.
     Intended use is to run the main program or one interactive statement.
@@ -44,67 +63,53 @@
                 stdout.write('\n')
 
     except SystemExit, e:
-        # exit if we catch a w_SystemExit
-        exitcode = e.code
-        if exitcode is None:
-            exitcode = 0
-        else:
-            try:
-                exitcode = int(exitcode)
-            except:
-                # not an integer: print it to stderr
-                try:
-                    stderr = sys.stderr
-                except AttributeError:
-                    pass   # too bad
-                else:
-                    print >> stderr, exitcode
-                exitcode = 1
-        raise SystemExit(exitcode)
+        handle_sys_exit(e)
+    except:
+        display_exception()
+        return False
+    return True   # success
+
+def display_exception():
+    etype, evalue, etraceback = sys.exc_info()
+    try:
+        # extra debugging info in case the code below goes very wrong
+        if DEBUG and hasattr(sys, 'stderr'):
+            s = getattr(etype, '__name__', repr(etype))
+            print >> sys.stderr, "debug: exception-type: ", s
+            print >> sys.stderr, "debug: exception-value:", str(evalue)
+            tbentry = etraceback
+            if tbentry:
+                while tbentry.tb_next:
+                    tbentry = tbentry.tb_next
+                lineno = tbentry.tb_lineno
+                filename = tbentry.tb_frame.f_code.co_filename
+                print >> sys.stderr, "debug: exception-tb:    %s:%d" % (
+                    filename, lineno)
+
+        # set the sys.last_xxx attributes
+        sys.last_type = etype
+        sys.last_value = evalue
+        sys.last_traceback = etraceback
+
+        # call sys.excepthook
+        hook = getattr(sys, 'excepthook', originalexcepthook)
+        hook(etype, evalue, etraceback)
+        return # done
 
     except:
-        etype, evalue, etraceback = sys.exc_info()
         try:
-            # extra debugging info in case the code below goes very wrong
-            if DEBUG and hasattr(sys, 'stderr'):
-                s = getattr(etype, '__name__', repr(etype))
-                print >> sys.stderr, "debug: exception-type: ", s
-                print >> sys.stderr, "debug: exception-value:", str(evalue)
-                tbentry = etraceback
-                if tbentry:
-                    while tbentry.tb_next:
-                        tbentry = tbentry.tb_next
-                    lineno = tbentry.tb_lineno
-                    filename = tbentry.tb_frame.f_code.co_filename
-                    print >> sys.stderr, "debug: exception-tb:    %s:%d" % (
-                        filename, lineno)
-
-            # set the sys.last_xxx attributes
-            sys.last_type = etype
-            sys.last_value = evalue
-            sys.last_traceback = etraceback
-
-            # call sys.excepthook
-            hook = getattr(sys, 'excepthook', originalexcepthook)
-            hook(etype, evalue, etraceback)
-            return False   # done
-
-        except:
-            try:
-                stderr = sys.stderr
-            except AttributeError:
-                pass   # too bad
-            else:
-                print >> stderr, 'Error calling sys.excepthook:'
-                originalexcepthook(*sys.exc_info())
-                print >> stderr
-                print >> stderr, 'Original exception was:'
+            stderr = sys.stderr
+        except AttributeError:
+            pass   # too bad
+        else:
+            print >> stderr, 'Error calling sys.excepthook:'
+            originalexcepthook(*sys.exc_info())
+            print >> stderr
+            print >> stderr, 'Original exception was:'
 
-        # we only get here if sys.excepthook didn't do its job
-        originalexcepthook(etype, evalue, etraceback)
-        return False
+    # we only get here if sys.excepthook didn't do its job
+    originalexcepthook(etype, evalue, etraceback)
 
-    return True   # success
 
 # ____________________________________________________________
 # Option parsing
@@ -481,14 +486,25 @@
             sys.path.insert(0, scriptdir)
             success = run_toplevel(execfile, sys.argv[0], mainmodule.__dict__)
 
-        # start a prompt if requested
+    except SystemExit, e:
+        status = e.code
         if inspect_requested():
+            display_exception()
+    else:
+        status = not success
+
+    # start a prompt if requested
+    if inspect_requested():
+        inteactive = False
+        try:
             from _pypy_interact import interactive_console
             success = run_toplevel(interactive_console, mainmodule)
-    except SystemExit, e:
-        return e.code
-    else:
-        return not success
+        except SystemExit, e:
+            status = e.code
+        else:
+            status = not success
+
+    return status
 
 def resolvedirof(filename):
     try:

Modified: pypy/branch/fast-forward/pypy/translator/goal/test2/test_app_main.py
==============================================================================
--- pypy/branch/fast-forward/pypy/translator/goal/test2/test_app_main.py	(original)
+++ pypy/branch/fast-forward/pypy/translator/goal/test2/test_app_main.py	Thu Oct 28 00:10:04 2010
@@ -241,6 +241,11 @@
         child.sendline('"pypy.translator.goal.test2.mymodule" in sys.modules')
         child.expect('False')
 
+    def test_option_i_noexit(self):
+        child = self.spawn(['-i', '-c', 'import sys; sys.exit(1)'])
+        child.expect('Traceback')
+        child.expect('SystemExit: 1')
+
     def test_options_u_i(self):
         if sys.platform == "win32":
             skip("close_fds is not supported on Windows platforms")


From afa at codespeak.net  Thu Oct 28 01:54:14 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 01:54:14 +0200 (CEST)
Subject: [pypy-svn] r78382 - in pypy/branch/fast-forward/pypy/module/rctime:
	. test
Message-ID: <20101027235414.4C310282BE0@codespeak.net>

Author: afa
Date: Thu Oct 28 01:54:11 2010
New Revision: 78382

Modified:
   pypy/branch/fast-forward/pypy/module/rctime/app_time.py
   pypy/branch/fast-forward/pypy/module/rctime/test/test_rctime.py
Log:
Ensure that time.strptime() returns a time structure

(the _strptime module changed in 2.7 to return an additional value)


Modified: pypy/branch/fast-forward/pypy/module/rctime/app_time.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/rctime/app_time.py	(original)
+++ pypy/branch/fast-forward/pypy/module/rctime/app_time.py	Thu Oct 28 01:54:11 2010
@@ -25,7 +25,7 @@
     (same as strftime())."""
 
     import _strptime     # from the CPython standard library
-    return _strptime._strptime(string, format)
+    return _strptime._strptime(string, format)[0]
 
 __doc__ = """This module provides various functions to manipulate time values.
 

Modified: pypy/branch/fast-forward/pypy/module/rctime/test/test_rctime.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/rctime/test/test_rctime.py	(original)
+++ pypy/branch/fast-forward/pypy/module/rctime/test/test_rctime.py	Thu Oct 28 01:54:11 2010
@@ -270,9 +270,11 @@
 
     def test_strptime(self):
         import time as rctime
-        
+
         t = rctime.time()
         tt = rctime.gmtime(t)
+        assert isinstance(rctime.strptime("", ""), type(tt))
+
         for directive in ('a', 'A', 'b', 'B', 'c', 'd', 'H', 'I',
                           'j', 'm', 'M', 'p', 'S',
                           'U', 'w', 'W', 'x', 'X', 'y', 'Y', 'Z', '%'):


From afa at codespeak.net  Thu Oct 28 02:02:54 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 02:02:54 +0200 (CEST)
Subject: [pypy-svn] r78383 - pypy/branch/fast-forward/lib_pypy
Message-ID: <20101028000254.94026282BE0@codespeak.net>

Author: afa
Date: Thu Oct 28 02:02:52 2010
New Revision: 78383

Modified:
   pypy/branch/fast-forward/lib_pypy/datetime.py
Log:
Update datetime module to use the raw _strptime function


Modified: pypy/branch/fast-forward/lib_pypy/datetime.py
==============================================================================
--- pypy/branch/fast-forward/lib_pypy/datetime.py	(original)
+++ pypy/branch/fast-forward/lib_pypy/datetime.py	Thu Oct 28 02:02:52 2010
@@ -1612,10 +1612,11 @@
     @classmethod
     def strptime(cls, date_string, format):
         'string, format -> new datetime parsed from a string (like time.strptime()).'
+        from _strptime import _strptime
         # _strptime._strptime returns a two-element tuple.  The first
         # element is a time.struct_time object.  The second is the
         # microseconds (which are not defined for time.struct_time).
-        struct, micros = _time.strptime(date_string, format)
+        struct, micros = _strptime(date_string, format)
         return cls(*(struct[0:6] + (micros,)))
 
     def utcoffset(self):


From afa at codespeak.net  Thu Oct 28 09:00:29 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 09:00:29 +0200 (CEST)
Subject: [pypy-svn] r78384 - in pypy/branch/fast-forward:
	lib-python/modified-2.7.0 pypy/interpreter pypy/interpreter/test
Message-ID: <20101028070029.EB507282BE0@codespeak.net>

Author: afa
Date: Thu Oct 28 09:00:27 2010
New Revision: 78384

Modified:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/inspect.py
   pypy/branch/fast-forward/pypy/interpreter/argument.py
   pypy/branch/fast-forward/pypy/interpreter/test/test_argument.py
Log:
CPython removed the "non-keyword" fragment from error messages.
Also fix a typo in inspect.py


Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/inspect.py
==============================================================================
--- pypy/branch/fast-forward/lib-python/modified-2.7.0/inspect.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/inspect.py	Thu Oct 28 09:00:27 2010
@@ -952,7 +952,7 @@
             f_name, 'at most' if defaults else 'exactly', num_args,
             'arguments' if num_args > 1 else 'argument', num_total))
     elif num_args == 0 and num_total:
-        raise TypeError('%s() takes no arguments (%d given)' %
+        raise TypeError('%s() takes no argument (%d given)' %
                         (f_name, num_total))
     for arg in args:
         if isinstance(arg, str) and arg in named:

Modified: pypy/branch/fast-forward/pypy/interpreter/argument.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/argument.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/argument.py	Thu Oct 28 09:00:27 2010
@@ -574,20 +574,17 @@
         args = None
         #args_w, kwds_w = args.unpack()
         if self.has_kwarg or (self.num_kwds and self.num_defaults):
-            msg2 = "non-keyword "
             if self.missing_args:
                 required_args = self.expected_nargs - self.num_defaults
                 nargs = required_args - self.missing_args
             else:
                 nargs = self.num_args
         else:
-            msg2 = ""
             nargs = self.num_args + self.num_kwds
         n = self.expected_nargs
         if n == 0:
-            msg = "%s() takes no %sargument (%d given)" % (
+            msg = "%s() takes no argument (%d given)" % (
                 fnname, 
-                msg2,
                 nargs)
         else:
             defcount = self.num_defaults
@@ -598,17 +595,14 @@
             else:
                 msg1 = "at least"
                 n -= defcount
-                if not self.num_kwds:  # msg "f() takes at least X non-keyword args"
-                    msg2 = ""   # is confusing if no kwd arg actually provided
             if n == 1:
                 plural = ""
             else:
                 plural = "s"
-            msg = "%s() takes %s %d %sargument%s (%d given)" % (
+            msg = "%s() takes %s %d argument%s (%d given)" % (
                 fnname,
                 msg1,
                 n,
-                msg2,
                 plural,
                 nargs)
         return msg

Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_argument.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/test/test_argument.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/test/test_argument.py	Thu Oct 28 09:00:27 2010
@@ -486,10 +486,10 @@
         assert s == "foo() takes at most 2 arguments (3 given)"
         err = ArgErrCount(0, 1, 2, True, False, ['a'], 1)
         s = err.getmsg('foo')
-        assert s == "foo() takes at least 1 non-keyword argument (0 given)"
+        assert s == "foo() takes at least 1 argument (0 given)"
         err = ArgErrCount(2, 1, 1, False, True, [], 0)
         s = err.getmsg('foo')
-        assert s == "foo() takes exactly 1 non-keyword argument (2 given)"
+        assert s == "foo() takes exactly 1 argument (2 given)"
 
 
     def test_unknown_keywords(self):


From afa at codespeak.net  Thu Oct 28 10:19:35 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 10:19:35 +0200 (CEST)
Subject: [pypy-svn] r78385 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101028081935.5D387282BE0@codespeak.net>

Author: afa
Date: Thu Oct 28 10:19:33 2010
New Revision: 78385

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Log:
file.read(n) may return a shorter string at end of file.
This also fixes readlines() when there is no \n at the end.

Too much code, and not enough tests :-(


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Thu Oct 28 10:19:33 2010
@@ -371,12 +371,14 @@
         self.read_end = 0
 
         while remaining > 0 and self.read_end < self.buffer_size:
-            # Read until EOF or until read() would block
             try:
                 size = self._fill_buffer(space)
             except BlockingIOError:
+                # EOF or read() would block
                 if written == 0:
                     return None
+                size = 0
+            if size == 0:
                 break
 
             if remaining > 0:

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Thu Oct 28 10:19:33 2010
@@ -22,6 +22,16 @@
         assert r == "a\nb\n"
         f.close()
 
+    def test_read_pieces(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile)
+        f = _io.BufferedReader(raw)
+        assert f.read(3) == "a\nb"
+        assert f.read(3) == "\nc"
+        assert f.read(3) == ""
+        assert f.read(3) == ""
+        f.close()
+
     def test_seek(self):
         import _io
         raw = _io.FileIO(self.tmpfile)
@@ -33,6 +43,12 @@
         assert f.read() == "\nc"
         f.close()
 
+    def test_readlines(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile)
+        f = _io.BufferedReader(raw)
+        assert f.readlines() == ['a\n', 'b\n', 'c']
+
 class AppTestBufferedWriter:
     def setup_class(cls):
         cls.space = gettestobjspace(usemodules=['_io'])


From arigo at codespeak.net  Thu Oct 28 10:42:07 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 28 Oct 2010 10:42:07 +0200 (CEST)
Subject: [pypy-svn] r78386 - in
	pypy/branch/bisect-module/pypy/module/_bisect: . test
Message-ID: <20101028084207.1136B282BE0@codespeak.net>

Author: arigo
Date: Thu Oct 28 10:42:05 2010
New Revision: 78386

Added:
   pypy/branch/bisect-module/pypy/module/_bisect/   (props changed)
   pypy/branch/bisect-module/pypy/module/_bisect/__init__.py   (contents, props changed)
   pypy/branch/bisect-module/pypy/module/_bisect/interp_bisect.py   (contents, props changed)
   pypy/branch/bisect-module/pypy/module/_bisect/test/   (props changed)
   pypy/branch/bisect-module/pypy/module/_bisect/test/__init__.py   (contents, props changed)
   pypy/branch/bisect-module/pypy/module/_bisect/test/test_bisect.py   (contents, props changed)
Log:
bisect_left, bisect_right


Added: pypy/branch/bisect-module/pypy/module/_bisect/__init__.py
==============================================================================
--- (empty file)
+++ pypy/branch/bisect-module/pypy/module/_bisect/__init__.py	Thu Oct 28 10:42:05 2010
@@ -0,0 +1,27 @@
+"""
+Mixed-module definition for the _bisect module.
+This is an optional module; if not present, bisect.py uses the
+pure Python version of these functions.
+"""
+
+from pypy.interpreter.mixedmodule import MixedModule
+
+
+class Module(MixedModule):
+    """\
+This module provides support for maintaining a list in sorted order without
+having to sort the list after each insertion. For long lists of items with
+expensive comparison operations, this can be an improvement over the more
+common approach."""
+
+    appleveldefs = {
+        }
+
+    interpleveldefs = {
+        'bisect':        'interp_bisect.bisect_right',
+        'bisect_left':   'interp_bisect.bisect_left',
+        'bisect_right':  'interp_bisect.bisect_right',
+        'insort':        'interp_bisect.insort_right',
+        'insort_left':   'interp_bisect.insort_left',
+        'insort_right':  'interp_bisect.insort_right',
+        }

Added: pypy/branch/bisect-module/pypy/module/_bisect/interp_bisect.py
==============================================================================
--- (empty file)
+++ pypy/branch/bisect-module/pypy/module/_bisect/interp_bisect.py	Thu Oct 28 10:42:05 2010
@@ -0,0 +1,45 @@
+from pypy.interpreter.gateway import ObjSpace, W_Root
+
+
+def bisect_left(space, w_a, w_x, lo=0, hi=-1):
+    """Return the index where to insert item x in list a, assuming a is sorted.
+
+The return value i is such that all e in a[:i] have e < x, and all e in
+a[i:] have e >= x.  So if x already appears in the list, i points just
+before the leftmost x already there.
+
+Optional args lo (default 0) and hi (default len(a)) bound the
+slice of a to be searched."""
+    if hi == -1:
+        hi = space.int_w(space.len(w_a))
+    while lo < hi:
+        mid = (lo + hi) >> 1
+        w_litem = space.getitem(w_a, space.wrap(mid))
+        if space.is_true(space.lt(w_litem, w_x)):
+            lo = mid + 1
+        else:
+            hi = mid
+    return space.wrap(lo)
+bisect_left.unwrap_spec = [ObjSpace, W_Root, W_Root, int, int]
+
+
+def bisect_right(space, w_a, w_x, lo=0, hi=-1):
+    """Return the index where to insert item x in list a, assuming a is sorted.
+
+The return value i is such that all e in a[:i] have e <= x, and all e in
+a[i:] have e > x.  So if x already appears in the list, i points just
+beyond the rightmost x already there
+
+Optional args lo (default 0) and hi (default len(a)) bound the
+slice of a to be searched."""
+    if hi == -1:
+        hi = space.int_w(space.len(w_a))
+    while lo < hi:
+        mid = (lo + hi) >> 1
+        w_litem = space.getitem(w_a, space.wrap(mid))
+        if space.is_true(space.lt(w_x, w_litem)):
+            hi = mid
+        else:
+            lo = mid + 1
+    return space.wrap(lo)
+bisect_right.unwrap_spec = [ObjSpace, W_Root, W_Root, int, int]

Added: pypy/branch/bisect-module/pypy/module/_bisect/test/__init__.py
==============================================================================

Added: pypy/branch/bisect-module/pypy/module/_bisect/test/test_bisect.py
==============================================================================
--- (empty file)
+++ pypy/branch/bisect-module/pypy/module/_bisect/test/test_bisect.py	Thu Oct 28 10:42:05 2010
@@ -0,0 +1,79 @@
+from pypy.conftest import gettestobjspace
+
+
+class AppTestBisect:
+
+    def setup_class(cls):
+        cls.space = gettestobjspace(usemodules=['_bisect'])
+
+    def test_bisect_left(self):
+        from _bisect import bisect_left
+        a = [0, 5, 6, 6, 6, 7]
+        assert bisect_left(a, None) == 0
+        assert bisect_left(a, -3) == 0
+        assert bisect_left(a, 0) == 0
+        assert bisect_left(a, 3) == 1
+        assert bisect_left(a, 5) == 1
+        assert bisect_left(a, 5.5) == 2
+        assert bisect_left(a, 6) == 2
+        assert bisect_left(a, 6.0) == 2
+        assert bisect_left(a, 6.1) == 5
+        assert bisect_left(a, 7) == 5
+        assert bisect_left(a, 8) == 6
+        a = []
+        assert bisect_left(a, 123) == 0
+        a = [9]
+        assert bisect_left(a, -123) == 0
+        assert bisect_left(a, 9) == 0
+        assert bisect_left(a, 123) == 1
+        a = [9, 9]
+        assert bisect_left(a, -123) == 0
+        assert bisect_left(a, 9) == 0
+        assert bisect_left(a, 123) == 2
+        a = [4, 6, 6, 9]
+        assert bisect_left(a, 6, 0) == 1
+        assert bisect_left(a, 6, 1) == 1
+        assert bisect_left(a, 6, 2) == 2
+        assert bisect_left(a, 6, 3) == 3
+        assert bisect_left(a, 6, 4) == 4
+        assert bisect_left(a, 6, 0, 0) == 0
+        assert bisect_left(a, 6, 0, 1) == 1
+        assert bisect_left(a, 6, 0, 2) == 1
+        assert bisect_left(a, 6, 0, 3) == 1
+        assert bisect_left(a, 6, 0, 4) == 1
+
+    def test_bisect_right(self):
+        from _bisect import bisect_right
+        a = [0, 5, 6, 6, 6, 7]
+        assert bisect_right(a, None) == 0
+        assert bisect_right(a, -3) == 0
+        assert bisect_right(a, 0) == 1
+        assert bisect_right(a, 3) == 1
+        assert bisect_right(a, 5) == 2
+        assert bisect_right(a, 5.5) == 2
+        assert bisect_right(a, 6) == 5
+        assert bisect_right(a, 6.0) == 5
+        assert bisect_right(a, 6.1) == 5
+        assert bisect_right(a, 7) == 6
+        assert bisect_right(a, 8) == 6
+        a = []
+        assert bisect_right(a, 123) == 0
+        a = [9]
+        assert bisect_right(a, -123) == 0
+        assert bisect_right(a, 9) == 1
+        assert bisect_right(a, 123) == 1
+        a = [9, 9]
+        assert bisect_right(a, -123) == 0
+        assert bisect_right(a, 9) == 2
+        assert bisect_right(a, 123) == 2
+        a = [4, 6, 6, 9]
+        assert bisect_right(a, 6, 0) == 3
+        assert bisect_right(a, 6, 1) == 3
+        assert bisect_right(a, 6, 2) == 3
+        assert bisect_right(a, 6, 3) == 3
+        assert bisect_right(a, 6, 4) == 4
+        assert bisect_right(a, 6, 0, 0) == 0
+        assert bisect_right(a, 6, 0, 1) == 1
+        assert bisect_right(a, 6, 0, 2) == 2
+        assert bisect_right(a, 6, 0, 3) == 3
+        assert bisect_right(a, 6, 0, 4) == 3


From arigo at codespeak.net  Thu Oct 28 10:48:42 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 28 Oct 2010 10:48:42 +0200 (CEST)
Subject: [pypy-svn] r78387 - in
	pypy/branch/bisect-module/pypy/module/_bisect: . test
Message-ID: <20101028084842.9D4745080C@codespeak.net>

Author: arigo
Date: Thu Oct 28 10:48:41 2010
New Revision: 78387

Added:
   pypy/branch/bisect-module/pypy/module/_bisect/app_bisect.py
Modified:
   pypy/branch/bisect-module/pypy/module/_bisect/__init__.py
   pypy/branch/bisect-module/pypy/module/_bisect/test/test_bisect.py
Log:
Insort.


Modified: pypy/branch/bisect-module/pypy/module/_bisect/__init__.py
==============================================================================
--- pypy/branch/bisect-module/pypy/module/_bisect/__init__.py	(original)
+++ pypy/branch/bisect-module/pypy/module/_bisect/__init__.py	Thu Oct 28 10:48:41 2010
@@ -15,13 +15,13 @@
 common approach."""
 
     appleveldefs = {
+        'insort':        'app_bisect.insort_right',
+        'insort_left':   'app_bisect.insort_left',
+        'insort_right':  'app_bisect.insort_right',
         }
 
     interpleveldefs = {
         'bisect':        'interp_bisect.bisect_right',
         'bisect_left':   'interp_bisect.bisect_left',
         'bisect_right':  'interp_bisect.bisect_right',
-        'insort':        'interp_bisect.insort_right',
-        'insort_left':   'interp_bisect.insort_left',
-        'insort_right':  'interp_bisect.insort_right',
         }

Added: pypy/branch/bisect-module/pypy/module/_bisect/app_bisect.py
==============================================================================
--- (empty file)
+++ pypy/branch/bisect-module/pypy/module/_bisect/app_bisect.py	Thu Oct 28 10:48:41 2010
@@ -0,0 +1,21 @@
+from _bisect import bisect_left, bisect_right
+
+
+def insort_left(a, x, lo=0, hi=-1):
+    """Insert item x in list a, and keep it sorted assuming a is sorted.
+
+If x is already in a, insert it to the left of the leftmost x.
+
+Optional args lo (default 0) and hi (default len(a)) bound the
+slice of a to be searched."""
+    a.insert(bisect_left(a, x, lo, hi), x)
+
+
+def insort_right(a, x, lo=0, hi=-1):
+    """Insert item x in list a, and keep it sorted assuming a is sorted.
+
+If x is already in a, insert it to the right of the rightmost x.
+
+Optional args lo (default 0) and hi (default len(a)) bound the
+slice of a to be searched."""
+    a.insert(bisect_right(a, x, lo, hi), x)

Modified: pypy/branch/bisect-module/pypy/module/_bisect/test/test_bisect.py
==============================================================================
--- pypy/branch/bisect-module/pypy/module/_bisect/test/test_bisect.py	(original)
+++ pypy/branch/bisect-module/pypy/module/_bisect/test/test_bisect.py	Thu Oct 28 10:48:41 2010
@@ -77,3 +77,17 @@
         assert bisect_right(a, 6, 0, 2) == 2
         assert bisect_right(a, 6, 0, 3) == 3
         assert bisect_right(a, 6, 0, 4) == 3
+
+    def test_insort_left(self):
+        from _bisect import insort_left
+        a = [0, 5, 6, 6, 6, 7]
+        insort_left(a, 6.0)
+        assert a == [0, 5, 6.0, 6, 6, 6, 7]
+        assert map(type, a) == [int, int, float, int, int, int, int]
+
+    def test_insort_right(self):
+        from _bisect import insort_right
+        a = [0, 5, 6, 6, 6, 7]
+        insort_right(a, 6.0)
+        assert a == [0, 5, 6, 6, 6, 6.0, 7]
+        assert map(type, a) == [int, int, int, int, int, float, int]


From arigo at codespeak.net  Thu Oct 28 10:49:23 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 28 Oct 2010 10:49:23 +0200 (CEST)
Subject: [pypy-svn] r78388 - pypy/branch/bisect-module/pypy/config
Message-ID: <20101028084923.29058282BE0@codespeak.net>

Author: arigo
Date: Thu Oct 28 10:49:21 2010
New Revision: 78388

Modified:
   pypy/branch/bisect-module/pypy/config/pypyoption.py
Log:
Enable this small module.


Modified: pypy/branch/bisect-module/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/bisect-module/pypy/config/pypyoption.py	(original)
+++ pypy/branch/bisect-module/pypy/config/pypyoption.py	Thu Oct 28 10:49:21 2010
@@ -30,7 +30,8 @@
       "rctime" , "select", "zipimport", "_lsprof",
      "crypt", "signal", "_rawffi", "termios", "zlib",
      "struct", "md5", "sha", "bz2", "_minimal_curses", "cStringIO",
-     "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array"]
+     "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array",
+     "_bisect"]
 ))
 
 working_oo_modules = default_modules.copy()


From afa at codespeak.net  Thu Oct 28 11:31:13 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 11:31:13 +0200 (CEST)
Subject: [pypy-svn] r78389 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101028093113.8DCD85080C@codespeak.net>

Author: afa
Date: Thu Oct 28 11:31:11 2010
New Revision: 78389

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Log:
Implement .fileno() and .detach()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Thu Oct 28 11:31:11 2010
@@ -30,11 +30,16 @@
     def write_w(self, space, w_data):
         self._unsupportedoperation(space, "write")
 
+    @unwrap_spec('self', ObjSpace)
+    def detach_w(self, space):
+        self._unsupportedoperation(space, "detach")
+
 W_BufferedIOBase.typedef = TypeDef(
     '_BufferedIOBase', W_IOBase.typedef,
     __new__ = generic_new_descr(W_BufferedIOBase),
     read = interp2app(W_BufferedIOBase.read_w),
     write = interp2app(W_BufferedIOBase.write_w),
+    detach = interp2app(W_BufferedIOBase.detach_w),
     )
 
 class BufferedMixin:
@@ -127,6 +132,7 @@
 
     @unwrap_spec('self', ObjSpace, r_longlong, int)
     def seek_w(self, space, pos, whence=0):
+        self._check_init(space)
         if whence not in (0, 1, 2):
             raise operationerrfmt(space.w_ValueError,
                 "whence must be between 0 and 2, not %d", whence)
@@ -244,6 +250,20 @@
             l.append(self.buffer[i])
         return self._write(space, ''.join(l))
 
+    @unwrap_spec('self', ObjSpace)
+    def detach_w(self, space):
+        self._check_init(space)
+        space.call_method(self, "flush")
+        raw = self.raw
+        self.raw = None
+        self.state = STATE_DETACHED
+        return space.wrap(raw)
+
+    @unwrap_spec('self', ObjSpace)
+    def fileno_w(self, space):
+        self._check_init(space)
+        return space.call_method(self.raw, "fileno")
+
 class W_BufferedReader(BufferedMixin, W_BufferedIOBase):
     @unwrap_spec('self', ObjSpace, W_Root, int)
     def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE):
@@ -261,6 +281,7 @@
 
     @unwrap_spec('self', ObjSpace, W_Root)
     def read_w(self, space, w_size=None):
+        self._check_init(space)
         self._check_closed(space, "read of closed file")
         size = convert_size(space, w_size)
 
@@ -417,6 +438,8 @@
     seek = interp2app(W_BufferedReader.seek_w),
     close = interp2app(W_BufferedReader.close_w),
     flush = interp2app(W_BufferedReader.flush_w),
+    detach = interp2app(W_BufferedReader.detach_w),
+    fileno = interp2app(W_BufferedReader.fileno_w),
     closed = GetSetProperty(W_BufferedReader.closed_get_w),
     )
 
@@ -442,6 +465,7 @@
 
     @unwrap_spec('self', ObjSpace, W_Root)
     def write_w(self, space, w_data):
+        self._check_init(space)
         self._check_closed(space, "write to closed file")
         data = space.str_w(w_data)
         size = len(data)
@@ -548,6 +572,7 @@
 
     @unwrap_spec('self', ObjSpace)
     def flush_w(self, space):
+        self._check_init(space)
         self._check_closed(space, "flush of closed file")
         with self.lock:
             self._writer_flush_unlocked(space)
@@ -568,6 +593,8 @@
     # from the mixin class
     seek = interp2app(W_BufferedWriter.seek_w),
     close = interp2app(W_BufferedWriter.close_w),
+    fileno = interp2app(W_BufferedWriter.fileno_w),
+    detach = interp2app(W_BufferedWriter.detach_w),
     closed = GetSetProperty(W_BufferedWriter.closed_get_w),
     )
 

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Thu Oct 28 11:31:11 2010
@@ -49,6 +49,19 @@
         f = _io.BufferedReader(raw)
         assert f.readlines() == ['a\n', 'b\n', 'c']
 
+    def test_detach(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile)
+        f = _io.BufferedReader(raw)
+        assert f.fileno() == raw.fileno()
+        assert f.detach() is raw
+        raises(ValueError, f.fileno)
+        raises(ValueError, f.close)
+        raises(ValueError, f.detach)
+        raises(ValueError, f.flush)
+        assert not raw.closed
+        raw.close()
+
 class AppTestBufferedWriter:
     def setup_class(cls):
         cls.space = gettestobjspace(usemodules=['_io'])


From afa at codespeak.net  Thu Oct 28 12:11:51 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 12:11:51 +0200 (CEST)
Subject: [pypy-svn] r78390 - in pypy/trunk/pypy/translator/c/gcc: . test
	test/darwin64
Message-ID: <20101028101151.6B81A282C1E@codespeak.net>

Author: afa
Date: Thu Oct 28 12:11:49 2010
New Revision: 78390

Added:
   pypy/trunk/pypy/translator/c/gcc/test/darwin64/
   pypy/trunk/pypy/translator/c/gcc/test/darwin64/track0.s
Modified:
   pypy/trunk/pypy/translator/c/gcc/test/test_trackgcroot.py
   pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
Log:
Add trackgcroot support for 64bit darwin


Added: pypy/trunk/pypy/translator/c/gcc/test/darwin64/track0.s
==============================================================================
--- (empty file)
+++ pypy/trunk/pypy/translator/c/gcc/test/darwin64/track0.s	Thu Oct 28 12:11:49 2010
@@ -0,0 +1,159 @@
+_PyObject_Realloc:
+LFB118:
+	movq	%rbx, -32(%rsp)
+LCFI86:
+	movq	%rbp, -24(%rsp)
+LCFI87:
+	movq	%r12, -16(%rsp)
+LCFI88:
+	movq	%r13, -8(%rsp)
+LCFI89:
+	subq	$40, %rsp
+LCFI90:
+	movq	%rdi, %rbx
+	testq	%rdi, %rdi
+	je	L384
+	movq	%rdi, %rbp
+	andq	$-4096, %rbp
+	movl	32(%rbp), %edx
+	movl	_narenas(%rip), %eax
+	cmpl	%eax, %edx
+	jae	L360
+	movq	_arenas(%rip), %rax
+	mov	%edx, %edx
+	movq	%rdi, %rcx
+	subq	(%rax,%rdx,8), %rcx
+	cmpq	$262143, %rcx
+	ja	L360
+	movl	36(%rbp), %ecx
+	incl	%ecx
+	leal	0(,%rcx,8), %r12d
+	mov	%r12d, %eax
+	cmpq	%rax, %rsi
+	ja	L363
+	leaq	0(,%rsi,4), %rdx
+	sall	$4, %ecx
+	leal	(%rcx,%r12), %eax
+	mov	%eax, %eax
+	cmpq	%rax, %rdx
+	ja	L365
+	movl	%esi, %r12d
+L363:
+	movq	%rsi, %rdi
+	call	_PyObject_Malloc
+	;; expected {40(%rsp) | 8(%rsp), 24(%rsp), 32(%rsp), %r14, %r15, 16(%rsp) | }
+	testq	%rax, %rax
+	je	L367
+	mov	%r12d, %edx
+	movq	%rbx, %rsi
+	movq	%rax, %rdi
+	call	_memcpy
+	;; expected {40(%rsp) | 8(%rsp), 24(%rsp), 32(%rsp), %r14, %r15, 16(%rsp) | }
+	movl	32(%rbp), %edx
+	movl	_narenas(%rip), %eax
+	cmpl	%eax, %edx
+	jae	L369
+	movq	_arenas(%rip), %rax
+	mov	%edx, %edx
+	movq	%rbx, %rcx
+	subq	(%rax,%rdx,8), %rcx
+	cmpq	$262143, %rcx
+	jbe	L385
+L369:
+	movq	%rbx, %rdi
+	call	_free
+	;; expected {40(%rsp) | 8(%rsp), 24(%rsp), 32(%rsp), %r14, %r15, 16(%rsp) | }
+	movq	%r13, %rbx
+	jmp	L365
+	.align 4,0x90
+L360:
+	testq	%rsi, %rsi
+	jne	L386
+	movl	$1, %esi
+	movq	%rbx, %rdi
+	call	_realloc
+	;; expected {40(%rsp) | 8(%rsp), 24(%rsp), 32(%rsp), %r14, %r15, 16(%rsp) | }
+	testq	%rax, %rax
+	cmovne	%rax, %rbx
+L365:
+	movq	%rbx, %rax
+	movq	8(%rsp), %rbx
+	movq	16(%rsp), %rbp
+	movq	24(%rsp), %r12
+	movq	32(%rsp), %r13
+	addq	$40, %rsp
+	ret
+	.align 4,0x90
+L386:
+	movq	%rbx, %rdi
+	movq	8(%rsp), %rbx
+	movq	16(%rsp), %rbp
+	movq	24(%rsp), %r12
+	movq	32(%rsp), %r13
+	addq	$40, %rsp
+	jmp	_realloc
+L384:
+	movq	%rsi, %rdi
+	movq	8(%rsp), %rbx
+	movq	16(%rsp), %rbp
+	movq	24(%rsp), %r12
+	movq	32(%rsp), %r13
+	addq	$40, %rsp
+	jmp	_PyObject_Malloc
+L367:
+	movq	%r13, %rbx
+	jmp	L365
+L385:
+	movl	(%rbp), %esi
+	testl	%esi, %esi
+	je	L387
+	movq	8(%rbp), %rax
+	movq	%rax, (%rbx)
+	movq	%rbx, 8(%rbp)
+	testq	%rax, %rax
+	je	L374
+	movl	(%rbp), %eax
+	decl	%eax
+	movl	%eax, (%rbp)
+	testl	%eax, %eax
+	jne	L367
+	movq	16(%rbp), %rdx
+	movq	24(%rbp), %rax
+	movq	%rax, 24(%rdx)
+	movq	%rdx, 16(%rax)
+	movq	_freepools(%rip), %rax
+	movq	%rax, 16(%rbp)
+	movq	%rbp, _freepools(%rip)
+	movq	%r13, %rbx
+	jmp	L365
+L374:
+	movl	(%rbp), %eax
+	decl	%eax
+	movl	%eax, (%rbp)
+	testl	%eax, %eax
+	je	L388
+	movl	36(%rbp), %eax
+	addl	%eax, %eax
+	mov	%eax, %eax
+	leaq	_usedpools(%rip), %rdx
+	movq	(%rdx,%rax,8), %rax
+	movq	24(%rax), %rdx
+	movq	%rax, 16(%rbp)
+	movq	%rdx, 24(%rbp)
+	movq	%rbp, 24(%rax)
+	movq	%rbp, 16(%rdx)
+	movq	%r13, %rbx
+	jmp	L365
+L387:
+	leaq	LC6(%rip), %rcx
+	movl	$744, %edx
+	leaq	LC7(%rip), %rsi
+	leaq	___func__.207211(%rip), %rdi
+	call	___assert_rtn
+L388:
+	leaq	LC6(%rip), %rcx
+	movl	$783, %edx
+	leaq	LC7(%rip), %rsi
+	leaq	___func__.207211(%rip), %rdi
+	call	___assert_rtn
+LFE118:

Modified: pypy/trunk/pypy/translator/c/gcc/test/test_trackgcroot.py
==============================================================================
--- pypy/trunk/pypy/translator/c/gcc/test/test_trackgcroot.py	(original)
+++ pypy/trunk/pypy/translator/c/gcc/test/test_trackgcroot.py	Thu Oct 28 12:11:49 2010
@@ -109,7 +109,7 @@
  
 def test_computegcmaptable():
     tests = []
-    for format in ('elf', 'darwin', 'msvc', 'elf64'):
+    for format in ('elf', 'elf64', 'darwin', 'darwin64', 'msvc'):
         for path in this_dir.join(format).listdir("track*.s"):
             n = path.purebasename[5:]
             try:

Modified: pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
==============================================================================
--- pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	(original)
+++ pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	Thu Oct 28 12:11:49 2010
@@ -1090,7 +1090,7 @@
 
 ElfFunctionGcRootTracker64.init_regexp()
 
-class DarwinFunctionGcRootTracker(ElfFunctionGcRootTracker32):
+class DarwinFunctionGcRootTracker32(ElfFunctionGcRootTracker32):
     format = 'darwin'
     function_names_prefix = '_'
 
@@ -1102,7 +1102,19 @@
         funcname = '_' + match.group(1)
         FunctionGcRootTracker32.__init__(self, funcname, lines, filetag)
 
-class Mingw32FunctionGcRootTracker(DarwinFunctionGcRootTracker):
+class DarwinFunctionGcRootTracker64(ElfFunctionGcRootTracker64):
+    format = 'darwin64'
+    function_names_prefix = '_'
+
+    r_functionstart = re.compile(r"_(\w+):\s*$")
+    OFFSET_LABELS   = 0
+
+    def __init__(self, lines, filetag=0):
+        match = self.r_functionstart.match(lines[0])
+        funcname = '_' + match.group(1)
+        FunctionGcRootTracker64.__init__(self, funcname, lines, filetag)
+
+class Mingw32FunctionGcRootTracker(DarwinFunctionGcRootTracker32):
     format = 'mingw32'
     function_names_prefix = '_'
 
@@ -1373,7 +1385,7 @@
 
 class DarwinAssemblerParser(AssemblerParser):
     format = "darwin"
-    FunctionGcRootTracker = DarwinFunctionGcRootTracker
+    FunctionGcRootTracker = DarwinFunctionGcRootTracker32
 
     r_textstart = re.compile(r"\t.text\s*$")
 
@@ -1419,6 +1431,10 @@
         return super(DarwinAssemblerParser, self).process_function(
             lines, entrypoint, filename)
 
+class DarwinAssemblerParser64(DarwinAssemblerParser):
+    format = "darwin64"
+    FunctionGcRootTracker = DarwinFunctionGcRootTracker64
+
 class Mingw32AssemblerParser(DarwinAssemblerParser):
     format = "mingw32"
     FunctionGcRootTracker = Mingw32FunctionGcRootTracker
@@ -1542,6 +1558,7 @@
     'elf': ElfAssemblerParser,
     'elf64': ElfAssemblerParser64,
     'darwin': DarwinAssemblerParser,
+    'darwin64': DarwinAssemblerParser64,
     'mingw32': Mingw32AssemblerParser,
     'msvc': MsvcAssemblerParser,
     }
@@ -1885,7 +1902,10 @@
     shuffle = False
     output_raw_table = False
     if sys.platform == 'darwin':
-        format = 'darwin'
+        if sys.maxint > 2147483647:
+            format = 'darwin64'
+        else:
+            format = 'darwin'
     elif sys.platform == 'win32':
         format = 'mingw32'
     else:


From arigo at codespeak.net  Thu Oct 28 12:17:39 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 28 Oct 2010 12:17:39 +0200 (CEST)
Subject: [pypy-svn] r78391 - pypy/branch/bisect-module/pypy/module/_bisect
Message-ID: <20101028101739.1B6EC5080C@codespeak.net>

Author: arigo
Date: Thu Oct 28 12:17:38 2010
New Revision: 78391

Modified:
   pypy/branch/bisect-module/pypy/module/_bisect/app_bisect.py
Log:
Recorder things that way.  Should fix CPython's test_bisect.py.


Modified: pypy/branch/bisect-module/pypy/module/_bisect/app_bisect.py
==============================================================================
--- pypy/branch/bisect-module/pypy/module/_bisect/app_bisect.py	(original)
+++ pypy/branch/bisect-module/pypy/module/_bisect/app_bisect.py	Thu Oct 28 12:17:38 2010
@@ -8,7 +8,8 @@
 
 Optional args lo (default 0) and hi (default len(a)) bound the
 slice of a to be searched."""
-    a.insert(bisect_left(a, x, lo, hi), x)
+    n = bisect_left(a, x, lo, hi)
+    a.insert(n, x)
 
 
 def insort_right(a, x, lo=0, hi=-1):
@@ -18,4 +19,5 @@
 
 Optional args lo (default 0) and hi (default len(a)) bound the
 slice of a to be searched."""
-    a.insert(bisect_right(a, x, lo, hi), x)
+    n = bisect_right(a, x, lo, hi)
+    a.insert(n, x)


From arigo at codespeak.net  Thu Oct 28 12:17:47 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 28 Oct 2010 12:17:47 +0200 (CEST)
Subject: [pypy-svn] r78392 - pypy/branch/bisect-module/lib-python
Message-ID: <20101028101747.7DE20282C1E@codespeak.net>

Author: arigo
Date: Thu Oct 28 12:17:46 2010
New Revision: 78392

Modified:
   pypy/branch/bisect-module/lib-python/conftest.py
Log:
Add a usemodules line.


Modified: pypy/branch/bisect-module/lib-python/conftest.py
==============================================================================
--- pypy/branch/bisect-module/lib-python/conftest.py	(original)
+++ pypy/branch/bisect-module/lib-python/conftest.py	Thu Oct 28 12:17:46 2010
@@ -144,7 +144,7 @@
     RegrTest('test_binhex.py'),
 
     RegrTest('test_binop.py', core=True),
-    RegrTest('test_bisect.py', core=True),
+    RegrTest('test_bisect.py', core=True, usemodules='_bisect'),
     RegrTest('test_bool.py', core=True),
     RegrTest('test_bsddb.py', skip="unsupported extension module"),
     RegrTest('test_bsddb185.py', skip="unsupported extension module"),


From arigo at codespeak.net  Thu Oct 28 12:18:55 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 28 Oct 2010 12:18:55 +0200 (CEST)
Subject: [pypy-svn] r78393 - in pypy/trunk: lib-python pypy/config
	pypy/module/_bisect pypy/module/_bisect/test
Message-ID: <20101028101855.4A9F6282C1E@codespeak.net>

Author: arigo
Date: Thu Oct 28 12:18:53 2010
New Revision: 78393

Added:
   pypy/trunk/pypy/module/_bisect/   (props changed)
      - copied from r78392, pypy/branch/bisect-module/pypy/module/_bisect/
Modified:
   pypy/trunk/lib-python/conftest.py
   pypy/trunk/pypy/config/pypyoption.py
   pypy/trunk/pypy/module/_bisect/test/   (props changed)
Log:
Merge branch/bisect-module, writing the '_bisect' module like CPython.


Modified: pypy/trunk/lib-python/conftest.py
==============================================================================
--- pypy/trunk/lib-python/conftest.py	(original)
+++ pypy/trunk/lib-python/conftest.py	Thu Oct 28 12:18:53 2010
@@ -144,7 +144,7 @@
     RegrTest('test_binhex.py'),
 
     RegrTest('test_binop.py', core=True),
-    RegrTest('test_bisect.py', core=True),
+    RegrTest('test_bisect.py', core=True, usemodules='_bisect'),
     RegrTest('test_bool.py', core=True),
     RegrTest('test_bsddb.py', skip="unsupported extension module"),
     RegrTest('test_bsddb185.py', skip="unsupported extension module"),

Modified: pypy/trunk/pypy/config/pypyoption.py
==============================================================================
--- pypy/trunk/pypy/config/pypyoption.py	(original)
+++ pypy/trunk/pypy/config/pypyoption.py	Thu Oct 28 12:18:53 2010
@@ -30,7 +30,8 @@
       "rctime" , "select", "zipimport", "_lsprof",
      "crypt", "signal", "_rawffi", "termios", "zlib",
      "struct", "md5", "sha", "bz2", "_minimal_curses", "cStringIO",
-     "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array"]
+     "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array",
+     "_bisect"]
 ))
 
 working_oo_modules = default_modules.copy()


From arigo at codespeak.net  Thu Oct 28 12:19:19 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 28 Oct 2010 12:19:19 +0200 (CEST)
Subject: [pypy-svn] r78394 - pypy/branch/bisect-module
Message-ID: <20101028101919.1A1865080C@codespeak.net>

Author: arigo
Date: Thu Oct 28 12:19:17 2010
New Revision: 78394

Removed:
   pypy/branch/bisect-module/
Log:
Remove merged branch.



From arigo at codespeak.net  Thu Oct 28 12:39:15 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 28 Oct 2010 12:39:15 +0200 (CEST)
Subject: [pypy-svn] r78395 - pypy/trunk/pypy/module/pypyjit/test
Message-ID: <20101028103915.57DC0282C1D@codespeak.net>

Author: arigo
Date: Thu Oct 28 12:39:13 2010
New Revision: 78395

Modified:
   pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
Log:
Fix and document this test.

Modified: pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/trunk/pypy/module/pypyjit/test/test_pypy_c.py	Thu Oct 28 12:39:13 2010
@@ -294,7 +294,11 @@
 
         ops = self.get_by_bytecode("LOAD_ATTR")
         assert len(ops) == 2
-        assert ops[0].get_opnames() == ["getfield_gc", "getarrayitem_gc",
+        # With mapdict, we get fast access to (so far) the 5 first
+        # attributes, which means it is done with only the following
+        # operations.  (For the other attributes there is additionally
+        # a getarrayitem_gc.)
+        assert ops[0].get_opnames() == ["getfield_gc",
                                         "guard_nonnull_class"]
         assert not ops[1] # second LOAD_ATTR folded away
 


From arigo at codespeak.net  Thu Oct 28 13:32:07 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Thu, 28 Oct 2010 13:32:07 +0200 (CEST)
Subject: [pypy-svn] r78397 - in pypy/trunk/pypy/jit/codewriter: . test
Message-ID: <20101028113207.0DFE05080C@codespeak.net>

Author: arigo
Date: Thu Oct 28 13:32:06 2010
New Revision: 78397

Modified:
   pypy/trunk/pypy/jit/codewriter/format.py
   pypy/trunk/pypy/jit/codewriter/jtransform.py
   pypy/trunk/pypy/jit/codewriter/test/test_list.py
Log:
Minimal support for void lists: remove the operations that are no-ops.


Modified: pypy/trunk/pypy/jit/codewriter/format.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/format.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/format.py	Thu Oct 28 13:32:06 2010
@@ -80,7 +80,8 @@
 
 def assert_format(ssarepr, expected):
     asm = format_assembler(ssarepr)
-    expected = str(py.code.Source(expected)).strip() + '\n'
+    if expected != '':
+        expected = str(py.code.Source(expected)).strip() + '\n'
     asmlines = asm.split("\n")
     explines = expected.split("\n")
     for asm, exp in zip(asmlines, explines):

Modified: pypy/trunk/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/jtransform.py	Thu Oct 28 13:32:06 2010
@@ -894,17 +894,21 @@
             prefix = 'do_resizable_'
             ARRAY = LIST.items.TO
             if self._array_of_voids(ARRAY):
-                raise NotSupported("resizable lists of voids")
-            descrs = (self.cpu.arraydescrof(ARRAY),
-                      self.cpu.fielddescrof(LIST, 'length'),
-                      self.cpu.fielddescrof(LIST, 'items'),
-                      self.cpu.sizeof(LIST))
+                prefix += 'void_'
+                descrs = ()
+            else:
+                descrs = (self.cpu.arraydescrof(ARRAY),
+                          self.cpu.fielddescrof(LIST, 'length'),
+                          self.cpu.fielddescrof(LIST, 'items'),
+                          self.cpu.sizeof(LIST))
         else:
             prefix = 'do_fixed_'
             if self._array_of_voids(LIST):
-                raise NotSupported("fixed lists of voids")
-            arraydescr = self.cpu.arraydescrof(LIST)
-            descrs = (arraydescr,)
+                prefix += 'void_'
+                descrs = ()
+            else:
+                arraydescr = self.cpu.arraydescrof(LIST)
+                descrs = (arraydescr,)
         #
         try:
             meth = getattr(self, prefix + oopspec_name.replace('.', '_'))
@@ -943,6 +947,11 @@
                                              descr, args[1]], v_posindex)
             return v_posindex, [op0, op1]
 
+    def _prepare_void_list_getset(self, op):
+        non_negative, can_raise = self._get_list_nonneg_canraise_flags(op)
+        if can_raise:
+            raise NotSupported("list operation can raise")
+
     def _get_initial_newlist_length(self, op, args):
         # normalize number of arguments to the 'newlist' function
         if len(args) > 1:
@@ -1014,6 +1023,12 @@
     def do_fixed_list_ll_arraycopy(self, op, args, arraydescr):
         return self._handle_oopspec_call(op, args, EffectInfo.OS_ARRAYCOPY)
 
+    def do_fixed_void_list_getitem(self, op, args):
+        self._prepare_void_list_getset(op)
+        return []
+    do_fixed_void_list_getitem_foldable = do_fixed_void_list_getitem
+    do_fixed_void_list_setitem = do_fixed_void_list_getitem
+
     # ---------- resizable lists ----------
 
     def do_resizable_newlist(self, op, args, arraydescr, lengthdescr,
@@ -1049,6 +1064,12 @@
         return SpaceOperation('getfield_gc_i',
                               [args[0], lengthdescr], op.result)
 
+    def do_resizable_void_list_getitem(self, op, args):
+        self._prepare_void_list_getset(op)
+        return []
+    do_resizable_void_list_getitem_foldable = do_resizable_void_list_getitem
+    do_resizable_void_list_setitem = do_resizable_void_list_getitem
+
     # ----------
     # Strings and Unicodes.
 

Modified: pypy/trunk/pypy/jit/codewriter/test/test_list.py
==============================================================================
--- pypy/trunk/pypy/jit/codewriter/test/test_list.py	(original)
+++ pypy/trunk/pypy/jit/codewriter/test/test_list.py	Thu Oct 28 13:32:06 2010
@@ -19,6 +19,7 @@
 class FakeCPU:
     class arraydescrof(AbstractDescr):
         def __init__(self, ARRAY):
+            assert ARRAY.OF != lltype.Void
             self.ARRAY = ARRAY
         def __repr__(self):
             return ''


From afa at codespeak.net  Thu Oct 28 14:59:11 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 14:59:11 +0200 (CEST)
Subject: [pypy-svn] r78398 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101028125911.25630282C1E@codespeak.net>

Author: afa
Date: Thu Oct 28 14:59:09 2010
New Revision: 78398

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Log:
BufferedReader.read1


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Thu Oct 28 14:59:09 2010
@@ -27,6 +27,10 @@
         self._unsupportedoperation(space, "read")
 
     @unwrap_spec('self', ObjSpace, W_Root)
+    def read1_w(self, space, w_size):
+        self._unsupportedoperation(space, "read1")
+
+    @unwrap_spec('self', ObjSpace, W_Root)
     def write_w(self, space, w_data):
         self._unsupportedoperation(space, "write")
 
@@ -38,6 +42,7 @@
     '_BufferedIOBase', W_IOBase.typedef,
     __new__ = generic_new_descr(W_BufferedIOBase),
     read = interp2app(W_BufferedIOBase.read_w),
+    read1 = interp2app(W_BufferedIOBase.read1_w),
     write = interp2app(W_BufferedIOBase.write_w),
     detach = interp2app(W_BufferedIOBase.detach_w),
     )
@@ -296,6 +301,45 @@
                     res = self._read_generic(space, size)
         return space.wrap(res)
 
+    @unwrap_spec('self', ObjSpace, int)
+    def read1_w(self, space, size):
+        self._check_init(space)
+        self._check_closed(space, "read of closed file")
+
+        if size < 0:
+            raise OperationError(space.w_ValueError, space.wrap(
+                "read length must be positive"))
+        if size == 0:
+            return space.wrap("")
+
+        with self.lock:
+            if self.writable:
+                self._writer_flush_unlocked(space, restore_pos=True)
+
+            # Return up to n bytes.  If at least one byte is buffered, we only
+            # return buffered bytes.  Otherwise, we do one raw read.
+
+            # XXX: this mimicks the io.py implementation but is probably
+            # wrong. If we need to read from the raw stream, then we could
+            # actually read all `n` bytes asked by the caller (and possibly
+            # more, so as to fill our buffer for the next reads).
+
+            have = self._readahead()
+            if have == 0:
+                # Fill the buffer from the raw stream
+                self._reader_reset_buf()
+                self.pos = 0
+                try:
+                    have = self._fill_buffer(space)
+                except BlockingIOError:
+                    have = 0
+            if size > have:
+                size = have
+            data = rffi.charpsize2str(rffi.ptradd(self.buffer, self.pos),
+                                      size)
+            self.pos += size
+            return space.wrap(data)
+
     def _read_all(self, space):
         "Read all the file, don't update the cache"
         builder = StringBuilder()
@@ -433,6 +477,7 @@
     __init__  = interp2app(W_BufferedReader.descr_init),
 
     read = interp2app(W_BufferedReader.read_w),
+    read1 = interp2app(W_BufferedReader.read1_w),
 
     # from the mixin class
     seek = interp2app(W_BufferedReader.seek_w),

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Thu Oct 28 14:59:09 2010
@@ -32,6 +32,29 @@
         assert f.read(3) == ""
         f.close()
 
+    def test_read1(self):
+        import _io
+        class RecordingFileIO(_io.FileIO):
+            def read(self, size=-1):
+                self.nbreads += 1
+                return _io.FileIO.read(self, size)
+            def readinto(self, buf):
+                self.nbreads += 1
+                return _io.FileIO.readinto(self, buf)
+        raw = RecordingFileIO(self.tmpfile)
+        raw.nbreads = 0
+        f = _io.BufferedReader(raw, buffer_size=3)
+        assert f.read(1) == 'a'
+        assert f.read1(1) == '\n'
+        assert raw.nbreads == 1
+        assert f.read1(100) == 'b'
+        assert raw.nbreads == 1
+        assert f.read1(100) == '\nc'
+        assert raw.nbreads == 2
+        assert f.read1(100) == ''
+        assert raw.nbreads == 3
+        f.close()
+
     def test_seek(self):
         import _io
         raw = _io.FileIO(self.tmpfile)


From afa at codespeak.net  Thu Oct 28 15:15:38 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 15:15:38 +0200 (CEST)
Subject: [pypy-svn] r78399 - in pypy/trunk/pypy/translator/c/gcc: .
	test/darwin64
Message-ID: <20101028131538.3D443282C1E@codespeak.net>

Author: afa
Date: Thu Oct 28 15:15:36 2010
New Revision: 78399

Added:
   pypy/trunk/pypy/translator/c/gcc/test/darwin64/track1.s
Modified:
   pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
Log:
On darwin 64bit, fix pattern to recognise items of table-based switches


Added: pypy/trunk/pypy/translator/c/gcc/test/darwin64/track1.s
==============================================================================
--- (empty file)
+++ pypy/trunk/pypy/translator/c/gcc/test/darwin64/track1.s	Thu Oct 28 15:15:36 2010
@@ -0,0 +1,785 @@
+_pypy_g_PyObject_RichCompare:
+LFB302:
+	movq	%rbx, -48(%rsp)
+LCFI297:
+	movq	%rbp, -40(%rsp)
+LCFI298:
+	movq	%r12, -32(%rsp)
+LCFI299:
+	movq	%r13, -24(%rsp)
+LCFI300:
+	movq	%r14, -16(%rsp)
+LCFI301:
+	movq	%r15, -8(%rsp)
+LCFI302:
+	subq	$88, %rsp
+LCFI303:
+	movq	%rdi, %rbx
+	movq	%rsi, %rbp
+L970:
+	cmpl	$5, %edx
+	jbe	L1056
+L971:
+	movq	_pypy_g_pypy_rpython_memory_gc_minimark_MiniMarkGC at GOTPCREL(%rip), %rdi
+	movq	16(%rdi), %rdx
+	leaq	16(%rdx), %rax
+	movq	%rax, 16(%rdi)
+	cmpq	24(%rdi), %rax
+	ja	L1034
+	movq	%rdx, %rbx
+	movq	_pypy_g_ExcData at GOTPCREL(%rip), %r11
+L1036:
+	movq	$61224, (%rbx)
+	movq	$0, 8(%rdx)
+L1037:
+	movq	%r11, (%rsp)
+	call	_pypy_g_PyErr_BadInternalCall
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | %rbx}
+	movq	___gcmapend at GOTPCREL(%rip), %rax
+	/* GCROOT %rbx */
+	movq	(%rsp), %r11
+	movq	(%r11), %rbp
+	testq	%rbp, %rbp
+	je	L1057
+L1038:
+	movq	8(%r11), %r12
+	movq	_pypydtcount at GOTPCREL(%rip), %r10
+	movl	(%r10), %eax
+	movq	_pypy_debug_tracebacks at GOTPCREL(%rip), %r9
+	movslq	%eax,%rdx
+	salq	$4, %rdx
+	leaq	_loc.235055(%rip), %rcx
+	movq	%rcx, (%rdx,%r9)
+	movq	%rbp, 8(%rdx,%r9)
+	incl	%eax
+	andl	$127, %eax
+	movl	%eax, (%r10)
+	movq	_pypy_g_typeinfo at GOTPCREL(%rip), %r8
+	leaq	59688(%r8), %rax
+	cmpq	%rax, %rbp
+	je	L1040
+	leaq	59544(%r8), %rax
+	cmpq	%rax, %rbp
+	je	L1040
+L1042:
+	movq	$0, 8(%r11)
+	movq	$0, (%r11)
+	/* keepalive %r12 */
+	/* keepalive %rbx */
+L1043:
+L1044:
+	/* GC_NOCOLLECT pypy_g_ll_issubclass */
+L1046:
+L1047:
+	/* keepalive %rbx */
+	cmpq	$0, 8(%rbx)
+	jle	L1048
+L1049:
+	/* keepalive %rbx */
+L1050:
+	leaq	59688(%r8), %rcx
+	movq	%rcx, (%r11)
+	movq	_pypy_g_exceptions_AssertionError_6 at GOTPCREL(%rip), %rax
+	movq	%rax, 8(%r11)
+	movl	(%r10), %edx
+	movslq	%edx,%rax
+	salq	$4, %rax
+	movq	$0, (%rax,%r9)
+	movq	%rcx, 8(%rax,%r9)
+	incl	%edx
+	andl	$127, %edx
+	/* GC_NOCOLLECT pypy_g_RPyRaiseException */
+	movslq	%edx,%rax
+	salq	$4, %rax
+	leaq	_loc.235061(%rip), %rbx
+	movq	%rbx, (%rax,%r9)
+	movq	$0, 8(%rax,%r9)
+	incl	%edx
+	andl	$127, %edx
+	movl	%edx, (%r10)
+	xorl	%eax, %eax
+	.align 4,0x90
+L981:
+	movq	40(%rsp), %rbx
+	movq	48(%rsp), %rbp
+	movq	56(%rsp), %r12
+	movq	64(%rsp), %r13
+	movq	72(%rsp), %r14
+	movq	80(%rsp), %r15
+	addq	$88, %rsp
+	ret
+	.align 4,0x90
+L1056:
+	mov	%edx, %eax
+	leaq	L978(%rip), %rdx
+	movslq	(%rdx,%rax,4),%rax
+	addq	%rdx, %rax
+	jmp	*%rax
+	.align 2,0x90
+L978:
+	.long	L972-L978
+	.long	L973-L978
+	.long	L974-L978
+	.long	L975-L978
+	.long	L976-L978
+	.long	L977-L978
+L972:
+	/* keepalive %rdi */
+	/* keepalive %rsi */
+	call	_pypy_g___mm_lt_0_perform_call
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | %rbx, %rbp}
+	movq	___gcmapend at GOTPCREL(%rip), %rdx
+	movq	%rbx, %r15
+	/* GCROOT %r15 */
+	movq	%rbp, %r14
+	/* GCROOT %r14 */
+	movq	%rbx, %r12
+	/* GCROOT %r12 */
+	/* GCROOT %rbp */
+	movq	_pypy_g_ExcData at GOTPCREL(%rip), %r11
+	movq	(%r11), %rbx
+	testq	%rbx, %rbx
+	je	L981
+L979:
+	movq	8(%r11), %r13
+	movq	_pypydtcount at GOTPCREL(%rip), %r10
+	movl	(%r10), %eax
+	movq	_pypy_debug_tracebacks at GOTPCREL(%rip), %r9
+	movslq	%eax,%rdx
+	salq	$4, %rdx
+	leaq	_loc.234970(%rip), %rcx
+	movq	%rcx, (%rdx,%r9)
+	movq	%rbx, 8(%rdx,%r9)
+	incl	%eax
+	andl	$127, %eax
+	movl	%eax, (%r10)
+	movq	_pypy_g_typeinfo at GOTPCREL(%rip), %r8
+	leaq	59688(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L982
+	leaq	59544(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L982
+L984:
+	movq	$0, 8(%r11)
+	movq	$0, (%r11)
+	/* keepalive %r13 */
+	/* keepalive %r15 */
+	/* keepalive %r14 */
+	/* keepalive %r12 */
+	/* keepalive %rbp */
+L985:
+	movq	1712(%r8), %rcx
+	movq	(%rbx), %rdx
+	subq	%rcx, %rdx
+	movq	1720(%r8), %rax
+L986:
+	/* GC_NOCOLLECT pypy_g_ll_issubclass */
+	subq	%rcx, %rax
+	cmpq	%rax, %rdx
+	jae	L1058
+L987:
+	/* keepalive %r12 */
+	/* keepalive %rbp */
+	movq	%rbp, %rsi
+	movq	%r12, %rdi
+	movq	40(%rsp), %rbx
+	movq	48(%rsp), %rbp
+	movq	56(%rsp), %r12
+	movq	64(%rsp), %r13
+	movq	72(%rsp), %r14
+	movq	80(%rsp), %r15
+	addq	$88, %rsp
+	jmp	_pypy_g_comparison_lt_impl
+L973:
+	/* keepalive %rdi */
+	/* keepalive %rsi */
+	call	_pypy_g___mm_le_0_perform_call
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | %rbx, %rbp}
+	movq	___gcmapend at GOTPCREL(%rip), %rdx
+	movq	%rbx, %r15
+	/* GCROOT %r15 */
+	movq	%rbp, %r14
+	/* GCROOT %r14 */
+	movq	%rbx, %r12
+	/* GCROOT %r12 */
+	/* GCROOT %rbp */
+	movq	_pypy_g_ExcData at GOTPCREL(%rip), %r11
+	movq	(%r11), %rbx
+	testq	%rbx, %rbx
+	je	L981
+L989:
+	movq	8(%r11), %r13
+	movq	_pypydtcount at GOTPCREL(%rip), %r10
+	movl	(%r10), %eax
+	movq	_pypy_debug_tracebacks at GOTPCREL(%rip), %r9
+	movslq	%eax,%rdx
+	salq	$4, %rdx
+	leaq	_loc.234982(%rip), %rcx
+	movq	%rcx, (%rdx,%r9)
+	movq	%rbx, 8(%rdx,%r9)
+	incl	%eax
+	andl	$127, %eax
+	movl	%eax, (%r10)
+	movq	_pypy_g_typeinfo at GOTPCREL(%rip), %r8
+	leaq	59688(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L991
+	leaq	59544(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L991
+L993:
+	movq	$0, 8(%r11)
+	movq	$0, (%r11)
+	/* keepalive %r13 */
+	/* keepalive %r15 */
+	/* keepalive %r14 */
+	/* keepalive %r12 */
+	/* keepalive %rbp */
+L994:
+	movq	1712(%r8), %rcx
+	movq	(%rbx), %rdx
+	subq	%rcx, %rdx
+	movq	1720(%r8), %rax
+L995:
+	/* GC_NOCOLLECT pypy_g_ll_issubclass */
+	subq	%rcx, %rax
+	cmpq	%rax, %rdx
+	jae	L1059
+L996:
+	/* keepalive %r12 */
+	/* keepalive %rbp */
+	movq	%rbp, %rsi
+	movq	%r12, %rdi
+	movq	40(%rsp), %rbx
+	movq	48(%rsp), %rbp
+	movq	56(%rsp), %r12
+	movq	64(%rsp), %r13
+	movq	72(%rsp), %r14
+	movq	80(%rsp), %r15
+	addq	$88, %rsp
+	jmp	_pypy_g_comparison_le_impl
+L974:
+	/* keepalive %rdi */
+	/* keepalive %rsi */
+	call	_pypy_g___mm_eq_0_perform_call
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | %rbx, %rbp}
+	movq	___gcmapend at GOTPCREL(%rip), %rdx
+	movq	%rbx, %r15
+	/* GCROOT %r15 */
+	movq	%rbp, %r14
+	/* GCROOT %r14 */
+	movq	%rbx, %r12
+	/* GCROOT %r12 */
+	/* GCROOT %rbp */
+	movq	_pypy_g_ExcData at GOTPCREL(%rip), %r11
+	movq	(%r11), %rbx
+	testq	%rbx, %rbx
+	je	L981
+L998:
+	movq	8(%r11), %r13
+	movq	_pypydtcount at GOTPCREL(%rip), %r10
+	movl	(%r10), %eax
+	movq	_pypy_debug_tracebacks at GOTPCREL(%rip), %r9
+	movslq	%eax,%rdx
+	salq	$4, %rdx
+	leaq	_loc.234994(%rip), %rcx
+	movq	%rcx, (%rdx,%r9)
+	movq	%rbx, 8(%rdx,%r9)
+	incl	%eax
+	andl	$127, %eax
+	movl	%eax, (%r10)
+	movq	_pypy_g_typeinfo at GOTPCREL(%rip), %r8
+	leaq	59688(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L1000
+	leaq	59544(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L1000
+L1002:
+	movq	$0, 8(%r11)
+	movq	$0, (%r11)
+	/* keepalive %r13 */
+	/* keepalive %r15 */
+	/* keepalive %r14 */
+	/* keepalive %rbp */
+	/* keepalive %r12 */
+L1003:
+	movq	1712(%r8), %rcx
+	movq	(%rbx), %rdx
+	subq	%rcx, %rdx
+	movq	1720(%r8), %rax
+L1004:
+	/* GC_NOCOLLECT pypy_g_ll_issubclass */
+	subq	%rcx, %rax
+	cmpq	%rax, %rdx
+	jae	L1060
+L1005:
+	/* keepalive %r12 */
+	/* keepalive %rbp */
+	movq	%rbp, %rsi
+	movq	%r12, %rdi
+	movq	40(%rsp), %rbx
+	movq	48(%rsp), %rbp
+	movq	56(%rsp), %r12
+	movq	64(%rsp), %r13
+	movq	72(%rsp), %r14
+	movq	80(%rsp), %r15
+	addq	$88, %rsp
+	jmp	_pypy_g_comparison_eq_impl
+L975:
+	/* keepalive %rdi */
+	/* keepalive %rsi */
+	call	_pypy_g___mm_ne_0_perform_call
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | %rbx, %rbp}
+	movq	___gcmapend at GOTPCREL(%rip), %rdx
+	movq	%rbx, %r15
+	/* GCROOT %r15 */
+	movq	%rbp, %r14
+	/* GCROOT %r14 */
+	movq	%rbx, %r12
+	/* GCROOT %r12 */
+	/* GCROOT %rbp */
+	movq	_pypy_g_ExcData at GOTPCREL(%rip), %r11
+	movq	(%r11), %rbx
+	testq	%rbx, %rbx
+	je	L981
+L1007:
+	movq	8(%r11), %r13
+	movq	_pypydtcount at GOTPCREL(%rip), %r10
+	movl	(%r10), %eax
+	movq	_pypy_debug_tracebacks at GOTPCREL(%rip), %r9
+	movslq	%eax,%rdx
+	salq	$4, %rdx
+	leaq	_loc.235006(%rip), %rcx
+	movq	%rcx, (%rdx,%r9)
+	movq	%rbx, 8(%rdx,%r9)
+	incl	%eax
+	andl	$127, %eax
+	movl	%eax, (%r10)
+	movq	_pypy_g_typeinfo at GOTPCREL(%rip), %r8
+	leaq	59688(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L1009
+	leaq	59544(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L1009
+L1011:
+	movq	$0, 8(%r11)
+	movq	$0, (%r11)
+	/* keepalive %r13 */
+	/* keepalive %r15 */
+	/* keepalive %r14 */
+	/* keepalive %r12 */
+	/* keepalive %rbp */
+L1012:
+	movq	1712(%r8), %rcx
+	movq	(%rbx), %rdx
+	subq	%rcx, %rdx
+	movq	1720(%r8), %rax
+L1013:
+	/* GC_NOCOLLECT pypy_g_ll_issubclass */
+	subq	%rcx, %rax
+	cmpq	%rax, %rdx
+	jae	L1061
+L1014:
+	/* keepalive %r12 */
+	/* keepalive %rbp */
+	movq	%rbp, %rsi
+	movq	%r12, %rdi
+	movq	40(%rsp), %rbx
+	movq	48(%rsp), %rbp
+	movq	56(%rsp), %r12
+	movq	64(%rsp), %r13
+	movq	72(%rsp), %r14
+	movq	80(%rsp), %r15
+	addq	$88, %rsp
+	jmp	_pypy_g_comparison_ne_impl
+L976:
+	/* keepalive %rdi */
+	/* keepalive %rsi */
+	call	_pypy_g___mm_gt_0_perform_call
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | %rbx, %rbp}
+	movq	___gcmapend at GOTPCREL(%rip), %rdx
+	movq	%rbx, %r15
+	/* GCROOT %r15 */
+	movq	%rbp, %r14
+	/* GCROOT %r14 */
+	movq	%rbx, %r12
+	/* GCROOT %r12 */
+	/* GCROOT %rbp */
+	movq	_pypy_g_ExcData at GOTPCREL(%rip), %r11
+	movq	(%r11), %rbx
+	testq	%rbx, %rbx
+	je	L981
+L1016:
+	movq	8(%r11), %r13
+	movq	_pypydtcount at GOTPCREL(%rip), %r10
+	movl	(%r10), %eax
+	movq	_pypy_debug_tracebacks at GOTPCREL(%rip), %r9
+	movslq	%eax,%rdx
+	salq	$4, %rdx
+	leaq	_loc.235018(%rip), %rcx
+	movq	%rcx, (%rdx,%r9)
+	movq	%rbx, 8(%rdx,%r9)
+	incl	%eax
+	andl	$127, %eax
+	movl	%eax, (%r10)
+	movq	_pypy_g_typeinfo at GOTPCREL(%rip), %r8
+	leaq	59688(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L1018
+	leaq	59544(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L1018
+L1020:
+	movq	$0, 8(%r11)
+	movq	$0, (%r11)
+	/* keepalive %r13 */
+	/* keepalive %r15 */
+	/* keepalive %r14 */
+	/* keepalive %rbp */
+	/* keepalive %r12 */
+L1021:
+	movq	1712(%r8), %rcx
+	movq	(%rbx), %rdx
+	subq	%rcx, %rdx
+	movq	1720(%r8), %rax
+L1022:
+	/* GC_NOCOLLECT pypy_g_ll_issubclass */
+	subq	%rcx, %rax
+	cmpq	%rax, %rdx
+	jae	L1062
+L1023:
+	/* keepalive %r12 */
+	/* keepalive %rbp */
+	movq	%rbp, %rsi
+	movq	%r12, %rdi
+	movq	40(%rsp), %rbx
+	movq	48(%rsp), %rbp
+	movq	56(%rsp), %r12
+	movq	64(%rsp), %r13
+	movq	72(%rsp), %r14
+	movq	80(%rsp), %r15
+	addq	$88, %rsp
+	jmp	_pypy_g_comparison_gt_impl
+L977:
+	/* keepalive %rdi */
+	/* keepalive %rsi */
+	call	_pypy_g___mm_ge_0_perform_call
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | %rbx, %rbp}
+	movq	___gcmapend at GOTPCREL(%rip), %rdx
+	movq	%rbx, %r15
+	/* GCROOT %r15 */
+	movq	%rbp, %r14
+	/* GCROOT %r14 */
+	movq	%rbx, %r12
+	/* GCROOT %r12 */
+	/* GCROOT %rbp */
+	movq	_pypy_g_ExcData at GOTPCREL(%rip), %r11
+	movq	(%r11), %rbx
+	testq	%rbx, %rbx
+	je	L981
+L1025:
+	movq	8(%r11), %r13
+	movq	_pypydtcount at GOTPCREL(%rip), %r10
+	movl	(%r10), %eax
+	movq	_pypy_debug_tracebacks at GOTPCREL(%rip), %r9
+	movslq	%eax,%rdx
+	salq	$4, %rdx
+	leaq	_loc.235030(%rip), %rcx
+	movq	%rcx, (%rdx,%r9)
+	movq	%rbx, 8(%rdx,%r9)
+	incl	%eax
+	andl	$127, %eax
+	movl	%eax, (%r10)
+	movq	_pypy_g_typeinfo at GOTPCREL(%rip), %r8
+	leaq	59688(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L1027
+	leaq	59544(%r8), %rax
+	cmpq	%rax, %rbx
+	je	L1027
+L1029:
+	movq	$0, 8(%r11)
+	movq	$0, (%r11)
+	/* keepalive %r13 */
+	/* keepalive %r15 */
+	/* keepalive %r14 */
+	/* keepalive %r12 */
+	/* keepalive %rbp */
+L1030:
+	movq	1712(%r8), %rcx
+	movq	(%rbx), %rdx
+	subq	%rcx, %rdx
+	movq	1720(%r8), %rax
+L1031:
+	/* GC_NOCOLLECT pypy_g_ll_issubclass */
+	subq	%rcx, %rax
+	cmpq	%rax, %rdx
+	jae	L1063
+L1032:
+	/* keepalive %r12 */
+	/* keepalive %rbp */
+	movq	%rbp, %rsi
+	movq	%r12, %rdi
+	movq	40(%rsp), %rbx
+	movq	48(%rsp), %rbp
+	movq	56(%rsp), %r12
+	movq	64(%rsp), %r13
+	movq	72(%rsp), %r14
+	movq	80(%rsp), %r15
+	addq	$88, %rsp
+	jmp	_pypy_g_comparison_ge_impl
+L1045:
+L1034:
+	movl	$16, %esi
+	call	_pypy_g_MiniMarkGC_collect_and_reserve
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | }
+	movq	_pypy_g_ExcData at GOTPCREL(%rip), %r11
+	movq	%rax, %rdx
+	movq	%rax, %rbx
+	cmpq	$0, (%r11)
+	je	L1036
+L1053:
+	movq	_pypydtcount at GOTPCREL(%rip), %rsi
+	movl	(%rsi), %edx
+	movq	_pypy_debug_tracebacks at GOTPCREL(%rip), %rcx
+	movslq	%edx,%rax
+	salq	$4, %rax
+	leaq	_loc.235064(%rip), %rbx
+	movq	%rbx, (%rax,%rcx)
+	movq	$0, 8(%rcx,%rax)
+	incl	%edx
+	andl	$127, %edx
+L1052:
+	movslq	%edx,%rax
+	salq	$4, %rax
+	leaq	_loc.235062(%rip), %rbx
+	movq	%rbx, (%rax,%rcx)
+	movq	$0, 8(%rax,%rcx)
+	incl	%edx
+	andl	$127, %edx
+	movl	%edx, (%rsi)
+	xorl	%eax, %eax
+	jmp	L981
+L1027:
+	movq	%r8, 24(%rsp)
+	movq	%r9, 16(%rsp)
+	movq	%r10, 8(%rsp)
+	movq	%r11, (%rsp)
+	call	_pypy_debug_catch_fatal_exception
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | }
+	movq	(%rsp), %r11
+	movq	8(%rsp), %r10
+	movq	16(%rsp), %r9
+	movq	24(%rsp), %r8
+	jmp	L1029
+L1018:
+	movq	%r8, 24(%rsp)
+	movq	%r9, 16(%rsp)
+	movq	%r10, 8(%rsp)
+	movq	%r11, (%rsp)
+	call	_pypy_debug_catch_fatal_exception
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | }
+	movq	(%rsp), %r11
+	movq	8(%rsp), %r10
+	movq	16(%rsp), %r9
+	movq	24(%rsp), %r8
+	jmp	L1020
+L1009:
+	movq	%r8, 24(%rsp)
+	movq	%r9, 16(%rsp)
+	movq	%r10, 8(%rsp)
+	movq	%r11, (%rsp)
+	call	_pypy_debug_catch_fatal_exception
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | }
+	movq	(%rsp), %r11
+	movq	8(%rsp), %r10
+	movq	16(%rsp), %r9
+	movq	24(%rsp), %r8
+	jmp	L1011
+L1000:
+	movq	%r8, 24(%rsp)
+	movq	%r9, 16(%rsp)
+	movq	%r10, 8(%rsp)
+	movq	%r11, (%rsp)
+	call	_pypy_debug_catch_fatal_exception
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | }
+	movq	(%rsp), %r11
+	movq	8(%rsp), %r10
+	movq	16(%rsp), %r9
+	movq	24(%rsp), %r8
+	jmp	L1002
+L991:
+	movq	%r8, 24(%rsp)
+	movq	%r9, 16(%rsp)
+	movq	%r10, 8(%rsp)
+	movq	%r11, (%rsp)
+	call	_pypy_debug_catch_fatal_exception
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | }
+	movq	(%rsp), %r11
+	movq	8(%rsp), %r10
+	movq	16(%rsp), %r9
+	movq	24(%rsp), %r8
+	jmp	L993
+L982:
+	movq	%r8, 24(%rsp)
+	movq	%r9, 16(%rsp)
+	movq	%r10, 8(%rsp)
+	movq	%r11, (%rsp)
+	call	_pypy_debug_catch_fatal_exception
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | }
+	movq	(%rsp), %r11
+	movq	8(%rsp), %r10
+	movq	16(%rsp), %r9
+	movq	24(%rsp), %r8
+	jmp	L984
+L1059:
+L997:
+	movq	%rbx, (%r11)
+	movq	%r13, 8(%r11)
+	movl	(%r10), %edx
+	movslq	%edx,%rax
+	salq	$4, %rax
+	movq	$-1, (%rax,%r9)
+	movq	%rbx, 8(%rax,%r9)
+	incl	%edx
+	andl	$127, %edx
+	movl	%edx, (%r10)
+	/* GC_NOCOLLECT pypy_g_RPyReRaiseException */
+	xorl	%eax, %eax
+	jmp	L981
+L1058:
+L988:
+	movq	%rbx, (%r11)
+	movq	%r13, 8(%r11)
+	movl	(%r10), %edx
+	movslq	%edx,%rax
+	salq	$4, %rax
+	movq	$-1, (%rax,%r9)
+	movq	%rbx, 8(%rax,%r9)
+	incl	%edx
+	andl	$127, %edx
+	movl	%edx, (%r10)
+	/* GC_NOCOLLECT pypy_g_RPyReRaiseException */
+	xorl	%eax, %eax
+	jmp	L981
+L1061:
+L1015:
+	movq	%rbx, (%r11)
+	movq	%r13, 8(%r11)
+	movl	(%r10), %edx
+	movslq	%edx,%rax
+	salq	$4, %rax
+	movq	$-1, (%rax,%r9)
+	movq	%rbx, 8(%rax,%r9)
+	incl	%edx
+	andl	$127, %edx
+	movl	%edx, (%r10)
+	/* GC_NOCOLLECT pypy_g_RPyReRaiseException */
+	xorl	%eax, %eax
+	jmp	L981
+L1060:
+L1006:
+	movq	%rbx, (%r11)
+	movq	%r13, 8(%r11)
+	movl	(%r10), %edx
+	movslq	%edx,%rax
+	salq	$4, %rax
+	movq	$-1, (%rax,%r9)
+	movq	%rbx, 8(%rax,%r9)
+	incl	%edx
+	andl	$127, %edx
+	movl	%edx, (%r10)
+	/* GC_NOCOLLECT pypy_g_RPyReRaiseException */
+	xorl	%eax, %eax
+	jmp	L981
+L1063:
+L1033:
+	movq	%rbx, (%r11)
+	movq	%r13, 8(%r11)
+	movl	(%r10), %edx
+	movslq	%edx,%rax
+	salq	$4, %rax
+	movq	$-1, (%rax,%r9)
+	movq	%rbx, 8(%rax,%r9)
+	incl	%edx
+	andl	$127, %edx
+	movl	%edx, (%r10)
+	/* GC_NOCOLLECT pypy_g_RPyReRaiseException */
+	xorl	%eax, %eax
+	jmp	L981
+L1062:
+L1024:
+	movq	%rbx, (%r11)
+	movq	%r13, 8(%r11)
+	movl	(%r10), %edx
+	movslq	%edx,%rax
+	salq	$4, %rax
+	movq	$-1, (%rax,%r9)
+	movq	%rbx, 8(%rax,%r9)
+	incl	%edx
+	andl	$127, %edx
+	movl	%edx, (%r10)
+	/* GC_NOCOLLECT pypy_g_RPyReRaiseException */
+	xorl	%eax, %eax
+	jmp	L981
+L1040:
+	movq	%r8, 24(%rsp)
+	movq	%r9, 16(%rsp)
+	movq	%r10, 8(%rsp)
+	movq	%r11, (%rsp)
+	call	_pypy_debug_catch_fatal_exception
+	;; expected {88(%rsp) | 40(%rsp), 56(%rsp), 64(%rsp), 72(%rsp), 80(%rsp), 48(%rsp) | }
+	movq	(%rsp), %r11
+	movq	8(%rsp), %r10
+	movq	16(%rsp), %r9
+	movq	24(%rsp), %r8
+	jmp	L1042
+L1048:
+	/* keepalive %rbx */
+	/* keepalive %r12 */
+L1051:
+	movq	%rbp, (%r11)
+	movq	%r12, 8(%r11)
+	movl	(%r10), %edx
+	movslq	%edx,%rax
+	salq	$4, %rax
+	movq	$-1, (%rax,%r9)
+	movq	%rbp, 8(%rax,%r9)
+	incl	%edx
+	andl	$127, %edx
+	movl	%edx, (%r10)
+	/* GC_NOCOLLECT pypy_g_RPyReRaiseException */
+	xorl	%eax, %eax
+	jmp	L981
+L1057:
+L1039:
+	movq	_pypy_g_typeinfo at GOTPCREL(%rip), %rsi
+	addq	$59688, %rsi
+	movq	%rsi, (%r11)
+	movq	_pypy_g_exceptions_AssertionError_5 at GOTPCREL(%rip), %rax
+	movq	%rax, 8(%r11)
+	movq	_pypydtcount at GOTPCREL(%rip), %rdi
+	movl	(%rdi), %edx
+	movq	_pypy_debug_tracebacks at GOTPCREL(%rip), %rcx
+	movslq	%edx,%rax
+	salq	$4, %rax
+	movq	$0, (%rax,%rcx)
+	movq	%rsi, 8(%rcx,%rax)
+	incl	%edx
+	andl	$127, %edx
+	/* GC_NOCOLLECT pypy_g_RPyRaiseException */
+	movslq	%edx,%rax
+	salq	$4, %rax
+	leaq	_loc.235054(%rip), %rbx
+	movq	%rbx, (%rax,%rcx)
+	movq	$0, 8(%rcx,%rax)
+	incl	%edx
+	andl	$127, %edx
+	movl	%edx, (%rdi)
+	xorl	%eax, %eax
+	jmp	L981
+LFE302:
+	.align 4,0x90

Modified: pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
==============================================================================
--- pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	(original)
+++ pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	Thu Oct 28 15:15:36 2010
@@ -1106,6 +1106,9 @@
     format = 'darwin64'
     function_names_prefix = '_'
 
+    LABEL = ElfFunctionGcRootTracker32.LABEL
+    r_jmptable_item = re.compile(r"\t.(?:long|quad)\t"+LABEL+"(-\"?[A-Za-z0-9$]+\"?)?\s*$")
+
     r_functionstart = re.compile(r"_(\w+):\s*$")
     OFFSET_LABELS   = 0
 


From afa at codespeak.net  Thu Oct 28 15:51:56 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 15:51:56 +0200 (CEST)
Subject: [pypy-svn] r78401 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101028135156.6F5DC282C1E@codespeak.net>

Author: afa
Date: Thu Oct 28 15:51:55 2010
New Revision: 78401

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_io.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
Log:
Test and fix for BlockingIOError


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_io.py	Thu Oct 28 15:51:55 2010
@@ -18,7 +18,7 @@
         self.written = written
 
 W_BlockingIOError.typedef = TypeDef(
-    'BlockingIOError',
+    'BlockingIOError', W_IOError.typedef,
     __doc__ = ("Exception raised when I/O would block "
                "on a non-blocking I/O stream"),
     __new__  = generic_new_descr(W_BlockingIOError),

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	Thu Oct 28 15:51:55 2010
@@ -42,6 +42,16 @@
         import _io
         e = _io.UnsupportedOperation("seek")
 
+    def test_blockingerror(self):
+        import _io
+        try:
+            raise _io.BlockingIOError(42, "test blocking", 123)
+        except IOError, e:
+            assert isinstance(e, _io.BlockingIOError)
+            assert e.errno == 42
+            assert e.strerror == "test blocking"
+            assert e.characters_written == 123
+
     def test_destructor(self):
         import io
         io.IOBase()


From afa at codespeak.net  Thu Oct 28 16:04:20 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 16:04:20 +0200 (CEST)
Subject: [pypy-svn] r78402 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101028140420.B603836E076@codespeak.net>

Author: afa
Date: Thu Oct 28 16:04:19 2010
New Revision: 78402

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Log:
BufferedReader.readinto()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Thu Oct 28 16:04:19 2010
@@ -38,6 +38,19 @@
     def detach_w(self, space):
         self._unsupportedoperation(space, "detach")
 
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def readinto_w(self, space, w_buffer):
+        rwbuffer = space.rwbuffer_w(w_buffer)
+        length = rwbuffer.getlength()
+        w_data = space.call_method(self, "read", space.wrap(length))
+
+        if not space.isinstance_w(w_data, space.w_str):
+            raise OperationError(space.w_TypeError, space.wrap(
+                "read() should return bytes"))
+        data = space.str_w(w_data)
+        rwbuffer.setslice(0, data)
+        return space.wrap(len(data))
+
 W_BufferedIOBase.typedef = TypeDef(
     '_BufferedIOBase', W_IOBase.typedef,
     __new__ = generic_new_descr(W_BufferedIOBase),
@@ -45,6 +58,7 @@
     read1 = interp2app(W_BufferedIOBase.read1_w),
     write = interp2app(W_BufferedIOBase.write_w),
     detach = interp2app(W_BufferedIOBase.detach_w),
+    readinto = interp2app(W_BufferedIOBase.readinto_w),
     )
 
 class BufferedMixin:

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Thu Oct 28 16:04:19 2010
@@ -55,6 +55,15 @@
         assert raw.nbreads == 3
         f.close()
 
+    def test_readinto(self):
+        import _io
+        a = bytearray('x' * 10)
+        raw = _io.FileIO(self.tmpfile)
+        f = _io.BufferedReader(raw)
+        assert f.readinto(a) == 5
+        f.close()
+        assert a == 'a\nb\ncxxxxx'
+
     def test_seek(self):
         import _io
         raw = _io.FileIO(self.tmpfile)


From afa at codespeak.net  Thu Oct 28 16:15:44 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 16:15:44 +0200 (CEST)
Subject: [pypy-svn] r78403 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101028141544.BC367282C1E@codespeak.net>

Author: afa
Date: Thu Oct 28 16:15:43 2010
New Revision: 78403

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Log:
Deprecate BufferedWriter.max_buffer_size


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Thu Oct 28 16:15:43 2010
@@ -123,6 +123,10 @@
         self._check_init(space)
         W_IOBase._check_closed(self, space, message)
 
+    def _deprecated_max_buffer_size(self, space):
+        space.warn("max_buffer_size is deprecated",
+                   space.w_DeprecationWarning)
+
     def _raw_tell(self, space):
         w_pos = space.call_method(self.raw, "tell")
         pos = space.r_longlong_w(w_pos)
@@ -503,8 +507,12 @@
     )
 
 class W_BufferedWriter(BufferedMixin, W_BufferedIOBase):
-    @unwrap_spec('self', ObjSpace, W_Root, int)
-    def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE):
+    @unwrap_spec('self', ObjSpace, W_Root, int, int)
+    def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE,
+                   max_buffer_size=-234):
+        if max_buffer_size != -234:
+            self._deprecated_max_buffer_size(space)
+
         self.state = STATE_ZERO
         raw = space.interp_w(W_IOBase, w_raw)
         raw.check_writable_w(space)

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Thu Oct 28 16:15:43 2010
@@ -131,6 +131,17 @@
         raises(ValueError, b.flush)
         raises(ValueError, b.close)
 
+    def test_deprecated_max_buffer_size(self):
+        import _io, warnings
+        raw = _io.FileIO(self.tmpfile, 'w')
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter("always")
+            f = _io.BufferedWriter(raw, max_buffer_size=8192)
+        f.close()
+        assert len(w) == 1
+        assert str(w[0].message) == "max_buffer_size is deprecated"
+        assert w[0].category is DeprecationWarning
+
     def test_check_several_writes(self):
         import _io
         raw = _io.FileIO(self.tmpfile, 'w')


From afa at codespeak.net  Thu Oct 28 16:36:56 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 16:36:56 +0200 (CEST)
Subject: [pypy-svn] r78404 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101028143656.9759836E076@codespeak.net>

Author: afa
Date: Thu Oct 28 16:36:55 2010
New Revision: 78404

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Log:
implement tell().
Fix an issue when seek() crosses the buffer boundaries


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Thu Oct 28 16:36:55 2010
@@ -153,6 +153,12 @@
             return self.raw_pos - self.pos
         return 0
 
+    @unwrap_spec('self', ObjSpace)
+    def tell_w(self, space):
+        self._check_init(space)
+        pos = self._raw_tell(space) - self._raw_offset()
+        return space.wrap(pos)
+
     @unwrap_spec('self', ObjSpace, r_longlong, int)
     def seek_w(self, space, pos, whence=0):
         self._check_init(space)
@@ -425,6 +431,8 @@
             data = rffi.charpsize2str(rffi.ptradd(self.buffer, self.pos),
                                       current_size)
             builder.append(data)
+            remaining -= len(data)
+            written += len(data)
         self._reader_reset_buf()
 
         # XXX potential bug in CPython? The following is not enabled.
@@ -446,6 +454,7 @@
             size = len(data)
             if size == 0:
                 return builder.build()
+            builder.append(data)
             remaining -= size
             written += size
 
@@ -499,6 +508,7 @@
 
     # from the mixin class
     seek = interp2app(W_BufferedReader.seek_w),
+    tell = interp2app(W_BufferedReader.tell_w),
     close = interp2app(W_BufferedReader.close_w),
     flush = interp2app(W_BufferedReader.flush_w),
     detach = interp2app(W_BufferedReader.detach_w),
@@ -659,6 +669,7 @@
 
     # from the mixin class
     seek = interp2app(W_BufferedWriter.seek_w),
+    tell = interp2app(W_BufferedWriter.tell_w),
     close = interp2app(W_BufferedWriter.close_w),
     fileno = interp2app(W_BufferedWriter.fileno_w),
     detach = interp2app(W_BufferedWriter.detach_w),

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Thu Oct 28 16:36:55 2010
@@ -94,6 +94,22 @@
         assert not raw.closed
         raw.close()
 
+    def test_tell(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile)
+        f = _io.BufferedReader(raw, buffer_size=2)
+        assert f.tell() == 0
+        d1 = f.read(1)
+        assert f.tell() == 1
+        d2 = f.read(2)
+        assert f.tell() == 3
+        assert f.seek(0) == 0
+        assert f.tell() == 0
+        d3 = f.read(3)
+        assert f.tell() == 3
+        assert d1 + d2 == d3
+        f.close()
+
 class AppTestBufferedWriter:
     def setup_class(cls):
         cls.space = gettestobjspace(usemodules=['_io'])


From afa at codespeak.net  Thu Oct 28 17:03:15 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 17:03:15 +0200 (CEST)
Subject: [pypy-svn] r78409 - pypy/trunk/pypy/translator/c/gcc
Message-ID: <20101028150315.94620282C1E@codespeak.net>

Author: afa
Date: Thu Oct 28 17:03:14 2010
New Revision: 78409

Modified:
   pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
Log:
Add darwin64 variant for the pypy_asm_stackwalk() function


Modified: pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
==============================================================================
--- pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	(original)
+++ pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	Thu Oct 28 17:03:14 2010
@@ -1655,7 +1655,8 @@
         elif self.format == 'elf64':
             print >> output, "\t.text"
             print >> output, "\t.globl %s" % _globalname('pypy_asm_stackwalk')
-            print >> output, "\t.type pypy_asm_stackwalk, @function"
+            _variant(elf64='.type pypy_asm_stackwalk, @function',
+                     darwin64='')
             print >> output, "%s:" % _globalname('pypy_asm_stackwalk')
 
             print >> output, """\
@@ -1700,8 +1701,9 @@
             /* the return value is the one of the 'call' above, */
             /* because %rax (and possibly %rdx) are unmodified  */
             ret
-            .size pypy_asm_stackwalk, .-pypy_asm_stackwalk
             """
+            _variant(elf64='.size pypy_asm_stackwalk, .-pypy_asm_stackwalk',
+                     darwin64='')
         else:
             print >> output, "\t.text"
             print >> output, "\t.globl %s" % _globalname('pypy_asm_stackwalk')
@@ -1828,6 +1830,7 @@
             _variant(elf='.section\t.rodata',
                      elf64='.section\t.rodata',
                      darwin='.const',
+                     darwin64='.const',
                      mingw32='')
 
             print >> output, """\


From afa at codespeak.net  Thu Oct 28 17:12:25 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 17:12:25 +0200 (CEST)
Subject: [pypy-svn] r78410 - pypy/trunk/pypy/translator/c/gcc
Message-ID: <20101028151225.1741436E076@codespeak.net>

Author: afa
Date: Thu Oct 28 17:12:24 2010
New Revision: 78410

Modified:
   pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
Log:
Fix pypy_asm_stackwalk() for darwin64, again


Modified: pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
==============================================================================
--- pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	(original)
+++ pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	Thu Oct 28 17:12:24 2010
@@ -1652,7 +1652,7 @@
                }
             }
             """
-        elif self.format == 'elf64':
+        elif self.format in ('elf64', 'darwin64'):
             print >> output, "\t.text"
             print >> output, "\t.globl %s" % _globalname('pypy_asm_stackwalk')
             _variant(elf64='.type pypy_asm_stackwalk, @function',


From afa at codespeak.net  Thu Oct 28 17:24:24 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 17:24:24 +0200 (CEST)
Subject: [pypy-svn] r78412 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101028152424.6FE5D282BE8@codespeak.net>

Author: afa
Date: Thu Oct 28 17:24:23 2010
New Revision: 78412

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
Log:
implement truncate() on buffered files


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bufferedio.py	Thu Oct 28 17:24:23 2010
@@ -170,7 +170,9 @@
             # Check if seeking leaves us inside the current buffer, so as to
             # return quickly if possible. Also, we needn't take the lock in
             # this fast path.
-            current = self._raw_tell(space)
+            if self.abs_pos == -1:
+                self._raw_tell(space)
+            current = self.abs_pos
             available = self._readahead()
             if available > 0:
                 if whence == 0:
@@ -293,6 +295,23 @@
         self._check_init(space)
         return space.call_method(self.raw, "fileno")
 
+    @unwrap_spec('self', ObjSpace, W_Root)
+    def truncate_w(self, space, w_size=None):
+        self._check_init(space)
+        with self.lock:
+            if self.writable:
+                self._writer_flush_unlocked(space)
+            if self.readable:
+                if space.is_w(w_size, space.w_None):
+                    # Rewind the raw stream so that its position corresponds
+                    # to the current logical position
+                    self._raw_seek(space, -self._raw_offset(), 1)
+                self._reader_reset_buf()
+            # invalidate cached position
+            self.abs_pos = -1
+
+            return space.call_method(self.raw, "truncate", w_size)
+
 class W_BufferedReader(BufferedMixin, W_BufferedIOBase):
     @unwrap_spec('self', ObjSpace, W_Root, int)
     def descr_init(self, space, w_raw, buffer_size=DEFAULT_BUFFER_SIZE):
@@ -512,6 +531,7 @@
     close = interp2app(W_BufferedReader.close_w),
     flush = interp2app(W_BufferedReader.flush_w),
     detach = interp2app(W_BufferedReader.detach_w),
+    truncate = interp2app(W_BufferedReader.truncate_w),
     fileno = interp2app(W_BufferedReader.fileno_w),
     closed = GetSetProperty(W_BufferedReader.closed_get_w),
     )
@@ -673,6 +693,7 @@
     close = interp2app(W_BufferedWriter.close_w),
     fileno = interp2app(W_BufferedWriter.fileno_w),
     detach = interp2app(W_BufferedWriter.detach_w),
+    truncate = interp2app(W_BufferedWriter.truncate_w),
     closed = GetSetProperty(W_BufferedWriter.closed_get_w),
     )
 

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bufferedio.py	Thu Oct 28 17:24:23 2010
@@ -187,3 +187,11 @@
         import gc; gc.collect()
         assert record == [1, 2, 3]
 
+    def test_truncate(self):
+        import _io
+        raw = _io.FileIO(self.tmpfile, 'w+')
+        raw.write('x' * 20)
+        b = _io.BufferedReader(raw)
+        assert b.seek(8) == 8
+        assert b.truncate() == 8
+        assert b.tell() == 8


From afa at codespeak.net  Thu Oct 28 18:35:45 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 18:35:45 +0200 (CEST)
Subject: [pypy-svn] r78418 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101028163545.DDE2736E076@codespeak.net>

Author: afa
Date: Thu Oct 28 18:35:44 2010
New Revision: 78418

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py
Log:
BytesIO seek() and tell()


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py	Thu Oct 28 18:35:44 2010
@@ -2,8 +2,11 @@
     TypeDef, generic_new_descr)
 from pypy.interpreter.gateway import interp2app, unwrap_spec
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
+from pypy.interpreter.error import OperationError, operationerrfmt
+from pypy.rlib.rarithmetic import r_longlong
 from pypy.module._io.interp_bufferedio import W_BufferedIOBase
 from pypy.module._io.interp_iobase import convert_size
+import sys
 
 def buffer2string(buffer, start, end):
     from pypy.rlib.rstring import StringBuilder
@@ -84,6 +87,38 @@
         self._check_closed(space)
         return space.wrap(buffer2string(self.buf, 0, self.string_size))
 
+    @unwrap_spec('self', ObjSpace)
+    def tell_w(self, space):
+        self._check_closed(space)
+        return space.wrap(self.pos)
+
+    @unwrap_spec('self', ObjSpace, r_longlong, int)
+    def seek_w(self, space, pos, whence=0):
+        self._check_closed(space)
+
+        if whence == 0:
+            if pos < 0:
+                raise OperationError(space.w_ValueError, space.wrap(
+                    "negative seek value"))
+        if whence == 1:
+            if pos > sys.maxint - self.pos:
+                raise OperationError(space.w_OverflowError, space.wrap(
+                    "new position too large"))
+            pos += self.pos
+        elif whence == 2:
+            if pos > sys.maxint - self.string_size:
+                raise OperationError(space.w_OverflowError, space.wrap(
+                    "new position too large"))
+            pos += self.string_size
+        else:
+            raise operationerrfmt(space.w_ValueError,
+                "whence must be between 0 and 2, not %d", whence)
+
+        if pos >= 0:
+            self.pos = pos
+        else:
+            self.pos = 0
+        return space.wrap(self.pos)
 
 W_BytesIO.typedef = TypeDef(
     'BytesIO', W_BufferedIOBase.typedef,
@@ -93,5 +128,7 @@
     read = interp2app(W_BytesIO.read_w),
     write = interp2app(W_BytesIO.write_w),
     getvalue = interp2app(W_BytesIO.getvalue_w),
+    seek = interp2app(W_BytesIO.seek_w),
+    tell = interp2app(W_BytesIO.tell_w),
     )
 

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py	Thu Oct 28 18:35:44 2010
@@ -19,7 +19,9 @@
         import gc; gc.collect()
         f.close()
 
-    ## def test_seek(self):
-    ##     import _io
-    ##     f = _io.BytesIO("hello")
-    ##     assert f.seek(-1, 2) == 4
+    def test_seek(self):
+        import _io
+        f = _io.BytesIO("hello")
+        assert f.tell() == 0
+        assert f.seek(-1, 2) == 4
+        assert f.tell() == 4


From afa at codespeak.net  Thu Oct 28 21:41:14 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 21:41:14 +0200 (CEST)
Subject: [pypy-svn] r78420 -
	pypy/branch/fast-forward/lib-python/modified-2.7.0/test
Message-ID: <20101028194114.030E0282BE8@codespeak.net>

Author: afa
Date: Thu Oct 28 21:41:12 2010
New Revision: 78420

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_file.py
      - copied, changed from r78419, pypy/branch/fast-forward/lib-python/2.7.0/test/test_file.py
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_file2k.py
      - copied, changed from r78419, pypy/branch/fast-forward/lib-python/2.7.0/test/test_file2k.py
Log:
call gc.collect() before checking dead weakrefs


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_file.py (from r78419, pypy/branch/fast-forward/lib-python/2.7.0/test/test_file.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_file.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_file.py	Thu Oct 28 21:41:12 2010
@@ -12,7 +12,7 @@
 import io
 import _pyio as pyio
 
-from test.test_support import TESTFN, run_unittest
+from test.test_support import TESTFN, run_unittest, gc_collect
 from UserList import UserList
 
 class AutoFileTests(unittest.TestCase):
@@ -33,6 +33,7 @@
         self.assertEquals(self.f.tell(), p.tell())
         self.f.close()
         self.f = None
+        gc_collect()
         self.assertRaises(ReferenceError, getattr, p, 'tell')
 
     def testAttributes(self):

Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_file2k.py (from r78419, pypy/branch/fast-forward/lib-python/2.7.0/test/test_file2k.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_file2k.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_file2k.py	Thu Oct 28 21:41:12 2010
@@ -11,7 +11,7 @@
     threading = None
 
 from test import test_support
-from test.test_support import TESTFN, run_unittest
+from test.test_support import TESTFN, run_unittest, gc_collect
 from UserList import UserList
 
 class AutoFileTests(unittest.TestCase):
@@ -32,6 +32,7 @@
         self.assertEquals(self.f.tell(), p.tell())
         self.f.close()
         self.f = None
+        gc_collect()
         self.assertRaises(ReferenceError, getattr, p, 'tell')
 
     def testAttributes(self):


From afa at codespeak.net  Thu Oct 28 22:13:29 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 22:13:29 +0200 (CEST)
Subject: [pypy-svn] r78421 - in pypy/branch/fast-forward/pypy: interpreter
	interpreter/astcompiler objspace/std
Message-ID: <20101028201329.71066282BE8@codespeak.net>

Author: afa
Date: Thu Oct 28 22:13:27 2010
New Revision: 78421

Modified:
   pypy/branch/fast-forward/pypy/interpreter/astcompiler/codegen.py
   pypy/branch/fast-forward/pypy/interpreter/pyopcode.py
   pypy/branch/fast-forward/pypy/objspace/std/frame.py
Log:
Generate the same LIST_APPEND opcode as cpython.
Also remove a duplicated function.


Modified: pypy/branch/fast-forward/pypy/interpreter/astcompiler/codegen.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/astcompiler/codegen.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/astcompiler/codegen.py	Thu Oct 28 22:13:27 2010
@@ -1060,7 +1060,7 @@
             self._listcomp_generator(gens, gen_index, elt)
         else:
             elt.walkabout(self)
-            self.emit_op_arg(ops.LIST_APPEND, gen_index)
+            self.emit_op_arg(ops.LIST_APPEND, gen_index + 1)
             self.use_next_block(skip)
         self.use_next_block(if_cleanup)
         self.emit_jump(ops.JUMP_ABSOLUTE, start, True)

Modified: pypy/branch/fast-forward/pypy/interpreter/pyopcode.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyopcode.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyopcode.py	Thu Oct 28 22:13:27 2010
@@ -1037,7 +1037,7 @@
 
     def LIST_APPEND(self, oparg, next_instr):
         w = self.popvalue()
-        v = self.peekvalue(oparg)
+        v = self.peekvalue(oparg - 1)
         self.space.call_method(v, 'append', w)
 
     def SET_ADD(self, oparg, next_instr):
@@ -1124,11 +1124,6 @@
         w_dict = self.space.newdict()
         self.pushvalue(w_dict)
 
-    def LIST_APPEND(self, oparg, next_instr):
-        w_value = self.popvalue()
-        w_list = self.peekvalue(oparg + 1)
-        self.space.call_method(w_list, "append", w_value)
-
     def STORE_MAP(self, zero, next_instr):
         if sys.version_info >= (2, 6):
             w_key = self.popvalue()

Modified: pypy/branch/fast-forward/pypy/objspace/std/frame.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/frame.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/frame.py	Thu Oct 28 22:13:27 2010
@@ -18,7 +18,7 @@
 
     def LIST_APPEND(f, oparg, next_instr):
         w = f.popvalue()
-        v = f.peekvalue(oparg)
+        v = f.peekvalue(oparg - 1)
         if type(v) is W_ListObject:
             v.append(w)
         else:


From afa at codespeak.net  Thu Oct 28 23:05:53 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 23:05:53 +0200 (CEST)
Subject: [pypy-svn] r78422 -
	pypy/branch/fast-forward/pypy/interpreter/pyparser/test
Message-ID: <20101028210553.D050A282C1E@codespeak.net>

Author: afa
Date: Thu Oct 28 23:05:52 2010
New Revision: 78422

Modified:
   pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py
Log:
Fix tests after I changed the error message


Modified: pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/pyparser/test/test_pyparse.py	Thu Oct 28 23:05:52 2010
@@ -69,9 +69,9 @@
         assert exc.offset == 5
         assert exc.text.startswith("name another for")
         exc = py.test.raises(SyntaxError, parse, "\"blah").value
-        assert exc.msg == "EOL while scanning single-quoted string"
+        assert exc.msg == "EOL while scanning string literal"
         exc = py.test.raises(SyntaxError, parse, "'''\n").value
-        assert exc.msg == "EOF while scanning triple-quoted string"
+        assert exc.msg == "EOF while scanning triple-quoted string literal"
         for input in ("())", "(()", "((", "))"):
             py.test.raises(SyntaxError, parse, input)
 


From afa at codespeak.net  Thu Oct 28 23:33:49 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 23:33:49 +0200 (CEST)
Subject: [pypy-svn] r78424 - pypy/branch/fast-forward/pypy/objspace/std
Message-ID: <20101028213349.95DCC282C1E@codespeak.net>

Author: afa
Date: Thu Oct 28 23:33:48 2010
New Revision: 78424

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/formatting.py
Log:
Implement the same kind of limit as cpython,
(except that cpython limit is 117)
this avoids a memory error with the '%.123456789d' format


Modified: pypy/branch/fast-forward/pypy/objspace/std/formatting.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/formatting.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/formatting.py	Thu Oct 28 23:33:48 2010
@@ -41,6 +41,10 @@
     def std_wp_int(self, r, prefix='', keep_zero=False):
         # use self.prec to add some '0' on the left of the number
         if self.prec >= 0:
+            if self.prec > 1000:
+                raise OperationError(
+                    self.space.w_OverflowError, self.space.wrap(
+                    'formatted integer is too long (precision too large?)'))
             sign = r[0] == '-'
             padding = self.prec - (len(r)-int(sign))
             if padding > 0:


From afa at codespeak.net  Thu Oct 28 23:41:23 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 23:41:23 +0200 (CEST)
Subject: [pypy-svn] r78425 - in pypy/branch/fast-forward/pypy/objspace/std:
	. test
Message-ID: <20101028214123.786F036E076@codespeak.net>

Author: afa
Date: Thu Oct 28 23:41:22 2010
New Revision: 78425

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/strutil.py
   pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py
Log:
'0x' is not a correct hexadecimal number.


Modified: pypy/branch/fast-forward/pypy/objspace/std/strutil.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/strutil.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/strutil.py	Thu Oct 28 23:41:22 2010
@@ -62,14 +62,14 @@
             raise ParseStringError, "%s() base must be >= 2 and <= 36" % (fname,)
         self.base = base
 
-        if not s:
-            self.error()
         if base == 16 and (s.startswith('0x') or s.startswith('0X')):
             s = s[2:]
         if base == 8 and (s.startswith('0o') or s.startswith('0O')):
             s = s[2:]
         if base == 2 and (s.startswith('0b') or s.startswith('0B')):
             s = s[2:]
+        if not s:
+            self.error()
         self.s = s
         self.n = len(s)
         self.i = 0

Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py	Thu Oct 28 23:41:22 2010
@@ -88,6 +88,8 @@
             raises(ParseStringError, string_to_int, s+'  ')
             raises(ParseStringError, string_to_int, '+'+s)
             raises(ParseStringError, string_to_int, '-'+s)
+        raises(ParseStringError, string_to_int, '0x', 16)
+        raises(ParseStringError, string_to_int, '-0x', 16)
 
     def test_string_to_int_overflow(self):
         import sys


From afa at codespeak.net  Thu Oct 28 23:42:49 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Thu, 28 Oct 2010 23:42:49 +0200 (CEST)
Subject: [pypy-svn] r78426 - pypy/branch/fast-forward/pypy/objspace/std/test
Message-ID: <20101028214249.A0D32282C1E@codespeak.net>

Author: afa
Date: Thu Oct 28 23:42:48 2010
New Revision: 78426

Modified:
   pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py
Log:
update: '0x' is *no more* a correct hexadecimal number


Modified: pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py
==============================================================================
--- pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py	(original)
+++ pypy/branch/fast-forward/pypy/objspace/std/test/test_strutil.py	Thu Oct 28 23:42:48 2010
@@ -48,10 +48,6 @@
                  ('0x10', 0, 16),
                  ('0XE',  0, 14),
                  ('0',    0, 0),
-                 ('0x',   0, 0),    # according to CPython so far
-                 ('0X',   0, 0),    #     "           "
-                 ('0x',  16, 0),    #     "           "
-                 ('0X',  16, 0),    #     "           "
                  ('0b11', 2, 3),
                  ('0B10', 2, 2),
                  ('0o77', 8, 63),


From afa at codespeak.net  Fri Oct 29 00:35:44 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 00:35:44 +0200 (CEST)
Subject: [pypy-svn] r78427 -
	pypy/branch/fast-forward/lib-python/modified-2.7.0/test
Message-ID: <20101028223544.004CB36E076@codespeak.net>

Author: afa
Date: Fri Oct 29 00:35:43 2010
New Revision: 78427

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_weakref.py
      - copied, changed from r78426, pypy/branch/fast-forward/lib-python/2.7.0/test/test_weakref.py
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_weakset.py
      - copied, changed from r78426, pypy/branch/fast-forward/lib-python/2.7.0/test/test_weakset.py
Log:
Fix many weakref tests by forcing garbage collection when needed


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_weakref.py (from r78426, pypy/branch/fast-forward/lib-python/2.7.0/test/test_weakref.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_weakref.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_weakref.py	Fri Oct 29 00:35:43 2010
@@ -1,4 +1,3 @@
-import gc
 import sys
 import unittest
 import UserList
@@ -6,6 +5,7 @@
 import operator
 
 from test import test_support
+from test.test_support import gc_collect
 
 # Used in ReferencesTestCase.test_ref_created_during_del() .
 ref_from_del = None
@@ -70,6 +70,7 @@
         ref1 = weakref.ref(o, self.callback)
         ref2 = weakref.ref(o, self.callback)
         del o
+        gc_collect()
         self.assertTrue(ref1() is None,
                      "expected reference to be invalidated")
         self.assertTrue(ref2() is None,
@@ -101,13 +102,16 @@
         ref1 = weakref.proxy(o, self.callback)
         ref2 = weakref.proxy(o, self.callback)
         del o
+        gc_collect()
 
         def check(proxy):
             proxy.bar
 
         self.assertRaises(weakref.ReferenceError, check, ref1)
         self.assertRaises(weakref.ReferenceError, check, ref2)
-        self.assertRaises(weakref.ReferenceError, bool, weakref.proxy(C()))
+        ref3 = weakref.proxy(C())
+        gc_collect()
+        self.assertRaises(weakref.ReferenceError, bool, ref3)
         self.assertTrue(self.cbcalled == 2)
 
     def check_basic_ref(self, factory):
@@ -124,6 +128,7 @@
         o = factory()
         ref = weakref.ref(o, self.callback)
         del o
+        gc_collect()
         self.assertTrue(self.cbcalled == 1,
                      "callback did not properly set 'cbcalled'")
         self.assertTrue(ref() is None,
@@ -148,6 +153,7 @@
         self.assertTrue(weakref.getweakrefcount(o) == 2,
                      "wrong weak ref count for object")
         del proxy
+        gc_collect()
         self.assertTrue(weakref.getweakrefcount(o) == 1,
                      "wrong weak ref count for object after deleting proxy")
 
@@ -325,6 +331,7 @@
                      "got wrong number of weak reference objects")
 
         del ref1, ref2, proxy1, proxy2
+        gc_collect()
         self.assertTrue(weakref.getweakrefcount(o) == 0,
                      "weak reference objects not unlinked from"
                      " referent when discarded.")
@@ -338,6 +345,7 @@
         ref1 = weakref.ref(o, self.callback)
         ref2 = weakref.ref(o, self.callback)
         del ref1
+        gc_collect()
         self.assertTrue(weakref.getweakrefs(o) == [ref2],
                      "list of refs does not match")
 
@@ -345,10 +353,12 @@
         ref1 = weakref.ref(o, self.callback)
         ref2 = weakref.ref(o, self.callback)
         del ref2
+        gc_collect()
         self.assertTrue(weakref.getweakrefs(o) == [ref1],
                      "list of refs does not match")
 
         del ref1
+        gc_collect()
         self.assertTrue(weakref.getweakrefs(o) == [],
                      "list of refs not cleared")
 
@@ -400,13 +410,11 @@
         # when the second attempt to remove the instance from the "list
         # of all objects" occurs.
 
-        import gc
-
         class C(object):
             pass
 
         c = C()
-        wr = weakref.ref(c, lambda ignore: gc.collect())
+        wr = weakref.ref(c, lambda ignore: gc_collect())
         del c
 
         # There endeth the first part.  It gets worse.
@@ -414,7 +422,7 @@
 
         c1 = C()
         c1.i = C()
-        wr = weakref.ref(c1.i, lambda ignore: gc.collect())
+        wr = weakref.ref(c1.i, lambda ignore: gc_collect())
 
         c2 = C()
         c2.c1 = c1
@@ -430,8 +438,6 @@
         del c2
 
     def test_callback_in_cycle_1(self):
-        import gc
-
         class J(object):
             pass
 
@@ -467,11 +473,9 @@
         # search II.__mro__, but that's NULL.   The result was a segfault in
         # a release build, and an assert failure in a debug build.
         del I, J, II
-        gc.collect()
+        gc_collect()
 
     def test_callback_in_cycle_2(self):
-        import gc
-
         # This is just like test_callback_in_cycle_1, except that II is an
         # old-style class.  The symptom is different then:  an instance of an
         # old-style class looks in its own __dict__ first.  'J' happens to
@@ -496,11 +500,9 @@
         I.wr = weakref.ref(J, I.acallback)
 
         del I, J, II
-        gc.collect()
+        gc_collect()
 
     def test_callback_in_cycle_3(self):
-        import gc
-
         # This one broke the first patch that fixed the last two.  In this
         # case, the objects reachable from the callback aren't also reachable
         # from the object (c1) *triggering* the callback:  you can get to
@@ -520,11 +522,9 @@
         c2.wr = weakref.ref(c1, c2.cb)
 
         del c1, c2
-        gc.collect()
+        gc_collect()
 
     def test_callback_in_cycle_4(self):
-        import gc
-
         # Like test_callback_in_cycle_3, except c2 and c1 have different
         # classes.  c2's class (C) isn't reachable from c1 then, so protecting
         # objects reachable from the dying object (c1) isn't enough to stop
@@ -548,11 +548,9 @@
         c2.wr = weakref.ref(c1, c2.cb)
 
         del c1, c2, C, D
-        gc.collect()
+        gc_collect()
 
     def test_callback_in_cycle_resurrection(self):
-        import gc
-
         # Do something nasty in a weakref callback:  resurrect objects
         # from dead cycles.  For this to be attempted, the weakref and
         # its callback must also be part of the cyclic trash (else the
@@ -583,7 +581,7 @@
         del c1, c2, C   # make them all trash
         self.assertEqual(alist, [])  # del isn't enough to reclaim anything
 
-        gc.collect()
+        gc_collect()
         # c1.wr and c2.wr were part of the cyclic trash, so should have
         # been cleared without their callbacks executing.  OTOH, the weakref
         # to C is bound to a function local (wr), and wasn't trash, so that
@@ -593,12 +591,10 @@
         self.assertEqual(wr(), None)
 
         del alist[:]
-        gc.collect()
+        gc_collect()
         self.assertEqual(alist, [])
 
     def test_callbacks_on_callback(self):
-        import gc
-
         # Set up weakref callbacks *on* weakref callbacks.
         alist = []
         def safe_callback(ignore):
@@ -626,12 +622,12 @@
 
         del callback, c, d, C
         self.assertEqual(alist, [])  # del isn't enough to clean up cycles
-        gc.collect()
+        gc_collect()
         self.assertEqual(alist, ["safe_callback called"])
         self.assertEqual(external_wr(), None)
 
         del alist[:]
-        gc.collect()
+        gc_collect()
         self.assertEqual(alist, [])
 
     def test_gc_during_ref_creation(self):
@@ -641,9 +637,11 @@
         self.check_gc_during_creation(weakref.proxy)
 
     def check_gc_during_creation(self, makeref):
-        thresholds = gc.get_threshold()
-        gc.set_threshold(1, 1, 1)
-        gc.collect()
+        if test_support.check_impl_detail():
+            import gc
+            thresholds = gc.get_threshold()
+            gc.set_threshold(1, 1, 1)
+        gc_collect()
         class A:
             pass
 
@@ -663,7 +661,8 @@
             weakref.ref(referenced, callback)
 
         finally:
-            gc.set_threshold(*thresholds)
+            if test_support.check_impl_detail():
+                gc.set_threshold(*thresholds)
 
     def test_ref_created_during_del(self):
         # Bug #1377858
@@ -683,7 +682,7 @@
         r = weakref.ref(Exception)
         self.assertRaises(TypeError, r.__init__, 0, 0, 0, 0, 0)
         # No exception should be raised here
-        gc.collect()
+        gc_collect()
 
     def test_classes(self):
         # Check that both old-style classes and new-style classes
@@ -696,12 +695,12 @@
         weakref.ref(int)
         a = weakref.ref(A, l.append)
         A = None
-        gc.collect()
+        gc_collect()
         self.assertEqual(a(), None)
         self.assertEqual(l, [a])
         b = weakref.ref(B, l.append)
         B = None
-        gc.collect()
+        gc_collect()
         self.assertEqual(b(), None)
         self.assertEqual(l, [a, b])
 
@@ -722,6 +721,7 @@
         self.assertTrue(mr.called)
         self.assertEqual(mr.value, 24)
         del o
+        gc_collect()
         self.assertTrue(mr() is None)
         self.assertTrue(mr.called)
 
@@ -839,15 +839,18 @@
         del items1, items2
         self.assertTrue(len(dict) == self.COUNT)
         del objects[0]
+        gc_collect()
         self.assertTrue(len(dict) == (self.COUNT - 1),
                      "deleting object did not cause dictionary update")
         del objects, o
+        gc_collect()
         self.assertTrue(len(dict) == 0,
                      "deleting the values did not clear the dictionary")
         # regression on SF bug #447152:
         dict = weakref.WeakValueDictionary()
         self.assertRaises(KeyError, dict.__getitem__, 1)
         dict[2] = C()
+        gc_collect()
         self.assertRaises(KeyError, dict.__getitem__, 2)
 
     def test_weak_keys(self):
@@ -868,9 +871,11 @@
         del items1, items2
         self.assertTrue(len(dict) == self.COUNT)
         del objects[0]
+        gc_collect()
         self.assertTrue(len(dict) == (self.COUNT - 1),
                      "deleting object did not cause dictionary update")
         del objects, o
+        gc_collect()
         self.assertTrue(len(dict) == 0,
                      "deleting the keys did not clear the dictionary")
         o = Object(42)
@@ -1137,6 +1142,7 @@
         for o in objs:
             count += 1
             del d[o]
+        gc_collect()
         self.assertEqual(len(d), 0)
         self.assertEqual(count, 2)
 

Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_weakset.py (from r78426, pypy/branch/fast-forward/lib-python/2.7.0/test/test_weakset.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_weakset.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_weakset.py	Fri Oct 29 00:35:43 2010
@@ -57,6 +57,7 @@
         self.assertEqual(len(self.s), len(self.d))
         self.assertEqual(len(self.fs), 1)
         del self.obj
+        test_support.gc_collect()
         self.assertEqual(len(self.fs), 0)
 
     def test_contains(self):
@@ -65,6 +66,7 @@
         self.assertRaises(TypeError, self.s.__contains__, [[]])
         self.assertIn(self.obj, self.fs)
         del self.obj
+        test_support.gc_collect()
         self.assertNotIn(SomeClass('F'), self.fs)
 
     def test_union(self):
@@ -203,6 +205,7 @@
         self.assertEqual(self.s, dup)
         self.assertRaises(TypeError, self.s.add, [])
         self.fs.add(Foo())
+        test_support.gc_collect()
         self.assertTrue(len(self.fs) == 1)
         self.fs.add(self.obj)
         self.assertTrue(len(self.fs) == 1)


From afa at codespeak.net  Fri Oct 29 00:47:54 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 00:47:54 +0200 (CEST)
Subject: [pypy-svn] r78429 -
	pypy/branch/fast-forward/lib-python/modified-2.7.0/test
Message-ID: <20101028224754.07E07282BE8@codespeak.net>

Author: afa
Date: Fri Oct 29 00:47:53 2010
New Revision: 78429

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_tempfile.py
      - copied, changed from r78426, pypy/branch/fast-forward/lib-python/2.7.0/test/test_tempfile.py
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/tf_inherit_check.py
      - copied unchanged from r78426, pypy/branch/fast-forward/lib-python/2.7.0/test/tf_inherit_check.py
Log:
More fixes with gc.collect()


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_tempfile.py (from r78426, pypy/branch/fast-forward/lib-python/2.7.0/test/test_tempfile.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_tempfile.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_tempfile.py	Fri Oct 29 00:47:53 2010
@@ -244,6 +244,7 @@
         dir = tempfile.mkdtemp()
         try:
             self.do_create(dir=dir).write("blat")
+            test_support.gc_collect()
         finally:
             os.rmdir(dir)
 
@@ -528,12 +529,15 @@
         self.do_create(suf="b")
         self.do_create(pre="a", suf="b")
         self.do_create(pre="aa", suf=".txt")
+        test_support.gc_collect()
 
     def test_many(self):
         # mktemp can choose many usable file names (stochastic)
         extant = range(TEST_FILES)
         for i in extant:
             extant[i] = self.do_create(pre="aa")
+        del extant
+        test_support.gc_collect()
 
 ##     def test_warning(self):
 ##         # mktemp issues a warning when used


From afa at codespeak.net  Fri Oct 29 01:01:08 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 01:01:08 +0200 (CEST)
Subject: [pypy-svn] r78430 - pypy/branch/fast-forward/pypy/module/pyexpat
Message-ID: <20101028230108.85CD7282BE8@codespeak.net>

Author: afa
Date: Fri Oct 29 01:01:06 2010
New Revision: 78430

Modified:
   pypy/branch/fast-forward/pypy/module/pyexpat/interp_pyexpat.py
Log:
Format pyexpat error messages like CPython


Modified: pypy/branch/fast-forward/pypy/module/pyexpat/interp_pyexpat.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/pyexpat/interp_pyexpat.py	(original)
+++ pypy/branch/fast-forward/pypy/module/pyexpat/interp_pyexpat.py	Fri Oct 29 01:01:06 2010
@@ -505,7 +505,7 @@
         err = rffi.charp2strn(XML_ErrorString(code), 200)
         lineno = XML_GetCurrentLineNumber(self.itself)
         colno = XML_GetCurrentColumnNumber(self.itself)
-        msg = "%s: line: %d, column: %d" % (err, lineno, colno)
+        msg = "%s: line %d, column %d" % (err, lineno, colno)
         w_module = space.getbuiltinmodule('pyexpat')
         w_errorcls = space.getattr(w_module, space.wrap('error'))
         w_error = space.call_function(


From afa at codespeak.net  Fri Oct 29 01:15:35 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 01:15:35 +0200 (CEST)
Subject: [pypy-svn] r78431 - in
	pypy/branch/fast-forward/pypy/module/pyexpat: . test
Message-ID: <20101028231535.DE65E36E076@codespeak.net>

Author: afa
Date: Fri Oct 29 01:15:34 2010
New Revision: 78431

Modified:
   pypy/branch/fast-forward/pypy/module/pyexpat/interp_pyexpat.py
   pypy/branch/fast-forward/pypy/module/pyexpat/test/test_parser.py
Log:
pyexpat: add parser.buffer_size


Modified: pypy/branch/fast-forward/pypy/module/pyexpat/interp_pyexpat.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/pyexpat/interp_pyexpat.py	(original)
+++ pypy/branch/fast-forward/pypy/module/pyexpat/interp_pyexpat.py	Fri Oct 29 01:15:34 2010
@@ -527,6 +527,16 @@
     def descr_ErrorByteIndex(space, self):
         return space.wrap(XML_GetErrorByteIndex(self.itself))
 
+    def get_buffer_size(space, self):
+        return space.wrap(self.buffer_size)
+    def set_buffer_size(space, self, w_value):
+        value = space.int_w(w_value)
+        if value <= 0:
+            raise OperationError(space.w_ValueError, space.wrap(
+                "buffer_size must be greater than zero"))
+        self.flush_character_buffer(space)
+        self.buffer_size = value
+
     def get_buffer_text(space, self):
         return space.wrap(self.buffer_w is not None)
     def set_buffer_text(space, self, w_value):
@@ -561,6 +571,9 @@
     ordered_attributes = bool_property('ordered_attributes', W_XMLParserType),
     specified_attributes = bool_property('specified_attributes', W_XMLParserType),
     intern = GetSetProperty(W_XMLParserType.get_intern, cls=W_XMLParserType),
+    buffer_size = GetSetProperty(W_XMLParserType.get_buffer_size,
+                                 W_XMLParserType.set_buffer_size,
+                                 cls=W_XMLParserType),
     buffer_text = GetSetProperty(W_XMLParserType.get_buffer_text,
                                  W_XMLParserType.set_buffer_text, cls=W_XMLParserType),
 

Modified: pypy/branch/fast-forward/pypy/module/pyexpat/test/test_parser.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/pyexpat/test/test_parser.py	(original)
+++ pypy/branch/fast-forward/pypy/module/pyexpat/test/test_parser.py	Fri Oct 29 01:15:34 2010
@@ -15,3 +15,9 @@
         assert res == 1
 
         raises(pyexpat.ExpatError, p.Parse, "3")
+
+    def test_set_buffersize(self):
+        import pyexpat
+        p = pyexpat.ParserCreate()
+        p.buffer_size = 150
+        assert p.buffer_size == 150


From agaynor at codespeak.net  Fri Oct 29 06:49:43 2010
From: agaynor at codespeak.net (agaynor at codespeak.net)
Date: Fri, 29 Oct 2010 06:49:43 +0200 (CEST)
Subject: [pypy-svn] r78433 - pypy/trunk/pypy/doc/config
Message-ID: <20101029044943.0513A282BE8@codespeak.net>

Author: agaynor
Date: Fri Oct 29 06:49:41 2010
New Revision: 78433

Added:
   pypy/trunk/pypy/doc/config/objspace.usemodules._bisect.txt
Log:
Added docs for the usemodule _bisect option.


Added: pypy/trunk/pypy/doc/config/objspace.usemodules._bisect.txt
==============================================================================
--- (empty file)
+++ pypy/trunk/pypy/doc/config/objspace.usemodules._bisect.txt	Fri Oct 29 06:49:41 2010
@@ -0,0 +1,4 @@
+Use the '_bisect' module.
+Used, optionally,  by the 'bisect' standard lib module. This module is expected to be working and is included by default.
+
+


From david at codespeak.net  Fri Oct 29 09:36:55 2010
From: david at codespeak.net (david at codespeak.net)
Date: Fri, 29 Oct 2010 09:36:55 +0200 (CEST)
Subject: [pypy-svn] r78438 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . test
Message-ID: <20101029073655.8EA4336E076@codespeak.net>

Author: david
Date: Fri Oct 29 09:36:53 2010
New Revision: 78438

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/locations.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py
Log:
Implement int_mul, int_floordiv and int_mod operations.

- Added support to codebuilder to allocate new memory for instructions
- Implement int_div and int_mod as functions called from assembler, because doesn't provide instructions for them



Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py	Fri Oct 29 09:36:53 2010
@@ -1,2 +1,20 @@
+from pypy.rpython.lltypesystem import lltype
+
 FUNC_ALIGN=8
 WORD=4
+
+arm_int_div_sign = lltype.Ptr(lltype.FuncType([lltype.Signed, lltype.Signed], lltype.Signed))
+arm_int_mod_sign = arm_int_div_sign
+
+def arm_int_div(a, b):
+    return int(a/float(b))
+
+def arm_int_mod(a, b):
+    sign = 1
+    if a < 0:
+        a = -1 * a
+        sign = -1
+    if b < 0:
+        b = -1 * b
+    res = a % b
+    return sign * res

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Fri Oct 29 09:36:53 2010
@@ -3,7 +3,7 @@
 from pypy.jit.backend.arm import registers as r
 from pypy.jit.backend.arm.arch import WORD, FUNC_ALIGN
 from pypy.jit.backend.arm.codebuilder import ARMv7Builder, ARMv7InMemoryBuilder
-from pypy.jit.backend.arm.regalloc import ARMRegisterManager
+from pypy.jit.backend.arm.regalloc import ARMRegisterManager, ARMFrameManager
 from pypy.jit.backend.llsupport.regalloc import compute_vars_longevity
 from pypy.jit.metainterp.history import ConstInt, BoxInt, Box, BasicFailDescr
 from pypy.jit.metainterp.resoperation import rop
@@ -25,6 +25,7 @@
         self.fail_boxes_int = values_array(lltype.Signed, failargs_limit)
         self._debug_asm = True
 
+        self._exit_code_addr = self.mc.curraddr()
         self._gen_exit_path()
         self.align()
         self.mc._start_addr = self.mc.curraddr()
@@ -136,7 +137,7 @@
         n = self.cpu.get_fail_descr_number(op.getdescr())
         self.encode32(mem, j+1, n)
         self.mc.gen_load_int(r.lr.value, memaddr, cond=fcond)
-        self.mc.gen_load_int(reg.value, self.mc.baseaddr(), cond=fcond)
+        self.mc.gen_load_int(reg.value, self._exit_code_addr, cond=fcond)
         self.mc.MOV_rr(r.pc.value, reg.value, cond=fcond)
 
         # This register is used for patching when assembling a bridge
@@ -168,7 +169,7 @@
     # cpu interface
     def assemble_loop(self, inputargs, operations, looptoken):
         longevity = compute_vars_longevity(inputargs, operations)
-        regalloc = ARMRegisterManager(longevity, assembler=self.mc)
+        regalloc = ARMRegisterManager(longevity, assembler=self.mc, frame_manager=ARMFrameManager())
         self.align()
         loop_start=self.mc.curraddr()
         self.gen_func_prolog()
@@ -189,7 +190,7 @@
     def assemble_bridge(self, faildescr, inputargs, operations):
         enc = rffi.cast(rffi.CCHARP, faildescr._failure_recovery_code)
         longevity = compute_vars_longevity(inputargs, operations)
-        regalloc = ARMRegisterManager(longevity, assembler=self.mc)
+        regalloc = ARMRegisterManager(longevity, assembler=self.mc, frame_manager=ARMFrameManager())
 
         regalloc.update_bindings(enc, inputargs)
         bridge_head = self.mc.curraddr()

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Fri Oct 29 09:36:53 2010
@@ -1,10 +1,13 @@
 from pypy.jit.backend.arm import conditions as cond
 from pypy.jit.backend.arm import registers as reg
-from pypy.jit.backend.arm.arch import WORD
+from pypy.jit.backend.arm.arch import (WORD, FUNC_ALIGN, arm_int_div,
+                                        arm_int_div_sign, arm_int_mod_sign, arm_int_mod)
 from pypy.jit.backend.arm.instruction_builder import define_instructions
 
 from pypy.rlib.rmmap import alloc, PTR
+from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import lltype, rffi
+from pypy.jit.metainterp.history import ConstInt, BoxInt, Box, BasicFailDescr
 
 class AbstractARMv7Builder(object):
     def _init(self, data, map_size):
@@ -40,6 +43,26 @@
     def BKPT(self, cond=cond.AL):
         self.write32(cond << 28 | 0x1200070)
 
+    def DIV(self, cond=cond.AL):
+        """Generates a call to a helper function used for division, takes its
+        arguments in r0 and r1, result is placed in r0"""
+        self.PUSH(range(2, 12), cond=cond)
+        div_addr = rffi.cast(lltype.Signed, llhelper(arm_int_div_sign, arm_int_div))
+        self.gen_load_int(reg.r2.value, div_addr, cond=cond)
+        self.gen_load_int(reg.lr.value, self.curraddr()+self.size_of_gen_load_int+WORD, cond=cond)
+        self.MOV_rr(reg.pc.value, reg.r2.value, cond=cond)
+        self.LDM(reg.sp.value, range(2, 12), w=1, cond=cond) # XXX Replace with POP instr. someday
+
+    def MOD(self, cond=cond.AL):
+        """Generate a call to a helper function used for modulo, takes its
+        arguments in r0 and r1, result is placed in r0"""
+        self.PUSH(range(2, 12), cond=cond)
+        mod_addr = rffi.cast(lltype.Signed, llhelper(arm_int_mod_sign, arm_int_mod))
+        self.gen_load_int(reg.r2.value, mod_addr, cond=cond)
+        self.gen_load_int(reg.lr.value, self.curraddr()+self.size_of_gen_load_int+WORD, cond=cond)
+        self.MOV_rr(reg.pc.value, reg.r2.value, cond=cond)
+        self.LDM(reg.sp.value, range(2, 12), w=1, cond=cond) # XXX Replace with POP instr. someday
+
     def _encode_reg_list(self, instr, regs):
         for reg in regs:
             instr |= 0x1 << reg
@@ -73,16 +96,22 @@
         """r is the register number, value is the value to be loaded to the
         register"""
         assert r != reg.ip.value, 'ip is used to load int'
-        self.MOV_ri(r, (value & 0xFF), cond=cond)
         ip = reg.ip.value
 
+        self.MOV_ri(r, (value & 0xFF), cond=cond)
         for offset in range(8, 25, 8):
             t = (value >> offset) & 0xFF
-            #if t == 0:
-            #    continue
             self.MOV_ri(ip, t, cond=cond)
             self.ORR_rr(r, r, ip, offset, cond=cond)
 
+    # regalloc support
+    def regalloc_mov(self, prev_loc, loc):
+        if isinstance(prev_loc, ConstInt):
+            # XXX check size of imm for current instr
+            self.gen_load_int(loc.value, prev_loc.getint())
+        else:
+            self.MOV_rr(loc.value, prev_loc.value)
+
 class ARMv7InMemoryBuilder(AbstractARMv7Builder):
     def __init__(self, start, end):
         map_size = end - start
@@ -96,5 +125,28 @@
         data = alloc(map_size)
         self._pos = 0
         self._init(data, map_size)
+        self.checks = True
+        self.n_data=0
+
+    _space_for_jump = 10 * WORD
+    def writechar(self, char):
+        if self.checks and not self._pos < self._size - self._space_for_jump:
+            self.checks = False
+            self._add_more_mem()
+            self.checks = True
+        assert self._pos < self._size
+        AbstractARMv7Builder.writechar(self, char)
+
+    def _add_more_mem(self):
+        new_mem = alloc(self._size)
+        new_mem_addr = rffi.cast(lltype.Signed, new_mem)
+        self.PUSH([reg.r0.value])
+        self.gen_load_int(reg.r0.value, new_mem_addr)
+        self.MOV_rr(reg.pc.value, reg.r0.value)
+        self._dump_trace('data%d.asm' % self.n_data)
+        self.n_data+=1
+        self._data = new_mem
+        self._pos = 0
+        self.LDM(reg.sp.value, [reg.r0.value], w=1) # XXX Replace with POP instr. someday
 
 define_instructions(AbstractARMv7Builder)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/locations.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/locations.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/locations.py	Fri Oct 29 09:36:53 2010
@@ -5,3 +5,6 @@
 
     def __init__(self, value):
         self.value = value
+
+    def __repr__(self):
+        return 'r%d' % self.value

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	Fri Oct 29 09:36:53 2010
@@ -1,7 +1,8 @@
 from pypy.jit.backend.arm import conditions as c
 from pypy.jit.backend.arm import locations
 from pypy.jit.backend.arm import registers as r
-from pypy.jit.backend.arm.arch import WORD, FUNC_ALIGN
+from pypy.jit.backend.arm.arch import (WORD, FUNC_ALIGN, arm_int_div,
+                                        arm_int_div_sign, arm_int_mod_sign, arm_int_mod)
 from pypy.jit.backend.arm.codebuilder import ARMv7Builder, ARMv7InMemoryBuilder
 from pypy.jit.backend.arm.regalloc import ARMRegisterManager
 from pypy.jit.backend.llsupport.regalloc import compute_vars_longevity
@@ -68,7 +69,44 @@
         return fcond
 
     def emit_op_int_mul(self, op, regalloc, fcond):
-        import pdb; pdb.set_trace()
+        res = regalloc.try_allocate_reg(op.result)
+        reg1 = self._put_in_reg(op.getarg(0), regalloc)
+        reg2 = self._put_in_reg(op.getarg(1), regalloc)
+        self.mc.MUL(res.value, reg1.value, reg2.value)
+        regalloc.possibly_free_var(reg1)
+        regalloc.possibly_free_var(reg2)
+        return fcond
+
+    def emit_op_int_floordiv(self, op, regalloc, fcond):
+        arg1 = regalloc.make_sure_var_in_reg(op.getarg(0), selected_reg=r.r0)
+        arg2 = regalloc.make_sure_var_in_reg(op.getarg(1), selected_reg=r.r1)
+        assert arg1 == r.r0
+        assert arg2 == r.r1
+        res = regalloc.try_allocate_reg(op.result)
+        self.mc.DIV(fcond)
+        self.mc.MOV_rr(res.value, r.r0.value, cond=fcond)
+        regalloc.possibly_free_vars_for_op(op)
+        return fcond
+
+    def emit_op_int_mod(self, op, regalloc, fcond):
+        arg1 = regalloc.make_sure_var_in_reg(op.getarg(0), selected_reg=r.r0)
+        arg2 = regalloc.make_sure_var_in_reg(op.getarg(1), selected_reg=r.r1)
+        assert arg1 == r.r0
+        assert arg2 == r.r1
+        res = regalloc.try_allocate_reg(op.result)
+        self.mc.MOD(fcond)
+        self.mc.MOV_rr(res.value, r.r0.value, cond=fcond)
+        regalloc.possibly_free_vars_for_op(op)
+        return fcond
+
+    def _put_in_reg(self, box, regalloc):
+        if isinstance(box, ConstInt):
+            t = Box()
+            reg = regalloc.try_allocate_reg(t)
+            self.mc.gen_load_int(reg.value, box.getint())
+        else:
+            reg = regalloc.try_allocate_reg(box)
+        return reg
 
 class GuardOpAssembler(object):
     _mixin_ = True

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/regalloc.py	Fri Oct 29 09:36:53 2010
@@ -19,6 +19,9 @@
             self.try_allocate_reg(inputargs[i], r.all_regs[ord(enc[j])])
             j += 1
 
+    def convert_to_imm(self, c):
+        return c
+
 class ARMFrameManager(FrameManager):
     @staticmethod
     def frame_pos(loc, type):

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/support.py	Fri Oct 29 09:36:53 2010
@@ -13,6 +13,7 @@
     addr = asm.mc._start_addr
     assert addr % 8 == 0
     func = rffi.cast(lltype.Ptr(BOOTSTRAP_TP), addr)
+    asm.mc._dump_trace('test.asm')
     return func()
 
 def skip_unless_arm():

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py	Fri Oct 29 09:36:53 2010
@@ -1,3 +1,4 @@
+from pypy.jit.backend.arm.arch import arm_int_div, arm_int_div_sign
 from pypy.jit.backend.arm import conditions as c
 from pypy.jit.backend.arm import registers as r
 from pypy.jit.backend.arm.arch import WORD
@@ -37,6 +38,18 @@
         self.a.gen_func_epilog()
         assert run_asm(self.a) == -123
 
+    def test_load_neg_int_to_reg(self):
+        self.a.gen_func_prolog()
+        self.a.mc.gen_load_int(r.r0.value, -110)
+        self.a.gen_func_epilog()
+        assert run_asm(self.a) == -110
+
+    def test_load_neg_int_to_reg2(self):
+        self.a.gen_func_prolog()
+        self.a.mc.gen_load_int(r.r0.value, -3)
+        self.a.gen_func_epilog()
+        assert run_asm(self.a) == -3
+
 
     def test_or(self):
         self.a.gen_func_prolog()
@@ -108,6 +121,47 @@
         self.a.gen_func_epilog()
         assert run_asm(self.a) == 133
 
+    def test_division(self):
+        self.a.gen_func_prolog()
+        self.a.mc.MOV_ri(r.r0.value, 123)
+        self.a.mc.MOV_ri(r.r1.value, 2)
+
+        # call to div
+        self.a.mc.PUSH(range(2, 12))
+        div_addr = rffi.cast(lltype.Signed, llhelper(arm_int_div_sign, arm_int_div))
+        self.a.mc.gen_load_int(r.r10.value, div_addr)
+        self.a.mc.gen_load_int(r.lr.value, self.a.mc.curraddr()+self.a.mc.size_of_gen_load_int+WORD)
+        self.a.mc.MOV_rr(r.pc.value, r.r10.value)
+        self.a.mc.LDM(r.sp.value, range(2, 12), w=1) # XXX Replace with POP instr. someday
+        self.a.gen_func_epilog()
+        assert run_asm(self.a) == 61
+
+    def test_DIV(self):
+        self.a.gen_func_prolog()
+        self.a.mc.MOV_ri(r.r0.value, 123)
+        self.a.mc.MOV_ri(r.r1.value, 2)
+        self.a.mc.DIV()
+        self.a.gen_func_epilog()
+        assert run_asm(self.a) == 61
+
+    def test_DIV2(self):
+        self.a.gen_func_prolog()
+        self.a.mc.gen_load_int(r.r0.value, -110)
+        self.a.mc.gen_load_int(r.r1.value, 3)
+        self.a.mc.DIV()
+        self.a.gen_func_epilog()
+        assert run_asm(self.a) == -36
+
+    def test_DIV3(self):
+        self.a.gen_func_prolog()
+        self.a.mc.gen_load_int(r.r8.value, 110)
+        self.a.mc.gen_load_int(r.r9.value, -3)
+        self.a.mc.MOV_rr(r.r0.value, r.r8.value)
+        self.a.mc.MOV_rr(r.r1.value, r.r9.value)
+        self.a.mc.DIV()
+        self.a.gen_func_epilog()
+        assert run_asm(self.a) == -36
+
 def callme(inp):
     i = inp + 10
     return i


From afa at codespeak.net  Fri Oct 29 10:16:24 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 10:16:24 +0200 (CEST)
Subject: [pypy-svn] r78442 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101029081624.996F436E0C1@codespeak.net>

Author: afa
Date: Fri Oct 29 10:16:23 2010
New Revision: 78442

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py
Log:
Bah. Test and fix


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_bytesio.py	Fri Oct 29 10:16:23 2010
@@ -100,7 +100,7 @@
             if pos < 0:
                 raise OperationError(space.w_ValueError, space.wrap(
                     "negative seek value"))
-        if whence == 1:
+        elif whence == 1:
             if pos > sys.maxint - self.pos:
                 raise OperationError(space.w_OverflowError, space.wrap(
                     "new position too large"))

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_bytesio.py	Fri Oct 29 10:16:23 2010
@@ -25,3 +25,4 @@
         assert f.tell() == 0
         assert f.seek(-1, 2) == 4
         assert f.tell() == 4
+        assert f.seek(0) == 0


From cfbolz at codespeak.net  Fri Oct 29 10:18:37 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 29 Oct 2010 10:18:37 +0200 (CEST)
Subject: [pypy-svn] r78443 - pypy/extradoc/sprintinfo/ddorf2010
Message-ID: <20101029081837.C077236E0CB@codespeak.net>

Author: cfbolz
Date: Fri Oct 29 10:18:36 2010
New Revision: 78443

Modified:
   pypy/extradoc/sprintinfo/ddorf2010/planning.txt
Log:
(all): planning for today


Modified: pypy/extradoc/sprintinfo/ddorf2010/planning.txt
==============================================================================
--- pypy/extradoc/sprintinfo/ddorf2010/planning.txt	(original)
+++ pypy/extradoc/sprintinfo/ddorf2010/planning.txt	Fri Oct 29 10:18:36 2010
@@ -1,29 +1,32 @@
 people present:
     Anto
-    David
     Carl Friedrich
     Ronny
     Armin
+    H?kan
     Lukas
 
+coming later?:
+    Sven
+
+not there today:
+    David
 
 topics:
 
-- get mmap to work on Beagleboard DONE
-- bridges in the ARM backend SORT OF DONE
-- add spilling and location support IN PROGRESS (David, Armin around)
-- ARM int trace operations (David)
-- hg conversion WE HAVE MORE OF A PLAN (Anto, Ronny)
-- memory benchmarks DONE
-- refine heap analysis CONFUSING (Carl Friedrich, Lukas)
-- (Re)write _bisect module (Armin)
+- add spilling and location support IN PROGRESS
+- ARM int trace operations SOME PROGRESS
+- hg conversion WE HAVE AN EVEN DIFFERENT PLAN (Anto, Ronny)
+- refine heap analysis GETTING BETTER (Carl Friedrich, Lukas)
+- (Re)write _bisect module DONE
 - buildbot improvements, locks, email on failing tests
 - run pypy-c-nojit in the benchmarks
 - make it possible to run benchmark on branches
 - understand problems of mapdict (Carl Friedrich, Armin)
 - rewrite ctypes to use _ffi
-- can classes with non-type metaclasses use typeversion? DONE
-
+- fix the remaining bugs on the jit-unroll-loops branch (Armin, H?kan)
+- have an implementation of Python longs with C long longs
+- work in Pyrolog (Sven, Carl Friedrich around)
 
 scheduled discussions:
 


From fijal at codespeak.net  Fri Oct 29 10:34:33 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Fri, 29 Oct 2010 10:34:33 +0200 (CEST)
Subject: [pypy-svn] r78444 - pypy/trunk/pypy/jit/metainterp
Message-ID: <20101029083433.C159236E0CB@codespeak.net>

Author: fijal
Date: Fri Oct 29 10:34:32 2010
New Revision: 78444

Modified:
   pypy/trunk/pypy/jit/metainterp/compile.py
Log:
kill silly pass


Modified: pypy/trunk/pypy/jit/metainterp/compile.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/compile.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/compile.py	Fri Oct 29 10:34:32 2010
@@ -123,7 +123,6 @@
     if not we_are_translated():
         show_loop(metainterp_sd)
         TreeLoop.check_consistency_of(inputargs, operations)
-        pass
     metainterp_sd.profiler.start_backend()
     debug_start("jit-backend")
     try:


From afa at codespeak.net  Fri Oct 29 10:42:14 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 10:42:14 +0200 (CEST)
Subject: [pypy-svn] r78445 - in pypy/branch/fast-forward/pypy/module/_io: .
	test
Message-ID: <20101029084214.2259236E076@codespeak.net>

Author: afa
Date: Fri Oct 29 10:42:13 2010
New Revision: 78445

Modified:
   pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
   pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
Log:
implement abstract tell() == seek(0, 1)


Modified: pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/interp_iobase.py	Fri Oct 29 10:42:13 2010
@@ -74,6 +74,10 @@
                 space.wrap("I/O operation on closed file"))
 
     @unwrap_spec('self', ObjSpace)
+    def tell_w(self, space):
+        return space.call_method(self, "seek", space.wrap(0), space.wrap(1))
+
+    @unwrap_spec('self', ObjSpace)
     def enter_w(self, space):
         self._check_closed(space)
         return space.wrap(self)
@@ -225,6 +229,7 @@
     next = interp2app(W_IOBase.next_w),
     close = interp2app(W_IOBase.close_w),
     flush = interp2app(W_IOBase.flush_w),
+    tell = interp2app(W_IOBase.tell_w),
     isatty = interp2app(W_IOBase.isatty_w),
     readable = interp2app(W_IOBase.readable_w),
     writable = interp2app(W_IOBase.writable_w),

Modified: pypy/branch/fast-forward/pypy/module/_io/test/test_io.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_io/test/test_io.py	Fri Oct 29 10:42:13 2010
@@ -71,6 +71,13 @@
         import gc; gc.collect()
         assert record == [1, 2, 3]
 
+    def test_tell(self):
+        import io
+        class MyIO(io.IOBase):
+            def seek(self, pos, whence=0):
+                return 42
+        assert MyIO().tell() == 42
+
 class AppTestOpen:
     def setup_class(cls):
         tmpfile = udir.join('tmpfile').ensure()


From arigo at codespeak.net  Fri Oct 29 10:49:38 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 10:49:38 +0200 (CEST)
Subject: [pypy-svn] r78446 - pypy/trunk/pypy/jit/codewriter/test
Message-ID: <20101029084938.C39F136E076@codespeak.net>

Author: arigo
Date: Fri Oct 29 10:49:37 2010
New Revision: 78446

Added:
   pypy/trunk/pypy/jit/codewriter/test/test_void_list.py
Log:
Oups, forgot to check in the test.


Added: pypy/trunk/pypy/jit/codewriter/test/test_void_list.py
==============================================================================
--- (empty file)
+++ pypy/trunk/pypy/jit/codewriter/test/test_void_list.py	Fri Oct 29 10:49:37 2010
@@ -0,0 +1,142 @@
+from pypy.rpython.lltypesystem import lltype
+from pypy.translator.unsimplify import varoftype
+from pypy.objspace.flow.model import Constant
+from pypy.jit.codewriter.jtransform import NotSupported
+
+from pypy.jit.codewriter.test.test_list import builtin_test
+
+# ____________________________________________________________
+# XXX support for lists of voids is limited right now
+
+FIXEDLIST = lltype.Ptr(lltype.GcArray(lltype.Void))
+VARLIST = lltype.Ptr(lltype.GcStruct('VARLIST',
+                                     ('length', lltype.Signed),
+                                     ('items', FIXEDLIST),
+                                     adtmeths={"ITEM": lltype.Void}))
+
+# ____________________________________________________________
+# Fixed lists
+
+def test_newlist():
+    builtin_test('newlist', [], FIXEDLIST,
+                 NotSupported)
+    builtin_test('newlist', [Constant(5, lltype.Signed)], FIXEDLIST,
+                 NotSupported)
+    builtin_test('newlist', [varoftype(lltype.Signed)], FIXEDLIST,
+                 NotSupported)
+    builtin_test('newlist', [Constant(5, lltype.Signed),
+                             Constant(0, lltype.Signed)], FIXEDLIST,
+                 NotSupported)
+    builtin_test('newlist', [Constant(5, lltype.Signed),
+                             Constant(1, lltype.Signed)], FIXEDLIST,
+                 NotSupported)
+    builtin_test('newlist', [Constant(5, lltype.Signed),
+                             varoftype(lltype.Signed)], FIXEDLIST,
+                 NotSupported)
+
+def test_fixed_ll_arraycopy():
+    builtin_test('list.ll_arraycopy',
+                 [varoftype(FIXEDLIST),
+                  varoftype(FIXEDLIST),
+                  varoftype(lltype.Signed), 
+                  varoftype(lltype.Signed), 
+                  varoftype(lltype.Signed)],
+                 lltype.Void,
+                 NotSupported)
+
+def test_fixed_getitem():
+    builtin_test('list.getitem/NONNEG',
+                 [varoftype(FIXEDLIST), varoftype(lltype.Signed)],
+                 lltype.Void, "")
+    builtin_test('list.getitem/NEG',
+                 [varoftype(FIXEDLIST), varoftype(lltype.Signed)],
+                 lltype.Void, "")
+    builtin_test('list.getitem/CANRAISE',
+                 [varoftype(FIXEDLIST), varoftype(lltype.Signed)],
+                 lltype.Void, NotSupported)
+
+def test_fixed_getitem_foldable():
+    builtin_test('list.getitem_foldable/NONNEG',
+                 [varoftype(FIXEDLIST), varoftype(lltype.Signed)],
+                 lltype.Void, "")
+    builtin_test('list.getitem_foldable/NEG',
+                 [varoftype(FIXEDLIST), varoftype(lltype.Signed)],
+                 lltype.Void, "")
+    builtin_test('list.getitem_foldable/CANRAISE',
+                 [varoftype(FIXEDLIST), varoftype(lltype.Signed)],
+                 lltype.Void, NotSupported)
+
+def test_fixed_setitem():
+    builtin_test('list.setitem/NONNEG', [varoftype(FIXEDLIST),
+                                         varoftype(lltype.Signed),
+                                         varoftype(lltype.Void)],
+                 lltype.Void, "")
+    builtin_test('list.setitem/NEG', [varoftype(FIXEDLIST),
+                                      varoftype(lltype.Signed),
+                                      varoftype(lltype.Void)],
+                 lltype.Void, "")
+    builtin_test('list.setitem/CANRAISE', [varoftype(FIXEDLIST),
+                                           varoftype(lltype.Signed),
+                                           varoftype(lltype.Void)],
+                 lltype.Void, NotSupported)
+
+def test_fixed_len():
+    builtin_test('list.len', [varoftype(FIXEDLIST)], lltype.Signed,
+                 NotSupported)
+
+def test_fixed_len_foldable():
+    builtin_test('list.len_foldable', [varoftype(FIXEDLIST)], lltype.Signed,
+                 NotSupported)
+
+# ____________________________________________________________
+# Resizable lists
+
+def test_resizable_newlist():
+    builtin_test('newlist', [], VARLIST,
+                 NotSupported)
+    builtin_test('newlist', [Constant(5, lltype.Signed)], VARLIST,
+                 NotSupported)
+    builtin_test('newlist', [varoftype(lltype.Signed)], VARLIST,
+                 NotSupported)
+    builtin_test('newlist', [Constant(5, lltype.Signed),
+                             Constant(0, lltype.Signed)], VARLIST,
+                 NotSupported)
+    builtin_test('newlist', [Constant(5, lltype.Signed),
+                             Constant(1, lltype.Signed)], VARLIST,
+                 NotSupported)
+    builtin_test('newlist', [Constant(5, lltype.Signed),
+                             varoftype(lltype.Signed)], VARLIST,
+                 NotSupported)
+
+def test_resizable_getitem():
+    builtin_test('list.getitem/NONNEG',
+                 [varoftype(VARLIST), varoftype(lltype.Signed)],
+                 lltype.Void, "")
+    builtin_test('list.getitem/NEG',
+                 [varoftype(VARLIST), varoftype(lltype.Signed)],
+                 lltype.Void, "")
+    builtin_test('list.getitem/CANRAISE',
+                 [varoftype(VARLIST), varoftype(lltype.Signed)],
+                 lltype.Void, NotSupported)
+
+def test_resizable_setitem():
+    builtin_test('list.setitem/NONNEG', [varoftype(VARLIST),
+                                         varoftype(lltype.Signed),
+                                         varoftype(lltype.Void)],
+                 lltype.Void, "")
+    builtin_test('list.setitem/NEG', [varoftype(VARLIST),
+                                      varoftype(lltype.Signed),
+                                      varoftype(lltype.Void)],
+                 lltype.Void, "")
+    builtin_test('list.setitem/CANRAISE', [varoftype(VARLIST),
+                                           varoftype(lltype.Signed),
+                                           varoftype(lltype.Void)],
+                 lltype.Void, NotSupported)
+
+def test_resizable_len():
+    builtin_test('list.len', [varoftype(VARLIST)], lltype.Signed,
+                 NotSupported)
+
+def test_resizable_unsupportedop():
+    builtin_test('list.foobar', [varoftype(VARLIST)], lltype.Signed,
+                 NotSupported)


From arigo at codespeak.net  Fri Oct 29 10:54:38 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 10:54:38 +0200 (CEST)
Subject: [pypy-svn] r78447 - in pypy/trunk/pypy/rlib: . test
Message-ID: <20101029085438.A9B26282BE8@codespeak.net>

Author: arigo
Date: Fri Oct 29 10:54:37 2010
New Revision: 78447

Modified:
   pypy/trunk/pypy/rlib/rbigint.py
   pypy/trunk/pypy/rlib/test/test_rbigint.py
Log:
Move here with a good implementation some logic that is so
far in objspace/std/strutil.py and horribly slow.


Modified: pypy/trunk/pypy/rlib/rbigint.py
==============================================================================
--- pypy/trunk/pypy/rlib/rbigint.py	(original)
+++ pypy/trunk/pypy/rlib/rbigint.py	Fri Oct 29 10:54:37 2010
@@ -1596,11 +1596,16 @@
 
 # a few internal helpers
 
-DEC_PER_DIGIT = 1
-while int('9' * DEC_PER_DIGIT) < MASK:
-    DEC_PER_DIGIT += 1
-DEC_PER_DIGIT -= 1
-DEC_MAX = 10 ** DEC_PER_DIGIT
+def digits_max_for_base(base):
+    dec_per_digit = 1
+    while base ** dec_per_digit < MASK:
+        dec_per_digit += 1
+    dec_per_digit -= 1
+    return base ** dec_per_digit
+
+BASE_MAX = [0, 0] + [digits_max_for_base(_base) for _base in range(2, 37)]
+DEC_MAX = digits_max_for_base(10)
+assert DEC_MAX == BASE_MAX[10]
 
 def _decimalstr_to_bigint(s):
     # a string that has been already parsed to be decimal and valid,
@@ -1615,7 +1620,6 @@
         p += 1
 
     a = rbigint.fromint(0)
-    cnt = DEC_PER_DIGIT
     tens = 1
     dig = 0
     ord0 = ord('0')
@@ -1631,4 +1635,23 @@
         a.sign = -1
     return a
 
-
+def parse_digit_string(parser):
+    # helper for objspace.std.strutil
+    a = rbigint.fromint(0)
+    base = parser.base
+    digitmax = BASE_MAX[base]
+    tens, dig = 1, 0
+    while True:
+        digit = parser.next_digit()
+        if tens == digitmax or digit < 0:
+            a = _muladd1(a, tens, dig)
+            if digit < 0:
+                break
+            dig = digit
+            tens = base
+        else:
+            dig = dig * base + digit
+            tens *= base
+    if parser.sign == -1:
+        a.sign = -1
+    return a

Modified: pypy/trunk/pypy/rlib/test/test_rbigint.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_rbigint.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_rbigint.py	Fri Oct 29 10:54:37 2010
@@ -448,6 +448,30 @@
         assert (rbigint.fromlong(-9**50).ulonglongmask() ==
                 r_ulonglong(-9**50))
 
+    def test_parse_digit_string(self):
+        from pypy.rlib.rbigint import parse_digit_string
+        class Parser:
+            def __init__(self, base, sign, digits):
+                self.base = base
+                self.sign = sign
+                self.next_digit = iter(digits + [-1]).next
+        x = parse_digit_string(Parser(10, 1, [6]))
+        assert x.eq(rbigint.fromint(6))
+        x = parse_digit_string(Parser(10, 1, [6, 2, 3]))
+        assert x.eq(rbigint.fromint(623))
+        x = parse_digit_string(Parser(10, -1, [6, 2, 3]))
+        assert x.eq(rbigint.fromint(-623))
+        x = parse_digit_string(Parser(16, 1, [0xA, 0x4, 0xF]))
+        assert x.eq(rbigint.fromint(0xA4F))
+        num = 0
+        for i in range(36):
+            x = parse_digit_string(Parser(36, 1, range(i)))
+            assert x.eq(rbigint.fromlong(num))
+            num = num * 36 + i
+        x = parse_digit_string(Parser(16, -1, range(15,-1,-1)*99))
+        assert x.eq(rbigint.fromlong(long('-0x' + 'FEDCBA9876543210'*99, 16)))
+
+
 BASE = 2 ** SHIFT
 
 class TestTranslatable(object):


From arigo at codespeak.net  Fri Oct 29 10:58:01 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 10:58:01 +0200 (CEST)
Subject: [pypy-svn] r78448 - pypy/trunk/pypy/objspace/std
Message-ID: <20101029085801.5A678282BE8@codespeak.net>

Author: arigo
Date: Fri Oct 29 10:58:00 2010
New Revision: 78448

Modified:
   pypy/trunk/pypy/objspace/std/smallintobject.py
Log:
Kill most import; they are unneeded.


Modified: pypy/trunk/pypy/objspace/std/smallintobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/smallintobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/smallintobject.py	Fri Oct 29 10:58:00 2010
@@ -2,18 +2,13 @@
 Implementation of small ints, stored as odd-valued pointers in the
 translated PyPy.  To enable them, see inttype.py.
 """
-import types
-from pypy.interpreter.error import OperationError
 from pypy.objspace.std import intobject
 from pypy.objspace.std.model import registerimplementation, W_Object
 from pypy.objspace.std.register_all import register_all
-from pypy.objspace.std.multimethod import FailedToImplementArgs
 from pypy.objspace.std.noneobject import W_NoneObject
-from pypy.rlib.rarithmetic import ovfcheck, ovfcheck_lshift, LONG_BIT, r_uint
-from pypy.objspace.std.inttype import wrapint
-from pypy.objspace.std.intobject import W_IntObject, _impl_int_int_pow
+from pypy.objspace.std.intobject import W_IntObject
 from pypy.rlib.objectmodel import UnboxedValue
-from pypy.rlib.rbigint import rbigint
+from pypy.tool.sourcetools import func_with_new_name
 
 
 class W_SmallIntObject(W_Object, UnboxedValue):
@@ -46,9 +41,7 @@
             new_name = name.replace("Int", "SmallInt")
             # Copy the function, so the annotator specializes it for
             # W_SmallIntObject.
-            ns[new_name] = types.FunctionType(func.func_code, ns, new_name,
-                                              func.func_defaults,
-                                              func.func_closure)
+            ns[new_name] = func_with_new_name(func, new_name)
     ns["get_integer"] = ns["pos__SmallInt"] = ns["int__SmallInt"]
     ns["get_negint"] = ns["neg__SmallInt"]
 


From arigo at codespeak.net  Fri Oct 29 10:58:49 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 10:58:49 +0200 (CEST)
Subject: [pypy-svn] r78449 - pypy/trunk/pypy/objspace/std/test
Message-ID: <20101029085849.3940A36E0CB@codespeak.net>

Author: arigo
Date: Fri Oct 29 10:58:47 2010
New Revision: 78449

Modified:
   pypy/trunk/pypy/objspace/std/test/test_longobject.py
Log:
This test contains tons of operations that are actually constant-folded
by the CPython compiler.  So it doesn't really test anything :-(


Modified: pypy/trunk/pypy/objspace/std/test/test_longobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_longobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_longobject.py	Fri Oct 29 10:58:47 2010
@@ -41,56 +41,64 @@
 
 class AppTestLong:
     def test_add(self):
-        assert int(123L + 12443L) == 123 + 12443
-        assert -20 + 2 + 3L + True == -14L
+        x = 123L
+        assert int(x + 12443L) == 123 + 12443
+        x = -20
+        assert x + 2 + 3L + True == -14L
 
     def test_sub(self):
-        assert int(58543L - 12332L) == 58543 - 12332
-        assert 237123838281233L * 12 == 237123838281233L * 12L
+        x = 58543L
+        assert int(x - 12332L) == 58543 - 12332
+        x = 237123838281233L
+        assert x * 12 == x * 12L
 
     def test_mul(self):
-        assert 363L * 2 ** 40 == 363L << 40
+        x = 363L
+        assert x * 2 ** 40 == x << 40
 
     def test_truediv(self):
         exec "from __future__ import division; a = 31415926L / 10000000L"
         assert a == 3.1415926
 
     def test_floordiv(self):
-        a = 31415926L // 10000000L
+        x = 31415926L
+        a = x // 10000000L
         assert a == 3L
 
     def test_compare(self):
+        Z = 0
+        ZL = 0L
         for BIG in (1L, 1L << 62, 1L << 9999):
-            assert 0 == 0L
-            assert not (0 != 0L)
-            assert 0L == 0
-            assert not (0L != 0)
-            assert not (0 == BIG)
-            assert 0 != BIG
-            assert not (BIG == 0)
-            assert BIG != 0
-            assert not (0L == BIG)
-            assert 0L != BIG
-            assert 0 <= 0L
-            assert not (0 < 0L)
-            assert 0 <= BIG
-            assert 0 < BIG
-            assert not (BIG <= 0)
-            assert not (BIG < 0)
-            assert 0L <= 0L
-            assert not (0L < 0L)
-            assert 0L <= BIG
-            assert 0L < BIG
-            assert not (BIG <= 0L)
-            assert not (BIG < 0L)
-            assert not (0 <= -BIG)
-            assert not (0 < -BIG)
-            assert -BIG <= 0
-            assert -BIG < 0
-            assert not (0L <= -BIG)
-            assert not (0L < -BIG)
-            assert -BIG <= 0L
-            assert -BIG < 0L
+            assert Z == ZL
+            assert not (Z != ZL)
+            assert ZL == Z
+            assert not (ZL != Z)
+            assert not (Z == BIG)
+            assert Z != BIG
+            assert not (BIG == Z)
+            assert BIG != Z
+            assert not (ZL == BIG)
+            assert ZL != BIG
+            assert Z <= ZL
+            assert not (Z < ZL)
+            assert Z <= BIG
+            assert Z < BIG
+            assert not (BIG <= Z)
+            assert not (BIG < Z)
+            assert ZL <= ZL
+            assert not (ZL < ZL)
+            assert ZL <= BIG
+            assert ZL < BIG
+            assert not (BIG <= ZL)
+            assert not (BIG < ZL)
+            assert not (Z <= -BIG)
+            assert not (Z < -BIG)
+            assert -BIG <= Z
+            assert -BIG < Z
+            assert not (ZL <= -BIG)
+            assert not (ZL < -BIG)
+            assert -BIG <= ZL
+            assert -BIG < ZL
             #
             assert not (BIG <  int(BIG))
             assert     (BIG <= int(BIG))
@@ -137,7 +145,8 @@
     def test_conversion(self):
         class long2(long):
             pass
-        x = long2(1L<<100)
+        x = 1L
+        x = long2(x<<100)
         y = int(x)
         assert type(y) == long
         assert type(+long2(5)) is long
@@ -152,7 +161,8 @@
         assert type(long2(5) // 1) is long
 
     def test_pow(self):
-        assert pow(0L, 0L, 1L) == 0L
+        x = 0L
+        assert pow(x, 0L, 1L) == 0L
 
     def test_getnewargs(self):
         assert  0L .__getnewargs__() == (0L,)
@@ -193,13 +203,14 @@
         assert oct(01234567012345670L) == '01234567012345670L'
 
     def test_bits(self):
-        assert 0xAAAAAAAAL | 0x55555555L == 0xFFFFFFFFL
-        assert 0xAAAAAAAAL & 0x55555555L == 0x00000000L
-        assert 0xAAAAAAAAL ^ 0x55555555L == 0xFFFFFFFFL
-        assert -0xAAAAAAAAL | 0x55555555L == -0xAAAAAAA9L
-        assert 0xAAAAAAAAL | 0x555555555L == 0x5FFFFFFFFL
-        assert 0xAAAAAAAAL & 0x555555555L == 0x000000000L
-        assert 0xAAAAAAAAL ^ 0x555555555L == 0x5FFFFFFFFL
+        x = 0xAAAAAAAAL
+        assert x | 0x55555555L == 0xFFFFFFFFL
+        assert x & 0x55555555L == 0x00000000L
+        assert x ^ 0x55555555L == 0xFFFFFFFFL
+        assert -x | 0x55555555L == -0xAAAAAAA9L
+        assert x | 0x555555555L == 0x5FFFFFFFFL
+        assert x & 0x555555555L == 0x000000000L
+        assert x ^ 0x555555555L == 0x5FFFFFFFFL
 
     def test_hash(self):
         # ints have the same hash as equal longs
@@ -229,7 +240,8 @@
 
     def test_huge_longs(self):
         import operator
-        huge = 1L << 40000L
+        x = 1L
+        huge = x << 40000L
         raises(OverflowError, float, huge)
         raises(OverflowError, operator.truediv, huge, 3)
         raises(OverflowError, operator.truediv, huge, 3L)


From arigo at codespeak.net  Fri Oct 29 10:59:46 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 10:59:46 +0200 (CEST)
Subject: [pypy-svn] r78450 - pypy/trunk/pypy/objspace/std
Message-ID: <20101029085946.E255C282C1E@codespeak.net>

Author: arigo
Date: Fri Oct 29 10:59:45 2010
New Revision: 78450

Modified:
   pypy/trunk/pypy/objspace/std/inttype.py
   pypy/trunk/pypy/objspace/std/longtype.py
   pypy/trunk/pypy/objspace/std/strutil.py
Log:
Use the more efficient parse_digit_string() from rbigint.


Modified: pypy/trunk/pypy/objspace/std/inttype.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/inttype.py	(original)
+++ pypy/trunk/pypy/objspace/std/inttype.py	Fri Oct 29 10:59:45 2010
@@ -1,7 +1,7 @@
 from pypy.interpreter import gateway
 from pypy.interpreter.error import OperationError
 from pypy.objspace.std.stdtypedef import StdTypeDef
-from pypy.objspace.std.strutil import (string_to_int, string_to_w_long,
+from pypy.objspace.std.strutil import (string_to_int, string_to_bigint,
                                        ParseStringError,
                                        ParseStringOverflowError)
 from pypy.rlib.rarithmetic import r_uint
@@ -45,10 +45,12 @@
 def retry_to_w_long(space, parser, base=0):
     parser.rewind()
     try:
-        return string_to_w_long(space, None, base=base, parser=parser)
+        bigint = string_to_bigint(None, base=base, parser=parser)
     except ParseStringError, e:
         raise OperationError(space.w_ValueError,
                              space.wrap(e.msg))
+    from pypy.objspace.std.longobject import W_LongObject
+    return W_LongObject(bigint)
 
 def descr__new__(space, w_inttype, w_x=0, w_base=gateway.NoneNotWrapped):
     from pypy.objspace.std.intobject import W_IntObject

Modified: pypy/trunk/pypy/objspace/std/longtype.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/longtype.py	(original)
+++ pypy/trunk/pypy/objspace/std/longtype.py	Fri Oct 29 10:59:45 2010
@@ -1,7 +1,7 @@
 from pypy.interpreter.error import OperationError
 from pypy.interpreter import gateway
 from pypy.objspace.std.stdtypedef import StdTypeDef
-from pypy.objspace.std.strutil import string_to_w_long, ParseStringError
+from pypy.objspace.std.strutil import string_to_bigint, ParseStringError
 
 def descr__new__(space, w_longtype, w_x=0, w_base=gateway.NoneNotWrapped):
     from pypy.objspace.std.longobject import W_LongObject
@@ -9,10 +9,10 @@
     if w_base is None:
         # check for easy cases
         if type(w_value) is W_LongObject:
-            pass
+            bigint = w_value.num
         elif space.is_true(space.isinstance(w_value, space.w_str)):
             try:
-                w_value = string_to_w_long(space, space.str_w(w_value))
+                bigint = string_to_bigint(space.str_w(w_value))
             except ParseStringError, e:
                 raise OperationError(space.w_ValueError,
                                      space.wrap(e.msg))
@@ -22,7 +22,7 @@
                     from pypy.objspace.std.ropeunicodeobject import unicode_to_decimal_w
                 else:
                     from pypy.objspace.std.unicodeobject import unicode_to_decimal_w
-                w_value = string_to_w_long(space, unicode_to_decimal_w(space, w_value))
+                bigint = string_to_bigint(unicode_to_decimal_w(space, w_value))
             except ParseStringError, e:
                 raise OperationError(space.w_ValueError,
                                      space.wrap(e.msg))
@@ -35,10 +35,11 @@
             if space.is_true(space.isinstance(w_obj, space.w_long)):
                 assert isinstance(w_obj, W_LongObject)  # XXX this could fail!
                 # XXX find a way to do that even if w_obj is not a W_LongObject
-                w_value = w_obj
+                bigint = w_obj.num
             elif space.is_true(space.isinstance(w_obj, space.w_int)):
+                from pypy.rlib.rbigint import rbigint
                 intval = space.int_w(w_obj)
-                w_value = W_LongObject.fromint(space, intval)
+                bigint = rbigint.fromint(intval)
             else:
                 raise OperationError(space.w_ValueError,
                                     space.wrap("value can't be converted to long"))
@@ -56,13 +57,13 @@
                                      space.wrap("long() can't convert non-string "
                                                 "with explicit base"))
         try:
-            w_value = string_to_w_long(space, s, base)
+            bigint = string_to_bigint(s, base)
         except ParseStringError, e:
             raise OperationError(space.w_ValueError,
                                  space.wrap(e.msg))
 
     w_obj = space.allocate_instance(W_LongObject, w_longtype)
-    W_LongObject.__init__(w_obj, w_value.num)
+    W_LongObject.__init__(w_obj, bigint)
     return w_obj
 
 # ____________________________________________________________

Modified: pypy/trunk/pypy/objspace/std/strutil.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/strutil.py	(original)
+++ pypy/trunk/pypy/objspace/std/strutil.py	Fri Oct 29 10:59:45 2010
@@ -4,6 +4,7 @@
 
 from pypy.rlib.rarithmetic import ovfcheck, break_up_float, parts_to_float,\
      INFINITY, NAN
+from pypy.rlib.rbigint import rbigint, parse_digit_string
 from pypy.interpreter.error import OperationError
 import math
 
@@ -91,9 +92,10 @@
             return -1
 
 def string_to_int(s, base=10):
-    """Utility to converts a string to an integer (or possibly a long).
+    """Utility to converts a string to an integer.
     If base is 0, the proper base is guessed based on the leading
     characters of 's'.  Raises ParseStringError in case of error.
+    Raises ParseStringOverflowError in case the result does not fit.
     """
     s = literal = strip_spaces(s)
     p = NumberStringParser(s, literal, base, 'int')
@@ -113,11 +115,9 @@
         except OverflowError:
             raise ParseStringOverflowError(p)
 
-def string_to_long(space, s, base=10, parser=None):
-    return string_to_w_long(space, s, base, parser).longval()
-
-def string_to_w_long(space, s, base=10, parser=None):
-    """As string_to_int(), but ignores an optional 'l' or 'L' suffix."""
+def string_to_bigint(s, base=10, parser=None):
+    """As string_to_int(), but ignores an optional 'l' or 'L' suffix
+    and returns an rbigint."""
     if parser is None:
         s = literal = strip_spaces(s)
         if (s.endswith('l') or s.endswith('L')) and base < 22:
@@ -126,18 +126,7 @@
         p = NumberStringParser(s, literal, base, 'long')
     else:
         p = parser
-    w_base = space.newlong(p.base)
-    w_result = space.newlong(0)
-    while True:
-        digit = p.next_digit()
-        if digit == -1:
-            if p.sign == -1:
-                w_result = space.neg(w_result)
-            # XXX grumble
-            from pypy.objspace.std.longobject import W_LongObject
-            assert isinstance(w_result, W_LongObject)
-            return w_result
-        w_result = space.add(space.mul(w_result,w_base), space.newlong(digit))
+    return parse_digit_string(p)
 
 # Tim's comment:
 # 57 bits are more than needed in any case.


From afa at codespeak.net  Fri Oct 29 11:23:03 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 11:23:03 +0200 (CEST)
Subject: [pypy-svn] r78451 -
	pypy/branch/fast-forward/lib-python/modified-2.7.0/test
Message-ID: <20101029092303.8C97236E3BC@codespeak.net>

Author: afa
Date: Fri Oct 29 11:23:01 2010
New Revision: 78451

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_socket.py
      - copied, changed from r78441, pypy/branch/fast-forward/lib-python/2.7.0/test/test_socket.py
Modified:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py
Log:
More gc_collect() to avoid a deadlock and make more tests pass


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_socket.py (from r78441, pypy/branch/fast-forward/lib-python/2.7.0/test/test_socket.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_socket.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_socket.py	Fri Oct 29 11:23:01 2010
@@ -160,6 +160,7 @@
 
     def clientTearDown(self):
         self.done.set()
+        test_support.gc_collect()
         thread.exit()
 
 class ThreadedTCPSocketTest(SocketTCPTest, ThreadableTest):
@@ -246,6 +247,7 @@
         self.assertEqual(p.fileno(), s.fileno())
         s.close()
         s = None
+        test_support.gc_collect()
         try:
             p.fileno()
         except ReferenceError:
@@ -727,6 +729,7 @@
     def testClose(self):
         conn, addr = self.serv.accept()
         conn.close()
+        test_support.gc_collect() # not sure why
 
         sd = self.cli
         read, write, err = select.select([sd], [], [], 1.0)

Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py
==============================================================================
--- pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py	Fri Oct 29 11:23:01 2010
@@ -281,7 +281,9 @@
     tempsock = socket.socket(family, socktype)
     port = bind_port(tempsock)
     tempsock.close()
+    # Be sure to close the socket so the port can be reused
     del tempsock
+    gc_collect()
     return port
 
 def bind_port(sock, host=HOST):


From arigo at codespeak.net  Fri Oct 29 11:35:42 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 11:35:42 +0200 (CEST)
Subject: [pypy-svn] r78452 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101029093542.0AF8B282BE8@codespeak.net>

Author: arigo
Date: Fri Oct 29 11:35:40 2010
New Revision: 78452

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py
Log:
Fix this test.

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py	Fri Oct 29 11:35:40 2010
@@ -599,19 +599,19 @@
                            get_printable_location = lambda codeno : str(codeno))
 
         def portal(codeno, j):
-            i = 0
+            i = 1
             while 1:
                 driver.jit_merge_point(codeno=codeno, i=i, j=j)
-                if i == 1:
+                if (i >> 1) == 1:
                     if j == 0:
                         return
                     portal(2, j - 1)
-                elif i == 3:
+                elif i == 5:
                     return
                 i += 1
                 driver.can_enter_jit(codeno=codeno, i=i, j=j)
 
-        portal(2, 50)
+        portal(2, 5)
 
         from pypy.jit.metainterp import compile, pyjitpl
         pyjitpl._warmrunnerdesc = None
@@ -623,8 +623,8 @@
         original_ctc = compile.compile_tmp_callback
         try:
             compile.compile_tmp_callback = my_ctc
-            self.meta_interp(portal, [2, 20], inline=True)
-            self.check_loops(call_assembler=1, call_may_force=0,
+            self.meta_interp(portal, [2, 5], inline=True)
+            self.check_loops(call_assembler=2, call_may_force=0,
                              everywhere=True)
         finally:
             compile.compile_tmp_callback = original_ctc


From hakanardo at codespeak.net  Fri Oct 29 11:40:02 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Fri, 29 Oct 2010 11:40:02 +0200 (CEST)
Subject: [pypy-svn] r78453 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101029094002.6D5AF36E0CB@codespeak.net>

Author: hakanardo
Date: Fri Oct 29 11:40:00 2010
New Revision: 78453

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py
Log:
repeated fixed

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py	Fri Oct 29 11:40:00 2010
@@ -652,14 +652,14 @@
                            get_printable_location = lambda codeno : str(codeno))
 
         def portal(codeno, frame):
-            i = 0
+            i = 1
             while 1:
                 driver.jit_merge_point(codeno=codeno, i=i, frame=frame)
-                if i == 1:
+                if (i >> 1) == 1:
                     if frame.j == 0:
                         return
                     portal(2, Frame(frame.j - 1))
-                elif i == 3:
+                elif i == 5:
                     return
                 i += 1
                 driver.can_enter_jit(codeno=codeno, i=i, frame=frame)
@@ -667,7 +667,7 @@
         def main(codeno, j):
             portal(codeno, Frame(j))
 
-        main(2, 50)
+        main(2, 5)
 
         from pypy.jit.metainterp import compile, pyjitpl
         pyjitpl._warmrunnerdesc = None
@@ -679,8 +679,8 @@
         original_ctc = compile.compile_tmp_callback
         try:
             compile.compile_tmp_callback = my_ctc
-            self.meta_interp(main, [2, 20], inline=True)
-            self.check_loops(call_assembler=1, call_may_force=0,
+            self.meta_interp(main, [2, 5], inline=True)
+            self.check_loops(call_assembler=2, call_may_force=0,
                              everywhere=True)
         finally:
             compile.compile_tmp_callback = original_ctc


From afa at codespeak.net  Fri Oct 29 11:51:42 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 11:51:42 +0200 (CEST)
Subject: [pypy-svn] r78454 -
	pypy/branch/fast-forward/lib-python/modified-2.7.0/test
Message-ID: <20101029095142.DCC85282BE8@codespeak.net>

Author: afa
Date: Fri Oct 29 11:51:41 2010
New Revision: 78454

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_zlib.py
      - copied, changed from r78441, pypy/branch/fast-forward/lib-python/2.7.0/test/test_zlib.py
Log:
Use os.urandom() instead of complex stuff.
This allows the test to run without timing out...


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_zlib.py (from r78441, pypy/branch/fast-forward/lib-python/2.7.0/test/test_zlib.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_zlib.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_zlib.py	Fri Oct 29 11:51:41 2010
@@ -1,6 +1,7 @@
 import unittest
 from test import test_support
 import binascii
+import os
 import random
 from test.test_support import precisionbigmemtest, _1G
 
@@ -93,14 +94,7 @@
 
 class BaseCompressTestCase(object):
     def check_big_compress_buffer(self, size, compress_func):
-        _1M = 1024 * 1024
-        fmt = "%%0%dx" % (2 * _1M)
-        # Generate 10MB worth of random, and expand it by repeating it.
-        # The assumption is that zlib's memory is not big enough to exploit
-        # such spread out redundancy.
-        data = ''.join([binascii.a2b_hex(fmt % random.getrandbits(8 * _1M))
-                        for i in range(10)])
-        data = data * (size // len(data) + 1)
+        data = os.urandom(size)
         try:
             compress_func(data)
         finally:


From hakanardo at codespeak.net  Fri Oct 29 11:59:21 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Fri, 29 Oct 2010 11:59:21 +0200 (CEST)
Subject: [pypy-svn] r78455 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101029095921.CBEAA36E076@codespeak.net>

Author: hakanardo
Date: Fri Oct 29 11:59:20 2010
New Revision: 78455

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitprof.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitprof.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitprof.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitprof.py	Fri Oct 29 11:59:20 2010
@@ -55,6 +55,8 @@
             TRACING,
             BACKEND,
             ~ BACKEND,
+            BACKEND,
+            ~ BACKEND,
             ~ TRACING,
             RUNNING,
             ~ RUNNING,
@@ -62,8 +64,8 @@
             ~ BLACKHOLE
             ]
         assert profiler.events == expected
-        assert profiler.times == [2, 1, 1, 1]
-        assert profiler.counters == [1, 1, 1, 1, 3, 3, 1, 7, 1, 0, 0, 0,
+        assert profiler.times == [3, 2, 1, 1]
+        assert profiler.counters == [1, 2, 1, 1, 3, 3, 1, 13, 2, 0, 0, 0,
                                      0, 0, 0, 0]
 
     def test_simple_loop_with_call(self):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py	Fri Oct 29 11:59:20 2010
@@ -1016,7 +1016,7 @@
         res = self.meta_interp(portal, [2, 0], inline=True,
                                policy=StopAtXPolicy(residual))
         assert res == portal(2, 0)
-        self.check_loops(call_assembler=2)
+        self.check_loops(call_assembler=3, everywhere=True)
 
     def test_inline_without_hitting_the_loop(self):
         driver = JitDriver(greens = ['codeno'], reds = ['i'],


From cfbolz at codespeak.net  Fri Oct 29 12:03:21 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 29 Oct 2010 12:03:21 +0200 (CEST)
Subject: [pypy-svn] r78456 - in pypy/trunk/pypy: interpreter
	objspace/std/test
Message-ID: <20101029100321.7A0E2282C1E@codespeak.net>

Author: cfbolz
Date: Fri Oct 29 12:03:20 2010
New Revision: 78456

Modified:
   pypy/trunk/pypy/interpreter/typedef.py
   pypy/trunk/pypy/objspace/std/test/test_mapdict.py
Log:
fix one of the nightly failures


Modified: pypy/trunk/pypy/interpreter/typedef.py
==============================================================================
--- pypy/trunk/pypy/interpreter/typedef.py	(original)
+++ pypy/trunk/pypy/interpreter/typedef.py	Fri Oct 29 12:03:20 2010
@@ -203,6 +203,7 @@
         from pypy.objspace.std.mapdict import BaseMapdictObject, ObjectMixin
         add(BaseMapdictObject)
         add(ObjectMixin)
+        body["user_overridden_class"] = True
         features = ()
 
     if "user" in features:     # generic feature needed by all subcls

Modified: pypy/trunk/pypy/objspace/std/test/test_mapdict.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_mapdict.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_mapdict.py	Fri Oct 29 12:03:20 2010
@@ -777,3 +777,19 @@
         assert res == (0, 0, 1)
         res = self.check(f, 'x')
         assert res == (0, 0, 1)
+
+class TestDictSubclassShortcutBug(object):
+    def setup_class(cls):
+        cls.space = gettestobjspace(
+            **{"objspace.std.withmapdict": True,
+               "objspace.std.withmethodcachecounter": True})
+
+    def test_bug(self):
+        w_dict = self.space.appexec([], """():
+                class A(dict):
+                    def __getitem__(self, key):
+                        return 1
+                assert eval("a", globals(), A()) == 1
+                return A()
+                """)
+        assert w_dict.user_overridden_class


From arigo at codespeak.net  Fri Oct 29 13:26:45 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 13:26:45 +0200 (CEST)
Subject: [pypy-svn] r78457 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101029112645.4AFF736E076@codespeak.net>

Author: arigo
Date: Fri Oct 29 13:26:43 2010
New Revision: 78457

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
Log:
(hakanardo, arigo)
Change the interface of get_forced_boxes() to enum_forced_boxes().
Implement it for VStrings too.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	Fri Oct 29 13:26:43 2010
@@ -16,7 +16,7 @@
                      OptIntBounds(),
                      OptRewrite(),
                      OptVirtualize(),
-#                     OptString(),
+                     OptString(),
                      OptHeap(),
                     ]
     optimizer = Optimizer(metainterp_sd, loop, optimizations, virtuals)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	Fri Oct 29 13:26:43 2010
@@ -47,14 +47,11 @@
     def get_key_box(self):
         return self.box
 
-    # FIXME: Reuse get_args_for_fail instead?
-    def get_forced_boxes(self, already_seen):
+    def enum_forced_boxes(self, boxes, already_seen):
         key = self.get_key_box()
-        if key in already_seen:
-            return []
-        box = self.force_box()
-        already_seen.append(self.get_key_box())
-        return [box]
+        if key not in already_seen:
+            boxes.append(self.force_box())
+            already_seen[self.get_key_box()] = None
 
     def get_args_for_fail(self, modifier):
         pass

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py	Fri Oct 29 13:26:43 2010
@@ -108,6 +108,17 @@
             for value in self._chars:
                 value.get_args_for_fail(modifier)
 
+    def enum_forced_boxes(self, boxes, already_seen):
+        key = self.get_key_box()
+        if key in already_seen:
+            return
+        already_seen.append(key)
+        if self.box is None:
+            for box in self._chars:
+                box.enum_forced_boxes(boxes, already_seen)
+        else:
+            boxes.append(self.box)
+
     def _make_virtual(self, modifier):
         return modifier.make_vstrplain()
 
@@ -149,6 +160,17 @@
             self.left.get_args_for_fail(modifier)
             self.right.get_args_for_fail(modifier)
 
+    def enum_forced_boxes(self, boxes, already_seen):
+        key = self.get_key_box()
+        if key in already_seen:
+            return
+        already_seen.append(key)
+        if self.box is None:
+            self.left.enum_forced_boxes(boxes, already_seen)
+            self.right.enum_forced_boxes(boxes, already_seen)
+        else:
+            boxes.append(self.box)
+
     def _make_virtual(self, modifier):
         return modifier.make_vstrconcat()
 
@@ -194,6 +216,18 @@
             self.vstart.get_args_for_fail(modifier)
             self.vlength.get_args_for_fail(modifier)
 
+    def enum_forced_boxes(self, boxes, already_seen):
+        key = self.get_key_box()
+        if key in already_seen:
+            return
+        already_seen.append(key)
+        if self.box is None:
+            self.vstr.enum_forced_boxes(boxes, already_seen)
+            self.vstart.enum_forced_boxes(boxes, already_seen)
+            self.vlength.enum_forced_boxes(boxes, already_seen)
+        else:
+            boxes.append(self.box)
+
     def _make_virtual(self, modifier):
         return modifier.make_vstrslice()
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Fri Oct 29 13:26:43 2010
@@ -51,9 +51,11 @@
         self.snapshot_map ={None: None}
         
         inputargs = []
-        seen = []
+        seen = {}
         for arg in jump_args:
-            for a in self.getvalue(arg).get_forced_boxes(seen):
+            boxes = []
+            self.getvalue(arg).enum_forced_boxes(boxes, seen)
+            for a in boxes:
                 if not isinstance(a, Const):
                     inputargs.append(a)
 
@@ -75,8 +77,9 @@
             if newop.getopnum() == rop.JUMP:
                 args = []
                 #for arg in newop.getarglist():
-                for arg in [argmap[a] for a in inputargs]:
-                    args.extend(self.getvalue(arg).get_forced_boxes([]))
+                for arg in inputargs:
+                    arg = argmap[arg]
+                    self.getvalue(arg).enum_forced_boxes(args, {})
                 newop.initarglist(args + inputargs[len(args):])
 
             #print 'P: ', str(newop)
@@ -105,8 +108,9 @@
                             jmp = self.optimizer.newoperations[-1]
                             if jmp.getopnum() == rop.JUMP:
                                 newval = self.getvalue(argmap[a])
-                                newarg = newval.get_forced_boxes([])
-                                jmp.initarglist(jmp.getarglist() + newarg)
+                                boxes = jmp.getarglist()[:]
+                                newval.enum_forced_boxes(boxes, {})
+                                jmp.initarglist(boxes)
 
         return inputargs
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Fri Oct 29 13:26:43 2010
@@ -121,20 +121,17 @@
                 fieldvalue = self._fields[ofs]
                 fieldvalue.get_args_for_fail(modifier)
 
-    def get_forced_boxes(self, already_seen):
+    def enum_forced_boxes(self, boxes, already_seen):
         key = self.get_key_box()
         if key in already_seen:
-            return []
-        already_seen.append(key)
+            return
+        already_seen[key] = None
         if self.box is None:
             lst = self._get_field_descr_list()
-            fieldboxes = []
             for ofs in lst:
-                boxes = self._fields[ofs].get_forced_boxes(already_seen)
-                fieldboxes.extend(boxes)
-            return fieldboxes
+                self._fields[ofs].enum_forced_boxes(boxes, already_seen)
         else:
-            return [self.box]
+            boxes.append(self.box)
 
 
 class VirtualValue(AbstractVirtualStructValue):
@@ -208,18 +205,16 @@
     def _make_virtual(self, modifier):
         return modifier.make_varray(self.arraydescr)
 
-    def get_forced_boxes(self, already_seen):
+    def enum_forced_boxes(self, boxes, already_seen):
         key = self.get_key_box()
         if key in already_seen:
-            return []
+            return
         already_seen.append(key)
         if self.box is None:
-            boxes = []
             for itemvalue in self._items:
-                boxes.extend(itemvalue.get_forced_boxes(already_seen))
-            return boxes
+                itemvalue.enum_forced_boxes(boxes, already_seen)
         else:
-            return [self.box]
+            boxes.append(self.box)
 
 
 class __extend__(SpecNode):


From hakanardo at codespeak.net  Fri Oct 29 13:28:31 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Fri, 29 Oct 2010 13:28:31 +0200 (CEST)
Subject: [pypy-svn] r78458 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101029112831.A7534282BE8@codespeak.net>

Author: hakanardo
Date: Fri Oct 29 13:28:30 2010
New Revision: 78458

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_list.py
Log:
(hakanardo, arigo) fixed test

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_list.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_list.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_list.py	Fri Oct 29 13:28:30 2010
@@ -216,4 +216,4 @@
             return a * b
         res = self.meta_interp(f, [37])
         assert res == f(37)
-        self.check_loops(getfield_gc=1)
+        self.check_loops(getfield_gc=1, everywhere=True)


From arigo at codespeak.net  Fri Oct 29 13:29:01 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 13:29:01 +0200 (CEST)
Subject: [pypy-svn] r78459 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101029112901.D414936E076@codespeak.net>

Author: arigo
Date: Fri Oct 29 13:29:00 2010
New Revision: 78459

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
Log:
Oups.  Tests are useful, if only they passed originally :-/


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py	Fri Oct 29 13:29:00 2010
@@ -112,7 +112,7 @@
         key = self.get_key_box()
         if key in already_seen:
             return
-        already_seen.append(key)
+        already_seen[key] = None
         if self.box is None:
             for box in self._chars:
                 box.enum_forced_boxes(boxes, already_seen)
@@ -164,7 +164,7 @@
         key = self.get_key_box()
         if key in already_seen:
             return
-        already_seen.append(key)
+        already_seen[key] = None
         if self.box is None:
             self.left.enum_forced_boxes(boxes, already_seen)
             self.right.enum_forced_boxes(boxes, already_seen)
@@ -220,7 +220,7 @@
         key = self.get_key_box()
         if key in already_seen:
             return
-        already_seen.append(key)
+        already_seen[key] = None
         if self.box is None:
             self.vstr.enum_forced_boxes(boxes, already_seen)
             self.vstart.enum_forced_boxes(boxes, already_seen)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Fri Oct 29 13:29:00 2010
@@ -209,7 +209,7 @@
         key = self.get_key_box()
         if key in already_seen:
             return
-        already_seen.append(key)
+        already_seen[key] = None
         if self.box is None:
             for itemvalue in self._items:
                 itemvalue.enum_forced_boxes(boxes, already_seen)


From afa at codespeak.net  Fri Oct 29 13:30:07 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 13:30:07 +0200 (CEST)
Subject: [pypy-svn] r78460 - pypy/trunk/pypy/translator/c/gcc
Message-ID: <20101029113007.00160282BE8@codespeak.net>

Author: afa
Date: Fri Oct 29 13:30:06 2010
New Revision: 78460

Modified:
   pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
Log:
gcmaptable now uses ".quad" in all 64bit platforms


Modified: pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
==============================================================================
--- pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	(original)
+++ pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	Fri Oct 29 13:30:06 2010
@@ -1599,7 +1599,7 @@
             txt = kwargs[self.format]
             print >> output, "\t%s" % txt
 
-        if self.format == 'elf64':
+        if self.format in ('elf64', 'darwin64'):
             word_decl = '.quad'
         else:
             word_decl = '.long'


From arigo at codespeak.net  Fri Oct 29 13:34:19 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 13:34:19 +0200 (CEST)
Subject: [pypy-svn] r78461 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101029113419.4707136E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 29 13:34:17 2010
New Revision: 78461

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_spec.py
Log:
Fix the test.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_spec.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_spec.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_spec.py	Fri Oct 29 13:34:17 2010
@@ -7,7 +7,6 @@
     optimizer = OPTIMIZER_FULL
     automatic_promotion_result = {
         'int_add' : 3, 'int_gt' : 1, 'guard_false' : 1, 'jump' : 1, 
-        'guard_value' : 1
     }
 
     # ====> test_loop.py


From arigo at codespeak.net  Fri Oct 29 13:44:31 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 13:44:31 +0200 (CEST)
Subject: [pypy-svn] r78462 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101029114431.09FC9282BE8@codespeak.net>

Author: arigo
Date: Fri Oct 29 13:44:30 2010
New Revision: 78462

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
Log:
(hakanardo, arigo)  Fix.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Fri Oct 29 13:44:30 2010
@@ -79,7 +79,7 @@
                 #for arg in newop.getarglist():
                 for arg in inputargs:
                     arg = argmap[arg]
-                    self.getvalue(arg).enum_forced_boxes(args, {})
+                    args.append(self.getvalue(arg).force_box())
                 newop.initarglist(args + inputargs[len(args):])
 
             #print 'P: ', str(newop)
@@ -108,8 +108,8 @@
                             jmp = self.optimizer.newoperations[-1]
                             if jmp.getopnum() == rop.JUMP:
                                 newval = self.getvalue(argmap[a])
-                                boxes = jmp.getarglist()[:]
-                                newval.enum_forced_boxes(boxes, {})
+                                boxes = jmp.getarglist()
+                                boxes.append(newval.force_box())
                                 jmp.initarglist(boxes)
 
         return inputargs


From hakanardo at codespeak.net  Fri Oct 29 13:47:19 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Fri, 29 Oct 2010 13:47:19 +0200 (CEST)
Subject: [pypy-svn] r78463 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101029114719.A2B1E36E3BC@codespeak.net>

Author: hakanardo
Date: Fri Oct 29 13:47:18 2010
New Revision: 78463

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py
Log:
(hakanardo, arigo) skip this for now

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtual.py	Fri Oct 29 13:47:18 2010
@@ -164,6 +164,7 @@
                                 getfield_gc=0, setfield_gc=0)
 
     def test_two_loops_with_virtual(self):
+        py.test.skip("We don't know how to virtualize across bridges right now")
         myjitdriver = JitDriver(greens = [], reds = ['n', 'node'])
         def f(n):
             node = self._new()


From cfbolz at codespeak.net  Fri Oct 29 13:48:11 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 29 Oct 2010 13:48:11 +0200 (CEST)
Subject: [pypy-svn] r78464 - in pypy/trunk/pypy/objspace/std: . test
Message-ID: <20101029114811.37C0F282BE8@codespeak.net>

Author: cfbolz
Date: Fri Oct 29 13:48:09 2010
New Revision: 78464

Modified:
   pypy/trunk/pypy/objspace/std/mapdict.py
   pypy/trunk/pypy/objspace/std/test/test_mapdict.py
Log:
fix failing weakref assertion error


Modified: pypy/trunk/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/mapdict.py	(original)
+++ pypy/trunk/pypy/objspace/std/mapdict.py	Fri Oct 29 13:48:09 2010
@@ -361,7 +361,8 @@
 
     def setweakref(self, space, weakreflifeline):
         from pypy.module._weakref.interp__weakref import WeakrefLifeline
-        assert isinstance(weakreflifeline, WeakrefLifeline)
+        assert (isinstance(weakreflifeline, WeakrefLifeline) or
+                    weakreflifeline is None)
         self._get_mapdict_map().write(self, ("weakref", SPECIAL), weakreflifeline)
 
 class ObjectMixin(object):

Modified: pypy/trunk/pypy/objspace/std/test/test_mapdict.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_mapdict.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_mapdict.py	Fri Oct 29 13:48:09 2010
@@ -160,6 +160,13 @@
     assert obj.getweakref() is lifeline1
     assert obj.getdictvalue(space, "weakref") == 41
 
+    lifeline1 = WeakrefLifeline(space)
+    obj = c.instantiate()
+    assert obj.getweakref() is None
+    obj.setweakref(space, lifeline1)
+    obj.setweakref(space, None)
+
+
 
 def test_slots():
     cls = Class()


From arigo at codespeak.net  Fri Oct 29 14:02:34 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 14:02:34 +0200 (CEST)
Subject: [pypy-svn] r78466 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101029120234.7B8A236E0CB@codespeak.net>

Author: arigo
Date: Fri Oct 29 14:02:33 2010
New Revision: 78466

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
Log:
(hakanardo, arigo)
Argh, these tests.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	Fri Oct 29 14:02:33 2010
@@ -149,9 +149,9 @@
             def foo(self):
                 return 3
         def externfn(y):
-            if y % 4 == 0: return W1()
-            elif y % 4 == 3: return W2()
-            else: return W3()
+            lst = [W1, W1, W2, W2, W3, W3, W2, W1, W3]
+            W = lst[y % len(lst)]
+            return W()
         def f(y):
             while y > 0:
                 myjitdriver.can_enter_jit(y=y)
@@ -164,7 +164,8 @@
         for j in range(69, 75):
             res = self.meta_interp(f, [j], policy=policy)
             assert res == 42
-            self.check_loop_count(3)
+            self.check_enter_count(5)
+            self.check_loop_count(5)
 
     def test_oosend_guard_failure(self):
         myjitdriver = JitDriver(greens = [], reds = ['x', 'y', 'w'])


From afa at codespeak.net  Fri Oct 29 14:08:46 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 14:08:46 +0200 (CEST)
Subject: [pypy-svn] r78467 - in pypy/branch/fast-forward/pypy/rlib: . test
Message-ID: <20101029120846.798CA282C1E@codespeak.net>

Author: afa
Date: Fri Oct 29 14:08:44 2010
New Revision: 78467

Modified:
   pypy/branch/fast-forward/pypy/rlib/rzlib.py
   pypy/branch/fast-forward/pypy/rlib/test/test_rzlib.py
Log:
Improve zlib error messages: in the Z_FINISH case, 
call decompress() again with a size of zero.


Modified: pypy/branch/fast-forward/pypy/rlib/rzlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rzlib.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rzlib.py	Fri Oct 29 14:08:44 2010
@@ -22,7 +22,7 @@
 
 
 constantnames = '''
-    Z_OK  Z_STREAM_ERROR  Z_BUF_ERROR  Z_MEM_ERROR  Z_STREAM_END
+    Z_OK  Z_STREAM_ERROR  Z_BUF_ERROR  Z_MEM_ERROR  Z_STREAM_END Z_DATA_ERROR
     Z_DEFLATED  Z_DEFAULT_STRATEGY  Z_DEFAULT_COMPRESSION
     Z_NO_FLUSH  Z_FINISH  Z_SYNC_FLUSH  Z_FULL_FLUSH
     MAX_WBITS  MAX_MEM_LEVEL
@@ -200,10 +200,17 @@
         """
         if stream.c_msg:
             reason = rffi.charp2str(stream.c_msg)
+        elif err == Z_MEM_ERROR:
+            reason = "out of memory"
+        elif err == Z_BUF_ERROR:
+            reason = "incomplete or truncated stream"
+        elif err == Z_STREAM_ERROR:
+            reason = "inconsistent stream state"
+        elif err == Z_DATA_ERROR:
+            reason = "invalid input data"
         else:
             reason = ""
-        if not reason and err == Z_MEM_ERROR:
-            reason = "out of memory"
+
         if reason:
             delim = ": "
         else:
@@ -313,15 +320,17 @@
         should_finish = True
     else:
         should_finish = False
-    result = _operate(stream, data, flush, max_length, _inflate,
-                      "while decompressing")
+    while_doing = "while decompressing data"
+    data, err, avail_in = _operate(stream, data, flush, max_length, _inflate,
+                                   while_doing)
     if should_finish:
-        # detect incomplete input in the Z_FINISHED case
-        finished = result[1]
-        if not finished:
-            raise RZlibError("the input compressed stream of data is "
-                             "incomplete")
-    return result
+        # detect incomplete input
+        rffi.setintfield(stream, 'c_avail_in', 0)
+        err = _inflate(stream, Z_FINISH)
+        if err < 0:
+            raise RZlibError.fromstream(stream, err, while_doing)
+    finished = (err == Z_STREAM_END)
+    return data, finished, avail_in
 
 
 def _operate(stream, data, flush, max_length, cfunc, while_doing):
@@ -393,5 +402,5 @@
     # then the zlib simply returns Z_OK and waits for more.  If it is
     # complete it returns Z_STREAM_END.
     return (''.join(result),
-            err == Z_STREAM_END,
+            err,
             rffi.cast(lltype.Signed, stream.c_avail_in))

Modified: pypy/branch/fast-forward/pypy/rlib/test/test_rzlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/test/test_rzlib.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/test/test_rzlib.py	Fri Oct 29 14:08:44 2010
@@ -165,9 +165,11 @@
     assert finished2 is False
     assert unused2 == 0
     assert expanded.startswith(data)
-    py.test.raises(rzlib.RZlibError,
-                   rzlib.decompress, stream, compressed[2000:-500],
-                   rzlib.Z_FINISH)
+    exc = py.test.raises(
+        rzlib.RZlibError,
+        rzlib.decompress, stream, compressed[2000:-500], rzlib.Z_FINISH)
+    msg = "Error -5 while decompressing data: incomplete or truncated stream"
+    assert str(exc.value) == msg
     rzlib.inflateEnd(stream)
 
 


From cfbolz at codespeak.net  Fri Oct 29 15:01:44 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Fri, 29 Oct 2010 15:01:44 +0200 (CEST)
Subject: [pypy-svn] r78468 - pypy/trunk/lib-python/modified-2.5.2/test
Message-ID: <20101029130144.D184036E0C1@codespeak.net>

Author: cfbolz
Date: Fri Oct 29 15:01:43 2010
New Revision: 78468

Modified:
   pypy/trunk/lib-python/modified-2.5.2/test/test_descr.py
Log:
(arigo, cfbolz): we don't care that in pypy you can take weakrefs to objects
with slots without __weakref__


Modified: pypy/trunk/lib-python/modified-2.5.2/test/test_descr.py
==============================================================================
--- pypy/trunk/lib-python/modified-2.5.2/test/test_descr.py	(original)
+++ pypy/trunk/lib-python/modified-2.5.2/test/test_descr.py	Fri Oct 29 15:01:43 2010
@@ -2028,7 +2028,9 @@
     except TypeError, msg:
         verify(str(msg).find("weak reference") >= 0)
     else:
-        verify(0, "weakref.ref(no) should be illegal")
+        # in PyPy it is (sometimes) possible to take a weakref here
+        #verify(0, "weakref.ref(no) should be illegal")
+        pass
     class Weak(object):
         __slots__ = ['foo', '__weakref__']
     yes = Weak()


From arigo at codespeak.net  Fri Oct 29 15:11:14 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 15:11:14 +0200 (CEST)
Subject: [pypy-svn] r78469 - in pypy/trunk/pypy: objspace/std/test rlib
	rlib/test
Message-ID: <20101029131114.CB553282BE8@codespeak.net>

Author: arigo
Date: Fri Oct 29 15:11:13 2010
New Revision: 78469

Modified:
   pypy/trunk/pypy/objspace/std/test/test_longobject.py
   pypy/trunk/pypy/rlib/rbigint.py
   pypy/trunk/pypy/rlib/test/test_rbigint.py
Log:
-0L == 0L.


Modified: pypy/trunk/pypy/objspace/std/test/test_longobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_longobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_longobject.py	Fri Oct 29 15:11:13 2010
@@ -254,3 +254,7 @@
         class myotherlong(long):
             pass
         assert long(myotherlong(21)) == 21L
+
+    def test_negative_zero(self):
+        x = eval("-0L")
+        assert x == 0L

Modified: pypy/trunk/pypy/rlib/rbigint.py
==============================================================================
--- pypy/trunk/pypy/rlib/rbigint.py	(original)
+++ pypy/trunk/pypy/rlib/rbigint.py	Fri Oct 29 15:11:13 2010
@@ -1631,7 +1631,7 @@
             a = _muladd1(a, tens, dig)
             tens = 1
             dig = 0
-    if sign:
+    if sign and a.sign == 1:
         a.sign = -1
     return a
 
@@ -1652,6 +1652,5 @@
         else:
             dig = dig * base + digit
             tens *= base
-    if parser.sign == -1:
-        a.sign = -1
+    a.sign *= parser.sign
     return a

Modified: pypy/trunk/pypy/rlib/test/test_rbigint.py
==============================================================================
--- pypy/trunk/pypy/rlib/test/test_rbigint.py	(original)
+++ pypy/trunk/pypy/rlib/test/test_rbigint.py	Fri Oct 29 15:11:13 2010
@@ -112,6 +112,23 @@
         assert rbigint.fromrarith_int(r_uint(2*sys.maxint+1)).eq(
             rbigint.fromlong(2*sys.maxint+1))
 
+    def test_fromdecimalstr(self):
+        x = rbigint.fromdecimalstr("12345678901234567890523897987")
+        assert x.tolong() == 12345678901234567890523897987L
+        assert x.tobool() is True
+        x = rbigint.fromdecimalstr("+12345678901234567890523897987")
+        assert x.tolong() == 12345678901234567890523897987L
+        assert x.tobool() is True
+        x = rbigint.fromdecimalstr("-12345678901234567890523897987")
+        assert x.tolong() == -12345678901234567890523897987L
+        assert x.tobool() is True
+        x = rbigint.fromdecimalstr("+0")
+        assert x.tolong() == 0
+        assert x.tobool() is False
+        x = rbigint.fromdecimalstr("-0")
+        assert x.tolong() == 0
+        assert x.tobool() is False
+
     def test_add(self):
         x = 123456789123456789000000L
         y = 123858582373821923936744221L
@@ -470,6 +487,11 @@
             num = num * 36 + i
         x = parse_digit_string(Parser(16, -1, range(15,-1,-1)*99))
         assert x.eq(rbigint.fromlong(long('-0x' + 'FEDCBA9876543210'*99, 16)))
+        assert x.tobool() is True
+        x = parse_digit_string(Parser(7, 1, [0, 0, 0]))
+        assert x.tobool() is False
+        x = parse_digit_string(Parser(7, -1, [0, 0, 0]))
+        assert x.tobool() is False
 
 
 BASE = 2 ** SHIFT


From antocuni at codespeak.net  Fri Oct 29 15:13:03 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Fri, 29 Oct 2010 15:13:03 +0200 (CEST)
Subject: [pypy-svn] r78470 - pypy/build/bot2-hg
Message-ID: <20101029131303.3E3C936E0CB@codespeak.net>

Author: antocuni
Date: Fri Oct 29 15:13:01 2010
New Revision: 78470

Added:
   pypy/build/bot2-hg/
      - copied from r78469, pypy/build/bot2/
Log:
a branch to make buildbot pulling pypy from a mercurial repo



From antocuni at codespeak.net  Fri Oct 29 15:17:07 2010
From: antocuni at codespeak.net (antocuni at codespeak.net)
Date: Fri, 29 Oct 2010 15:17:07 +0200 (CEST)
Subject: [pypy-svn] r78471 - pypy/build/bot2-hg
Message-ID: <20101029131707.E1B7A282C24@codespeak.net>

Author: antocuni
Date: Fri Oct 29 15:17:06 2010
New Revision: 78471

Removed:
   pypy/build/bot2-hg/
Log:
now that I think better of it, it makes more sense to develop this branch
directly on mercurial, doesn't it? :-)




From afa at codespeak.net  Fri Oct 29 15:18:00 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 15:18:00 +0200 (CEST)
Subject: [pypy-svn] r78472 - in pypy/branch/fast-forward/pypy/module/imp: .
	test
Message-ID: <20101029131800.2CB1336E3F0@codespeak.net>

Author: afa
Date: Fri Oct 29 15:17:58 2010
New Revision: 78472

Modified:
   pypy/branch/fast-forward/pypy/module/imp/__init__.py
   pypy/branch/fast-forward/pypy/module/imp/importing.py
   pypy/branch/fast-forward/pypy/module/imp/test/test_app.py
   pypy/branch/fast-forward/pypy/module/imp/test/test_import.py
Log:
Add imp.NullImporter. This also fixes "./test_all.py -A module/imp" 
when run on top of CPython 2.7


Modified: pypy/branch/fast-forward/pypy/module/imp/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/imp/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/imp/__init__.py	Fri Oct 29 15:17:58 2010
@@ -26,6 +26,7 @@
         'is_builtin':      'interp_imp.is_builtin',
         'is_frozen':       'interp_imp.is_frozen',
         'reload':          'importing.reload',
+        'NullImporter':    'importing.W_NullImporter',
 
         'lock_held':       'interp_imp.lock_held',
         'acquire_lock':    'interp_imp.acquire_lock',

Modified: pypy/branch/fast-forward/pypy/module/imp/importing.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/imp/importing.py	(original)
+++ pypy/branch/fast-forward/pypy/module/imp/importing.py	Fri Oct 29 15:17:58 2010
@@ -5,9 +5,10 @@
 import sys, os, stat
 
 from pypy.interpreter.module import Module
-from pypy.interpreter import gateway
+from pypy.interpreter.gateway import Arguments, interp2app, unwrap_spec
+from pypy.interpreter.typedef import TypeDef, generic_new_descr
 from pypy.interpreter.error import OperationError, operationerrfmt
-from pypy.interpreter.baseobjspace import W_Root, ObjSpace
+from pypy.interpreter.baseobjspace import W_Root, ObjSpace, Wrappable
 from pypy.interpreter.eval import Code
 from pypy.rlib import streamio, jit
 from pypy.rlib.streamio import StreamErrors
@@ -256,8 +257,7 @@
     w_path_importer_cache = space.sys.get("path_importer_cache")
     w_importer = space.finditem(w_path_importer_cache, w_pathitem)
     if w_importer is None:
-        w_importer = space.w_None
-        space.setitem(w_path_importer_cache, w_pathitem, w_importer)
+        space.setitem(w_path_importer_cache, w_pathitem, space.w_None)
         for w_hook in space.unpackiterable(space.sys.get("path_hooks")):
             try:
                 w_importer = space.call_function(w_hook, w_pathitem)
@@ -266,6 +266,8 @@
                     raise
             else:
                 break
+        if w_importer is None:
+            w_importer = space.wrap(W_NullImporter(space))
         if space.is_true(w_importer):
             space.setitem(w_path_importer_cache, w_pathitem, w_importer)
     if space.is_true(w_importer):
@@ -273,6 +275,37 @@
         if space.is_true(w_loader):
             return w_loader
 
+
+class W_NullImporter(Wrappable):
+    def __init__(self, space):
+        pass
+
+    @unwrap_spec('self', ObjSpace, str)
+    def descr_init(self, space, path):
+        if not path:
+            raise OperationError(space.w_ImportError, space.wrap(
+                "empty pathname"))
+
+        # Directory should not exist
+        try:
+            st = os.stat(path)
+        except OSError:
+            return # ok
+        if stat.S_ISDIR(st.st_mode):
+            raise OperationError(space.w_ImportError, space.wrap(
+                "existing directory"))
+
+    @unwrap_spec('self', ObjSpace, Arguments)
+    def find_module_w(self, space, __args__):
+        return space.wrap(None)
+
+W_NullImporter.typedef = TypeDef(
+    'imp.NullImporter',
+    __new__=generic_new_descr(W_NullImporter),
+    __init__=interp2app(W_NullImporter.descr_init),
+    find_module=interp2app(W_NullImporter.find_module_w),
+    )
+
 class FindInfo:
     def __init__(self, modtype, filename, stream,
                  suffix="", filemode="", w_loader=None):

Modified: pypy/branch/fast-forward/pypy/module/imp/test/test_app.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/imp/test/test_app.py	(original)
+++ pypy/branch/fast-forward/pypy/module/imp/test/test_app.py	Fri Oct 29 15:17:58 2010
@@ -122,3 +122,9 @@
         mod2 = self.imp.load_module('test_imp_extra_AUTO6', f, fn, descr)
         f.close()
         assert mod2 is mod
+
+    def test_nullimporter(self):
+        import os
+        importer = self.imp.NullImporter("path")
+        assert importer.find_module(1, 2, 3, 4) is None
+        raises(ImportError, self.imp.NullImporter, os.getcwd())

Modified: pypy/branch/fast-forward/pypy/module/imp/test/test_import.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/imp/test/test_import.py	(original)
+++ pypy/branch/fast-forward/pypy/module/imp/test/test_import.py	Fri Oct 29 15:17:58 2010
@@ -869,7 +869,7 @@
             if path == "xxx":
                 return Importer()
             raise ImportError()
-        import sys
+        import sys, imp
         try:
             sys.path_hooks.append(importer_for_path)
             sys.path.insert(0, "yyy")
@@ -879,7 +879,8 @@
                 import b
             except ImportError:
                 pass
-            assert sys.path_importer_cache['yyy'] is None
+            assert isinstance(sys.path_importer_cache['yyy'],
+                              imp.NullImporter)
         finally:
             sys.path.pop(0)
             sys.path.pop(0)


From afa at codespeak.net  Fri Oct 29 15:21:25 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 15:21:25 +0200 (CEST)
Subject: [pypy-svn] r78473 - pypy/branch/fast-forward/pypy/module/imp
Message-ID: <20101029132125.DF9AE282C24@codespeak.net>

Author: afa
Date: Fri Oct 29 15:21:24 2010
New Revision: 78473

Modified:
   pypy/branch/fast-forward/pypy/module/imp/importing.py
Log:
More robust syntax (I don't like "return" hidden in the code)


Modified: pypy/branch/fast-forward/pypy/module/imp/importing.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/imp/importing.py	(original)
+++ pypy/branch/fast-forward/pypy/module/imp/importing.py	Fri Oct 29 15:21:24 2010
@@ -290,10 +290,11 @@
         try:
             st = os.stat(path)
         except OSError:
-            return # ok
-        if stat.S_ISDIR(st.st_mode):
-            raise OperationError(space.w_ImportError, space.wrap(
-                "existing directory"))
+            pass
+        else:
+            if stat.S_ISDIR(st.st_mode):
+                raise OperationError(space.w_ImportError, space.wrap(
+                    "existing directory"))
 
     @unwrap_spec('self', ObjSpace, Arguments)
     def find_module_w(self, space, __args__):


From arigo at codespeak.net  Fri Oct 29 16:23:29 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 16:23:29 +0200 (CEST)
Subject: [pypy-svn] r78474 - pypy/trunk/pypy/jit/metainterp/optimizeopt
Message-ID: <20101029142329.0D2EB36E0C1@codespeak.net>

Author: arigo
Date: Fri Oct 29 16:23:27 2010
New Revision: 78474

Modified:
   pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py
Log:
Simplify the logic and stick to the invariant
"getint() must *never* be called on a BoxInt, only on a ConstInt"


Modified: pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/optimizeopt/rewrite.py	Fri Oct 29 16:23:27 2010
@@ -244,14 +244,11 @@
         self.optimizer.exception_might_have_happened = False
 
     def optimize_CALL_LOOPINVARIANT(self, op):
-        funcvalue = self.getvalue(op.getarg(0))
-        if not funcvalue.is_constant():
-            # XXX this code path is never executed in tests nor in production.
-            # in fact, it can't even happen since residual_call in codewriter
-            # expects a compile-time constant
-            self.emit_operation(op)
-            return
-        key = make_hashable_int(op.getarg(0).getint())
+        arg = op.getarg(0)
+        # 'arg' must be a Const, because residual_call in codewriter
+        # expects a compile-time constant
+        assert isinstance(arg, Const)
+        key = make_hashable_int(arg.getint())
         resvalue = self.optimizer.loop_invariant_results.get(key, None)
         if resvalue is not None:
             self.make_equal_to(op.result, resvalue)


From hakanardo at codespeak.net  Fri Oct 29 16:43:28 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Fri, 29 Oct 2010 16:43:28 +0200 (CEST)
Subject: [pypy-svn] r78475 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: . optimizeopt test
Message-ID: <20101029144328.2EB3F282C25@codespeak.net>

Author: hakanardo
Date: Fri Oct 29 16:43:27 2010
New Revision: 78475

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/oparser.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(armin, hakanardo) separate into 3 loops

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Fri Oct 29 16:43:27 2010
@@ -9,8 +9,11 @@
     distinction anymore)"""
     
     def setup(self, virtuals):
-        self.enabled = virtuals
-
+        self.enabled = virtuals        
+        self.cloned_operations = []
+        for op in self.optimizer.loop.operations:
+            self.cloned_operations.append(op.clone())
+            
     def propagate_forward(self, op):
         if not self.enabled:
             self.emit_operation(op)
@@ -22,8 +25,7 @@
             self.optimizer.newoperations = []
             jump_args = op.getarglist()
             op.initarglist([])
-            inputargs = self.inline(loop.operations,
-                                    #loop.preamble.operations + [op],
+            inputargs = self.inline(self.cloned_operations,
                                     loop.inputargs, jump_args)
             loop.inputargs = inputargs
             jmp = ResOperation(rop.JUMP, loop.inputargs[:], None)
@@ -51,27 +53,24 @@
         self.snapshot_map ={None: None}
         
         inputargs = []
-        seen = {}
+        seen_inputargs = {}
         for arg in jump_args:
             boxes = []
-            self.getvalue(arg).enum_forced_boxes(boxes, seen)
+            self.getvalue(arg).enum_forced_boxes(boxes, seen_inputargs)
             for a in boxes:
                 if not isinstance(a, Const):
                     inputargs.append(a)
 
-        for op in loop_operations:
+        for newop in loop_operations:
             #import pdb; pdb.set_trace()
-            newop = op.clone()
             newop.initarglist([self.inline_arg(a) for a in newop.getarglist()])
-            if op.result:
-                newop.result = op.result.clonebox()
-                argmap[op.result] = newop.result
+            if newop.result:
+                old_result = newop.result
+                newop.result = newop.result.clonebox()
+                argmap[old_result] = newop.result
+
             descr = newop.getdescr()
             if isinstance(descr, ResumeGuardDescr):
-                orgdescr = op.getdescr()
-                assert isinstance(orgdescr, ResumeGuardDescr)
-                orgdescr.rd_snapshot = None #FIXME: In the right place?
-                descr.rd_numb = None
                 descr.rd_snapshot = self.inline_snapshot(descr.rd_snapshot)
                 
             if newop.getopnum() == rop.JUMP:
@@ -83,34 +82,55 @@
                 newop.initarglist(args + inputargs[len(args):])
 
             #print 'P: ', str(newop)
-            current = len(self.optimizer.newoperations)
+            #current = len(self.optimizer.newoperations)
             self.emit_operation(newop)
 
+
             # FIXME: force_lazy_setfield in heap.py may reorder last ops
-            current = max(current-1, 0)
-                
-            for op in self.optimizer.newoperations[current:]:
-                #print 'E: ', str(op)
-                if op.is_guard():
-                    descr = op.getdescr()
-                    assert isinstance(descr, ResumeGuardDescr)
-                    descr.rd_snapshot = None #FIXME: In the right place?
-                args = op.getarglist()
-                if op.is_guard():
-                    args = args + op.getfailargs()
-                #if op.getopnum() == rop.SETFIELD_GC:
-                #    import pdb; pdb.set_trace()
-                for a in args:
-                    if not isinstance(a, Const) and a in self.optimizer.values:
-                        v = self.getvalue(a)
-                        if v.fromstart and a not in inputargs:
+            #current = max(current-1, 0)
+        
+        jmp = self.optimizer.newoperations[-1]
+        assert jmp.getopnum() == rop.JUMP
+        jumpargs = jmp.getarglist()
+        for op in self.optimizer.newoperations:
+            print 'E: ', str(op)
+            #if op.is_guard():
+            #    descr = op.getdescr()
+            #    assert isinstance(descr, ResumeGuardDescr)
+            #    descr.rd_snapshot = None #FIXME: In the right place?
+            args = op.getarglist()
+            if op.is_guard():
+                #new_failargs = []
+                #for a in op.getfailargs():
+                #    new_failargs.append(self.getvalue(self.inline_arg(a)).force_box())
+                #op.setfailargs(new_failargs)
+                    
+                args = args + op.getfailargs()
+            print "Args: ", args
+            
+            for a in args:
+                if not isinstance(a, Const) and a in self.optimizer.values:
+                    v = self.getvalue(a)
+                    if v.fromstart and not v.is_constant():
+                        a = v.force_box()
+                        if a not in inputargs:
                             inputargs.append(a)
-                            jmp = self.optimizer.newoperations[-1]
-                            if jmp.getopnum() == rop.JUMP:
-                                newval = self.getvalue(argmap[a])
-                                boxes = jmp.getarglist()
-                                boxes.append(newval.force_box())
-                                jmp.initarglist(boxes)
+                            newval = self.getvalue(argmap[a])
+                            jumpargs.append(newval.force_box())
+
+
+                        #boxes = []
+                        #v.enum_forced_boxes(boxes, seen_inputargs)
+                        #for b in boxes:
+                        #    if not isinstance(b, Const):
+                        #        inputargs.append(b)
+                        #        jumpargs.append(b)
+                                
+                        #newval = self.getvalue(argmap[a])
+                        #jumpargs.append(newval.force_box())
+        jmp.initarglist(jumpargs)
+
+        print "Inputargs: ", inputargs
 
         return inputargs
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py	Fri Oct 29 16:43:27 2010
@@ -298,7 +298,7 @@
         self.liveboxes_from_env = liveboxes_from_env
         self.liveboxes = {}
         storage.rd_numb = numb
-        #storage.rd_snapshot = None FIXME!!
+        storage.rd_snapshot = None
 
         # collect liveboxes and virtuals
         n = len(liveboxes_from_env) - v

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/oparser.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/oparser.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/oparser.py	Fri Oct 29 16:43:27 2010
@@ -31,6 +31,9 @@
     def getopnum(self):
         return self.OPNUM
 
+    def clone(self):
+        return ESCAPE_OP(self.OPNUM, self.getarglist()[:], self.result, self.getdescr())
+
 class ExtendedTreeLoop(TreeLoop):
 
     def getboxes(self):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Fri Oct 29 16:43:27 2010
@@ -3092,8 +3092,8 @@
         jump(i2)
         '''
         expected = '''
-        [i0]
-        jump(1)
+        []
+        jump()
         '''
         self.optimize_loop(ops, 'Not', expected)
 
@@ -3109,28 +3109,30 @@
         jump(i2)
         '''
         expected = '''
-        [i0]
-        jump(3)
+        []
+        jump()
         '''
         self.optimize_loop(ops, 'Not', expected)
 
     def test_arraycopy_not_virtual(self):
         ops = '''
-        [p0]
+        []
         p1 = new_array(3, descr=arraydescr)
         p2 = new_array(3, descr=arraydescr)
         setarrayitem_gc(p1, 2, 10, descr=arraydescr)
         setarrayitem_gc(p2, 2, 13, descr=arraydescr)
         call(0, p1, p2, 0, 0, 3, descr=arraycopydescr)
-        jump(p2)
+        escape(p2)
+        jump()
         '''
         expected = '''
-        [p0]
+        []
         p2 = new_array(3, descr=arraydescr)
         setarrayitem_gc(p2, 2, 10, descr=arraydescr)
-        jump(p2)
+        escape(p2)
+        jump()
         '''
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, '', expected)
 
     def test_arraycopy_no_elem(self):
         """ this was actually observed in the wild


From hakanardo at codespeak.net  Fri Oct 29 16:49:36 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Fri, 29 Oct 2010 16:49:36 +0200 (CEST)
Subject: [pypy-svn] r78476 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101029144936.4B94F36E0C1@codespeak.net>

Author: hakanardo
Date: Fri Oct 29 16:49:34 2010
New Revision: 78476

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
Log:
(armin, hakanardo) cleanups

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Fri Oct 29 16:49:34 2010
@@ -48,10 +48,8 @@
         for op in self.optimizer.pure_operations.values():
             v = self.getvalue(op.result)
             v.fromstart = True
-            
 
-        self.snapshot_map ={None: None}
-        
+        self.snapshot_map = {None: None}
         inputargs = []
         seen_inputargs = {}
         for arg in jump_args:
@@ -62,7 +60,6 @@
                     inputargs.append(a)
 
         for newop in loop_operations:
-            #import pdb; pdb.set_trace()
             newop.initarglist([self.inline_arg(a) for a in newop.getarglist()])
             if newop.result:
                 old_result = newop.result
@@ -75,38 +72,22 @@
                 
             if newop.getopnum() == rop.JUMP:
                 args = []
-                #for arg in newop.getarglist():
                 for arg in inputargs:
                     arg = argmap[arg]
                     args.append(self.getvalue(arg).force_box())
                 newop.initarglist(args + inputargs[len(args):])
 
             #print 'P: ', str(newop)
-            #current = len(self.optimizer.newoperations)
             self.emit_operation(newop)
 
-
-            # FIXME: force_lazy_setfield in heap.py may reorder last ops
-            #current = max(current-1, 0)
-        
         jmp = self.optimizer.newoperations[-1]
         assert jmp.getopnum() == rop.JUMP
         jumpargs = jmp.getarglist()
         for op in self.optimizer.newoperations:
-            print 'E: ', str(op)
-            #if op.is_guard():
-            #    descr = op.getdescr()
-            #    assert isinstance(descr, ResumeGuardDescr)
-            #    descr.rd_snapshot = None #FIXME: In the right place?
+            #print 'E: ', str(op)
             args = op.getarglist()
             if op.is_guard():
-                #new_failargs = []
-                #for a in op.getfailargs():
-                #    new_failargs.append(self.getvalue(self.inline_arg(a)).force_box())
-                #op.setfailargs(new_failargs)
-                    
                 args = args + op.getfailargs()
-            print "Args: ", args
             
             for a in args:
                 if not isinstance(a, Const) and a in self.optimizer.values:
@@ -118,20 +99,7 @@
                             newval = self.getvalue(argmap[a])
                             jumpargs.append(newval.force_box())
 
-
-                        #boxes = []
-                        #v.enum_forced_boxes(boxes, seen_inputargs)
-                        #for b in boxes:
-                        #    if not isinstance(b, Const):
-                        #        inputargs.append(b)
-                        #        jumpargs.append(b)
-                                
-                        #newval = self.getvalue(argmap[a])
-                        #jumpargs.append(newval.force_box())
         jmp.initarglist(jumpargs)
-
-        print "Inputargs: ", inputargs
-
         return inputargs
 
     def inline_arg(self, arg):


From arigo at codespeak.net  Fri Oct 29 17:02:10 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 17:02:10 +0200 (CEST)
Subject: [pypy-svn] r78478 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101029150210.83BB5282C1E@codespeak.net>

Author: arigo
Date: Fri Oct 29 17:02:08 2010
New Revision: 78478

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
clone_if_mutable() must really clone now.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Fri Oct 29 17:02:08 2010
@@ -215,7 +215,7 @@
     def __eq__(self, other):
         return type(self) is type(other)      # xxx obscure
     def clone_if_mutable(self):
-        return self
+        return Storage(self.metainterp_sd, self.original_greenkey)
 
 def _sortboxes(boxes):
     _kind2count = {history.INT: 1, history.REF: 2, history.FLOAT: 3}


From arigo at codespeak.net  Fri Oct 29 17:19:45 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Fri, 29 Oct 2010 17:19:45 +0200 (CEST)
Subject: [pypy-svn] r78479 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101029151945.2854236E0CB@codespeak.net>

Author: arigo
Date: Fri Oct 29 17:19:43 2010
New Revision: 78479

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
Whack whack more.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Fri Oct 29 17:19:43 2010
@@ -215,7 +215,9 @@
     def __eq__(self, other):
         return type(self) is type(other)      # xxx obscure
     def clone_if_mutable(self):
-        return Storage(self.metainterp_sd, self.original_greenkey)
+        res = Storage(self.metainterp_sd, self.original_greenkey)
+        self.copy_all_attrbutes_into(res)
+        return res
 
 def _sortboxes(boxes):
     _kind2count = {history.INT: 1, history.REF: 2, history.FLOAT: 3}


From hakanardo at codespeak.net  Fri Oct 29 17:28:58 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Fri, 29 Oct 2010 17:28:58 +0200 (CEST)
Subject: [pypy-svn] r78480 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101029152858.797FA282BE8@codespeak.net>

Author: hakanardo
Date: Fri Oct 29 17:28:51 2010
New Revision: 78480

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(armin, hakanardo) fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Fri Oct 29 17:28:51 2010
@@ -242,7 +242,8 @@
         assert equaloplists(optimized.operations,
                             expected.operations, False, remap)
 
-    def optimize_loop(self, ops, spectext, optops, checkspecnodes=True):
+    def optimize_loop(self, ops, spectext, optops, checkspecnodes=True,
+                      expected_preamble=None):
         loop = self.parse(ops)
         #
         if checkspecnodes:
@@ -259,6 +260,7 @@
         #
         self.loop = loop
         loop.preamble = TreeLoop('preamble')
+        loop.preamble.inputargs = loop.inputargs
         metainterp_sd = FakeMetaInterpStaticData(self.cpu)
         if hasattr(self, 'vrefinfo'):
             metainterp_sd.virtualref_info = self.vrefinfo
@@ -270,20 +272,20 @@
         print '\n'.join([str(o) for o in loop.operations])
         self.assert_equal(loop, expected)
 
+        if expected_preamble:
+            expected = self.parse(expected_preamble)
+            self.assert_equal(loop.preamble, expected)
+
     def test_simple(self):
         ops = """
-        [i]
-        i0 = int_sub(i, 1)
-        guard_value(i0, 0) [i0]
-        jump(i)
-        """
-        expected = """
-        [i]
-        i0 = int_sub(i, 1)
-        guard_value(i0, 0) [i0]
-        jump(1)
+        []
+        f = escape()
+        f0 = float_sub(f, 1.0)
+        guard_value(f0, 0.0) [f0]
+        escape(f)
+        jump()
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, '', ops)
 
     def test_constant_propagate(self):
         ops = """
@@ -363,12 +365,16 @@
         guard_class(p0, ConstClass(node_vtable)) []
         jump(p0)
         """
-        expected = """
+        preamble = """
         [p0]
         guard_class(p0, ConstClass(node_vtable)) []
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [p0]
+        jump(p0)
+        """
+        self.optimize_loop(ops, 'Not', expected, expected_preamble=preamble)
 
     def test_remove_guard_class_2(self):
         ops = """
@@ -3718,41 +3724,48 @@
 
     def test_bound_eq(self):
         ops = """
-        [i0, i1]
+        []
+        i0 = escape()
+        i1 = escape()
         i2 = int_le(i0, 4)
         guard_true(i2) []
         i3 = int_eq(i0, i1)
         guard_true(i3) []
         i4 = int_lt(i1, 5)
         guard_true(i4) []
-        jump(i0, i1)
+        jump()
         """
         expected = """
-        [i0, i1]
+        []
+        i0 = escape()
+        i1 = escape()
         i2 = int_le(i0, 4)
         guard_true(i2) []
         i3 = int_eq(i0, i1)
         guard_true(i3) []
-        jump(i0, i1)
+        jump()
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, '', expected)
 
     def test_bound_eq_const(self):
         ops = """
-        [i0]
+        []
+        i0 = escape()
         i1 = int_eq(i0, 7)
         guard_true(i1) []
         i2 = int_add(i0, 3)
-        jump(i2)
+        escape(i2)
+        jump()
         """
         expected = """
-        [i0]
+        []
+        i0 = escape()
         i1 = int_eq(i0, 7)
         guard_true(i1) []
-        jump(10)
-
+        escape(10)
+        jump()
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, '', expected)
 
     def test_bound_eq_const_not(self):
         ops = """


From fijal at codespeak.net  Fri Oct 29 18:33:06 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Fri, 29 Oct 2010 18:33:06 +0200 (CEST)
Subject: [pypy-svn] r78481 - pypy/trunk/pypy/jit/tool
Message-ID: <20101029163306.98AA536E3BC@codespeak.net>

Author: fijal
Date: Fri Oct 29 18:33:04 2010
New Revision: 78481

Modified:
   pypy/trunk/pypy/jit/tool/showstats.py
Log:
fix


Modified: pypy/trunk/pypy/jit/tool/showstats.py
==============================================================================
--- pypy/trunk/pypy/jit/tool/showstats.py	(original)
+++ pypy/trunk/pypy/jit/tool/showstats.py	Fri Oct 29 18:33:04 2010
@@ -10,9 +10,9 @@
 
 def main(argv):
     log = logparser.parse_log_file(argv[0])
-    parts = logparser.extract_category(log, "jit-log-opt-")
+    parts = logparser.extract_category(log, "jit-log-noopt-")
     for i, oplist in enumerate(parts):
-        loop = parse(oplist, no_namespace=True)
+        loop = parse(oplist, no_namespace=True, nonstrict=True)
         num_ops = 0
         num_dmp = 0
         num_guards = 0
@@ -23,7 +23,10 @@
                 num_ops += 1
             if op.is_guard():
                 num_guards += 1
-        print "Loop #%d, length: %d, opcodes: %d, guards: %d, %f" % (i, num_ops, num_dmp, num_guards, num_ops/num_dmp)
+        if num_dmp == 0:
+            print "Loop #%d, length: %d, opcodes: %d, guards: %d" % (i, num_ops, num_dmp, num_guards)
+        else:
+            print "Loop #%d, length: %d, opcodes: %d, guards: %d, %f" % (i, num_ops, num_dmp, num_guards, num_ops/num_dmp)
 
 if __name__ == '__main__':
     main(sys.argv[1:])


From afa at codespeak.net  Fri Oct 29 18:36:48 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 18:36:48 +0200 (CEST)
Subject: [pypy-svn] r78482 - in
	pypy/branch/fast-forward/pypy/module/zipimport: . test
Message-ID: <20101029163648.1B92A282BE8@codespeak.net>

Author: afa
Date: Fri Oct 29 18:36:46 2010
New Revision: 78482

Modified:
   pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py
   pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py
Log:
Fix in zipimport: get_code() is supposed to return a Code object,
not the content of the .pyc file!


Modified: pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py	(original)
+++ pypy/branch/fast-forward/pypy/module/zipimport/interp_zipimport.py	Fri Oct 29 18:36:46 2010
@@ -183,12 +183,20 @@
             return False
         return True
 
+    def can_use_pyc(self, space, filename, magic, timestamp):
+        if magic != importing.get_pyc_magic(space):
+            return False
+        if self.check_newer_pyfile(space, filename[:-1], timestamp):
+            return False
+        if not self.check_compatible_mtime(space, filename, timestamp):
+            return False
+        return True
+
     def import_pyc_file(self, space, modname, filename, buf, pkgpath):
         w = space.wrap
         magic = importing._get_long(buf[:4])
         timestamp = importing._get_long(buf[4:8])
-        if (self.check_newer_pyfile(space, filename[:-1], timestamp) or
-            not self.check_compatible_mtime(space, filename, timestamp)):
+        if not self.can_use_pyc(space, filename, magic, timestamp):
             return self.import_py_file(space, modname, filename[:-1], buf,
                                        pkgpath)
         buf = buf[8:] # XXX ugly copy, should use sequential read instead
@@ -276,16 +284,22 @@
 
     def get_code(self, space, fullname):
         filename = self.make_filename(fullname)
-        w = space.wrap
         for compiled, _, ext in ENUMERATE_EXTS:
             if self.have_modulefile(space, filename + ext):
+                w_source = self.get_data(space, filename + ext)
+                source = space.str_w(w_source)
                 if compiled:
-                    return self.get_data(space, filename + ext)
+                    magic = importing._get_long(source[:4])
+                    timestamp = importing._get_long(source[4:8])
+                    if not self.can_use_pyc(space, filename + ext,
+                                            magic, timestamp):
+                        continue
+                    code_w = importing.read_compiled_module(
+                        space, filename + ext, source[8:])
                 else:
-                    w_source = self.get_data(space, filename + ext)
-                    w_code = space.builtin.call('compile', w_source,
-                                                w(filename + ext), w('exec'))
-                    return w_code
+                    code_w = importing.parse_source_module(
+                        space, filename + ext, source)
+                return space.wrap(code_w)
         raise operationerrfmt(self.w_ZipImportError,
             "Cannot find source or code for %s in %s", filename, self.name)
     get_code.unwrap_spec = ['self', ObjSpace, str]
@@ -386,7 +400,7 @@
     get_data    = interp2app(W_ZipImporter.get_data),
     get_code    = interp2app(W_ZipImporter.get_code),
     get_source  = interp2app(W_ZipImporter.get_source),
-    _get_filename = interp2app(W_ZipImporter.get_filename),
+    get_filename = interp2app(W_ZipImporter.get_filename),
     is_package  = interp2app(W_ZipImporter.is_package),
     load_module = interp2app(W_ZipImporter.load_module),
     archive     = GetSetProperty(W_ZipImporter.getarchive),

Modified: pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py	(original)
+++ pypy/branch/fast-forward/pypy/module/zipimport/test/test_zipimport.py	Fri Oct 29 18:36:46 2010
@@ -175,7 +175,12 @@
         assert mod.__file__.endswith('.zip'+os.sep+'uuu.pyc')
         assert mod.get_file() == mod.__file__
         assert mod.get_name() == mod.__name__
-                                
+        #
+        import zipimport
+        z = zipimport.zipimporter(self.zipfile)
+        code = z.get_code('uuu')
+        assert isinstance(code, type((lambda:0).func_code))
+
     def test_bad_pyc(self):
         import zipimport
         import sys
@@ -206,6 +211,7 @@
         z = zipimport.zipimporter(self.zipfile)
         sys.modules['uuu'] = lambda x : x + 1
         raises(ImportError, z.load_module, 'uuu')
+        raises(zipimport.ZipImportError, z.get_code, 'uuu')
 
     def test_package(self):
         import os, sys
@@ -270,7 +276,7 @@
         assert z.get_source('xx') == "5"
         assert z.archive == self.zipfile
         mod = z.load_module('xx')
-        assert z._get_filename('xx') == mod.__file__
+        assert z.get_filename('xx') == mod.__file__
 
     def test_archive(self):
         """
@@ -300,7 +306,7 @@
         z = zipimport.zipimporter(self.zipfile + "/directory")
         mod = z.load_module("package")
         assert z.is_package("package")
-        assert z._get_filename("package") == mod.__file__
+        assert z.get_filename("package") == mod.__file__
 
     def test_zip_directory_cache(self):
         """ Check full dictionary interface


From fijal at codespeak.net  Fri Oct 29 20:50:52 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Fri, 29 Oct 2010 20:50:52 +0200 (CEST)
Subject: [pypy-svn] r78484 - in pypy/trunk/pypy/jit: backend backend/llgraph
	backend/x86 metainterp
Message-ID: <20101029185052.29F3636E0CB@codespeak.net>

Author: fijal
Date: Fri Oct 29 20:50:49 2010
New Revision: 78484

Modified:
   pypy/trunk/pypy/jit/backend/llgraph/runner.py
   pypy/trunk/pypy/jit/backend/model.py
   pypy/trunk/pypy/jit/backend/x86/assembler.py
   pypy/trunk/pypy/jit/backend/x86/runner.py
   pypy/trunk/pypy/jit/metainterp/compile.py
Log:
Introduce log=bool flag on compile_loop and compile_bridge. The idea is that
if we want to log something about the loop (like say number of times it was
called), we want to skip loops that are not logged to normal log.


Modified: pypy/trunk/pypy/jit/backend/llgraph/runner.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/llgraph/runner.py	(original)
+++ pypy/trunk/pypy/jit/backend/llgraph/runner.py	Fri Oct 29 20:50:49 2010
@@ -118,13 +118,13 @@
             self._descrs[key] = descr
             return descr
 
-    def compile_bridge(self, faildescr, inputargs, operations):
+    def compile_bridge(self, faildescr, inputargs, operations, log=True):
         c = llimpl.compile_start()
         self._compile_loop_or_bridge(c, inputargs, operations)
         old, oldindex = faildescr._compiled_fail
         llimpl.compile_redirect_fail(old, oldindex, c)
 
-    def compile_loop(self, inputargs, operations, loopdescr):
+    def compile_loop(self, inputargs, operations, loopdescr, log=True):
         """In a real assembler backend, this should assemble the given
         list of operations.  Here we just generate a similar CompiledLoop
         instance.  The code here is RPython, whereas the code in llimpl

Modified: pypy/trunk/pypy/jit/backend/model.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/model.py	(original)
+++ pypy/trunk/pypy/jit/backend/model.py	Fri Oct 29 20:50:49 2010
@@ -33,14 +33,14 @@
         pass
 
 
-    def compile_loop(self, inputargs, operations, looptoken):
+    def compile_loop(self, inputargs, operations, looptoken, log=True):
         """Assemble the given loop.
         Extra attributes should be put in the LoopToken to
         point to the compiled loop in assembler.
         """
         raise NotImplementedError
 
-    def compile_bridge(self, faildescr, inputargs, operations):
+    def compile_bridge(self, faildescr, inputargs, operations, log=True):
         """Assemble the bridge.
         The FailDescr is the descr of the original guard that failed.
         """

Modified: pypy/trunk/pypy/jit/backend/x86/assembler.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/assembler.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/assembler.py	Fri Oct 29 20:50:49 2010
@@ -295,7 +295,7 @@
         self.mc.RET()
         self.mc.done()
 
-    def assemble_loop(self, inputargs, operations, looptoken):
+    def assemble_loop(self, inputargs, operations, looptoken, log):
         """adds the following attributes to looptoken:
                _x86_loop_code       (an integer giving an address)
                _x86_bootstrap_code  (an integer giving an address)
@@ -310,10 +310,11 @@
 
         self.setup()
         funcname = self._find_debug_merge_point(operations)
-
+        if log:
+            self._register_counter()
+            operations = self._inject_debugging_code(operations)
         
         regalloc = RegAlloc(self, self.cpu.translate_support_code)
-        operations = self._inject_debugging_code(operations)
         arglocs = regalloc.prepare_loop(inputargs, operations, looptoken)
         looptoken._x86_arglocs = arglocs
 
@@ -340,13 +341,16 @@
         self.mc.end_function()
         self.write_pending_failure_recoveries()
         
-    def assemble_bridge(self, faildescr, inputargs, operations):
+    def assemble_bridge(self, faildescr, inputargs, operations, log):
         if not we_are_translated():
             # Arguments should be unique
             assert len(set(inputargs)) == len(inputargs)
 
         self.setup()
         funcname = self._find_debug_merge_point(operations)
+        if log:
+            self._register_counter()
+            operations = self._inject_debugging_code(operations)
 
         arglocs = self.rebuild_faillocs_from_descr(
             faildescr._x86_failure_recovery_bytecode)
@@ -401,12 +405,14 @@
         else:
             funcname = "" % len(self.loop_run_counters)
         # invent the counter, so we don't get too confused
+        return funcname
+
+    def _register_counter(self):
         if self._debug:
             struct = lltype.malloc(DEBUG_COUNTER, flavor='raw',
                                    track_allocation=False)   # known to leak
             struct.i = 0
             self.loop_run_counters.append((len(self.loop_run_counters), struct))
-        return funcname
         
     def patch_jump_for_descr(self, faildescr, adr_new_target):
         adr_jump_offset = faildescr._x86_adr_jump_offset

Modified: pypy/trunk/pypy/jit/backend/x86/runner.py
==============================================================================
--- pypy/trunk/pypy/jit/backend/x86/runner.py	(original)
+++ pypy/trunk/pypy/jit/backend/x86/runner.py	Fri Oct 29 20:50:49 2010
@@ -49,11 +49,13 @@
         self.assembler.finish_once()
         self.profile_agent.shutdown()
 
-    def compile_loop(self, inputargs, operations, looptoken):
-        self.assembler.assemble_loop(inputargs, operations, looptoken)
-
-    def compile_bridge(self, faildescr, inputargs, operations):
-        self.assembler.assemble_bridge(faildescr, inputargs, operations)
+    def compile_loop(self, inputargs, operations, looptoken, log=True):
+        self.assembler.assemble_loop(inputargs, operations, looptoken,
+                                     log=log)
+
+    def compile_bridge(self, faildescr, inputargs, operations, log=True):
+        self.assembler.assemble_bridge(faildescr, inputargs, operations,
+                                       log=log)
 
     def set_future_value_int(self, index, intvalue):
         self.assembler.fail_boxes_int.setitem(index, intvalue)

Modified: pypy/trunk/pypy/jit/metainterp/compile.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/compile.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/compile.py	Fri Oct 29 20:50:49 2010
@@ -599,5 +599,5 @@
         ResOperation(rop.FINISH, finishargs, None, descr=jd.portal_finishtoken)
         ]
     operations[1].setfailargs([])
-    cpu.compile_loop(inputargs, operations, loop_token)
+    cpu.compile_loop(inputargs, operations, loop_token, log=False)
     return loop_token


From agaynor at codespeak.net  Fri Oct 29 20:59:16 2010
From: agaynor at codespeak.net (agaynor at codespeak.net)
Date: Fri, 29 Oct 2010 20:59:16 +0200 (CEST)
Subject: [pypy-svn] r78485 - pypy/extradoc/talk/pycon2011
Message-ID: <20101029185916.8CA3B282B9D@codespeak.net>

Author: agaynor
Date: Fri Oct 29 20:59:15 2010
New Revision: 78485

Modified:
   pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
Log:
MOre work on pycon-talk.

Modified: pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
==============================================================================
--- pypy/extradoc/talk/pycon2011/pypy-optimizations.txt	(original)
+++ pypy/extradoc/talk/pycon2011/pypy-optimizations.txt	Fri Oct 29 20:59:15 2010
@@ -1,60 +1,44 @@
 Title: Optimizations in PyPy
 Description:
 
-PyPy is a virtual machine for Python, featuring an advanced just in time compiler, which can deliver exceptional performance.  This talk is going to be a deep dive into what exactly makes Python such a hard language to optimize, how PyPy is organized, and what optimizations our JIT can do for your code.
+PyPy is a virtual machine for Python, featuring an advanced just in time
+compiler, which can deliver exceptional performance.  This talk is going to be
+a deep dive into what exactly makes Python such a hard language to optimize,
+how PyPy is organized, and what optimizations our JIT can do for your code.
 
 Session type: Talk
 Classification: Discuss in depth
 Abstract:
 
-The talk would contain a list of features that are hard to optimize
-in Python, why and how we're fixing it in PyPy. This also might be
-used as a rough guide what features in Python are optimizable right
-now and what are not.
+The talk would contain a list of features that are hard to optimize in Python,
+why and how we're fixing it in PyPy. This also might be used as a rough guide
+what features in Python are optimizable right now and what are not.
 
 The list of mentioned features is not exhaustive, however we will try to
 focus at least on the following:
 
-* Dynamic language - In Python code we have no known types, like a
-  static language.  Even operations like "a + b" can do anything,
-  unless we know more about the code.
-
-* Frame introspection - Frame objects need to be allocated for every
-  function call, and all local variables are stored on the frame, and
-  must be accessible from further down the call stack.
-
-* Object model - All user defined Python objects have a dictionary which
-  stores their attributes, as does every type.  When Python does an
-  attribute lookup this requires 2 dictionary lookups.
+* Dynamic language - In Python code we have no known types, like a static
+  language.  Even operations like "a + b" can do anything, unless we know more
+  about the code.
+
+* Frame introspection - Frame objects need to be allocated for every function
+  call, and all local variables are stored on the frame, and must be accessible
+  from further down the call stack.
+
+* Object model - All user defined Python objects have a dictionary which stores
+  their attributes, as does every type.  When Python does an attribute lookup
+  this requires 2 dictionary lookups.
 
   In PyPy we use an approach similar to one used by v8 with hidden classes
   (except more pypy specific) called shared dictionaries.
 
-* FFI calls - calling C from Python is costly and hard to optimize.
-  In PyPy we decided to go via ctypes, this part will explain how are
-  we going to handle ctypes calls to be fast.
-
-This talk is 3 parts, what's hard to optimize about Python, how PyPy is organized, and what optimizations our JIT can do.
- 
-I. Why Python is hard (8 minutes)
-    A) Dynamic language - No known types, even operations such as "+" can
-    B) Frame introspection.
-    C) Object model.
-    D) CPython
-        1) Interpretation
-        2) Boxing
-II. How PyPy is organized (5 minutes)
-    A) Written in Python
-    B) RPython
-    C) The JIT
-        1) Source transformation on the code
-        2) Orthogonal to the interpreter
-III. Optimizations the JIT does (13 minutes)
-    A) Unboxing
-    B) Attribute lookup
-        1) "Hidden classes"
-    C) Globals/builtin "freezing"
-    D) array module
-    E) The future
-        1) ctypes
-IV. Questions (4 minutes)
+* FFI calls - calling C from Python is costly and hard to optimize. In PyPy we
+  decided to go via ctypes, this part will explain how are we going to handle
+  ctypes calls to be fast.
+
+* `array` module - users of the CPython array module probably know it can save
+  them quiet a bit of memory, however it's also slower than using a list, due
+  to the overhead of boxing and unboxing on every operations. Here we will tie
+  everything together and describe how the ``array`` module is much faster with
+  PyPy's JIT, combining our optimizations to: unbox values, remove the
+  dynamicism within traces, and deliver great performance.


From afa at codespeak.net  Fri Oct 29 22:15:08 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 22:15:08 +0200 (CEST)
Subject: [pypy-svn] r78486 - pypy/branch/fast-forward/pypy/module/zlib
Message-ID: <20101029201508.60985282B9D@codespeak.net>

Author: afa
Date: Fri Oct 29 22:15:05 2010
New Revision: 78486

Modified:
   pypy/branch/fast-forward/pypy/module/zlib/interp_zlib.py
Log:
Even it the decompressobj.flush() always returns "", mimic the checks done by CPython.


Modified: pypy/branch/fast-forward/pypy/module/zlib/interp_zlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/zlib/interp_zlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/zlib/interp_zlib.py	Fri Oct 29 22:15:05 2010
@@ -321,6 +321,9 @@
         because each call to decompress() immediately returns as much
         data as possible.
         """
+        if length <= 0:
+            raise OperationError(self.space.w_ValueError, self.space.wrap(
+                "length must be greater than zero"))
         # We could call rzlib.decompress(self.stream, '', rzlib.Z_FINISH)
         # which would complain if the input stream so far is not complete;
         # however CPython's zlib module does not behave like that.


From afa at codespeak.net  Fri Oct 29 22:30:01 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 22:30:01 +0200 (CEST)
Subject: [pypy-svn] r78487 - pypy/branch/fast-forward/pypy/module/_hashlib
Message-ID: <20101029203001.C30D2282B9E@codespeak.net>

Author: afa
Date: Fri Oct 29 22:30:00 2010
New Revision: 78487

Modified:
   pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py
Log:
Expose the same _hashlib items as CPython


Modified: pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/__init__.py	Fri Oct 29 22:30:00 2010
@@ -5,14 +5,13 @@
 class Module(MixedModule):
     interpleveldefs = {
         'new' : 'interp_hashlib.new',
-        'HASH': 'interp_hashlib.W_Hash',
         }
 
     appleveldefs = {
         }
 
     for name in algorithms:
-        interpleveldefs[name] = 'interp_hashlib.new_%s' % (name,)
+        interpleveldefs['openssl_' + name] = 'interp_hashlib.new_' + name
 
     def startup(self, space):
         from pypy.rlib.ropenssl import init_digests


From afa at codespeak.net  Fri Oct 29 22:40:52 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 22:40:52 +0200 (CEST)
Subject: [pypy-svn] r78488 -
	pypy/branch/fast-forward/pypy/module/_hashlib/test
Message-ID: <20101029204052.D84D3282B9E@codespeak.net>

Author: afa
Date: Fri Oct 29 22:40:51 2010
New Revision: 78488

Modified:
   pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py
Log:
Fix test. hum


Modified: pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py	Fri Oct 29 22:40:51 2010
@@ -7,7 +7,7 @@
 
     def test_simple(self):
         import _hashlib
-        assert isinstance(_hashlib.new('md5'), _hashlib.HASH)
+        assert _hashlib.new('md5').__class__.__name__ == 'HASH'
         assert len(_hashlib.new('md5').hexdigest()) == 32
 
     def test_attributes(self):
@@ -59,5 +59,5 @@
 
     def test_unicode(self):
         import _hashlib
-        assert isinstance(_hashlib.new('sha1', u'xxx'), _hashlib.HASH)
+        assert _hashlib.new('sha1', u'xxx').__class__.__name__ == 'HASH'
 


From afa at codespeak.net  Fri Oct 29 23:05:43 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 23:05:43 +0200 (CEST)
Subject: [pypy-svn] r78489 - in pypy/branch/fast-forward/pypy/interpreter: .
	test
Message-ID: <20101029210543.78979282BAD@codespeak.net>

Author: afa
Date: Fri Oct 29 23:05:41 2010
New Revision: 78489

Modified:
   pypy/branch/fast-forward/pypy/interpreter/generator.py
   pypy/branch/fast-forward/pypy/interpreter/test/test_generator.py
   pypy/branch/fast-forward/pypy/interpreter/typedef.py
Log:
add a __repr__ to generators


Modified: pypy/branch/fast-forward/pypy/interpreter/generator.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/generator.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/generator.py	Fri Oct 29 23:05:41 2010
@@ -13,6 +13,12 @@
         self.frame = frame
         self.running = False
 
+    def descr__repr__(self, space):
+        code_name = self.frame.pycode.co_name
+        addrstring = self.getaddrstring(space)
+        return space.wrap("" %
+                          (code_name, addrstring))
+
     def descr__reduce__(self, space):
         from pypy.interpreter.mixedmodule import MixedModule
         w_mod    = space.getbuiltinmodule('_pickle_support')

Modified: pypy/branch/fast-forward/pypy/interpreter/test/test_generator.py
==============================================================================
--- pypy/branch/fast-forward/pypy/interpreter/test/test_generator.py	(original)
+++ pypy/branch/fast-forward/pypy/interpreter/test/test_generator.py	Fri Oct 29 23:05:41 2010
@@ -241,3 +241,8 @@
 res = f()
 """ in d
         assert d['res'] == (10, 7)
+
+    def test_repr(self):
+        def myFunc():
+            yield 1
+        assert repr(myFunc()).startswith("

Author: afa
Date: Fri Oct 29 23:10:24 2010
New Revision: 78490

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_genexps.py
      - copied, changed from r78483, pypy/branch/fast-forward/lib-python/2.7.0/test/test_genexps.py
Log:
Skip one CPython-specific test and update docstrings


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_genexps.py (from r78483, pypy/branch/fast-forward/lib-python/2.7.0/test/test_genexps.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_genexps.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_genexps.py	Fri Oct 29 23:10:24 2010
@@ -128,9 +128,9 @@
 
 Verify re-use of tuples (a side benefit of using genexps over listcomps)
 
-    >>> tupleids = map(id, ((i,i) for i in xrange(10)))
-    >>> int(max(tupleids) - min(tupleids))
-    0
+##  >>> tupleids = map(id, ((i,i) for i in xrange(10)))
+##  >>> int(max(tupleids) - min(tupleids))
+##  0
 
 Verify that syntax error's are raised for genexps used as lvalues
 
@@ -204,7 +204,7 @@
         g.next()
       File "", line 1, in 
         g = (10 // i for i in (5, 0, 2))
-    ZeroDivisionError: integer division or modulo by zero
+    ZeroDivisionError: integer division by zero
     >>> g.next()
     Traceback (most recent call last):
       File "", line 1, in -toplevel-
@@ -224,7 +224,7 @@
     True
 
     >>> print g.next.__doc__
-    x.next() -> the next value, or raise StopIteration
+    next() -> the next value, or raise StopIteration
     >>> import types
     >>> isinstance(g, types.GeneratorType)
     True


From afa at codespeak.net  Fri Oct 29 23:28:58 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 23:28:58 +0200 (CEST)
Subject: [pypy-svn] r78491 - in pypy/branch/fast-forward/pypy/rlib: . test
Message-ID: <20101029212858.BDC44282B9E@codespeak.net>

Author: afa
Date: Fri Oct 29 23:28:57 2010
New Revision: 78491

Modified:
   pypy/branch/fast-forward/pypy/rlib/rmmap.py
   pypy/branch/fast-forward/pypy/rlib/test/test_rmmap.py
Log:
Implement fix for CPython bug #2111:
mmap segfaults when trying to write a block opened with PROT_READ


Modified: pypy/branch/fast-forward/pypy/rlib/rmmap.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rmmap.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rmmap.py	Fri Oct 29 23:28:57 2010
@@ -591,6 +591,9 @@
         else:
             raise RValueError("mmap invalid access parameter.")
 
+        if prot == PROT_READ:
+            access = ACCESS_READ
+
         # check file size
         try:
             st = os.fstat(fd)

Modified: pypy/branch/fast-forward/pypy/rlib/test/test_rmmap.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/test/test_rmmap.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/test/test_rmmap.py	Fri Oct 29 23:28:57 2010
@@ -229,6 +229,16 @@
         interpret(func, [f.fileno()])
         f.close()
 
+    def test_write_readonly(self):
+        if os.name == "nt":
+            skip("Needs PROT_READ")
+        f = open(self.tmpname + "l", "w+")
+        f.write("foobar")
+        f.flush()
+        m = mmap.mmap(f.fileno(), 6, prot=mmap.PROT_READ)
+        raises(RTypeError, m.write, "foo")
+        f.close()
+
     def test_size(self):
         f = open(self.tmpname + "l", "w+")
         


From afa at codespeak.net  Fri Oct 29 23:41:28 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 23:41:28 +0200 (CEST)
Subject: [pypy-svn] r78492 -
	pypy/branch/fast-forward/lib-python/modified-2.7.0/test
Message-ID: <20101029214128.1B21E282B9D@codespeak.net>

Author: afa
Date: Fri Oct 29 23:41:26 2010
New Revision: 78492

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_mmap.py
      - copied, changed from r78483, pypy/branch/fast-forward/lib-python/2.7.0/test/test_mmap.py
Log:
Flush data before reopening the same file


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_mmap.py (from r78483, pypy/branch/fast-forward/lib-python/2.7.0/test/test_mmap.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/test/test_mmap.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_mmap.py	Fri Oct 29 23:41:26 2010
@@ -118,7 +118,8 @@
     def test_access_parameter(self):
         # Test for "access" keyword parameter
         mapsize = 10
-        open(TESTFN, "wb").write("a"*mapsize)
+        with open(TESTFN, "wb") as f:
+            f.write("a"*mapsize)
         f = open(TESTFN, "rb")
         m = mmap.mmap(f.fileno(), mapsize, access=mmap.ACCESS_READ)
         self.assertEqual(m[:], 'a'*mapsize, "Readonly memory map data incorrect.")
@@ -494,7 +495,8 @@
         if not hasattr(mmap, 'PROT_READ'):
             return
         mapsize = 10
-        open(TESTFN, "wb").write("a"*mapsize)
+        with open(TESTFN, "wb") as f:
+            f.write("a"*mapsize)
         f = open(TESTFN, "rb")
         m = mmap.mmap(f.fileno(), mapsize, prot=mmap.PROT_READ)
         self.assertRaises(TypeError, m.write, "foo")
@@ -506,7 +508,8 @@
 
     def test_io_methods(self):
         data = "0123456789"
-        open(TESTFN, "wb").write("x"*len(data))
+        with open(TESTFN, "wb") as f:
+            f.write("x"*len(data))
         f = open(TESTFN, "r+b")
         m = mmap.mmap(f.fileno(), len(data))
         f.close()


From afa at codespeak.net  Fri Oct 29 23:42:27 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 23:42:27 +0200 (CEST)
Subject: [pypy-svn] r78493 - pypy/branch/fast-forward/pypy/rlib
Message-ID: <20101029214227.7FA72282BEA@codespeak.net>

Author: afa
Date: Fri Oct 29 23:42:25 2010
New Revision: 78493

Modified:
   pypy/branch/fast-forward/pypy/rlib/rmmap.py
Log:
Array bound check in mmap.write_byte()


Modified: pypy/branch/fast-forward/pypy/rlib/rmmap.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rmmap.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rmmap.py	Fri Oct 29 23:42:25 2010
@@ -421,8 +421,11 @@
         
         if len(byte) != 1:
             raise RTypeError("write_byte() argument must be char")
-        
+
         self.check_writeable()
+        if self.pos >= self.size:
+            raise RValueError("write byte out of range")
+
         self.data[self.pos] = byte[0]
         self.pos += 1
 


From afa at codespeak.net  Fri Oct 29 23:53:20 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 23:53:20 +0200 (CEST)
Subject: [pypy-svn] r78494 - in pypy/branch/fast-forward/pypy/module/mmap: .
	test
Message-ID: <20101029215320.B248B36E0C1@codespeak.net>

Author: afa
Date: Fri Oct 29 23:53:19 2010
New Revision: 78494

Modified:
   pypy/branch/fast-forward/pypy/module/mmap/__init__.py
   pypy/branch/fast-forward/pypy/module/mmap/interp_mmap.py
   pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py
Log:
mmap.mmap is now a class (it was a function)


Modified: pypy/branch/fast-forward/pypy/module/mmap/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/mmap/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/mmap/__init__.py	Fri Oct 29 23:53:19 2010
@@ -4,7 +4,7 @@
 class Module(MixedModule):
     interpleveldefs = {
         'PAGESIZE': 'space.wrap(interp_mmap.PAGESIZE)',
-        'mmap': 'interp_mmap.mmap'
+        'mmap': 'interp_mmap.W_MMap'
     }
 
     appleveldefs = {

Modified: pypy/branch/fast-forward/pypy/module/mmap/interp_mmap.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/mmap/interp_mmap.py	(original)
+++ pypy/branch/fast-forward/pypy/module/mmap/interp_mmap.py	Fri Oct 29 23:53:19 2010
@@ -188,7 +188,43 @@
         return space.wrap(StringLikeBuffer(space, space.wrap(self)))
     descr_buffer.unwrap_spec = ['self']
 
+if rmmap._POSIX:
+
+    def mmap(space, w_subtype, fileno, length, flags=rmmap.MAP_SHARED,
+             prot=rmmap.PROT_WRITE | rmmap.PROT_READ,
+             access=rmmap._ACCESS_DEFAULT):
+        self = space.allocate_instance(W_MMap, w_subtype)
+        try:
+            W_MMap.__init__(self, space,
+                            rmmap.mmap(fileno, length, flags, prot, access))
+        except OSError, e:
+            raise wrap_oserror(space, e, exception_name='w_EnvironmentError')
+        except RValueError, e:
+            raise OperationError(space.w_ValueError, space.wrap(e.message))
+        except RTypeError, e:
+            raise OperationError(space.w_TypeError, space.wrap(e.message))
+        return space.wrap(self)
+    mmap.unwrap_spec = [ObjSpace, W_Root, int, 'index', int, int, int]
+
+elif rmmap._MS_WINDOWS:
+
+    def mmap(space, w_subtype, fileno, length, tagname="",
+             access=rmmap._ACCESS_DEFAULT):
+        self = space.allocate_instance(W_MMap, w_subtype)
+        try:
+            W_MMap.__init__(self, space,
+                            rmmap.mmap(fileno, length, tagname, access))
+        except OSError, e:
+            raise wrap_oserror(space, e, exception_name='w_EnvironmentError')
+        except RValueError, e:
+            raise OperationError(space.w_ValueError, space.wrap(e.message))
+        except RTypeError, e:
+            raise OperationError(space.w_TypeError, space.wrap(e.message))
+        return space.wrap(self)
+    mmap.unwrap_spec = [ObjSpace, W_Root, int, 'index', str, int]
+
 W_MMap.typedef = TypeDef("mmap",
+    __new__ = interp2app(mmap),
     close = interp2app(W_MMap.close),
     read_byte = interp2app(W_MMap.read_byte),
     readline = interp2app(W_MMap.readline),
@@ -210,43 +246,5 @@
     __buffer__ = interp2app(W_MMap.descr_buffer),
 )
 
-def _check_map_size(space, size):
-    if size < 0:
-        raise OperationError(space.w_TypeError,
-            space.wrap("memory mapped size must be positive"))
-    if rffi.cast(size_t, size) != size:
-        raise OperationError(space.w_OverflowError,
-            space.wrap("memory mapped size is too large (limited by C int)"))
-
-if rmmap._POSIX:
-    
-    def mmap(space, fileno, length, flags=rmmap.MAP_SHARED,
-        prot=rmmap.PROT_WRITE | rmmap.PROT_READ, access=rmmap._ACCESS_DEFAULT):
-
-        try:
-            return space.wrap(W_MMap(space, rmmap.mmap(fileno, length,
-                                                       flags, prot, access)))
-        except OSError, e:
-            raise wrap_oserror(space, e, exception_name='w_EnvironmentError')
-        except RValueError, e:
-            raise OperationError(space.w_ValueError, space.wrap(e.message))
-        except RTypeError, e:
-            raise OperationError(space.w_TypeError, space.wrap(e.message))
-    mmap.unwrap_spec = [ObjSpace, int, 'index', int, int, int]
-
-elif rmmap._MS_WINDOWS:
-
-    def mmap(space, fileno, length, tagname="", access=rmmap._ACCESS_DEFAULT):
-        try:
-            return space.wrap(W_MMap(space, rmmap.mmap(fileno, length,
-                                                       tagname, access)))
-        except OSError, e:
-            raise wrap_oserror(space, e, exception_name='w_EnvironmentError')
-        except RValueError, e:
-            raise OperationError(space.w_ValueError, space.wrap(e.message))
-        except RTypeError, e:
-            raise OperationError(space.w_TypeError, space.wrap(e.message))
-    mmap.unwrap_spec = [ObjSpace, int, 'index', str, int]
-
 constants = rmmap.constants
 PAGESIZE = rmmap.PAGESIZE

Modified: pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py	(original)
+++ pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py	Fri Oct 29 23:53:19 2010
@@ -34,7 +34,8 @@
         from mmap import mmap
         import os
         import sys
-        
+
+        assert isinstance(mmap, type)
         raises(TypeError, mmap, "foo")
         raises(TypeError, mmap, 0, "foo")
              


From afa at codespeak.net  Fri Oct 29 23:54:59 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Fri, 29 Oct 2010 23:54:59 +0200 (CEST)
Subject: [pypy-svn] r78495 - pypy/branch/fast-forward/pypy/module/mmap/test
Message-ID: <20101029215459.16D62282C24@codespeak.net>

Author: afa
Date: Fri Oct 29 23:54:57 2010
New Revision: 78495

Modified:
   pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py
Log:
Add a real test for a mmap subclass


Modified: pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py	(original)
+++ pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py	Fri Oct 29 23:54:57 2010
@@ -35,7 +35,6 @@
         import os
         import sys
 
-        assert isinstance(mmap, type)
         raises(TypeError, mmap, "foo")
         raises(TypeError, mmap, 0, "foo")
              
@@ -53,6 +52,13 @@
             raises(TypeError, mmap, 0, 1, access="foo")
             raises(ValueError, mmap, 0, 1, access=-1)
 
+    def test_subclass(self):
+        import mmap
+        class anon_mmap(mmap.mmap):
+            def __new__(klass, *args, **kwargs):
+                return mmap.mmap.__new__(klass, -1, *args, **kwargs)
+        anon_mmap(mmap.PAGESIZE)
+
     def test_file_size(self):
         import os
         if os.name == "nt":


From afa at codespeak.net  Sat Oct 30 00:13:56 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sat, 30 Oct 2010 00:13:56 +0200 (CEST)
Subject: [pypy-svn] r78496 - in pypy/branch/fast-forward/pypy: module/mmap
	module/mmap/test rlib
Message-ID: <20101029221356.0DE8D282BD6@codespeak.net>

Author: afa
Date: Sat Oct 30 00:13:55 2010
New Revision: 78496

Modified:
   pypy/branch/fast-forward/pypy/module/mmap/__init__.py
   pypy/branch/fast-forward/pypy/module/mmap/interp_mmap.py
   pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py
   pypy/branch/fast-forward/pypy/rlib/rmmap.py
Log:
Add mmap.ALLOCATIONGRANULARITY


Modified: pypy/branch/fast-forward/pypy/module/mmap/__init__.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/mmap/__init__.py	(original)
+++ pypy/branch/fast-forward/pypy/module/mmap/__init__.py	Sat Oct 30 00:13:55 2010
@@ -4,6 +4,7 @@
 class Module(MixedModule):
     interpleveldefs = {
         'PAGESIZE': 'space.wrap(interp_mmap.PAGESIZE)',
+        'ALLOCATIONGRANULARITY': 'space.wrap(interp_mmap.ALLOCATIONGRANULARITY)',
         'mmap': 'interp_mmap.W_MMap'
     }
 

Modified: pypy/branch/fast-forward/pypy/module/mmap/interp_mmap.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/mmap/interp_mmap.py	(original)
+++ pypy/branch/fast-forward/pypy/module/mmap/interp_mmap.py	Sat Oct 30 00:13:55 2010
@@ -248,3 +248,4 @@
 
 constants = rmmap.constants
 PAGESIZE = rmmap.PAGESIZE
+ALLOCATIONGRANULARITY = rmmap.ALLOCATIONGRANULARITY

Modified: pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py	(original)
+++ pypy/branch/fast-forward/pypy/module/mmap/test/test_mmap.py	Sat Oct 30 00:13:55 2010
@@ -11,8 +11,11 @@
     def test_page_size(self):
         import mmap
         assert mmap.PAGESIZE > 0
+        assert mmap.ALLOCATIONGRANULARITY > 0
         assert isinstance(mmap.PAGESIZE, int)
-        
+        assert isinstance(mmap.ALLOCATIONGRANULARITY, int)
+        assert mmap.ALLOCATIONGRANULARITY % mmap.PAGESIZE == 0
+
     def test_attributes(self):
         import mmap
         import os

Modified: pypy/branch/fast-forward/pypy/rlib/rmmap.py
==============================================================================
--- pypy/branch/fast-forward/pypy/rlib/rmmap.py	(original)
+++ pypy/branch/fast-forward/pypy/rlib/rmmap.py	Sat Oct 30 00:13:55 2010
@@ -119,6 +119,7 @@
 
     # this one is always safe
     _, _get_page_size = external('getpagesize', [], rffi.INT)
+    _get_allocation_granularity = _get_page_size
 
     def _get_error_no():
         return rposix.get_errno()
@@ -195,7 +196,7 @@
     VirtualFree = winexternal('VirtualFree',
                               [rffi.VOIDP, rffi.SIZE_T, DWORD], BOOL)
 
-    
+
     def _get_page_size():
         try:
             si = rffi.make(SYSTEM_INFO)
@@ -203,7 +204,15 @@
             return int(si.c_dwPageSize)
         finally:
             lltype.free(si, flavor="raw")
-    
+
+    def _get_allocation_granularity():
+        try:
+            si = rffi.make(SYSTEM_INFO)
+            GetSystemInfo(si)
+            return int(si.c_dwAllocationGranularity)
+        finally:
+            lltype.free(si, flavor="raw")
+
     def _get_file_size(handle):
         # XXX use native Windows types like WORD
         high_ref = lltype.malloc(LPDWORD.TO, 1, flavor='raw')
@@ -232,6 +241,7 @@
     INVALID_HANDLE = INVALID_HANDLE_VALUE
 
 PAGESIZE = _get_page_size()
+ALLOCATIONGRANULARITY = _get_allocation_granularity()
 NULL = lltype.nullptr(PTR.TO)
 NODATA = lltype.nullptr(PTR.TO)
 


From afa at codespeak.net  Sat Oct 30 01:06:43 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sat, 30 Oct 2010 01:06:43 +0200 (CEST)
Subject: [pypy-svn] r78499 - in
	pypy/branch/fast-forward/lib-python/modified-2.7.0: . test
Message-ID: <20101029230643.78BDE282B9D@codespeak.net>

Author: afa
Date: Sat Oct 30 01:06:41 2010
New Revision: 78499

Added:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/socket.py
      - copied, changed from r78496, pypy/branch/fast-forward/lib-python/2.7.0/socket.py
Modified:
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_socket.py
   pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py
Log:
Try to introduce py3k's idea of explicit reference counting of sockets

This is not the same because in py3k:
- py3k uses inheritance of the raw _socket.socket, not delegation
- the C module implements a real dup(), even on Windows
- socket.makefile does not take ownership of the fd

But it seems to work, and I could even remove most gc.collect() from the tests.


Copied: pypy/branch/fast-forward/lib-python/modified-2.7.0/socket.py (from r78496, pypy/branch/fast-forward/lib-python/2.7.0/socket.py)
==============================================================================
--- pypy/branch/fast-forward/lib-python/2.7.0/socket.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/socket.py	Sat Oct 30 01:06:41 2010
@@ -180,12 +180,15 @@
 
     __doc__ = _realsocket.__doc__
 
-    __slots__ = ["_sock", "__weakref__"] + list(_delegate_methods)
+    __slots__ = ["_sock", "__weakref__", "_io_refs", "_closed"
+                 ] + list(_delegate_methods)
 
     def __init__(self, family=AF_INET, type=SOCK_STREAM, proto=0, _sock=None):
         if _sock is None:
             _sock = _realsocket(family, type, proto)
         self._sock = _sock
+        self._io_refs = 0
+        self._closed = False
         for method in _delegate_methods:
             setattr(self, method, getattr(_sock, method))
 
@@ -216,6 +219,22 @@
         and bufsize arguments are as for the built-in open() function."""
         return _fileobject(self._sock, mode, bufsize)
 
+    def _decref_socketios(self):
+        if self._io_refs > 0:
+            self._io_refs -= 1
+        if self._closed:
+            self.close()
+
+    def _real_close(self):
+        # This function should not reference any globals. See issue #808164.
+        self._sock.close()
+
+    def close(self):
+        # This function should not reference any globals. See issue #808164.
+        self._closed = True
+        if self._io_refs <= 0:
+            self._real_close()
+
     family = property(lambda self: self._sock.family, doc="the socket family")
     type = property(lambda self: self._sock.type, doc="the socket type")
     proto = property(lambda self: self._sock.proto, doc="the socket protocol")

Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_socket.py
==============================================================================
--- pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_socket.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_socket.py	Sat Oct 30 01:06:41 2010
@@ -160,7 +160,6 @@
 
     def clientTearDown(self):
         self.done.set()
-        test_support.gc_collect()
         thread.exit()
 
 class ThreadedTCPSocketTest(SocketTCPTest, ThreadableTest):
@@ -729,7 +728,6 @@
     def testClose(self):
         conn, addr = self.serv.accept()
         conn.close()
-        test_support.gc_collect() # not sure why
 
         sd = self.cli
         read, write, err = select.select([sd], [], [], 1.0)

Modified: pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py
==============================================================================
--- pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py	(original)
+++ pypy/branch/fast-forward/lib-python/modified-2.7.0/test/test_support.py	Sat Oct 30 01:06:41 2010
@@ -281,9 +281,7 @@
     tempsock = socket.socket(family, socktype)
     port = bind_port(tempsock)
     tempsock.close()
-    # Be sure to close the socket so the port can be reused
     del tempsock
-    gc_collect()
     return port
 
 def bind_port(sock, host=HOST):


From hakanardo at codespeak.net  Sat Oct 30 11:07:29 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 11:07:29 +0200 (CEST)
Subject: [pypy-svn] r78503 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030090729.8D581282BAD@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 11:07:28 2010
New Revision: 78503

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(armin, hakanardo) fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 11:07:28 2010
@@ -242,21 +242,9 @@
         assert equaloplists(optimized.operations,
                             expected.operations, False, remap)
 
-    def optimize_loop(self, ops, spectext, optops, checkspecnodes=True,
-                      expected_preamble=None):
+    def optimize_loop(self, ops, spectext, optops, expected_preamble=None):
         loop = self.parse(ops)
-        #
-        if checkspecnodes:
-            # verify that 'spectext' is indeed what optimizefindnode would
-            # compute for this loop
-            cpu = self.cpu
-            perfect_specialization_finder = PerfectSpecializationFinder(cpu)
-            perfect_specialization_finder.find_nodes_loop(loop)
-            self.check_specnodes(loop.token.specnodes, spectext)
-        else:
-            # for cases where we want to see how optimizeopt behaves with
-            # combinations different from the one computed by optimizefindnode
-            loop.token.specnodes = self.unpack_specnodes(spectext)
+        loop.token.specnodes = self.unpack_specnodes(spectext)
         #
         self.loop = loop
         loop.preamble = TreeLoop('preamble')
@@ -267,11 +255,16 @@
         optimize_loop_1(metainterp_sd, loop)
         #
         expected = self.parse(optops)
+
         print
-        print "Ops: "
+        print loop.preamble.inputargs
+        print '\n'.join([str(o) for o in loop.preamble.operations])
+        print 
+        print loop.inputargs
         print '\n'.join([str(o) for o in loop.operations])
+        print
+        
         self.assert_equal(loop, expected)
-
         if expected_preamble:
             expected = self.parse(expected_preamble)
             self.assert_equal(loop.preamble, expected)
@@ -414,13 +407,17 @@
         guard_false(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_lt(i0, 0)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, 'Not', expected, expected_preamble=preamble)
 
     def test_constant_boolrewrite_gt(self):
         ops = """
@@ -431,13 +428,17 @@
         guard_false(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_gt(i0, 0)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, 'Not', expected, expected_preamble=preamble)
 
     def test_constant_boolrewrite_reflex(self):
         ops = """
@@ -448,13 +449,17 @@
         guard_true(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_gt(i0, 0)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, 'Not', expected, expected_preamble=preamble)
 
     def test_constant_boolrewrite_reflex_invers(self):
         ops = """
@@ -465,13 +470,17 @@
         guard_false(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_gt(i0, 0)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, 'Not', expected, expected_preamble=preamble)
 
     def test_remove_consecutive_guard_value_constfold(self):
         ops = """
@@ -497,13 +506,13 @@
         ops = """
         [p1]
         guard_value(p1, ConstPtr(myptr)) []
-        jump(ConstPtr(myptr))
+        jump(p1)
         """
         expected = """
         []
         jump()
         """
-        self.optimize_loop(ops, 'Constant(myptr)', expected)
+        self.optimize_loop(ops, 'Not', expected)
 
     def test_ooisnull_oononnull_1(self):
         ops = """
@@ -512,12 +521,16 @@
         guard_nonnull(p0) []
         jump(p0)
         """
-        expected = """
+        preamble = """
         [p0]
         guard_class(p0, ConstClass(node_vtable)) []
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [p0]
+        jump(p0)
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_int_is_true_1(self):
         ops = """
@@ -528,13 +541,17 @@
         guard_true(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_is_true(i0)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_int_is_true_is_zero(self):
         py.test.skip("XXX implement me")
@@ -561,12 +578,16 @@
         guard_nonnull(p0) []
         jump(p0)
         """
-        expected = """
+        preamble = """
         [p0]
         guard_nonnull(p0) []
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [p0]
+        jump(p0)
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_ooisnull_on_null_ptr_1(self):
         ops = """
@@ -594,12 +615,16 @@
         guard_nonnull(p1) []
         jump(p0)
         """
-        expected = """
+        preamble = """
         [p0]
         guard_nonnull(p0) []
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [p0]
+        jump(p0)
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_oois_1(self):
         ops = """
@@ -615,12 +640,16 @@
         guard_false(i1) []
         jump(p0)
         """
-        expected = """
+        preamble = """
         [p0]
         guard_class(p0, ConstClass(node_vtable)) []
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [p0]
+        jump(p0)
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_nonnull_1(self):
         ops = """
@@ -673,13 +702,17 @@
         guard_value(i1, 1) [i]
         jump(i)
         """
-        expected = """
+        preamble = """
         [i]
         i1 = int_lt(i, 3)
         guard_true(i1) [i]
         jump(i)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [i]
+        jump(i)
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_guard_value_to_guard_false(self):
         ops = """
@@ -688,13 +721,17 @@
         guard_value(i1, 0) [i]
         jump(i)
         """
-        expected = """
+        preamble = """
         [i]
         i1 = int_is_true(i)
         guard_false(i1) [i]
         jump(i)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [i]
+        jump(i)
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_guard_value_on_nonbool(self):
         ops = """
@@ -703,13 +740,17 @@
         guard_value(i1, 0) [i]
         jump(i)
         """
-        expected = """
+        preamble = """
         [i]
         i1 = int_add(i, 3)
         guard_value(i1, 0) [i]
-        jump(-3)
+        jump()
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        []
+        jump()
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_int_is_true_of_bool(self):
         ops = """
@@ -720,13 +761,17 @@
         guard_value(i4, 0) [i0, i1]
         jump(i0, i1)
         """
-        expected = """
+        preamble = """
         [i0, i1]
         i2 = int_gt(i0, i1)
         guard_false(i2) [i0, i1]
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        expected = """
+        [i0, i1]
+        jump(i0, i1)
+        """
+        self.optimize_loop(ops, 'Not, Not', expected, preamble)
 
 
 
@@ -740,8 +785,22 @@
         setfield_gc(p1, i1, descr=valuedescr)
         jump(i1, p1, p2)
         """
+        preamble = """
+        [i1, p2, p3]
+        i3 = getfield_gc(p3, descr=valuedescr)
+        escape(i3)
+        jump(i1, p2)
+        """
+        expected = """
+        [i1, p2]
+        i3 = getfield_gc(p2, descr=valuedescr)
+        escape(i3)
+        p3 = new_with_vtable(ConstClass(node_vtable))
+        setfield_gc(p3, i1, descr=valuedescr)        
+        jump(i1, p3)
+        """
         # We cannot track virtuals that survive for more than two iterations.
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
 
     def test_p123_nested(self):
         ops = """
@@ -757,7 +816,24 @@
         """
         # The same as test_p123_simple, but with a virtual containing another
         # virtual.
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        preamble = """
+        [i1, p2, p3]
+        i3 = getfield_gc(p3, descr=valuedescr)
+        escape(i3)
+        jump(i1, p2)
+        """
+        expected = """
+        [i1, p2]
+        i3 = getfield_gc(p2, descr=valuedescr)
+        escape(i3)
+        p4 = new_with_vtable(ConstClass(node_vtable))
+        setfield_gc(p4, i1, descr=valuedescr)
+        p1sub = new_with_vtable(ConstClass(node_vtable2))
+        setfield_gc(p1sub, i1, descr=valuedescr)
+        setfield_gc(p4, p1sub, descr=nextdescr)
+        jump(i1, p4)
+        """
+        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
 
     def test_p123_anti_nested(self):
         ops = """
@@ -773,7 +849,23 @@
         """
         # The same as test_p123_simple, but in the end the "old" p2 contains
         # a "young" virtual p2sub.  Make sure it is all forced.
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        preamble = """
+        [i1, p2, p3]
+        p3sub = getfield_gc(p3, descr=nextdescr)
+        i3 = getfield_gc(p3sub, descr=valuedescr)
+        escape(i3)
+        jump(i1, p2)
+        """
+        expected = """
+        [i1, p3]
+        p2sub = new_with_vtable(ConstClass(node_vtable2))
+        setfield_gc(p2sub, i1, descr=valuedescr)
+        setfield_gc(p3, p2sub, descr=nextdescr)
+        escape(i1)
+        p4 = new_with_vtable(ConstClass(node_vtable))
+        jump(i1, p4)
+        """
+        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
 
     # ----------
 


From cfbolz at codespeak.net  Sat Oct 30 11:52:26 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 11:52:26 +0200 (CEST)
Subject: [pypy-svn] r78504 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030095226.0067D282BD4@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 11:52:25 2010
New Revision: 78504

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(hakanardo, cfbolz) a test to try to understand how unrolling works with delayed
setfields


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 11:52:25 2010
@@ -1932,6 +1932,34 @@
         """
         self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected)
 
+    def test_duplicate_setfield_virtual(self):
+        # the difference with the previous test is that the field value is
+        # a virtual, which we try hard to keep virtual
+        ops = """
+        [p1, i2, i3, p4]
+        p2 = new_with_vtable(ConstClass(node_vtable))
+        setfield_gc(p2, p4, descr=nextdescr)
+        setfield_gc(p1, p2, descr=nextdescr)
+        guard_true(i3) []
+        i4 = int_neg(i2)
+        jump(p1, i2, i4, p4)
+        """
+        preamble = """
+        [p1, i2, i3, p4]
+        guard_true(i3) [p1, p4]
+        i4 = int_neg(i2)
+        jump(p1, i2, i4, p4)
+        """
+        expected = """
+        [p1, i2, i4, p4]
+        guard_true(i4) [p1, p4]
+        p2 = new_with_vtable(ConstClass(node_vtable))
+        setfield_gc(p2, p4, descr=nextdescr)
+        setfield_gc(p1, p2, descr=nextdescr)
+        jump(p1, i2, 1, p4)
+        """
+        self.optimize_loop(ops, 'Not, Not, Not, Not', expected, preamble)
+
     def test_bug_1(self):
         ops = """
         [i0, p1]


From cfbolz at codespeak.net  Sat Oct 30 11:52:38 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 11:52:38 +0200 (CEST)
Subject: [pypy-svn] r78505 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101030095238.2796B282BDB@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 11:52:36 2010
New Revision: 78505

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
Log:
(hakanardo, cfbolz): an XXX and a typo


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Sat Oct 30 11:52:36 2010
@@ -4,7 +4,7 @@
 from pypy.jit.metainterp.resume import Snapshot
 
 class OptUnroll(Optimization):
-    """Unroll the loop into two itterations. The first one will
+    """Unroll the loop into two iterations. The first one will
     become the preamble or entry bridge (don't think there is a
     distinction anymore)"""
     
@@ -43,7 +43,7 @@
         for v in self.optimizer.values.values():
             v.last_guard_index = -1 # FIXME: Are there any more indexes stored?
             if not v.is_constant() and v.box:
-                v.fromstart = True
+                v.fromstart = True # XXX do we really need to use an attribute?
 
         for op in self.optimizer.pure_operations.values():
             v = self.getvalue(op.result)


From cfbolz at codespeak.net  Sat Oct 30 12:01:27 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 12:01:27 +0200 (CEST)
Subject: [pypy-svn] r78506 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030100127.3754D282BDD@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 12:01:25 2010
New Revision: 78506

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
kill copy-pasted comment


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 12:01:25 2010
@@ -1933,8 +1933,6 @@
         self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected)
 
     def test_duplicate_setfield_virtual(self):
-        # the difference with the previous test is that the field value is
-        # a virtual, which we try hard to keep virtual
         ops = """
         [p1, i2, i3, p4]
         p2 = new_with_vtable(ConstClass(node_vtable))


From arigo at codespeak.net  Sat Oct 30 12:18:12 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 12:18:12 +0200 (CEST)
Subject: [pypy-svn] r78507 - in pypy/trunk/pypy: annotation objspace/std
	objspace/std/test rlib rpython/lltypesystem rpython/test
Message-ID: <20101030101812.7B7C1282BE8@codespeak.net>

Author: arigo
Date: Sat Oct 30 12:18:10 2010
New Revision: 78507

Modified:
   pypy/trunk/pypy/annotation/unaryop.py
   pypy/trunk/pypy/objspace/std/dictmultiobject.py
   pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py
   pypy/trunk/pypy/rlib/objectmodel.py
   pypy/trunk/pypy/rpython/lltypesystem/rdict.py
   pypy/trunk/pypy/rpython/test/test_rdict.py
Log:
Add dict.popitem() as an RPython operation, which gives it a better implementation
(see POPITEMINDEX in rdict.py)



Modified: pypy/trunk/pypy/annotation/unaryop.py
==============================================================================
--- pypy/trunk/pypy/annotation/unaryop.py	(original)
+++ pypy/trunk/pypy/annotation/unaryop.py	Sat Oct 30 12:18:10 2010
@@ -434,6 +434,9 @@
     def method_clear(dct):
         pass
 
+    def method_popitem(dct):
+        return dct.getanyitem('items')
+
     def _can_only_throw(dic, *ignore):
         if dic1.dictdef.dictkey.custom_eq_hash:
             return None    # r_dict: can throw anything

Modified: pypy/trunk/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/dictmultiobject.py	Sat Oct 30 12:18:10 2010
@@ -157,6 +157,16 @@
         key = OPTIMIZED_BUILTINS[i]
         return self.impl_getitem_str(key)
 
+    def impl_popitem(self):
+        # default implementation
+        space = self.space
+        iterator = self.impl_iter()
+        w_key, w_value = iterator.next()
+        if w_key is None:
+            raise KeyError
+        self.impl_delitem(w_key)
+        return w_key, w_value
+
     # _________________________________________________________________
     # fallback implementation methods
 
@@ -196,6 +206,9 @@
         key = OPTIMIZED_BUILTINS[i]
         return self.impl_fallback_getitem_str(key)
 
+    def impl_fallback_popitem(self):
+        return self.r_dict_content.popitem()
+
 
 implementation_methods = [
     ("getitem", 1),
@@ -210,6 +223,7 @@
     ("keys", 0),
     ("clear", 0),
     ("get_builtin_indexed", 1),
+    ("popitem", 0),
 ]
 
 
@@ -779,16 +793,11 @@
         return w_item
 
 def dict_popitem__DictMulti(space, w_dict):
-    # XXX should somehow use the same trick as CPython: saving the index
-    # of the last popped item in the hash table, so that the next call to
-    # popitem() can be more efficient, instead of always starting from the
-    # beginning of the hash table.
-    iterator = w_dict.iter()
-    w_key, w_value = iterator.next()
-    if w_key is None:
+    try:
+        w_key, w_value = w_dict.popitem()
+    except KeyError:
         raise OperationError(space.w_KeyError,
                              space.wrap("popitem(): dictionary is empty"))
-    w_dict.delitem(w_key)
     return space.newtuple([w_key, w_value])
 
 app = gateway.applevel('''

Modified: pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_dictmultiobject.py	Sat Oct 30 12:18:10 2010
@@ -238,7 +238,16 @@
         assert len(d) == 0
         assert (it!=it1) and (it1==(1,2) or it1==(3,4))
         raises(KeyError, d.popitem)
-    
+
+    def test_popitem_2(self):
+        class A(object):
+            pass
+        d = A().__dict__
+        d['x'] = 5
+        it1 = d.popitem()
+        assert it1 == ('x', 5)
+        raises(KeyError, d.popitem)
+
     def test_setdefault(self):
         d = {1:2, 3:4}
         dd = d.copy()

Modified: pypy/trunk/pypy/rlib/objectmodel.py
==============================================================================
--- pypy/trunk/pypy/rlib/objectmodel.py	(original)
+++ pypy/trunk/pypy/rlib/objectmodel.py	Sat Oct 30 12:18:10 2010
@@ -475,6 +475,10 @@
     def setdefault(self, key, default):
         return self._dict.setdefault(_r_dictkey(self, key), default)
 
+    def popitem(self):
+        dk, value = self._dict.popitem()
+        return dk.key, value
+
     def copy(self):
         result = r_dict(self.key_eq, self.key_hash)
         result.update(self)

Modified: pypy/trunk/pypy/rpython/lltypesystem/rdict.py
==============================================================================
--- pypy/trunk/pypy/rpython/lltypesystem/rdict.py	(original)
+++ pypy/trunk/pypy/rpython/lltypesystem/rdict.py	Sat Oct 30 12:18:10 2010
@@ -306,6 +306,13 @@
         hop.exception_cannot_occur()
         return hop.gendirectcall(ll_clear, v_dict)
 
+    def rtype_method_popitem(self, hop):
+        v_dict, = hop.inputargs(self)
+        r_tuple = hop.r_result
+        cTUPLE = hop.inputconst(lltype.Void, r_tuple.lowleveltype)
+        hop.exception_is_here()
+        return hop.gendirectcall(ll_popitem, cTUPLE, v_dict)
+
 class __extend__(pairtype(DictRepr, rmodel.Repr)): 
 
     def rtype_getitem((r_dict, r_key), hop):
@@ -465,6 +472,10 @@
     i = ll_dict_lookup(d, key, d.keyhash(key))
     if not d.entries.valid(i):
         raise KeyError
+    _ll_dict_del(d, i)
+ll_dict_delitem.oopspec = 'dict.delitem(d, key)'
+
+def _ll_dict_del(d, i):
     d.entries.mark_deleted(i)
     d.num_items -= 1
     # clear the key and the value if they are GC pointers
@@ -481,7 +492,6 @@
     num_entries = len(d.entries)
     if num_entries > DICT_INITSIZE and d.num_items < num_entries / 4:
         ll_dict_resize(d)
-ll_dict_delitem.oopspec = 'dict.delitem(d, key)'
 
 def ll_dict_resize(d):
     old_entries = d.entries
@@ -810,3 +820,26 @@
     i = ll_dict_lookup(d, key, d.keyhash(key))
     return d.entries.valid(i)
 ll_contains.oopspec = 'dict.contains(d, key)'
+
+POPITEMINDEX = lltype.Struct('PopItemIndex', ('nextindex', lltype.Signed))
+global_popitem_index = lltype.malloc(POPITEMINDEX, zero=True, immortal=True)
+
+def ll_popitem(ELEM, dic):
+    entries = dic.entries
+    dmask = len(entries) - 1
+    base = global_popitem_index.nextindex
+    counter = 0
+    while counter <= dmask:
+        i = (base + counter) & dmask
+        counter += 1
+        if entries.valid(i):
+            break
+    else:
+        raise KeyError
+    global_popitem_index.nextindex += counter
+    entry = entries[i]
+    r = lltype.malloc(ELEM.TO)
+    r.item0 = recast(ELEM.TO.item0, entry.key)
+    r.item1 = recast(ELEM.TO.item1, entry.value)
+    _ll_dict_del(dic, i)
+    return r

Modified: pypy/trunk/pypy/rpython/test/test_rdict.py
==============================================================================
--- pypy/trunk/pypy/rpython/test/test_rdict.py	(original)
+++ pypy/trunk/pypy/rpython/test/test_rdict.py	Sat Oct 30 12:18:10 2010
@@ -682,6 +682,26 @@
         # if it does not crash, we are fine. It crashes if you forget the hash field.
         self.interpret(func, [])
 
+    def test_dict_popitem(self):
+        def func():
+            d = {}
+            d[5] = 2
+            d[6] = 3
+            k1, v1 = d.popitem()
+            assert len(d) == 1
+            k2, v2 = d.popitem()
+            try:
+                d.popitem()
+            except KeyError:
+                pass
+            else:
+                assert 0, "should have raised KeyError"
+            assert len(d) == 0
+            return k1*1000 + v1*100 + k2*10 + v2
+
+        res = self.interpret(func, [])
+        assert res in [5263, 6352]
+
     # ____________________________________________________________
 
     def test_opt_nullkeymarker(self):


From arigo at codespeak.net  Sat Oct 30 12:22:11 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 12:22:11 +0200 (CEST)
Subject: [pypy-svn] r78508 - pypy/trunk/pypy/objspace/std/test
Message-ID: <20101030102211.0EF07282BEA@codespeak.net>

Author: arigo
Date: Sat Oct 30 12:22:05 2010
New Revision: 78508

Modified:
   pypy/trunk/pypy/objspace/std/test/test_strutil.py
Log:
Fix the tests.


Modified: pypy/trunk/pypy/objspace/std/test/test_strutil.py
==============================================================================
--- pypy/trunk/pypy/objspace/std/test/test_strutil.py	(original)
+++ pypy/trunk/pypy/objspace/std/test/test_strutil.py	Sat Oct 30 12:22:05 2010
@@ -25,7 +25,7 @@
                  ]
         for s, expected in cases:
             assert string_to_int(s) == expected
-            assert string_to_w_long(space, s).longval() == expected
+            assert string_to_bigint(s).tolong() == expected
 
     def test_string_to_int_base(self):
         space = self.space        
@@ -119,17 +119,16 @@
             raises(ParseStringError, string_to_int, '+'+s, base)
             raises(ParseStringError, string_to_int, '-'+s, base)
 
-    def test_string_to_w_long(self):
-        space = self.space
-        assert string_to_w_long(space, '123L').longval() == 123
-        assert string_to_w_long(space, '123L  ').longval() == 123
-        raises(ParseStringError, string_to_w_long, space, 'L')
-        raises(ParseStringError, string_to_w_long, space, 'L  ')
-        assert string_to_w_long(space, '123L', 4).longval() == 27
-        assert string_to_w_long(space, '123L', 30).longval() == 27000 + 1800 + 90 + 21
-        assert string_to_w_long(space, '123L', 22).longval() == 10648 + 968 + 66 + 21
-        assert string_to_w_long(space, '123L', 21).longval() == 441 + 42 + 3
-        assert string_to_w_long(space, '1891234174197319').longval() == 1891234174197319
+    def test_string_to_bigint(self):
+        assert string_to_bigint('123L').tolong() == 123
+        assert string_to_bigint('123L  ').tolong() == 123
+        raises(ParseStringError, string_to_bigint, 'L')
+        raises(ParseStringError, string_to_bigint, 'L  ')
+        assert string_to_bigint('123L', 4).tolong() == 27
+        assert string_to_bigint('123L', 30).tolong() == 27000 + 1800 + 90 + 21
+        assert string_to_bigint('123L', 22).tolong() == 10648 + 968 + 66 + 21
+        assert string_to_bigint('123L', 21).tolong() == 441 + 42 + 3
+        assert string_to_bigint('1891234174197319').tolong() == 1891234174197319
 
     def test_string_to_float(self):
         def string_to_float(x):


From hakanardo at codespeak.net  Sat Oct 30 12:30:37 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 12:30:37 +0200 (CEST)
Subject: [pypy-svn] r78509 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101030103037.69EF8282BEB@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 12:30:35 2010
New Revision: 78509

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(hakanardo, cfbolz) force lazy setfileds at end of preamble

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py	Sat Oct 30 12:30:35 2010
@@ -148,6 +148,10 @@
         self.clean_caches()
 
 
+    def force_at_end_of_preamble(self):
+        self.force_all_lazy_setfields()
+        self.next_optimization.force_at_end_of_preamble()
+        
     def force_lazy_setfield(self, descr, before_guard=False):
         try:
             op = self.lazy_setfields[descr]

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	Sat Oct 30 12:30:35 2010
@@ -150,6 +150,8 @@
 oohelper.CVAL_NULLREF = ConstantValue(oohelper.CONST_NULL)
 
 class Optimization(object):
+    next_optimization = None
+    
     def propagate_forward(self, op):
         raise NotImplementedError
 
@@ -191,6 +193,10 @@
     def setup(self, virtuals):
         pass
 
+    def force_at_end_of_preamble(self):
+        if self.next_optimization:
+            self.next_optimization.force_at_end_of_preamble()
+
 class Optimizer(Optimization):
 
     def __init__(self, metainterp_sd, loop, optimizations=None, virtuals=True):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Sat Oct 30 12:30:35 2010
@@ -18,8 +18,9 @@
         if not self.enabled:
             self.emit_operation(op)
             return
-        
+
         if op.getopnum() == rop.JUMP:
+            self.force_at_end_of_preamble()
             loop = self.optimizer.loop
             loop.preamble.operations = self.optimizer.newoperations
             self.optimizer.newoperations = []
@@ -93,10 +94,13 @@
                 if not isinstance(a, Const) and a in self.optimizer.values:
                     v = self.getvalue(a)
                     if v.fromstart and not v.is_constant():
-                        a = v.force_box()
-                        if a not in inputargs:
-                            inputargs.append(a)
-                            newval = self.getvalue(argmap[a])
+                        b = v.force_box()
+                        if b not in inputargs:
+                        #boxes = []
+                        #v.enum_forced_boxes(boxes, seen_inputargs)
+                        #for b in boxes:
+                            inputargs.append(b)
+                            newval = self.getvalue(argmap[b])
                             jumpargs.append(newval.force_box())
 
         jmp.initarglist(jumpargs)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 12:30:35 2010
@@ -854,16 +854,20 @@
         p3sub = getfield_gc(p3, descr=nextdescr)
         i3 = getfield_gc(p3sub, descr=valuedescr)
         escape(i3)
-        jump(i1, p2)
-        """
-        expected = """
-        [i1, p3]
         p2sub = new_with_vtable(ConstClass(node_vtable2))
         setfield_gc(p2sub, i1, descr=valuedescr)
-        setfield_gc(p3, p2sub, descr=nextdescr)
-        escape(i1)
-        p4 = new_with_vtable(ConstClass(node_vtable))
-        jump(i1, p4)
+        setfield_gc(p2, p2sub, descr=nextdescr)        
+        jump(i1, p2, p2sub)
+        """
+        expected = """
+        [i1, p2, p2sub]
+        i3 = getfield_gc(p2sub, descr=valuedescr)
+        escape(i3)
+        p1 = new_with_vtable(ConstClass(node_vtable))
+        p3sub = new_with_vtable(ConstClass(node_vtable2))
+        setfield_gc(p3sub, i1, descr=valuedescr)
+        setfield_gc(p1, p3sub, descr=nextdescr)
+        jump(i1, p1, p3sub)
         """
         self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
 


From arigo at codespeak.net  Sat Oct 30 12:33:20 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 12:33:20 +0200 (CEST)
Subject: [pypy-svn] r78510 - pypy/branch/jit-unroll-loops/pypy/jit/metainterp
Message-ID: <20101030103320.A5020282BEC@codespeak.net>

Author: arigo
Date: Sat Oct 30 12:33:19 2010
New Revision: 78510

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py
Log:
Fix the message


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py	Sat Oct 30 12:33:19 2010
@@ -1680,7 +1680,7 @@
                 # raises in case it works -- which is the common case
                 self.compile(original_boxes, live_arg_boxes, start)
                 # creation of the loop was cancelled!
-                self.staticdata.log('cancelled, going on...')
+                self.staticdata.log('cancelled, tracing more...')
 
         # Otherwise, no loop found so far, so continue tracing.
         start = len(self.history.operations)


From cfbolz at codespeak.net  Sat Oct 30 13:01:07 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 13:01:07 +0200 (CEST)
Subject: [pypy-svn] r78511 - pypy/trunk/pypy/jit/metainterp/test
Message-ID: <20101030110107.447D2282BD4@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 13:01:05 2010
New Revision: 78511

Modified:
   pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
make this test test a bit more


Modified: pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 13:01:05 2010
@@ -807,10 +807,10 @@
         guard_value(i2, 1) []
         i3 = call_loopinvariant(1, i1, descr=nonwritedescr)
         guard_no_exception() []
-        guard_value(i2, 1) []
+        guard_value(i3, 1) []
         i4 = call_loopinvariant(1, i1, descr=nonwritedescr)
         guard_no_exception() []
-        guard_value(i2, 1) []
+        guard_value(i4, 1) []
         jump(i1)
         """
         expected = """


From cfbolz at codespeak.net  Sat Oct 30 13:15:25 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 13:15:25 +0200 (CEST)
Subject: [pypy-svn] r78512 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101030111525.A7683282BD4@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 13:15:24 2010
New Revision: 78512

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
Log:
 - only use unroll optimization if we are not in a bridge
 - rename the confusing flag "virtuals" to "not_a_bridge"



Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	Sat Oct 30 13:15:24 2010
@@ -6,20 +6,21 @@
 from pypy.jit.metainterp.optimizeopt.string import OptString
 from pypy.jit.metainterp.optimizeopt.unroll import OptUnroll
 
-def optimize_loop_1(metainterp_sd, loop, virtuals=True):
+def optimize_loop_1(metainterp_sd, loop, not_a_bridge=True):
     """Optimize loop.operations to make it match the input of loop.specnodes
     and to remove internal overheadish operations.  Note that loop.specnodes
     must be applicable to the loop; you will probably get an AssertionError
     if not.
     """
-    optimizations = [OptUnroll(),
-                     OptIntBounds(),
+    optimizations = [OptIntBounds(),
                      OptRewrite(),
                      OptVirtualize(),
                      OptString(),
                      OptHeap(),
                     ]
-    optimizer = Optimizer(metainterp_sd, loop, optimizations, virtuals)
+    if not_a_bridge:
+        optimizations.insert(0, OptUnroll())
+    optimizer = Optimizer(metainterp_sd, loop, optimizations, not_a_bridge)
     optimizer.propagate_all_forward()
 
 def optimize_bridge_1(metainterp_sd, bridge):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	Sat Oct 30 13:15:24 2010
@@ -190,7 +190,7 @@
         op = ResOperation(opnum, args, result)
         self.optimizer.pure_operations[self.optimizer.make_args_key(op)] = op
 
-    def setup(self, virtuals):
+    def setup(self, not_a_bridge):
         pass
 
     def force_at_end_of_preamble(self):
@@ -199,7 +199,7 @@
 
 class Optimizer(Optimization):
 
-    def __init__(self, metainterp_sd, loop, optimizations=None, virtuals=True):
+    def __init__(self, metainterp_sd, loop, optimizations=None, not_a_bridge=True):
         self.metainterp_sd = metainterp_sd
         self.cpu = metainterp_sd.cpu
         self.loop = loop
@@ -220,7 +220,7 @@
             optimizations[-1].next_optimization = self
             for o in optimizations:
                 o.optimizer = self
-                o.setup(virtuals)
+                o.setup(not_a_bridge)
         else:
             self.first_optimization = self
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Sat Oct 30 13:15:24 2010
@@ -278,8 +278,8 @@
 class OptVirtualize(Optimization):
     "Virtualize objects until they escape."
 
-    def setup(self, virtuals):
-        if not virtuals:
+    def setup(self, not_a_bridge):
+        if not not_a_bridge:
             return
 
         inputargs = self.optimizer.loop.inputargs


From cfbolz at codespeak.net  Sat Oct 30 13:18:32 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 13:18:32 +0200 (CEST)
Subject: [pypy-svn] r78513 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101030111832.711A236E0C1@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 13:18:30 2010
New Revision: 78513

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
Log:
this is no longer necessary now


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Sat Oct 30 13:18:30 2010
@@ -8,16 +8,13 @@
     become the preamble or entry bridge (don't think there is a
     distinction anymore)"""
     
-    def setup(self, virtuals):
-        self.enabled = virtuals        
+    def setup(self, not_a_bridge):
+        assert not_a_bridge
         self.cloned_operations = []
         for op in self.optimizer.loop.operations:
             self.cloned_operations.append(op.clone())
             
     def propagate_forward(self, op):
-        if not self.enabled:
-            self.emit_operation(op)
-            return
 
         if op.getopnum() == rop.JUMP:
             self.force_at_end_of_preamble()


From arigo at codespeak.net  Sat Oct 30 13:20:40 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 13:20:40 +0200 (CEST)
Subject: [pypy-svn] r78514 - pypy/trunk/lib-python/modified-2.5.2/test
Message-ID: <20101030112040.A395D282B90@codespeak.net>

Author: arigo
Date: Sat Oct 30 13:20:39 2010
New Revision: 78514

Added:
   pypy/trunk/lib-python/modified-2.5.2/test/test_dict.py
      - copied, changed from r78508, pypy/trunk/lib-python/2.5.2/test/test_dict.py
Log:
Implementation detail.


Copied: pypy/trunk/lib-python/modified-2.5.2/test/test_dict.py (from r78508, pypy/trunk/lib-python/2.5.2/test/test_dict.py)
==============================================================================
--- pypy/trunk/lib-python/2.5.2/test/test_dict.py	(original)
+++ pypy/trunk/lib-python/modified-2.5.2/test/test_dict.py	Sat Oct 30 13:20:39 2010
@@ -300,7 +300,8 @@
                     self.assertEqual(va, int(ka))
                     kb, vb = tb = b.popitem()
                     self.assertEqual(vb, int(kb))
-                    self.assert_(not(copymode < 0 and ta != tb))
+                    if test_support.check_impl_detail():
+                        self.assert_(not(copymode < 0 and ta != tb))
                 self.assert_(not a)
                 self.assert_(not b)
 


From fijal at codespeak.net  Sat Oct 30 13:31:16 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Sat, 30 Oct 2010 13:31:16 +0200 (CEST)
Subject: [pypy-svn] r78515 - pypy/trunk/pypy/objspace
Message-ID: <20101030113116.E89EB282BAD@codespeak.net>

Author: fijal
Date: Sat Oct 30 13:31:15 2010
New Revision: 78515

Modified:
   pypy/trunk/pypy/objspace/descroperation.py
Log:
Make everything newstyle in descroperation


Modified: pypy/trunk/pypy/objspace/descroperation.py
==============================================================================
--- pypy/trunk/pypy/objspace/descroperation.py	(original)
+++ pypy/trunk/pypy/objspace/descroperation.py	Sat Oct 30 13:31:15 2010
@@ -51,7 +51,7 @@
     return space.is_w(w_typ1, w_typ2)
 
 
-class Object:
+class Object(object):
     def descr__getattribute__(space, w_obj, w_name):
         name = space.str_w(w_name)
         w_descr = space.lookup(w_obj, name)
@@ -96,7 +96,7 @@
     def descr__init__(space, w_obj, __args__):
         pass
 
-class DescrOperation:
+class DescrOperation(object):
     _mixin_ = True
 
     def is_data_descr(space, w_obj):
@@ -541,7 +541,7 @@
 
 # what is the maximum value slices can get on CPython?
 # we need to stick to that value, because fake.py etc.
-class Temp:
+class Temp(object):
     def __getslice__(self, i, j):
         return j
 slice_max = Temp()[:]


From fijal at codespeak.net  Sat Oct 30 13:32:51 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Sat, 30 Oct 2010 13:32:51 +0200 (CEST)
Subject: [pypy-svn] r78516 - pypy/trunk/pypy/interpreter
Message-ID: <20101030113251.62367282BD4@codespeak.net>

Author: fijal
Date: Sat Oct 30 13:32:49 2010
New Revision: 78516

Modified:
   pypy/trunk/pypy/interpreter/gateway.py
Log:
MAke those newstyle as well


Modified: pypy/trunk/pypy/interpreter/gateway.py
==============================================================================
--- pypy/trunk/pypy/interpreter/gateway.py	(original)
+++ pypy/trunk/pypy/interpreter/gateway.py	Sat Oct 30 13:32:49 2010
@@ -28,7 +28,7 @@
 # internal non-translatable parts: 
 import py
 
-class SignatureBuilder:
+class SignatureBuilder(object):
     "NOT_RPYTHON"
     def __init__(self, func=None, argnames=None, varargname=None,
                  kwargname=None, name = None):
@@ -51,7 +51,7 @@
 
 #________________________________________________________________
 
-class UnwrapSpecRecipe:
+class UnwrapSpecRecipe(object):
     "NOT_RPYTHON"
 
     bases_order = [Wrappable, W_Root, ObjSpace, Arguments, object]


From fijal at codespeak.net  Sat Oct 30 13:35:43 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Sat, 30 Oct 2010 13:35:43 +0200 (CEST)
Subject: [pypy-svn] r78517 - pypy/trunk/pypy/annotation
Message-ID: <20101030113543.5F70F36E0CB@codespeak.net>

Author: fijal
Date: Sat Oct 30 13:35:41 2010
New Revision: 78517

Modified:
   pypy/trunk/pypy/annotation/bookkeeper.py
   pypy/trunk/pypy/annotation/classdef.py
   pypy/trunk/pypy/annotation/description.py
   pypy/trunk/pypy/annotation/dictdef.py
   pypy/trunk/pypy/annotation/listdef.py
   pypy/trunk/pypy/annotation/specialize.py
Log:
sprinkle (object) around annotation as well


Modified: pypy/trunk/pypy/annotation/bookkeeper.py
==============================================================================
--- pypy/trunk/pypy/annotation/bookkeeper.py	(original)
+++ pypy/trunk/pypy/annotation/bookkeeper.py	Sat Oct 30 13:35:41 2010
@@ -24,7 +24,7 @@
 from pypy.rpython import extregistry
 from pypy.tool.identity_dict import identity_dict
 
-class Stats:
+class Stats(object):
 
     def __init__(self, bookkeeper):
         self.bookkeeper = bookkeeper
@@ -137,7 +137,7 @@
     def consider_dict_delitem(self, dic):
         return dic
 
-class Bookkeeper:
+class Bookkeeper(object):
     """The log of choices that have been made while analysing the operations.
     It ensures that the same 'choice objects' will be returned if we ask
     again during reflowing.  Like ExecutionContext, there is an implicit
@@ -736,7 +736,7 @@
         return True
 
 # for parsing call arguments
-class RPythonCallsSpace:
+class RPythonCallsSpace(object):
     """Pseudo Object Space providing almost no real operation.
     For the Arguments class: if it really needs other operations, it means
     that the call pattern is too complex for R-Python.

Modified: pypy/trunk/pypy/annotation/classdef.py
==============================================================================
--- pypy/trunk/pypy/annotation/classdef.py	(original)
+++ pypy/trunk/pypy/annotation/classdef.py	Sat Oct 30 13:35:41 2010
@@ -58,7 +58,7 @@
 #        same name in all subclasses of A, if any.  (Parent class attributes can
 #        be visible in reads from instances of subclasses.)
 
-class Attribute:
+class Attribute(object):
     # readonly-ness
     # SomeThing-ness
     # NB.  an attribute is readonly if it is a constant class attribute.
@@ -402,7 +402,7 @@
 
 # ____________________________________________________________
 
-class InstanceSource:
+class InstanceSource(object):
     instance_level = True
 
     def __init__(self, bookkeeper, obj):

Modified: pypy/trunk/pypy/annotation/description.py
==============================================================================
--- pypy/trunk/pypy/annotation/description.py	(original)
+++ pypy/trunk/pypy/annotation/description.py	Sat Oct 30 13:35:41 2010
@@ -6,7 +6,7 @@
 from pypy.tool.sourcetools import valid_identifier
 from pypy.tool.pairtype import extendabletype
 
-class CallFamily:
+class CallFamily(object):
     """A family of Desc objects that could be called from common call sites.
     The call families are conceptually a partition of all (callable) Desc
     objects, where the equivalence relation is the transitive closure of
@@ -51,7 +51,7 @@
             self.total_calltable_size += 1
 
 
-class FrozenAttrFamily:
+class FrozenAttrFamily(object):
     """A family of FrozenDesc objects that have any common 'getattr' sites.
     The attr families are conceptually a partition of FrozenDesc objects,
     where the equivalence relation is the transitive closure of:
@@ -80,7 +80,7 @@
         self.attrs[attrname] = s_value
 
 
-class ClassAttrFamily:
+class ClassAttrFamily(object):
     """A family of ClassDesc objects that have common 'getattr' sites for a
     given attribute name.  The attr families are conceptually a partition
     of ClassDesc objects, where the equivalence relation is the transitive

Modified: pypy/trunk/pypy/annotation/dictdef.py
==============================================================================
--- pypy/trunk/pypy/annotation/dictdef.py	(original)
+++ pypy/trunk/pypy/annotation/dictdef.py	Sat Oct 30 13:35:41 2010
@@ -77,7 +77,7 @@
             dictdef.dictvalue = self
 
 
-class DictDef:
+class DictDef(object):
     """A dict definition remembers how general the keys and values in that
     particular dict have to be.  Every dict creation makes a new DictDef,
     and the union of two dicts merges the DictKeys and DictValues that each

Modified: pypy/trunk/pypy/annotation/listdef.py
==============================================================================
--- pypy/trunk/pypy/annotation/listdef.py	(original)
+++ pypy/trunk/pypy/annotation/listdef.py	Sat Oct 30 13:35:41 2010
@@ -6,7 +6,7 @@
 class TooLateForChange(Exception):
     pass
 
-class ListItem:
+class ListItem(object):
     mutated = False    # True for lists mutated after creation
     resized = False    # True for lists resized after creation
     range_step = None  # the step -- only for lists only created by a range()
@@ -117,7 +117,7 @@
         return updated
 
 
-class ListDef:
+class ListDef(object):
     """A list definition remembers how general the items in that particular
     list have to be.  Every list creation makes a new ListDef, and the union
     of two lists merges the ListItems that each ListDef stores."""

Modified: pypy/trunk/pypy/annotation/specialize.py
==============================================================================
--- pypy/trunk/pypy/annotation/specialize.py	(original)
+++ pypy/trunk/pypy/annotation/specialize.py	Sat Oct 30 13:35:41 2010
@@ -100,7 +100,7 @@
 # ____________________________________________________________________________
 # specializations
 
-class MemoTable:
+class MemoTable(object):
     def __init__(self, funcdesc, args, value):
         self.funcdesc = funcdesc
         self.table = {args: value}


From fijal at codespeak.net  Sat Oct 30 13:36:00 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Sat, 30 Oct 2010 13:36:00 +0200 (CEST)
Subject: [pypy-svn] r78518 - pypy/branch/faster-compilation
Message-ID: <20101030113600.708BC282BDB@codespeak.net>

Author: fijal
Date: Sat Oct 30 13:35:59 2010
New Revision: 78518

Removed:
   pypy/branch/faster-compilation/
Log:
Everything worth noting has been re-applied to trunk



From cfbolz at codespeak.net  Sat Oct 30 13:43:27 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 13:43:27 +0200 (CEST)
Subject: [pypy-svn] r78519 -
	pypy/branch/mapdict-without-jit/pypy/objspace/std
Message-ID: <20101030114327.63AE6282BEB@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 13:43:26 2010
New Revision: 78519

Modified:
   pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
Log:
cannot use jit.purefunction here, because the argument is a tuple. hack
differently.


Modified: pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	Sat Oct 30 13:43:26 2010
@@ -41,9 +41,9 @@
     def delete(self, obj, selector):
         return None
 
-    @jit.purefunction
     def index(self, selector):
-        if self.space.config.objspace.std.withmethodcache:
+        if (self.space.config.objspace.std.withmethodcache and 
+                not jit.we_are_jitted()):
             return self._index_cache(selector)
         else:
             return self._index(selector)


From cfbolz at codespeak.net  Sat Oct 30 13:45:09 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 13:45:09 +0200 (CEST)
Subject: [pypy-svn] r78520 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101030114509.36BFE282BEF@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 13:45:07 2010
New Revision: 78520

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
Log:
import * is evil


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Sat Oct 30 13:45:07 2010
@@ -1,18 +1,20 @@
+from pypy.jit.metainterp.history import Const, ConstInt
 from pypy.jit.metainterp.specnode import SpecNode, NotSpecNode, ConstantSpecNode
 from pypy.jit.metainterp.specnode import AbstractVirtualStructSpecNode
 from pypy.jit.metainterp.specnode import VirtualInstanceSpecNode
 from pypy.jit.metainterp.specnode import VirtualArraySpecNode
 from pypy.jit.metainterp.specnode import VirtualStructSpecNode
 from pypy.jit.metainterp.resoperation import rop, ResOperation
-from pypy.jit.metainterp.optimizeutil import _findall
+from pypy.jit.metainterp.optimizeutil import _findall, sort_descrs
+from pypy.jit.metainterp.optimizeutil import descrlist_dict
 from pypy.rlib.objectmodel import we_are_translated
-from pypy.jit.metainterp.optimizeopt.optimizer import *
+from pypy.jit.metainterp.optimizeopt import optimizer
 
 
-class AbstractVirtualValue(OptValue):
+class AbstractVirtualValue(optimizer.OptValue):
     _attrs_ = ('optimizer', 'keybox', 'source_op', '_cached_vinfo')
     box = None
-    level = LEVEL_NONNULL
+    level = optimizer.LEVEL_NONNULL
     _cached_vinfo = None
 
     def __init__(self, optimizer, keybox, source_op=None):
@@ -67,7 +69,7 @@
         return self._fields.get(ofs, default)
 
     def setfield(self, ofs, fieldvalue):
-        assert isinstance(fieldvalue, OptValue)
+        assert isinstance(fieldvalue, optimizer.OptValue)
         self._fields[ofs] = fieldvalue
 
     def _really_force(self):
@@ -135,7 +137,7 @@
 
 
 class VirtualValue(AbstractVirtualStructValue):
-    level = LEVEL_KNOWNCLASS
+    level = optimizer.LEVEL_KNOWNCLASS
 
     def __init__(self, optimizer, known_class, keybox, source_op=None):
         AbstractVirtualStructValue.__init__(self, optimizer, keybox, source_op)
@@ -172,7 +174,7 @@
         return res
 
     def setitem(self, index, itemvalue):
-        assert isinstance(itemvalue, OptValue)
+        assert isinstance(itemvalue, optimizer.OptValue)
         self._items[index] = itemvalue
 
     def _really_force(self):
@@ -275,7 +277,7 @@
             subspecnode = self.items[index]
             subspecnode.teardown_virtual_node(optimizer, subvalue, newexitargs)
 
-class OptVirtualize(Optimization):
+class OptVirtualize(optimizer.Optimization):
     "Virtualize objects until they escape."
 
     def setup(self, not_a_bridge):


From arigo at codespeak.net  Sat Oct 30 13:48:46 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 13:48:46 +0200 (CEST)
Subject: [pypy-svn] r78521 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030114846.279F7282BF0@codespeak.net>

Author: arigo
Date: Sat Oct 30 13:48:44 2010
New Revision: 78521

Added:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
      - copied, changed from r77505, pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
A copy of test_optimizeopt from trunk, which makes sense
to keep running without the OptUnroll optimization.


Copied: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py (from r77505, pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py)
==============================================================================
--- pypy/trunk/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	Sat Oct 30 13:48:44 2010
@@ -6,7 +6,6 @@
 from pypy.jit.metainterp.optimizefindnode import PerfectSpecializationFinder
 import pypy.jit.metainterp.optimizeopt.optimizer as optimizeopt
 import pypy.jit.metainterp.optimizeopt.virtualize as virtualize
-from pypy.jit.metainterp.optimizeopt import optimize_loop_1
 from pypy.jit.metainterp.optimizeutil import InvalidLoop
 from pypy.jit.metainterp.history import AbstractDescr, ConstInt, BoxInt
 from pypy.jit.metainterp.jitprof import EmptyProfiler
@@ -219,7 +218,7 @@
     _kind2count = {history.INT: 1, history.REF: 2, history.FLOAT: 3}
     return sorted(boxes, key=lambda box: _kind2count[box.type])
 
-class BaseTestOptimizeOpt(BaseTest):
+class BaseTestOptimizeBasic(BaseTest):
 
     def invent_fail_descr(self, fail_args):
         if fail_args is None:
@@ -240,24 +239,29 @@
 
     def optimize_loop(self, ops, spectext, optops, checkspecnodes=True):
         loop = self.parse(ops)
-        #
-        if checkspecnodes:
-            # verify that 'spectext' is indeed what optimizefindnode would
-            # compute for this loop
-            cpu = self.cpu
-            perfect_specialization_finder = PerfectSpecializationFinder(cpu)
-            perfect_specialization_finder.find_nodes_loop(loop)
-            self.check_specnodes(loop.token.specnodes, spectext)
-        else:
-            # for cases where we want to see how optimizeopt behaves with
-            # combinations different from the one computed by optimizefindnode
-            loop.token.specnodes = self.unpack_specnodes(spectext)
+        loop.token.specnodes = self.unpack_specnodes(spectext)
         #
         self.loop = loop
         metainterp_sd = FakeMetaInterpStaticData(self.cpu)
         if hasattr(self, 'vrefinfo'):
             metainterp_sd.virtualref_info = self.vrefinfo
-        optimize_loop_1(metainterp_sd, loop)
+        #
+        # XXX list the exact optimizations that are needed for each test
+        from pypy.jit.metainterp.optimizeopt import (OptIntBounds,
+                                                     OptRewrite,
+                                                     OptVirtualize,
+                                                     OptString,
+                                                     OptHeap,
+                                                     Optimizer)
+        optimizations = [OptIntBounds(),
+                         OptRewrite(),
+                         OptVirtualize(),
+                         OptString(),
+                         OptHeap(),
+                         ]
+        optimizer = Optimizer(metainterp_sd, loop, optimizations,
+                              virtuals=True)
+        optimizer.propagate_all_forward()
         #
         expected = self.parse(optops)
         print '\n'.join([str(o) for o in loop.operations])
@@ -490,6 +494,7 @@
         []
         jump()
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Constant(myptr)', expected)
 
     def test_ooisnull_oononnull_1(self):
@@ -752,10 +757,10 @@
         p3sub = getfield_gc(p3, descr=nextdescr)
         i3 = getfield_gc(p3sub, descr=valuedescr)
         escape(i3)
-        p1 = new_with_vtable(ConstClass(node_vtable))
         p2sub = new_with_vtable(ConstClass(node_vtable2))
         setfield_gc(p2sub, i1, descr=valuedescr)
         setfield_gc(p2, p2sub, descr=nextdescr)
+        p1 = new_with_vtable(ConstClass(node_vtable))
         jump(i1, p1, p2)
         """
         # The same as test_p123_simple, but in the end the "old" p2 contains
@@ -827,8 +832,9 @@
         i1 = int_add(i2, i)
         jump(i, i1)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Not, Virtual(node_vtable, valuedescr=Not)',
-                           expected, checkspecnodes=False)
+                           expected)
 
     def test_virtual_float(self):
         ops = """
@@ -843,6 +849,7 @@
         f1 = float_add(f2, f)
         jump(f, f1)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Not, Virtual(node_vtable, floatdescr=Not)',
                            expected, checkspecnodes=False)
 
@@ -860,6 +867,7 @@
         i1 = int_add(i2, i)
         jump(i, i1)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Not, Virtual(node_vtable, valuedescr=Not)',
                            expected)
 
@@ -894,6 +902,7 @@
         # all constant-folded :-)
         jump(p2)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, '''Virtual(node_vtable),
                                    Virtual(node_vtable),
                                    Not''',
@@ -931,6 +940,7 @@
         """
         # the 'expected' is sub-optimal, but it should be done by another later
         # optimization step.  See test_find_nodes_default_field() for why.
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Virtual(node_vtable, valuedescr=Not)',
                            expected)
 
@@ -967,6 +977,7 @@
         i3 = int_add(i0, i1)
         jump(i3, i2)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Not, Virtual(node_vtable, valuedescr=Not)',
                            expected)
 
@@ -990,6 +1001,7 @@
         i3 = int_add(i0, i1)
         jump(i3, i2, i1)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops,
             '''Not, Virtual(node_vtable,
                             valuedescr=Not,
@@ -1241,6 +1253,7 @@
         guard_value(i3, 15) []
         jump(i0, 20, i0)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Not, VArray(arraydescr, Not, Not)', expected)
 
     def test_p123_array(self):
@@ -1291,6 +1304,7 @@
         escape(i2)
         jump(i1, i1)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Not, VStruct(ssize, adescr=Not)', expected)
 
     def test_p123_vstruct(self):
@@ -1472,6 +1486,7 @@
         escape(i1)
         jump()
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Constant(myptr)', expected)
 
     def test_duplicate_getfield_sideeffects_1(self):
@@ -1693,6 +1708,7 @@
         setfield_gc(ConstPtr(myptr), i2, descr=valuedescr)
         jump(i1, i2)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Constant(myptr), Not, Not', expected)
 
     def test_duplicate_getarrayitem_1(self):
@@ -1847,6 +1863,7 @@
         p3 = escape()
         jump(i0, p3)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Not, Virtual(node_vtable, nextdescr=Not)',
                            expected)
 
@@ -1870,6 +1887,7 @@
         p3 = escape()
         jump(i0, p3)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Not, VArray(arraydescr2, Not)',
                            expected)
 
@@ -1903,6 +1921,7 @@
         p2a = new_with_vtable(ConstClass(node_vtable))
         jump(p2a, p3a)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Virtual(node_vtable2, nextdescr=Not, otherdescr=Not)', expected)
 
     def test_bug_3bis(self):
@@ -1935,6 +1954,7 @@
         escape(p3a)
         jump(p2a, p3a)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, 'Virtual(node_vtable2, nextdescr=Not, otherdescr=Not)', expected)
 
     def test_invalid_loop_1(self):
@@ -1945,6 +1965,7 @@
         p2 = new_with_vtable(ConstClass(node_vtable))
         jump(p2)
         """
+        py.test.skip("XXX")
         py.test.raises(InvalidLoop, self.optimize_loop,
                        ops, 'Virtual(node_vtable)', None)
 
@@ -1972,6 +1993,7 @@
         setfield_gc(p3, p4, descr=nextdescr)
         jump(p3)
         """
+        py.test.skip("XXX")
         py.test.raises(InvalidLoop, self.optimize_loop, ops,
                        'Virtual(node_vtable, nextdescr=Virtual(node_vtable))',
                        None)
@@ -2501,6 +2523,7 @@
         guard_true(i0, descr=fdescr) [i1b]
         jump(i1, i1, i1)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, '''Virtual(node_vtable, valuedescr=Not),
                                    Not, Not''', expected)
         self.check_expanded_fail_descr('''p0
@@ -2575,6 +2598,7 @@
         guard_true(i1, descr=fdescr) [ia, iv, i2]
         jump(1, 1, i2, NULL, i2)
         """
+        py.test.skip("XXX")
         self.optimize_loop(ops, '''
             Not,
             VArray(arraydescr2,
@@ -2643,7 +2667,7 @@
             ''', rop.GUARD_TRUE)
 
 
-class TestLLtype(BaseTestOptimizeOpt, LLtypeMixin):
+class TestLLtype(BaseTestOptimizeBasic, LLtypeMixin):
 
     def test_residual_call_does_not_invalidate_caches(self):
         ops = """
@@ -4495,7 +4519,7 @@
     # XXX str2unicode
 
 
-##class TestOOtype(BaseTestOptimizeOpt, OOtypeMixin):
+##class TestOOtype(BaseTestOptimizeBasic, OOtypeMixin):
 
 ##    def test_instanceof(self):
 ##        ops = """


From arigo at codespeak.net  Sat Oct 30 13:54:45 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 13:54:45 +0200 (CEST)
Subject: [pypy-svn] r78522 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101030115445.9C7A8282BAD@codespeak.net>

Author: arigo
Date: Sat Oct 30 13:54:44 2010
New Revision: 78522

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py
Log:
For now, kill {get,set}field_raw support.  If we want it we need to be more careful.
For example, a signal handler could have changed the field -- so we would need some
'volatile' declarations like C.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py	Sat Oct 30 13:54:44 2010
@@ -107,7 +107,6 @@
             return
         opnum = op.getopnum()
         if (opnum == rop.SETFIELD_GC or
-            opnum == rop.SETFIELD_RAW or
             opnum == rop.SETARRAYITEM_GC or
             opnum == rop.DEBUG_MERGE_POINT):
             return
@@ -241,10 +240,6 @@
         # remember the result of future reads of the field
         self.cache_field_value(op.getdescr(), value, fieldvalue, write=True)
 
-    optimize_GETFIELD_RAW = optimize_GETFIELD_GC
-    optimize_SETFIELD_RAW = optimize_SETFIELD_GC
-    # FIXME: Do we need separate caches for raw and gc?
-
     def optimize_GETARRAYITEM_GC(self, op):
         value = self.getvalue(op.getarg(0))
         indexvalue = self.getvalue(op.getarg(1))


From david at codespeak.net  Sat Oct 30 13:54:56 2010
From: david at codespeak.net (david at codespeak.net)
Date: Sat, 30 Oct 2010 13:54:56 +0200 (CEST)
Subject: [pypy-svn] r78523 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101030115456.6D6CD282BF0@codespeak.net>

Author: david
Date: Sat Oct 30 13:54:54 2010
New Revision: 78523

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
Log:
Implement int_and, int_or, int_xor operations

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Sat Oct 30 13:54:54 2010
@@ -169,7 +169,7 @@
     # cpu interface
     def assemble_loop(self, inputargs, operations, looptoken):
         longevity = compute_vars_longevity(inputargs, operations)
-        regalloc = ARMRegisterManager(longevity, assembler=self.mc, frame_manager=ARMFrameManager())
+        regalloc = ARMRegisterManager(longevity, assembler=self, frame_manager=ARMFrameManager())
         self.align()
         loop_start=self.mc.curraddr()
         self.gen_func_prolog()
@@ -190,7 +190,7 @@
     def assemble_bridge(self, faildescr, inputargs, operations):
         enc = rffi.cast(rffi.CCHARP, faildescr._failure_recovery_code)
         longevity = compute_vars_longevity(inputargs, operations)
-        regalloc = ARMRegisterManager(longevity, assembler=self.mc, frame_manager=ARMFrameManager())
+        regalloc = ARMRegisterManager(longevity, assembler=self, frame_manager=ARMFrameManager())
 
         regalloc.update_bindings(enc, inputargs)
         bridge_head = self.mc.curraddr()
@@ -218,6 +218,13 @@
         b.gen_load_int(reg.value, bridge_addr, fcond)
         b.MOV_rr(r.pc.value, reg.value, cond=fcond)
 
+    # regalloc support
+    def regalloc_mov(self, prev_loc, loc):
+        if isinstance(prev_loc, ConstInt):
+            # XXX check size of imm for current instr
+            self.mc.gen_load_int(loc.value, prev_loc.getint())
+        else:
+            self.mc.MOV_rr(loc.value, prev_loc.value)
 
 def make_operation_list():
     def notimplemented(self, op, regalloc, fcond):

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Sat Oct 30 13:54:54 2010
@@ -104,13 +104,6 @@
             self.MOV_ri(ip, t, cond=cond)
             self.ORR_rr(r, r, ip, offset, cond=cond)
 
-    # regalloc support
-    def regalloc_mov(self, prev_loc, loc):
-        if isinstance(prev_loc, ConstInt):
-            # XXX check size of imm for current instr
-            self.gen_load_int(loc.value, prev_loc.getint())
-        else:
-            self.MOV_rr(loc.value, prev_loc.value)
 
 class ARMv7InMemoryBuilder(AbstractARMv7Builder):
     def __init__(self, start, end):
@@ -121,7 +114,7 @@
 class ARMv7Builder(AbstractARMv7Builder):
 
     def __init__(self):
-        map_size = 1024
+        map_size = 4096
         data = alloc(map_size)
         self._pos = 0
         self._init(data, map_size)
@@ -140,13 +133,13 @@
     def _add_more_mem(self):
         new_mem = alloc(self._size)
         new_mem_addr = rffi.cast(lltype.Signed, new_mem)
-        self.PUSH([reg.r0.value])
+        self.PUSH([reg.r0.value, reg.ip.value])
         self.gen_load_int(reg.r0.value, new_mem_addr)
         self.MOV_rr(reg.pc.value, reg.r0.value)
         self._dump_trace('data%d.asm' % self.n_data)
         self.n_data+=1
         self._data = new_mem
         self._pos = 0
-        self.LDM(reg.sp.value, [reg.r0.value], w=1) # XXX Replace with POP instr. someday
+        self.LDM(reg.sp.value, [reg.r0.value, reg.ip.value], w=1) # XXX Replace with POP instr. someday
 
 define_instructions(AbstractARMv7Builder)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	Sat Oct 30 13:54:54 2010
@@ -34,7 +34,7 @@
 }
 
 data_proc_imm = {
-    'ADD_ri': {'op': 0, 'rncond':'', 'result':True, 'base':True},
+    'AND_ri': {'op': 0, 'rncond':'', 'result':True, 'base':True},
     'EOR_ri': {'op': 0x2, 'rncond':'', 'result':True, 'base':True},
     'SUB_ri': {'op': 0x4, 'rncond':'!0xF', 'result':True, 'base':True},
     #'ADR_ri': {'op': 0x4, 'rncond':'0xF', 'result':True, 'base':True},

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	Sat Oct 30 13:54:54 2010
@@ -12,6 +12,35 @@
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import lltype, rffi, llmemory
 
+def gen_emit_op_ri(opname):
+    def f(self, op, regalloc, fcond):
+        ri_op = getattr(self.mc, '%s_ri' % opname)
+        rr_op = getattr(self.mc, '%s_rr' % opname)
+
+        arg0 = op.getarg(0)
+        arg1 = op.getarg(1)
+        res = regalloc.try_allocate_reg(op.result)
+        if self._check_imm_arg(arg0, 0xFF) and not isinstance(arg1, ConstInt):
+            print 'arg0 is imm'
+            reg = regalloc.try_allocate_reg(arg1)
+            ri_op(res.value, reg.value, imm=arg0.getint(), cond=fcond)
+        elif self._check_imm_arg(arg1, 0xFF) and not isinstance(arg0, ConstInt):
+            print 'arg1 is imm'
+            box = Box()
+            reg = regalloc.try_allocate_reg(arg0)
+            ri_op(res.value, reg.value, imm=arg1.getint(), cond=fcond)
+        else:
+            print 'generating rr'
+            reg = self._put_in_reg(arg0, regalloc)
+            reg2 = self._put_in_reg(arg1, regalloc)
+            rr_op(res.value, reg.value, reg2.value)
+            regalloc.possibly_free_var(reg2)
+
+        regalloc.possibly_free_var(res)
+        regalloc.possibly_free_var(reg)
+        return fcond
+    return f
+
 class IntOpAsslember(object):
     _mixin_ = True
 
@@ -89,11 +118,11 @@
         return fcond
 
     def emit_op_int_mod(self, op, regalloc, fcond):
+        res = regalloc.force_allocate_reg(op.result)
         arg1 = regalloc.make_sure_var_in_reg(op.getarg(0), selected_reg=r.r0)
         arg2 = regalloc.make_sure_var_in_reg(op.getarg(1), selected_reg=r.r1)
         assert arg1 == r.r0
         assert arg2 == r.r1
-        res = regalloc.try_allocate_reg(op.result)
         self.mc.MOD(fcond)
         self.mc.MOV_rr(res.value, r.r0.value, cond=fcond)
         regalloc.possibly_free_vars_for_op(op)
@@ -108,6 +137,14 @@
             reg = regalloc.try_allocate_reg(box)
         return reg
 
+    emit_op_int_and = gen_emit_op_ri('AND')
+    emit_op_int_or = gen_emit_op_ri('ORR')
+    emit_op_int_xor = gen_emit_op_ri('EOR')
+
+    def _check_imm_arg(self, arg, size):
+        return isinstance(arg, ConstInt) and arg.getint() <= size and arg.getint() >= 0
+
+
 class GuardOpAssembler(object):
     _mixin_ = True
 


From hakanardo at codespeak.net  Sat Oct 30 13:59:27 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 13:59:27 +0200 (CEST)
Subject: [pypy-svn] r78524 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101030115927.4D3F4282BAD@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 13:59:25 2010
New Revision: 78524

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(hakanardo, cfbolz, arigo) dont propagate guard_no_exception from one iteration to the next of a loop

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Sat Oct 30 13:59:25 2010
@@ -23,6 +23,8 @@
             self.optimizer.newoperations = []
             jump_args = op.getarglist()
             op.initarglist([])
+            # Exceptions not caught in one itteration should not propagate to the next
+            self.optimizer.exception_might_have_happened = False 
             inputargs = self.inline(self.cloned_operations,
                                     loop.inputargs, jump_args)
             loop.inputargs = inputargs

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 13:59:25 2010
@@ -893,7 +893,7 @@
         i3 = call(i2, descr=nonwritedescr)
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, 'Not', expected, expected)
 
     # ----------
 
@@ -905,20 +905,24 @@
         guard_value(i2, 1) []
         i3 = call_loopinvariant(1, i1, descr=nonwritedescr)
         guard_no_exception() []
-        guard_value(i2, 1) []
+        guard_value(i3, 1) []
         i4 = call_loopinvariant(1, i1, descr=nonwritedescr)
         guard_no_exception() []
-        guard_value(i2, 1) []
+        guard_value(i4, 1) []
         jump(i1)
         """
-        expected = """
+        preamble = """
         [i1]
         i2 = call(1, i1, descr=nonwritedescr)
         guard_no_exception() []
         guard_value(i2, 1) []
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        [i1]
+        jump(i1)
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
 
     # ----------
@@ -928,49 +932,44 @@
         [i, p0]
         i0 = getfield_gc(p0, descr=valuedescr)
         i1 = int_add(i0, i)
-        setfield_gc(p0, i1, descr=valuedescr)
-        jump(i, p0)
+        p1 = new_with_vtable(ConstClass(node_vtable))
+        setfield_gc(p1, i1, descr=valuedescr)
+        jump(i, p1)
+        """
+        preamble = """
+        [i, p0]
+        i0 = getfield_gc(p0, descr=valuedescr)        
+        i1 = int_add(i0, i)
+        jump(i, i1)
         """
         expected = """
         [i, i2]
         i1 = int_add(i2, i)
         jump(i, i1)
         """
-        self.optimize_loop(ops, 'Not, Virtual(node_vtable, valuedescr=Not)',
-                           expected, checkspecnodes=False)
+        self.optimize_loop(ops, 'Not, Not', expected, preamble)
 
     def test_virtual_float(self):
         ops = """
         [f, p0]
         f0 = getfield_gc(p0, descr=floatdescr)
         f1 = float_add(f0, f)
-        setfield_gc(p0, f1, descr=floatdescr)
-        jump(f, p0)
+        p1 = new_with_vtable(ConstClass(node_vtable))
+        setfield_gc(p1, f1, descr=floatdescr)
+        jump(f, p1)
         """
-        expected = """
-        [f, f2]
+        preamble = """
+        [f, p0]
+        f2 = getfield_gc(p0, descr=floatdescr)
         f1 = float_add(f2, f)
         jump(f, f1)
         """
-        self.optimize_loop(ops, 'Not, Virtual(node_vtable, floatdescr=Not)',
-                           expected, checkspecnodes=False)
-
-    def test_virtual_2(self):
-        ops = """
-        [i, p0]
-        i0 = getfield_gc(p0, descr=valuedescr)
-        i1 = int_add(i0, i)
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, i1, descr=valuedescr)
-        jump(i, p1)
-        """
         expected = """
-        [i, i2]
-        i1 = int_add(i2, i)
-        jump(i, i1)
+        [f, f2]
+        f1 = float_add(f2, f)
+        jump(f, f1)
         """
-        self.optimize_loop(ops, 'Not, Virtual(node_vtable, valuedescr=Not)',
-                           expected)
+        self.optimize_loop(ops, 'Not, Not', expected, preamble)
 
     def test_virtual_oois(self):
         ops = """
@@ -999,14 +998,10 @@
         jump(p0, p1, p2)
         """
         expected = """
-        [p2]
+        [p0, p1, p2]
         # all constant-folded :-)
-        jump(p2)
+        jump(p0, p1, p2)
         """
-        self.optimize_loop(ops, '''Virtual(node_vtable),
-                                   Virtual(node_vtable),
-                                   Not''',
-                           expected, checkspecnodes=False)
         #
         # to be complete, we also check the no-opt case where most comparisons
         # are not removed.  The exact set of comparisons removed depends on
@@ -1022,7 +1017,7 @@
         guard_true(i11) []
         jump(p0, p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected2)
+        self.optimize_loop(ops, 'Not, Not, Not', expected, expected2)
 
     def test_virtual_default_field(self):
         ops = """
@@ -1033,15 +1028,17 @@
         # the field 'value' has its default value of 0
         jump(p1)
         """
+        preamble = """
+        [p0]
+        i0 = getfield_gc(p0, descr=valuedescr)
+        guard_value(i0, 0) []
+        jump(p1)
+        """
         expected = """
-        [i]
-        guard_value(i, 0) []
-        jump(0)
+        []
+        jump()
         """
-        # the 'expected' is sub-optimal, but it should be done by another later
-        # optimization step.  See test_find_nodes_default_field() for why.
-        self.optimize_loop(ops, 'Virtual(node_vtable, valuedescr=Not)',
-                           expected)
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_virtual_3(self):
         ops = """


From arigo at codespeak.net  Sat Oct 30 14:00:10 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 14:00:10 +0200 (CEST)
Subject: [pypy-svn] r78525 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030120010.9FF5E282BF0@codespeak.net>

Author: arigo
Date: Sat Oct 30 14:00:09 2010
New Revision: 78525

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
Reduce this file's length by importing stuff from test_optimizebasic.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 14:00:09 2010
@@ -13,14 +13,7 @@
 from pypy.jit.metainterp import executor, compile, resume, history
 from pypy.jit.metainterp.resoperation import rop, opname, ResOperation
 from pypy.jit.metainterp.test.oparser import pure_parse
-
-##class FakeFrame(object):
-##    parent_resumedata_snapshot = None
-##    parent_resumedata_frame_info_list = None
-
-##    def __init__(self, code="", pc=0):
-##        self.jitcode = code
-##        self.pc = pc
+from pypy.jit.metainterp.test.test_optimizebasic import equaloplists
 
 class Fake(object):
     failargs_limit = 1000
@@ -34,175 +27,6 @@
         self.options = Fake()
         self.globaldata = Fake()
 
-def test_store_final_boxes_in_guard():
-    from pypy.jit.metainterp.compile import ResumeGuardDescr
-    from pypy.jit.metainterp.resume import tag, TAGBOX
-    b0 = BoxInt()
-    b1 = BoxInt()
-    opt = optimizeopt.Optimizer(FakeMetaInterpStaticData(LLtypeMixin.cpu),
-                                None)
-    fdescr = ResumeGuardDescr(None, None)
-    op = ResOperation(rop.GUARD_TRUE, ['dummy'], None, descr=fdescr)
-    # setup rd data
-    fi0 = resume.FrameInfo(None, "code0", 11)
-    fdescr.rd_frame_info_list = resume.FrameInfo(fi0, "code1", 33)
-    snapshot0 = resume.Snapshot(None, [b0])
-    fdescr.rd_snapshot = resume.Snapshot(snapshot0, [b1])
-    #
-    opt.store_final_boxes_in_guard(op)
-    if op.getfailargs() == [b0, b1]:
-        assert fdescr.rd_numb.nums      == [tag(1, TAGBOX)]
-        assert fdescr.rd_numb.prev.nums == [tag(0, TAGBOX)]
-    else:
-        assert op.getfailargs() == [b1, b0]
-        assert fdescr.rd_numb.nums      == [tag(0, TAGBOX)]
-        assert fdescr.rd_numb.prev.nums == [tag(1, TAGBOX)]
-    assert fdescr.rd_virtuals is None
-    assert fdescr.rd_consts == []
-
-def test_sharing_field_lists_of_virtual():
-    class FakeOptimizer(object):
-        class cpu(object):
-            pass
-    opt = FakeOptimizer()
-    virt1 = virtualize.AbstractVirtualStructValue(opt, None)
-    lst1 = virt1._get_field_descr_list()
-    assert lst1 == []
-    lst2 = virt1._get_field_descr_list()
-    assert lst1 is lst2
-    virt1.setfield(LLtypeMixin.valuedescr, optimizeopt.OptValue(None))
-    lst3 = virt1._get_field_descr_list()
-    assert lst3 == [LLtypeMixin.valuedescr]
-    lst4 = virt1._get_field_descr_list()
-    assert lst3 is lst4
-
-    virt2 = virtualize.AbstractVirtualStructValue(opt, None)
-    lst5 = virt2._get_field_descr_list()
-    assert lst5 is lst1
-    virt2.setfield(LLtypeMixin.valuedescr, optimizeopt.OptValue(None))
-    lst6 = virt1._get_field_descr_list()
-    assert lst6 is lst3
-
-def test_reuse_vinfo():
-    class FakeVInfo(object):
-        def set_content(self, fieldnums):
-            self.fieldnums = fieldnums
-        def equals(self, fieldnums):
-            return self.fieldnums == fieldnums
-    class FakeVirtualValue(virtualize.AbstractVirtualValue):
-        def _make_virtual(self, *args):
-            return FakeVInfo()
-    v1 = FakeVirtualValue(None, None, None)
-    vinfo1 = v1.make_virtual_info(None, [1, 2, 4])
-    vinfo2 = v1.make_virtual_info(None, [1, 2, 4])
-    assert vinfo1 is vinfo2
-    vinfo3 = v1.make_virtual_info(None, [1, 2, 6])
-    assert vinfo3 is not vinfo2
-    vinfo4 = v1.make_virtual_info(None, [1, 2, 6])
-    assert vinfo3 is vinfo4
-
-def test_descrlist_dict():
-    from pypy.jit.metainterp import optimizeutil
-    h1 = optimizeutil.descrlist_hash([])
-    h2 = optimizeutil.descrlist_hash([LLtypeMixin.valuedescr])
-    h3 = optimizeutil.descrlist_hash(
-            [LLtypeMixin.valuedescr, LLtypeMixin.nextdescr])
-    assert h1 != h2
-    assert h2 != h3
-    assert optimizeutil.descrlist_eq([], [])
-    assert not optimizeutil.descrlist_eq([], [LLtypeMixin.valuedescr])
-    assert optimizeutil.descrlist_eq([LLtypeMixin.valuedescr],
-                                     [LLtypeMixin.valuedescr])
-    assert not optimizeutil.descrlist_eq([LLtypeMixin.valuedescr],
-                                         [LLtypeMixin.nextdescr])
-    assert optimizeutil.descrlist_eq([LLtypeMixin.valuedescr, LLtypeMixin.nextdescr],
-                                     [LLtypeMixin.valuedescr, LLtypeMixin.nextdescr])
-    assert not optimizeutil.descrlist_eq([LLtypeMixin.nextdescr, LLtypeMixin.valuedescr],
-                                         [LLtypeMixin.valuedescr, LLtypeMixin.nextdescr])
-
-    # descrlist_eq should compare by identity of the descrs, not by the result
-    # of sort_key
-    class FakeDescr(object):
-        def sort_key(self):
-            return 1
-
-    assert not optimizeutil.descrlist_eq([FakeDescr()], [FakeDescr()])
-
-
-# ____________________________________________________________
-
-def equaloplists(oplist1, oplist2, strict_fail_args=True, remap={}):
-    print '-'*20, 'Comparing lists', '-'*20
-    for op1, op2 in zip(oplist1, oplist2):
-        txt1 = str(op1)
-        txt2 = str(op2)
-        while txt1 or txt2:
-            print '%-39s| %s' % (txt1[:39], txt2[:39])
-            txt1 = txt1[39:]
-            txt2 = txt2[39:]
-        assert op1.getopnum() == op2.getopnum()
-        assert op1.numargs() == op2.numargs()
-        for i in range(op1.numargs()):
-            x = op1.getarg(i)
-            y = op2.getarg(i)
-            assert x == remap.get(y, y)
-        if op2.result in remap:
-            assert op1.result == remap[op2.result]
-        else:
-            remap[op2.result] = op1.result
-        if op1.getopnum() != rop.JUMP:      # xxx obscure
-            assert op1.getdescr() == op2.getdescr()
-        if op1.getfailargs() or op2.getfailargs():
-            assert len(op1.getfailargs()) == len(op2.getfailargs())
-            if strict_fail_args:
-                for x, y in zip(op1.getfailargs(), op2.getfailargs()):
-                    assert x == remap.get(y, y)
-            else:
-                fail_args1 = set(op1.getfailargs())
-                fail_args2 = set([remap.get(y, y) for y in op2.getfailargs()])
-                assert fail_args1 == fail_args2
-    assert len(oplist1) == len(oplist2)
-    print '-'*57
-    return True
-
-def test_equaloplists():
-    ops = """
-    [i0]
-    i1 = int_add(i0, 1)
-    i2 = int_add(i1, 1)
-    guard_true(i1) [i2]
-    jump(i1)
-    """
-    namespace = {}
-    loop1 = pure_parse(ops, namespace=namespace)
-    loop2 = pure_parse(ops, namespace=namespace)
-    loop3 = pure_parse(ops.replace("i2 = int_add", "i2 = int_sub"),
-                       namespace=namespace)
-    assert equaloplists(loop1.operations, loop2.operations)
-    py.test.raises(AssertionError,
-                   "equaloplists(loop1.operations, loop3.operations)")
-
-def test_equaloplists_fail_args():
-    ops = """
-    [i0]
-    i1 = int_add(i0, 1)
-    i2 = int_add(i1, 1)
-    guard_true(i1) [i2, i1]
-    jump(i1)
-    """
-    namespace = {}
-    loop1 = pure_parse(ops, namespace=namespace)
-    loop2 = pure_parse(ops.replace("[i2, i1]", "[i1, i2]"),
-                       namespace=namespace)
-    py.test.raises(AssertionError,
-                   "equaloplists(loop1.operations, loop2.operations)")
-    assert equaloplists(loop1.operations, loop2.operations,
-                        strict_fail_args=False)
-    loop3 = pure_parse(ops.replace("[i2, i1]", "[i2, i0]"),
-                       namespace=namespace)
-    py.test.raises(AssertionError,
-                   "equaloplists(loop1.operations, loop3.operations)")
-
 # ____________________________________________________________
 
 class Storage(compile.ResumeGuardDescr):


From david at codespeak.net  Sat Oct 30 16:02:18 2010
From: david at codespeak.net (david at codespeak.net)
Date: Sat, 30 Oct 2010 16:02:18 +0200 (CEST)
Subject: [pypy-svn] r78526 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . test
Message-ID: <20101030140218.96EB2282BDD@codespeak.net>

Author: david
Date: Sat Oct 30 16:02:16 2010
New Revision: 78526

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
Log:
Implement shift operations int_lshift, int_rshift, uint_rshift

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	Sat Oct 30 16:02:16 2010
@@ -94,6 +94,42 @@
                         | reg_operation(rd, rn, rm, imm, s, shifttype))
     return f
 
+def define_data_proc_register_shifted(name, table):
+    n = ((0x1 << 4) | (table['op1'] & 0x1F) << 20 | (table['op2'] & 0x3) << 5)
+    if 'result' in table and not table['result']:
+        result = False
+    else:
+        result = True
+    if name[-2:] == 'sr':
+        if result:
+            def f(self, rd, rn, rm, rs, cond=cond.AL, s=0, shifttype=0):
+                self.write32(n
+                            | cond << 28
+                            | (s & 0x1) << 20
+                            | (rn & 0xF) << 16
+                            | (rd & 0xF) << 12
+                            | (rs & 0xF) << 8
+                            | (shifttype & 0x3) << 5
+                            | (rm & 0xF))
+        else:
+            def f(self, rn, rm, rs, cond=cond.AL, s=0, shifttype=0):
+                self.write32(n
+                            | cond << 28
+                            | (s & 0x1) << 20
+                            | (rn & 0xF) << 16
+                            | (rs & 0xF) << 8
+                            | (shifttype & 0x3) << 5
+                            | (rm & 0xF))
+    else:
+        def f(self, rd, rn, rm, cond=cond.AL, s=0):
+            self.write32(n
+                        | cond << 28
+                        | (s & 0x1) << 20
+                        | (rd & 0xF) << 12
+                        | (rm & 0xF) << 8
+                        | (rn & 0xF))
+    return f
+
 def define_supervisor_and_coproc(name, table):
     n = (0x3 << 26 | (table['op1'] & 0x3F) << 20 | (table['op'] & 0x1) << 4)
     def f(self, coproc, opc1, rt, crn, crm, opc2=0, cond=cond.AL):
@@ -162,7 +198,8 @@
                 (instructions.data_proc, define_data_proc),
                 (instructions.data_proc_imm, define_data_proc_imm),
                 (instructions.supervisor_and_coproc, define_supervisor_and_coproc),
-                (instructions.multiply, define_multiply_instructions)]
+                (instructions.multiply, define_multiply_instructions),
+                (instructions.data_proc_reg_shift_reg, define_data_proc_register_shifted)]
 
     for inss, gen in i_g_map:
         for key, val in inss.iteritems():

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	Sat Oct 30 16:02:16 2010
@@ -31,6 +31,30 @@
     'ASR_ri': {'op1':0x1A, 'op2':0, 'op3':0x2, 'op2cond':'', 'result':False, 'base':True},
     #'RRX_ri': {'op1':0x1A, 'op2':0, 'op3':0x3, 'op2cond':'0', 'result':False, 'base':True},
     'ROR_ri': {'op1':0x1A, 'op2':0x0, 'op3':0x3, 'op2cond':'!0', 'result':True, 'base':False},
+    #BIC
+    #MVN
+}
+
+data_proc_reg_shift_reg = {
+    'AND_rr_sr': {'op1':0x0,  'op2':0},
+    'EOR_rr_sr': {'op1':0x2,  'op2':0},
+    'SUB_rr_sr': {'op1':0x4,  'op2':0},
+    'RSB_rr_sr': {'op1':0x6,  'op2':0},
+    'ADD_rr_sr': {'op1':0x8,  'op2':0},
+    'ADC_rr_sr': {'op1':0xA,  'op2':0},
+    'SBC_rr_sr': {'op1':0xC,  'op2':0},
+    'RSC_rr_sr': {'op1':0xE,  'op2':0},
+    'TST_rr_sr': {'op1':0x11, 'op2':0, 'result': False},
+    'TEQ_rr_sr': {'op1':0x13, 'op2':0, 'result': False},
+    'CMP_rr_sr': {'op1':0x15, 'op2':0, 'result': False},
+    'CMN_rr_sr': {'op1':0x17, 'op2':0, 'result': False},
+    'ORR_rr_sr': {'op1':0x18, 'op2':0},
+    'LSL_rr': {'op1':0x1A, 'op2':0, },
+    'LSR_rr': {'op1':0x1A, 'op2':0x1},
+    'ASR_rr': {'op1':0x1A, 'op2':0x2},
+    #'RRX_rr': {'op1':0x1A, 'op2':0,},
+    'ROR_rr': {'op1':0x1A, 'op2':0x3},
+    # BIC, MVN
 }
 
 data_proc_imm = {

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	Sat Oct 30 16:02:16 2010
@@ -12,7 +12,7 @@
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import lltype, rffi, llmemory
 
-def gen_emit_op_ri(opname):
+def gen_emit_op_ri(opname, imm_size=0xFF, commutative=True):
     def f(self, op, regalloc, fcond):
         ri_op = getattr(self.mc, '%s_ri' % opname)
         rr_op = getattr(self.mc, '%s_rr' % opname)
@@ -20,17 +20,15 @@
         arg0 = op.getarg(0)
         arg1 = op.getarg(1)
         res = regalloc.try_allocate_reg(op.result)
-        if self._check_imm_arg(arg0, 0xFF) and not isinstance(arg1, ConstInt):
-            print 'arg0 is imm'
+        if (commutative
+                and self._check_imm_arg(arg0, imm_size)
+                and not isinstance(arg1, ConstInt)):
             reg = regalloc.try_allocate_reg(arg1)
             ri_op(res.value, reg.value, imm=arg0.getint(), cond=fcond)
-        elif self._check_imm_arg(arg1, 0xFF) and not isinstance(arg0, ConstInt):
-            print 'arg1 is imm'
-            box = Box()
+        elif self._check_imm_arg(arg1, imm_size) and not isinstance(arg0, ConstInt):
             reg = regalloc.try_allocate_reg(arg0)
             ri_op(res.value, reg.value, imm=arg1.getint(), cond=fcond)
         else:
-            print 'generating rr'
             reg = self._put_in_reg(arg0, regalloc)
             reg2 = self._put_in_reg(arg1, regalloc)
             rr_op(res.value, reg.value, reg2.value)
@@ -140,9 +138,14 @@
     emit_op_int_and = gen_emit_op_ri('AND')
     emit_op_int_or = gen_emit_op_ri('ORR')
     emit_op_int_xor = gen_emit_op_ri('EOR')
+    emit_op_int_lshift = gen_emit_op_ri('LSL', imm_size=0x1F, commutative=False)
+    emit_op_int_rshift = gen_emit_op_ri('ASR', imm_size=0x1F, commutative=False)
+    emit_op_uint_rshift = gen_emit_op_ri('LSR', imm_size=0x1F, commutative=False)
+
 
     def _check_imm_arg(self, arg, size):
-        return isinstance(arg, ConstInt) and arg.getint() <= size and arg.getint() >= 0
+        #XXX check ranges for different operations
+        return isinstance(arg, ConstInt) and arg.getint() <= size and arg.getint() > 0
 
 
 class GuardOpAssembler(object):

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	Sat Oct 30 16:02:16 2010
@@ -126,23 +126,6 @@
     def setup_method(self, ffuu_method):
         self.cb = CodeBuilder()
 
-def build_tests():
-    test_name = 'test_generated_%s'
-    for key, value in instructions.load_store.iteritems():
-        if value['imm']:
-            f = gen_test_imm_func
-        else:
-            f = gen_test_reg_func
-        build_test(f, key, value, test_name)
-
-    for key, value, in instructions.data_proc.iteritems():
-        build_test(gen_test_data_reg_func, key, value, test_name)
-
-    for key, value, in instructions.data_proc_imm.iteritems():
-        build_test(gen_test_data_proc_imm_func, key, value, test_name)
-
-    for key, value, in instructions.multiply.iteritems():
-        build_test(gen_test_mul_func, key, value, test_name)
 # XXX refactor this functions
 
 def build_test(builder, key, value, test_name):
@@ -200,6 +183,31 @@
             self.assert_equal('%s r3, r7, r12' % name)
     return f
 
+def gen_test_data_reg_shift_reg_func(name, table):
+    if name[-2:] == 'rr':
+        def f(self):
+            func = getattr(self.cb, name)
+            func(r.r3.value, r.r7.value, r.r12.value)
+            self.assert_equal('%s r3, r7, r12' % name[:name.index('_')])
+
+    else:
+        if 'result' in table and not table['result']:
+            result = False
+        else:
+            result = True
+        if result:
+            def f(self):
+                func = getattr(self.cb, name)
+                func(r.r3.value, r.r7.value, r.r8.value, r.r11.value, shifttype=0x2)
+                self.assert_equal('%s r3, r7, r8, ASR r11' % name[:name.index('_')])
+        else:
+            def f(self):
+                func = getattr(self.cb, name)
+                func(r.r3.value, r.r7.value, r.r11.value, shifttype=0x2)
+                self.assert_equal('%s r3, r7, ASR r11' % name[:name.index('_')])
+
+    return f
+
 def gen_test_data_reg_func(name, table):
     if name[-2:] == 'ri':
         def f(self):
@@ -220,4 +228,25 @@
 
     return f
 
+def build_tests():
+    test_name = 'test_generated_%s'
+    for key, value in instructions.load_store.iteritems():
+        if value['imm']:
+            f = gen_test_imm_func
+        else:
+            f = gen_test_reg_func
+        build_test(f, key, value, test_name)
+
+    for key, value, in instructions.data_proc.iteritems():
+        build_test(gen_test_data_reg_func, key, value, test_name)
+
+    for key, value, in instructions.data_proc_reg_shift_reg.iteritems():
+        build_test(gen_test_data_reg_shift_reg_func, key, value, test_name)
+
+    for key, value, in instructions.data_proc_imm.iteritems():
+        build_test(gen_test_data_proc_imm_func, key, value, test_name)
+
+    for key, value, in instructions.multiply.iteritems():
+        build_test(gen_test_mul_func, key, value, test_name)
+
 build_tests()


From hakanardo at codespeak.net  Sat Oct 30 16:03:55 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 16:03:55 +0200 (CEST)
Subject: [pypy-svn] r78527 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101030140355.22BC3282BEF@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 16:03:53 2010
New Revision: 78527

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(hakanardo, cfbolz, arigo) default values of virtuals

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Sat Oct 30 16:03:53 2010
@@ -371,7 +371,8 @@
             # optimizefindnode should ensure that fieldvalue is found
             assert isinstance(value, AbstractVirtualValue)
             fieldvalue = value.getfield(op.getdescr(), None)
-            assert fieldvalue is not None
+            if fieldvalue is None:
+                fieldvalue = self.optimizer.new_const(op.getdescr())
             self.make_equal_to(op.result, fieldvalue)
         else:
             value.ensure_nonnull()

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 16:03:53 2010
@@ -856,7 +856,7 @@
         [p0]
         i0 = getfield_gc(p0, descr=valuedescr)
         guard_value(i0, 0) []
-        jump(p1)
+        jump()
         """
         expected = """
         []


From hakanardo at codespeak.net  Sat Oct 30 16:20:28 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 16:20:28 +0200 (CEST)
Subject: [pypy-svn] r78528 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101030142028.AD2AC36C394@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 16:20:26 2010
New Revision: 78528

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Sat Oct 30 16:20:26 2010
@@ -13,6 +13,7 @@
         self.cloned_operations = []
         for op in self.optimizer.loop.operations:
             self.cloned_operations.append(op.clone())
+        
             
     def propagate_forward(self, op):
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 16:20:26 2010
@@ -891,14 +891,21 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(i3, p1)
         """
+        preamble = """
+        [i0, p0]
+        guard_class(p0, ConstClass(node_vtable)) []
+        i1 = getfield_gc(p0, descr=valuedescr)
+        i2 = int_sub(i1, 1)
+        i3 = int_add(i0, i1)
+        jump(i3, i2)
+        """
         expected = """
         [i0, i1]
         i2 = int_sub(i1, 1)
         i3 = int_add(i0, i1)
         jump(i3, i2)
         """
-        self.optimize_loop(ops, 'Not, Virtual(node_vtable, valuedescr=Not)',
-                           expected)
+        self.optimize_loop(ops, 'Not, Not', expected, preamble)
 
     def test_virtual_5(self):
         ops = """
@@ -914,18 +921,21 @@
         setfield_gc(p1, p2, descr=nextdescr)
         jump(i3, p1)
         """
+        preamble = """
+        [i0, p0]
+        guard_class(p0, ConstClass(node_vtable)) []
+        i1 = getfield_gc(p0, descr=valuedescr)
+        i2 = int_sub(i1, 1)
+        i3 = int_add(i0, i1)
+        jump(i3, i2, i1)
+        """
         expected = """
         [i0, i1, i1bis]
         i2 = int_sub(i1, 1)
         i3 = int_add(i0, i1)
         jump(i3, i2, i1)
         """
-        self.optimize_loop(ops,
-            '''Not, Virtual(node_vtable,
-                            valuedescr=Not,
-                            nextdescr=Virtual(node_vtable2,
-                                              valuedescr=Not))''',
-                           expected)
+        self.optimize_loop(ops, 'Not, Not', expected, preamble)
 
     def test_virtual_constant_isnull(self):
         ops = """


From arigo at codespeak.net  Sat Oct 30 16:28:33 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 16:28:33 +0200 (CEST)
Subject: [pypy-svn] r78529 - pypy/trunk/pypy/translator/c/gcc
Message-ID: <20101030142833.CBEDF282BEC@codespeak.net>

Author: arigo
Date: Sat Oct 30 16:28:32 2010
New Revision: 78529

Modified:
   pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
Log:
Add psllX.


Modified: pypy/trunk/pypy/translator/c/gcc/trackgcroot.py
==============================================================================
--- pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	(original)
+++ pypy/trunk/pypy/translator/c/gcc/trackgcroot.py	Sat Oct 30 16:28:32 2010
@@ -455,7 +455,7 @@
         'inc', 'dec', 'not', 'neg', 'or', 'and', 'sbb', 'adc',
         'shl', 'shr', 'sal', 'sar', 'rol', 'ror', 'mul', 'imul', 'div', 'idiv',
         'bswap', 'bt', 'rdtsc',
-        'punpck', 'pshufd', 
+        'punpck', 'pshufd', 'psll',
         # zero-extending moves should not produce GC pointers
         'movz',
         ])


From hakanardo at codespeak.net  Sat Oct 30 16:41:30 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 16:41:30 +0200 (CEST)
Subject: [pypy-svn] r78530 - in pypy/branch/jit-unroll-loops: . dotviewer
	lib-python lib-python/modified-2.5.2/test lib_pypy/_ctypes
	pypy pypy/annotation pypy/annotation/test pypy/config
	pypy/config/test pypy/doc pypy/doc/config pypy/interpreter
	pypy/interpreter/astcompiler/test pypy/interpreter/pyparser
	pypy/interpreter/test pypy/jit/backend
	pypy/jit/backend/llgraph pypy/jit/backend/llsupport
	pypy/jit/backend/llsupport/test pypy/jit/backend/test
	pypy/jit/backend/x86 pypy/jit/backend/x86/test
	pypy/jit/codewriter pypy/jit/codewriter/test
	pypy/jit/metainterp pypy/jit/metainterp/optimizeopt
	pypy/jit/metainterp/test pypy/jit/tl pypy/jit/tl/spli
	pypy/jit/tl/tla pypy/jit/tool pypy/module/__builtin__
	pypy/module/__builtin__/test pypy/module/_bisect
	pypy/module/_bisect/test pypy/module/_ffi
	pypy/module/_ffi/test pypy/module/_rawffi
	pypy/module/_rawffi/test pypy/module/_socket/test
	pypy/module/_sre pypy/module/_weakref pypy/module/_winreg
	pypy/module/array pypy/module/array/benchmark
	pypy/module/array/test pypy/module/bz2 pypy/module/cpyext
	pypy/module/cpyext/test pypy/module/gc pypy/module/gc/test
	pypy/module/imp pypy/module/imp/test pypy/module/parser/test
	pypy/module/pypyjit pypy/module/pypyjit/test
	pypy/module/signal pypy/module/sys
	pypy/module/test_lib_pypy/ctypes_tests pypy/module/thread
	pypy/objspace pypy/objspace/flow pypy/objspace/flow/test
	pypy/objspace/std pypy/objspace/std/test pypy/objspace/test
	pypy/rlib pypy/rlib/rsre pypy/rlib/rsre/test pypy/rlib/test
	pypy/rpython pypy/rpython/lltypesystem
	pypy/rpython/lltypesystem/test pypy/rpython/memory
	pypy/rpython/memory/gc pypy/rpython/memory/gc/test
	pypy/rpython/memory/gctransform pypy/rpython/memory/test
	pypy/rpython/module pypy/rpython/test pypy/tool
	pypy/tool/release pypy/tool/release/test pypy/tool/test
	pypy/translator pypy/translator/c pypy/translator/c/gcc
	pypy/translator/c/gcc/test pypy/translator/c/gcc/test/darwin64
	pypy/translator/c/gcc/test/elf
	pypy/translator/c/gcc/test/elf64 pypy/translator/c/src
	pypy/translator/c/test pypy/translator/goal pypy/translator/jvm/test
	pypy/translator/oosupport/test_template pypy/translator/platform
Message-ID: <20101030144130.B614C282BF0@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 16:41:17 2010
New Revision: 78530

Added:
   pypy/branch/jit-unroll-loops/lib-python/modified-2.5.2/test/test_dict.py
      - copied unchanged from r78528, pypy/trunk/lib-python/modified-2.5.2/test/test_dict.py
   pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.std.withmapdict.txt
      - copied unchanged from r78528, pypy/trunk/pypy/doc/config/objspace.std.withmapdict.txt
   pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.std.withstrbuf.txt
      - copied unchanged from r78528, pypy/trunk/pypy/doc/config/objspace.std.withstrbuf.txt
   pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.usemodules._bisect.txt
      - copied unchanged from r78528, pypy/trunk/pypy/doc/config/objspace.usemodules._bisect.txt
   pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.usemodules._ffi.txt
      - copied unchanged from r78528, pypy/trunk/pypy/doc/config/objspace.usemodules._ffi.txt
   pypy/branch/jit-unroll-loops/pypy/interpreter/pyparser/autopath.py
      - copied unchanged from r78528, pypy/trunk/pypy/interpreter/pyparser/autopath.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/pyparser/genpytokenize.py
      - copied unchanged from r78528, pypy/trunk/pypy/interpreter/pyparser/genpytokenize.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/pyparser/pylexer.py
      - copied unchanged from r78528, pypy/trunk/pypy/interpreter/pyparser/pylexer.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/conftest.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/backend/conftest.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/ffisupport.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/backend/llsupport/ffisupport.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/test/test_ffisupport.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/backend/llsupport/test/test_ffisupport.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_void_list.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/codewriter/test/test_void_list.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/greenfield.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/metainterp/greenfield.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/fficall.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/metainterp/optimizeopt/fficall.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_fficall.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/metainterp/test/test_fficall.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_greenfield.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/metainterp/test/test_greenfield.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefficall.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/metainterp/test/test_optimizefficall.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/jittest.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/tl/jittest.py
   pypy/branch/jit-unroll-loops/pypy/jit/tool/oparser.py
      - copied unchanged from r78528, pypy/trunk/pypy/jit/tool/oparser.py
   pypy/branch/jit-unroll-loops/pypy/jit/tool/pypytrace.vim
      - copied unchanged from r78528, pypy/trunk/pypy/jit/tool/pypytrace.vim
   pypy/branch/jit-unroll-loops/pypy/module/_bisect/   (props changed)
      - copied from r78528, pypy/trunk/pypy/module/_bisect/
   pypy/branch/jit-unroll-loops/pypy/module/_ffi/   (props changed)
      - copied from r78528, pypy/trunk/pypy/module/_ffi/
   pypy/branch/jit-unroll-loops/pypy/objspace/std/mapdict.py
      - copied unchanged from r78528, pypy/trunk/pypy/objspace/std/mapdict.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/strbufobject.py
      - copied unchanged from r78528, pypy/trunk/pypy/objspace/std/strbufobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_mapdict.py
      - copied unchanged from r78528, pypy/trunk/pypy/objspace/std/test/test_mapdict.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_methodcache.py
      - copied unchanged from r78528, pypy/trunk/pypy/objspace/std/test/test_methodcache.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_strbufobject.py
      - copied unchanged from r78528, pypy/trunk/pypy/objspace/std/test/test_strbufobject.py
   pypy/branch/jit-unroll-loops/pypy/rlib/clibffi.py
      - copied unchanged from r78528, pypy/trunk/pypy/rlib/clibffi.py
   pypy/branch/jit-unroll-loops/pypy/rlib/libffi.py
      - copied unchanged from r78528, pypy/trunk/pypy/rlib/libffi.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rerased.py
      - copied unchanged from r78528, pypy/trunk/pypy/rlib/rerased.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rsre/rsre_jit.py
      - copied unchanged from r78528, pypy/trunk/pypy/rlib/rsre/rsre_jit.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rsre/test/conftest.py
      - copied unchanged from r78528, pypy/trunk/pypy/rlib/rsre/test/conftest.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rsre/test/test_zjit.py
      - copied unchanged from r78528, pypy/trunk/pypy/rlib/rsre/test/test_zjit.py
   pypy/branch/jit-unroll-loops/pypy/rlib/test/test_clibffi.py
      - copied unchanged from r78528, pypy/trunk/pypy/rlib/test/test_clibffi.py
   pypy/branch/jit-unroll-loops/pypy/rlib/test/test_libffi.py
      - copied unchanged from r78528, pypy/trunk/pypy/rlib/test/test_libffi.py
   pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rerased.py
      - copied unchanged from r78528, pypy/trunk/pypy/rlib/test/test_rerased.py
   pypy/branch/jit-unroll-loops/pypy/tool/leakfinder.py
      - copied unchanged from r78528, pypy/trunk/pypy/tool/leakfinder.py
   pypy/branch/jit-unroll-loops/pypy/tool/test/test_leakfinder.py
      - copied unchanged from r78528, pypy/trunk/pypy/tool/test/test_leakfinder.py
   pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/darwin64/
      - copied from r78528, pypy/trunk/pypy/translator/c/gcc/test/darwin64/
   pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/elf/track10.s
      - copied unchanged from r78528, pypy/trunk/pypy/translator/c/gcc/test/elf/track10.s
   pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/elf/track11.s
      - copied unchanged from r78528, pypy/trunk/pypy/translator/c/gcc/test/elf/track11.s
   pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/elf64/track_loadconst.s
      - copied unchanged from r78528, pypy/trunk/pypy/translator/c/gcc/test/elf64/track_loadconst.s
   pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s
      - copied unchanged from r78528, pypy/trunk/pypy/translator/c/gcc/test/elf64/track_rpyassertfailed.s
   pypy/branch/jit-unroll-loops/pypy/translator/c/src/debug_alloc.h
      - copied unchanged from r78528, pypy/trunk/pypy/translator/c/src/debug_alloc.h
Removed:
   pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.std.withinlineddict.txt
   pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.std.withshadowtracking.txt
   pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.std.withsharingdict.txt
   pypy/branch/jit-unroll-loops/pypy/jit/backend/test/conftest.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/oparser.py
   pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_find.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/inlinedict.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/sharingdict.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_inlinedict.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_shadowtracking.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_sharingdict.py
   pypy/branch/jit-unroll-loops/pypy/rpython/rspecialcase.py
   pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rspecialcase.py
Modified:
   pypy/branch/jit-unroll-loops/   (props changed)
   pypy/branch/jit-unroll-loops/dotviewer/drawgraph.py
   pypy/branch/jit-unroll-loops/lib-python/conftest.py
   pypy/branch/jit-unroll-loops/lib-python/modified-2.5.2/test/test_descr.py
   pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/function.py
   pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/structure.py
   pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/union.py
   pypy/branch/jit-unroll-loops/pypy/   (props changed)
   pypy/branch/jit-unroll-loops/pypy/annotation/bookkeeper.py
   pypy/branch/jit-unroll-loops/pypy/annotation/builtin.py
   pypy/branch/jit-unroll-loops/pypy/annotation/classdef.py
   pypy/branch/jit-unroll-loops/pypy/annotation/description.py
   pypy/branch/jit-unroll-loops/pypy/annotation/dictdef.py
   pypy/branch/jit-unroll-loops/pypy/annotation/listdef.py
   pypy/branch/jit-unroll-loops/pypy/annotation/model.py
   pypy/branch/jit-unroll-loops/pypy/annotation/policy.py
   pypy/branch/jit-unroll-loops/pypy/annotation/specialize.py
   pypy/branch/jit-unroll-loops/pypy/annotation/test/test_annrpython.py
   pypy/branch/jit-unroll-loops/pypy/annotation/test/test_model.py
   pypy/branch/jit-unroll-loops/pypy/annotation/unaryop.py
   pypy/branch/jit-unroll-loops/pypy/config/pypyoption.py
   pypy/branch/jit-unroll-loops/pypy/config/test/test_pypyoption.py
   pypy/branch/jit-unroll-loops/pypy/conftest.py
   pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt
   pypy/branch/jit-unroll-loops/pypy/doc/docindex.txt
   pypy/branch/jit-unroll-loops/pypy/doc/interpreter-optimizations.txt
   pypy/branch/jit-unroll-loops/pypy/interpreter/astcompiler/test/test_compiler.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/baseobjspace.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/function.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/gateway.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/pycode.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/pyframe.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/pyopcode.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/pyparser/pytokenize.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/test/test_function.py
   pypy/branch/jit-unroll-loops/pypy/interpreter/typedef.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/detect_cpu.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/llgraph/llimpl.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/llgraph/runner.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/descr.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/gc.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/llmodel.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/symbolic.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/test/test_descr.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/test/test_gc.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/model.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/test/runner_test.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/test/test_ll_random.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/test/test_random.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/assembler.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/regalloc.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/regloc.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/runner.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/rx86.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_assembler.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_gc_integration.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_regalloc.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_runner.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_string.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_zrpy_gc.py
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_ztranslation.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/call.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/codewriter.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/effectinfo.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/format.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/jtransform.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/support.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_codewriter.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_effectinfo.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_jtransform.py
   pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_list.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/blackhole.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/executor.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/graphpage.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/history.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/jitdriver.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py   (contents, props changed)
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py   (props changed)
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resoperation.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/simple_optimize.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitdriver.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_logger.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nopspec.py   (props changed)
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_oparser.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefindnode.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_string.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualref.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_warmspot.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/virtualref.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmspot.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/spli/interpreter.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/tla/test_tla.py
   pypy/branch/jit-unroll-loops/pypy/jit/tool/loopviewer.py
   pypy/branch/jit-unroll-loops/pypy/jit/tool/showstats.py
   pypy/branch/jit-unroll-loops/pypy/jit/tool/traceviewer.py
   pypy/branch/jit-unroll-loops/pypy/module/__builtin__/interp_classobj.py
   pypy/branch/jit-unroll-loops/pypy/module/__builtin__/test/test_classobj.py
   pypy/branch/jit-unroll-loops/pypy/module/_bisect/test/   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/_ffi/test/   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/_rawffi/__init__.py
   pypy/branch/jit-unroll-loops/pypy/module/_rawffi/array.py
   pypy/branch/jit-unroll-loops/pypy/module/_rawffi/callback.py
   pypy/branch/jit-unroll-loops/pypy/module/_rawffi/interp_rawffi.py
   pypy/branch/jit-unroll-loops/pypy/module/_rawffi/structure.py
   pypy/branch/jit-unroll-loops/pypy/module/_rawffi/test/test__rawffi.py
   pypy/branch/jit-unroll-loops/pypy/module/_socket/test/test_sock_app.py
   pypy/branch/jit-unroll-loops/pypy/module/_sre/interp_sre.py
   pypy/branch/jit-unroll-loops/pypy/module/_weakref/interp__weakref.py
   pypy/branch/jit-unroll-loops/pypy/module/_winreg/interp_winreg.py
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/Makefile   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/intimg.c   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/intimgtst.c   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/intimgtst.py   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/loop.c   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/sum.c   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/sumtst.c   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/array/benchmark/sumtst.py   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/array/interp_array.py
   pypy/branch/jit-unroll-loops/pypy/module/array/test/test_array_old.py   (props changed)
   pypy/branch/jit-unroll-loops/pypy/module/bz2/interp_bz2.py
   pypy/branch/jit-unroll-loops/pypy/module/cpyext/classobject.py
   pypy/branch/jit-unroll-loops/pypy/module/cpyext/test/test_cpyext.py
   pypy/branch/jit-unroll-loops/pypy/module/cpyext/test/test_unicodeobject.py
   pypy/branch/jit-unroll-loops/pypy/module/gc/interp_gc.py
   pypy/branch/jit-unroll-loops/pypy/module/gc/referents.py
   pypy/branch/jit-unroll-loops/pypy/module/gc/test/test_gc.py
   pypy/branch/jit-unroll-loops/pypy/module/imp/importing.py
   pypy/branch/jit-unroll-loops/pypy/module/imp/test/test_import.py
   pypy/branch/jit-unroll-loops/pypy/module/parser/test/test_parser.py
   pypy/branch/jit-unroll-loops/pypy/module/pypyjit/__init__.py
   pypy/branch/jit-unroll-loops/pypy/module/pypyjit/interp_jit.py
   pypy/branch/jit-unroll-loops/pypy/module/pypyjit/policy.py
   pypy/branch/jit-unroll-loops/pypy/module/pypyjit/test/test_pypy_c.py
   pypy/branch/jit-unroll-loops/pypy/module/signal/interp_signal.py
   pypy/branch/jit-unroll-loops/pypy/module/sys/__init__.py
   pypy/branch/jit-unroll-loops/pypy/module/sys/state.py
   pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c
   pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py
   pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py
   pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py
   pypy/branch/jit-unroll-loops/pypy/module/thread/ll_thread.py
   pypy/branch/jit-unroll-loops/pypy/objspace/descroperation.py
   pypy/branch/jit-unroll-loops/pypy/objspace/flow/flowcontext.py
   pypy/branch/jit-unroll-loops/pypy/objspace/flow/test/test_objspace.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/callmethod.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/celldict.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/dictmultiobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/dicttype.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/fake.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/frozensettype.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/inttype.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/listobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/longtype.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/model.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/objspace.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/proxyobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/setobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/settype.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/smallintobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/stringobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/stringtype.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/strutil.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_dictmultiobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_longobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_setobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_strutil.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_typeobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_versionedtype.py
   pypy/branch/jit-unroll-loops/pypy/objspace/std/typeobject.py
   pypy/branch/jit-unroll-loops/pypy/objspace/test/test_descroperation.py
   pypy/branch/jit-unroll-loops/pypy/rlib/_rsocket_rffi.py
   pypy/branch/jit-unroll-loops/pypy/rlib/jit.py
   pypy/branch/jit-unroll-loops/pypy/rlib/objectmodel.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rbigint.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rgc.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rmmap.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rsre/rsre_char.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rsre/rsre_core.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rsre/test/test_match.py
   pypy/branch/jit-unroll-loops/pypy/rlib/rstring.py
   pypy/branch/jit-unroll-loops/pypy/rlib/test/test_jit.py
   pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rbigint.py
   pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rdynload.py
   pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rsocket.py
   pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rstring.py
   pypy/branch/jit-unroll-loops/pypy/rlib/test/test_runicode.py
   pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rzlib.py
   pypy/branch/jit-unroll-loops/pypy/rpython/llinterp.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/ll2ctypes.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/llarena.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/llmemory.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/lloperation.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/lltype.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rbuilder.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rclass.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rdict.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rffi.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_ll2ctypes.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_llmemory.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_lltype.py
   pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_rffi.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/base.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/generation.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/markcompact.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/marksweep.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/minimark.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/minimarkpage.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/semispace.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/test/test_direct.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/asmgcroot.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/framework.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/transform.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/gcwrapper.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/support.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/test/test_gc.py
   pypy/branch/jit-unroll-loops/pypy/rpython/memory/test/test_transformed_gc.py
   pypy/branch/jit-unroll-loops/pypy/rpython/module/ll_time.py
   pypy/branch/jit-unroll-loops/pypy/rpython/rbuilder.py
   pypy/branch/jit-unroll-loops/pypy/rpython/rbuiltin.py
   pypy/branch/jit-unroll-loops/pypy/rpython/rpbc.py
   pypy/branch/jit-unroll-loops/pypy/rpython/rtyper.py
   pypy/branch/jit-unroll-loops/pypy/rpython/test/test_llinterp.py
   pypy/branch/jit-unroll-loops/pypy/rpython/test/test_nongc.py
   pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rbuilder.py
   pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rclass.py
   pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rdict.py
   pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rptr.py
   pypy/branch/jit-unroll-loops/pypy/tool/release/   (props changed)
   pypy/branch/jit-unroll-loops/pypy/tool/release/__init__.py   (props changed)
   pypy/branch/jit-unroll-loops/pypy/tool/release/force-builds.py   (props changed)
   pypy/branch/jit-unroll-loops/pypy/tool/release/make_release.py   (props changed)
   pypy/branch/jit-unroll-loops/pypy/tool/release/test/   (props changed)
   pypy/branch/jit-unroll-loops/pypy/tool/release/test/__init__.py   (props changed)
   pypy/branch/jit-unroll-loops/pypy/tool/release/test/test_make_release.py   (props changed)
   pypy/branch/jit-unroll-loops/pypy/translator/c/funcgen.py
   pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/instruction.py
   pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/elf/track5.s
   pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/test_trackgcroot.py
   pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/trackgcroot.py
   pypy/branch/jit-unroll-loops/pypy/translator/c/src/g_include.h
   pypy/branch/jit-unroll-loops/pypy/translator/c/src/main.h
   pypy/branch/jit-unroll-loops/pypy/translator/c/src/signals.h
   pypy/branch/jit-unroll-loops/pypy/translator/c/src/stack.h
   pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_genc.py
   pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_lltyped.py
   pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_newgc.py
   pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_standalone.py
   pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_typed.py
   pypy/branch/jit-unroll-loops/pypy/translator/driver.py
   pypy/branch/jit-unroll-loops/pypy/translator/goal/ann_override.py
   pypy/branch/jit-unroll-loops/pypy/translator/goal/app_main.py
   pypy/branch/jit-unroll-loops/pypy/translator/goal/translate.py
   pypy/branch/jit-unroll-loops/pypy/translator/jvm/test/test_class.py
   pypy/branch/jit-unroll-loops/pypy/translator/oosupport/test_template/class_.py
   pypy/branch/jit-unroll-loops/pypy/translator/platform/darwin.py
   pypy/branch/jit-unroll-loops/pypy/translator/transform.py
Log:
svn merge -r77505:HEAD svn+ssh://hakanardo at codespeak.net/svn/pypy/trunk

Modified: pypy/branch/jit-unroll-loops/dotviewer/drawgraph.py
==============================================================================
--- pypy/branch/jit-unroll-loops/dotviewer/drawgraph.py	(original)
+++ pypy/branch/jit-unroll-loops/dotviewer/drawgraph.py	Sat Oct 30 16:41:17 2010
@@ -423,20 +423,43 @@
         else:
             for line in lines:
                 raw_line = line.replace('\\l','').replace('\r','') or ' '
-                img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
-                w, h = img.get_size()
-                if w>wmax: wmax = w
-                if raw_line.strip():
-                    if line.endswith('\\l'):
-                        def cmd(img=img, y=hmax):
-                            img.draw(xleft, ytop+y)
-                    elif line.endswith('\r'):
-                        def cmd(img=img, y=hmax, w=w):
-                            img.draw(xright-w, ytop+y)
-                    else:
-                        def cmd(img=img, y=hmax, w=w):
-                            img.draw(xcenter-w//2, ytop+y)
+                if '\f' in raw_line:   # grayed out parts of the line
+                    imgs = []
+                    graytext = True
+                    h = 16
+                    w_total = 0
+                    for linepart in raw_line.split('\f'):
+                        graytext = not graytext
+                        if not linepart.strip():
+                            continue
+                        if graytext:
+                            fgcolor = (128, 160, 160)
+                        else:
+                            fgcolor = (0, 0, 0)
+                        img = TextSnippet(self, linepart, fgcolor, bgcolor)
+                        imgs.append((w_total, img))
+                        w, h = img.get_size()
+                        w_total += w
+                    if w_total > wmax: wmax = w_total
+                    def cmd(imgs=imgs, y=hmax):
+                        for x, img in imgs:
+                            img.draw(xleft+x, ytop+y)
                     commands.append(cmd)
+                else:
+                    img = TextSnippet(self, raw_line, (0, 0, 0), bgcolor)
+                    w, h = img.get_size()
+                    if w>wmax: wmax = w
+                    if raw_line.strip():
+                        if line.endswith('\\l'):
+                            def cmd(img=img, y=hmax):
+                                img.draw(xleft, ytop+y)
+                        elif line.endswith('\r'):
+                            def cmd(img=img, y=hmax, w=w):
+                                img.draw(xright-w, ytop+y)
+                        else:
+                            def cmd(img=img, y=hmax, w=w):
+                                img.draw(xcenter-w//2, ytop+y)
+                        commands.append(cmd)
                 hmax += h
                 #hmax += 8
 

Modified: pypy/branch/jit-unroll-loops/lib-python/conftest.py
==============================================================================
--- pypy/branch/jit-unroll-loops/lib-python/conftest.py	(original)
+++ pypy/branch/jit-unroll-loops/lib-python/conftest.py	Sat Oct 30 16:41:17 2010
@@ -144,7 +144,7 @@
     RegrTest('test_binhex.py'),
 
     RegrTest('test_binop.py', core=True),
-    RegrTest('test_bisect.py', core=True),
+    RegrTest('test_bisect.py', core=True, usemodules='_bisect'),
     RegrTest('test_bool.py', core=True),
     RegrTest('test_bsddb.py', skip="unsupported extension module"),
     RegrTest('test_bsddb185.py', skip="unsupported extension module"),
@@ -464,7 +464,12 @@
     RegrTest('test_coding.py'),
     RegrTest('test_complex_args.py'),
     RegrTest('test_contextlib.py', usemodules="thread"),
-    RegrTest('test_ctypes.py', usemodules="_rawffi"),
+    # we skip test ctypes, since we adapted it massively in order
+    # to test what we want to support. There are real failures,
+    # but it's about missing features that we don't want to support
+    # now
+    RegrTest('test_ctypes.py', usemodules="_rawffi",
+             skip="missing features that we don't want to support now"),
     RegrTest('test_defaultdict.py'),
     RegrTest('test_email_renamed.py'),
     RegrTest('test_exception_variations.py'),

Modified: pypy/branch/jit-unroll-loops/lib-python/modified-2.5.2/test/test_descr.py
==============================================================================
--- pypy/branch/jit-unroll-loops/lib-python/modified-2.5.2/test/test_descr.py	(original)
+++ pypy/branch/jit-unroll-loops/lib-python/modified-2.5.2/test/test_descr.py	Sat Oct 30 16:41:17 2010
@@ -2028,7 +2028,9 @@
     except TypeError, msg:
         verify(str(msg).find("weak reference") >= 0)
     else:
-        verify(0, "weakref.ref(no) should be illegal")
+        # in PyPy it is (sometimes) possible to take a weakref here
+        #verify(0, "weakref.ref(no) should be illegal")
+        pass
     class Weak(object):
         __slots__ = ['foo', '__weakref__']
     yes = Weak()

Modified: pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/function.py
==============================================================================
--- pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/function.py	(original)
+++ pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/function.py	Sat Oct 30 16:41:17 2010
@@ -171,7 +171,7 @@
         return self._build_result(restype, resbuffer, argtypes, argsandobjs)
 
     def _getfuncptr(self, argtypes, restype, thisarg=None):
-        if self._ptr is not None:
+        if self._ptr is not None and argtypes is self._argtypes_:
             return self._ptr
         if restype is None or not isinstance(restype, _CDataMeta):
             import ctypes

Modified: pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/structure.py
==============================================================================
--- pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/structure.py	(original)
+++ pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/structure.py	Sat Oct 30 16:41:17 2010
@@ -7,7 +7,7 @@
 def round_up(size, alignment):
     return (size + alignment - 1) & -alignment
 
-def size_alignment_pos(fields):
+def size_alignment_pos(fields, is_union=False):
     import ctypes
     size = 0
     alignment = 1
@@ -15,14 +15,19 @@
     for fieldname, ctype in fields:
         fieldsize = ctypes.sizeof(ctype)
         fieldalignment = ctypes.alignment(ctype)
-        size = round_up(size, fieldalignment)
         alignment = max(alignment, fieldalignment)
-        pos.append(size)
-        size += fieldsize
+        if is_union:
+            pos.append(0)
+            size = max(size, fieldsize)
+        else:
+            size = round_up(size, fieldalignment)
+            pos.append(size)
+            size += fieldsize
     size = round_up(size, alignment)
     return size, alignment, pos
 
-def names_and_fields(_fields_, superclass, zero_offset=False, anon=None):
+def names_and_fields(_fields_, superclass, zero_offset=False, anon=None,
+                     is_union=False):
     if isinstance(_fields_, tuple):
         _fields_ = list(_fields_)
     for _, tp in _fields_:
@@ -36,7 +41,7 @@
     rawfields = [(name, ctype._ffishape)
                  for name, ctype in all_fields]
     if not zero_offset:
-        _, _, pos = size_alignment_pos(all_fields)
+        _, _, pos = size_alignment_pos(all_fields, is_union)
     else:
         pos = [0] * len(all_fields)
     fields = {}
@@ -73,8 +78,8 @@
 
 # ________________________________________________________________
 
-def _set_shape(tp, rawfields):
-    tp._ffistruct = _rawffi.Structure(rawfields)
+def _set_shape(tp, rawfields, is_union=False):
+    tp._ffistruct = _rawffi.Structure(rawfields, is_union)
     tp._ffiargshape = tp._ffishape = (tp._ffistruct, 1)
     tp._fficompositesize = tp._ffistruct.size
 
@@ -92,13 +97,14 @@
             raise AttributeError("Structure or union cannot contain itself")
         self._names, rawfields, self._fieldtypes = names_and_fields(
             value, self.__bases__[0], False,
-            self.__dict__.get('_anonymous_', None))
+            self.__dict__.get('_anonymous_', None), self._is_union)
         _CDataMeta.__setattr__(self, '_fields_', value)
-        _set_shape(self, rawfields)
+        _set_shape(self, rawfields, self._is_union)
         return
     _CDataMeta.__setattr__(self, name, value)
 
-class StructureMeta(_CDataMeta):
+class StructOrUnionMeta(_CDataMeta):
+
     def __new__(self, name, cls, typedict):
         res = type.__new__(self, name, cls, typedict)
         if '_fields_' in typedict:
@@ -109,8 +115,8 @@
                     raise AttributeError("Anonymous field not found")
             res._names, rawfields, res._fieldtypes = names_and_fields(
                 typedict['_fields_'], cls[0], False,
-                typedict.get('_anonymous_', None))
-            _set_shape(res, rawfields)
+                typedict.get('_anonymous_', None), self._is_union)
+            _set_shape(res, rawfields, self._is_union)
 
         return res
 
@@ -150,8 +156,8 @@
         res.__dict__['_index'] = -1
         return res
 
-class Structure(_CData):
-    __metaclass__ = StructureMeta
+class StructOrUnion(_CData):
+    __metaclass__ = StructOrUnionMeta
 
     def __new__(cls, *args, **kwds):
         if not hasattr(cls, '_ffistruct'):
@@ -213,3 +219,10 @@
 
     def _get_buffer_value(self):
         return self._buffer.buffer
+
+
+class StructureMeta(StructOrUnionMeta):
+    _is_union = False
+
+class Structure(StructOrUnion):
+    __metaclass__ = StructureMeta

Modified: pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/union.py
==============================================================================
--- pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/union.py	(original)
+++ pypy/branch/jit-unroll-loops/lib_pypy/_ctypes/union.py	Sat Oct 30 16:41:17 2010
@@ -1,118 +1,7 @@
+from _ctypes import structure
 
+class UnionMeta(structure.StructOrUnionMeta):
+    _is_union = True
 
-import _rawffi
-from _ctypes.basics import _CData, _CDataMeta, store_reference, keepalive_key
-from _ctypes.basics import ensure_objects
-from _ctypes.structure import round_up, names_and_fields, struct_getattr,\
-     struct_setattr
-
-
-def _set_shape(tp):
-    size = tp._sizeofinstances()
-    alignment = tp._alignmentofinstances()
-    tp._ffiopaque = _rawffi.Structure((size, alignment)) # opaque
-    tp._ffiargshape = tp._ffishape = (tp._ffiopaque, 1)
-    tp._fficompositesize = tp._ffiopaque.size
-    # we need to create an array of size one for each
-    # of our elements
-    tp._ffiarrays = {}
-    for name, field in tp._fieldtypes.iteritems():
-        tp._ffiarrays[name] = _rawffi.Array(field.ctype._ffishape)
-        
-class UnionMeta(_CDataMeta):
-    def __new__(self, name, cls, typedict):
-        res = type.__new__(self, name, cls, typedict)
-        if '_fields_' in typedict:
-            res._names, rawfields, res._fieldtypes = names_and_fields(
-                typedict['_fields_'], cls[0], True,
-                typedict.get('_anonymous_', None))
-            _set_shape(res)
-
-        def __init__(self): # don't allow arguments by now
-            if not hasattr(self, '_ffiarrays'):
-                raise TypeError("Cannot instantiate union, has no type")
-            # malloc size
-            size = self.__class__._sizeofinstances()
-            self.__dict__['_objects'] = {}
-            self.__dict__['_buffer'] = self._ffiopaque(autofree=True)
-        res.__init__ = __init__
-        return res
-
-    def _sizeofinstances(self):
-        if not hasattr(self, '_size_'):
-            self._size_ = max([field.size for field in
-                               self._fieldtypes.values()] + [0])
-        return self._size_
-
-    def _alignmentofinstances(self):
-        from ctypes import alignment
-        if not hasattr(self, '_alignment_'):
-            self._alignment_ = max([alignment(field.ctype) for field in
-                                    self._fieldtypes.values()] + [1])
-        return self._alignment_
-    
-    __getattr__ = struct_getattr
-
-    def __setattr__(self, name, value):
-        if name == '_fields_':
-            if self.__dict__.get('_fields_', None):
-                raise AttributeError("_fields_ is final")
-            if self in [v for k, v in value]:
-                raise AttributeError("Union cannot contain itself")
-            self._names, rawfields, self._fieldtypes = names_and_fields(
-                value, self.__bases__[0], True,
-                self.__dict__.get('_anonymous_', None))
-            _CDataMeta.__setattr__(self, '_fields_', value)
-            _set_shape(self)
-        _CDataMeta.__setattr__(self, name, value)
-
-    def _CData_output(self, resarray, base=None, index=-1):
-        res = self.__new__(self)
-        ffiopaque = self._ffiopaque.fromaddress(resarray.buffer)
-        res.__dict__['_buffer'] = ffiopaque
-        res.__dict__['_base'] = base
-        res.__dict__['_index'] = index
-        return res
-    
-    def _CData_retval(self, resbuffer):
-        res = self.__new__(self)
-        res.__dict__['_buffer'] = resbuffer
-        res.__dict__['_base'] = None
-        res.__dict__['_index'] = -1
-        return res
-
-
-class Union(_CData):
+class Union(structure.StructOrUnion):
     __metaclass__ = UnionMeta
-
-    def __getattr__(self, name):
-        try:
-            field = self._fieldtypes[name]
-        except KeyError:
-            raise AttributeError(name)
-        fieldtype = field.ctype
-        val = self._ffiarrays[name].fromaddress(self._buffer.buffer, 1)
-        offset = field.num
-        return fieldtype._CData_output(val, self, offset)
-
-    def __setattr__(self, name, value):
-        try:
-            field = self._fieldtypes[name]
-        except KeyError:
-            raise AttributeError(name)
-        fieldtype = field.ctype
-        cobj = fieldtype.from_param(value)
-        if ensure_objects(cobj) is not None:
-            key = keepalive_key(field.num)
-            store_reference(self, key, cobj._objects)
-        arg = cobj._get_buffer_value()
-        if fieldtype._fficompositesize is not None:
-            from ctypes import memmove
-            dest = self._buffer.buffer
-            memmove(dest, arg, fieldtype._fficompositesize)
-        else:
-            buf = self._ffiarrays[name].fromaddress(self._buffer.buffer, 1)
-            buf[0] = arg
-
-    def _get_buffer_value(self):
-        return self._buffer.buffer

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/bookkeeper.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/bookkeeper.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/bookkeeper.py	Sat Oct 30 16:41:17 2010
@@ -24,7 +24,7 @@
 from pypy.rpython import extregistry
 from pypy.tool.identity_dict import identity_dict
 
-class Stats:
+class Stats(object):
 
     def __init__(self, bookkeeper):
         self.bookkeeper = bookkeeper
@@ -137,7 +137,7 @@
     def consider_dict_delitem(self, dic):
         return dic
 
-class Bookkeeper:
+class Bookkeeper(object):
     """The log of choices that have been made while analysing the operations.
     It ensures that the same 'choice objects' will be returned if we ask
     again during reflowing.  Like ExecutionContext, there is an implicit
@@ -736,7 +736,7 @@
         return True
 
 # for parsing call arguments
-class RPythonCallsSpace:
+class RPythonCallsSpace(object):
     """Pseudo Object Space providing almost no real operation.
     For the Arguments class: if it really needs other operations, it means
     that the call pattern is too complex for R-Python.

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/builtin.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/builtin.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/builtin.py	Sat Oct 30 16:41:17 2010
@@ -423,7 +423,7 @@
 from pypy.annotation.model import SomePtr
 from pypy.rpython.lltypesystem import lltype
 
-def malloc(s_T, s_n=None, s_flavor=None, s_zero=None):
+def malloc(s_T, s_n=None, s_flavor=None, s_zero=None, s_track_allocation=None):
     assert (s_n is None or s_n.knowntype == int
             or issubclass(s_n.knowntype, pypy.rlib.rarithmetic.base_int))
     assert s_T.is_constant()
@@ -438,13 +438,15 @@
         r = SomePtr(lltype.typeOf(p))
     else:
         assert s_flavor.is_constant()
+        assert s_track_allocation is None or s_track_allocation.is_constant()
         # not sure how to call malloc() for the example 'p' in the
         # presence of s_extraargs
         r = SomePtr(lltype.Ptr(s_T.const))
     return r
 
-def free(s_p, s_flavor):
+def free(s_p, s_flavor, s_track_allocation=None):
     assert s_flavor.is_constant()
+    assert s_track_allocation is None or s_track_allocation.is_constant()
     # same problem as in malloc(): some flavors are not easy to
     # malloc-by-example
     #T = s_p.ll_ptrtype.TO

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/classdef.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/classdef.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/classdef.py	Sat Oct 30 16:41:17 2010
@@ -58,7 +58,7 @@
 #        same name in all subclasses of A, if any.  (Parent class attributes can
 #        be visible in reads from instances of subclasses.)
 
-class Attribute:
+class Attribute(object):
     # readonly-ness
     # SomeThing-ness
     # NB.  an attribute is readonly if it is a constant class attribute.
@@ -402,7 +402,7 @@
 
 # ____________________________________________________________
 
-class InstanceSource:
+class InstanceSource(object):
     instance_level = True
 
     def __init__(self, bookkeeper, obj):

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/description.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/description.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/description.py	Sat Oct 30 16:41:17 2010
@@ -6,7 +6,7 @@
 from pypy.tool.sourcetools import valid_identifier
 from pypy.tool.pairtype import extendabletype
 
-class CallFamily:
+class CallFamily(object):
     """A family of Desc objects that could be called from common call sites.
     The call families are conceptually a partition of all (callable) Desc
     objects, where the equivalence relation is the transitive closure of
@@ -51,7 +51,7 @@
             self.total_calltable_size += 1
 
 
-class FrozenAttrFamily:
+class FrozenAttrFamily(object):
     """A family of FrozenDesc objects that have any common 'getattr' sites.
     The attr families are conceptually a partition of FrozenDesc objects,
     where the equivalence relation is the transitive closure of:
@@ -80,7 +80,7 @@
         self.attrs[attrname] = s_value
 
 
-class ClassAttrFamily:
+class ClassAttrFamily(object):
     """A family of ClassDesc objects that have common 'getattr' sites for a
     given attribute name.  The attr families are conceptually a partition
     of ClassDesc objects, where the equivalence relation is the transitive

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/dictdef.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/dictdef.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/dictdef.py	Sat Oct 30 16:41:17 2010
@@ -77,7 +77,7 @@
             dictdef.dictvalue = self
 
 
-class DictDef:
+class DictDef(object):
     """A dict definition remembers how general the keys and values in that
     particular dict have to be.  Every dict creation makes a new DictDef,
     and the union of two dicts merges the DictKeys and DictValues that each

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/listdef.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/listdef.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/listdef.py	Sat Oct 30 16:41:17 2010
@@ -6,7 +6,7 @@
 class TooLateForChange(Exception):
     pass
 
-class ListItem:
+class ListItem(object):
     mutated = False    # True for lists mutated after creation
     resized = False    # True for lists resized after creation
     range_step = None  # the step -- only for lists only created by a range()
@@ -117,7 +117,7 @@
         return updated
 
 
-class ListDef:
+class ListDef(object):
     """A list definition remembers how general the items in that particular
     list have to be.  Every list creation makes a new ListDef, and the union
     of two lists merges the ListItems that each ListDef stores."""

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/model.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/model.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/model.py	Sat Oct 30 16:41:17 2010
@@ -574,11 +574,11 @@
         
 NUMBER = object()
 annotation_to_ll_map = [
+    (SomeSingleFloat(), lltype.SingleFloat),
     (s_None, lltype.Void),   # also matches SomeImpossibleValue()
     (s_Bool, lltype.Bool),
     (SomeInteger(knowntype=r_ulonglong), NUMBER),    
     (SomeFloat(), lltype.Float),
-    (SomeSingleFloat(), lltype.SingleFloat),
     (SomeChar(), lltype.Char),
     (SomeUnicodeCodePoint(), lltype.UniChar),
     (SomeAddress(), llmemory.Address),

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/policy.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/policy.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/policy.py	Sat Oct 30 16:41:17 2010
@@ -1,4 +1,4 @@
-# base annotation policy for overrides and specialization
+# base annotation policy for specialization
 from pypy.annotation.specialize import default_specialize as default
 from pypy.annotation.specialize import specialize_argvalue, specialize_argtype, specialize_arglistitemtype
 from pypy.annotation.specialize import memo
@@ -41,7 +41,7 @@
         if directive is None:
             return pol.default_specialize
 
-        # specialize|override:name[(args)]
+        # specialize[(args)]
         directive_parts = directive.split('(', 1)
         if len(directive_parts) == 1:
             [name] = directive_parts
@@ -60,14 +60,6 @@
         except AttributeError:
             raise AttributeError("%r specialize tag not defined in annotation"
                                  "policy %s" % (name, pol))
-        if directive.startswith('override:'):
-            # different signature: override__xyz(*args_s)
-            if parms:
-                raise Exception, "override:* specialisations don't support parameters"
-            def specialize_override(funcdesc, args_s):
-                funcdesc.overridden = True
-                return specializer(*args_s)
-            return specialize_override
         else:
             if not parms:
                 return specializer
@@ -92,9 +84,5 @@
         from pypy.rpython.annlowlevel import LowLevelAnnotatorPolicy
         return LowLevelAnnotatorPolicy.specialize__ll_and_arg(*args)
 
-    def override__ignore(pol, *args):
-        bk = getbookkeeper()
-        return bk.immutablevalue(None)
-
 class StrictAnnotatorPolicy(AnnotatorPolicy):
     allow_someobjects = False

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/specialize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/specialize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/specialize.py	Sat Oct 30 16:41:17 2010
@@ -100,7 +100,7 @@
 # ____________________________________________________________________________
 # specializations
 
-class MemoTable:
+class MemoTable(object):
     def __init__(self, funcdesc, args, value):
         self.funcdesc = funcdesc
         self.table = {args: value}

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/test/test_annrpython.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/test/test_annrpython.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/test/test_annrpython.py	Sat Oct 30 16:41:17 2010
@@ -1,4 +1,4 @@
-
+from __future__ import with_statement
 import autopath
 import py.test
 import sys
@@ -766,28 +766,6 @@
         s = a.build_types(f, [list])
         assert s.classdef is a.bookkeeper.getuniqueclassdef(IndexError)  # KeyError ignored because l is a list
 
-    def test_overrides(self):
-        excs = []
-        def record_exc(e):
-            """NOT_RPYTHON"""
-            excs.append(sys.exc_info)
-        record_exc._annspecialcase_ = "override:record_exc"
-        def g():
-            pass
-        def f():
-            try:
-                g()
-            except Exception, e:
-                record_exc(e)
-        class MyAnnotatorPolicy(policy.AnnotatorPolicy):
-
-            def override__record_exc(pol, s_e):
-                return a.bookkeeper.immutablevalue(None)
-            
-        a = self.RPythonAnnotator(policy=MyAnnotatorPolicy())
-        s = a.build_types(f, [])
-        assert s.const is None
-
     def test_freeze_protocol(self):
         class Stuff:
             def __init__(self, flag):
@@ -3359,6 +3337,26 @@
         s = a.build_types(f, [])
         assert isinstance(s, annmodel.SomeChar)
 
+    def test_context_manager(self):
+        class C:
+            def __init__(self):
+                pass
+            def __enter__(self):
+                self.x = 1
+            def __exit__(self, *args):
+                self.x = 3
+        def f():
+            c = C()
+            with c:
+                pass
+            return c.x
+
+        a = self.RPythonAnnotator()
+        s = a.build_types(f, [])
+        assert isinstance(s, annmodel.SomeInteger)
+        # not a constant: both __enter__ and __exit__ have been annotated
+        assert not s.is_constant()
+
 
 def g(n):
     return [0,1,2,n]

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/test/test_model.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/test/test_model.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/test/test_model.py	Sat Oct 30 16:41:17 2010
@@ -128,7 +128,7 @@
     assert isinstance(s_p, SomeOOInstance) and s_p.ootype == C
 
 def test_annotation_to_lltype():
-    from pypy.rlib.rarithmetic import r_uint
+    from pypy.rlib.rarithmetic import r_uint, r_singlefloat
     s_i = SomeInteger()
     s_pos = SomeInteger(nonneg=True)
     s_1 = SomeInteger(nonneg=True); s_1.const = 1
@@ -151,6 +151,9 @@
     C = ootype.Instance('C', ROOT, {})
     ref = SomeOOInstance(C)
     assert annotation_to_lltype(ref) == C
+    s_singlefloat = SomeSingleFloat()
+    s_singlefloat.const = r_singlefloat(0.0)
+    assert annotation_to_lltype(s_singlefloat) == lltype.SingleFloat
     
 def test_ll_union():
     PS1 = lltype.Ptr(lltype.GcStruct('s'))

Modified: pypy/branch/jit-unroll-loops/pypy/annotation/unaryop.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/annotation/unaryop.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/annotation/unaryop.py	Sat Oct 30 16:41:17 2010
@@ -434,6 +434,9 @@
     def method_clear(dct):
         pass
 
+    def method_popitem(dct):
+        return dct.getanyitem('items')
+
     def _can_only_throw(dic, *ignore):
         if dic1.dictdef.dictkey.custom_eq_hash:
             return None    # r_dict: can throw anything

Modified: pypy/branch/jit-unroll-loops/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/config/pypyoption.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/config/pypyoption.py	Sat Oct 30 16:41:17 2010
@@ -30,7 +30,8 @@
       "rctime" , "select", "zipimport", "_lsprof",
      "crypt", "signal", "_rawffi", "termios", "zlib",
      "struct", "md5", "sha", "bz2", "_minimal_curses", "cStringIO",
-     "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array"]
+     "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array",
+     "_bisect"]
 ))
 
 working_oo_modules = default_modules.copy()
@@ -73,9 +74,10 @@
     }
 
 module_import_dependencies = {
-    # no _rawffi if importing pypy.rlib.libffi raises ImportError
+    # no _rawffi if importing pypy.rlib.clibffi raises ImportError
     # or CompilationError
-    "_rawffi"   : ["pypy.rlib.libffi"],
+    "_rawffi"   : ["pypy.rlib.clibffi"],
+    "_ffi"      : ["pypy.rlib.clibffi"],
 
     "zlib"      : ["pypy.rlib.rzlib"],
     "bz2"       : ["pypy.module.bz2.interp_bz2"],
@@ -198,6 +200,9 @@
         BoolOption("withstrslice", "use strings optimized for slicing",
                    default=False),
 
+        BoolOption("withstrbuf", "use strings optimized for addition (ver 2)",
+                   default=False),
+
         BoolOption("withprebuiltchar",
                    "use prebuilt single-character string objects",
                    default=False),
@@ -210,7 +215,8 @@
         BoolOption("withrope", "use ropes as the string implementation",
                    default=False,
                    requires=[("objspace.std.withstrslice", False),
-                             ("objspace.std.withstrjoin", False)],
+                             ("objspace.std.withstrjoin", False),
+                             ("objspace.std.withstrbuf", False)],
                    suggests=[("objspace.std.withprebuiltchar", True),
                              ("objspace.std.sharesmallstr", True)]),
 
@@ -224,19 +230,17 @@
                    requires=[("objspace.opcodes.CALL_LIKELY_BUILTIN", False),
                              ("objspace.honor__builtins__", False)]),
 
-        BoolOption("withsharingdict",
-                   "use dictionaries that share the keys part",
-                   default=False),
-
         BoolOption("withdictmeasurement",
                    "create huge files with masses of information "
                    "about dictionaries",
                    default=False),
 
-        BoolOption("withinlineddict",
-                   "make instances more compact by revoming a level of indirection",
+        BoolOption("withmapdict",
+                   "make instances really small but slow without the JIT",
                    default=False,
-                   requires=[("objspace.std.withshadowtracking", False)]),
+                   requires=[("objspace.std.getattributeshortcut", True),
+                             ("objspace.std.withtypeversion", True),
+                       ]),
 
         BoolOption("withrangelist",
                    "enable special range list implementation that does not "
@@ -251,12 +255,6 @@
                    # weakrefs needed, because of get_subclasses()
                    requires=[("translation.rweakref", True)]),
 
-        BoolOption("withshadowtracking",
-                   "track whether an instance attribute shadows a type"
-                   " attribute",
-                   default=False,
-                   requires=[("objspace.std.withtypeversion", True),
-                             ("translation.rweakref", True)]),
         BoolOption("withmethodcache",
                    "try to cache method lookups",
                    default=False,
@@ -328,9 +326,6 @@
         config.objspace.std.suggest(optimized_list_getitem=True)
         config.objspace.std.suggest(getattributeshortcut=True)
         config.objspace.std.suggest(newshortcut=True)        
-        if type_system != 'ootype':
-            config.objspace.std.suggest(withsharingdict=True)
-        config.objspace.std.suggest(withinlineddict=True)
 
     # extra costly optimizations only go in level 3
     if level == '3':
@@ -343,7 +338,7 @@
         config.objspace.std.suggest(withprebuiltint=True)
         config.objspace.std.suggest(withrangelist=True)
         config.objspace.std.suggest(withprebuiltchar=True)
-        config.objspace.std.suggest(withinlineddict=True)
+        config.objspace.std.suggest(withmapdict=True)
         config.objspace.std.suggest(withstrslice=True)
         config.objspace.std.suggest(withstrjoin=True)
         # xxx other options? ropes maybe?
@@ -359,6 +354,7 @@
     # extra optimizations with the JIT
     if level == 'jit':
         config.objspace.std.suggest(withcelldict=True)
+        config.objspace.std.suggest(withmapdict=True)
 
 
 def enable_allworkingmodules(config):

Modified: pypy/branch/jit-unroll-loops/pypy/config/test/test_pypyoption.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/config/test/test_pypyoption.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/config/test/test_pypyoption.py	Sat Oct 30 16:41:17 2010
@@ -47,7 +47,7 @@
 def test_set_pypy_opt_level():
     conf = get_pypy_config()
     set_pypy_opt_level(conf, '2')
-    assert conf.objspace.std.withsharingdict
+    assert conf.objspace.std.newshortcut
     conf = get_pypy_config()
     set_pypy_opt_level(conf, '0')
     assert not conf.objspace.std.newshortcut
@@ -59,7 +59,6 @@
 
     assert not conf.objspace.std.withtypeversion
     assert not conf.objspace.std.withmethodcache
-    assert not conf.objspace.std.withshadowtracking
 
 def test_check_documentation():
     def check_file_exists(fn):

Modified: pypy/branch/jit-unroll-loops/pypy/conftest.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/conftest.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/conftest.py	Sat Oct 30 16:41:17 2010
@@ -7,6 +7,7 @@
 from inspect import isclass, getmro
 from pypy.tool.udir import udir
 from pypy.tool.autopath import pypydir
+from pypy.tool import leakfinder
 
 # pytest settings
 pytest_plugins = "resultlog",
@@ -354,7 +355,14 @@
 
     def runtest(self):
         try:
-            super(IntTestFunction, self).runtest()
+            leakfinder.start_tracking_allocations()
+            try:
+                super(IntTestFunction, self).runtest()
+            finally:
+                if leakfinder.TRACK_ALLOCATIONS:
+                    leaks = leakfinder.stop_tracking_allocations(False)
+                else:
+                    leaks = None   # stop_tracking_allocations() already called
         except OperationError, e:
             check_keyboard_interrupt(e)
             raise
@@ -373,6 +381,8 @@
                 _pygame_imported = True
                 assert option.view, ("should not invoke Pygame "
                                      "if conftest.option.view is False")
+        if leaks:        # check for leaks, but only if the test passed so far
+            raise leakfinder.MallocMismatch(leaks)
 
 class AppTestFunction(PyPyTestFunction):
     def _prunetraceback(self, traceback):

Modified: pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt	(original)
+++ pypy/branch/jit-unroll-loops/pypy/doc/config/objspace.opcodes.CALL_METHOD.txt	Sat Oct 30 16:41:17 2010
@@ -5,8 +5,6 @@
 case.  So far, this only works for calls with no keyword, no ``*arg``
 and no ``**arg`` but it would be easy to extend.
 
-Gives the best results combined with :config:`objspace.std.withshadowtracking`.
-
 For more information, see the section in `Standard Interpreter Optimizations`_.
 
 .. _`Standard Interpreter Optimizations`: ../interpreter-optimizations.html#lookup-method-call-method

Modified: pypy/branch/jit-unroll-loops/pypy/doc/docindex.txt
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/doc/docindex.txt	(original)
+++ pypy/branch/jit-unroll-loops/pypy/doc/docindex.txt	Sat Oct 30 16:41:17 2010
@@ -84,7 +84,7 @@
 
 PyPy's own tests `summary`_, daily updated, run through BuildBot infrastructure.
 You can also find CPython's compliance tests run with compiled ``pypy-c``
-exeuctables there.
+executables there.
 
 information dating from early 2007: 
 

Modified: pypy/branch/jit-unroll-loops/pypy/doc/interpreter-optimizations.txt
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/doc/interpreter-optimizations.txt	(original)
+++ pypy/branch/jit-unroll-loops/pypy/doc/interpreter-optimizations.txt	Sat Oct 30 16:41:17 2010
@@ -153,8 +153,8 @@
 dicts:
 the representation of the instance dict contains only a list of values.
 
-You can enable this feature with the :config:`objspace.std.withsharingdict`
-option.
+A more advanced version of sharing dicts, called *map dicts,* is available
+with the :config:`objspace.std.withmapdict` option.
 
 Builtin-Shadowing
 +++++++++++++++++
@@ -219,8 +219,7 @@
 shadowing the class attribute. If we know that there is no shadowing (since
 instance dict tells us that) we can save this lookup on the instance dictionary.
 
-You can enable this feature with the :config:`objspace.std.withshadowtracking`
-option.
+*This was deprecated and is no longer available.*
 
 
 Method Caching

Modified: pypy/branch/jit-unroll-loops/pypy/interpreter/astcompiler/test/test_compiler.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/interpreter/astcompiler/test/test_compiler.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/interpreter/astcompiler/test/test_compiler.py	Sat Oct 30 16:41:17 2010
@@ -41,7 +41,7 @@
         co_expr = compile(evalexpr, '', 'eval')
         space = self.space
         pyco_expr = PyCode._from_code(space, co_expr)
-        w_res = pyco_expr.exec_code(space, w_dict, w_dict)
+        w_res = pyco_expr.exec_host_bytecode(space, w_dict, w_dict)
         res = space.str_w(space.repr(w_res))
         if not isinstance(expected, float):
             assert res == repr(expected)

Modified: pypy/branch/jit-unroll-loops/pypy/interpreter/baseobjspace.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/interpreter/baseobjspace.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/interpreter/baseobjspace.py	Sat Oct 30 16:41:17 2010
@@ -36,13 +36,10 @@
             return space.finditem_str(w_dict, attr)
         return None
 
-    def getdictvalue_attr_is_in_class(self, space, attr):
-        return self.getdictvalue(space, attr)
-
-    def setdictvalue(self, space, attr, w_value, shadows_type=True):
+    def setdictvalue(self, space, attr, w_value):
         w_dict = self.getdict()
         if w_dict is not None:
-            space.setitem_str(w_dict, attr, w_value, shadows_type)
+            space.setitem_str(w_dict, attr, w_value)
             return True
         return False
 
@@ -168,6 +165,20 @@
     def _call_builtin_destructor(self):
         pass     # method overridden in typedef.py
 
+    # hooks that the mapdict implementations needs:
+    def _get_mapdict_map(self):
+        return None
+    def _set_mapdict_map(self, map):
+        raise NotImplementedError
+    def _mapdict_read_storage(self, index):
+        raise NotImplementedError
+    def _mapdict_write_storage(self, index, value):
+        raise NotImplementedError
+    def _mapdict_storage_length(self):
+        raise NotImplementedError
+    def _set_mapdict_storage_and_map(self, storage, map):
+        raise NotImplementedError
+
 
 class Wrappable(W_Root):
     """A subclass of Wrappable is an internal, interpreter-level class
@@ -643,7 +654,7 @@
         """shortcut for space.int_w(space.hash(w_obj))"""
         return self.int_w(self.hash(w_obj))
 
-    def setitem_str(self, w_obj, key, w_value, shadows_type=True):
+    def setitem_str(self, w_obj, key, w_value):
         return self.setitem(w_obj, self.wrap(key), w_value)
 
     def finditem_str(self, w_obj, key):
@@ -725,7 +736,7 @@
 
     def unpackiterable(self, w_iterable, expected_length=-1):
         """Unpack an iterable object into a real (interpreter-level) list.
-        Raise a real (subclass of) ValueError if the length is wrong."""
+        Raise an OperationError(w_ValueError) if the length is wrong."""
         w_iterator = self.iter(w_iterable)
         items = []
         while True:

Modified: pypy/branch/jit-unroll-loops/pypy/interpreter/function.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/interpreter/function.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/interpreter/function.py	Sat Oct 30 16:41:17 2010
@@ -430,8 +430,11 @@
         self.w_class = w_class         # possibly space.w_None
 
     def descr_method__new__(space, w_subtype, w_function, w_instance, w_class=None):
-        if space.is_w( w_instance, space.w_None ):
+        if space.is_w(w_instance, space.w_None):
             w_instance = None
+        if w_instance is None and space.is_w(w_class, space.w_None):
+            raise OperationError(space.w_TypeError,
+                                 space.wrap("unbound methods must have class"))
         method = space.allocate_instance(Method, w_subtype)
         Method.__init__(method, space, w_function, w_instance, w_class)
         return space.wrap(method)
@@ -586,12 +589,14 @@
             w_klass = space.type(w_obj)
         return space.wrap(Method(space, self.w_function, w_klass, space.w_None))
 
-    def descr_classmethod__new__(space, w_type, w_function):
+    def descr_classmethod__new__(space, w_subtype, w_function):
         if not space.is_true(space.callable(w_function)):
             typename = space.type(w_function).getname(space, '?')
             raise operationerrfmt(space.w_TypeError,
                                   "'%s' object is not callable", typename)
-        return space.wrap(ClassMethod(w_function))
+        instance = space.allocate_instance(ClassMethod, w_subtype)
+        instance.__init__(w_function)
+        return space.wrap(instance)
 
 class FunctionWithFixedCode(Function):
     can_change_code = False

Modified: pypy/branch/jit-unroll-loops/pypy/interpreter/gateway.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/interpreter/gateway.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/interpreter/gateway.py	Sat Oct 30 16:41:17 2010
@@ -28,7 +28,7 @@
 # internal non-translatable parts: 
 import py
 
-class SignatureBuilder:
+class SignatureBuilder(object):
     "NOT_RPYTHON"
     def __init__(self, func=None, argnames=None, varargname=None,
                  kwargname=None, name = None):
@@ -51,7 +51,7 @@
 
 #________________________________________________________________
 
-class UnwrapSpecRecipe:
+class UnwrapSpecRecipe(object):
     "NOT_RPYTHON"
 
     bases_order = [Wrappable, W_Root, ObjSpace, Arguments, object]

Modified: pypy/branch/jit-unroll-loops/pypy/interpreter/pycode.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/interpreter/pycode.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/interpreter/pycode.py	Sat Oct 30 16:41:17 2010
@@ -117,6 +117,10 @@
 
         self._compute_flatcall()
 
+        if self.space.config.objspace.std.withmapdict:
+            from pypy.objspace.std.mapdict import init_mapdict_cache
+            init_mapdict_cache(self)
+
     def _freeze_(self):
         if (self.magic == cpython_magic and
             '__pypy__' not in sys.builtin_module_names):
@@ -253,6 +257,12 @@
                          tuple(self.co_freevars),
                          tuple(self.co_cellvars) )
 
+    def exec_host_bytecode(self, w_dict, w_globals, w_locals):
+        from pypy.interpreter.pyframe import CPythonFrame
+        frame = CPythonFrame(self.space, self, w_globals, None)
+        frame.setdictscope(w_locals)
+        return frame.run()
+
     def dump(self):
         """A dis.dis() dump of the code object."""
         co = self._to_code()

Modified: pypy/branch/jit-unroll-loops/pypy/interpreter/pyframe.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/interpreter/pyframe.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/interpreter/pyframe.py	Sat Oct 30 16:41:17 2010
@@ -13,6 +13,7 @@
 from pypy.rlib.rarithmetic import intmask
 from pypy.rlib import jit, rstack
 from pypy.tool import stdlib_opcode
+from pypy.tool.stdlib_opcode import host_bytecode_spec
 
 # Define some opcodes used
 g = globals()
@@ -140,7 +141,8 @@
         # the following 'assert' is an annotation hint: it hides from
         # the annotator all methods that are defined in PyFrame but
         # overridden in the {,Host}FrameClass subclasses of PyFrame.
-        assert isinstance(self, self.space.FrameClass)
+        assert (isinstance(self, self.space.FrameClass) or
+                not self.space.config.translating)
         executioncontext = self.space.getexecutioncontext()
         executioncontext.enter(self)
         try:
@@ -634,6 +636,18 @@
             return space.wrap(self.builtin is not space.builtin)
         return space.w_False
 
+class CPythonFrame(PyFrame):
+    """
+    Execution of host (CPython) opcodes.
+    """
+
+    bytecode_spec = host_bytecode_spec
+    opcode_method_names = host_bytecode_spec.method_names
+    opcodedesc = host_bytecode_spec.opcodedesc
+    opdescmap = host_bytecode_spec.opdescmap
+    HAVE_ARGUMENT = host_bytecode_spec.HAVE_ARGUMENT
+
+
 # ____________________________________________________________
 
 def get_block_class(opname):

Modified: pypy/branch/jit-unroll-loops/pypy/interpreter/pyopcode.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/interpreter/pyopcode.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/interpreter/pyopcode.py	Sat Oct 30 16:41:17 2010
@@ -164,6 +164,9 @@
             next_instr = block.handle(self, unroller)
             return next_instr
 
+    def call_contextmanager_exit_function(self, w_func, w_typ, w_val, w_tb):
+        return self.space.call_function(w_func, w_typ, w_val, w_tb)
+
     @jit.unroll_safe
     def dispatch_bytecode(self, co_code, next_instr, ec):
         space = self.space
@@ -710,9 +713,14 @@
 
     def LOAD_ATTR(self, nameindex, next_instr):
         "obj.attributename"
-        w_attributename = self.getname_w(nameindex)
         w_obj = self.popvalue()
-        w_value = self.space.getattr(w_obj, w_attributename)
+        if (self.space.config.objspace.std.withmapdict
+            and not jit.we_are_jitted()):
+            from pypy.objspace.std.mapdict import LOAD_ATTR_caching
+            w_value = LOAD_ATTR_caching(self.getcode(), w_obj, nameindex)
+        else:
+            w_attributename = self.getname_w(nameindex)
+            w_value = self.space.getattr(w_obj, w_attributename)
         self.pushvalue(w_value)
     LOAD_ATTR._always_inline_ = True
 
@@ -868,23 +876,43 @@
 
     def WITH_CLEANUP(self, oparg, next_instr):
         # see comment in END_FINALLY for stack state
-        w_exitfunc = self.popvalue()
-        w_unroller = self.peekvalue(2)
+        # This opcode changed a lot between CPython versions
+        if (self.pycode.magic >= 0xa0df2ef
+            # Implementation since 2.7a0: 62191 (introduce SETUP_WITH)
+            or self.pycode.magic >= 0xa0df2d1):
+            # implementation since 2.6a1: 62161 (WITH_CLEANUP optimization)
+            self.popvalue()
+            self.popvalue()
+            w_unroller = self.popvalue()
+            w_exitfunc = self.popvalue()
+            self.pushvalue(w_unroller)
+            self.pushvalue(self.space.w_None)
+            self.pushvalue(self.space.w_None)
+        elif self.pycode.magic >= 0xa0df28c:
+            # Implementation since 2.5a0: 62092 (changed WITH_CLEANUP opcode)
+            w_exitfunc = self.popvalue()
+            w_unroller = self.peekvalue(2)
+        else:
+            raise NotImplementedError("WITH_CLEANUP for CPython <= 2.4")
+
         unroller = self.space.interpclass_w(w_unroller)
         if isinstance(unroller, SApplicationException):
             operr = unroller.operr
-            w_result = self.space.call_function(w_exitfunc,
-                                                operr.w_type,
-                                                operr.get_w_value(self.space),
-                                                operr.application_traceback)
+            w_traceback = self.space.wrap(operr.application_traceback)
+            w_result = self.call_contextmanager_exit_function(
+                w_exitfunc,
+                operr.w_type,
+                operr.get_w_value(self.space),
+                w_traceback)
             if self.space.is_true(w_result):
                 # __exit__() returned True -> Swallow the exception.
                 self.settopvalue(self.space.w_None, 2)
         else:
-            self.space.call_function(w_exitfunc,
-                                     self.space.w_None,
-                                     self.space.w_None,
-                                     self.space.w_None)
+            self.call_contextmanager_exit_function(
+                w_exitfunc,
+                self.space.w_None,
+                self.space.w_None,
+                self.space.w_None)
 
     @jit.unroll_safe
     def call_function(self, oparg, w_star=None, w_starstar=None):
@@ -1225,10 +1253,18 @@
         frame.pushvalue(frame.space.w_None)
         return self.handlerposition   # jump to the handler
 
+class WithBlock(FinallyBlock):
+
+    def really_handle(self, frame, unroller):
+        if (frame.space.full_exceptions and
+            isinstance(unroller, SApplicationException)):
+            unroller.operr.normalize_exception(frame.space)
+        return FinallyBlock.really_handle(self, frame, unroller)
 
 block_classes = {'SETUP_LOOP': LoopBlock,
                  'SETUP_EXCEPT': ExceptBlock,
-                 'SETUP_FINALLY': FinallyBlock}
+                 'SETUP_FINALLY': FinallyBlock,
+                 'SETUP_WITH': WithBlock}
 
 ### helpers written at the application-level ###
 # Some of these functions are expected to be generally useful if other

Modified: pypy/branch/jit-unroll-loops/pypy/interpreter/pyparser/pytokenize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/interpreter/pyparser/pytokenize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/interpreter/pyparser/pytokenize.py	Sat Oct 30 16:41:17 2010
@@ -21,40 +21,42 @@
 __all__ = [ "tokenize" ]
 
 # ______________________________________________________________________
-# Automatically generated DFA's (with one or two hand tweeks):
-pseudoStatesAccepts = [True, True, True, True, True, True, True, True,
-                       True, True, False, True, True, True, False, False,
-                       False, False, True, False, False, True, True, False,
-                       True, False, True, False, True, False, True, False,
-                       False, False, True, False, False, False, True]
-
-pseudoStates = [
-    {'\t': 0, '\n': 13, '\x0c': 0, '\r': 14, ' ': 0, '!': 10,
-     '"': 16, '#': 18, '%': 12, '&': 12,
-     "'": 15, '(': 13, ')': 13, '*': 7,
-     '+': 12, ',': 13, '-': 12, '.': 6,
-     '/': 11, '0': 4, '1': 5, '2': 5,
-     '3': 5, '4': 5, '5': 5, '6': 5,
-     '7': 5, '8': 5, '9': 5, ':': 13,
-     ';': 13, '<': 9, '=': 12, '>': 8,
-     '@': 13, 'A': 1,
-     'B': 1, 'C': 1, 'D': 1, 'E': 1,
-     'F': 1, 'G': 1, 'H': 1, 'I': 1,
-     'J': 1, 'K': 1, 'L': 1, 'M': 1,
-     'N': 1, 'O': 1, 'P': 1, 'Q': 1,
-     'R': 2, 'S': 1, 'T': 1, 'U': 3,
-     'V': 1, 'W': 1, 'X': 1, 'Y': 1,
-     'Z': 1, '[': 13, '\\': 17, ']': 13,
-     '^': 12, '_': 1, '`': 13, 'a': 1,
-     'b': 1, 'c': 1, 'd': 1, 'e': 1,
-     'f': 1, 'g': 1, 'h': 1, 'i': 1,
-     'j': 1, 'k': 1, 'l': 1, 'm': 1,
-     'n': 1, 'o': 1, 'p': 1, 'q': 1,
-     'r': 2, 's': 1, 't': 1, 'u': 3,
-     'v': 1, 'w': 1, 'x': 1, 'y': 1,
-     'z': 1, '{': 13, '|': 12, '}': 13,
-     '~': 13},
+# Automatically generated DFA's
 
+accepts = [True, True, True, True, True, True, True, True,
+           True, True, False, True, True, True, False, False,
+           False, False, True, True, True, False, True,
+           False, True, False, True, False, False, True,
+           False, False, False, False, True, False, False,
+           False, True]
+states = [
+    # 0
+    {'\t': 0, '\n': 13, '\x0c': 0,
+     '\r': 14, ' ': 0, '!': 10, '"': 16,
+     '#': 18, '%': 12, '&': 12, "'": 15,
+     '(': 13, ')': 13, '*': 7, '+': 12,
+     ',': 13, '-': 12, '.': 6, '/': 11,
+     '0': 4, '1': 5, '2': 5, '3': 5,
+     '4': 5, '5': 5, '6': 5, '7': 5,
+     '8': 5, '9': 5, ':': 13, ';': 13,
+     '<': 9, '=': 12, '>': 8, '@': 13,
+     'A': 1, 'B': 1, 'C': 1, 'D': 1,
+     'E': 1, 'F': 1, 'G': 1, 'H': 1,
+     'I': 1, 'J': 1, 'K': 1, 'L': 1,
+     'M': 1, 'N': 1, 'O': 1, 'P': 1,
+     'Q': 1, 'R': 2, 'S': 1, 'T': 1,
+     'U': 3, 'V': 1, 'W': 1, 'X': 1,
+     'Y': 1, 'Z': 1, '[': 13, '\\': 17,
+     ']': 13, '^': 12, '_': 1, '`': 13,
+     'a': 1, 'b': 1, 'c': 1, 'd': 1,
+     'e': 1, 'f': 1, 'g': 1, 'h': 1,
+     'i': 1, 'j': 1, 'k': 1, 'l': 1,
+     'm': 1, 'n': 1, 'o': 1, 'p': 1,
+     'q': 1, 'r': 2, 's': 1, 't': 1,
+     'u': 3, 'v': 1, 'w': 1, 'x': 1,
+     'y': 1, 'z': 1, '{': 13, '|': 12,
+     '}': 13, '~': 13},
+    # 1
     {'0': 1, '1': 1, '2': 1, '3': 1,
      '4': 1, '5': 1, '6': 1, '7': 1,
      '8': 1, '9': 1, 'A': 1, 'B': 1,
@@ -71,7 +73,7 @@
      'p': 1, 'q': 1, 'r': 1, 's': 1,
      't': 1, 'u': 1, 'v': 1, 'w': 1,
      'x': 1, 'y': 1, 'z': 1},
-
+    # 2
     {'"': 16, "'": 15, '0': 1, '1': 1,
      '2': 1, '3': 1, '4': 1, '5': 1,
      '6': 1, '7': 1, '8': 1, '9': 1,
@@ -89,7 +91,7 @@
      'r': 1, 's': 1, 't': 1, 'u': 1,
      'v': 1, 'w': 1, 'x': 1, 'y': 1,
      'z': 1},
-
+    # 3
     {'"': 16, "'": 15, '0': 1, '1': 1,
      '2': 1, '3': 1, '4': 1, '5': 1,
      '6': 1, '7': 1, '8': 1, '9': 1,
@@ -107,158 +109,182 @@
      'r': 2, 's': 1, 't': 1, 'u': 1,
      'v': 1, 'w': 1, 'x': 1, 'y': 1,
      'z': 1},
-
-    {'.': 24, '0': 22, '1': 22, '2': 22,
-     '3': 22, '4': 22, '5': 22, '6': 22,
-     '7': 22, '8': 23, '9': 23, 'E': 25,
-     'J': 13, 'L': 13, 'X': 21, 'e': 25,
-     'j': 13, 'l': 13, 'x': 21},
-
-    {'.': 24, '0': 5, '1': 5, '2': 5,
+    # 4
+    {'.': 22, '0': 20, '1': 20, '2': 20,
+     '3': 20, '4': 20, '5': 20, '6': 20,
+     '7': 20, '8': 21, '9': 21, 'E': 23,
+     'J': 13, 'L': 13, 'X': 19, 'e': 23,
+     'j': 13, 'l': 13, 'x': 19},
+    # 5
+    {'.': 22, '0': 5, '1': 5, '2': 5,
      '3': 5, '4': 5, '5': 5, '6': 5,
-     '7': 5, '8': 5, '9': 5, 'E': 25,
-     'J': 13, 'L': 13, 'e': 25, 'j': 13,
+     '7': 5, '8': 5, '9': 5, 'E': 23,
+     'J': 13, 'L': 13, 'e': 23, 'j': 13,
      'l': 13},
-
-    {'0': 26, '1': 26, '2': 26, '3': 26,
-     '4': 26, '5': 26, '6': 26, '7': 26,
-     '8': 26, '9': 26},
-
+    # 6
+    {'0': 24, '1': 24, '2': 24, '3': 24,
+     '4': 24, '5': 24, '6': 24, '7': 24,
+     '8': 24, '9': 24},
+    # 7
     {'*': 12, '=': 13},
-
+    # 8
     {'=': 13, '>': 12},
-
-    {'=': 13, '<': 12, '>': 13},
-
+    # 9
+    {'<': 12, '=': 13, '>': 13},
+    # 10
     {'=': 13},
-
-    {'=': 13, '/': 12},
-
+    # 11
+    {'/': 12, '=': 13},
+    # 12
     {'=': 13},
-
+    # 13
     {},
-
+    # 14
     {'\n': 13},
-
-    {automata.DEFAULT: 19, '\n': 27, '\\': 29, "'": 28},
-
-    {automata.DEFAULT: 20, '"': 30, '\n': 27, '\\': 31},
-
+    # 15
+    {automata.DEFAULT: 28, '\n': 25, "'": 26, '\\': 27},
+    # 16
+    {automata.DEFAULT: 31, '\n': 25, '"': 29, '\\': 30},
+    # 17
     {'\n': 13, '\r': 14},
-
-    {automata.DEFAULT: 18, '\n': 27, '\r': 27},
-
-    {automata.DEFAULT: 19, '\n': 27, '\\': 29, "'": 13},
-
-    {automata.DEFAULT: 20, '"': 13, '\n': 27, '\\': 31},
-                
-    {'0': 21, '1': 21, '2': 21, '3': 21,
-     '4': 21, '5': 21, '6': 21, '7': 21,
-     '8': 21, '9': 21, 'A': 21, 'B': 21,
-     'C': 21, 'D': 21, 'E': 21, 'F': 21,
-     'L': 13, 'a': 21, 'b': 21, 'c': 21,
-     'd': 21, 'e': 21, 'f': 21, 'l': 13},
-    
-    {'.': 24, '0': 22, '1': 22, '2': 22,
-     '3': 22, '4': 22, '5': 22, '6': 22,
-     '7': 22, '8': 23, '9': 23, 'E': 25,
-     'J': 13, 'L': 13, 'e': 25, 'j': 13,
+    # 18
+    {automata.DEFAULT: 18, '\n': 25, '\r': 25},
+    # 19
+    {'0': 19, '1': 19, '2': 19, '3': 19,
+     '4': 19, '5': 19, '6': 19, '7': 19,
+     '8': 19, '9': 19, 'A': 19, 'B': 19,
+     'C': 19, 'D': 19, 'E': 19, 'F': 19,
+     'L': 13, 'a': 19, 'b': 19, 'c': 19,
+     'd': 19, 'e': 19, 'f': 19, 'l': 13},
+    # 20
+    {'.': 22, '0': 20, '1': 20, '2': 20,
+     '3': 20, '4': 20, '5': 20, '6': 20,
+     '7': 20, '8': 21, '9': 21, 'E': 23,
+     'J': 13, 'L': 13, 'e': 23, 'j': 13,
      'l': 13},
-    
-    {'.': 24, '0': 23, '1': 23, '2': 23,
-     '3': 23, '4': 23, '5': 23, '6': 23,
-     '7': 23, '8': 23, '9': 23, 'E': 25,
-     'J': 13, 'e': 25, 'j': 13},
-    
-    {'0': 24, '1': 24, '2': 24, '3': 24,
-     '4': 24, '5': 24, '6': 24, '7': 24,
-     '8': 24, '9': 24, 'E': 32, 'J': 13,
+    # 21
+    {'.': 22, '0': 21, '1': 21, '2': 21,
+     '3': 21, '4': 21, '5': 21, '6': 21,
+     '7': 21, '8': 21, '9': 21, 'E': 23,
+     'J': 13, 'e': 23, 'j': 13},
+    # 22
+    {'0': 22, '1': 22, '2': 22, '3': 22,
+     '4': 22, '5': 22, '6': 22, '7': 22,
+     '8': 22, '9': 22, 'E': 32, 'J': 13,
      'e': 32, 'j': 13},
-    
+    # 23
     {'+': 33, '-': 33, '0': 34, '1': 34,
      '2': 34, '3': 34, '4': 34, '5': 34,
      '6': 34, '7': 34, '8': 34, '9': 34},
-    
-    {'0': 26, '1': 26, '2': 26, '3': 26,
-     '4': 26, '5': 26, '6': 26, '7': 26,
-     '8': 26, '9': 26, 'E': 32, 'J': 13,
+    # 24
+    {'0': 24, '1': 24, '2': 24, '3': 24,
+     '4': 24, '5': 24, '6': 24, '7': 24,
+     '8': 24, '9': 24, 'E': 32, 'J': 13,
      'e': 32, 'j': 13},
-    
+    # 25
     {},
-
+    # 26
     {"'": 13},
-
+    # 27
     {automata.DEFAULT: 35, '\n': 13, '\r': 14},
-
+    # 28
+    {automata.DEFAULT: 28, '\n': 25, "'": 13, '\\': 27},
+    # 29
     {'"': 13},
-
+    # 30
     {automata.DEFAULT: 36, '\n': 13, '\r': 14},
-
+    # 31
+    {automata.DEFAULT: 31, '\n': 25, '"': 13, '\\': 30},
+    # 32
     {'+': 37, '-': 37, '0': 38, '1': 38,
      '2': 38, '3': 38, '4': 38, '5': 38,
      '6': 38, '7': 38, '8': 38, '9': 38},
-    
-    
+    # 33
     {'0': 34, '1': 34, '2': 34, '3': 34,
      '4': 34, '5': 34, '6': 34, '7': 34,
      '8': 34, '9': 34},
-    
+    # 34
     {'0': 34, '1': 34, '2': 34, '3': 34,
      '4': 34, '5': 34, '6': 34, '7': 34,
      '8': 34, '9': 34, 'J': 13, 'j': 13},
-    
-    {automata.DEFAULT: 35, '\n': 27, '\\': 29, "'": 13},
-    
-    {automata.DEFAULT: 36, '"': 13, '\n': 27, '\\': 31},
-    
+    # 35
+    {automata.DEFAULT: 35, '\n': 25, "'": 13, '\\': 27},
+    # 36
+    {automata.DEFAULT: 36, '\n': 25, '"': 13, '\\': 30},
+    # 37
     {'0': 38, '1': 38, '2': 38, '3': 38,
      '4': 38, '5': 38, '6': 38, '7': 38,
      '8': 38, '9': 38},
-
+    # 38
     {'0': 38, '1': 38, '2': 38, '3': 38,
      '4': 38, '5': 38, '6': 38, '7': 38,
      '8': 38, '9': 38, 'J': 13, 'j': 13},
     ]
+pseudoDFA = automata.DFA(states, accepts)
 
-pseudoDFA = automata.DFA(pseudoStates, pseudoStatesAccepts)
-
-double3StatesAccepts = [False, False, False, False, False, True]
-double3States = [
+accepts = [False, False, False, False, False, True]
+states = [
+    # 0
     {automata.DEFAULT: 0, '"': 1, '\\': 2},
+    # 1
     {automata.DEFAULT: 4, '"': 3, '\\': 2},
+    # 2
     {automata.DEFAULT: 4},
+    # 3
     {automata.DEFAULT: 4, '"': 5, '\\': 2},
+    # 4
     {automata.DEFAULT: 4, '"': 1, '\\': 2},
+    # 5
     {automata.DEFAULT: 4, '"': 5, '\\': 2},
     ]
-double3DFA = automata.NonGreedyDFA(double3States, double3StatesAccepts)
+double3DFA = automata.NonGreedyDFA(states, accepts)
 
-single3StatesAccepts = [False, False, False, False, False, True]
-single3States = [
-    {automata.DEFAULT: 0, '\\': 2, "'": 1},
-    {automata.DEFAULT: 4, '\\': 2, "'": 3},
+accepts = [False, False, False, False, False, True]
+states = [
+    # 0
+    {automata.DEFAULT: 0, "'": 1, '\\': 2},
+    # 1
+    {automata.DEFAULT: 4, "'": 3, '\\': 2},
+    # 2
     {automata.DEFAULT: 4},
-    {automata.DEFAULT: 4, '\\': 2, "'": 5},
-    {automata.DEFAULT: 4, '\\': 2, "'": 1},
-    {automata.DEFAULT: 4, '\\': 2, "'": 5},
+    # 3
+    {automata.DEFAULT: 4, "'": 5, '\\': 2},
+    # 4
+    {automata.DEFAULT: 4, "'": 1, '\\': 2},
+    # 5
+    {automata.DEFAULT: 4, "'": 5, '\\': 2},
     ]
-single3DFA = automata.NonGreedyDFA(single3States, single3StatesAccepts)
+single3DFA = automata.NonGreedyDFA(states, accepts)
 
-singleStatesAccepts = [False, True, False]
-singleStates = [
-    {automata.DEFAULT: 0, '\\': 2, "'": 1},
+accepts = [False, True, False, False]
+states = [
+    # 0
+    {automata.DEFAULT: 0, "'": 1, '\\': 2},
+    # 1
     {},
-    {automata.DEFAULT: 0},
+    # 2
+    {automata.DEFAULT: 3},
+    # 3
+    {automata.DEFAULT: 3, "'": 1, '\\': 2},
     ]
-singleDFA = automata.DFA(singleStates, singleStatesAccepts)
+singleDFA = automata.DFA(states, accepts)
 
-doubleStatesAccepts = [False, True, False]
-doubleStates = [
+accepts = [False, True, False, False]
+states = [
+    # 0
     {automata.DEFAULT: 0, '"': 1, '\\': 2},
+    # 1
     {},
-    {automata.DEFAULT: 0},
+    # 2
+    {automata.DEFAULT: 3},
+    # 3
+    {automata.DEFAULT: 3, '"': 1, '\\': 2},
     ]
-doubleDFA = automata.DFA(doubleStates, doubleStatesAccepts)
+doubleDFA = automata.DFA(states, accepts)
+
+
+#_______________________________________________________________________
+# End of automatically generated DFA's
 
 endDFAs = {"'" : singleDFA,
            '"' : doubleDFA,

Modified: pypy/branch/jit-unroll-loops/pypy/interpreter/test/test_function.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/interpreter/test/test_function.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/interpreter/test/test_function.py	Sat Oct 30 16:41:17 2010
@@ -482,6 +482,11 @@
         raises(TypeError, m, MyInst(None))
         raises(TypeError, m, MyInst(42))
 
+    def test_invalid_creation(self):
+        import new
+        def f(): pass
+        raises(TypeError, new.instancemethod, f, None)
+
 
 class TestMethod: 
     def setup_method(self, method):

Modified: pypy/branch/jit-unroll-loops/pypy/interpreter/typedef.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/interpreter/typedef.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/interpreter/typedef.py	Sat Oct 30 16:41:17 2010
@@ -133,6 +133,13 @@
     typedef = cls.typedef
     if wants_dict and typedef.hasdict:
         wants_dict = False
+    if config.objspace.std.withmapdict and not typedef.hasdict:
+        # mapdict only works if the type does not already have a dict
+        if wants_del:
+            parentcls = get_unique_interplevel_subclass(config, cls, True, True,
+                                                        False, True)
+            return _usersubclswithfeature(config, parentcls, "del")
+        return _usersubclswithfeature(config, cls, "user", "dict", "weakref", "slots")
     # Forest of if's - see the comment above.
     if wants_del:
         if wants_dict:
@@ -186,10 +193,21 @@
 
     def add(Proto):
         for key, value in Proto.__dict__.items():
-            if not key.startswith('__') or key == '__del__':
+            if (not key.startswith('__') and not key.startswith('_mixin_') 
+                    or key == '__del__'):
+                if hasattr(value, "func_name"):
+                    value = func_with_new_name(value, value.func_name)
                 body[key] = value
 
+    if (config.objspace.std.withmapdict and "dict" in features):
+        from pypy.objspace.std.mapdict import BaseMapdictObject, ObjectMixin
+        add(BaseMapdictObject)
+        add(ObjectMixin)
+        body["user_overridden_class"] = True
+        features = ()
+
     if "user" in features:     # generic feature needed by all subcls
+
         class Proto(object):
             user_overridden_class = True
 
@@ -245,16 +263,10 @@
                 return self.slots_w[index]
         add(Proto)
 
-    wantdict = "dict" in features
-    if wantdict and config.objspace.std.withinlineddict:
-        from pypy.objspace.std.objectobject import W_ObjectObject
-        from pypy.objspace.std.inlinedict import make_mixin
-        if supercls is W_ObjectObject:
-            Mixin = make_mixin(config)
-            add(Mixin)
-            wantdict = False
-
-    if wantdict:
+    if "dict" in features:
+        base_user_setup = supercls.user_setup.im_func
+        if "user_setup" in body:
+            base_user_setup = body["user_setup"]
         class Proto(object):
             def getdict(self):
                 return self.w__dict__
@@ -263,24 +275,13 @@
                 self.w__dict__ = check_new_dictionary(space, w_dict)
             
             def user_setup(self, space, w_subtype):
-                self.space = space
-                self.w__class__ = w_subtype
                 self.w__dict__ = space.newdict(
                     instance=True, classofinstance=w_subtype)
-                self.user_setup_slots(w_subtype.nslots)
+                base_user_setup(self, space, w_subtype)
 
             def setclass(self, space, w_subtype):
                 # only used by descr_set___class__
                 self.w__class__ = w_subtype
-                if space.config.objspace.std.withshadowtracking:
-                    self.w__dict__.set_shadows_anything()
-
-            def getdictvalue_attr_is_in_class(self, space, name):
-                w_dict = self.w__dict__
-                if space.config.objspace.std.withshadowtracking:
-                    if not w_dict.shadows_anything():
-                        return None
-                return space.finditem_str(w_dict, name)
 
         add(Proto)
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/detect_cpu.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/detect_cpu.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/detect_cpu.py	Sat Oct 30 16:41:17 2010
@@ -23,11 +23,6 @@
         mach = os.popen('uname -m', 'r').read().strip()
         if not mach:
             raise ProcessorAutodetectError, "cannot run 'uname -m'"
-    if mach == 'x86_64':
-        if sys.maxint == 2147483647:
-            mach = 'x86'     # it's a 64-bit processor but in 32-bits mode, maybe
-        else:
-            assert sys.maxint == 2 ** 63 - 1
     try:
         return {'i386': 'x86',
                 'i486': 'x86',
@@ -36,17 +31,21 @@
                 'i86pc': 'x86',    # Solaris/Intel
                 'x86':   'x86',    # Apple
                 'Power Macintosh': 'ppc',
-                'x86_64': 'x86_64', 
+                'x86_64': 'x86', 
                 }[mach]
     except KeyError:
         raise ProcessorAutodetectError, "unsupported processor '%s'" % mach
 
 def autodetect():
     model = autodetect_main_model()
-    if model == 'x86':
-        from pypy.jit.backend.x86.detect_sse2 import detect_sse2
-        if not detect_sse2():
-            model = 'x86-without-sse2'
+    if sys.maxint == 2**63-1:
+        model += '_64'
+    else:
+        assert sys.maxint == 2**31-1
+        if model == 'x86':
+            from pypy.jit.backend.x86.detect_sse2 import detect_sse2
+            if not detect_sse2():
+                model = 'x86-without-sse2'
     return model
 
 def getcpuclassname(backend_name="auto"):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/llgraph/llimpl.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/llgraph/llimpl.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/llgraph/llimpl.py	Sat Oct 30 16:41:17 2010
@@ -10,7 +10,7 @@
                                          BoxInt, BoxPtr, BoxObj, BoxFloat,
                                          REF, INT, FLOAT)
 from pypy.jit.codewriter import heaptracker
-from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr
+from pypy.rpython.lltypesystem import lltype, llmemory, rclass, rstr, rffi
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython.module.support import LLSupport, OOSupport
 from pypy.rpython.llinterp import LLException
@@ -129,7 +129,7 @@
     'arraylen_gc'     : (('ref',), 'int'),
     'call'            : (('ref', 'varargs'), 'intorptr'),
     'call_assembler'  : (('varargs',), 'intorptr'),
-    'cond_call_gc_wb' : (('ptr',), None),
+    'cond_call_gc_wb' : (('ptr', 'ptr'), None),
     'oosend'          : (('varargs',), 'intorptr'),
     'oosend_pure'     : (('varargs',), 'intorptr'),
     'guard_true'      : (('bool',), None),
@@ -305,12 +305,12 @@
     loop = _from_opaque(loop)
     loop.operations.append(Operation(opnum))
 
-def compile_add_descr(loop, ofs, type):
+def compile_add_descr(loop, ofs, type, arg_types):
     from pypy.jit.backend.llgraph.runner import Descr
     loop = _from_opaque(loop)
     op = loop.operations[-1]
     assert isinstance(type, str) and len(type) == 1
-    op.descr = Descr(ofs, type)
+    op.descr = Descr(ofs, type, arg_types=arg_types)
 
 def compile_add_loop_token(loop, descr):
     if we_are_translated():
@@ -801,7 +801,7 @@
             else:
                 raise TypeError(x)
         try:
-            return _do_call_common(func, args_in_order)
+            return _do_call_common(func, args_in_order, calldescr)
         except LLException, lle:
             _last_exception = lle
             d = {'v': None,
@@ -810,7 +810,7 @@
                  FLOAT: 0.0}
             return d[calldescr.typeinfo]
 
-    def op_cond_call_gc_wb(self, descr, a):
+    def op_cond_call_gc_wb(self, descr, a, b):
         py.test.skip("cond_call_gc_wb not supported")
 
     def op_oosend(self, descr, obj, *args):
@@ -1018,6 +1018,9 @@
     if isinstance(TYPE, lltype.Ptr):
         if isinstance(x, (int, long, llmemory.AddressAsInt)):
             x = llmemory.cast_int_to_adr(x)
+        if TYPE is rffi.VOIDP:
+            # assume that we want a "C-style" cast, without typechecking the value
+            return rffi.cast(TYPE, x)
         return llmemory.cast_adr_to_ptr(x, TYPE)
     elif TYPE == llmemory.Address:
         if isinstance(x, (int, long, llmemory.AddressAsInt)):
@@ -1411,10 +1414,26 @@
 def do_call_pushfloat(x):
     _call_args_f.append(x)
 
-def _do_call_common(f, args_in_order=None):
+kind2TYPE = {
+    'i': lltype.Signed,
+    'f': lltype.Float,
+    'v': lltype.Void,
+    }
+
+def _do_call_common(f, args_in_order=None, calldescr=None):
     ptr = llmemory.cast_int_to_adr(f).ptr
-    FUNC = lltype.typeOf(ptr).TO
-    ARGS = FUNC.ARGS
+    PTR = lltype.typeOf(ptr)
+    if PTR == rffi.VOIDP:
+        # it's a pointer to a C function, so we don't have a precise
+        # signature: create one from the descr
+        ARGS = map(kind2TYPE.get, calldescr.arg_types)
+        RESULT = kind2TYPE[calldescr.typeinfo]
+        FUNC = lltype.FuncType(ARGS, RESULT)
+        func_to_call = rffi.cast(lltype.Ptr(FUNC), ptr)
+    else:
+        FUNC = PTR.TO
+        ARGS = FUNC.ARGS
+        func_to_call = ptr._obj._callable
     args = cast_call_args(ARGS, _call_args_i, _call_args_r, _call_args_f,
                           args_in_order)
     del _call_args_i[:]
@@ -1426,7 +1445,7 @@
         result = llinterp.eval_graph(ptr._obj.graph, args)
         # ^^^ may raise, in which case we get an LLException
     else:
-        result = ptr._obj._callable(*args)
+        result = func_to_call(*args)
     return result
 
 def do_call_void(f):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/llgraph/runner.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/llgraph/runner.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/llgraph/runner.py	Sat Oct 30 16:41:17 2010
@@ -118,13 +118,13 @@
             self._descrs[key] = descr
             return descr
 
-    def compile_bridge(self, faildescr, inputargs, operations):
+    def compile_bridge(self, faildescr, inputargs, operations, log=True):
         c = llimpl.compile_start()
         self._compile_loop_or_bridge(c, inputargs, operations)
         old, oldindex = faildescr._compiled_fail
         llimpl.compile_redirect_fail(old, oldindex, c)
 
-    def compile_loop(self, inputargs, operations, loopdescr):
+    def compile_loop(self, inputargs, operations, loopdescr, log=True):
         """In a real assembler backend, this should assemble the given
         list of operations.  Here we just generate a similar CompiledLoop
         instance.  The code here is RPython, whereas the code in llimpl
@@ -154,7 +154,7 @@
             llimpl.compile_add(c, op.getopnum())
             descr = op.getdescr()
             if isinstance(descr, Descr):
-                llimpl.compile_add_descr(c, descr.ofs, descr.typeinfo)
+                llimpl.compile_add_descr(c, descr.ofs, descr.typeinfo, descr.arg_types)
             if isinstance(descr, history.LoopToken) and op.getopnum() != rop.JUMP:
                 llimpl.compile_add_loop_token(c, descr)
             if self.is_oo and isinstance(descr, (OODescr, MethDescr)):
@@ -297,6 +297,18 @@
         return self.getdescr(0, token[0], extrainfo=extrainfo,
                              arg_types=''.join(arg_types))
 
+    def calldescrof_dynamic(self, ffi_args, ffi_result, extrainfo=None):
+        from pypy.jit.backend.llsupport.ffisupport import get_ffi_type_kind
+        arg_types = []
+        for arg in ffi_args:
+            kind = get_ffi_type_kind(arg)
+            if kind != history.VOID:
+                arg_types.append(kind)
+        reskind = get_ffi_type_kind(ffi_result)
+        return self.getdescr(0, reskind, extrainfo=extrainfo,
+                             arg_types=''.join(arg_types))
+
+
     def grab_exc_value(self):
         return llimpl.grab_exc_value()
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/descr.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/descr.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/descr.py	Sat Oct 30 16:41:17 2010
@@ -82,6 +82,7 @@
 
     _is_pointer_field = False   # unless overridden by GcPtrFieldDescr
     _is_float_field = False     # unless overridden by FloatFieldDescr
+    _is_field_signed = False    # unless overridden by XxxFieldDescr
 
     def is_pointer_field(self):
         return self._is_pointer_field
@@ -89,6 +90,9 @@
     def is_float_field(self):
         return self._is_float_field
 
+    def is_field_signed(self):
+        return self._is_field_signed
+
     def repr_of_descr(self):
         return '<%s %s %s>' % (self._clsname, self.name, self.offset)
 
@@ -105,7 +109,7 @@
 def getFieldDescrClass(TYPE):
     return getDescrClass(TYPE, BaseFieldDescr, GcPtrFieldDescr,
                          NonGcPtrFieldDescr, 'Field', 'get_field_size',
-                         '_is_float_field')
+                         '_is_float_field', '_is_field_signed')
 
 def get_field_descr(gccache, STRUCT, fieldname):
     cache = gccache._cache_field
@@ -144,6 +148,7 @@
 
     _is_array_of_pointers = False      # unless overridden by GcPtrArrayDescr
     _is_array_of_floats   = False      # unless overridden by FloatArrayDescr
+    _is_item_signed       = False      # unless overridden by XxxArrayDescr
 
     def is_array_of_pointers(self):
         return self._is_array_of_pointers
@@ -151,6 +156,9 @@
     def is_array_of_floats(self):
         return self._is_array_of_floats
 
+    def is_item_signed(self):
+        return self._is_item_signed
+
     def repr_of_descr(self):
         return '<%s>' % self._clsname
 
@@ -186,12 +194,12 @@
 def getArrayDescrClass(ARRAY):
     return getDescrClass(ARRAY.OF, BaseArrayDescr, GcPtrArrayDescr,
                          NonGcPtrArrayDescr, 'Array', 'get_item_size',
-                         '_is_array_of_floats')
+                         '_is_array_of_floats', '_is_item_signed')
 
 def getArrayNoLengthDescrClass(ARRAY):
     return getDescrClass(ARRAY.OF, BaseArrayNoLengthDescr, GcPtrArrayNoLengthDescr,
                          NonGcPtrArrayNoLengthDescr, 'ArrayNoLength', 'get_item_size',
-                         '_is_array_of_floats')
+                         '_is_array_of_floats', '_is_item_signed')
 
 def get_array_descr(gccache, ARRAY):
     cache = gccache._cache_array
@@ -242,6 +250,9 @@
     def get_result_size(self, translate_support_code):
         raise NotImplementedError
 
+    def is_result_signed(self):
+        return False    # unless overridden
+
     def create_call_stub(self, rtyper, RESULT):
         def process(c):
             arg = 'args_%s[%d]' % (c, seen[c])
@@ -307,6 +318,30 @@
     _return_type = history.INT
     call_stub = staticmethod(lambda func, args_i, args_r, args_f: 0)
 
+    _is_result_signed = False      # can be overridden in XxxCallDescr
+    def is_result_signed(self):
+        return self._is_result_signed
+
+class DynamicIntCallDescr(BaseIntCallDescr):
+    """
+    calldescr that works for every integer type, by explicitly passing it the
+    size of the result. Used only by get_call_descr_dynamic
+    """
+    _clsname = 'DynamicIntCallDescr'
+
+    def __init__(self, arg_classes, result_size, result_sign, extrainfo=None):
+        BaseIntCallDescr.__init__(self, arg_classes, extrainfo)
+        assert isinstance(result_sign, bool)
+        self._result_size = chr(result_size)
+        self._result_sign = result_sign
+
+    def get_result_size(self, translate_support_code):
+        return ord(self._result_size)
+
+    def is_result_signed(self):
+        return self._result_sign
+
+
 class NonGcPtrCallDescr(BaseIntCallDescr):
     _clsname = 'NonGcPtrCallDescr'
     def get_result_size(self, translate_support_code):
@@ -341,7 +376,8 @@
         return FloatCallDescr
     return getDescrClass(RESULT, BaseIntCallDescr, GcPtrCallDescr,
                          NonGcPtrCallDescr, 'Call', 'get_result_size',
-                         Ellipsis)  # <= floatattrname should not be used here
+                         Ellipsis,  # <= floatattrname should not be used here
+                         '_is_result_signed')
 
 def get_call_descr(gccache, ARGS, RESULT, extrainfo=None):
     arg_classes = []
@@ -368,7 +404,8 @@
 # ____________________________________________________________
 
 def getDescrClass(TYPE, BaseDescr, GcPtrDescr, NonGcPtrDescr,
-                  nameprefix, methodname, floatattrname, _cache={}):
+                  nameprefix, methodname, floatattrname, signedattrname,
+                  _cache={}):
     if isinstance(TYPE, lltype.Ptr):
         if TYPE.TO._gckind == 'gc':
             return GcPtrDescr
@@ -388,6 +425,8 @@
         #
         if TYPE is lltype.Float:
             setattr(Descr, floatattrname, True)
+        elif TYPE is not lltype.Bool and rffi.cast(TYPE, -1) == -1:
+            setattr(Descr, signedattrname, True)
         #
         _cache[nameprefix, TYPE] = Descr
         return Descr

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/gc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/gc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/gc.py	Sat Oct 30 16:41:17 2010
@@ -158,7 +158,7 @@
         # used to avoid too many duplications in the GCREF_LISTs.
         self.hashtable = lltype.malloc(self.HASHTABLE,
                                        self.HASHTABLE_SIZE+1,
-                                       flavor='raw')
+                                       flavor='raw', track_allocation=False)
         dummy = lltype.direct_ptradd(lltype.direct_arrayitems(self.hashtable),
                                      self.HASHTABLE_SIZE)
         dummy = llmemory.cast_ptr_to_adr(dummy)
@@ -252,14 +252,15 @@
 
     def _enlarge_gcmap(self):
         newlength = 250 + self._gcmap_maxlength * 2
-        newgcmap = lltype.malloc(self.GCMAP_ARRAY, newlength, flavor='raw')
+        newgcmap = lltype.malloc(self.GCMAP_ARRAY, newlength, flavor='raw',
+                                 track_allocation=False)
         oldgcmap = self._gcmap
         for i in range(self._gcmap_curlength):
             newgcmap[i] = oldgcmap[i]
         self._gcmap = newgcmap
         self._gcmap_maxlength = newlength
         if oldgcmap:
-            lltype.free(oldgcmap, flavor='raw')
+            lltype.free(oldgcmap, flavor='raw', track_allocation=False)
 
     def get_basic_shape(self, is_64_bit=False):
         # XXX: Should this code even really know about stack frame layout of
@@ -308,7 +309,8 @@
         # them inside bigger arrays) and we never try to share them.
         length = len(shape)
         compressed = lltype.malloc(self.CALLSHAPE_ARRAY, length,
-                                   flavor='raw')
+                                   flavor='raw',
+                                   track_allocation=False)   # memory leak
         for i in range(length):
             compressed[length-1-i] = rffi.cast(rffi.UCHAR, shape[i])
         return llmemory.cast_ptr_to_adr(compressed)
@@ -404,7 +406,7 @@
         self.GC_MALLOC_BASIC = lltype.Ptr(lltype.FuncType(
             [lltype.Signed, lltype.Signed], llmemory.GCREF))
         self.WB_FUNCPTR = lltype.Ptr(lltype.FuncType(
-            [llmemory.Address], lltype.Void))
+            [llmemory.Address, llmemory.Address], lltype.Void))
         self.write_barrier_descr = WriteBarrierDescr(self)
         #
         def malloc_array(itemsize, tid, num_elem):
@@ -550,7 +552,8 @@
             # the GC, and call it immediately
             llop1 = self.llop1
             funcptr = llop1.get_write_barrier_failing_case(self.WB_FUNCPTR)
-            funcptr(llmemory.cast_ptr_to_adr(gcref_struct))
+            funcptr(llmemory.cast_ptr_to_adr(gcref_struct),
+                    llmemory.cast_ptr_to_adr(gcref_newptr))
 
     def rewrite_assembler(self, cpu, operations):
         # Perform two kinds of rewrites in parallel:
@@ -589,7 +592,7 @@
                 v = op.getarg(1)
                 if isinstance(v, BoxPtr) or (isinstance(v, ConstPtr) and
                                              bool(v.value)): # store a non-NULL
-                    self._gen_write_barrier(newops, op.getarg(0))
+                    self._gen_write_barrier(newops, op.getarg(0), v)
                     op = op.copy_and_change(rop.SETFIELD_RAW)
             # ---------- write barrier for SETARRAYITEM_GC ----------
             if op.getopnum() == rop.SETARRAYITEM_GC:
@@ -598,15 +601,15 @@
                                              bool(v.value)): # store a non-NULL
                     # XXX detect when we should produce a
                     # write_barrier_from_array
-                    self._gen_write_barrier(newops, op.getarg(0))
+                    self._gen_write_barrier(newops, op.getarg(0), v)
                     op = op.copy_and_change(rop.SETARRAYITEM_RAW)
             # ----------
             newops.append(op)
         del operations[:]
         operations.extend(newops)
 
-    def _gen_write_barrier(self, newops, v_base):
-        args = [v_base]
+    def _gen_write_barrier(self, newops, v_base, v_value):
+        args = [v_base, v_value]
         newops.append(ResOperation(rop.COND_CALL_GC_WB, args, None,
                                    descr=self.write_barrier_descr))
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/llmodel.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/llmodel.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/llmodel.py	Sat Oct 30 16:41:17 2010
@@ -17,6 +17,7 @@
 from pypy.jit.backend.llsupport.descr import get_call_descr
 from pypy.jit.backend.llsupport.descr import BaseIntCallDescr, GcPtrCallDescr
 from pypy.jit.backend.llsupport.descr import FloatCallDescr, VoidCallDescr
+from pypy.jit.backend.llsupport.ffisupport import get_call_descr_dynamic
 from pypy.rpython.annlowlevel import cast_instance_to_base_ptr
 
 
@@ -82,7 +83,8 @@
         # read back by the machine code reading at the address given by
         # pos_exception() and pos_exc_value().
         _exception_emulator = lltype.malloc(rffi.CArray(lltype.Signed), 2,
-                                            zero=True, flavor='raw')
+                                            zero=True, flavor='raw',
+                                            immortal=True)
         self._exception_emulator = _exception_emulator
 
         def _store_exception(lle):
@@ -210,7 +212,8 @@
         assert isinstance(fielddescr, BaseFieldDescr)
         ofs = fielddescr.offset
         size = fielddescr.get_field_size(self.translate_support_code)
-        return ofs, size
+        sign = fielddescr.is_field_signed()
+        return ofs, size, sign
     unpack_fielddescr_size._always_inline_ = True
 
     def arraydescrof(self, A):
@@ -225,12 +228,16 @@
         assert isinstance(arraydescr, BaseArrayDescr)
         ofs = arraydescr.get_base_size(self.translate_support_code)
         size = arraydescr.get_item_size(self.translate_support_code)
-        return ofs, size
+        sign = arraydescr.is_item_signed()
+        return ofs, size, sign
     unpack_arraydescr_size._always_inline_ = True
 
     def calldescrof(self, FUNC, ARGS, RESULT, extrainfo=None):
         return get_call_descr(self.gc_ll_descr, ARGS, RESULT, extrainfo)
 
+    def calldescrof_dynamic(self, ffi_args, ffi_result, extrainfo=None):
+        return get_call_descr_dynamic(ffi_args, ffi_result, extrainfo)
+
     def get_overflow_error(self):
         ovf_vtable = self.cast_adr_to_int(self._ovf_error_vtable)
         ovf_inst = lltype.cast_opaque_ptr(llmemory.GCREF,
@@ -252,15 +259,21 @@
 
     @specialize.argtype(2)
     def bh_getarrayitem_gc_i(self, arraydescr, gcref, itemindex):
-        ofs, size = self.unpack_arraydescr_size(arraydescr)
+        ofs, size, sign = self.unpack_arraydescr_size(arraydescr)
         # --- start of GC unsafe code (no GC operation!) ---
         items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for STYPE, UTYPE, itemsize in unroll_basic_sizes:
             if size == itemsize:
-                items = rffi.cast(rffi.CArrayPtr(TYPE), items) 
-                val = items[itemindex]
+                if sign:
+                    items = rffi.cast(rffi.CArrayPtr(STYPE), items)
+                    val = items[itemindex]
+                    val = rffi.cast(lltype.Signed, val)
+                else:
+                    items = rffi.cast(rffi.CArrayPtr(UTYPE), items)
+                    val = items[itemindex]
+                    val = rffi.cast(lltype.Signed, val)
                 # --- end of GC unsafe code ---
-                return rffi.cast(lltype.Signed, val)
+                return val
         else:
             raise NotImplementedError("size = %d" % size)
 
@@ -285,10 +298,10 @@
 
     @specialize.argtype(2)
     def bh_setarrayitem_gc_i(self, arraydescr, gcref, itemindex, newvalue):
-        ofs, size = self.unpack_arraydescr_size(arraydescr)
+        ofs, size, sign = self.unpack_arraydescr_size(arraydescr)
         # --- start of GC unsafe code (no GC operation!) ---
         items = rffi.ptradd(rffi.cast(rffi.CCHARP, gcref), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for TYPE, _, itemsize in unroll_basic_sizes:
             if size == itemsize:
                 items = rffi.cast(rffi.CArrayPtr(TYPE), items)
                 items[itemindex] = rffi.cast(TYPE, newvalue)
@@ -339,14 +352,22 @@
 
     @specialize.argtype(1)
     def _base_do_getfield_i(self, struct, fielddescr):
-        ofs, size = self.unpack_fielddescr_size(fielddescr)
+        ofs, size, sign = self.unpack_fielddescr_size(fielddescr)
         # --- start of GC unsafe code (no GC operation!) ---
         fieldptr = rffi.ptradd(rffi.cast(rffi.CCHARP, struct), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for STYPE, UTYPE, itemsize in unroll_basic_sizes:
             if size == itemsize:
-                val = rffi.cast(rffi.CArrayPtr(TYPE), fieldptr)[0]
+                # Note that in the common case where size==sizeof(Signed),
+                # both cases of what follows are doing the same thing.
+                # But gcc is clever enough to figure this out :-)
+                if sign:
+                    val = rffi.cast(rffi.CArrayPtr(STYPE), fieldptr)[0]
+                    val = rffi.cast(lltype.Signed, val)
+                else:
+                    val = rffi.cast(rffi.CArrayPtr(UTYPE), fieldptr)[0]
+                    val = rffi.cast(lltype.Signed, val)
                 # --- end of GC unsafe code ---
-                return rffi.cast(lltype.Signed, val)
+                return val
         else:
             raise NotImplementedError("size = %d" % size)
 
@@ -378,10 +399,10 @@
 
     @specialize.argtype(1)
     def _base_do_setfield_i(self, struct, fielddescr, newvalue):
-        ofs, size = self.unpack_fielddescr_size(fielddescr)
+        ofs, size, sign = self.unpack_fielddescr_size(fielddescr)
         # --- start of GC unsafe code (no GC operation!) ---
         fieldptr = rffi.ptradd(rffi.cast(rffi.CCHARP, struct), ofs)
-        for TYPE, itemsize in unroll_basic_sizes:
+        for TYPE, _, itemsize in unroll_basic_sizes:
             if size == itemsize:
                 fieldptr = rffi.cast(rffi.CArrayPtr(TYPE), fieldptr)
                 fieldptr[0] = rffi.cast(TYPE, newvalue)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/symbolic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/symbolic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/symbolic.py	Sat Oct 30 16:41:17 2010
@@ -69,8 +69,9 @@
 SIZEOF_INT   = get_size(rffi.INT, False)
 SIZEOF_FLOAT = get_size(lltype.Float, False)
 
-unroll_basic_sizes = unrolling_iterable([(lltype.Signed, WORD),
-                                         (lltype.Char,   SIZEOF_CHAR),
-                                         (rffi.SHORT,    SIZEOF_SHORT),
-                                         (rffi.INT,      SIZEOF_INT)])
+unroll_basic_sizes = unrolling_iterable([
+    (lltype.Signed,   lltype.Unsigned, WORD),
+    (rffi.SIGNEDCHAR, lltype.Char,     SIZEOF_CHAR),
+    (rffi.SHORT,      rffi.USHORT,     SIZEOF_SHORT),
+    (rffi.INT,        rffi.UINT,       SIZEOF_INT)])
 # does not contain Float ^^^ which must be special-cased

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/test/test_descr.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/test/test_descr.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/test/test_descr.py	Sat Oct 30 16:41:17 2010
@@ -83,6 +83,18 @@
         assert     descr_f.is_float_field()
 
 
+def test_get_field_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        S = lltype.GcStruct('S', ('x', RESTYPE))
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            descr_x = get_field_descr(c2, S, 'x')
+            assert descr_x.is_field_signed() == signed
+
+
 def test_get_array_descr():
     U = lltype.Struct('U')
     T = lltype.GcStruct('T')
@@ -164,6 +176,25 @@
     assert descr.get_base_size(False) == 0
     assert descr.get_ofs_length(False) == -1
 
+
+def test_get_array_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        A = lltype.GcArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            arraydescr = get_array_descr(c2, A)
+            assert arraydescr.is_item_signed() == signed
+        #
+        RA = rffi.CArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            arraydescr = get_array_descr(c2, RA)
+            assert arraydescr.is_item_signed() == signed
+
+
 def test_get_call_descr_not_translated():
     c0 = GcCache(False)
     descr1 = get_call_descr(c0, [lltype.Char, lltype.Signed], lltype.Char)
@@ -219,6 +250,17 @@
     extrainfo = descr3.get_extra_info()
     assert extrainfo is None
 
+def test_get_call_descr_sign():
+    for RESTYPE, signed in [(rffi.SIGNEDCHAR, True), (rffi.UCHAR,  False),
+                            (rffi.SHORT,      True), (rffi.USHORT, False),
+                            (rffi.INT,        True), (rffi.UINT,   False),
+                            (rffi.LONG,       True), (rffi.ULONG,  False)]:
+        A = lltype.GcArray(RESTYPE)
+        for tsc in [False, True]:
+            c2 = GcCache(tsc)
+            descr1 = get_call_descr(c2, [], RESTYPE)
+            assert descr1.is_result_signed() == signed
+
 
 def test_repr_of_descr():
     c0 = GcCache(False)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/test/test_gc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/test/test_gc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/llsupport/test/test_gc.py	Sat Oct 30 16:41:17 2010
@@ -141,8 +141,8 @@
                             repr(offset_to_length), p))
         return p
 
-    def _write_barrier_failing_case(self, adr_struct):
-        self.record.append(('barrier', adr_struct))
+    def _write_barrier_failing_case(self, adr_struct, adr_newptr):
+        self.record.append(('barrier', adr_struct, adr_newptr))
 
     def get_write_barrier_failing_case(self, FPTRTYPE):
         return llhelper(FPTRTYPE, self._write_barrier_failing_case)
@@ -239,6 +239,7 @@
         s_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, s)
         r_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, r)
         s_adr = llmemory.cast_ptr_to_adr(s)
+        r_adr = llmemory.cast_ptr_to_adr(r)
         #
         s_hdr.tid &= ~gc_ll_descr.GCClass.JIT_WB_IF_FLAG
         gc_ll_descr.do_write_barrier(s_gcref, r_gcref)
@@ -246,7 +247,7 @@
         #
         s_hdr.tid |= gc_ll_descr.GCClass.JIT_WB_IF_FLAG
         gc_ll_descr.do_write_barrier(s_gcref, r_gcref)
-        assert self.llop1.record == [('barrier', s_adr)]
+        assert self.llop1.record == [('barrier', s_adr, r_adr)]
 
     def test_gen_write_barrier(self):
         gc_ll_descr = self.gc_ll_descr
@@ -254,11 +255,13 @@
         #
         newops = []
         v_base = BoxPtr()
-        gc_ll_descr._gen_write_barrier(newops, v_base)
+        v_value = BoxPtr()
+        gc_ll_descr._gen_write_barrier(newops, v_base, v_value)
         assert llop1.record == []
         assert len(newops) == 1
         assert newops[0].getopnum() == rop.COND_CALL_GC_WB
         assert newops[0].getarg(0) == v_base
+        assert newops[0].getarg(1) == v_value
         assert newops[0].result is None
         wbdescr = newops[0].getdescr()
         assert isinstance(wbdescr.jit_wb_if_flag, int)
@@ -358,6 +361,7 @@
         #
         assert operations[0].getopnum() == rop.COND_CALL_GC_WB
         assert operations[0].getarg(0) == v_base
+        assert operations[0].getarg(1) == v_value
         assert operations[0].result is None
         #
         assert operations[1].getopnum() == rop.SETFIELD_RAW
@@ -381,6 +385,7 @@
         #
         assert operations[0].getopnum() == rop.COND_CALL_GC_WB
         assert operations[0].getarg(0) == v_base
+        assert operations[0].getarg(1) == v_value
         assert operations[0].result is None
         #
         assert operations[1].getopnum() == rop.SETARRAYITEM_RAW

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/model.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/model.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/model.py	Sat Oct 30 16:41:17 2010
@@ -33,14 +33,14 @@
         pass
 
 
-    def compile_loop(self, inputargs, operations, looptoken):
+    def compile_loop(self, inputargs, operations, looptoken, log=True):
         """Assemble the given loop.
         Extra attributes should be put in the LoopToken to
         point to the compiled loop in assembler.
         """
         raise NotImplementedError
 
-    def compile_bridge(self, faildescr, inputargs, operations):
+    def compile_bridge(self, faildescr, inputargs, operations, log=True):
         """Assemble the bridge.
         The FailDescr is the descr of the original guard that failed.
         """

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/test/runner_test.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/test/runner_test.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/test/runner_test.py	Sat Oct 30 16:41:17 2010
@@ -9,12 +9,13 @@
                                          ConstObj, BoxFloat, ConstFloat)
 from pypy.jit.metainterp.resoperation import ResOperation, rop
 from pypy.jit.metainterp.typesystem import deref
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.rpython.lltypesystem import lltype, llmemory, rstr, rffi, rclass
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.llinterp import LLException
 from pypy.jit.codewriter import heaptracker
+from pypy.rlib.rarithmetic import intmask
 
 
 class Runner(object):
@@ -421,6 +422,7 @@
             assert x == 3.5 - 42
 
     def test_call(self):
+        from pypy.rlib.libffi import types
 
         def func_int(a, b):
             return a + b
@@ -428,23 +430,31 @@
             return chr(ord(c) + ord(c1))
 
         functions = [
-            (func_int, lltype.Signed, 655360),
-            (func_int, rffi.SHORT, 1213),
-            (func_char, lltype.Char, 12)
+            (func_int, lltype.Signed, types.sint, 655360),
+            (func_int, rffi.SHORT, types.sint16, 1213),
+            (func_char, lltype.Char, types.uchar, 12)
             ]
 
-        for func, TP, num in functions:
+        for func, TP, ffi_type, num in functions:
             cpu = self.cpu
             #
             FPTR = self.Ptr(self.FuncType([TP, TP], TP))
             func_ptr = llhelper(FPTR, func)
             FUNC = deref(FPTR)
-            calldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
             funcbox = self.get_funcbox(cpu, func_ptr)
+            # first, try it with the "normal" calldescr
+            calldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
             res = self.execute_operation(rop.CALL,
                                          [funcbox, BoxInt(num), BoxInt(num)],
                                          'int', descr=calldescr)
             assert res.value == 2 * num
+            # then, try it with the dynamic calldescr
+            dyn_calldescr = cpu.calldescrof_dynamic([ffi_type, ffi_type], ffi_type)
+            res = self.execute_operation(rop.CALL,
+                                         [funcbox, BoxInt(num), BoxInt(num)],
+                                         'int', descr=dyn_calldescr)
+            assert res.value == 2 * num
+            
 
         if cpu.supports_floats:
             def func(f0, f1, f2, f3, f4, f5, f6, i0, i1, f7, f8, f9):
@@ -507,6 +517,24 @@
                                          'int', descr=calldescr)
             assert res.value == func_ints(*args)
 
+    def test_call_to_c_function(self):
+        from pypy.rlib.libffi import CDLL, types, ArgChain
+        from pypy.rpython.lltypesystem.ll2ctypes import libc_name
+        libc = CDLL(libc_name)
+        c_tolower = libc.getpointer('tolower', [types.uchar], types.sint)
+        argchain = ArgChain().arg(ord('A'))
+        assert c_tolower.call(argchain, rffi.INT) == ord('a')
+
+        func_adr = llmemory.cast_ptr_to_adr(c_tolower.funcsym)
+        funcbox = ConstInt(heaptracker.adr2int(func_adr))
+        calldescr = self.cpu.calldescrof_dynamic([types.uchar], types.sint)
+        res = self.execute_operation(rop.CALL,
+                                     [funcbox, BoxInt(ord('A'))],
+                                     'int',
+                                     descr=calldescr)
+        assert res.value == ord('a')
+
+
     def test_field_basic(self):
         t_box, T_box = self.alloc_instance(self.T)
         fielddescr = self.cpu.fielddescrof(self.S, 'value')
@@ -833,6 +861,23 @@
                                                         length_box], 'void')
                                 assert self.look_string(r_box) == "!??cdef?!"
 
+    def test_copyunicodecontent(self):
+        s_box = self.alloc_unicode(u"abcdef")
+        for s_box in [s_box, s_box.constbox()]:
+            for srcstart_box in [BoxInt(2), ConstInt(2)]:
+                for dststart_box in [BoxInt(3), ConstInt(3)]:
+                    for length_box in [BoxInt(4), ConstInt(4)]:
+                        for r_box_is_const in [False, True]:
+                            r_box = self.alloc_unicode(u"!???????!")
+                            if r_box_is_const:
+                                r_box = r_box.constbox()
+                                self.execute_operation(rop.COPYUNICODECONTENT,
+                                                       [s_box, r_box,
+                                                        srcstart_box,
+                                                        dststart_box,
+                                                        length_box], 'void')
+                                assert self.look_unicode(r_box) == u"!??cdef?!"
+
     def test_do_unicode_basic(self):
         u = self.cpu.bh_newunicode(5)
         self.cpu.bh_unicodesetitem(u, 4, 123)
@@ -1227,6 +1272,10 @@
         u_box = BoxPtr(lltype.cast_opaque_ptr(llmemory.GCREF, u))
         return u_box
 
+    def look_unicode(self, unicode_box):
+        u = unicode_box.getref(lltype.Ptr(rstr.UNICODE))
+        return u''.join(u.chars)
+
 
     def test_casts(self):
         py.test.skip("xxx fix or kill")
@@ -1283,6 +1332,7 @@
                                    descr=fd)
             res = self.execute_operation(get_op, [s_box], 'int', descr=fd)
             assert res.getint()  == 32
+        lltype.free(s, flavor='raw')
 
     def test_new_with_vtable(self):
         cpu = self.cpu
@@ -1427,12 +1477,12 @@
         assert not excvalue
 
     def test_cond_call_gc_wb(self):
-        def func_void(a):
-            record.append(a)
+        def func_void(a, b):
+            record.append((a, b))
         record = []
         #
         S = lltype.GcStruct('S', ('tid', lltype.Signed))
-        FUNC = self.FuncType([lltype.Ptr(S)], lltype.Void)
+        FUNC = self.FuncType([lltype.Ptr(S), lltype.Signed], lltype.Void)
         func_ptr = llhelper(lltype.Ptr(FUNC), func_void)
         funcbox = self.get_funcbox(self.cpu, func_ptr)
         class WriteBarrierDescr(AbstractDescr):
@@ -1453,10 +1503,10 @@
             sgcref = lltype.cast_opaque_ptr(llmemory.GCREF, s)
             del record[:]
             self.execute_operation(rop.COND_CALL_GC_WB,
-                                   [BoxPtr(sgcref)],
+                                   [BoxPtr(sgcref), ConstInt(-2121)],
                                    'void', descr=WriteBarrierDescr())
             if cond:
-                assert record == [s]
+                assert record == [(s, -2121)]
             else:
                 assert record == []
 
@@ -1980,6 +2030,196 @@
         assert self.cpu.get_latest_value_float(0) == 13.5
         assert called
 
+    def test_short_result_of_getfield_direct(self):
+        # Test that a getfield that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            S = lltype.GcStruct('S', ('x', RESTYPE))
+            descrfld_x = cpu.fielddescrof(S, 'x')
+            s = lltype.malloc(S)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            s.x = rffi.cast(RESTYPE, value)
+            x = cpu.bh_getfield_gc_i(lltype.cast_opaque_ptr(llmemory.GCREF, s),
+                                     descrfld_x)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_getfield_compiled(self):
+        # Test that a getfield that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            S = lltype.GcStruct('S', ('x', RESTYPE))
+            descrfld_x = cpu.fielddescrof(S, 'x')
+            s = lltype.malloc(S)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            s.x = rffi.cast(RESTYPE, value)
+            s_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, s)
+            res = self.execute_operation(rop.GETFIELD_GC, [BoxPtr(s_gcref)],
+                                         'int', descr=descrfld_x)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
+    def test_short_result_of_getarrayitem_direct(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = lltype.GcArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            x = cpu.bh_getarrayitem_gc_i(
+                descrarray, lltype.cast_opaque_ptr(llmemory.GCREF, a), 3)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_getarrayitem_compiled(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = lltype.GcArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_gcref = lltype.cast_opaque_ptr(llmemory.GCREF, a)
+            res = self.execute_operation(rop.GETARRAYITEM_GC,
+                                         [BoxPtr(a_gcref), BoxInt(3)],
+                                         'int', descr=descrarray)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
+    def test_short_result_of_getarrayitem_raw_direct(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Direct bh_xxx test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = rffi.CArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5, flavor='raw')
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_rawint = heaptracker.adr2int(llmemory.cast_ptr_to_adr(a))
+            x = cpu.bh_getarrayitem_raw_i(descrarray, a_rawint, 3)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+            lltype.free(a, flavor='raw')
+
+    def test_short_result_of_getarrayitem_raw_compiled(self):
+        # Test that a getarrayitem that returns a CHAR, SHORT or INT, signed
+        # or unsigned, properly gets zero-extended or sign-extended.
+        # Machine code compilation test.
+        cpu = self.cpu
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            A = rffi.CArray(RESTYPE)
+            descrarray = cpu.arraydescrof(A)
+            a = lltype.malloc(A, 5, flavor='raw')
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value))
+            a[3] = rffi.cast(RESTYPE, value)
+            a_rawint = heaptracker.adr2int(llmemory.cast_ptr_to_adr(a))
+            res = self.execute_operation(rop.GETARRAYITEM_RAW,
+                                         [BoxInt(a_rawint), BoxInt(3)],
+                                         'int', descr=descrarray)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+            lltype.free(a, flavor='raw')
+
+    def test_short_result_of_call_direct(self):
+        # Test that calling a function that returns a CHAR, SHORT or INT,
+        # signed or unsigned, properly gets zero-extended or sign-extended.
+        from pypy.translator.tool.cbuild import ExternalCompilationInfo
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            # Tested with a function that intentionally does not cast the
+            # result to RESTYPE, but makes sure that we return the whole
+            # value in eax or rax.
+            eci = ExternalCompilationInfo(separate_module_sources=["""
+                long fn_test_result_of_call(long x)
+                {
+                    return x + 1;
+                }
+            """])
+            f = rffi.llexternal('fn_test_result_of_call', [lltype.Signed],
+                                RESTYPE, compilation_info=eci, _nowrapper=True)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value + 1))
+            assert intmask(f(value)) == expected
+            #
+            FUNC = self.FuncType([lltype.Signed], RESTYPE)
+            FPTR = self.Ptr(FUNC)
+            calldescr = self.cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
+            x = self.cpu.bh_call_i(self.get_funcbox(self.cpu, f).value,
+                                   calldescr, [value], None, None)
+            assert x == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, x, expected))
+
+    def test_short_result_of_call_compiled(self):
+        # Test that calling a function that returns a CHAR, SHORT or INT,
+        # signed or unsigned, properly gets zero-extended or sign-extended.
+        from pypy.translator.tool.cbuild import ExternalCompilationInfo
+        for RESTYPE in [rffi.SIGNEDCHAR, rffi.UCHAR,
+                        rffi.SHORT, rffi.USHORT,
+                        rffi.INT, rffi.UINT,
+                        rffi.LONG, rffi.ULONG]:
+            # Tested with a function that intentionally does not cast the
+            # result to RESTYPE, but makes sure that we return the whole
+            # value in eax or rax.
+            eci = ExternalCompilationInfo(separate_module_sources=["""
+                long fn_test_result_of_call(long x)
+                {
+                    return x + 1;
+                }
+            """])
+            f = rffi.llexternal('fn_test_result_of_call', [lltype.Signed],
+                                RESTYPE, compilation_info=eci, _nowrapper=True)
+            value = intmask(0xFFEEDDCCBBAA9988)
+            expected = rffi.cast(lltype.Signed, rffi.cast(RESTYPE, value + 1))
+            assert intmask(f(value)) == expected
+            #
+            FUNC = self.FuncType([lltype.Signed], RESTYPE)
+            FPTR = self.Ptr(FUNC)
+            calldescr = self.cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT)
+            funcbox = self.get_funcbox(self.cpu, f)
+            res = self.execute_operation(rop.CALL, [funcbox, BoxInt(value)],
+                                         'int', descr=calldescr)
+            assert res.value == expected, (
+                "%r: got %r, expected %r" % (RESTYPE, res.value, expected))
+
 
 class OOtypeBackendTest(BaseBackendTest):
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/test/test_ll_random.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/test/test_ll_random.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/test/test_ll_random.py	Sat Oct 30 16:41:17 2010
@@ -386,6 +386,20 @@
         v_string = self.get_string(builder, r)
         builder.do(self.opnum, [v_string])
 
+class AbstractCopyContentOperation(AbstractStringOperation):
+    def produce_into(self, builder, r):
+        v_srcstring = self.get_string(builder, r)
+        v_dststring = self.get_string(builder, r)
+        if v_srcstring.value == v_dststring.value:    # because it's not a
+            raise test_random.CannotProduceOperation  # memmove(), but memcpy()
+        srclen = len(v_srcstring.getref(self.ptr).chars)
+        dstlen = len(v_dststring.getref(self.ptr).chars)
+        v_length = builder.get_index(min(srclen, dstlen), r)
+        v_srcstart = builder.get_index(srclen - v_length.value + 1, r)
+        v_dststart = builder.get_index(dstlen - v_length.value + 1, r)
+        builder.do(self.opnum, [v_srcstring, v_dststring,
+                                v_srcstart, v_dststart, v_length])
+
 class StrGetItemOperation(AbstractGetItemOperation, _StrOperation):
     pass
 
@@ -404,6 +418,13 @@
 class UnicodeLenOperation(AbstractStringLenOperation, _UnicodeOperation):
     pass
 
+class CopyStrContentOperation(AbstractCopyContentOperation, _StrOperation):
+    pass
+
+class CopyUnicodeContentOperation(AbstractCopyContentOperation,
+                                  _UnicodeOperation):
+    pass
+
 
 # there are five options in total:
 # 1. non raising call and guard_no_exception
@@ -577,6 +598,8 @@
     OPERATIONS.append(UnicodeSetItemOperation(rop.UNICODESETITEM))
     OPERATIONS.append(StrLenOperation(rop.STRLEN))
     OPERATIONS.append(UnicodeLenOperation(rop.UNICODELEN))
+    OPERATIONS.append(CopyStrContentOperation(rop.COPYSTRCONTENT))
+    OPERATIONS.append(CopyUnicodeContentOperation(rop.COPYUNICODECONTENT))
 
 for i in range(2):
     OPERATIONS.append(GuardClassOperation(rop.GUARD_CLASS))

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/test/test_random.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/test/test_random.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/test/test_random.py	Sat Oct 30 16:41:17 2010
@@ -1,7 +1,7 @@
 import py, sys
 from pypy.rlib.rarithmetic import intmask, LONG_BIT
 from pypy.rpython.lltypesystem import llmemory
-from pypy.jit.backend.test import conftest as demo_conftest
+from pypy.jit.backend import conftest as demo_conftest
 from pypy.jit.metainterp.history import BasicFailDescr, TreeLoop
 from pypy.jit.metainterp.history import BoxInt, ConstInt, LoopToken
 from pypy.jit.metainterp.history import BoxPtr, ConstPtr
@@ -102,7 +102,7 @@
             elif isinstance(v, ConstFloat):
                 args.append('ConstFloat(%r)' % v.value)
             elif isinstance(v, ConstInt):
-                args.append('ConstInt(%d)' % v.value)
+                args.append('ConstInt(%s)' % v.value)
             else:
                 raise NotImplementedError(v)
         if op.getdescr() is None:
@@ -113,7 +113,7 @@
             except AttributeError:
                 descrstr = ', descr=...'
         print >>s, '        ResOperation(rop.%s, [%s], %s%s),' % (
-            opname[op.opnum], ', '.join(args), names[op.result], descrstr)
+            opname[op.getopnum()], ', '.join(args), names[op.result], descrstr)
         #if getattr(op, 'suboperations', None) is not None:
         #    subops.append(op)
 
@@ -189,7 +189,7 @@
                                                                        v.value)
         print >>s, '    op = cpu.execute_token(looptoken)'
         if self.should_fail_by is None:
-            fail_args = self.loop.operations[-1].args
+            fail_args = self.loop.operations[-1].getarglist()
         else:
             fail_args = self.should_fail_by.getfailargs()
         for i, v in enumerate(fail_args):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/assembler.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/assembler.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/assembler.py	Sat Oct 30 16:41:17 2010
@@ -8,7 +8,8 @@
 from pypy.rpython.annlowlevel import llhelper
 from pypy.tool.uid import fixid
 from pypy.jit.backend.x86.regalloc import (RegAlloc, X86RegisterManager,
-                                           X86XMMRegisterManager, get_ebp_ofs)
+                                           X86XMMRegisterManager, get_ebp_ofs,
+                                           _get_scale)
 
 from pypy.jit.backend.x86.arch import (FRAME_FIXED_SIZE, FORCE_INDEX_OFS, WORD,
                                        IS_X86_32, IS_X86_64)
@@ -22,7 +23,8 @@
                                          X86_64_SCRATCH_REG,
                                          X86_64_XMM_SCRATCH_REG,
                                          RegLoc, StackLoc, ConstFloatLoc,
-                                         ImmedLoc, AddressLoc, imm)
+                                         ImmedLoc, AddressLoc, imm,
+                                         imm0, imm1)
 
 from pypy.rlib.objectmodel import we_are_translated, specialize
 from pypy.jit.backend.x86 import rx86, regloc, codebuf
@@ -244,12 +246,13 @@
             f = open_file_as_stream(output_log, "w")
             for i in range(len(self.loop_run_counters)):
                 name, struct = self.loop_run_counters[i]
-                f.write(str(struct.i) + " " * (8 - len(str(struct.i))) + name + "\n")
+                f.write(str(name) + ":" +  str(struct.i) + "\n")
             f.close()
 
     def _build_float_constants(self):
         # 44 bytes: 32 bytes for the data, and up to 12 bytes for alignment
-        addr = lltype.malloc(rffi.CArray(lltype.Char), 44, flavor='raw')
+        addr = lltype.malloc(rffi.CArray(lltype.Char), 44, flavor='raw',
+                             track_allocation=False)
         if not we_are_translated():
             self._keepalive_malloced_float_consts = addr
         float_constants = rffi.cast(lltype.Signed, addr)
@@ -292,7 +295,7 @@
         self.mc.RET()
         self.mc.done()
 
-    def assemble_loop(self, inputargs, operations, looptoken):
+    def assemble_loop(self, inputargs, operations, looptoken, log):
         """adds the following attributes to looptoken:
                _x86_loop_code       (an integer giving an address)
                _x86_bootstrap_code  (an integer giving an address)
@@ -307,10 +310,11 @@
 
         self.setup()
         funcname = self._find_debug_merge_point(operations)
-
+        if log:
+            self._register_counter()
+            operations = self._inject_debugging_code(operations)
         
         regalloc = RegAlloc(self, self.cpu.translate_support_code)
-        operations = self._inject_debugging_code(operations)
         arglocs = regalloc.prepare_loop(inputargs, operations, looptoken)
         looptoken._x86_arglocs = arglocs
 
@@ -337,13 +341,16 @@
         self.mc.end_function()
         self.write_pending_failure_recoveries()
         
-    def assemble_bridge(self, faildescr, inputargs, operations):
+    def assemble_bridge(self, faildescr, inputargs, operations, log):
         if not we_are_translated():
             # Arguments should be unique
             assert len(set(inputargs)) == len(inputargs)
 
         self.setup()
         funcname = self._find_debug_merge_point(operations)
+        if log:
+            self._register_counter()
+            operations = self._inject_debugging_code(operations)
 
         arglocs = self.rebuild_faillocs_from_descr(
             faildescr._x86_failure_recovery_bytecode)
@@ -398,11 +405,14 @@
         else:
             funcname = "" % len(self.loop_run_counters)
         # invent the counter, so we don't get too confused
+        return funcname
+
+    def _register_counter(self):
         if self._debug:
-            struct = lltype.malloc(DEBUG_COUNTER, flavor='raw')
+            struct = lltype.malloc(DEBUG_COUNTER, flavor='raw',
+                                   track_allocation=False)   # known to leak
             struct.i = 0
-            self.loop_run_counters.append((funcname, struct))
-        return funcname
+            self.loop_run_counters.append((len(self.loop_run_counters), struct))
         
     def patch_jump_for_descr(self, faildescr, adr_new_target):
         adr_jump_offset = faildescr._x86_adr_jump_offset
@@ -440,7 +450,7 @@
             # self.mc.PUSH(eax)
             # adr = rffi.cast(lltype.Signed, self.loop_run_counters[-1][1])
             # self.mc.MOV(eax, heap(adr))
-            # self.mc.ADD(eax, imm(1))
+            # self.mc.ADD(eax, imm1)
             # self.mc.MOV(heap(adr), eax)
             # self.mc.POP(eax)
         return operations
@@ -711,7 +721,7 @@
         self.regalloc_perform_with_guard(None, guard_op, faillocs, arglocs,
                                          resloc, current_depths)
 
-    def load_effective_addr(self, sizereg, baseofs, scale, result, frm=imm(0)):
+    def load_effective_addr(self, sizereg, baseofs, scale, result, frm=imm0):
         self.mc.LEA(result, addr_add(frm, sizereg, baseofs, scale))
 
     def _unaryop(asmop):
@@ -973,28 +983,28 @@
 
     def genop_guard_int_is_true(self, op, guard_op, guard_token, arglocs, resloc):
         guard_opnum = guard_op.getopnum()
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         if guard_opnum == rop.GUARD_TRUE:
             return self.implement_guard(guard_token, 'Z')
         else:
             return self.implement_guard(guard_token, 'NZ')
 
     def genop_int_is_true(self, op, arglocs, resloc):
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         rl = resloc.lowest8bits()
         self.mc.SET_ir(rx86.Conditions['NE'], rl.value)
         self.mc.MOVZX8(resloc, rl)
 
     def genop_guard_int_is_zero(self, op, guard_op, guard_token, arglocs, resloc):
         guard_opnum = guard_op.getopnum()
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         if guard_opnum == rop.GUARD_TRUE:
             return self.implement_guard(guard_token, 'NZ')
         else:
             return self.implement_guard(guard_token, 'Z')
 
     def genop_int_is_zero(self, op, arglocs, resloc):
-        self.mc.CMP(arglocs[0], imm(0))
+        self.mc.CMP(arglocs[0], imm0)
         rl = resloc.lowest8bits()
         self.mc.SET_ir(rx86.Conditions['E'], rl.value)
         self.mc.MOVZX8(resloc, rl)
@@ -1050,50 +1060,66 @@
         assert result_loc is eax
         self.call(self.malloc_unicode_func_addr, arglocs, eax)
 
-    def genop_getfield_gc(self, op, arglocs, resloc):
-        base_loc, ofs_loc, size_loc = arglocs
-        assert isinstance(size_loc, ImmedLoc)
+    # ----------
+
+    def load_from_mem(self, resloc, source_addr, size_loc, sign_loc):
         assert isinstance(resloc, RegLoc)
         size = size_loc.value
-
-        source_addr = AddressLoc(base_loc, ofs_loc)
+        sign = sign_loc.value
         if resloc.is_xmm:
             self.mc.MOVSD(resloc, source_addr)
+        elif size == WORD:
+            self.mc.MOV(resloc, source_addr)
         elif size == 1:
-            self.mc.MOVZX8(resloc, source_addr)
+            if sign:
+                self.mc.MOVSX8(resloc, source_addr)
+            else:
+                self.mc.MOVZX8(resloc, source_addr)
         elif size == 2:
-            self.mc.MOVZX16(resloc, source_addr)
+            if sign:
+                self.mc.MOVSX16(resloc, source_addr)
+            else:
+                self.mc.MOVZX16(resloc, source_addr)
+        elif IS_X86_64 and size == 4:
+            if sign:
+                self.mc.MOVSX32(resloc, source_addr)
+            else:
+                self.mc.MOV32(resloc, source_addr)    # zero-extending
+        else:
+            not_implemented("load_from_mem size = %d" % size)
+
+    def save_into_mem(self, dest_addr, value_loc, size_loc):
+        size = size_loc.value
+        if isinstance(value_loc, RegLoc) and value_loc.is_xmm:
+            self.mc.MOVSD(dest_addr, value_loc)
+        elif size == 1:
+            self.mc.MOV8(dest_addr, value_loc.lowest8bits())
+        elif size == 2:
+            self.mc.MOV16(dest_addr, value_loc)
         elif size == 4:
-            # MOV32 is zero-extending on 64-bit, so this is okay
-            self.mc.MOV32(resloc, source_addr)
+            self.mc.MOV32(dest_addr, value_loc)
         elif IS_X86_64 and size == 8:
-            self.mc.MOV(resloc, source_addr)
+            self.mc.MOV(dest_addr, value_loc)
         else:
-            raise NotImplementedError("getfield size = %d" % size)
+            not_implemented("save_into_mem size = %d" % size)
+
+    def genop_getfield_gc(self, op, arglocs, resloc):
+        base_loc, ofs_loc, size_loc, sign_loc = arglocs
+        assert isinstance(size_loc, ImmedLoc)
+        source_addr = AddressLoc(base_loc, ofs_loc)
+        self.load_from_mem(resloc, source_addr, size_loc, sign_loc)
 
     genop_getfield_raw = genop_getfield_gc
     genop_getfield_raw_pure = genop_getfield_gc
     genop_getfield_gc_pure = genop_getfield_gc
 
     def genop_getarrayitem_gc(self, op, arglocs, resloc):
-        base_loc, ofs_loc, scale, ofs = arglocs
+        base_loc, ofs_loc, size_loc, ofs, sign_loc = arglocs
         assert isinstance(ofs, ImmedLoc)
-        assert isinstance(scale, ImmedLoc)
-        src_addr = addr_add(base_loc, ofs_loc, ofs.value, scale.value)
-        if op.result.type == FLOAT:
-            self.mc.MOVSD(resloc, src_addr)
-        else:
-            if scale.value == 0:
-                self.mc.MOVZX8(resloc, src_addr)
-            elif scale.value == 1:
-                self.mc.MOVZX16(resloc, src_addr)
-            elif scale.value == 2:
-                self.mc.MOV32(resloc, src_addr)
-            elif IS_X86_64 and scale.value == 3:
-                self.mc.MOV(resloc, src_addr)
-            else:
-                print "[asmgen]getarrayitem unsupported size: %d" % scale.value
-                raise NotImplementedError()
+        assert isinstance(size_loc, ImmedLoc)
+        scale = _get_scale(size_loc.value)
+        src_addr = addr_add(base_loc, ofs_loc, ofs.value, scale)
+        self.load_from_mem(resloc, src_addr, size_loc, sign_loc)
 
     genop_getarrayitem_gc_pure = genop_getarrayitem_gc
     genop_getarrayitem_raw = genop_getarrayitem_gc
@@ -1101,40 +1127,16 @@
     def genop_discard_setfield_gc(self, op, arglocs):
         base_loc, ofs_loc, size_loc, value_loc = arglocs
         assert isinstance(size_loc, ImmedLoc)
-        size = size_loc.value
         dest_addr = AddressLoc(base_loc, ofs_loc)
-        if isinstance(value_loc, RegLoc) and value_loc.is_xmm:
-            self.mc.MOVSD(dest_addr, value_loc)
-        elif IS_X86_64 and size == 8:
-            self.mc.MOV(dest_addr, value_loc)
-        elif size == 4:
-            self.mc.MOV32(dest_addr, value_loc)
-        elif size == 2:
-            self.mc.MOV16(dest_addr, value_loc)
-        elif size == 1:
-            self.mc.MOV8(dest_addr, value_loc.lowest8bits())
-        else:
-            print "[asmgen]setfield addr size %d" % size
-            raise NotImplementedError("Addr size %d" % size)
+        self.save_into_mem(dest_addr, value_loc, size_loc)
 
     def genop_discard_setarrayitem_gc(self, op, arglocs):
-        base_loc, ofs_loc, value_loc, scale_loc, baseofs = arglocs
+        base_loc, ofs_loc, value_loc, size_loc, baseofs = arglocs
         assert isinstance(baseofs, ImmedLoc)
-        assert isinstance(scale_loc, ImmedLoc)
-        dest_addr = AddressLoc(base_loc, ofs_loc, scale_loc.value, baseofs.value)
-        if op.getarg(2).type == FLOAT:
-            self.mc.MOVSD(dest_addr, value_loc)
-        else:
-            if IS_X86_64 and scale_loc.value == 3:
-                self.mc.MOV(dest_addr, value_loc)
-            elif scale_loc.value == 2:
-                self.mc.MOV32(dest_addr, value_loc)
-            elif scale_loc.value == 1:
-                self.mc.MOV16(dest_addr, value_loc)
-            elif scale_loc.value == 0:
-                self.mc.MOV8(dest_addr, value_loc.lowest8bits())
-            else:
-                raise NotImplementedError("scale = %d" % scale_loc.value)
+        assert isinstance(size_loc, ImmedLoc)
+        scale = _get_scale(size_loc.value)
+        dest_addr = AddressLoc(base_loc, ofs_loc, scale, baseofs.value)
+        self.save_into_mem(dest_addr, value_loc, size_loc)
 
     def genop_discard_strsetitem(self, op, arglocs):
         base_loc, ofs_loc, val_loc = arglocs
@@ -1201,7 +1203,7 @@
 
     def genop_guard_guard_no_exception(self, ign_1, guard_op, guard_token,
                                        locs, ign_2):
-        self.mc.CMP(heap(self.cpu.pos_exception()), imm(0))
+        self.mc.CMP(heap(self.cpu.pos_exception()), imm0)
         return self.implement_guard(guard_token, 'NZ')
 
     def genop_guard_guard_exception(self, ign_1, guard_op, guard_token,
@@ -1213,8 +1215,8 @@
         addr = self.implement_guard(guard_token, 'NE')
         if resloc is not None:
             self.mc.MOV(resloc, heap(self.cpu.pos_exc_value()))
-        self.mc.MOV(heap(self.cpu.pos_exception()), imm(0))
-        self.mc.MOV(heap(self.cpu.pos_exc_value()), imm(0))
+        self.mc.MOV(heap(self.cpu.pos_exception()), imm0)
+        self.mc.MOV(heap(self.cpu.pos_exc_value()), imm0)
         return addr
 
     def _gen_guard_overflow(self, guard_op, guard_token):
@@ -1224,8 +1226,8 @@
         elif guard_opnum == rop.GUARD_OVERFLOW:
             return self.implement_guard(guard_token, 'NO')
         else:
-            print "int_xxx_ovf followed by", guard_op.getopname()
-            raise AssertionError
+            not_implemented("int_xxx_ovf followed by %s" %
+                            guard_op.getopname())
 
     def genop_guard_int_add_ovf(self, op, guard_op, guard_token, arglocs, result_loc):
         self.genop_int_add(op, arglocs, result_loc)
@@ -1288,7 +1290,7 @@
     def genop_guard_guard_nonnull_class(self, ign_1, guard_op,
                                         guard_token, locs, ign_2):
         self.mc.ensure_bytes_available(256)
-        self.mc.CMP(locs[0], imm(1))
+        self.mc.CMP(locs[0], imm1)
         # Patched below
         self.mc.J_il8(rx86.Conditions['B'], 0)
         jb_location = self.mc.get_relative_pos()
@@ -1637,25 +1639,34 @@
         sizeloc = arglocs[0]
         assert isinstance(sizeloc, ImmedLoc)
         size = sizeloc.value
+        signloc = arglocs[1]
 
         if isinstance(op.getarg(0), Const):
             x = imm(op.getarg(0).getint())
         else:
-            x = arglocs[1]
+            x = arglocs[2]
         if x is eax:
             tmp = ecx
         else:
             tmp = eax
         
-        self._emit_call(x, arglocs, 2, tmp=tmp)
+        self._emit_call(x, arglocs, 3, tmp=tmp)
+
+        if IS_X86_32 and isinstance(resloc, StackLoc) and resloc.width == 8:
+            self.mc.FSTP_b(resloc.value)   # float return
+        elif size == WORD:
+            assert resloc is eax or resloc is xmm0    # a full word
+        elif size == 0:
+            pass    # void return
+        else:
+            # use the code in load_from_mem to do the zero- or sign-extension
+            assert resloc is eax
+            if size == 1:
+                srcloc = eax.lowest8bits()
+            else:
+                srcloc = eax
+            self.load_from_mem(eax, srcloc, sizeloc, signloc)
 
-        if isinstance(resloc, StackLoc) and resloc.width == 8 and IS_X86_32:
-            self.mc.FSTP_b(resloc.value)
-        elif size == 1:
-            self.mc.AND_ri(eax.value, 0xff)
-        elif size == 2:
-            self.mc.AND_ri(eax.value, 0xffff)
-    
     def genop_guard_call_may_force(self, op, guard_op, guard_token,
                                    arglocs, result_loc):
         faildescr = guard_op.getdescr()
@@ -1780,11 +1791,12 @@
                     self.mc.PUSH_i32(loc.getint())
         
         if IS_X86_64:
-            # We clobber this register to pass the arguments, but that's
+            # We clobber these registers to pass the arguments, but that's
             # okay, because consider_cond_call_gc_wb makes sure that any
             # caller-save registers with values in them are present in arglocs,
             # so they are saved on the stack above and restored below 
             self.mc.MOV_rs(edi.value, 0)
+            self.mc.MOV_rs(esi.value, 8)
 
         # misaligned stack in the call, but it's ok because the write barrier
         # is not going to call anything more.  Also, this assumes that the
@@ -1807,20 +1819,16 @@
         self.mc.LEA_rb(resloc.value, FORCE_INDEX_OFS)
 
     def not_implemented_op_discard(self, op, arglocs):
-        msg = "not implemented operation: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation: %s" % op.getopname())
 
     def not_implemented_op(self, op, arglocs, resloc):
-        msg = "not implemented operation with res: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation with res: %s" %
+                        op.getopname())
 
     def not_implemented_op_guard(self, op, guard_op,
                                  failaddr, arglocs, resloc):
-        msg = "not implemented operation (guard): %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation (guard): %s" %
+                        op.getopname())
 
     def mark_gc_roots(self):
         gcrootmap = self.cpu.gc_ll_descr.gcrootmap
@@ -1904,3 +1912,7 @@
 
 def heap(addr):
     return AddressLoc(ImmedLoc(addr), ImmedLoc(0), 0, 0)
+
+def not_implemented(msg):
+    os.write(2, '[x86/asm] %s\n' % msg)
+    raise NotImplementedError(msg)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/regalloc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/regalloc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/regalloc.py	Sat Oct 30 16:41:17 2010
@@ -2,6 +2,7 @@
 """ Register allocation scheme.
 """
 
+import os
 from pypy.jit.metainterp.history import (Box, Const, ConstInt, ConstPtr,
                                          ResOperation, BoxPtr,
                                          LoopToken, INT, REF, FLOAT)
@@ -40,12 +41,10 @@
             return imm(c.value)
         elif isinstance(c, ConstPtr):
             if we_are_translated() and c.value and rgc.can_move(c.value):
-                print "convert_to_imm: ConstPtr needs special care"
-                raise AssertionError
+                not_implemented("convert_to_imm: ConstPtr needs special care")
             return imm(rffi.cast(lltype.Signed, c.value))
         else:
-            print "convert_to_imm: got a %s" % c
-            raise AssertionError
+            not_implemented("convert_to_imm: got a %s" % c)
 
 class X86_64_RegisterManager(X86RegisterManager):
     # r11 omitted because it's used as scratch
@@ -70,8 +69,9 @@
 
     def _get_new_array(self):
         n = self.BASE_CONSTANT_SIZE
+        # known to leak
         self.cur_array = lltype.malloc(rffi.CArray(lltype.Float), n,
-                                       flavor='raw')
+                                       flavor='raw', track_allocation=False)
         self.cur_array_free = n
     _get_new_array._dont_inline_ = True
 
@@ -359,8 +359,8 @@
             if op.is_ovf():
                 if (operations[i + 1].getopnum() != rop.GUARD_NO_OVERFLOW and
                     operations[i + 1].getopnum() != rop.GUARD_OVERFLOW):
-                    print "int_xxx_ovf not followed by guard_(no)_overflow"
-                    raise AssertionError
+                    not_implemented("int_xxx_ovf not followed by "
+                                    "guard_(no)_overflow")
                 return True
             return False
         if (operations[i + 1].getopnum() != rop.GUARD_TRUE and
@@ -412,8 +412,8 @@
                 arg = op.getarg(j)
                 if isinstance(arg, Box):
                     if arg not in start_live:
-                        print "Bogus arg in operation %d at %d" % (op.getopnum(), i)
-                        raise AssertionError
+                        not_implemented("Bogus arg in operation %d at %d" %
+                                        (op.getopnum(), i))
                     longevity[arg] = (start_live[arg], i)
             if op.is_guard():
                 for arg in op.getfailargs():
@@ -421,8 +421,8 @@
                         continue
                     assert isinstance(arg, Box)
                     if arg not in start_live:
-                        print "Bogus arg in guard %d at %d" % (op.getopnum(), i)
-                        raise AssertionError
+                        not_implemented("Bogus arg in guard %d at %d" %
+                                        (op.getopnum(), i))
                     longevity[arg] = (start_live[arg], i)
         for arg in inputargs:
             if arg not in longevity:
@@ -667,7 +667,13 @@
         assert isinstance(calldescr, BaseCallDescr)
         assert len(calldescr.arg_classes) == op.numargs() - 1
         size = calldescr.get_result_size(self.translate_support_code)
-        self._call(op, [imm(size)] + [self.loc(op.getarg(i)) for i in range(op.numargs())],
+        sign = calldescr.is_result_signed()
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self._call(op, [imm(size), sign_loc] +
+                       [self.loc(op.getarg(i)) for i in range(op.numargs())],
                    guard_not_forced_op=guard_not_forced_op)
 
     def consider_call(self, op):
@@ -688,7 +694,7 @@
             self.rm._sync_var(op.getarg(vable_index))
             vable = self.fm.loc(op.getarg(vable_index))
         else:
-            vable = imm(0)
+            vable = imm0
         self._call(op, [imm(size), vable] +
                    [self.loc(op.getarg(i)) for i in range(op.numargs())],
                    guard_not_forced_op=guard_op)
@@ -696,9 +702,13 @@
     def consider_cond_call_gc_wb(self, op):
         assert op.result is None
         args = op.getarglist()
+        loc_newvalue = self.rm.make_sure_var_in_reg(op.getarg(1), args)
+        # ^^^ we force loc_newvalue in a reg (unless it's a Const),
+        # because it will be needed anyway by the following setfield_gc.
+        # It avoids loading it twice from the memory.
         loc_base = self.rm.make_sure_var_in_reg(op.getarg(0), args,
                                                 imm_fine=False)
-        arglocs = [loc_base]
+        arglocs = [loc_base, loc_newvalue]
         # add eax, ecx and edx as extra "arguments" to ensure they are
         # saved and restored.  Fish in self.rm to know which of these
         # registers really need to be saved (a bit of a hack).  Moreover,
@@ -774,15 +784,11 @@
             loc = self.loc(op.getarg(0))
             return self._call(op, [loc])
         # boehm GC (XXX kill the following code at some point)
-        ofs_items, itemsize, ofs = symbolic.get_array_token(rstr.UNICODE, self.translate_support_code)
-        if itemsize == 4:
-            return self._malloc_varsize(ofs_items, ofs, 2, op.getarg(0),
-                                        op.result)
-        elif itemsize == 2:
-            return self._malloc_varsize(ofs_items, ofs, 1, op.getarg(0),
-                                        op.result)
-        else:
-            assert False, itemsize
+        ofs_items, _, ofs = symbolic.get_array_token(rstr.UNICODE,
+                                                   self.translate_support_code)
+        scale = self._get_unicode_item_scale()
+        return self._malloc_varsize(ofs_items, ofs, scale, op.getarg(0),
+                                    op.result)
 
     def _malloc_varsize(self, ofs_items, ofs_length, scale, v, res_v):
         # XXX kill this function at some point
@@ -814,8 +820,9 @@
             arglocs.append(self.loc(op.getarg(0)))
             return self._call(op, arglocs)
         # boehm GC (XXX kill the following code at some point)
-        scale_of_field, basesize, ofs_length, _ = (
+        itemsize, basesize, ofs_length, _, _ = (
             self._unpack_arraydescr(op.getdescr()))
+        scale_of_field = _get_scale(itemsize)
         return self._malloc_varsize(basesize, ofs_length, scale_of_field,
                                     op.getarg(0), op.result)
 
@@ -825,21 +832,19 @@
         ofs = arraydescr.get_base_size(self.translate_support_code)
         size = arraydescr.get_item_size(self.translate_support_code)
         ptr = arraydescr.is_array_of_pointers()
-        scale = 0
-        while (1 << scale) < size:
-            scale += 1
-        assert (1 << scale) == size
-        return scale, ofs, ofs_length, ptr
+        sign = arraydescr.is_item_signed()
+        return size, ofs, ofs_length, ptr, sign
 
     def _unpack_fielddescr(self, fielddescr):
         assert isinstance(fielddescr, BaseFieldDescr)
         ofs = fielddescr.offset
         size = fielddescr.get_field_size(self.translate_support_code)
         ptr = fielddescr.is_pointer_field()
-        return imm(ofs), imm(size), ptr
+        sign = fielddescr.is_field_signed()
+        return imm(ofs), imm(size), ptr, sign
 
     def consider_setfield_gc(self, op):
-        ofs_loc, size_loc, ptr = self._unpack_fielddescr(op.getdescr())
+        ofs_loc, size_loc, _, _ = self._unpack_fielddescr(op.getdescr())
         assert isinstance(size_loc, ImmedLoc)
         if size_loc.value == 1:
             need_lower_byte = True
@@ -866,10 +871,10 @@
     consider_unicodesetitem = consider_strsetitem
 
     def consider_setarrayitem_gc(self, op):
-        scale, ofs, _, ptr = self._unpack_arraydescr(op.getdescr())
+        itemsize, ofs, _, _, _ = self._unpack_arraydescr(op.getdescr())
         args = op.getarglist()
         base_loc  = self.rm.make_sure_var_in_reg(op.getarg(0), args)
-        if scale == 0:
+        if itemsize == 1:
             need_lower_byte = True
         else:
             need_lower_byte = False
@@ -878,30 +883,39 @@
         ofs_loc = self.rm.make_sure_var_in_reg(op.getarg(1), args)
         self.possibly_free_vars(args)
         self.PerformDiscard(op, [base_loc, ofs_loc, value_loc,
-                                 imm(scale), imm(ofs)])
+                                 imm(itemsize), imm(ofs)])
 
     consider_setarrayitem_raw = consider_setarrayitem_gc
 
     def consider_getfield_gc(self, op):
-        ofs_loc, size_loc, _ = self._unpack_fielddescr(op.getdescr())
+        ofs_loc, size_loc, _, sign = self._unpack_fielddescr(op.getdescr())
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
         self.rm.possibly_free_vars(args)
         result_loc = self.force_allocate_reg(op.result)
-        self.Perform(op, [base_loc, ofs_loc, size_loc], result_loc)
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self.Perform(op, [base_loc, ofs_loc, size_loc, sign_loc], result_loc)
 
     consider_getfield_raw = consider_getfield_gc
     consider_getfield_raw_pure = consider_getfield_gc
     consider_getfield_gc_pure = consider_getfield_gc
 
     def consider_getarrayitem_gc(self, op):
-        scale, ofs, _, _ = self._unpack_arraydescr(op.getdescr())
+        itemsize, ofs, _, _, sign = self._unpack_arraydescr(op.getdescr())
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(op.getarg(0), args)
         ofs_loc = self.rm.make_sure_var_in_reg(op.getarg(1), args)
         self.rm.possibly_free_vars_for_op(op)
         result_loc = self.force_allocate_reg(op.result)
-        self.Perform(op, [base_loc, ofs_loc, imm(scale), imm(ofs)], result_loc)
+        if sign:
+            sign_loc = imm1
+        else:
+            sign_loc = imm0
+        self.Perform(op, [base_loc, ofs_loc, imm(itemsize), imm(ofs),
+                          sign_loc], result_loc)
 
     consider_getarrayitem_raw = consider_getarrayitem_gc
     consider_getarrayitem_gc_pure = consider_getarrayitem_gc
@@ -955,41 +969,85 @@
     consider_unicodegetitem = consider_strgetitem
 
     def consider_copystrcontent(self, op):
+        self._consider_copystrcontent(op, is_unicode=False)
+
+    def consider_copyunicodecontent(self, op):
+        self._consider_copystrcontent(op, is_unicode=True)
+
+    def _consider_copystrcontent(self, op, is_unicode):
         # compute the source address
         args = op.getarglist()
         base_loc = self.rm.make_sure_var_in_reg(args[0], args)
         ofs_loc = self.rm.make_sure_var_in_reg(args[2], args)
+        assert args[0] is not args[1]    # forbidden case of aliasing
         self.rm.possibly_free_var(args[0])
-        self.rm.possibly_free_var(args[2])
+        if args[3] is not args[2] is not args[4]:  # MESS MESS MESS: don't free
+            self.rm.possibly_free_var(args[2])     # it if ==args[3] or args[4]
         srcaddr_box = TempBox()
-        srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box)
-        self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc)
+        forbidden_vars = [args[1], args[3], args[4], srcaddr_box]
+        srcaddr_loc = self.rm.force_allocate_reg(srcaddr_box, forbidden_vars)
+        self._gen_address_inside_string(base_loc, ofs_loc, srcaddr_loc,
+                                        is_unicode=is_unicode)
         # compute the destination address
-        base_loc = self.rm.make_sure_var_in_reg(args[1], args)
-        ofs_loc = self.rm.make_sure_var_in_reg(args[3], args)
+        base_loc = self.rm.make_sure_var_in_reg(args[1], forbidden_vars)
+        ofs_loc = self.rm.make_sure_var_in_reg(args[3], forbidden_vars)
         self.rm.possibly_free_var(args[1])
-        self.rm.possibly_free_var(args[3])
+        if args[3] is not args[4]:     # more of the MESS described above
+            self.rm.possibly_free_var(args[3])
+        forbidden_vars = [args[4], srcaddr_box]
         dstaddr_box = TempBox()
-        dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box)
-        self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc)
+        dstaddr_loc = self.rm.force_allocate_reg(dstaddr_box, forbidden_vars)
+        self._gen_address_inside_string(base_loc, ofs_loc, dstaddr_loc,
+                                        is_unicode=is_unicode)
+        # compute the length in bytes
+        length_box = args[4]
+        length_loc = self.loc(length_box)
+        if is_unicode:
+            self.rm.possibly_free_var(length_box)
+            forbidden_vars = [srcaddr_box, dstaddr_box]
+            bytes_box = TempBox()
+            bytes_loc = self.rm.force_allocate_reg(bytes_box, forbidden_vars)
+            scale = self._get_unicode_item_scale()
+            if not (isinstance(length_loc, ImmedLoc) or
+                    isinstance(length_loc, RegLoc)):
+                self.assembler.mov(length_loc, bytes_loc)
+                length_loc = bytes_loc
+            self.assembler.load_effective_addr(length_loc, 0, scale, bytes_loc)
+            length_box = bytes_box
+            length_loc = bytes_loc
         # call memcpy()
-        length_loc = self.loc(args[4])
         self.rm.before_call()
         self.xrm.before_call()
         self.assembler._emit_call(imm(self.assembler.memcpy_addr),
                                   [dstaddr_loc, srcaddr_loc, length_loc])
-        self.rm.possibly_free_var(args[4])
+        self.rm.possibly_free_var(length_box)
         self.rm.possibly_free_var(dstaddr_box)
         self.rm.possibly_free_var(srcaddr_box)
 
-    def _gen_address_inside_string(self, baseloc, ofsloc, resloc):
+    def _gen_address_inside_string(self, baseloc, ofsloc, resloc, is_unicode):
         cpu = self.assembler.cpu
-        ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR,
+        if is_unicode:
+            ofs_items, _, _ = symbolic.get_array_token(rstr.UNICODE,
                                                   self.translate_support_code)
-        assert itemsize == 1
-        self.assembler.load_effective_addr(ofsloc, ofs_items, 0,
+            scale = self._get_unicode_item_scale()
+        else:
+            ofs_items, itemsize, _ = symbolic.get_array_token(rstr.STR,
+                                                  self.translate_support_code)
+            assert itemsize == 1
+            scale = 0
+        self.assembler.load_effective_addr(ofsloc, ofs_items, scale,
                                            resloc, baseloc)
 
+    def _get_unicode_item_scale(self):
+        _, itemsize, _ = symbolic.get_array_token(rstr.UNICODE,
+                                                  self.translate_support_code)
+        if itemsize == 4:
+            return 2
+        elif itemsize == 2:
+            return 1
+        else:
+            raise AssertionError("bad unicode item size")
+
     def consider_jump(self, op):
         assembler = self.assembler
         assert self.jump_target_descr is None
@@ -1024,6 +1082,9 @@
     def consider_debug_merge_point(self, op):
         pass
 
+    def consider_jit_debug(self, op):
+        pass
+
     def get_mark_gc_roots(self, gcrootmap):
         shape = gcrootmap.get_basic_shape(IS_X86_64)
         for v, val in self.fm.frame_bindings.items():
@@ -1043,15 +1104,11 @@
         self.Perform(op, [], loc)
 
     def not_implemented_op(self, op):
-        msg = "[regalloc] Not implemented operation: %s" % op.getopname()
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation: %s" % op.getopname())
 
     def not_implemented_op_with_guard(self, op, guard_op):
-        msg = "[regalloc] Not implemented operation with guard: %s" % (
-            op.getopname(),)
-        print msg
-        raise NotImplementedError(msg)
+        not_implemented("not implemented operation with guard: %s" % (
+            op.getopname(),))
 
 oplist = [RegAlloc.not_implemented_op] * rop._LAST
 oplist_with_guard = [RegAlloc.not_implemented_op_with_guard] * rop._LAST
@@ -1085,3 +1142,14 @@
     # Returns (ebp-20), (ebp-24), (ebp-28)...
     # i.e. the n'th word beyond the fixed frame size.
     return -WORD * (FRAME_FIXED_SIZE + position)
+
+def _get_scale(size):
+    assert size == 1 or size == 2 or size == 4 or size == 8
+    if size < 4:
+        return size - 1         # 1, 2 => 0, 1
+    else:
+        return (size >> 2) + 1  # 4, 8 => 2, 3
+
+def not_implemented(msg):
+    os.write(2, '[x86/regalloc] %s\n' % msg)
+    raise NotImplementedError(msg)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/regloc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/regloc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/regloc.py	Sat Oct 30 16:41:17 2010
@@ -442,8 +442,11 @@
     MOV8 = _binaryop('MOV8')
     MOV16 = _16_bit_binaryop('MOV')
     MOVZX8 = _binaryop('MOVZX8')
+    MOVSX8 = _binaryop('MOVSX8')
     MOVZX16 = _binaryop('MOVZX16')
+    MOVSX16 = _binaryop('MOVSX16')
     MOV32 = _binaryop('MOV32')
+    MOVSX32 = _binaryop('MOVSX32')
     XCHG = _binaryop('XCHG')
 
     PUSH = _unaryop('PUSH')
@@ -473,6 +476,9 @@
     else:
         return ImmedLoc(x)
 
+imm0 = imm(0)
+imm1 = imm(1)
+
 all_extra_instructions = [name for name in LocationCodeBuilder.__dict__
                           if name[0].isupper()]
 all_extra_instructions.sort()

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/runner.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/runner.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/runner.py	Sat Oct 30 16:41:17 2010
@@ -49,11 +49,13 @@
         self.assembler.finish_once()
         self.profile_agent.shutdown()
 
-    def compile_loop(self, inputargs, operations, looptoken):
-        self.assembler.assemble_loop(inputargs, operations, looptoken)
-
-    def compile_bridge(self, faildescr, inputargs, operations):
-        self.assembler.assemble_bridge(faildescr, inputargs, operations)
+    def compile_loop(self, inputargs, operations, looptoken, log=True):
+        self.assembler.assemble_loop(inputargs, operations, looptoken,
+                                     log=log)
+
+    def compile_bridge(self, faildescr, inputargs, operations, log=True):
+        self.assembler.assemble_bridge(faildescr, inputargs, operations,
+                                       log=log)
 
     def set_future_value_int(self, index, intvalue):
         self.assembler.fail_boxes_int.setitem(index, intvalue)
@@ -87,7 +89,9 @@
 
     def execute_token(self, executable_token):
         addr = executable_token._x86_bootstrap_code
+        #llop.debug_print(lltype.Void, ">>>> Entering", addr)
         func = rffi.cast(lltype.Ptr(self.BOOTSTRAP_TP), addr)
+        #llop.debug_print(lltype.Void, "<<<< Back")
         fail_index = self._execute_call(func)
         return self.get_fail_descr_from_number(fail_index)
 
@@ -99,10 +103,7 @@
             LLInterpreter.current_interpreter = self.debug_ll_interpreter
         res = 0
         try:
-            #llop.debug_print(lltype.Void, ">>>> Entering",
-            #                 rffi.cast(lltype.Signed, func))
             res = func()
-            #llop.debug_print(lltype.Void, "<<<< Back")
         finally:
             if not self.translate_support_code:
                 LLInterpreter.current_interpreter = prev_interpreter
@@ -114,7 +115,8 @@
         return CPU386.cast_adr_to_int(adr)
 
     all_null_registers = lltype.malloc(rffi.LONGP.TO, 24,
-                                       flavor='raw', zero=True)
+                                       flavor='raw', zero=True,
+                                       immortal=True)
 
     def force(self, addr_of_force_index):
         TP = rffi.CArrayPtr(lltype.Signed)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/rx86.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/rx86.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/rx86.py	Sat Oct 30 16:41:17 2010
@@ -642,7 +642,10 @@
 define_modrm_modes('MOV8_*i', [rex_w, '\xC6', orbyte(0<<3)], [immediate(2, 'b')], regtype='BYTE')
 
 define_modrm_modes('MOVZX8_r*', [rex_w, '\x0F\xB6', register(1, 8)], regtype='BYTE')
+define_modrm_modes('MOVSX8_r*', [rex_w, '\x0F\xBE', register(1, 8)], regtype='BYTE')
 define_modrm_modes('MOVZX16_r*', [rex_w, '\x0F\xB7', register(1, 8)])
+define_modrm_modes('MOVSX16_r*', [rex_w, '\x0F\xBF', register(1, 8)])
+define_modrm_modes('MOVSX32_r*', [rex_w, '\x63', register(1, 8)])
 
 define_modrm_modes('MOVSD_x*', ['\xF2', rex_nw, '\x0F\x10', register(1,8)], regtype='XMM')
 define_modrm_modes('MOVSD_*x', ['\xF2', rex_nw, '\x0F\x11', register(2,8)], regtype='XMM')

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_assembler.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_assembler.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_assembler.py	Sat Oct 30 16:41:17 2010
@@ -81,7 +81,8 @@
 
     # also test rebuild_faillocs_from_descr(), which should not
     # reproduce the holes at all
-    bytecode = lltype.malloc(rffi.UCHARP.TO, len(mc.content), flavor='raw')
+    bytecode = lltype.malloc(rffi.UCHARP.TO, len(mc.content), flavor='raw',
+                             immortal=True)
     for i in range(len(mc.content)):
         assert 0 <= mc.content[i] <= 255
         bytecode[i] = rffi.cast(rffi.UCHAR, mc.content[i])
@@ -115,7 +116,8 @@
         assert withfloats
         value = random.random() - 0.5
         # make sure it fits into 64 bits
-        tmp = lltype.malloc(rffi.LONGP.TO, 2, flavor='raw')
+        tmp = lltype.malloc(rffi.LONGP.TO, 2, flavor='raw',
+                            track_allocation=False)
         rffi.cast(rffi.DOUBLEP, tmp)[0] = value
         return rffi.cast(rffi.DOUBLEP, tmp)[0], tmp[0], tmp[1]
 
@@ -152,10 +154,12 @@
 
     # prepare the expected target arrays, the descr_bytecode,
     # the 'registers' and the 'stack' arrays according to 'content'
-    xmmregisters = lltype.malloc(rffi.LONGP.TO, 16+ACTUAL_CPU.NUM_REGS+1, flavor='raw')
+    xmmregisters = lltype.malloc(rffi.LONGP.TO, 16+ACTUAL_CPU.NUM_REGS+1,
+                                 flavor='raw', immortal=True)
     registers = rffi.ptradd(xmmregisters, 16)
     stacklen = baseloc + 10
-    stack = lltype.malloc(rffi.LONGP.TO, stacklen, flavor='raw')
+    stack = lltype.malloc(rffi.LONGP.TO, stacklen, flavor='raw',
+                          immortal=True)
     expected_ints = [0] * len(content)
     expected_ptrs = [lltype.nullptr(llmemory.GCREF.TO)] * len(content)
     expected_floats = [0.0] * len(content)
@@ -221,7 +225,7 @@
     descr_bytecode.append(0x00)
     descr_bytecode.append(0xCC)   # end marker
     descr_bytes = lltype.malloc(rffi.UCHARP.TO, len(descr_bytecode),
-                                flavor='raw')
+                                flavor='raw', immortal=True)
     for i in range(len(descr_bytecode)):
         assert 0 <= descr_bytecode[i] <= 255
         descr_bytes[i] = rffi.cast(rffi.UCHAR, descr_bytecode[i])

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_gc_integration.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_gc_integration.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_gc_integration.py	Sat Oct 30 16:41:17 2010
@@ -12,7 +12,7 @@
 from pypy.jit.backend.detect_cpu import getcpuclass
 from pypy.jit.backend.x86.regalloc import RegAlloc
 from pypy.jit.backend.x86.arch import WORD, FRAME_FIXED_SIZE
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.rpython.lltypesystem import lltype, llmemory, rffi
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import rclass, rstr

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_regalloc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_regalloc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_regalloc.py	Sat Oct 30 16:41:17 2010
@@ -11,7 +11,7 @@
 from pypy.jit.backend.x86.regalloc import RegAlloc, X86RegisterManager,\
      FloatConstants, is_comparison_or_ovf_op
 from pypy.jit.backend.x86.arch import IS_X86_32, IS_X86_64
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.rpython.lltypesystem import lltype, llmemory, rffi
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import rclass, rstr

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_runner.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_runner.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_runner.py	Sat Oct 30 16:41:17 2010
@@ -10,7 +10,7 @@
 from pypy.jit.metainterp.resoperation import rop
 from pypy.jit.metainterp.executor import execute
 from pypy.jit.backend.test.runner_test import LLtypeBackendTest
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.tool.udir import udir
 import ctypes
 import sys
@@ -506,8 +506,8 @@
         self.cpu.execute_token(ops.token)
         # check debugging info
         name, struct = self.cpu.assembler.loop_run_counters[0]
-        assert name == 'xyz'
+        assert name == 0       # 'xyz'
         assert struct.i == 10
         self.cpu.finish_once()
         lines = py.path.local(self.logfile + ".count").readlines()
-        assert lines[0] == '10      xyz\n'
+        assert lines[0] == '0:10\n'  # '10      xyz\n'

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_string.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_string.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_string.py	Sat Oct 30 16:41:17 2010
@@ -2,8 +2,12 @@
 from pypy.jit.metainterp.test import test_string
 from pypy.jit.backend.x86.test.test_basic import Jit386Mixin
 
-class TestString(Jit386Mixin, test_string.StringTests):
+class TestString(Jit386Mixin, test_string.TestLLtype):
     # for the individual tests see
     # ====> ../../../metainterp/test/test_string.py
-    CALL = 'call'
-    CALL_PURE = 'call_pure'
+    pass
+
+class TestUnicode(Jit386Mixin, test_string.TestLLtypeUnicode):
+    # for the individual tests see
+    # ====> ../../../metainterp/test/test_string.py
+    pass

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_zrpy_gc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_zrpy_gc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_zrpy_gc.py	Sat Oct 30 16:41:17 2010
@@ -191,6 +191,33 @@
     def run_orig(self, name, n, x):
         self.main_allfuncs(name, n, x)
 
+    def define_libffi_workaround(cls):
+        # XXX: this is a workaround for a bug in database.py.  It seems that
+        # the problem is triggered by optimizeopt/fficall.py, and in
+        # particular by the ``cast_base_ptr_to_instance(Func, llfunc)``: in
+        # these tests, that line is the only place where libffi.Func is
+        # referenced.
+        #
+        # The problem occurs because the gctransformer tries to annotate a
+        # low-level helper to call the __del__ of libffi.Func when it's too
+        # late.
+        #
+        # This workaround works by forcing the annotator (and all the rest of
+        # the toolchain) to see libffi.Func in a "proper" context, not just as
+        # the target of cast_base_ptr_to_instance.  Note that the function
+        # below is *never* called by any actual test, it's just annotated.
+        #
+        from pypy.rlib.libffi import get_libc_name, CDLL, types, ArgChain
+        libc_name = get_libc_name()
+        def f(n, x, *args):
+            libc = CDLL(libc_name)
+            ptr = libc.getpointer('labs', [types.slong], types.slong)
+            chain = ArgChain()
+            chain.arg(n)
+            n = ptr.call(chain, lltype.Signed)
+            return (n, x) + args
+        return None, f, None
+
     def define_compile_framework_1(cls):
         # a moving GC.  Supports malloc_varsize_nonmovable.  Simple test, works
         # without write_barriers and root stack enumeration.

Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_ztranslation.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_ztranslation.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_ztranslation.py	Sat Oct 30 16:41:17 2010
@@ -1,4 +1,4 @@
-import py, os
+import py, os, sys
 from pypy.tool.udir import udir
 from pypy.rlib.jit import JitDriver, OPTIMIZER_FULL, unroll_parameters
 from pypy.rlib.jit import PARAMETERS, dont_look_inside
@@ -63,8 +63,32 @@
                 if k - abs(j):  raise ValueError
                 if k - abs(-j): raise ValueError
             return total * 10
+        #
+        from pypy.rpython.lltypesystem import lltype, rffi
+        from pypy.rlib.libffi import types, CDLL, ArgChain
+        from pypy.rlib.test.test_libffi import get_libm_name
+        libm_name = get_libm_name(sys.platform)
+        jitdriver2 = JitDriver(greens=[], reds = ['i', 'func', 'res', 'x'])
+        def libffi_stuff(i, j):
+            lib = CDLL(libm_name)
+            func = lib.getpointer('fabs', [types.double], types.double)
+            res = 0.0
+            x = float(j)
+            while i > 0:
+                jitdriver2.jit_merge_point(i=i, res=res, func=func, x=x)
+                jitdriver2.can_enter_jit(i=i, res=res, func=func, x=x)
+                func = hint(func, promote=True)
+                argchain = ArgChain()
+                argchain.arg(x)
+                res = func.call(argchain, rffi.DOUBLE)
+                i -= 1
+            return res
+        #
+        def main(i, j):
+            return f(i, j) + libffi_stuff(i, j)
+        expected = f(40, -49)
         res = self.meta_interp(f, [40, -49])
-        assert res == f(40, -49)
+        assert res == expected
 
     def test_direct_assembler_call_translates(self):
         class Thing(object):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/codewriter/call.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/codewriter/call.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/codewriter/call.py	Sat Oct 30 16:41:17 2010
@@ -277,3 +277,11 @@
             return seen.pop()
         else:
             return None
+
+    def could_be_green_field(self, GTYPE, fieldname):
+        GTYPE_fieldname = (GTYPE, fieldname)
+        for jd in self.jitdrivers_sd:
+            if jd.greenfield_info is not None:
+                if GTYPE_fieldname in jd.greenfield_info.green_fields:
+                    return True
+        return False

Modified: pypy/branch/jit-unroll-loops/pypy/jit/codewriter/codewriter.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/codewriter/codewriter.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/codewriter/codewriter.py	Sat Oct 30 16:41:17 2010
@@ -95,18 +95,18 @@
             print '%s:' % (ssarepr.name,)
             print format_assembler(ssarepr)
         else:
-            dir = udir.ensure("jitcodes", dir=1)
-            if portal_jitdriver:
-                name = "%02d_portal_runner" % (portal_jitdriver.index,)
-            elif ssarepr.name and ssarepr.name != '?':
-                name = ssarepr.name
-            else:
-                name = 'unnamed' % id(ssarepr)
-            i = 1
-            extra = ''
-            while name+extra in self._seen_files:
-                i += 1
-                extra = '.%d' % i
-            self._seen_files.add(name+extra)
-            dir.join(name+extra).write(format_assembler(ssarepr))
             log.dot()
+        dir = udir.ensure("jitcodes", dir=1)
+        if portal_jitdriver:
+            name = "%02d_portal_runner" % (portal_jitdriver.index,)
+        elif ssarepr.name and ssarepr.name != '?':
+            name = ssarepr.name
+        else:
+            name = 'unnamed' % id(ssarepr)
+        i = 1
+        extra = ''
+        while name+extra in self._seen_files:
+            i += 1
+            extra = '.%d' % i
+        self._seen_files.add(name+extra)
+        dir.join(name+extra).write(format_assembler(ssarepr))

Modified: pypy/branch/jit-unroll-loops/pypy/jit/codewriter/effectinfo.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/codewriter/effectinfo.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/codewriter/effectinfo.py	Sat Oct 30 16:41:17 2010
@@ -18,19 +18,34 @@
     # the 'oopspecindex' field is one of the following values:
     OS_NONE                     = 0    # normal case, no oopspec
     OS_ARRAYCOPY                = 1    # "list.ll_arraycopy"
-    OS_STR_CONCAT               = 2    # "stroruni.concat"
-    OS_UNI_CONCAT               = 3    # "stroruni.concat"
-    OS_STR_SLICE                = 4    # "stroruni.slice"
-    OS_UNI_SLICE                = 5    # "stroruni.slice"
-    OS_STR_EQUAL                = 6    # "stroruni.equal"
-    OS_UNI_EQUAL                = 7    # "stroruni.equal"
-    OS_STREQ_SLICE_CHECKNULL    = 8    # s2!=NULL and s1[x:x+length]==s2
-    OS_STREQ_SLICE_NONNULL      = 9    # s1[x:x+length]==s2   (assert s2!=NULL)
-    OS_STREQ_SLICE_CHAR         = 10   # s1[x:x+length]==char
-    OS_STREQ_NONNULL            = 11   # s1 == s2    (assert s1!=NULL,s2!=NULL)
-    OS_STREQ_NONNULL_CHAR       = 12   # s1 == char  (assert s1!=NULL)
-    OS_STREQ_CHECKNULL_CHAR     = 13   # s1!=NULL and s1==char
-    OS_STREQ_LENGTHOK           = 14   # s1 == s2    (assert len(s1)==len(s2))
+    OS_STR2UNICODE              = 2    # "str.str2unicode"
+    #
+    OS_STR_CONCAT               = 22   # "stroruni.concat"
+    OS_STR_SLICE                = 23   # "stroruni.slice"
+    OS_STR_EQUAL                = 24   # "stroruni.equal"
+    OS_STREQ_SLICE_CHECKNULL    = 25   # s2!=NULL and s1[x:x+length]==s2
+    OS_STREQ_SLICE_NONNULL      = 26   # s1[x:x+length]==s2   (assert s2!=NULL)
+    OS_STREQ_SLICE_CHAR         = 27   # s1[x:x+length]==char
+    OS_STREQ_NONNULL            = 28   # s1 == s2    (assert s1!=NULL,s2!=NULL)
+    OS_STREQ_NONNULL_CHAR       = 29   # s1 == char  (assert s1!=NULL)
+    OS_STREQ_CHECKNULL_CHAR     = 30   # s1!=NULL and s1==char
+    OS_STREQ_LENGTHOK           = 31   # s1 == s2    (assert len(s1)==len(s2))
+    #
+    OS_UNI_CONCAT               = 42   #
+    OS_UNI_SLICE                = 43   #
+    OS_UNI_EQUAL                = 44   #
+    OS_UNIEQ_SLICE_CHECKNULL    = 45   #
+    OS_UNIEQ_SLICE_NONNULL      = 46   #
+    OS_UNIEQ_SLICE_CHAR         = 47   #
+    OS_UNIEQ_NONNULL            = 48   #   the same for unicode
+    OS_UNIEQ_NONNULL_CHAR       = 49   #   (must be the same amount as for
+    OS_UNIEQ_CHECKNULL_CHAR     = 50   #   STR, in the same order)
+    OS_UNIEQ_LENGTHOK           = 51   #
+    _OS_offset_uni              = OS_UNI_CONCAT - OS_STR_CONCAT
+    #
+    OS_LIBFFI_PREPARE           = 60
+    OS_LIBFFI_PUSH_ARG          = 61
+    OS_LIBFFI_CALL              = 62
 
     def __new__(cls, readonly_descrs_fields,
                 write_descrs_fields, write_descrs_arrays,

Modified: pypy/branch/jit-unroll-loops/pypy/jit/codewriter/format.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/codewriter/format.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/codewriter/format.py	Sat Oct 30 16:41:17 2010
@@ -80,7 +80,8 @@
 
 def assert_format(ssarepr, expected):
     asm = format_assembler(ssarepr)
-    expected = str(py.code.Source(expected)).strip() + '\n'
+    if expected != '':
+        expected = str(py.code.Source(expected)).strip() + '\n'
     asmlines = asm.split("\n")
     explines = expected.split("\n")
     for asm, exp in zip(asmlines, explines):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/codewriter/jtransform.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/codewriter/jtransform.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/codewriter/jtransform.py	Sat Oct 30 16:41:17 2010
@@ -172,6 +172,7 @@
 
     def rewrite_op_same_as(self, op): pass
     def rewrite_op_cast_pointer(self, op): pass
+    def rewrite_op_cast_opaque_ptr(self, op): pass   # rlib.rerased
     def rewrite_op_cast_primitive(self, op): pass
     def rewrite_op_cast_bool_to_int(self, op): pass
     def rewrite_op_cast_bool_to_uint(self, op): pass
@@ -316,8 +317,14 @@
             prepare = self._handle_list_call
         elif oopspec_name.startswith('stroruni.'):
             prepare = self._handle_stroruni_call
+        elif oopspec_name == 'str.str2unicode':
+            prepare = self._handle_str2unicode_call
         elif oopspec_name.startswith('virtual_ref'):
             prepare = self._handle_virtual_ref_call
+        elif oopspec_name.startswith('jit.'):
+            prepare = self._handle_jit_call
+        elif oopspec_name.startswith('libffi_'):
+            prepare = self._handle_libffi_call
         else:
             prepare = self.prepare_builtin_call
         try:
@@ -427,7 +434,8 @@
                                   op.result)
 
     def rewrite_op_free(self, op):
-        assert op.args[1].value == 'raw'
+        flags = op.args[1].value
+        assert flags['flavor'] == 'raw'
         ARRAY = op.args[0].concretetype.TO
         return self._do_builtin_call(op, 'raw_free', [op.args[0]],
                                      extra = (ARRAY,), extrakey = ARRAY)
@@ -519,7 +527,12 @@
         # check for deepfrozen structures that force constant-folding
         immut = v_inst.concretetype.TO._immutable_field(c_fieldname.value)
         if immut:
-            pure = '_pure'
+            if (self.callcontrol is not None and
+                self.callcontrol.could_be_green_field(v_inst.concretetype.TO,
+                                                      c_fieldname.value)):
+                pure = '_greenfield'
+            else:
+                pure = '_pure'
             if immut == "[*]":
                 self.immutable_arrays[op.result] = True
         else:
@@ -819,6 +832,8 @@
     def rewrite_op_jit_marker(self, op):
         key = op.args[0].value
         jitdriver = op.args[1].value
+        if not jitdriver.active:
+            return []
         return getattr(self, 'handle_jit_marker__%s' % key)(op, jitdriver)
 
     def handle_jit_marker__jit_merge_point(self, op, jitdriver):
@@ -852,6 +867,15 @@
                     (self.graph,))
         return []
 
+    def _handle_jit_call(self, op, oopspec_name, args):
+        if oopspec_name == 'jit.debug':
+            return SpaceOperation('jit_debug', args, None)
+        elif oopspec_name == 'jit.assert_green':
+            kind = getkind(args[0].concretetype)
+            return SpaceOperation('%s_assert_green' % kind, args, None)
+        else:
+            raise AssertionError("missing support for %r" % oopspec_name)
+
     # ----------
     # Lists.
 
@@ -870,17 +894,21 @@
             prefix = 'do_resizable_'
             ARRAY = LIST.items.TO
             if self._array_of_voids(ARRAY):
-                raise NotSupported("resizable lists of voids")
-            descrs = (self.cpu.arraydescrof(ARRAY),
-                      self.cpu.fielddescrof(LIST, 'length'),
-                      self.cpu.fielddescrof(LIST, 'items'),
-                      self.cpu.sizeof(LIST))
+                prefix += 'void_'
+                descrs = ()
+            else:
+                descrs = (self.cpu.arraydescrof(ARRAY),
+                          self.cpu.fielddescrof(LIST, 'length'),
+                          self.cpu.fielddescrof(LIST, 'items'),
+                          self.cpu.sizeof(LIST))
         else:
             prefix = 'do_fixed_'
             if self._array_of_voids(LIST):
-                raise NotSupported("fixed lists of voids")
-            arraydescr = self.cpu.arraydescrof(LIST)
-            descrs = (arraydescr,)
+                prefix += 'void_'
+                descrs = ()
+            else:
+                arraydescr = self.cpu.arraydescrof(LIST)
+                descrs = (arraydescr,)
         #
         try:
             meth = getattr(self, prefix + oopspec_name.replace('.', '_'))
@@ -919,6 +947,11 @@
                                              descr, args[1]], v_posindex)
             return v_posindex, [op0, op1]
 
+    def _prepare_void_list_getset(self, op):
+        non_negative, can_raise = self._get_list_nonneg_canraise_flags(op)
+        if can_raise:
+            raise NotSupported("list operation can raise")
+
     def _get_initial_newlist_length(self, op, args):
         # normalize number of arguments to the 'newlist' function
         if len(args) > 1:
@@ -990,6 +1023,12 @@
     def do_fixed_list_ll_arraycopy(self, op, args, arraydescr):
         return self._handle_oopspec_call(op, args, EffectInfo.OS_ARRAYCOPY)
 
+    def do_fixed_void_list_getitem(self, op, args):
+        self._prepare_void_list_getset(op)
+        return []
+    do_fixed_void_list_getitem_foldable = do_fixed_void_list_getitem
+    do_fixed_void_list_setitem = do_fixed_void_list_getitem
+
     # ---------- resizable lists ----------
 
     def do_resizable_newlist(self, op, args, arraydescr, lengthdescr,
@@ -1025,11 +1064,19 @@
         return SpaceOperation('getfield_gc_i',
                               [args[0], lengthdescr], op.result)
 
+    def do_resizable_void_list_getitem(self, op, args):
+        self._prepare_void_list_getset(op)
+        return []
+    do_resizable_void_list_getitem_foldable = do_resizable_void_list_getitem
+    do_resizable_void_list_setitem = do_resizable_void_list_getitem
+
     # ----------
     # Strings and Unicodes.
 
-    def _handle_oopspec_call(self, op, args, oopspecindex):
+    def _handle_oopspec_call(self, op, args, oopspecindex, extraeffect=None):
         calldescr = self.callcontrol.getcalldescr(op, oopspecindex)
+        if extraeffect:
+            calldescr.get_extra_info().extraeffect = extraeffect
         if isinstance(op.args[0].value, str):
             pass  # for tests only
         else:
@@ -1055,28 +1102,32 @@
                             [c_func] + [varoftype(T) for T in argtypes],
                             varoftype(resulttype))
         calldescr = self.callcontrol.getcalldescr(op, oopspecindex)
-        func = heaptracker.adr2int(
-            llmemory.cast_ptr_to_adr(c_func.value))
+        if isinstance(c_func.value, str):    # in tests only
+            func = c_func.value
+        else:
+            func = heaptracker.adr2int(
+                llmemory.cast_ptr_to_adr(c_func.value))
         _callinfo_for_oopspec[oopspecindex] = calldescr, func
 
     def _handle_stroruni_call(self, op, oopspec_name, args):
-        if args[0].concretetype.TO == rstr.STR:
+        SoU = args[0].concretetype     # Ptr(STR) or Ptr(UNICODE)
+        if SoU.TO == rstr.STR:
             dict = {"stroruni.concat": EffectInfo.OS_STR_CONCAT,
                     "stroruni.slice":  EffectInfo.OS_STR_SLICE,
                     "stroruni.equal":  EffectInfo.OS_STR_EQUAL,
                     }
-        elif args[0].concretetype.TO == rstr.UNICODE:
+            CHR = lltype.Char
+        elif SoU.TO == rstr.UNICODE:
             dict = {"stroruni.concat": EffectInfo.OS_UNI_CONCAT,
                     "stroruni.slice":  EffectInfo.OS_UNI_SLICE,
                     "stroruni.equal":  EffectInfo.OS_UNI_EQUAL,
                     }
+            CHR = lltype.UniChar
         else:
             assert 0, "args[0].concretetype must be STR or UNICODE"
         #
         if oopspec_name == "stroruni.equal":
-            SoU = args[0].concretetype     # Ptr(STR) or Ptr(UNICODE)
             for otherindex, othername, argtypes, resulttype in [
-
                 (EffectInfo.OS_STREQ_SLICE_CHECKNULL,
                      "str.eq_slice_checknull",
                      [SoU, lltype.Signed, lltype.Signed, SoU],
@@ -1087,7 +1138,7 @@
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_SLICE_CHAR,
                      "str.eq_slice_char",
-                     [SoU, lltype.Signed, lltype.Signed, lltype.Char],
+                     [SoU, lltype.Signed, lltype.Signed, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_NONNULL,
                      "str.eq_nonnull",
@@ -1095,22 +1146,27 @@
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_NONNULL_CHAR,
                      "str.eq_nonnull_char",
-                     [SoU, lltype.Char],
+                     [SoU, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_CHECKNULL_CHAR,
                      "str.eq_checknull_char",
-                     [SoU, lltype.Char],
+                     [SoU, CHR],
                      lltype.Signed),
                 (EffectInfo.OS_STREQ_LENGTHOK,
                      "str.eq_lengthok",
                      [SoU, SoU],
                      lltype.Signed),
                 ]:
+                if args[0].concretetype.TO == rstr.UNICODE:
+                    otherindex += EffectInfo._OS_offset_uni
                 self._register_extra_helper(otherindex, othername,
                                             argtypes, resulttype)
         #
         return self._handle_oopspec_call(op, args, dict[oopspec_name])
 
+    def _handle_str2unicode_call(self, op, oopspec_name, args):
+        return self._handle_oopspec_call(op, args, EffectInfo.OS_STR2UNICODE)
+
     # ----------
     # VirtualRefs.
 
@@ -1121,6 +1177,23 @@
                                           vrefinfo.JIT_VIRTUAL_REF)
         return SpaceOperation(oopspec_name, list(args), op.result)
 
+    # -----------
+    # rlib.libffi
+
+    def _handle_libffi_call(self, op, oopspec_name, args):
+        if oopspec_name == 'libffi_prepare_call':
+            oopspecindex = EffectInfo.OS_LIBFFI_PREPARE
+            extraeffect = EffectInfo.EF_CANNOT_RAISE
+        elif oopspec_name.startswith('libffi_push_'):
+            oopspecindex = EffectInfo.OS_LIBFFI_PUSH_ARG
+            extraeffect = EffectInfo.EF_CANNOT_RAISE
+        elif oopspec_name.startswith('libffi_call_'):
+            oopspecindex = EffectInfo.OS_LIBFFI_CALL
+            extraeffect = EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE
+        else:
+            assert False, 'unsupported oopspec: %s' % oopspec_name
+        return self._handle_oopspec_call(op, args, oopspecindex, extraeffect)
+
     def rewrite_op_jit_force_virtual(self, op):
         return self._do_builtin_call(op)
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/codewriter/support.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/codewriter/support.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/codewriter/support.py	Sat Oct 30 16:41:17 2010
@@ -1,5 +1,5 @@
 import sys
-from pypy.rpython.lltypesystem import lltype, rclass
+from pypy.rpython.lltypesystem import lltype, rclass, rffi
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython import rlist
 from pypy.rpython.lltypesystem import rstr as ll_rstr, rdict as ll_rdict
@@ -8,6 +8,7 @@
 from pypy.rpython.ootypesystem import rdict as oo_rdict
 from pypy.rpython.llinterp import LLInterpreter
 from pypy.rpython.extregistry import ExtRegistryEntry
+from pypy.rpython.annlowlevel import cast_base_ptr_to_instance
 from pypy.translator.simplify import get_funcobj
 from pypy.translator.unsimplify import split_block
 from pypy.objspace.flow.model import Constant
@@ -60,7 +61,7 @@
     return rtyper.annotator.translator.graphs[0]
 
 def split_before_jit_merge_point(graph, portalblock, portalopindex):
-    """Find the block with 'jit_merge_point' and split just before,
+    """Split the block just before the 'jit_merge_point',
     making sure the input args are in the canonical order.
     """
     # split the block just before the jit_merge_point()
@@ -217,6 +218,33 @@
     else:
         return x
 
+
+# libffi support
+# --------------
+
+def func(llfunc):
+    from pypy.rlib.libffi import Func
+    return cast_base_ptr_to_instance(Func, llfunc)
+
+def _ll_1_libffi_prepare_call(llfunc):
+    return func(llfunc)._prepare()
+
+def _ll_4_libffi_push_int(llfunc, value, ll_args, i):
+    return func(llfunc)._push_int(value, ll_args, i)
+
+def _ll_4_libffi_push_float(llfunc, value, ll_args, i):
+    return func(llfunc)._push_float(value, ll_args, i)
+
+def _ll_3_libffi_call_int(llfunc, funcsym, ll_args):
+    return func(llfunc)._do_call(funcsym, ll_args, rffi.LONG)
+
+def _ll_3_libffi_call_float(llfunc, funcsym, ll_args):
+    return func(llfunc)._do_call(funcsym, ll_args, rffi.DOUBLE)
+
+def _ll_3_libffi_call_void(llfunc, funcsym, ll_args):
+    return func(llfunc)._do_call(funcsym, ll_args, lltype.Void)
+
+
 # in the following calls to builtins, the JIT is allowed to look inside:
 inline_calls_to = [
     ('int_floordiv_ovf_zer', [lltype.Signed, lltype.Signed], lltype.Signed),

Modified: pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_codewriter.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_codewriter.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_codewriter.py	Sat Oct 30 16:41:17 2010
@@ -45,6 +45,7 @@
         self.portal_graph = portal_graph
         self.portal_runner_ptr = "???"
         self.virtualizable_info = None
+        self.greenfield_info = None
 
 
 def test_loop():

Modified: pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_effectinfo.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_effectinfo.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_effectinfo.py	Sat Oct 30 16:41:17 2010
@@ -1,7 +1,8 @@
 from pypy.rpython.lltypesystem.rclass import OBJECT
 from pypy.rpython.lltypesystem import lltype
 from pypy.rpython.ootypesystem import ootype
-from pypy.jit.codewriter.effectinfo import effectinfo_from_writeanalyze
+from pypy.jit.codewriter.effectinfo import effectinfo_from_writeanalyze,\
+    EffectInfo
 
 class FakeCPU:
     def fielddescrof(self, T, fieldname):
@@ -9,6 +10,14 @@
     def arraydescrof(self, A):
         return ('arraydescr', A)
 
+def test_no_oopspec_duplicate():
+    # check that all the various EffectInfo.OS_* have unique values
+    oopspecs = set()
+    for name, value in EffectInfo.__dict__.iteritems():
+        if name.startswith('OS_'):
+            assert value not in oopspecs
+            oopspecs.add(value)
+
 def test_include_read_field():
     S = lltype.GcStruct("S", ("a", lltype.Signed))
     effects = frozenset([("readstruct", lltype.Ptr(S), "a")])

Modified: pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_jtransform.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_jtransform.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_jtransform.py	Sat Oct 30 16:41:17 2010
@@ -77,7 +77,32 @@
 class FakeBuiltinCallControl:
     def guess_call_kind(self, op):
         return 'builtin'
-    def getcalldescr(self, op, oopspecindex):
+    def getcalldescr(self, op, oopspecindex=None):
+        assert oopspecindex is not None    # in this test
+        EI = effectinfo.EffectInfo
+        if oopspecindex != EI.OS_ARRAYCOPY:
+            PSTR = lltype.Ptr(rstr.STR)
+            PUNICODE = lltype.Ptr(rstr.UNICODE)
+            INT = lltype.Signed
+            UNICHAR = lltype.UniChar
+            argtypes = {
+             EI.OS_STR2UNICODE:([PSTR], PUNICODE),
+             EI.OS_STR_CONCAT: ([PSTR, PSTR], PSTR),
+             EI.OS_STR_SLICE:  ([PSTR, INT, INT], PSTR),
+             EI.OS_UNI_CONCAT: ([PUNICODE, PUNICODE], PUNICODE),
+             EI.OS_UNI_SLICE:  ([PUNICODE, INT, INT], PUNICODE),
+             EI.OS_UNI_EQUAL:  ([PUNICODE, PUNICODE], lltype.Bool),
+             EI.OS_UNIEQ_SLICE_CHECKNULL:([PUNICODE, INT, INT, PUNICODE], INT),
+             EI.OS_UNIEQ_SLICE_NONNULL:  ([PUNICODE, INT, INT, PUNICODE], INT),
+             EI.OS_UNIEQ_SLICE_CHAR:     ([PUNICODE, INT, INT, UNICHAR], INT),
+             EI.OS_UNIEQ_NONNULL:        ([PUNICODE, PUNICODE], INT),
+             EI.OS_UNIEQ_NONNULL_CHAR:   ([PUNICODE, UNICHAR], INT),
+             EI.OS_UNIEQ_CHECKNULL_CHAR: ([PUNICODE, UNICHAR], INT),
+             EI.OS_UNIEQ_LENGTHOK:       ([PUNICODE, PUNICODE], INT),
+            }
+            argtypes = argtypes[oopspecindex]
+            assert argtypes[0] == [v.concretetype for v in op.args[1:]]
+            assert argtypes[1] == op.result.concretetype
         return 'calldescr-%d' % oopspecindex
     def calldescr_canraise(self, calldescr):
         return False
@@ -662,6 +687,80 @@
     assert block.operations[1].result is None
     assert block.exits[0].args == [v1]
 
+def test_jit_merge_point_1():
+    class FakeJitDriverSD:
+        index = 42
+        class jitdriver:
+            active = True
+            greens = ['green1', 'green2', 'voidgreen3']
+            reds = ['red1', 'red2', 'voidred3']
+    jd = FakeJitDriverSD()
+    v1 = varoftype(lltype.Signed)
+    v2 = varoftype(lltype.Signed)
+    vvoid1 = varoftype(lltype.Void)
+    v3 = varoftype(lltype.Signed)
+    v4 = varoftype(lltype.Signed)
+    vvoid2 = varoftype(lltype.Void)
+    v5 = varoftype(lltype.Void)
+    op = SpaceOperation('jit_marker',
+                        [Constant('jit_merge_point', lltype.Void),
+                         Constant(jd.jitdriver, lltype.Void),
+                         v1, v2, vvoid1, v3, v4, vvoid2], v5)
+    tr = Transformer()
+    tr.portal_jd = jd
+    oplist = tr.rewrite_operation(op)
+    assert len(oplist) == 6
+    assert oplist[0].opname == '-live-'
+    assert oplist[1].opname == 'int_guard_value'
+    assert oplist[1].args   == [v1]
+    assert oplist[2].opname == '-live-'
+    assert oplist[3].opname == 'int_guard_value'
+    assert oplist[3].args   == [v2]
+    assert oplist[4].opname == 'jit_merge_point'
+    assert oplist[4].args[0].value == 42
+    assert list(oplist[4].args[1]) == [v1, v2]
+    assert list(oplist[4].args[4]) == [v3, v4]
+    assert oplist[5].opname == '-live-'
+
+def test_getfield_gc():
+    S = lltype.GcStruct('S', ('x', lltype.Char))
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
+def test_getfield_gc_pure():
+    S = lltype.GcStruct('S', ('x', lltype.Char),
+                        hints={'immutable': True})
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i_pure'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
+def test_getfield_gc_greenfield():
+    class FakeCC:
+        def get_vinfo(self, v):
+            return None
+        def could_be_green_field(self, S1, name1):
+            assert S1 is S
+            assert name1 == 'x'
+            return True
+    S = lltype.GcStruct('S', ('x', lltype.Char),
+                        hints={'immutable': True})
+    v1 = varoftype(lltype.Ptr(S))
+    v2 = varoftype(lltype.Char)
+    op = SpaceOperation('getfield', [v1, Constant('x', lltype.Void)], v2)
+    op1 = Transformer(FakeCPU(), FakeCC()).rewrite_operation(op)
+    assert op1.opname == 'getfield_gc_i_greenfield'
+    assert op1.args == [v1, ('fielddescr', S, 'x')]
+    assert op1.result == v2
+
 def test_int_abs():
     v1 = varoftype(lltype.Signed)
     v2 = varoftype(lltype.Signed)
@@ -766,6 +865,46 @@
     assert op1.args[3] == ListOfKind('ref', [v1])
     assert op1.result == v4
 
+def test_str2unicode():
+    # test that the oopspec is present and correctly transformed
+    PSTR = lltype.Ptr(rstr.STR)
+    PUNICODE = lltype.Ptr(rstr.UNICODE)
+    FUNC = lltype.FuncType([PSTR], PUNICODE)
+    func = lltype.functionptr(FUNC, 'll_str2unicode',
+                            _callable=rstr.LLHelpers.ll_str2unicode)
+    v1 = varoftype(PSTR)
+    v2 = varoftype(PUNICODE)
+    op = SpaceOperation('direct_call', [const(func), v1], v2)
+    tr = Transformer(FakeCPU(), FakeBuiltinCallControl())
+    op1 = tr.rewrite_operation(op)
+    assert op1.opname == 'residual_call_r_r'
+    assert op1.args[0].value == func
+    assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_STR2UNICODE
+    assert op1.args[2] == ListOfKind('ref', [v1])
+    assert op1.result == v2
+
+def test_unicode_eq_checknull_char():
+    # test that the oopspec is present and correctly transformed
+    PUNICODE = lltype.Ptr(rstr.UNICODE)
+    FUNC = lltype.FuncType([PUNICODE, PUNICODE], lltype.Bool)
+    func = lltype.functionptr(FUNC, 'll_streq',
+                              _callable=rstr.LLHelpers.ll_streq)
+    v1 = varoftype(PUNICODE)
+    v2 = varoftype(PUNICODE)
+    v3 = varoftype(lltype.Bool)
+    op = SpaceOperation('direct_call', [const(func), v1, v2], v3)
+    tr = Transformer(FakeCPU(), FakeBuiltinCallControl())
+    op1 = tr.rewrite_operation(op)
+    assert op1.opname == 'residual_call_r_i'
+    assert op1.args[0].value == func
+    assert op1.args[1] == 'calldescr-%d' % effectinfo.EffectInfo.OS_UNI_EQUAL
+    assert op1.args[2] == ListOfKind('ref', [v1, v2])
+    assert op1.result == v3
+    # test that the OS_UNIEQ_* functions are registered
+    cifo = effectinfo._callinfo_for_oopspec
+    assert effectinfo.EffectInfo.OS_UNIEQ_SLICE_NONNULL in cifo
+    assert effectinfo.EffectInfo.OS_UNIEQ_CHECKNULL_CHAR in cifo
+
 def test_list_ll_arraycopy():
     from pypy.rlib.rgc import ll_arraycopy
     LIST = lltype.GcArray(lltype.Signed)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_list.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_list.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/codewriter/test/test_list.py	Sat Oct 30 16:41:17 2010
@@ -19,6 +19,7 @@
 class FakeCPU:
     class arraydescrof(AbstractDescr):
         def __init__(self, ARRAY):
+            assert ARRAY.OF != lltype.Void
             self.ARRAY = ARRAY
         def __repr__(self):
             return ''

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/blackhole.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/blackhole.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/blackhole.py	Sat Oct 30 16:41:17 2010
@@ -760,6 +760,20 @@
     def bhimpl_debug_fatalerror(msg):
         llop.debug_fatalerror(lltype.Void, msg)
 
+    @arguments("r", "i", "i", "i", "i")
+    def bhimpl_jit_debug(string, arg1=0, arg2=0, arg3=0, arg4=0):
+        pass
+
+    @arguments("i")
+    def bhimpl_int_assert_green(x):
+        pass
+    @arguments("r")
+    def bhimpl_ref_assert_green(x):
+        pass
+    @arguments("f")
+    def bhimpl_float_assert_green(x):
+        pass
+
     # ----------
     # the main hints and recursive calls
 
@@ -1073,6 +1087,10 @@
     bhimpl_getfield_vable_r = bhimpl_getfield_gc_r
     bhimpl_getfield_vable_f = bhimpl_getfield_gc_f
 
+    bhimpl_getfield_gc_i_greenfield = bhimpl_getfield_gc_i
+    bhimpl_getfield_gc_r_greenfield = bhimpl_getfield_gc_r
+    bhimpl_getfield_gc_f_greenfield = bhimpl_getfield_gc_f
+
     @arguments("cpu", "i", "d", returns="i")
     def bhimpl_getfield_raw_i(cpu, struct, fielddescr):
         return cpu.bh_getfield_raw_i(struct, fielddescr)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py	Sat Oct 30 16:41:17 2010
@@ -140,7 +140,6 @@
     if not we_are_translated():
         show_loop(metainterp_sd)
         TreeLoop.check_consistency_of(inputargs, operations)
-        pass
     metainterp_sd.profiler.start_backend()
     debug_start("jit-backend")
     try:
@@ -387,7 +386,8 @@
         from pypy.jit.metainterp.resume import force_from_resumedata
         metainterp_sd = self.metainterp_sd
         vinfo = self.jitdriver_sd.virtualizable_info
-        all_virtuals = force_from_resumedata(metainterp_sd, self, vinfo)
+        ginfo = self.jitdriver_sd.greenfield_info
+        all_virtuals = force_from_resumedata(metainterp_sd, self, vinfo, ginfo)
         # The virtualizable data was stored on the real virtualizable above.
         # Handle all_virtuals: keep them for later blackholing from the
         # future failure of the GUARD_NOT_FORCED
@@ -616,5 +616,5 @@
         ResOperation(rop.FINISH, finishargs, None, descr=jd.portal_finishtoken)
         ]
     operations[1].setfailargs([])
-    cpu.compile_loop(inputargs, operations, loop_token)
+    cpu.compile_loop(inputargs, operations, loop_token, log=False)
     return loop_token

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/executor.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/executor.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/executor.py	Sat Oct 30 16:41:17 2010
@@ -80,6 +80,9 @@
 do_call_loopinvariant = do_call
 do_call_may_force = do_call
 
+def do_call_c(cpu, metainterp, argboxes, descr):
+    raise NotImplementedError("Should never be called directly")
+
 def do_getarrayitem_gc(cpu, _, arraybox, indexbox, arraydescr):
     array = arraybox.getref_base()
     index = indexbox.getint()
@@ -205,8 +208,8 @@
 
 def do_copystrcontent(cpu, _, srcbox, dstbox,
                       srcstartbox, dststartbox, lengthbox):
-    src = srcbox.getptr(lltype.Ptr(rstr.STR))
-    dst = dstbox.getptr(lltype.Ptr(rstr.STR))
+    src = srcbox.getref(lltype.Ptr(rstr.STR))
+    dst = dstbox.getref(lltype.Ptr(rstr.STR))
     srcstart = srcstartbox.getint()
     dststart = dststartbox.getint()
     length = lengthbox.getint()
@@ -214,8 +217,8 @@
 
 def do_copyunicodecontent(cpu, _, srcbox, dstbox,
                           srcstartbox, dststartbox, lengthbox):
-    src = srcbox.getptr(lltype.Ptr(rstr.UNICODE))
-    dst = dstbox.getptr(lltype.Ptr(rstr.UNICODE))
+    src = srcbox.getref(lltype.Ptr(rstr.UNICODE))
+    dst = dstbox.getref(lltype.Ptr(rstr.UNICODE))
     srcstart = srcstartbox.getint()
     dststart = dststartbox.getint()
     length = lengthbox.getint()
@@ -304,6 +307,7 @@
                          rop.CALL_ASSEMBLER,
                          rop.COND_CALL_GC_WB,
                          rop.DEBUG_MERGE_POINT,
+                         rop.JIT_DEBUG,
                          rop.SETARRAYITEM_RAW,
                          ):      # list of opcodes never executed by pyjitpl
                 continue
@@ -428,6 +432,10 @@
         if arity == 3:
             func = get_execute_funclist(3, False)[opnum]
             return func(cpu, metainterp, argboxes[0], argboxes[1], argboxes[2])
+        if arity == 5:    # copystrcontent, copyunicodecontent
+            func = get_execute_funclist(5, False)[opnum]
+            return func(cpu, metainterp, argboxes[0], argboxes[1],
+                        argboxes[2], argboxes[3], argboxes[4])
     raise NotImplementedError
 
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/graphpage.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/graphpage.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/graphpage.py	Sat Oct 30 16:41:17 2010
@@ -153,7 +153,7 @@
         opindex = opstartindex
         while True:
             op = operations[opindex]
-            lines.append(repr(op))
+            lines.append(op.repr(graytext=True))
             if is_interesting_guard(op):
                 tgt = op.getdescr()._debug_suboperations[0]
                 tgt_g, tgt_i = self.all_operations[tgt]

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/history.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/history.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/history.py	Sat Oct 30 16:41:17 2010
@@ -698,6 +698,21 @@
     return result
 _const_ptr_for_string = {}
 
+def get_const_ptr_for_unicode(s):
+    from pypy.rpython.annlowlevel import llunicode
+    if not we_are_translated():
+        try:
+            return _const_ptr_for_unicode[s]
+        except KeyError:
+            pass
+    if isinstance(s, str):
+        s = unicode(s)
+    result = ConstPtr(lltype.cast_opaque_ptr(llmemory.GCREF, llunicode(s)))
+    if not we_are_translated():
+        _const_ptr_for_unicode[s] = result
+    return result
+_const_ptr_for_unicode = {}
+
 # ____________________________________________________________
 
 # The TreeLoop class contains a loop or a generalized loop, i.e. a tree

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/jitdriver.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/jitdriver.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/jitdriver.py	Sat Oct 30 16:41:17 2010
@@ -13,8 +13,10 @@
     #    self.num_red_args      ... pypy.jit.metainterp.warmspot
     #    self.result_type       ... pypy.jit.metainterp.warmspot
     #    self.virtualizable_info... pypy.jit.metainterp.warmspot
+    #    self.greenfield_info   ... pypy.jit.metainterp.warmspot
     #    self.warmstate         ... pypy.jit.metainterp.warmspot
     #    self.handle_jitexc_from_bh pypy.jit.metainterp.warmspot
+    #    self.no_loop_header    ... pypy.jit.metainterp.warmspot
     #    self.portal_finishtoken... pypy.jit.metainterp.pyjitpl
     #    self.index             ... pypy.jit.codewriter.call
     #    self.mainjitcode       ... pypy.jit.codewriter.call

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py	Sat Oct 30 16:41:17 2010
@@ -14,6 +14,9 @@
 def _optimize_loop(metainterp_sd, old_loop_tokens, loop):
     cpu = metainterp_sd.cpu
     metainterp_sd.logger_noopt.log_loop(loop.inputargs, loop.operations)
+    # XXX the following lines are probably still needed, to discard invalid
+    # loops. bit silly to run a full perfect specialization and throw the
+    # result away.
     finder = PerfectSpecializationFinder(cpu)
     finder.find_nodes_loop(loop, False)
     if old_loop_tokens:
@@ -31,6 +34,7 @@
 def _optimize_bridge(metainterp_sd, old_loop_tokens, bridge):
     cpu = metainterp_sd.cpu    
     metainterp_sd.logger_noopt.log_loop(bridge.inputargs, bridge.operations)
+    # XXX same comment as above applies
     finder = BridgeSpecializationFinder(cpu)
     finder.find_nodes_bridge(bridge)
     if old_loop_tokens:

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	Sat Oct 30 16:41:17 2010
@@ -3,6 +3,7 @@
 from pypy.jit.metainterp.optimizeopt.intbounds import OptIntBounds
 from pypy.jit.metainterp.optimizeopt.virtualize import OptVirtualize
 from pypy.jit.metainterp.optimizeopt.heap import OptHeap
+from pypy.jit.metainterp.optimizeopt.fficall import OptFfiCall
 from pypy.jit.metainterp.optimizeopt.string import OptString
 from pypy.jit.metainterp.optimizeopt.unroll import OptUnroll
 
@@ -17,6 +18,7 @@
                      OptVirtualize(),
                      OptString(),
                      OptHeap(),
+                     OptFfiCall(),
                     ]
     if not_a_bridge:
         optimizations.insert(0, OptUnroll())

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py	Sat Oct 30 16:41:17 2010
@@ -205,6 +205,7 @@
         v1.intbound.make_ge(IntLowerBound(0))
 
     optimize_STRLEN = optimize_ARRAYLEN_GC
+    optimize_UNICODELEN = optimize_ARRAYLEN_GC
 
     def make_int_lt(self, box1, box2):
         v1 = self.getvalue(box1)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py	Sat Oct 30 16:41:17 2010
@@ -3,6 +3,7 @@
 from pypy.jit.metainterp.history import ConstInt
 from pypy.jit.metainterp.optimizeutil import _findall
 from pypy.jit.metainterp.resoperation import rop, ResOperation
+from pypy.jit.codewriter.effectinfo import EffectInfo
 
 class OptRewrite(Optimization):
     """Rewrite operations into equivalent, cheaper operations.
@@ -244,11 +245,11 @@
         self.optimizer.exception_might_have_happened = False
 
     def optimize_CALL_LOOPINVARIANT(self, op):
-        funcvalue = self.getvalue(op.getarg(0))
-        if not funcvalue.is_constant():
-            self.emit_operation(op)
-            return
-        key = make_hashable_int(op.getarg(0).getint())
+        arg = op.getarg(0)
+        # 'arg' must be a Const, because residual_call in codewriter
+        # expects a compile-time constant
+        assert isinstance(arg, Const)
+        key = make_hashable_int(arg.getint())
         resvalue = self.optimizer.loop_invariant_results.get(key, None)
         if resvalue is not None:
             self.make_equal_to(op.result, resvalue)
@@ -324,8 +325,37 @@
 ##            return
 ##        self.emit_operation(op)
 
-optimize_ops = _findall(OptRewrite, 'optimize_')
-        
+    def optimize_CALL(self, op):
+        # dispatch based on 'oopspecindex' to a method that handles
+        # specifically the given oopspec call.  For non-oopspec calls,
+        # oopspecindex is just zero.
+        effectinfo = op.getdescr().get_extra_info()
+        if effectinfo is not None:
+            oopspecindex = effectinfo.oopspecindex
+            if oopspecindex == EffectInfo.OS_ARRAYCOPY:
+                if self._optimize_CALL_ARRAYCOPY(op):
+                    return
+        self.emit_operation(op)
 
+    def _optimize_CALL_ARRAYCOPY(self, op):
+        source_value = self.getvalue(op.getarg(1))
+        dest_value = self.getvalue(op.getarg(2))
+        source_start_box = self.get_constant_box(op.getarg(3))
+        dest_start_box = self.get_constant_box(op.getarg(4))
+        length = self.get_constant_box(op.getarg(5))
+        if (source_value.is_virtual() and source_start_box and dest_start_box
+            and length and dest_value.is_virtual()):
+            # XXX optimize the case where dest value is not virtual,
+            #     but we still can avoid a mess
+            source_start = source_start_box.getint()
+            dest_start = dest_start_box.getint()
+            for index in range(length.getint()):
+                val = source_value.getitem(index + source_start)
+                dest_value.setitem(index + dest_start, val)
+            return True
+        if length and length.getint() == 0:
+            return True # 0-length arraycopy
+        return False
+
+optimize_ops = _findall(OptRewrite, 'optimize_')
 
-        

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py	Sat Oct 30 16:41:17 2010
@@ -3,6 +3,7 @@
 from pypy.jit.metainterp.history import Box, BoxInt, BoxPtr
 from pypy.jit.metainterp.history import Const, ConstInt, ConstPtr
 from pypy.jit.metainterp.history import get_const_ptr_for_string
+from pypy.jit.metainterp.history import get_const_ptr_for_unicode
 from pypy.jit.metainterp.resoperation import rop, ResOperation
 from pypy.jit.metainterp.optimizeopt import optimizer, virtualize
 from pypy.jit.metainterp.optimizeopt.optimizer import CONST_0, CONST_1
@@ -11,56 +12,106 @@
 from pypy.jit.codewriter.effectinfo import EffectInfo, callinfo_for_oopspec
 from pypy.jit.codewriter import heaptracker
 from pypy.rlib.unroll import unrolling_iterable
+from pypy.rlib.objectmodel import specialize, we_are_translated
+
+
+class StrOrUnicode(object):
+    def __init__(self, LLTYPE, hlstr, emptystr, chr,
+                 NEWSTR, STRLEN, STRGETITEM, STRSETITEM, COPYSTRCONTENT,
+                 OS_offset):
+        self.LLTYPE = LLTYPE
+        self.hlstr = hlstr
+        self.emptystr = emptystr
+        self.chr = chr
+        self.NEWSTR = NEWSTR
+        self.STRLEN = STRLEN
+        self.STRGETITEM = STRGETITEM
+        self.STRSETITEM = STRSETITEM
+        self.COPYSTRCONTENT = COPYSTRCONTENT
+        self.OS_offset = OS_offset
+
+    def _freeze_(self):
+        return True
+
+mode_string = StrOrUnicode(rstr.STR, annlowlevel.hlstr, '', chr,
+                           rop.NEWSTR, rop.STRLEN, rop.STRGETITEM,
+                           rop.STRSETITEM, rop.COPYSTRCONTENT, 0)
+mode_unicode = StrOrUnicode(rstr.UNICODE, annlowlevel.hlunicode, u'', unichr,
+                            rop.NEWUNICODE, rop.UNICODELEN, rop.UNICODEGETITEM,
+                            rop.UNICODESETITEM, rop.COPYUNICODECONTENT,
+                            EffectInfo._OS_offset_uni)
+
+# ____________________________________________________________
 
 
 class __extend__(optimizer.OptValue):
     """New methods added to the base class OptValue for this file."""
 
-    def getstrlen(self, newoperations):
-        s = self.get_constant_string()
-        if s is not None:
-            return ConstInt(len(s))
+    def getstrlen(self, newoperations, mode):
+        if mode is mode_string:
+            s = self.get_constant_string_spec(mode_string)
+            if s is not None:
+                return ConstInt(len(s))
         else:
-            if newoperations is None:
-                return None
-            self.ensure_nonnull()
-            box = self.force_box()
-            lengthbox = BoxInt()
-            newoperations.append(ResOperation(rop.STRLEN, [box], lengthbox))
-            return lengthbox
+            s = self.get_constant_string_spec(mode_unicode)
+            if s is not None:
+                return ConstInt(len(s))
+        if newoperations is None:
+            return None
+        self.ensure_nonnull()
+        box = self.force_box()
+        lengthbox = BoxInt()
+        newoperations.append(ResOperation(mode.STRLEN, [box], lengthbox))
+        return lengthbox
 
-    def get_constant_string(self):
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
         if self.is_constant():
-            s = self.box.getref(lltype.Ptr(rstr.STR))
-            return annlowlevel.hlstr(s)
+            s = self.box.getref(lltype.Ptr(mode.LLTYPE))
+            return mode.hlstr(s)
         else:
             return None
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
         # Copies the pointer-to-string 'self' into the target string
         # given by 'targetbox', at the specified offset.  Returns the offset
         # at the end of the copy.
-        lengthbox = self.getstrlen(newoperations)
+        lengthbox = self.getstrlen(newoperations, mode)
         srcbox = self.force_box()
         return copy_str_content(newoperations, srcbox, targetbox,
-                                CONST_0, offsetbox, lengthbox)
+                                CONST_0, offsetbox, lengthbox, mode)
 
 
 class VAbstractStringValue(virtualize.AbstractVirtualValue):
-    _attrs_ = ()
+    _attrs_ = ('mode',)
+
+    def __init__(self, optimizer, keybox, source_op, mode):
+        virtualize.AbstractVirtualValue.__init__(self, optimizer, keybox,
+                                                 source_op)
+        self.mode = mode
 
     def _really_force(self):
-        s = self.get_constant_string()
-        if s is not None:
-            c_s = get_const_ptr_for_string(s)
-            self.make_constant(c_s)
-            return
+        if self.mode is mode_string:
+            s = self.get_constant_string_spec(mode_string)
+            if s is not None:
+                c_s = get_const_ptr_for_string(s)
+                self.make_constant(c_s)
+                return
+        else:
+            s = self.get_constant_string_spec(mode_unicode)
+            if s is not None:
+                c_s = get_const_ptr_for_unicode(s)
+                self.make_constant(c_s)
+                return
         assert self.source_op is not None
         self.box = box = self.source_op.result
         newoperations = self.optimizer.newoperations
-        lengthbox = self.getstrlen(newoperations)
-        newoperations.append(ResOperation(rop.NEWSTR, [lengthbox], box))
-        self.string_copy_parts(newoperations, box, CONST_0)
+        lengthbox = self.getstrlen(newoperations, self.mode)
+        op = ResOperation(self.mode.NEWSTR, [lengthbox], box)
+        if not we_are_translated():
+            op.name = 'FORCE'
+        newoperations.append(op)
+        self.string_copy_parts(newoperations, box, CONST_0, self.mode)
 
 
 class VStringPlainValue(VAbstractStringValue):
@@ -74,7 +125,7 @@
         assert 0 <= start <= stop <= len(longerlist)
         self._chars = longerlist[start:stop]
 
-    def getstrlen(self, _):
+    def getstrlen(self, _, mode):
         if self._lengthbox is None:
             self._lengthbox = ConstInt(len(self._chars))
         return self._lengthbox
@@ -86,18 +137,21 @@
         assert isinstance(charvalue, optimizer.OptValue)
         self._chars[index] = charvalue
 
-    def get_constant_string(self):
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
         for c in self._chars:
             if c is optimizer.CVAL_UNINITIALIZED_ZERO or not c.is_constant():
                 return None
-        return ''.join([chr(c.box.getint()) for c in self._chars])
+        return mode.emptystr.join([mode.chr(c.box.getint())
+                                   for c in self._chars])
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
         for i in range(len(self._chars)):
             charbox = self._chars[i].force_box()
-            newoperations.append(ResOperation(rop.STRSETITEM, [targetbox,
-                                                               offsetbox,
-                                                               charbox], None))
+            newoperations.append(ResOperation(mode.STRSETITEM, [targetbox,
+                                                                offsetbox,
+                                                                charbox],
+                                              None))
             offsetbox = _int_add(newoperations, offsetbox, CONST_1)
         return offsetbox
 
@@ -120,7 +174,7 @@
             boxes.append(self.box)
 
     def _make_virtual(self, modifier):
-        return modifier.make_vstrplain()
+        return modifier.make_vstrplain(self.mode is mode_unicode)
 
 
 class VStringConcatValue(VAbstractStringValue):
@@ -131,23 +185,24 @@
         self.right = right
         self.lengthbox = lengthbox
 
-    def getstrlen(self, _):
+    def getstrlen(self, _, mode):
         return self.lengthbox
 
-    def get_constant_string(self):
-        s1 = self.left.get_constant_string()
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
+        s1 = self.left.get_constant_string_spec(mode)
         if s1 is None:
             return None
-        s2 = self.right.get_constant_string()
+        s2 = self.right.get_constant_string_spec(mode)
         if s2 is None:
             return None
         return s1 + s2
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
         offsetbox = self.left.string_copy_parts(newoperations, targetbox,
-                                                offsetbox)
+                                                offsetbox, mode)
         offsetbox = self.right.string_copy_parts(newoperations, targetbox,
-                                                 offsetbox)
+                                                 offsetbox, mode)
         return offsetbox
 
     def get_args_for_fail(self, modifier):
@@ -172,7 +227,7 @@
             boxes.append(self.box)
 
     def _make_virtual(self, modifier):
-        return modifier.make_vstrconcat()
+        return modifier.make_vstrconcat(self.mode is mode_unicode)
 
 
 class VStringSliceValue(VAbstractStringValue):
@@ -184,12 +239,13 @@
         self.vstart = vstart
         self.vlength = vlength
 
-    def getstrlen(self, _):
+    def getstrlen(self, _, mode):
         return self.vlength.force_box()
 
-    def get_constant_string(self):
+    @specialize.arg(1)
+    def get_constant_string_spec(self, mode):
         if self.vstart.is_constant() and self.vlength.is_constant():
-            s1 = self.vstr.get_constant_string()
+            s1 = self.vstr.get_constant_string_spec(mode)
             if s1 is None:
                 return None
             start = self.vstart.box.getint()
@@ -199,12 +255,12 @@
             return s1[start : start + length]
         return None
 
-    def string_copy_parts(self, newoperations, targetbox, offsetbox):
-        lengthbox = self.getstrlen(newoperations)
+    def string_copy_parts(self, newoperations, targetbox, offsetbox, mode):
+        lengthbox = self.getstrlen(newoperations, mode)
         return copy_str_content(newoperations,
                                 self.vstr.force_box(), targetbox,
                                 self.vstart.force_box(), offsetbox,
-                                lengthbox)
+                                lengthbox, mode)
 
     def get_args_for_fail(self, modifier):
         if self.box is None and not modifier.already_seen_virtual(self.keybox):
@@ -229,11 +285,11 @@
             boxes.append(self.box)
 
     def _make_virtual(self, modifier):
-        return modifier.make_vstrslice()
+        return modifier.make_vstrslice(self.mode is mode_unicode)
 
 
 def copy_str_content(newoperations, srcbox, targetbox,
-                     srcoffsetbox, offsetbox, lengthbox):
+                     srcoffsetbox, offsetbox, lengthbox, mode):
     if isinstance(srcbox, ConstPtr) and isinstance(srcoffsetbox, Const):
         M = 5
     else:
@@ -242,17 +298,18 @@
         # up to M characters are done "inline", i.e. with STRGETITEM/STRSETITEM
         # instead of just a COPYSTRCONTENT.
         for i in range(lengthbox.value):
-            charbox = _strgetitem(newoperations, srcbox, srcoffsetbox)
+            charbox = _strgetitem(newoperations, srcbox, srcoffsetbox, mode)
             srcoffsetbox = _int_add(newoperations, srcoffsetbox, CONST_1)
-            newoperations.append(ResOperation(rop.STRSETITEM, [targetbox,
-                                                               offsetbox,
-                                                               charbox], None))
+            newoperations.append(ResOperation(mode.STRSETITEM, [targetbox,
+                                                                offsetbox,
+                                                                charbox],
+                                              None))
             offsetbox = _int_add(newoperations, offsetbox, CONST_1)
     else:
         nextoffsetbox = _int_add(newoperations, offsetbox, lengthbox)
-        op = ResOperation(rop.COPYSTRCONTENT, [srcbox, targetbox,
-                                               srcoffsetbox, offsetbox,
-                                               lengthbox], None)
+        op = ResOperation(mode.COPYSTRCONTENT, [srcbox, targetbox,
+                                                srcoffsetbox, offsetbox,
+                                                lengthbox], None)
         newoperations.append(op)
         offsetbox = nextoffsetbox
     return offsetbox
@@ -279,12 +336,16 @@
     newoperations.append(ResOperation(rop.INT_SUB, [box1, box2], resbox))
     return resbox
 
-def _strgetitem(newoperations, strbox, indexbox):
+def _strgetitem(newoperations, strbox, indexbox, mode):
     if isinstance(strbox, ConstPtr) and isinstance(indexbox, ConstInt):
-        s = strbox.getref(lltype.Ptr(rstr.STR))
-        return ConstInt(ord(s.chars[indexbox.getint()]))
+        if mode is mode_string:
+            s = strbox.getref(lltype.Ptr(rstr.STR))
+            return ConstInt(ord(s.chars[indexbox.getint()]))
+        else:
+            s = strbox.getref(lltype.Ptr(rstr.UNICODE))
+            return ConstInt(ord(s.chars[indexbox.getint()]))
     resbox = BoxInt()
-    newoperations.append(ResOperation(rop.STRGETITEM, [strbox, indexbox],
+    newoperations.append(ResOperation(mode.STRGETITEM, [strbox, indexbox],
                                       resbox))
     return resbox
 
@@ -292,62 +353,34 @@
 class OptString(optimizer.Optimization):
     "Handling of strings and unicodes."
 
-    def make_vstring_plain(self, box, source_op=None):
-        vvalue = VStringPlainValue(self.optimizer, box, source_op)
+    def make_vstring_plain(self, box, source_op, mode):
+        vvalue = VStringPlainValue(self.optimizer, box, source_op, mode)
         self.make_equal_to(box, vvalue)
         return vvalue
 
-    def make_vstring_concat(self, box, source_op=None):
-        vvalue = VStringConcatValue(self.optimizer, box, source_op)
+    def make_vstring_concat(self, box, source_op, mode):
+        vvalue = VStringConcatValue(self.optimizer, box, source_op, mode)
         self.make_equal_to(box, vvalue)
         return vvalue
 
-    def make_vstring_slice(self, box, source_op=None):
-        vvalue = VStringSliceValue(self.optimizer, box, source_op)
+    def make_vstring_slice(self, box, source_op, mode):
+        vvalue = VStringSliceValue(self.optimizer, box, source_op, mode)
         self.make_equal_to(box, vvalue)
         return vvalue
 
-    def optimize_CALL(self, op):
-        # dispatch based on 'oopspecindex' to a method that handles
-        # specifically the given oopspec call.  For non-oopspec calls,
-        # oopspecindex is just zero.
-        effectinfo = op.getdescr().get_extra_info()
-        if effectinfo is not None:
-            oopspecindex = effectinfo.oopspecindex
-            for value, meth in opt_call_oopspec_ops:
-                if oopspecindex == value:
-                    if meth(self, op):
-                        return
-        self.emit_operation(op)
-
-    def opt_call_oopspec_ARRAYCOPY(self, op):
-        source_value = self.getvalue(op.getarg(1))
-        dest_value = self.getvalue(op.getarg(2))
-        source_start_box = self.get_constant_box(op.getarg(3))
-        dest_start_box = self.get_constant_box(op.getarg(4))
-        length = self.get_constant_box(op.getarg(5))
-        if (source_value.is_virtual() and source_start_box and dest_start_box
-            and length and dest_value.is_virtual()):
-            # XXX optimize the case where dest value is not virtual,
-            #     but we still can avoid a mess
-            source_start = source_start_box.getint()
-            dest_start = dest_start_box.getint()
-            for index in range(length.getint()):
-                val = source_value.getitem(index + source_start)
-                dest_value.setitem(index + dest_start, val)
-            return True
-        if length and length.getint() == 0:
-            return True # 0-length arraycopy
-        return False
-
     def optimize_NEWSTR(self, op):
+        self._optimize_NEWSTR(op, mode_string)
+    def optimize_NEWUNICODE(self, op):
+        self._optimize_NEWSTR(op, mode_unicode)
+
+    def _optimize_NEWSTR(self, op, mode):
         length_box = self.get_constant_box(op.getarg(0))
         if length_box:
             # if the original 'op' did not have a ConstInt as argument,
             # build a new one with the ConstInt argument
             if not isinstance(op.getarg(0), ConstInt):
-                op = ResOperation(rop.NEWSTR, [length_box], op.result)
-            vvalue = self.make_vstring_plain(op.result, op)
+                op = ResOperation(mode.NEWSTR, [length_box], op.result)
+            vvalue = self.make_vstring_plain(op.result, op, mode)
             vvalue.setup(length_box.getint())
         else:
             self.getvalue(op.result).ensure_nonnull()
@@ -363,13 +396,20 @@
         value.ensure_nonnull()
         self.emit_operation(op)
 
+    optimize_UNICODESETITEM = optimize_STRSETITEM
+
     def optimize_STRGETITEM(self, op):
+        self._optimize_STRGETITEM(op, mode_string)
+    def optimize_UNICODEGETITEM(self, op):
+        self._optimize_STRGETITEM(op, mode_unicode)
+
+    def _optimize_STRGETITEM(self, op, mode):
         value = self.getvalue(op.getarg(0))
         vindex = self.getvalue(op.getarg(1))
-        vresult = self.strgetitem(value, vindex)
+        vresult = self.strgetitem(value, vindex, mode)
         self.make_equal_to(op.result, vresult)
 
-    def strgetitem(self, value, vindex):
+    def strgetitem(self, value, vindex, mode):
         value.ensure_nonnull()
         #
         if value.is_virtual() and isinstance(value, VStringSliceValue):
@@ -384,28 +424,71 @@
                 return value.getitem(vindex.box.getint())
         #
         resbox = _strgetitem(self.optimizer.newoperations,
-                             value.force_box(),vindex.force_box())
+                             value.force_box(),vindex.force_box(), mode)
         return self.getvalue(resbox)
 
     def optimize_STRLEN(self, op):
+        self._optimize_STRLEN(op, mode_string)
+    def optimize_UNICODELEN(self, op):
+        self._optimize_STRLEN(op, mode_unicode)
+
+    def _optimize_STRLEN(self, op, mode):
         value = self.getvalue(op.getarg(0))
-        lengthbox = value.getstrlen(self.optimizer.newoperations)
+        lengthbox = value.getstrlen(self.optimizer.newoperations, mode)
         self.make_equal_to(op.result, self.getvalue(lengthbox))
 
-    def opt_call_oopspec_STR_CONCAT(self, op):
+    def optimize_CALL(self, op):
+        # dispatch based on 'oopspecindex' to a method that handles
+        # specifically the given oopspec call.  For non-oopspec calls,
+        # oopspecindex is just zero.
+        effectinfo = op.getdescr().get_extra_info()
+        if effectinfo is not None:
+            oopspecindex = effectinfo.oopspecindex
+            for value, meth in opt_call_oopspec_ops:
+                if oopspecindex == value:      # a match with the OS_STR_xxx
+                    if meth(self, op, mode_string):
+                        return
+                    break
+                if oopspecindex == value + EffectInfo._OS_offset_uni:
+                    # a match with the OS_UNI_xxx
+                    if meth(self, op, mode_unicode):
+                        return
+                    break
+            if oopspecindex == EffectInfo.OS_STR2UNICODE:
+                if self.opt_call_str_STR2UNICODE(op):
+                    return
+        self.emit_operation(op)
+
+    def opt_call_str_STR2UNICODE(self, op):
+        # Constant-fold unicode("constant string").
+        # More generally, supporting non-constant but virtual cases is
+        # not obvious, because of the exception UnicodeDecodeError that
+        # can be raised by ll_str2unicode()
+        varg = self.getvalue(op.getarg(1))
+        s = varg.get_constant_string_spec(mode_string)
+        if s is None:
+            return False
+        try:
+            u = unicode(s)
+        except UnicodeDecodeError:
+            return False
+        self.make_constant(op.result, get_const_ptr_for_unicode(u))
+        return True
+
+    def opt_call_stroruni_STR_CONCAT(self, op, mode):
         vleft = self.getvalue(op.getarg(1))
         vright = self.getvalue(op.getarg(2))
         vleft.ensure_nonnull()
         vright.ensure_nonnull()
         newoperations = self.optimizer.newoperations
-        len1box = vleft.getstrlen(newoperations)
-        len2box = vright.getstrlen(newoperations)
+        len1box = vleft.getstrlen(newoperations, mode)
+        len2box = vright.getstrlen(newoperations, mode)
         lengthbox = _int_add(newoperations, len1box, len2box)
-        value = self.make_vstring_concat(op.result, op)
+        value = self.make_vstring_concat(op.result, op, mode)
         value.setup(vleft, vright, lengthbox)
         return True
 
-    def opt_call_oopspec_STR_SLICE(self, op):
+    def opt_call_stroruni_STR_SLICE(self, op, mode):
         newoperations = self.optimizer.newoperations
         vstr = self.getvalue(op.getarg(1))
         vstart = self.getvalue(op.getarg(2))
@@ -414,7 +497,7 @@
         if (isinstance(vstr, VStringPlainValue) and vstart.is_constant()
             and vstop.is_constant()):
             # slicing with constant bounds of a VStringPlainValue
-            value = self.make_vstring_plain(op.result, op)
+            value = self.make_vstring_plain(op.result, op, mode)
             value.setup_slice(vstr._chars, vstart.box.getint(),
                                            vstop.box.getint())
             return True
@@ -432,16 +515,16 @@
                                 vstart.force_box())
             vstart = self.getvalue(startbox)
         #
-        value = self.make_vstring_slice(op.result, op)
+        value = self.make_vstring_slice(op.result, op, mode)
         value.setup(vstr, vstart, self.getvalue(lengthbox))
         return True
 
-    def opt_call_oopspec_STR_EQUAL(self, op):
+    def opt_call_stroruni_STR_EQUAL(self, op, mode):
         v1 = self.getvalue(op.getarg(1))
         v2 = self.getvalue(op.getarg(2))
         #
-        l1box = v1.getstrlen(None)
-        l2box = v2.getstrlen(None)
+        l1box = v1.getstrlen(None, mode)
+        l2box = v2.getstrlen(None, mode)
         if (l1box is not None and l2box is not None and
             isinstance(l1box, ConstInt) and
             isinstance(l2box, ConstInt) and
@@ -450,13 +533,13 @@
             self.make_constant(op.result, CONST_0)
             return True
         #
-        if self.handle_str_equal_level1(v1, v2, op.result):
+        if self.handle_str_equal_level1(v1, v2, op.result, mode):
             return True
-        if self.handle_str_equal_level1(v2, v1, op.result):
+        if self.handle_str_equal_level1(v2, v1, op.result, mode):
             return True
-        if self.handle_str_equal_level2(v1, v2, op.result):
+        if self.handle_str_equal_level2(v1, v2, op.result, mode):
             return True
-        if self.handle_str_equal_level2(v2, v1, op.result):
+        if self.handle_str_equal_level2(v2, v1, op.result, mode):
             return True
         #
         if v1.is_nonnull() and v2.is_nonnull():
@@ -468,37 +551,37 @@
             else:
                 do = EffectInfo.OS_STREQ_NONNULL
             self.generate_modified_call(do, [v1.force_box(),
-                                             v2.force_box()], op.result)
+                                             v2.force_box()], op.result, mode)
             return True
         return False
 
-    def handle_str_equal_level1(self, v1, v2, resultbox):
-        l2box = v2.getstrlen(None)
+    def handle_str_equal_level1(self, v1, v2, resultbox, mode):
+        l2box = v2.getstrlen(None, mode)
         if isinstance(l2box, ConstInt):
             if l2box.value == 0:
-                lengthbox = v1.getstrlen(self.optimizer.newoperations)
+                lengthbox = v1.getstrlen(self.optimizer.newoperations, mode)
                 seo = self.optimizer.send_extra_operation
                 seo(ResOperation(rop.INT_EQ, [lengthbox, CONST_0], resultbox))
                 return True
             if l2box.value == 1:
-                l1box = v1.getstrlen(None)
+                l1box = v1.getstrlen(None, mode)
                 if isinstance(l1box, ConstInt) and l1box.value == 1:
                     # comparing two single chars
-                    vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO)
-                    vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO)
+                    vchar1 = self.strgetitem(v1, optimizer.CVAL_ZERO, mode)
+                    vchar2 = self.strgetitem(v2, optimizer.CVAL_ZERO, mode)
                     seo = self.optimizer.send_extra_operation
                     seo(ResOperation(rop.INT_EQ, [vchar1.force_box(),
                                                   vchar2.force_box()],
                                      resultbox))
                     return True
                 if isinstance(v1, VStringSliceValue):
-                    vchar = self.strgetitem(v2, optimizer.CVAL_ZERO)
+                    vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode)
                     do = EffectInfo.OS_STREQ_SLICE_CHAR
                     self.generate_modified_call(do, [v1.vstr.force_box(),
                                                      v1.vstart.force_box(),
                                                      v1.vlength.force_box(),
                                                      vchar.force_box()],
-                                                resultbox)
+                                                resultbox, mode)
                     return True
         #
         if v2.is_null():
@@ -516,17 +599,18 @@
         #
         return False
 
-    def handle_str_equal_level2(self, v1, v2, resultbox):
-        l2box = v2.getstrlen(None)
+    def handle_str_equal_level2(self, v1, v2, resultbox, mode):
+        l2box = v2.getstrlen(None, mode)
         if isinstance(l2box, ConstInt):
             if l2box.value == 1:
-                vchar = self.strgetitem(v2, optimizer.CVAL_ZERO)
+                vchar = self.strgetitem(v2, optimizer.CVAL_ZERO, mode)
                 if v1.is_nonnull():
                     do = EffectInfo.OS_STREQ_NONNULL_CHAR
                 else:
                     do = EffectInfo.OS_STREQ_CHECKNULL_CHAR
                 self.generate_modified_call(do, [v1.force_box(),
-                                                 vchar.force_box()], resultbox)
+                                                 vchar.force_box()], resultbox,
+                                            mode)
                 return True
         #
         if v1.is_virtual() and isinstance(v1, VStringSliceValue):
@@ -537,11 +621,12 @@
             self.generate_modified_call(do, [v1.vstr.force_box(),
                                              v1.vstart.force_box(),
                                              v1.vlength.force_box(),
-                                             v2.force_box()], resultbox)
+                                             v2.force_box()], resultbox, mode)
             return True
         return False
 
-    def generate_modified_call(self, oopspecindex, args, result):
+    def generate_modified_call(self, oopspecindex, args, result, mode):
+        oopspecindex += mode.OS_offset
         calldescr, func = callinfo_for_oopspec(oopspecindex)
         op = ResOperation(rop.CALL, [ConstInt(func)] + args, result,
                           descr=calldescr)
@@ -559,7 +644,7 @@
 optimize_ops = _findall(OptString, 'optimize_')
 
 def _findall_call_oopspec():
-    prefix = 'opt_call_oopspec_'
+    prefix = 'opt_call_stroruni_'
     result = []
     for name in dir(OptString):
         if name.startswith(prefix):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Sat Oct 30 16:41:17 2010
@@ -76,6 +76,8 @@
         assert self.source_op is not None
         # ^^^ This case should not occur any more (see test_bug_3).
         #
+        if not we_are_translated():
+            self.source_op.name = 'FORCE ' + self.source_op.name
         newoperations = self.optimizer.newoperations
         newoperations.append(self.source_op)
         self.box = box = self.source_op.result
@@ -148,6 +150,11 @@
         fielddescrs = self._get_field_descr_list()
         return modifier.make_virtual(self.known_class, fielddescrs)
 
+    def __repr__(self):
+        cls_name = self.known_class.value.adr.ptr._obj._TYPE._name
+        field_names = [field.name for field in self._fields]
+        return "" % (cls_name, field_names)
+
 class VStructValue(AbstractVirtualStructValue):
 
     def __init__(self, optimizer, structdescr, keybox, source_op=None):
@@ -179,6 +186,8 @@
 
     def _really_force(self):
         assert self.source_op is not None
+        if not we_are_translated():
+            self.source_op.name = 'FORCE ' + self.source_op.name
         newoperations = self.optimizer.newoperations
         newoperations.append(self.source_op)
         self.box = box = self.source_op.result

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py	Sat Oct 30 16:41:17 2010
@@ -1,4 +1,4 @@
-import py, os
+import py, os, sys
 from pypy.rpython.lltypesystem import lltype, llmemory, rclass
 from pypy.rlib.objectmodel import we_are_translated
 from pypy.rlib.unroll import unrolling_iterable
@@ -498,6 +498,22 @@
     opimpl_getfield_gc_r_pure = _opimpl_getfield_gc_pure_any
     opimpl_getfield_gc_f_pure = _opimpl_getfield_gc_pure_any
 
+    @arguments("orgpc", "box", "descr")
+    def _opimpl_getfield_gc_greenfield_any(self, pc, box, fielddescr):
+        ginfo = self.metainterp.jitdriver_sd.greenfield_info
+        if (ginfo is not None and fielddescr in ginfo.green_field_descrs
+            and not self._nonstandard_virtualizable(pc, box)):
+            # fetch the result, but consider it as a Const box and don't
+            # record any operation
+            resbox = executor.execute(self.metainterp.cpu, self.metainterp,
+                                      rop.GETFIELD_GC_PURE, fielddescr, box)
+            return resbox.constbox()
+        # fall-back
+        return self.execute_with_descr(rop.GETFIELD_GC_PURE, fielddescr, box)
+    opimpl_getfield_gc_i_greenfield = _opimpl_getfield_gc_greenfield_any
+    opimpl_getfield_gc_r_greenfield = _opimpl_getfield_gc_greenfield_any
+    opimpl_getfield_gc_f_greenfield = _opimpl_getfield_gc_greenfield_any
+
     @arguments("box", "descr", "box")
     def _opimpl_setfield_gc_any(self, box, fielddescr, valuebox):
         self.execute_with_descr(rop.SETFIELD_GC, fielddescr, box, valuebox)
@@ -529,7 +545,8 @@
     def _nonstandard_virtualizable(self, pc, box):
         # returns True if 'box' is actually not the "standard" virtualizable
         # that is stored in metainterp.virtualizable_boxes[-1]
-        if self.metainterp.jitdriver_sd.virtualizable_info is None:
+        if (self.metainterp.jitdriver_sd.virtualizable_info is None and
+            self.metainterp.jitdriver_sd.greenfield_info is None):
             return True      # can occur in case of multiple JITs
         standard_box = self.metainterp.virtualizable_boxes[-1]
         if standard_box is box:
@@ -799,12 +816,16 @@
 
     @arguments("orgpc", "int", "boxes3", "boxes3")
     def opimpl_jit_merge_point(self, orgpc, jdindex, greenboxes, redboxes):
+        any_operation = len(self.metainterp.history.operations) > 0
         jitdriver_sd = self.metainterp.staticdata.jitdrivers_sd[jdindex]
         self.verify_green_args(jitdriver_sd, greenboxes)
         # xxx we may disable the following line in some context later
         self.debug_merge_point(jitdriver_sd, greenboxes)
         if self.metainterp.seen_loop_header_for_jdindex < 0:
-            return
+            if not jitdriver_sd.no_loop_header or not any_operation:
+                return
+            # automatically add a loop_header if there is none
+            self.metainterp.seen_loop_header_for_jdindex = jdindex
         #
         assert self.metainterp.seen_loop_header_for_jdindex == jdindex, (
             "found a loop_header for a JitDriver that does not match "
@@ -893,6 +914,40 @@
         msg = box.getref(lltype.Ptr(rstr.STR))
         lloperation.llop.debug_fatalerror(msg)
 
+    @arguments("box", "box", "box", "box", "box")
+    def opimpl_jit_debug(self, stringbox, arg1box, arg2box, arg3box, arg4box):
+        from pypy.rpython.lltypesystem import rstr
+        from pypy.rpython.annlowlevel import hlstr
+        msg = stringbox.getref(lltype.Ptr(rstr.STR))
+        debug_print('jit_debug:', hlstr(msg),
+                    arg1box.getint(), arg2box.getint(),
+                    arg3box.getint(), arg4box.getint())
+        args = [stringbox, arg1box, arg2box, arg3box, arg4box]
+        i = 4
+        while i > 0 and args[i].getint() == -sys.maxint-1:
+            i -= 1
+        assert i >= 0
+        op = self.metainterp.history.record(rop.JIT_DEBUG, args[:i+1], None)
+        self.metainterp.attach_debug_info(op)
+
+    @arguments("box")
+    def _opimpl_assert_green(self, box):
+        if not isinstance(box, Const):
+            msg = "assert_green failed at %s:%d" % (
+                self.jitcode.name,
+                self.pc)
+            if we_are_translated():
+                from pypy.rpython.annlowlevel import llstr
+                from pypy.rpython.lltypesystem import lloperation
+                lloperation.llop.debug_fatalerror(lltype.Void, llstr(msg))
+            else:
+                from pypy.rlib.jit import AssertGreenFailed
+                raise AssertGreenFailed(msg)
+
+    opimpl_int_assert_green   = _opimpl_assert_green
+    opimpl_ref_assert_green   = _opimpl_assert_green
+    opimpl_float_assert_green = _opimpl_assert_green
+
     @arguments("box")
     def opimpl_virtual_ref(self, box):
         # Details on the content of metainterp.virtualref_boxes:
@@ -998,7 +1053,8 @@
         guard_op = metainterp.history.record(opnum, moreargs, None,
                                              descr=resumedescr)
         virtualizable_boxes = None
-        if metainterp.jitdriver_sd.virtualizable_info is not None:
+        if (metainterp.jitdriver_sd.virtualizable_info is not None or
+            metainterp.jitdriver_sd.greenfield_info is not None):
             virtualizable_boxes = metainterp.virtualizable_boxes
         saved_pc = self.pc
         if resumepc >= 0:
@@ -1646,6 +1702,7 @@
                                               duplicates)
             live_arg_boxes += self.virtualizable_boxes
             live_arg_boxes.pop()
+        #
         assert len(self.virtualref_boxes) == 0, "missing virtual_ref_finish()?"
         # Called whenever we reach the 'loop_header' hint.
         # First, attempt to make a bridge:
@@ -1832,6 +1889,7 @@
         f.setup_call(original_boxes)
         assert self.in_recursion == 0
         self.virtualref_boxes = []
+        self.initialize_withgreenfields(original_boxes)
         self.initialize_virtualizable(original_boxes)
 
     def initialize_state_from_guard_failure(self, resumedescr):
@@ -1856,6 +1914,14 @@
             self.virtualizable_boxes.append(virtualizable_box)
             self.initialize_virtualizable_enter()
 
+    def initialize_withgreenfields(self, original_boxes):
+        ginfo = self.jitdriver_sd.greenfield_info
+        if ginfo is not None:
+            assert self.jitdriver_sd.virtualizable_info is None
+            index = (self.jitdriver_sd.num_green_args +
+                     ginfo.red_index)
+            self.virtualizable_boxes = [original_boxes[index]]
+
     def initialize_virtualizable_enter(self):
         vinfo = self.jitdriver_sd.virtualizable_info
         virtualizable_box = self.virtualizable_boxes[-1]
@@ -1949,8 +2015,10 @@
 
     def rebuild_state_after_failure(self, resumedescr):
         vinfo = self.jitdriver_sd.virtualizable_info
+        ginfo = self.jitdriver_sd.greenfield_info
         self.framestack = []
-        boxlists = resume.rebuild_from_resumedata(self, resumedescr, vinfo)
+        boxlists = resume.rebuild_from_resumedata(self, resumedescr, vinfo,
+                                                  ginfo)
         inputargs_and_holes, virtualizable_boxes, virtualref_boxes = boxlists
         #
         # virtual refs: make the vrefs point to the freshly allocated virtuals
@@ -1975,6 +2043,12 @@
             assert not virtualizable.vable_token
             # fill the virtualizable with the local boxes
             self.synchronize_virtualizable()
+        #
+        elif self.jitdriver_sd.greenfield_info:
+            self.virtualizable_boxes = virtualizable_boxes
+        else:
+            assert not virtualizable_boxes
+        #
         return inputargs_and_holes
 
     def check_synchronized_virtualizable(self):
@@ -2048,7 +2122,8 @@
         for i in range(len(boxes)):
             if boxes[i] is oldbox:
                 boxes[i] = newbox
-        if self.jitdriver_sd.virtualizable_info is not None:
+        if (self.jitdriver_sd.virtualizable_info is not None or
+            self.jitdriver_sd.greenfield_info is not None):
             boxes = self.virtualizable_boxes
             for i in range(len(boxes)):
                 if boxes[i] is oldbox:

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resoperation.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resoperation.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resoperation.py	Sat Oct 30 16:41:17 2010
@@ -93,7 +93,7 @@
     def __repr__(self):
         return self.repr()
 
-    def repr(self):
+    def repr(self, graytext=False):
         # RPython-friendly version
         if self.result is not None:
             sres = '%s = ' % (self.result,)
@@ -101,6 +101,8 @@
             sres = ''
         if self.name:
             prefix = "%s:%s   " % (self.name, self.pc)
+            if graytext:
+                prefix = "\f%s\f" % prefix
         else:
             prefix = ""
         args = self.getarglist()
@@ -455,15 +457,16 @@
     'UNICODESETITEM/3',
     'NEWUNICODE/1',
     #'RUNTIMENEW/1',     # ootype operation    
-    'COND_CALL_GC_WB/1d',  # [objptr]   (for the write barrier)
+    'COND_CALL_GC_WB/2d', # [objptr, newvalue]   (for the write barrier)
     'DEBUG_MERGE_POINT/1',      # debugging only
+    'JIT_DEBUG/*',              # debugging only
     'VIRTUAL_REF_FINISH/2',   # removed before it's passed to the backend
     'COPYSTRCONTENT/5',       # src, dst, srcstart, dststart, length
     'COPYUNICODECONTENT/5',
 
     '_CANRAISE_FIRST', # ----- start of can_raise operations -----
     'CALL/*d',
-    'CALL_ASSEMBLER/*d',
+    'CALL_ASSEMBLER/*d',  # call already compiled assembler
     'CALL_MAY_FORCE/*d',
     'CALL_LOOPINVARIANT/*d',
     #'OOSEND',                     # ootype operation

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py	Sat Oct 30 16:41:17 2010
@@ -255,13 +255,19 @@
     def make_varray(self, arraydescr):
         return VArrayInfo(arraydescr)
 
-    def make_vstrplain(self):
+    def make_vstrplain(self, is_unicode=False):
+        if is_unicode:
+            return VUniPlainInfo()
         return VStrPlainInfo()
 
-    def make_vstrconcat(self):
+    def make_vstrconcat(self, is_unicode=False):
+        if is_unicode:
+            return VUniConcatInfo()
         return VStrConcatInfo()
 
-    def make_vstrslice(self):
+    def make_vstrslice(self, is_unicode=False):
+        if is_unicode:
+            return VUniSliceInfo()
         return VStrSliceInfo()
 
     def register_virtual_fields(self, virtualbox, fieldboxes):
@@ -550,6 +556,60 @@
         for i in self.fieldnums:
             debug_print("\t\t", str(untag(i)))
 
+
+class VUniPlainInfo(AbstractVirtualInfo):
+    """Stands for the unicode string made out of the characters of all
+    fieldnums."""
+
+    @specialize.argtype(1)
+    def allocate(self, decoder, index):
+        length = len(self.fieldnums)
+        string = decoder.allocate_unicode(length)
+        decoder.virtuals_cache[index] = string
+        for i in range(length):
+            decoder.unicode_setitem(string, i, self.fieldnums[i])
+        return string
+
+    def debug_prints(self):
+        debug_print("\tvuniplaininfo length", len(self.fieldnums))
+
+
+class VUniConcatInfo(AbstractVirtualInfo):
+    """Stands for the unicode string made out of the concatenation of two
+    other unicode strings."""
+
+    @specialize.argtype(1)
+    def allocate(self, decoder, index):
+        # xxx for blackhole resuming, this will build all intermediate
+        # strings and throw them away immediately, which is a bit sub-
+        # efficient.  Not sure we care.
+        left, right = self.fieldnums
+        string = decoder.concat_unicodes(left, right)
+        decoder.virtuals_cache[index] = string
+        return string
+
+    def debug_prints(self):
+        debug_print("\tvuniconcatinfo")
+        for i in self.fieldnums:
+            debug_print("\t\t", str(untag(i)))
+
+
+class VUniSliceInfo(AbstractVirtualInfo):
+    """Stands for the unicode string made out of slicing another
+    unicode string."""
+
+    @specialize.argtype(1)
+    def allocate(self, decoder, index):
+        largerstr, start, length = self.fieldnums
+        string = decoder.slice_unicode(largerstr, start, length)
+        decoder.virtuals_cache[index] = string
+        return string
+
+    def debug_prints(self):
+        debug_print("\tvunisliceinfo")
+        for i in self.fieldnums:
+            debug_print("\t\t", str(untag(i)))
+
 # ____________________________________________________________
 
 class AbstractResumeDataReader(object):
@@ -629,9 +689,11 @@
 
 # ---------- when resuming for pyjitpl.py, make boxes ----------
 
-def rebuild_from_resumedata(metainterp, storage, virtualizable_info):
+def rebuild_from_resumedata(metainterp, storage, virtualizable_info,
+                            greenfield_info):
     resumereader = ResumeDataBoxReader(storage, metainterp)
-    boxes = resumereader.consume_vref_and_vable_boxes(virtualizable_info)
+    boxes = resumereader.consume_vref_and_vable_boxes(virtualizable_info,
+                                                      greenfield_info)
     virtualizable_boxes, virtualref_boxes = boxes
     frameinfo = storage.rd_frame_info_list
     while True:
@@ -676,15 +738,18 @@
         assert (end & 1) == 0
         return [self.decode_ref(nums[i]) for i in range(end)]
 
-    def consume_vref_and_vable_boxes(self, vinfo):
+    def consume_vref_and_vable_boxes(self, vinfo, ginfo):
         nums = self.cur_numb.nums
         self.cur_numb = self.cur_numb.prev
-        if vinfo is None:
-            virtualizable_boxes = None
-            end = len(nums)
-        else:
+        if vinfo is not None:
             virtualizable_boxes = self.consume_virtualizable_boxes(vinfo, nums)
             end = len(nums) - len(virtualizable_boxes)
+        elif ginfo is not None:
+            virtualizable_boxes = [self.decode_ref(nums[-1])]
+            end = len(nums) - 1
+        else:
+            virtualizable_boxes = None
+            end = len(nums)
         virtualref_boxes = self.consume_virtualref_boxes(nums, end)
         return virtualizable_boxes, virtualref_boxes
 
@@ -725,6 +790,32 @@
         return self.metainterp.execute_and_record_varargs(
             rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr)
 
+    def allocate_unicode(self, length):
+        return self.metainterp.execute_and_record(rop.NEWUNICODE,
+                                                  None, ConstInt(length))
+
+    def unicode_setitem(self, strbox, index, charnum):
+        charbox = self.decode_box(charnum, INT)
+        self.metainterp.execute_and_record(rop.UNICODESETITEM, None,
+                                           strbox, ConstInt(index), charbox)
+
+    def concat_unicodes(self, str1num, str2num):
+        calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_CONCAT)
+        str1box = self.decode_box(str1num, REF)
+        str2box = self.decode_box(str2num, REF)
+        return self.metainterp.execute_and_record_varargs(
+            rop.CALL, [ConstInt(func), str1box, str2box], calldescr)
+
+    def slice_unicode(self, strnum, startnum, lengthnum):
+        calldescr, func = callinfo_for_oopspec(EffectInfo.OS_UNI_SLICE)
+        strbox = self.decode_box(strnum, REF)
+        startbox = self.decode_box(startnum, INT)
+        lengthbox = self.decode_box(lengthnum, INT)
+        stopbox = self.metainterp.execute_and_record(rop.INT_ADD, None,
+                                                     startbox, lengthbox)
+        return self.metainterp.execute_and_record_varargs(
+            rop.CALL, [ConstInt(func), strbox, startbox, stopbox], calldescr)
+
     def setfield(self, descr, structbox, fieldnum):
         if descr.is_pointer_field():
             kind = REF
@@ -815,8 +906,9 @@
     resumereader = ResumeDataDirectReader(blackholeinterpbuilder.cpu, storage,
                                           all_virtuals)
     vinfo = jitdriver_sd.virtualizable_info
+    ginfo = jitdriver_sd.greenfield_info
     vrefinfo = blackholeinterpbuilder.metainterp_sd.virtualref_info
-    resumereader.consume_vref_and_vable(vrefinfo, vinfo)
+    resumereader.consume_vref_and_vable(vrefinfo, vinfo, ginfo)
     #
     # First get a chain of blackhole interpreters whose length is given
     # by the depth of rd_frame_info_list.  The first one we get must be
@@ -846,11 +938,11 @@
     resumereader.done()
     return firstbh
 
-def force_from_resumedata(metainterp_sd, storage, vinfo=None):
+def force_from_resumedata(metainterp_sd, storage, vinfo, ginfo):
     resumereader = ResumeDataDirectReader(metainterp_sd.cpu, storage)
     resumereader.handling_async_forcing()
     vrefinfo = metainterp_sd.virtualref_info
-    resumereader.consume_vref_and_vable(vrefinfo, vinfo)
+    resumereader.consume_vref_and_vable(vrefinfo, vinfo, ginfo)
     return resumereader.force_all_virtuals()
 
 class ResumeDataDirectReader(AbstractResumeDataReader):
@@ -925,11 +1017,12 @@
         return specialize_value(TYPE, x)
     load_value_of_type._annspecialcase_ = 'specialize:arg(1)'
 
-    def consume_vref_and_vable(self, vrefinfo, vinfo):
+    def consume_vref_and_vable(self, vrefinfo, vinfo, ginfo):
         nums = self.cur_numb.nums
         self.cur_numb = self.cur_numb.prev
         if self.resume_after_guard_not_forced != 2:
             end_vref = self.consume_vable_info(vinfo, nums)
+            if ginfo is not None: end_vref -= 1
             self.consume_virtualref_info(vrefinfo, nums, end_vref)
 
     def allocate_with_vtable(self, known_class):
@@ -967,6 +1060,31 @@
         result = funcptr(str, start, start + length)
         return lltype.cast_opaque_ptr(llmemory.GCREF, result)
 
+    def allocate_unicode(self, length):
+        return self.cpu.bh_newunicode(length)
+
+    def unicode_setitem(self, str, index, charnum):
+        char = self.decode_int(charnum)
+        self.cpu.bh_unicodesetitem(str, index, char)
+
+    def concat_unicodes(self, str1num, str2num):
+        str1 = self.decode_ref(str1num)
+        str2 = self.decode_ref(str2num)
+        str1 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str1)
+        str2 = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str2)
+        funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_CONCAT)
+        result = funcptr(str1, str2)
+        return lltype.cast_opaque_ptr(llmemory.GCREF, result)
+
+    def slice_unicode(self, strnum, startnum, lengthnum):
+        str = self.decode_ref(strnum)
+        start = self.decode_int(startnum)
+        length = self.decode_int(lengthnum)
+        str = lltype.cast_opaque_ptr(lltype.Ptr(rstr.UNICODE), str)
+        funcptr = funcptr_for_oopspec(EffectInfo.OS_UNI_SLICE)
+        result = funcptr(str, start, start + length)
+        return lltype.cast_opaque_ptr(llmemory.GCREF, result)
+
     def setfield(self, descr, struct, fieldnum):
         if descr.is_pointer_field():
             newvalue = self.decode_ref(fieldnum)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/simple_optimize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/simple_optimize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/simple_optimize.py	Sat Oct 30 16:41:17 2010
@@ -9,11 +9,13 @@
 
 def transform(op):
     from pypy.jit.metainterp.history import AbstractDescr
-    # Rename CALL_PURE to CALL.
+    # Rename CALL_PURE and CALL_LOOPINVARIANT to CALL.
     # Simplify the VIRTUAL_REF_* so that they don't show up in the backend.
     if op.getopnum() == rop.CALL_PURE:
         op = ResOperation(rop.CALL, op.getarglist()[1:], op.result,
                           op.getdescr())
+    elif op.getopnum() == rop.CALL_LOOPINVARIANT:
+        op = op.copy_and_change(rop.CALL)
     elif op.getopnum() == rop.VIRTUAL_REF:
         op = ResOperation(rop.SAME_AS, [op.getarg(0)], op.result)
     elif op.getopnum() == rop.VIRTUAL_REF_FINISH:

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_basic.py	Sat Oct 30 16:41:17 2010
@@ -2,6 +2,7 @@
 import sys
 from pypy.rlib.jit import JitDriver, we_are_jitted, hint, dont_look_inside
 from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_SIMPLE, loop_invariant
+from pypy.rlib.jit import jit_debug, assert_green, AssertGreenFailed
 from pypy.jit.metainterp.warmspot import ll_meta_interp, get_stats
 from pypy.jit.backend.llgraph import runner
 from pypy.jit.metainterp import pyjitpl, history
@@ -44,6 +45,7 @@
         num_green_args = 0
         portal_graph = graphs[0]
         virtualizable_info = None
+        greenfield_info = None
         result_type = result_kind
         portal_runner_ptr = "???"
 
@@ -1775,6 +1777,33 @@
         res = self.interp_operations(f, [10, 3.5])
         assert res == 3.5
 
+    def test_jit_debug(self):
+        myjitdriver = JitDriver(greens = [], reds = ['x'])
+        class A:
+            pass
+        def f(x):
+            while x > 0:
+                myjitdriver.can_enter_jit(x=x)
+                myjitdriver.jit_merge_point(x=x)
+                jit_debug("hi there:", x)
+                jit_debug("foobar")
+                x -= 1
+            return x
+        res = self.meta_interp(f, [8])
+        assert res == 0
+        self.check_loops(jit_debug=2)
+
+    def test_assert_green(self):
+        def f(x, promote):
+            if promote:
+                x = hint(x, promote=True)
+            assert_green(x)
+            return x
+        res = self.interp_operations(f, [8, 1])
+        assert res == 8
+        py.test.raises(AssertGreenFailed, self.interp_operations, f, [8, 0])
+
+
 class TestOOtype(BasicTests, OOJitMixin):
 
     def test_oohash(self):
@@ -1882,7 +1911,7 @@
             c = bool(p1)
             d = not bool(p2)
             return 1000*a + 100*b + 10*c + d
-        prebuilt = [lltype.malloc(TP, flavor='raw')] * 2
+        prebuilt = [lltype.malloc(TP, flavor='raw', immortal=True)] * 2
         expected = f(0, 1)
         assert self.interp_operations(f, [0, 1]) == expected
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py	Sat Oct 30 16:41:17 2010
@@ -6,8 +6,8 @@
 from pypy.jit.metainterp.compile import ResumeGuardCountersInt
 from pypy.jit.metainterp.compile import compile_tmp_callback
 from pypy.jit.metainterp import optimize_nopspec, jitprof, typesystem, compile
-from pypy.jit.metainterp.test.oparser import parse
 from pypy.jit.metainterp.test.test_optimizefindnode import LLtypeMixin
+from pypy.jit.tool.oparser import parse
 
 
 def test_insert_loop_token():

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitdriver.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitdriver.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_jitdriver.py	Sat Oct 30 16:41:17 2010
@@ -72,6 +72,33 @@
         # we expect no loop at all for 'loop1': it should always be inlined
         self.check_tree_loop_count(2)    # 1 x loop, 1 x enter bridge
 
+    def test_inactive_jitdriver(self):
+        myjitdriver1 = JitDriver(greens=[], reds=['n', 'm'],
+                                 get_printable_location = getloc1)
+        myjitdriver2 = JitDriver(greens=['g'], reds=['r'],
+                                 get_printable_location = getloc2)
+        #
+        myjitdriver1.active = False    # <===
+        #
+        def loop1(n, m):
+            while n > 0:
+                myjitdriver1.can_enter_jit(n=n, m=m)
+                myjitdriver1.jit_merge_point(n=n, m=m)
+                n -= m
+            return n
+        #
+        def loop2(g, r):
+            while r > 0:
+                myjitdriver2.can_enter_jit(g=g, r=r)
+                myjitdriver2.jit_merge_point(g=g, r=r)
+                r += loop1(r, g) + (-1)
+            return r
+        #
+        res = self.meta_interp(loop2, [4, 40], repeat=7, inline=True)
+        assert res == loop2(4, 40)
+        # we expect no int_sub, but a residual call
+        self.check_loops(int_sub=0, call=1)
+
 
 class TestLLtype(MultipleJitDriversTests, LLJitMixin):
     pass

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_logger.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_logger.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_logger.py	Sat Oct 30 16:41:17 2010
@@ -1,6 +1,6 @@
 import sys
 from pypy.rlib import debug
-from pypy.jit.metainterp.test.oparser import pure_parse
+from pypy.jit.tool.oparser import pure_parse
 from pypy.jit.metainterp import logger
 from pypy.jit.metainterp.typesystem import llhelper
 from StringIO import StringIO

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_oparser.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_oparser.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_oparser.py	Sat Oct 30 16:41:17 2010
@@ -1,7 +1,7 @@
 
 from pypy.rpython.lltypesystem import lltype, llmemory
 
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.jit.metainterp.resoperation import rop
 from pypy.jit.metainterp.history import AbstractDescr, BoxInt, LoopToken,\
      BoxFloat

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	Sat Oct 30 16:41:17 2010
@@ -131,14 +131,21 @@
 # ____________________________________________________________
 
 def equaloplists(oplist1, oplist2, strict_fail_args=True, remap={}):
-    print '-'*20, 'Comparing lists', '-'*20
+    # try to use the full width of the terminal to display the list
+    # unfortunately, does not work with the default capture method of py.test
+    # (which is fd), you you need to use either -s or --capture=sys, else you
+    # get the standard 80 columns width
+    totwidth = py.io.get_terminal_width()
+    width = totwidth / 2 - 1
+    print ' Comparing lists '.center(totwidth, '-')
+    print '%s| %s' % ('optimized'.center(width), 'expected'.center(width))
     for op1, op2 in zip(oplist1, oplist2):
         txt1 = str(op1)
         txt2 = str(op2)
         while txt1 or txt2:
-            print '%-39s| %s' % (txt1[:39], txt2[:39])
-            txt1 = txt1[39:]
-            txt2 = txt2[39:]
+            print '%s| %s' % (txt1[:width].ljust(width), txt2[:width])
+            txt1 = txt1[width:]
+            txt2 = txt2[width:]
         assert op1.getopnum() == op2.getopnum()
         assert op1.numargs() == op2.numargs()
         for i in range(op1.numargs()):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefindnode.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefindnode.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefindnode.py	Sat Oct 30 16:41:17 2010
@@ -18,7 +18,7 @@
 from pypy.jit.metainterp.specnode import ConstantSpecNode
 from pypy.jit.codewriter.effectinfo import EffectInfo
 from pypy.jit.codewriter.heaptracker import register_known_gctype, adr2int
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 
 def test_sort_descrs():
     class PseudoDescr(AbstractDescr):
@@ -117,33 +117,32 @@
                             EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE))
     arraycopydescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
                  EffectInfo([], [], [], oopspecindex=EffectInfo.OS_ARRAYCOPY))
-    strconcatdescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_CONCAT))
-    slicedescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_SLICE))
-    strequaldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_EQUAL))
-    streq_slice_checknull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_CHECKNULL))
-    streq_slice_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_NONNULL))
-    streq_slice_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_CHAR))
-    streq_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_NONNULL))
-    streq_nonnull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_NONNULL_CHAR))
-    streq_checknull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_CHECKNULL_CHAR))
-    streq_lengthok_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_LENGTHOK))
+
+    for _name, _os in [
+        ('strconcatdescr',               'OS_STR_CONCAT'),
+        ('strslicedescr',                'OS_STR_SLICE'),
+        ('strequaldescr',                'OS_STR_EQUAL'),
+        ('streq_slice_checknull_descr',  'OS_STREQ_SLICE_CHECKNULL'),
+        ('streq_slice_nonnull_descr',    'OS_STREQ_SLICE_NONNULL'),
+        ('streq_slice_char_descr',       'OS_STREQ_SLICE_CHAR'),
+        ('streq_nonnull_descr',          'OS_STREQ_NONNULL'),
+        ('streq_nonnull_char_descr',     'OS_STREQ_NONNULL_CHAR'),
+        ('streq_checknull_char_descr',   'OS_STREQ_CHECKNULL_CHAR'),
+        ('streq_lengthok_descr',         'OS_STREQ_LENGTHOK'),
+        ]:
+        _oopspecindex = getattr(EffectInfo, _os)
+        locals()[_name] = \
+            cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=_oopspecindex))
+        #
+        _oopspecindex = getattr(EffectInfo, _os.replace('STR', 'UNI'))
+        locals()[_name.replace('str', 'unicode')] = \
+            cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=_oopspecindex))
+
+    s2u_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR2UNICODE))
+    #
 
     class LoopToken(AbstractDescr):
         pass

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 16:41:17 2010
@@ -12,7 +12,7 @@
 from pypy.jit.metainterp.jitprof import EmptyProfiler
 from pypy.jit.metainterp import executor, compile, resume, history
 from pypy.jit.metainterp.resoperation import rop, opname, ResOperation
-from pypy.jit.metainterp.test.oparser import pure_parse
+from pypy.jit.tool.oparser import pure_parse
 from pypy.jit.metainterp.test.test_optimizebasic import equaloplists
 
 class Fake(object):
@@ -28,7 +28,6 @@
         self.globaldata = Fake()
 
 # ____________________________________________________________
-
 class Storage(compile.ResumeGuardDescr):
     "for tests."
     def __init__(self, metainterp_sd=None, original_greenkey=None):
@@ -93,6 +92,10 @@
             expected = self.parse(expected_preamble)
             self.assert_equal(loop.preamble, expected)
 
+        return loop
+
+class OptimizeOptTest(BaseTestOptimizeOpt):
+
     def test_simple(self):
         ops = """
         []
@@ -2609,7 +2612,7 @@
             ''', rop.GUARD_TRUE)
 
 
-class TestLLtype(BaseTestOptimizeOpt, LLtypeMixin):
+class TestLLtype(OptimizeOptTest, LLtypeMixin):
 
     def test_residual_call_does_not_invalidate_caches(self):
         ops = """
@@ -3455,7 +3458,7 @@
         i0 = strlen(p0)
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not', expected)
 
     def test_addsub_const(self):
         ops = """
@@ -3900,6 +3903,15 @@
         """
         self.optimize_loop(ops, 'Not, Not', expected)
 
+    # ----------
+    def optimize_strunicode_loop(self, ops, spectext, optops):
+        # check with the arguments passed in
+        self.optimize_loop(ops, spectext, optops)
+        # check with replacing 'str' with 'unicode' everywhere
+        self.optimize_loop(ops.replace('str','unicode').replace('s"', 'u"'),
+                           spectext,
+                           optops.replace('str','unicode').replace('s"', 'u"'))
+
     def test_newstr_1(self):
         ops = """
         [i0]
@@ -3912,7 +3924,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not', expected)
 
     def test_newstr_2(self):
         ops = """
@@ -3928,7 +3940,7 @@
         [i0, i1]
         jump(i1, i0)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_concat_1(self):
         ops = """
@@ -3949,7 +3961,7 @@
         copystrcontent(p2, p3, 0, i4, i5)
         jump(p2, p3)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_concat_vstr2_str(self):
         ops = """
@@ -3972,7 +3984,7 @@
         copystrcontent(p2, p3, 0, 2, i4)
         jump(i1, i0, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_concat_str_vstr2(self):
         ops = """
@@ -3996,7 +4008,7 @@
         i6 = int_add(i5, 1)      # will be killed by the backend
         jump(i1, i0, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_concat_str_str_str(self):
         ops = """
@@ -4023,12 +4035,12 @@
         copystrcontent(p3, p5, 0, i12b, i3b)
         jump(p2, p3, p5)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_concat_str_cstr1(self):
         ops = """
         [p2]
-        p3 = call(0, p2, "x", descr=strconcatdescr)
+        p3 = call(0, p2, s"x", descr=strconcatdescr)
         jump(p3)
         """
         expected = """
@@ -4042,28 +4054,28 @@
         i5 = int_add(i4, 1)      # will be killed by the backend
         jump(p3)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not', expected)
 
     def test_str_concat_consts(self):
         ops = """
         []
-        p1 = same_as("ab")
-        p2 = same_as("cde")
+        p1 = same_as(s"ab")
+        p2 = same_as(s"cde")
         p3 = call(0, p1, p2, descr=strconcatdescr)
         escape(p3)
         jump()
         """
         expected = """
         []
-        escape("abcde")
+        escape(s"abcde")
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_strunicode_loop(ops, '', expected)
 
     def test_str_slice_1(self):
         ops = """
         [p1, i1, i2]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
         jump(p2, i1, i2)
         """
         expected = """
@@ -4073,12 +4085,12 @@
         copystrcontent(p1, p2, i1, 0, i3)
         jump(p2, i1, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
 
     def test_str_slice_2(self):
         ops = """
         [p1, i2]
-        p2 = call(0, p1, 0, i2, descr=slicedescr)
+        p2 = call(0, p1, 0, i2, descr=strslicedescr)
         jump(p2, i2)
         """
         expected = """
@@ -4087,13 +4099,13 @@
         copystrcontent(p1, p2, 0, 0, i2)
         jump(p2, i2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_slice_3(self):
         ops = """
         [p1, i1, i2, i3, i4]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
-        p3 = call(0, p2, i3, i4, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
+        p3 = call(0, p2, i3, i4, descr=strslicedescr)
         jump(p3, i1, i2, i3, i4)
         """
         expected = """
@@ -4105,12 +4117,12 @@
         copystrcontent(p1, p3, i6, 0, i5)
         jump(p3, i1, i2, i3, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not, Not', expected)
 
     def test_str_slice_getitem1(self):
         ops = """
         [p1, i1, i2, i3]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
         i4 = strgetitem(p2, i3)
         escape(i4)
         jump(p1, i1, i2, i3)
@@ -4123,7 +4135,7 @@
         escape(i4)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected)
 
     def test_str_slice_plain(self):
         ops = """
@@ -4131,7 +4143,7 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i3)
         strsetitem(p1, 1, i4)
-        p2 = call(0, p1, 1, 2, descr=slicedescr)
+        p2 = call(0, p1, 1, 2, descr=strslicedescr)
         i5 = strgetitem(p2, 0)
         escape(i5)
         jump(i3, i4)
@@ -4141,12 +4153,12 @@
         escape(i4)
         jump(i3, i4)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
 
     def test_str_slice_concat(self):
         ops = """
         [p1, i1, i2, p2]
-        p3 = call(0, p1, i1, i2, descr=slicedescr)
+        p3 = call(0, p1, i1, i2, descr=strslicedescr)
         p4 = call(0, p3, p2, descr=strconcatdescr)
         jump(p4, i1, i2, p2)
         """
@@ -4162,10 +4174,10 @@
         copystrcontent(p2, p4, 0, i3, i4b)
         jump(p4, i1, i2, p2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected)
 
     # ----------
-    def optimize_loop_extradescrs(self, ops, spectext, optops):
+    def optimize_strunicode_loop_extradescrs(self, ops, spectext, optops):
         from pypy.jit.metainterp.optimizeopt import string
         def my_callinfo_for_oopspec(oopspecindex):
             calldescrtype = type(LLtypeMixin.strequaldescr)
@@ -4180,7 +4192,7 @@
         saved = string.callinfo_for_oopspec
         try:
             string.callinfo_for_oopspec = my_callinfo_for_oopspec
-            self.optimize_loop(ops, spectext, optops)
+            self.optimize_strunicode_loop(ops, spectext, optops)
         finally:
             string.callinfo_for_oopspec = saved
 
@@ -4191,7 +4203,7 @@
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', ops)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', ops)
 
     def test_str_equal_noop2(self):
         ops = """
@@ -4216,12 +4228,13 @@
         escape(i0)
         jump(p1, p2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice1(self):
         ops = """
         [p1, i1, i2, p3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p4, p3, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4233,12 +4246,13 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice2(self):
         ops = """
         [p1, i1, i2, p3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p3, p4, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4250,13 +4264,14 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice3(self):
         ops = """
         [p1, i1, i2, p3]
         guard_nonnull(p3) []
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p3, p4, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4269,13 +4284,14 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice4(self):
         ops = """
         [p1, i1, i2]
-        p3 = call(0, p1, i1, i2, descr=slicedescr)
-        i0 = call(0, p3, "x", descr=strequaldescr)
+        p3 = call(0, p1, i1, i2, descr=strslicedescr)
+        i0 = call(0, p3, s"x", descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2)
         """
@@ -4286,12 +4302,13 @@
         escape(i0)
         jump(p1, i1, i2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_slice5(self):
         ops = """
         [p1, i1, i2, i3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         p5 = newstr(1)
         strsetitem(p5, 0, i3)
         i0 = call(0, p5, p4, descr=strequaldescr)
@@ -4305,7 +4322,8 @@
         escape(i0)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+                                                  expected)
 
     def test_str_equal_none1(self):
         ops = """
@@ -4320,7 +4338,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_none2(self):
         ops = """
@@ -4335,30 +4353,30 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull1(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "hello world", descr=strequaldescr)
+        i0 = call(0, p1, s"hello world", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
         expected = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "hello world", descr=streq_nonnull_descr)
+        i0 = call(0, p1, s"hello world", descr=streq_nonnull_descr)
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull2(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "", descr=strequaldescr)
+        i0 = call(0, p1, s"", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4370,13 +4388,13 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull3(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "x", descr=strequaldescr)
+        i0 = call(0, p1, s"x", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4387,13 +4405,13 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_nonnull4(self):
         ops = """
         [p1, p2]
         p4 = call(0, p1, p2, descr=strconcatdescr)
-        i0 = call(0, "hello world", p4, descr=strequaldescr)
+        i0 = call(0, s"hello world", p4, descr=strequaldescr)
         escape(i0)
         jump(p1, p2)
         """
@@ -4408,17 +4426,17 @@
         i5 = strlen(p2)
         i6 = int_add(i4, i5)      # will be killed by the backend
         copystrcontent(p2, p4, 0, i4, i5)
-        i0 = call(0, "hello world", p4, descr=streq_nonnull_descr)
+        i0 = call(0, s"hello world", p4, descr=streq_nonnull_descr)
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected)
 
     def test_str_equal_chars0(self):
         ops = """
         [i1]
         p1 = newstr(0)
-        i0 = call(0, p1, "", descr=strequaldescr)
+        i0 = call(0, p1, s"", descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4427,14 +4445,14 @@
         escape(1)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_chars1(self):
         ops = """
         [i1]
         p1 = newstr(1)
         strsetitem(p1, 0, i1)
-        i0 = call(0, p1, "x", descr=strequaldescr)
+        i0 = call(0, p1, s"x", descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4444,7 +4462,7 @@
         escape(i0)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_chars2(self):
         ops = """
@@ -4452,7 +4470,7 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i1)
         strsetitem(p1, 1, i2)
-        i0 = call(0, p1, "xy", descr=strequaldescr)
+        i0 = call(0, p1, s"xy", descr=strequaldescr)
         escape(i0)
         jump(i1, i2)
         """
@@ -4461,16 +4479,16 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i1)
         strsetitem(p1, 1, i2)
-        i0 = call(0, p1, "xy", descr=streq_lengthok_descr)
+        i0 = call(0, p1, s"xy", descr=streq_lengthok_descr)
         escape(i0)
         jump(i1, i2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected)
 
     def test_str_equal_chars3(self):
         ops = """
         [p1]
-        i0 = call(0, "x", p1, descr=strequaldescr)
+        i0 = call(0, s"x", p1, descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4480,14 +4498,14 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
     def test_str_equal_lengthmismatch1(self):
         ops = """
         [i1]
         p1 = newstr(1)
         strsetitem(p1, 0, i1)
-        i0 = call(0, "xy", p1, descr=strequaldescr)
+        i0 = call(0, s"xy", p1, descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4496,13 +4514,36 @@
         escape(0)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
 
-    # XXX unicode operations
-    # XXX str2unicode
+    def test_str2unicode_constant(self):
+        ops = """
+        []
+        p0 = call(0, "xy", descr=s2u_descr)      # string -> unicode
+        escape(p0)
+        jump()
+        """
+        expected = """
+        []
+        escape(u"xy")
+        jump()
+        """
+        self.optimize_strunicode_loop_extradescrs(ops, '', expected)
+
+    def test_str2unicode_nonconstant(self):
+        ops = """
+        [p0]
+        p1 = call(0, p0, descr=s2u_descr)      # string -> unicode
+        escape(p1)
+        jump(p1)
+        """
+        self.optimize_strunicode_loop_extradescrs(ops, 'Not', ops)
+        # more generally, supporting non-constant but virtual cases is
+        # not obvious, because of the exception UnicodeDecodeError that
+        # can be raised by ll_str2unicode()
 
 
-##class TestOOtype(BaseTestOptimizeOpt, OOtypeMixin):
+##class TestOOtype(OptimizeOptTest, OOtypeMixin):
 
 ##    def test_instanceof(self):
 ##        ops = """

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py	Sat Oct 30 16:41:17 2010
@@ -240,6 +240,17 @@
         return FakeBuiltObject(strconcat=[left, right])
     def slice_string(self, str, start, length):
         return FakeBuiltObject(strslice=[str, start, length])
+    def allocate_unicode(self, length):
+        return FakeBuiltObject(unistring=[None]*length)
+    def unicode_setitem(self, unistring, i, fieldnum):
+        value, tag = untag(fieldnum)
+        assert tag == TAGINT
+        assert 0 <= i < len(unistring.unistring)
+        unistring.unistring[i] = value
+    def concat_unicodes(self, left, right):
+        return FakeBuiltObject(uniconcat=[left, right])
+    def slice_unicode(self, str, start, length):
+        return FakeBuiltObject(unislice=[str, start, length])
 
 class FakeBuiltObject(object):
     def __init__(self, **kwds):
@@ -304,6 +315,30 @@
     assert reader.force_all_virtuals() == [
         FakeBuiltObject(strslice=info.fieldnums)]
 
+def test_vuniplaininfo():
+    info = VUniPlainInfo()
+    info.fieldnums = [tag(60, TAGINT)]
+    reader = FakeResumeDataReader()
+    reader._prepare_virtuals([info])
+    assert reader.force_all_virtuals() == [
+        FakeBuiltObject(unistring=[60])]
+
+def test_vuniconcatinfo():
+    info = VUniConcatInfo()
+    info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX)]
+    reader = FakeResumeDataReader()
+    reader._prepare_virtuals([info])
+    assert reader.force_all_virtuals() == [
+        FakeBuiltObject(uniconcat=info.fieldnums)]
+
+def test_vunisliceinfo():
+    info = VUniSliceInfo()
+    info.fieldnums = [tag(10, TAGBOX), tag(20, TAGBOX), tag(30, TAGBOX)]
+    reader = FakeResumeDataReader()
+    reader._prepare_virtuals([info])
+    assert reader.force_all_virtuals() == [
+        FakeBuiltObject(unislice=info.fieldnums)]
+
 # ____________________________________________________________
 
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_string.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_string.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_string.py	Sat Oct 30 16:41:17 2010
@@ -6,14 +6,17 @@
 
 
 class StringTests:
+    _str, _chr = str, chr
+
     def test_eq_residual(self):
+        _str = self._str
         jitdriver = JitDriver(greens = [], reds = ['n', 'i', 's'])
-        global_s = "hello"
+        global_s = _str("hello")
         def f(n, b, s):
             if b:
-                s += "ello"
+                s += _str("ello")
             else:
-                s += "allo"
+                s += _str("allo")
             i = 0
             while n > 0:
                 jitdriver.can_enter_jit(s=s, n=n, i=i)
@@ -21,18 +24,19 @@
                 n -= 1 + (s == global_s)
                 i += 1
             return i
-        res = self.meta_interp(f, [10, True, 'h'], listops=True)
+        res = self.meta_interp(f, [10, True, _str('h')], listops=True)
         assert res == 5
         self.check_loops(**{self.CALL: 1, self.CALL_PURE: 0})
 
     def test_eq_folded(self):
+        _str = self._str
         jitdriver = JitDriver(greens = ['s'], reds = ['n', 'i'])
-        global_s = "hello"
+        global_s = _str("hello")
         def f(n, b, s):
             if b:
-                s += "ello"
+                s += _str("ello")
             else:
-                s += "allo"
+                s += _str("allo")
             i = 0
             while n > 0:
                 jitdriver.can_enter_jit(s=s, n=n, i=i)
@@ -40,31 +44,18 @@
                 n -= 1 + (s == global_s)
                 i += 1
             return i
-        res = self.meta_interp(f, [10, True, 'h'], listops=True)
+        res = self.meta_interp(f, [10, True, _str('h')], listops=True)
         assert res == 5
         self.check_loops(**{self.CALL: 0, self.CALL_PURE: 0})
 
     def test_newstr(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['n', 'm'])
         def f(n, m):
             while True:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                bytecode = 'adlfkj' + chr(n)
-                res = bytecode[n]
-                m -= 1
-                if m < 0:
-                    return ord(res)
-        res = self.meta_interp(f, [6, 10])
-        assert res == 6
-
-    def test_newunicode(self):
-        jitdriver = JitDriver(greens = [], reds = ['n', 'm'])
-        def f(n, m):
-            while True:
-                jitdriver.can_enter_jit(m=m, n=n)
-                jitdriver.jit_merge_point(m=m, n=n)
-                bytecode = u'adlfkj' + unichr(n)
+                bytecode = _str('adlfkj') + _chr(n)
                 res = bytecode[n]
                 m -= 1
                 if m < 0:
@@ -73,95 +64,96 @@
         assert res == 6
 
     def test_char2string_pure(self):
-        for dochr in [chr, ]: #unichr]:
-            jitdriver = JitDriver(greens = [], reds = ['n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            def f(n):
-                while n > 0:
-                    jitdriver.can_enter_jit(n=n)
-                    jitdriver.jit_merge_point(n=n)
-                    s = dochr(n)
-                    if not we_are_jitted():
-                        s += s     # forces to be a string
-                    if n > 100:
-                        escape(s)
-                    n -= 1
-                return 42
-            self.meta_interp(f, [6])
-            self.check_loops(newstr=0, strsetitem=0, strlen=0,
-                             newunicode=0, unicodesetitem=0, unicodelen=0)
+        _str, _chr = self._str, self._chr
+        jitdriver = JitDriver(greens = [], reds = ['n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        def f(n):
+            while n > 0:
+                jitdriver.can_enter_jit(n=n)
+                jitdriver.jit_merge_point(n=n)
+                s = _chr(n)
+                if not we_are_jitted():
+                    s += s     # forces to be a string
+                if n > 100:
+                    escape(s)
+                n -= 1
+            return 42
+        self.meta_interp(f, [6])
+        self.check_loops(newstr=0, strsetitem=0, strlen=0,
+                         newunicode=0, unicodesetitem=0, unicodelen=0)
 
     def test_char2string_escape(self):
-        for dochr in [chr, ]: #unichr]:
-            jitdriver = JitDriver(greens = [], reds = ['n', 'total'])
-            @dont_look_inside
-            def escape(x):
-                return ord(x[0])
-            def f(n):
-                total = 0
-                while n > 0:
-                    jitdriver.can_enter_jit(n=n, total=total)
-                    jitdriver.jit_merge_point(n=n, total=total)
-                    s = dochr(n)
-                    if not we_are_jitted():
-                        s += s    # forces to be a string
-                    total += escape(s)
-                    n -= 1
-                return total
-            res = self.meta_interp(f, [6])
-            assert res == 21
+        _str, _chr = self._str, self._chr
+        jitdriver = JitDriver(greens = [], reds = ['n', 'total'])
+        @dont_look_inside
+        def escape(x):
+            return ord(x[0])
+        def f(n):
+            total = 0
+            while n > 0:
+                jitdriver.can_enter_jit(n=n, total=total)
+                jitdriver.jit_merge_point(n=n, total=total)
+                s = _chr(n)
+                if not we_are_jitted():
+                    s += s    # forces to be a string
+                total += escape(s)
+                n -= 1
+            return total
+        res = self.meta_interp(f, [6])
+        assert res == 21
 
     def test_char2string2char(self):
-        for dochr in [chr, ]: #unichr]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'total'])
-            def f(m):
-                total = 0
-                while m > 0:
-                    jitdriver.can_enter_jit(m=m, total=total)
-                    jitdriver.jit_merge_point(m=m, total=total)
-                    string = dochr(m)
-                    if m > 100:
-                        string += string    # forces to be a string
-                    # read back the character
-                    c = string[0]
-                    total += ord(c)
-                    m -= 1
-                return total
-            res = self.meta_interp(f, [6])
-            assert res == 21
-            self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0,
-                             newunicode=0, unicodegetitem=0, unicodesetitem=0,
-                             unicodelen=0)
+        _str, _chr = self._str, self._chr
+        jitdriver = JitDriver(greens = [], reds = ['m', 'total'])
+        def f(m):
+            total = 0
+            while m > 0:
+                jitdriver.can_enter_jit(m=m, total=total)
+                jitdriver.jit_merge_point(m=m, total=total)
+                string = _chr(m)
+                if m > 100:
+                    string += string    # forces to be a string
+                # read back the character
+                c = string[0]
+                total += ord(c)
+                m -= 1
+            return total
+        res = self.meta_interp(f, [6])
+        assert res == 21
+        self.check_loops(newstr=0, strgetitem=0, strsetitem=0, strlen=0,
+                         newunicode=0, unicodegetitem=0, unicodesetitem=0,
+                         unicodelen=0)
 
     def test_strconcat_pure(self):
-        for somestr in ["abc", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            mylist = [somestr+str(i) for i in range(10)]
-            def f(n, m):
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = mylist[n] + mylist[m]
-                    if m > 100:
-                        escape(s)
-                    m -= 1
-                return 42
-            self.meta_interp(f, [6, 7])
-            self.check_loops(newstr=0, strsetitem=0,
-                             newunicode=0, unicodesetitem=0,
-                             call=0, call_pure=0)
+        _str = self._str
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        mylist = [_str("abc") + _str(i) for i in range(10)]
+        def f(n, m):
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = mylist[n] + mylist[m]
+                if m > 100:
+                    escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 7])
+        self.check_loops(newstr=0, strsetitem=0,
+                         newunicode=0, unicodesetitem=0,
+                         call=0, call_pure=0)
 
     def test_strconcat_escape_str_str(self):
+        _str = self._str
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
@@ -171,46 +163,64 @@
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=0, copystrcontent=2,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=0, copystrcontent=2,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=0,
+                             copyunicodecontent=2,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_str_char(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = mylist[n] + chr(m)
+                s = mylist[n] + _chr(m)
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=1,
+                             copyunicodecontent=1,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_char_str(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = chr(n) + mylist[m]
+                s = _chr(n) + mylist[m]
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=1, copystrcontent=1,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=1,
+                             copyunicodecontent=1,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_char_char(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
@@ -219,91 +229,132 @@
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = chr(n) + chr(m)
+                s = _chr(n) + _chr(m)
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=2, copystrcontent=0,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=2, copystrcontent=0,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=2,
+                             copyunicodecontent=0,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_escape_str_char_str(self):
+        _str, _chr = self._str, self._chr
         jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
         @dont_look_inside
         def escape(x):
             pass
-        mylist = ["somestr"+str(i) for i in range(10)]
+        mylist = [_str("somestr") + _str(i) for i in range(10)]
         def f(n, m):
             while m >= 0:
                 jitdriver.can_enter_jit(m=m, n=n)
                 jitdriver.jit_merge_point(m=m, n=n)
-                s = mylist[n] + chr(n) + mylist[m]
+                s = mylist[n] + _chr(n) + mylist[m]
                 escape(s)
                 m -= 1
             return 42
         self.meta_interp(f, [6, 7])
-        self.check_loops(newstr=1, strsetitem=1, copystrcontent=2,
-                         call=1, call_pure=0)   # escape
+        if _str is str:
+            self.check_loops(newstr=1, strsetitem=1, copystrcontent=2,
+                             call=1, call_pure=0)   # escape
+        else:
+            self.check_loops(newunicode=1, unicodesetitem=1,
+                             copyunicodecontent=2,
+                             call=1, call_pure=0)   # escape
 
     def test_strconcat_guard_fail(self):
-        for somestr in ["abc", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            mylist = [somestr+str(i) for i in range(12)]
-            def f(n, m):
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = mylist[n] + mylist[m]
-                    if m & 1:
-                        escape(s)
-                    m -= 1
-                return 42
-            self.meta_interp(f, [6, 10])
+        _str = self._str
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        mylist = [_str("abc") + _str(i) for i in range(12)]
+        def f(n, m):
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = mylist[n] + mylist[m]
+                if m & 1:
+                    escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 10])
 
     def test_strslice(self):
-        for somestr in ["abc", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            def f(n, m):
-                assert n >= 0
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = "foobarbazetc"[m:n]
-                    if m <= 5:
-                        escape(s)
-                    m -= 1
-                return 42
-            self.meta_interp(f, [10, 10])
+        _str = self._str
+        longstring = _str("foobarbazetc")
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        def f(n, m):
+            assert n >= 0
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = longstring[m:n]
+                if m <= 5:
+                    escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [10, 10])
 
     def test_streq_char(self):
-        for somestr in ["?abcdefg", ]: #u"def"]:
-            jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
-            @dont_look_inside
-            def escape(x):
-                pass
-            def f(n, m):
-                assert n >= 0
-                while m >= 0:
-                    jitdriver.can_enter_jit(m=m, n=n)
-                    jitdriver.jit_merge_point(m=m, n=n)
-                    s = somestr[:m]
-                    escape(s == "?")
-                    m -= 1
-                return 42
-            self.meta_interp(f, [6, 7])
-            self.check_loops(newstr=0, newunicode=0)
-
-
-class TestOOtype(StringTests, OOJitMixin):
-    CALL = "oosend"
-    CALL_PURE = "oosend_pure"
+        _str = self._str
+        longstring = _str("?abcdefg")
+        somechar = _str("?")
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        @dont_look_inside
+        def escape(x):
+            pass
+        def f(n, m):
+            assert n >= 0
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                s = longstring[:m]
+                escape(s == somechar)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 7])
+        self.check_loops(newstr=0, newunicode=0)
+
+
+#class TestOOtype(StringTests, OOJitMixin):
+#    CALL = "oosend"
+#    CALL_PURE = "oosend_pure"
 
 class TestLLtype(StringTests, LLJitMixin):
     CALL = "call"
     CALL_PURE = "call_pure"
+
+class TestLLtypeUnicode(TestLLtype):
+    _str, _chr = unicode, unichr
+
+    def test_str2unicode(self):
+        _str = self._str
+        jitdriver = JitDriver(greens = [], reds = ['m', 'n'])
+        class Foo:
+            pass
+        @dont_look_inside
+        def escape(x):
+            assert x == _str("6y")
+        def f(n, m):
+            while m >= 0:
+                jitdriver.can_enter_jit(m=m, n=n)
+                jitdriver.jit_merge_point(m=m, n=n)
+                foo = Foo()
+                foo.y = chr(m)
+                foo.y = "y"
+                s = _str(str(n)) + _str(foo.y)
+                escape(s)
+                m -= 1
+            return 42
+        self.meta_interp(f, [6, 7])
+        self.check_loops(call=3,    # str(), _str(), escape()
+                         newunicode=1, unicodegetitem=0,
+                         unicodesetitem=1, copyunicodecontent=1)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualref.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualref.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualref.py	Sat Oct 30 16:41:17 2010
@@ -93,7 +93,7 @@
         lst = []
         vrefinfo.continue_tracing = lambda vref, virtual: \
                                         lst.append((vref, virtual))
-        resumereader.consume_vref_and_vable(vrefinfo, None)
+        resumereader.consume_vref_and_vable(vrefinfo, None, None)
         del vrefinfo.continue_tracing
         assert len(lst) == 1
         lltype.cast_opaque_ptr(lltype.Ptr(JIT_VIRTUAL_REF),

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_warmspot.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_warmspot.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_warmspot.py	Sat Oct 30 16:41:17 2010
@@ -296,6 +296,69 @@
         assert res == 1
         self.check_loops(int_add=1)   # I get 13 without the loop_header()
 
+    def test_omit_can_enter_jit(self):
+        # Simple test comparing the effects of always giving a can_enter_jit(),
+        # or not giving any.  Mostly equivalent, except that if given, it is
+        # ignored the first time, and so it ends up taking one extra loop to
+        # start JITting.
+        mydriver = JitDriver(greens=[], reds=['m'])
+        #
+        for i2 in range(10):
+            def f2(m):
+                while m > 0:
+                    mydriver.jit_merge_point(m=m)
+                    m -= 1
+            self.meta_interp(f2, [i2])
+            try:
+                self.check_tree_loop_count(1)
+                break
+            except AssertionError:
+                print "f2: no loop generated for i2==%d" % i2
+        else:
+            raise     # re-raise the AssertionError: check_loop_count never 1
+        #
+        for i1 in range(10):
+            def f1(m):
+                while m > 0:
+                    mydriver.can_enter_jit(m=m)
+                    mydriver.jit_merge_point(m=m)
+                    m -= 1
+            self.meta_interp(f1, [i1])
+            try:
+                self.check_tree_loop_count(1)
+                break
+            except AssertionError:
+                print "f1: no loop generated for i1==%d" % i1
+        else:
+            raise     # re-raise the AssertionError: check_loop_count never 1
+        #
+        assert i1 - 1 == i2
+
+    def test_no_loop_at_all(self):
+        mydriver = JitDriver(greens=[], reds=['m'])
+        def f2(m):
+            mydriver.jit_merge_point(m=m)
+            return m - 1
+        def f1(m):
+            while m > 0:
+                m = f2(m)
+        self.meta_interp(f1, [8])
+        # it should generate one "loop" only, which ends in a FINISH
+        # corresponding to the return from f2.
+        self.check_tree_loop_count(1)
+        self.check_loop_count(0)
+
+    def test_simple_loop(self):
+        mydriver = JitDriver(greens=[], reds=['m'])
+        def f1(m):
+            while m > 0:
+                mydriver.jit_merge_point(m=m)
+                m = m - 1
+        self.meta_interp(f1, [8])
+        self.check_loop_count(1)
+        self.check_loops({'int_sub': 1, 'int_gt': 1, 'guard_true': 1,
+                          'jump': 1})
+
 
 class TestLLWarmspot(WarmspotTests, LLJitMixin):
     CPUClass = runner.LLtypeCPU

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/virtualref.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/virtualref.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/virtualref.py	Sat Oct 30 16:41:17 2010
@@ -16,7 +16,8 @@
             ('virtualref_index', lltype.Signed),
             ('forced', rclass.OBJECTPTR))
         self.jit_virtual_ref_vtable = lltype.malloc(rclass.OBJECT_VTABLE,
-                                                    zero=True, flavor='raw')
+                                                    zero=True, flavor='raw',
+                                                    immortal=True)
         self.jit_virtual_ref_vtable.name = rclass.alloc_array_name(
             'jit_virtual_ref')
         # build some constants

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmspot.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmspot.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmspot.py	Sat Oct 30 16:41:17 2010
@@ -67,9 +67,16 @@
 def jittify_and_run(interp, graph, args, repeat=1,
                     backendopt=False, trace_limit=sys.maxint,
                     debug_level=DEBUG_STEPS, inline=False, **kwds):
+    from pypy.config.config import ConfigError
     translator = interp.typer.annotator.translator
-    translator.config.translation.gc = "boehm"
-    translator.config.translation.list_comprehension_operations = True
+    try:
+        translator.config.translation.gc = "boehm"
+    except ConfigError:
+        pass
+    try:
+        translator.config.translation.list_comprehension_operations = True
+    except ConfigError:
+        pass
     warmrunnerdesc = WarmRunnerDesc(translator, backendopt=backendopt, **kwds)
     for jd in warmrunnerdesc.jitdrivers_sd:
         jd.warmstate.set_param_threshold(3)          # for tests
@@ -103,15 +110,16 @@
             for i in range(len(block.operations)):
                 op = block.operations[i]
                 if (op.opname == 'jit_marker' and
-                    op.args[0].value == marker_name):
+                    op.args[0].value == marker_name and
+                    op.args[1].value.active):   # the jitdriver
                     results.append((graph, block, i))
     return results
 
 def find_can_enter_jit(graphs):
-    results = _find_jit_marker(graphs, 'can_enter_jit')
-    if not results:
-        raise Exception("no can_enter_jit found!")
-    return results
+    return _find_jit_marker(graphs, 'can_enter_jit')
+
+def find_loop_headers(graphs):
+    return _find_jit_marker(graphs, 'loop_header')
 
 def find_jit_merge_points(graphs):
     results = _find_jit_marker(graphs, 'jit_merge_point')
@@ -204,9 +212,9 @@
                 "there are multiple jit_merge_points with the same jitdriver"
 
     def split_graph_and_record_jitdriver(self, graph, block, pos):
-        jd = JitDriverStaticData()
-        jd._jit_merge_point_pos = (graph, block, pos)
         op = block.operations[pos]
+        jd = JitDriverStaticData()
+        jd._jit_merge_point_pos = (graph, op)
         args = op.args[2:]
         s_binding = self.translator.annotator.binding
         jd._portal_args_s = [s_binding(v) for v in args]
@@ -279,10 +287,20 @@
     def make_virtualizable_infos(self):
         vinfos = {}
         for jd in self.jitdrivers_sd:
+            #
+            jd.greenfield_info = None
+            for name in jd.jitdriver.greens:
+                if '.' in name:
+                    from pypy.jit.metainterp.greenfield import GreenFieldInfo
+                    jd.greenfield_info = GreenFieldInfo(self.cpu, jd)
+                    break
+            #
             if not jd.jitdriver.virtualizables:
                 jd.virtualizable_info = None
                 jd.index_of_virtualizable = -1
                 continue
+            else:
+                assert jd.greenfield_info is None, "XXX not supported yet"
             #
             jitdriver = jd.jitdriver
             assert len(jitdriver.virtualizables) == 1    # for now
@@ -450,8 +468,7 @@
             self.make_args_specification(jd)
 
     def make_args_specification(self, jd):
-        graph, block, index = jd._jit_merge_point_pos
-        op = block.operations[index]
+        graph, op = jd._jit_merge_point_pos
         greens_v, reds_v = support.decode_hp_hint_args(op)
         ALLARGS = [v.concretetype for v in (greens_v + reds_v)]
         jd._green_args_spec = [v.concretetype for v in greens_v]
@@ -467,26 +484,37 @@
             [lltype.Signed, llmemory.GCREF], RESTYPE)
 
     def rewrite_can_enter_jits(self):
-        can_enter_jits = find_can_enter_jit(self.translator.graphs)
         sublists = {}
         for jd in self.jitdrivers_sd:
-            sublists[jd.jitdriver] = []
+            sublists[jd.jitdriver] = jd, []
+            jd.no_loop_header = True
+        #
+        loop_headers = find_loop_headers(self.translator.graphs)
+        for graph, block, index in loop_headers:
+            op = block.operations[index]
+            jitdriver = op.args[1].value
+            assert jitdriver in sublists, \
+                   "loop_header with no matching jit_merge_point"
+            jd, sublist = sublists[jitdriver]
+            jd.no_loop_header = False
+        #
+        can_enter_jits = find_can_enter_jit(self.translator.graphs)
         for graph, block, index in can_enter_jits:
             op = block.operations[index]
             jitdriver = op.args[1].value
             assert jitdriver in sublists, \
                    "can_enter_jit with no matching jit_merge_point"
+            jd, sublist = sublists[jitdriver]
             origportalgraph = jd._jit_merge_point_pos[0]
             if graph is not origportalgraph:
-                sublists[jitdriver].append((graph, block, index))
+                sublist.append((graph, block, index))
+                jd.no_loop_header = False
             else:
                 pass   # a 'can_enter_jit' before the 'jit-merge_point', but
                        # originally in the same function: we ignore it here
                        # see e.g. test_jitdriver.test_simple
         for jd in self.jitdrivers_sd:
-            sublist = sublists[jd.jitdriver]
-            assert len(sublist) > 0, \
-                   "found no can_enter_jit for %r" % (jd.jitdriver,)
+            _, sublist = sublists[jd.jitdriver]
             self.rewrite_can_enter_jit(jd, sublist)
 
     def rewrite_can_enter_jit(self, jd, can_enter_jits):
@@ -494,6 +522,19 @@
         FUNCPTR = jd._PTR_JIT_ENTER_FUNCTYPE
         jit_enter_fnptr = self.helper_func(FUNCPTR, jd._maybe_enter_jit_fn)
 
+        if len(can_enter_jits) == 0:
+            # see test_warmspot.test_no_loop_at_all
+            operations = jd.portal_graph.startblock.operations
+            op1 = operations[0]
+            assert (op1.opname == 'jit_marker' and
+                    op1.args[0].value == 'jit_merge_point')
+            op0 = SpaceOperation(
+                'jit_marker',
+                [Constant('can_enter_jit', lltype.Void)] + op1.args[1:],
+                None)
+            operations.insert(0, op0)
+            can_enter_jits = [(jd.portal_graph, jd.portal_graph.startblock, 0)]
+
         for graph, block, index in can_enter_jits:
             if graph is jd._jit_merge_point_pos[0]:
                 continue
@@ -702,8 +743,14 @@
         # ____________________________________________________________
         # Now mutate origportalgraph to end with a call to portal_runner_ptr
         #
-        _, origblock, origindex = jd._jit_merge_point_pos
-        op = origblock.operations[origindex]
+        _, op = jd._jit_merge_point_pos
+        for origblock in origportalgraph.iterblocks():
+            if op in origblock.operations:
+                break
+        else:
+            assert False, "lost the operation %r in the graph %r" % (
+                op, origportalgraph)
+        origindex = origblock.operations.index(op)
         assert op.opname == 'jit_marker'
         assert op.args[0].value == 'jit_merge_point'
         greens_v, reds_v = support.decode_hp_hint_args(op)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py	Sat Oct 30 16:41:17 2010
@@ -1,5 +1,5 @@
 import sys
-from pypy.rpython.lltypesystem import lltype, llmemory, rstr
+from pypy.rpython.lltypesystem import lltype, llmemory, rstr, rffi
 from pypy.rpython.ootypesystem import ootype
 from pypy.rpython.annlowlevel import hlstr, llstr, cast_base_ptr_to_instance
 from pypy.rpython.annlowlevel import cast_object_to_ptr
@@ -24,7 +24,11 @@
     """
     INPUT = lltype.typeOf(x)
     if INPUT is lltype.Signed:
-        return lltype.cast_primitive(TYPE, x)    # XXX missing: Ptr(non-gc)
+        if isinstance(TYPE, lltype.Ptr) and TYPE.TO._gckind == 'raw':
+            # non-gc pointer
+            return rffi.cast(TYPE, x)
+        else:
+            return lltype.cast_primitive(TYPE, x)
     elif INPUT is lltype.Float:
         assert TYPE is lltype.Float
         return x
@@ -172,6 +176,9 @@
             meth(default_value)
 
     def set_param_threshold(self, threshold):
+        if threshold < 0:
+            self.increment_threshold = 0   # never reach the THRESHOLD_LIMIT
+            return
         if threshold < 2:
             threshold = 2
         self.increment_threshold = (self.THRESHOLD_LIMIT // threshold) + 1

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit.py	Sat Oct 30 16:41:17 2010
@@ -40,8 +40,10 @@
 config.objspace.usemodules.array = True
 config.objspace.usemodules._weakref = False
 config.objspace.usemodules._sre = False
+#
+config.objspace.usemodules._ffi = True
+#
 set_pypy_opt_level(config, level='jit')
-config.objspace.std.withinlineddict = True
 
 if BACKEND == 'c':
     config.objspace.std.multimethods = 'mrd'

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py	Sat Oct 30 16:41:17 2010
@@ -16,7 +16,7 @@
     interp.heap.malloc_nonmovable = returns_null     # XXX
 
     from pypy.jit.backend.llgraph.runner import LLtypeCPU
-    LLtypeCPU.supports_floats = False    # for now
+    #LLtypeCPU.supports_floats = False    # for now
     apply_jit(interp, graph, LLtypeCPU)
 
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/spli/interpreter.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/spli/interpreter.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/spli/interpreter.py	Sat Oct 30 16:41:17 2010
@@ -1,12 +1,14 @@
 import os
-from pypy.tool import stdlib_opcode as opcode
+from pypy.tool import stdlib_opcode
 from pypy.jit.tl.spli import objects, pycode
-from pypy.tool.stdlib_opcode import unrolling_opcode_descs
-from pypy.tool.stdlib_opcode import opcode_method_names
 from pypy.rlib.unroll import unrolling_iterable
 from pypy.rlib.jit import JitDriver, hint, dont_look_inside
 from pypy.rlib.objectmodel import we_are_translated
 
+opcode_method_names = stdlib_opcode.host_bytecode_spec.method_names
+unrolling_opcode_descs = unrolling_iterable(
+    stdlib_opcode.host_bytecode_spec.ordered_opdescs)
+HAVE_ARGUMENT = stdlib_opcode.host_HAVE_ARGUMENT
 
 compare_ops = [
     "cmp_lt",   # "<"
@@ -79,7 +81,7 @@
             self.stack_depth = hint(self.stack_depth, promote=True)
             op = ord(code[instr_index])
             instr_index += 1
-            if op >= opcode.HAVE_ARGUMENT:
+            if op >= HAVE_ARGUMENT:
                 low = ord(code[instr_index])
                 hi = ord(code[instr_index + 1])
                 oparg = (hi << 8) | low
@@ -183,6 +185,12 @@
             next_instr += arg
         return next_instr
 
+    def POP_JUMP_IF_FALSE(self, arg, next_instr, code):
+        w_cond = self.pop()
+        if not w_cond.is_true():
+            next_instr = arg
+        return next_instr
+
     def JUMP_FORWARD(self, arg, next_instr, code):
         return next_instr + arg
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/tla/test_tla.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/tla/test_tla.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/tla/test_tla.py	Sat Oct 30 16:41:17 2010
@@ -156,7 +156,7 @@
 # ____________________________________________________________ 
 
 from pypy.jit.metainterp.test.test_basic import LLJitMixin
-from pypy.rlib.jit import OPTIMIZER_FULL
+from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_NO_PERFECTSPEC
 
 class TestLLtype(LLJitMixin):
     def test_loop(self):
@@ -178,5 +178,5 @@
             assert isinstance(w_result, tla.W_IntObject)
             return w_result.intvalue
         res = self.meta_interp(interp_w, [42], listops=True,
-                               optimizer=OPTIMIZER_FULL)
+                               optimizer=OPTIMIZER_NO_PERFECTSPEC)
         assert res == 0

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tool/loopviewer.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tool/loopviewer.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tool/loopviewer.py	Sat Oct 30 16:41:17 2010
@@ -9,7 +9,7 @@
 import optparse
 from pprint import pprint
 from pypy.tool import logparser
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.jit.metainterp.history import ConstInt
 from pypy.rpython.lltypesystem import llmemory, lltype
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tool/showstats.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tool/showstats.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tool/showstats.py	Sat Oct 30 16:41:17 2010
@@ -4,15 +4,15 @@
 import autopath
 import sys, py
 from pypy.tool import logparser
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 from pypy.jit.metainterp.resoperation import rop
 from pypy.rpython.lltypesystem import lltype, llmemory
 
 def main(argv):
     log = logparser.parse_log_file(argv[0])
-    parts = logparser.extract_category(log, "jit-log-opt-")
+    parts = logparser.extract_category(log, "jit-log-noopt-")
     for i, oplist in enumerate(parts):
-        loop = parse(oplist, no_namespace=True)
+        loop = parse(oplist, no_namespace=True, nonstrict=True)
         num_ops = 0
         num_dmp = 0
         num_guards = 0
@@ -23,7 +23,10 @@
                 num_ops += 1
             if op.is_guard():
                 num_guards += 1
-        print "Loop #%d, length: %d, opcodes: %d, guards: %d, %f" % (i, num_ops, num_dmp, num_guards, num_ops/num_dmp)
+        if num_dmp == 0:
+            print "Loop #%d, length: %d, opcodes: %d, guards: %d" % (i, num_ops, num_dmp, num_guards)
+        else:
+            print "Loop #%d, length: %d, opcodes: %d, guards: %d, %f" % (i, num_ops, num_dmp, num_guards, num_ops/num_dmp)
 
 if __name__ == '__main__':
     main(sys.argv[1:])

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tool/traceviewer.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tool/traceviewer.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tool/traceviewer.py	Sat Oct 30 16:41:17 2010
@@ -253,10 +253,10 @@
 def main(loopfile, use_threshold, view=True):
     countname = py.path.local(loopfile + '.count')
     if countname.check():
-        counts = [re.split('( 20 and use_threshold:
             counts.threshold = l[-20]

Modified: pypy/branch/jit-unroll-loops/pypy/module/__builtin__/interp_classobj.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/__builtin__/interp_classobj.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/__builtin__/interp_classobj.py	Sat Oct 30 16:41:17 2010
@@ -2,9 +2,11 @@
 from pypy.interpreter.error import OperationError, operationerrfmt
 from pypy.interpreter.gateway import ObjSpace, W_Root, NoneNotWrapped, applevel
 from pypy.interpreter.gateway import interp2app, ObjSpace
-from pypy.interpreter.typedef import TypeDef, make_weakref_descr
+from pypy.interpreter.typedef import TypeDef
 from pypy.interpreter.argument import Arguments
 from pypy.interpreter.baseobjspace import Wrappable
+from pypy.interpreter.typedef import GetSetProperty, descr_get_dict
+from pypy.interpreter.typedef import descr_set_dict
 from pypy.rlib.rarithmetic import r_uint, intmask
 from pypy.rlib.objectmodel import compute_identity_hash
 from pypy.rlib.debug import make_sure_not_resized
@@ -57,6 +59,14 @@
         self.bases_w = bases
         self.w_dict = w_dict
  
+    def instantiate(self, space):
+        cache = space.fromcache(Cache)
+        if self.lookup(space, '__del__') is not None:
+            w_inst = cache.cls_with_del(space, self)
+        else:
+            w_inst = cache.cls_without_del(space, self)
+        return w_inst
+
     def getdict(self):
         return self.w_dict
 
@@ -100,15 +110,15 @@
         return False
 
     @jit.unroll_safe
-    def lookup(self, space, w_attr):
+    def lookup(self, space, attr):
         # returns w_value or interplevel None
-        w_result = space.finditem(self.w_dict, w_attr)
+        w_result = space.finditem_str(self.w_dict, attr)
         if w_result is not None:
             return w_result
         for base in self.bases_w:
             # XXX fix annotation of bases_w to be a list of W_ClassObjects
             assert isinstance(base, W_ClassObject)
-            w_result = base.lookup(space, w_attr)
+            w_result = base.lookup(space, attr)
             if w_result is not None:
                 return w_result
         return None
@@ -122,7 +132,7 @@
                 return space.wrap(self.name)
             elif name == "__bases__":
                 return space.newtuple(self.bases_w)
-        w_value = self.lookup(space, w_attr)
+        w_value = self.lookup(space, name)
         if w_value is None:
             raise operationerrfmt(
                 space.w_AttributeError,
@@ -147,7 +157,7 @@
                 self.setbases(space, w_value)
                 return
             elif name == "__del__":
-                if self.lookup(space, w_attr) is None:
+                if self.lookup(space, name) is None:
                     msg = ("a __del__ method added to an existing class "
                            "will not be called")
                     space.warn(msg, space.w_RuntimeWarning)
@@ -195,13 +205,20 @@
         # NOT_RPYTHON
         return '' % self.name
 
+class Cache:
+    def __init__(self, space):
+        from pypy.interpreter.typedef import _usersubclswithfeature
+        # evil
+        self.cls_without_del = _usersubclswithfeature(
+                space.config, W_InstanceObject, "dict", "weakref")
+        self.cls_with_del = _usersubclswithfeature(
+                space.config, self.cls_without_del, "del")
+
+
 def class_descr_call(space, w_self, __args__):
     self = space.interp_w(W_ClassObject, w_self)
-    if self.lookup(space, space.wrap('__del__')) is not None:
-        w_inst = W_InstanceObjectWithDel(space, self)
-    else:
-        w_inst = W_InstanceObject(space, self)
-    w_init = w_inst.getattr_from_class(space, space.wrap('__init__'))
+    w_inst = self.instantiate(space)
+    w_init = w_inst.getattr_from_class(space, '__init__')
     if w_init is not None:
         w_result = space.call_args(w_init, __args__)
         if not space.is_w(w_result, space.w_None):
@@ -234,7 +251,7 @@
 
 def make_unary_instance_method(name):
     def unaryop(self, space):
-        w_meth = self.getattr(space, space.wrap(name), True)
+        w_meth = self.getattr(space, name, True)
         return space.call_function(w_meth)
     unaryop.func_name = name
     return unaryop
@@ -242,7 +259,7 @@
 def make_binary_returning_notimplemented_instance_method(name):
     def binaryop(self, space, w_other):
         try:
-            w_meth = self.getattr(space, space.wrap(name), False)
+            w_meth = self.getattr(space, name, False)
         except OperationError, e:
             if e.match(space, space.w_AttributeError):
                 return space.w_NotImplemented
@@ -267,7 +284,7 @@
             w_a = self
             w_b = w_other
         if w_a is self:
-            w_meth = self.getattr(space, space.wrap(specialname), False)
+            w_meth = self.getattr(space, specialname, False)
             if w_meth is None:
                 return space.w_NotImplemented
             return space.call_function(w_meth, w_b)
@@ -278,7 +295,7 @@
     def rbinaryop(self, space, w_other):
         w_a, w_b = _coerce_helper(space, self, w_other)
         if w_a is None or w_a is self:
-            w_meth = self.getattr(space, space.wrap(rspecialname), False)
+            w_meth = self.getattr(space, rspecialname, False)
             if w_meth is None:
                 return space.w_NotImplemented
             return space.call_function(w_meth, w_other)
@@ -302,46 +319,34 @@
         raise OperationError(
             space.w_TypeError,
             space.wrap("instance() first arg must be class"))
-    if space.is_w(w_dict, space.w_None):
-        w_dict = None
-    elif not space.is_true(space.isinstance(w_dict, space.w_dict)):
-        raise OperationError(
-            space.w_TypeError,
-            space.wrap("instance() second arg must be dictionary or None"))
-    return W_InstanceObject(space, w_class, w_dict)
+    w_result = w_class.instantiate(space)
+    if not space.is_w(w_dict, space.w_None):
+        w_result.setdict(space, w_dict)
+    return w_result
 
 class W_InstanceObject(Wrappable):
-    def __init__(self, space, w_class, w_dict=None):
-        if w_dict is None:
-            w_dict = space.newdict(instance=True)
+    def __init__(self, space, w_class):
+        # note that user_setup is overridden by the typedef.py machinery
+        self.user_setup(space, space.gettypeobject(self.typedef))
         assert isinstance(w_class, W_ClassObject)
         self.w_class = w_class
-        self.w_dict = w_dict
-        self.space = space
-
-    def getdict(self):
-        return self.w_dict
 
-    def setdict(self, space, w_dict):
-        if (w_dict is None or
-            not space.is_true(space.isinstance(w_dict, space.w_dict))):
-            raise OperationError(
-                space.w_TypeError,
-                space.wrap("__dict__ must be a dictionary object"))
-        self.w_dict = w_dict
+    def user_setup(self, space, w_subtype):
+        self.space = space
 
-    def setclass(self, space, w_class):
+    def set_oldstyle_class(self, space, w_class):
         if w_class is None or not isinstance(w_class, W_ClassObject):
             raise OperationError(
                 space.w_TypeError,
                 space.wrap("__class__ must be set to a class"))
         self.w_class = w_class
 
-    def getattr_from_class(self, space, w_name):
+    def getattr_from_class(self, space, name):
         # Look up w_name in the class dict, and call its __get__.
         # This method ignores the instance dict and the __getattr__.
         # Returns None if not found.
-        w_value = self.w_class.lookup(space, w_name)
+        assert isinstance(name, str)
+        w_value = self.w_class.lookup(space, name)
         if w_value is None:
             return None
         w_descr_get = space.lookup(w_value, '__get__')
@@ -349,19 +354,20 @@
             return w_value
         return space.call_function(w_descr_get, w_value, self, self.w_class)
 
-    def getattr(self, space, w_name, exc=True):
+    def getattr(self, space, name, exc=True):
         # Normal getattr rules: look up w_name in the instance dict,
         # in the class dict, and then via a call to __getatttr__.
-        w_result = space.finditem(self.w_dict, w_name)
+        assert isinstance(name, str)
+        w_result = self.getdictvalue(space, name)
         if w_result is not None:
             return w_result
-        w_result = self.getattr_from_class(space, w_name)
+        w_result = self.getattr_from_class(space, name)
         if w_result is not None:
             return w_result
-        w_meth = self.getattr_from_class(space, space.wrap('__getattr__'))
+        w_meth = self.getattr_from_class(space, '__getattr__')
         if w_meth is not None:
             try:
-                return space.call_function(w_meth, w_name)
+                return space.call_function(w_meth, space.wrap(name))
             except OperationError, e:
                 if not exc and e.match(space, space.w_AttributeError):
                     return None     # eat the AttributeError
@@ -371,7 +377,7 @@
             raise operationerrfmt(
                 space.w_AttributeError,
                 "%s instance has no attribute '%s'",
-                self.w_class.name, space.str_w(w_name))
+                self.w_class.name, name)
         else:
             return None
 
@@ -379,24 +385,25 @@
         name = space.str_w(w_attr)
         if len(name) >= 8 and name[0] == '_':
             if name == "__dict__":
-                return self.w_dict
+                return self.getdict()
             elif name == "__class__":
                 return self.w_class
-        return self.getattr(space, w_attr)
+        return self.getattr(space, name)
 
     def descr_setattr(self, space, w_name, w_value):
         name = unwrap_attr(space, w_name)
-        w_meth = self.getattr_from_class(space, space.wrap('__setattr__'))
+        w_meth = self.getattr_from_class(space, '__setattr__')
         if name and name[0] == "_":
             if name == '__dict__':
                 self.setdict(space, w_value)
                 return
             if name == '__class__':
-                self.setclass(space, w_value)
+                self.set_oldstyle_class(space, w_value)
                 return
             if name == '__del__' and w_meth is None:
-                if (not isinstance(self, W_InstanceObjectWithDel)
-                    and space.finditem(self.w_dict, w_name) is None):
+                cache = space.fromcache(Cache)
+                if (not isinstance(self, cache.cls_with_del)
+                    and self.getdictvalue(space, '__del__') is None):
                     msg = ("a __del__ method added to an instance "
                            "with no __del__ in the class will not be called")
                     space.warn(msg, space.w_RuntimeWarning)
@@ -410,13 +417,13 @@
         if name and name[0] == "_":
             if name == '__dict__':
                 # use setdict to raise the error
-                self.setdict(space, None)
+                self.setdict(space, space.w_None)
                 return
             elif name == '__class__':
-                # use setclass to raise the error
-                self.setclass(space, None)
+                # use set_oldstyle_class to raise the error
+                self.set_oldstyle_class(space, None)
                 return
-        w_meth = self.getattr_from_class(space, space.wrap('__delattr__'))
+        w_meth = self.getattr_from_class(space, '__delattr__')
         if w_meth is not None:
             space.call_function(w_meth, w_name)
         else:
@@ -427,7 +434,7 @@
                     self.w_class.name, name)
 
     def descr_repr(self, space):
-        w_meth = self.getattr(space, space.wrap('__repr__'), False)
+        w_meth = self.getattr(space, '__repr__', False)
         if w_meth is None:
             w_class = self.w_class
             mod = w_class.get_module_string(space)
@@ -435,19 +442,19 @@
         return space.call_function(w_meth)
 
     def descr_str(self, space):
-        w_meth = self.getattr(space, space.wrap('__str__'), False)
+        w_meth = self.getattr(space, '__str__', False)
         if w_meth is None:
             return self.descr_repr(space)
         return space.call_function(w_meth)
 
     def descr_unicode(self, space):
-        w_meth = self.getattr(space, space.wrap('__unicode__'), False)
+        w_meth = self.getattr(space, '__unicode__', False)
         if w_meth is None:
             return self.descr_str(space)
         return space.call_function(w_meth)
 
     def descr_len(self, space):
-        w_meth = self.getattr(space, space.wrap('__len__'))
+        w_meth = self.getattr(space, '__len__')
         w_result = space.call_function(w_meth)
         if space.is_true(space.isinstance(w_result, space.w_int)):
             if space.is_true(space.lt(w_result, space.wrap(0))):
@@ -460,22 +467,22 @@
             space.wrap("__len__() should return an int"))
 
     def descr_getitem(self, space, w_key):
-        w_meth = self.getattr(space, space.wrap('__getitem__'))
+        w_meth = self.getattr(space, '__getitem__')
         return space.call_function(w_meth, w_key)
 
     def descr_setitem(self, space, w_key, w_value):
-        w_meth = self.getattr(space, space.wrap('__setitem__'))
+        w_meth = self.getattr(space, '__setitem__')
         space.call_function(w_meth, w_key, w_value)
 
     def descr_delitem(self, space, w_key):
-        w_meth = self.getattr(space, space.wrap('__delitem__'))
+        w_meth = self.getattr(space, '__delitem__')
         space.call_function(w_meth, w_key)
 
     def descr_iter(self, space):
-        w_meth = self.getattr(space, space.wrap('__iter__'), False)
+        w_meth = self.getattr(space, '__iter__', False)
         if w_meth is not None:
             return space.call_function(w_meth)
-        w_meth = self.getattr(space, space.wrap('__getitem__'), False)
+        w_meth = self.getattr(space, '__getitem__', False)
         if w_meth is None:
             raise OperationError(
                 space.w_TypeError,
@@ -485,14 +492,14 @@
     # don't see the point
 
     def descr_getslice(self, space, w_i, w_j):
-        w_meth = self.getattr(space, space.wrap('__getslice__'), False)
+        w_meth = self.getattr(space, '__getslice__', False)
         if w_meth is not None:
             return space.call_function(w_meth, w_i, w_j)
         else:
             return space.getitem(self, space.newslice(w_i, w_j, space.w_None))
 
     def descr_setslice(self, space, w_i, w_j, w_sequence):
-        w_meth = self.getattr(space, space.wrap('__setslice__'), False)
+        w_meth = self.getattr(space, '__setslice__', False)
         if w_meth is not None:
             space.call_function(w_meth, w_i, w_j, w_sequence)
         else:
@@ -500,20 +507,20 @@
                           w_sequence)
 
     def descr_delslice(self, space, w_i, w_j):
-        w_meth = self.getattr(space, space.wrap('__delslice__'), False)
+        w_meth = self.getattr(space, '__delslice__', False)
         if w_meth is not None:
             space.call_function(w_meth, w_i, w_j)
         else:
             return space.delitem(self, space.newslice(w_i, w_j, space.w_None))
 
     def descr_call(self, space, __args__):
-        w_meth = self.getattr(space, space.wrap('__call__'))
+        w_meth = self.getattr(space, '__call__')
         return space.call_args(w_meth, __args__)
 
     def descr_nonzero(self, space):
-        w_func = self.getattr(space, space.wrap('__nonzero__'), False)
+        w_func = self.getattr(space, '__nonzero__', False)
         if w_func is None:
-            w_func = self.getattr(space, space.wrap('__len__'), False)
+            w_func = self.getattr(space, '__len__', False)
             if w_func is None:
                 return space.w_True
         w_result = space.call_function(w_func)
@@ -537,7 +544,7 @@
                 not isinstance(w_b, W_InstanceObject)):
                 return space.cmp(w_a, w_b)
         if isinstance(w_a, W_InstanceObject):
-            w_func = w_a.getattr(space, space.wrap('__cmp__'), False)
+            w_func = w_a.getattr(space, '__cmp__', False)
             if w_func is not None:
                 w_res = space.call_function(w_func, w_b)
                 if space.is_w(w_res, space.w_NotImplemented):
@@ -556,7 +563,7 @@
                     return space.wrap(-1)
                 return space.wrap(0)
         if isinstance(w_b, W_InstanceObject):
-            w_func = w_b.getattr(space, space.wrap('__cmp__'), False)
+            w_func = w_b.getattr(space, '__cmp__', False)
             if w_func is not None:
                 w_res = space.call_function(w_func, w_a)
                 if space.is_w(w_res, space.w_NotImplemented):
@@ -577,10 +584,10 @@
         return space.w_NotImplemented
 
     def descr_hash(self, space):
-        w_func = self.getattr(space, space.wrap('__hash__'), False)
+        w_func = self.getattr(space, '__hash__', False)
         if w_func is None:
-            w_eq =  self.getattr(space, space.wrap('__eq__'), False)
-            w_cmp =  self.getattr(space, space.wrap('__cmp__'), False)
+            w_eq =  self.getattr(space, '__eq__', False)
+            w_cmp =  self.getattr(space, '__cmp__', False)
             if w_eq is not None or w_cmp is not None:
                 raise OperationError(space.w_TypeError,
                                      space.wrap("unhashable instance"))
@@ -595,7 +602,7 @@
         return w_ret
 
     def descr_index(self, space):
-        w_func = self.getattr(space, space.wrap('__index__'), False)
+        w_func = self.getattr(space, '__index__', False)
         if w_func is not None:
             return space.call_function(w_func)
         raise OperationError(
@@ -603,7 +610,7 @@
             space.wrap("object cannot be interpreted as an index"))
 
     def descr_contains(self, space, w_obj):
-        w_func = self.getattr(space, space.wrap('__contains__'), False)
+        w_func = self.getattr(space, '__contains__', False)
         if w_func is not None:
             return space.wrap(space.is_true(space.call_function(w_func, w_obj)))
         # now do it ourselves
@@ -626,7 +633,7 @@
                 w_a = self
                 w_b = w_other
             if w_a is self:
-                w_func = self.getattr(space, space.wrap('__pow__'), False)
+                w_func = self.getattr(space, '__pow__', False)
                 if w_func is not None:
                     return space.call_function(w_func, w_other)
                 return space.w_NotImplemented
@@ -634,7 +641,7 @@
                 return space.pow(w_a, w_b, space.w_None)
         else:
             # CPython also doesn't try coercion in this case
-            w_func = self.getattr(space, space.wrap('__pow__'), False)
+            w_func = self.getattr(space, '__pow__', False)
             if w_func is not None:
                 return space.call_function(w_func, w_other, w_modulo)
             return space.w_NotImplemented
@@ -646,7 +653,7 @@
                 w_a = self
                 w_b = w_other
             if w_a is self:
-                w_func = self.getattr(space, space.wrap('__rpow__'), False)
+                w_func = self.getattr(space, '__rpow__', False)
                 if w_func is not None:
                     return space.call_function(w_func, w_other)
                 return space.w_NotImplemented
@@ -654,13 +661,13 @@
                 return space.pow(w_b, w_a, space.w_None)
         else:
             # CPython also doesn't try coercion in this case
-            w_func = self.getattr(space, space.wrap('__rpow__'), False)
+            w_func = self.getattr(space, '__rpow__', False)
             if w_func is not None:
                 return space.call_function(w_func, w_other, w_modulo)
             return space.w_NotImplemented
 
     def descr_next(self, space):
-        w_func = self.getattr(space, space.wrap('next'), False)
+        w_func = self.getattr(space, 'next', False)
         if w_func is None:
             raise OperationError(space.w_TypeError,
                                  space.wrap("instance has no next() method"))
@@ -669,10 +676,9 @@
     def descr_del(self, space):
         # Note that this is called from executioncontext.UserDelAction
         # via the space.userdel() method.
-        w_name = space.wrap('__del__')
-        w_func = space.finditem(self.w_dict, w_name)
+        w_func = self.getdictvalue(space, '__del__')
         if w_func is None:
-            w_func = self.getattr_from_class(space, w_name)
+            w_func = self.getattr_from_class(space, '__del__')
         if w_func is not None:
             space.call_function(w_func)
 
@@ -717,6 +723,14 @@
         rmeth,
         unwrap_spec=["self", ObjSpace, W_Root])
 
+
+def descr_del_dict(space, w_inst):
+    # use setdict to raise the error
+    w_inst.setdict(space, space.w_None)
+
+dict_descr = GetSetProperty(descr_get_dict, descr_set_dict, descr_del_dict)
+dict_descr.name = '__dict__'
+
 W_InstanceObject.typedef = TypeDef("instance",
     __new__ = interp2app(descr_instance_new),
     __getattribute__ = interp2app(W_InstanceObject.descr_getattribute,
@@ -766,12 +780,9 @@
                          unwrap_spec=['self', ObjSpace, W_Root, W_Root]),
     next = interp2app(W_InstanceObject.descr_next,
                       unwrap_spec=['self', ObjSpace]),
-    __weakref__ = make_weakref_descr(W_InstanceObject),
     __del__ = interp2app(W_InstanceObject.descr_del,
                          unwrap_spec=['self', ObjSpace]),
+    __dict__ = dict_descr,
     **rawdict
 )
-
-class W_InstanceObjectWithDel(W_InstanceObject):
-    def __del__(self):
-        self._enqueue_for_destruction(self.space)
+W_InstanceObject.typedef.acceptable_as_base_class = False

Modified: pypy/branch/jit-unroll-loops/pypy/module/__builtin__/test/test_classobj.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/__builtin__/test/test_classobj.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/__builtin__/test/test_classobj.py	Sat Oct 30 16:41:17 2010
@@ -928,29 +928,33 @@
         assert x is b
         assert y == 5
 
-
-class AppTestOldStyleSharing(AppTestOldstyle):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withsharingdict": True})
-        if option.runappdirect:
-            py.test.skip("can only be run on py.py")
-        def is_sharing(space, w_inst):
-            from pypy.objspace.std.sharingdict import SharedDictImplementation
-            w_d = w_inst.getdict()
-            return space.wrap(isinstance(w_d, SharedDictImplementation) and w_d.r_dict_content is None)
-        cls.w_is_sharing = cls.space.wrap(gateway.interp2app(is_sharing))
-
-
-    def test_real_sharing(self):
+    def test_cant_subclass_instance(self):
         class A:
-            def __init__(self):
-                self.x = 42
-        A1, A2, A3 = A(), A(), A()
-        assert self.is_sharing(A3)
-        assert self.is_sharing(A2)
-        assert self.is_sharing(A1)
+            pass
+        try:
+            class B(type(A())):
+                pass
+        except TypeError:
+            pass
+        else:
+            assert 0, "should have raised"
+
+    def test_dict_descriptor(self):
+        import sys
+        if not hasattr(sys, 'pypy_objspaceclass'):
+            skip("on CPython old-style instances don't have a __dict__ descriptor")
+        class A:
+            pass
+        a = A()
+        a.x = 1
+        descr = type(a).__dict__['__dict__']
+        assert descr.__get__(a) == {'x': 1}
+        descr.__set__(a, {'x': 2})
+        assert a.x == 2
+        raises(TypeError, descr.__delete__, a)
+
 
-class AppTestOldStyleModDict(object):
+class AppTestOldStyleClassStrDict(object):
     def setup_class(cls):
         if option.runappdirect:
             py.test.skip("can only be run on py.py")
@@ -966,3 +970,22 @@
             a = 1
             b = 2
         assert self.is_strdict(A)
+
+class AppTestOldStyleMapDict(AppTestOldstyle):
+    def setup_class(cls):
+        cls.space = gettestobjspace(**{"objspace.std.withmapdict": True})
+        if option.runappdirect:
+            py.test.skip("can only be run on py.py")
+        def has_mapdict(space, w_inst):
+            return space.wrap(w_inst._get_mapdict_map() is not None)
+        cls.w_has_mapdict = cls.space.wrap(gateway.interp2app(has_mapdict))
+
+
+    def test_has_mapdict(self):
+        class A:
+            def __init__(self):
+                self.x = 42
+        a = A()
+        assert a.x == 42
+        assert self.has_mapdict(a)
+

Modified: pypy/branch/jit-unroll-loops/pypy/module/_rawffi/__init__.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/_rawffi/__init__.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/_rawffi/__init__.py	Sat Oct 30 16:41:17 2010
@@ -1,5 +1,5 @@
 
-""" Low-level interface to libffi
+""" Low-level interface to clibffi
 """
 
 from pypy.interpreter.mixedmodule import MixedModule
@@ -39,11 +39,11 @@
         if hasattr(interp_rawffi, 'check_HRESULT'):
             Module.interpleveldefs['check_HRESULT'] = 'interp_rawffi.check_HRESULT'
 
-        from pypy.rlib import libffi
+        from pypy.rlib import clibffi
         for name in ['FUNCFLAG_STDCALL', 'FUNCFLAG_CDECL', 'FUNCFLAG_PYTHONAPI',
                      ]:
-            if hasattr(libffi, name):
-                Module.interpleveldefs[name] = "space.wrap(%r)" % getattr(libffi, name)
+            if hasattr(clibffi, name):
+                Module.interpleveldefs[name] = "space.wrap(%r)" % getattr(clibffi, name)
                 
         super(Module, cls).buildloaders()
     buildloaders = classmethod(buildloaders)

Modified: pypy/branch/jit-unroll-loops/pypy/module/_rawffi/array.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/_rawffi/array.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/_rawffi/array.py	Sat Oct 30 16:41:17 2010
@@ -97,7 +97,15 @@
 
 class W_ArrayInstance(W_DataInstance):
     def __init__(self, space, shape, length, address=r_uint(0)):
-        W_DataInstance.__init__(self, space, shape.size * length, address)
+        # Workaround for a strange behavior of libffi: make sure that
+        # we always have at least 8 bytes.  For W_ArrayInstances that are
+        # used as the result value of a function call, ffi_call() writes
+        # 8 bytes into it even if the function's result type asks for less.
+        # This strange behavior is documented.
+        memsize = shape.size * length
+        if memsize < 8:
+            memsize = 8
+        W_DataInstance.__init__(self, space, memsize, address)
         self.length = length
         self.shape = shape
 

Modified: pypy/branch/jit-unroll-loops/pypy/module/_rawffi/callback.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/_rawffi/callback.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/_rawffi/callback.py	Sat Oct 30 16:41:17 2010
@@ -8,8 +8,8 @@
 from pypy.module._rawffi.array import get_elem, push_elem
 from pypy.module._rawffi.interp_rawffi import W_DataInstance, letter2tp, \
      wrap_value, unwrap_value, unwrap_truncate_int
-from pypy.rlib.libffi import USERDATA_P, CallbackFuncPtr, FUNCFLAG_CDECL
-from pypy.rlib.libffi import ffi_type_void
+from pypy.rlib.clibffi import USERDATA_P, CallbackFuncPtr, FUNCFLAG_CDECL
+from pypy.rlib.clibffi import ffi_type_void
 from pypy.module._rawffi.tracker import tracker
 from pypy.interpreter.error import OperationError
 from pypy.interpreter import gateway

Modified: pypy/branch/jit-unroll-loops/pypy/module/_rawffi/interp_rawffi.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/_rawffi/interp_rawffi.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/_rawffi/interp_rawffi.py	Sat Oct 30 16:41:17 2010
@@ -5,7 +5,7 @@
 from pypy.interpreter.gateway import interp2app, NoneNotWrapped
 from pypy.interpreter.typedef import TypeDef, GetSetProperty
 
-from pypy.rlib.libffi import *
+from pypy.rlib.clibffi import *
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.rlib.unroll import unrolling_iterable
 

Modified: pypy/branch/jit-unroll-loops/pypy/module/_rawffi/structure.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/_rawffi/structure.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/_rawffi/structure.py	Sat Oct 30 16:41:17 2010
@@ -15,7 +15,7 @@
 from pypy.module._rawffi.interp_rawffi import wrap_value, unwrap_value
 from pypy.module._rawffi.interp_rawffi import unpack_shape_with_length
 from pypy.module._rawffi.interp_rawffi import size_alignment
-from pypy.rlib import libffi
+from pypy.rlib import clibffi
 from pypy.rlib.rarithmetic import intmask, r_uint
 
 def unpack_fields(space, w_fields):
@@ -34,7 +34,7 @@
 def round_up(size, alignment):
     return (size + alignment - 1) & -alignment
 
-def size_alignment_pos(fields):
+def size_alignment_pos(fields, is_union=False):
     size = 0
     alignment = 1
     pos = []
@@ -42,16 +42,20 @@
         # fieldtype is a W_Array
         fieldsize = fieldtype.size
         fieldalignment = fieldtype.alignment
-        size = round_up(size, fieldalignment)
         alignment = max(alignment, fieldalignment)
-        pos.append(size)
-        size += intmask(fieldsize)
+        if is_union:
+            pos.append(0)
+            size = max(size, fieldsize)
+        else:
+            size = round_up(size, fieldalignment)
+            pos.append(size)
+            size += intmask(fieldsize)
     size = round_up(size, alignment)
     return size, alignment, pos
 
 
 class W_Structure(W_DataShape):
-    def __init__(self, space, fields, size, alignment):
+    def __init__(self, space, fields, size, alignment, is_union=False):
         name_to_index = {}
         if fields is not None:
             for i in range(len(fields)):
@@ -60,7 +64,7 @@
                     raise operationerrfmt(space.w_ValueError,
                         "duplicate field name %s", name)
                 name_to_index[name] = i
-            size, alignment, pos = size_alignment_pos(fields)
+            size, alignment, pos = size_alignment_pos(fields, is_union)
         else: # opaque case
             fields = []
             pos = []
@@ -104,14 +108,14 @@
     descr_fieldoffset.unwrap_spec = ['self', ObjSpace, str]
 
     # get the corresponding ffi_type
-    ffi_struct = lltype.nullptr(libffi.FFI_STRUCT_P.TO)
+    ffi_struct = lltype.nullptr(clibffi.FFI_STRUCT_P.TO)
 
     def get_basic_ffi_type(self):
         if not self.ffi_struct:
             # Repeated fields are delicate.  Consider for example
             #     struct { int a[5]; }
             # or  struct { struct {int x;} a[5]; }
-            # Seeing no corresponding doc in libffi, let's just repeat
+            # Seeing no corresponding doc in clibffi, let's just repeat
             # the field 5 times...
             fieldtypes = []
             for name, tp in self.fields:
@@ -122,7 +126,7 @@
                 while count + basic_size <= total_size:
                     fieldtypes.append(basic_ffi_type)
                     count += basic_size
-            self.ffi_struct = libffi.make_struct_ffitype_e(self.size,
+            self.ffi_struct = clibffi.make_struct_ffitype_e(self.size,
                                                            self.alignment,
                                                            fieldtypes)
         return self.ffi_struct.ffistruct
@@ -133,15 +137,17 @@
     
 
 
-def descr_new_structure(space, w_type, w_shapeinfo):
+def descr_new_structure(space, w_type, w_shapeinfo, union=0):
+    is_union = bool(union)
     if space.is_true(space.isinstance(w_shapeinfo, space.w_tuple)):
         w_size, w_alignment = space.fixedview(w_shapeinfo, expected_length=2)
         S = W_Structure(space, None, space.int_w(w_size),
-                                     space.int_w(w_alignment))
+                                     space.int_w(w_alignment), is_union)
     else:
         fields = unpack_fields(space, w_shapeinfo)
-        S = W_Structure(space, fields, 0, 0)
+        S = W_Structure(space, fields, 0, 0, is_union)
     return space.wrap(S)
+descr_new_structure.unwrap_spec = [ObjSpace, W_Root, W_Root, int]
 
 W_Structure.typedef = TypeDef(
     'Structure',

Modified: pypy/branch/jit-unroll-loops/pypy/module/_rawffi/test/test__rawffi.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/_rawffi/test/test__rawffi.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/_rawffi/test/test__rawffi.py	Sat Oct 30 16:41:17 2010
@@ -70,7 +70,7 @@
            return s[num];
         }
 
-        char *char_check(char x, char y)
+        const char *char_check(char x, char y)
         {
            if (y == static_str[0])
               return static_str;
@@ -185,13 +185,14 @@
                      sum_x_y
                      give perturb get_s2a check_s2a
                      AAA_first_ordinal_function
+                     ret_un_func
                   """.split()
         eci = ExternalCompilationInfo(export_symbols=symbols)
         return str(platform.compile([c_file], eci, 'x', standalone=False))
     prepare_c_example = staticmethod(prepare_c_example)
     
     def setup_class(cls):
-        from pypy.rlib.libffi import get_libc_name
+        from pypy.rlib.clibffi import get_libc_name
         space = gettestobjspace(usemodules=('_rawffi', 'struct'))
         cls.space = space
         cls.w_lib_name = space.wrap(cls.prepare_c_example())
@@ -945,14 +946,15 @@
         assert a[4] == 't'
 
     def test_union(self):
-        skip("segfaulting")
         import _rawffi
         longsize = _rawffi.sizeof('l')
-        S = _rawffi.Structure((longsize, longsize))
+        S = _rawffi.Structure([('x', 'h'), ('y', 'l')], union=True)
         s = S(autofree=False)
+        s.x = 12345
         lib = _rawffi.CDLL(self.lib_name)
         f = lib.ptr('ret_un_func', [(S, 1)], (S, 1))
         ret = f(s)
+        assert ret.y == 1234500, "ret.y == %d" % (ret.y,)
         s.free()
 
 class AppTestAutoFree:

Modified: pypy/branch/jit-unroll-loops/pypy/module/_socket/test/test_sock_app.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/_socket/test/test_sock_app.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/_socket/test/test_sock_app.py	Sat Oct 30 16:41:17 2010
@@ -254,6 +254,7 @@
     s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
     fd = s.fileno()
     w_obj = rsocket.make_address(c_addr, addrlen).as_object(fd, space)
+    lltype.free(c_addr_ll, flavor='raw')
     assert space.is_true(space.eq(w_obj, space.newtuple([
         space.wrap('lo'),
         space.wrap(socket.ntohs(8)),

Modified: pypy/branch/jit-unroll-loops/pypy/module/_sre/interp_sre.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/_sre/interp_sre.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/_sre/interp_sre.py	Sat Oct 30 16:41:17 2010
@@ -290,10 +290,8 @@
 def SRE_Pattern__new__(space, w_subtype, w_pattern, flags, w_code,
               groups=0, w_groupindex=None, w_indexgroup=None):
     n = space.int_w(space.len(w_code))
-    code = [0] * n
-    for i in range(n):
-        x = space.uint_w(space.getitem(w_code, space.wrap(i)))
-        code[i] = intmask(x)
+    code = [intmask(space.uint_w(space.getitem(w_code, space.wrap(i))))
+            for i in range(n)]
     #
     w_srepat = space.allocate_instance(W_SRE_Pattern, w_subtype)
     srepat = space.interp_w(W_SRE_Pattern, w_srepat)

Modified: pypy/branch/jit-unroll-loops/pypy/module/_weakref/interp__weakref.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/_weakref/interp__weakref.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/_weakref/interp__weakref.py	Sat Oct 30 16:41:17 2010
@@ -7,7 +7,7 @@
 import weakref
 
 
-class WeakrefLifeline(object):
+class WeakrefLifeline(W_Root):
     def __init__(self, space):
         self.space = space       # this is here for W_Root.clear_all_weakrefs()
         self.refs_weak = []

Modified: pypy/branch/jit-unroll-loops/pypy/module/_winreg/interp_winreg.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/_winreg/interp_winreg.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/_winreg/interp_winreg.py	Sat Oct 30 16:41:17 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 from pypy.interpreter.baseobjspace import Wrappable
 from pypy.interpreter.baseobjspace import ObjSpace, W_Root
 from pypy.interpreter.gateway import interp2app
@@ -212,13 +213,10 @@
         subkey = None
     else:
         subkey = space.str_w(w_subkey)
-    dataptr = rffi.str2charp(value)
-    try:
+    with rffi.scoped_str2charp(value) as dataptr:
         ret = rwinreg.RegSetValue(hkey, subkey, rwinreg.REG_SZ, dataptr, len(value))
-    finally:
-        rffi.free_charp(dataptr)
-    if ret != 0:
-        raiseWindowsError(space, ret, 'RegSetValue')
+        if ret != 0:
+            raiseWindowsError(space, ret, 'RegSetValue')
 SetValue.unwrap_spec = [ObjSpace, W_Root, W_Root, int, str]
 
 def QueryValue(space, w_hkey, w_subkey):
@@ -237,23 +235,15 @@
         subkey = None
     else:
         subkey = space.str_w(w_subkey)
-    bufsize_p = lltype.malloc(rwin32.PLONG.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwin32.PLONG.TO, 1) as bufsize_p:
         ret = rwinreg.RegQueryValue(hkey, subkey, None, bufsize_p)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegQueryValue')
-        buf = lltype.malloc(rffi.CCHARP.TO, bufsize_p[0], flavor='raw')
-        try:
+        with lltype.scoped_alloc(rffi.CCHARP.TO, bufsize_p[0]) as buf:
             ret = rwinreg.RegQueryValue(hkey, subkey, buf, bufsize_p)
             if ret != 0:
                 raiseWindowsError(space, ret, 'RegQueryValue')
             return space.wrap(rffi.charp2strn(buf, bufsize_p[0] - 1))
-        finally:
-            lltype.free(buf, flavor='raw')
-    finally:
-        lltype.free(bufsize_p, flavor='raw')
-    if ret != 0:
-        raiseWindowsError(space, ret, 'RegQueryValue')
 QueryValue.unwrap_spec = [ObjSpace, W_Root, W_Root]
 
 def convert_to_regdata(space, w_value, typ):
@@ -412,16 +402,14 @@
 value_name is a string indicating the value to query"""
     hkey = hkey_w(w_hkey, space)
     null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
-    retDataSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retDataSize:
         ret = rwinreg.RegQueryValueEx(hkey, subkey, null_dword, null_dword,
                                       None, retDataSize)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegQueryValueEx')
-        databuf = lltype.malloc(rffi.CCHARP.TO, retDataSize[0], flavor='raw')
-        try:
-            retType = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-            try:
+
+        with lltype.scoped_alloc(rffi.CCHARP.TO, retDataSize[0]) as databuf:
+            with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retType:
 
                 ret = rwinreg.RegQueryValueEx(hkey, subkey, null_dword,
                                               retType, databuf, retDataSize)
@@ -432,12 +420,6 @@
                                          retDataSize[0], retType[0]),
                     space.wrap(retType[0]),
                     ])
-            finally:
-                lltype.free(retType, flavor='raw')
-        finally:
-            lltype.free(databuf, flavor='raw')
-    finally:
-        lltype.free(retDataSize, flavor='raw')
 
 QueryValueEx.unwrap_spec = [ObjSpace, W_Root, str]
 
@@ -454,14 +436,11 @@
 The return value is the handle of the opened key.
 If the function fails, an exception is raised."""
     hkey = hkey_w(w_hkey, space)
-    rethkey = lltype.malloc(rwinreg.PHKEY.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwinreg.PHKEY.TO, 1) as rethkey:
         ret = rwinreg.RegCreateKey(hkey, subkey, rethkey)
         if ret != 0:
             raiseWindowsError(space, ret, 'CreateKey')
         return space.wrap(W_HKEY(rethkey[0]))
-    finally:
-        lltype.free(rethkey, flavor='raw')
 CreateKey.unwrap_spec = [ObjSpace, W_Root, str]
 
 def DeleteKey(space, w_hkey, subkey):
@@ -504,14 +483,11 @@
 The result is a new handle to the specified key
 If the function fails, an EnvironmentError exception is raised."""
     hkey = hkey_w(w_hkey, space)
-    rethkey = lltype.malloc(rwinreg.PHKEY.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwinreg.PHKEY.TO, 1) as rethkey:
         ret = rwinreg.RegOpenKeyEx(hkey, subkey, res, sam, rethkey)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegOpenKeyEx')
         return space.wrap(W_HKEY(rethkey[0]))
-    finally:
-        lltype.free(rethkey, flavor='raw')
 OpenKey.unwrap_spec = [ObjSpace, W_Root, str, int, rffi.r_uint]
 
 def EnumValue(space, w_hkey, index):
@@ -531,10 +507,8 @@
     hkey = hkey_w(w_hkey, space)
     null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
 
-    retValueSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-    try:
-        retDataSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-        try:
+    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retValueSize:
+        with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retDataSize:
             ret = rwinreg.RegQueryInfoKey(
                 hkey, None, null_dword, null_dword,
                 null_dword, null_dword, null_dword,
@@ -546,14 +520,9 @@
             retValueSize[0] += 1
             retDataSize[0] += 1
 
-            valuebuf = lltype.malloc(rffi.CCHARP.TO, retValueSize[0],
-                                     flavor='raw')
-            try:
-                databuf = lltype.malloc(rffi.CCHARP.TO, retDataSize[0],
-                                        flavor='raw')
-                try:
-                    retType = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-                    try:
+            with lltype.scoped_alloc(rffi.CCHARP.TO, retValueSize[0]) as valuebuf:
+                with lltype.scoped_alloc(rffi.CCHARP.TO, retDataSize[0]) as databuf:
+                    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retType:
                         ret = rwinreg.RegEnumValue(
                             hkey, index, valuebuf, retValueSize,
                             null_dword, retType, databuf, retDataSize)
@@ -566,16 +535,6 @@
                                                  retDataSize[0], retType[0]),
                             space.wrap(retType[0]),
                             ])
-                    finally:
-                        lltype.free(retType, flavor='raw')
-                finally:
-                    lltype.free(databuf, flavor='raw')
-            finally:
-                lltype.free(valuebuf, flavor='raw')
-        finally:
-            lltype.free(retDataSize, flavor='raw')
-    finally:
-        lltype.free(retValueSize, flavor='raw')
 
 EnumValue.unwrap_spec = [ObjSpace, W_Root, int]
 
@@ -592,10 +551,8 @@
     null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
 
     # max key name length is 255
-    buf = lltype.malloc(rffi.CCHARP.TO, 256, flavor='raw')
-    try:
-        retValueSize = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-        try:
+    with lltype.scoped_alloc(rffi.CCHARP.TO, 256) as buf:
+        with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as retValueSize:
             retValueSize[0] = 256 # includes NULL terminator
             ret = rwinreg.RegEnumKeyEx(hkey, index, buf, retValueSize,
                                        null_dword, None, null_dword,
@@ -603,10 +560,6 @@
             if ret != 0:
                 raiseWindowsError(space, ret, 'RegEnumKeyEx')
             return space.wrap(rffi.charp2str(buf))
-        finally:
-            lltype.free(retValueSize, flavor='raw')
-    finally:
-        lltype.free(buf, flavor='raw')
 
 EnumKey.unwrap_spec = [ObjSpace, W_Root, int]
 
@@ -621,12 +574,9 @@
 A long integer that identifies when the key was last modified (if available)
  as 100's of nanoseconds since Jan 1, 1600."""
     hkey = hkey_w(w_hkey, space)
-    nSubKeys = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-    try:
-        nValues = lltype.malloc(rwin32.LPDWORD.TO, 1, flavor='raw')
-        try:
-            ft = lltype.malloc(rwin32.PFILETIME.TO, 1, flavor='raw')
-            try:
+    with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as nSubKeys:
+        with lltype.scoped_alloc(rwin32.LPDWORD.TO, 1) as nValues:
+            with lltype.scoped_alloc(rwin32.PFILETIME.TO, 1) as ft:
                 null_dword = lltype.nullptr(rwin32.LPDWORD.TO)
                 ret = rwinreg.RegQueryInfoKey(
                     hkey, None, null_dword, null_dword,
@@ -640,12 +590,6 @@
                 return space.newtuple([space.wrap(nSubKeys[0]),
                                        space.wrap(nValues[0]),
                                        space.wrap(l)])
-            finally:
-                lltype.free(ft, flavor='raw')
-        finally:
-            lltype.free(nValues, flavor='raw')
-    finally:
-        lltype.free(nSubKeys, flavor='raw')
 QueryInfoKey.unwrap_spec = [ObjSpace, W_Root]
 
 def str_or_None_w(space, w_obj):
@@ -666,12 +610,9 @@
 If the function fails, an EnvironmentError exception is raised."""
     machine = str_or_None_w(space, w_machine)
     hkey = hkey_w(w_hkey, space)
-    rethkey = lltype.malloc(rwinreg.PHKEY.TO, 1, flavor='raw')
-    try:
+    with lltype.scoped_alloc(rwinreg.PHKEY.TO, 1) as rethkey:
         ret = rwinreg.RegConnectRegistry(machine, hkey, rethkey)
         if ret != 0:
             raiseWindowsError(space, ret, 'RegConnectRegistry')
         return space.wrap(W_HKEY(rethkey[0]))
-    finally:
-        lltype.free(rethkey, flavor='raw')
 ConnectRegistry.unwrap_spec = [ObjSpace, W_Root, W_Root]

Modified: pypy/branch/jit-unroll-loops/pypy/module/array/interp_array.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/array/interp_array.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/array/interp_array.py	Sat Oct 30 16:41:17 2010
@@ -192,32 +192,30 @@
                           mytype.bytes
                     raise OperationError(space.w_OverflowError,
                                          space.wrap(msg))
-            elif mytype.unwrap == 'str_w' or mytype.unwrap == 'unicode_w':
+                return rffi.cast(mytype.itemtype, item)
+            if mytype.unwrap == 'str_w' or mytype.unwrap == 'unicode_w':
                 if len(item) != 1:
                     msg = 'array item must be char'
                     raise OperationError(space.w_TypeError, space.wrap(msg))
                 item = item[0]
-
+                return rffi.cast(mytype.itemtype, item)
+            #
+            # "regular" case: it fits in an rpython integer (lltype.Signed)
+            result = rffi.cast(mytype.itemtype, item)
             if mytype.canoverflow:
-                msg = None
-                if mytype.signed:
-                    if item < -1 << (mytype.bytes * 8 - 1):
+                if rffi.cast(lltype.Signed, result) != item:
+                    # overflow.  build the correct message
+                    if item < 0:
                         msg = ('signed %d-byte integer is less than minimum' %
                                mytype.bytes)
-                    elif item > (1 << (mytype.bytes * 8 - 1)) - 1:
+                    else:
                         msg = ('signed %d-byte integer is greater than maximum'
                                % mytype.bytes)
-                else:
-                    if item < 0:
-                        msg = ('unsigned %d-byte integer is less than minimum'
-                               % mytype.bytes)
-                    elif item > (1 << (mytype.bytes * 8)) - 1:
-                        msg = ('unsigned %d-byte integer is greater'
-                               ' than maximum' % mytype.bytes)
-                if msg is not None:
+                    if not mytype.signed:
+                        msg = 'un' + msg      # 'signed' => 'unsigned'
                     raise OperationError(space.w_OverflowError,
                                          space.wrap(msg))
-            return rffi.cast(mytype.itemtype, item)
+            return result
 
         def __del__(self):
             self.setlen(0)

Modified: pypy/branch/jit-unroll-loops/pypy/module/bz2/interp_bz2.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/bz2/interp_bz2.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/bz2/interp_bz2.py	Sat Oct 30 16:41:17 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 from pypy.rpython.tool import rffi_platform as platform
 from pypy.rpython.lltypesystem import rffi
 from pypy.rpython.lltypesystem import lltype
@@ -225,6 +226,11 @@
         if self.current_size > 0:
             rffi.keep_buffer_alive_until_here(self.raw_buf, self.gc_buf)
 
+    def __enter__(self):
+        return self
+    def __exit__(self, *args):
+        self.free()
+
 # ____________________________________________________________
 #
 # Make the BZ2File type by internally inheriting from W_File.
@@ -531,33 +537,30 @@
         if not self.running:
             raise OperationError(self.space.w_ValueError,
                 self.space.wrap("this object was already flushed"))
-        
-        out = OutBuffer(self.bzs)
+
         in_bufsize = datasize
-        in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-        for i in range(datasize):
-            in_buf[i] = data[i]
 
-        try:
-        
-            self.bzs.c_next_in = in_buf
-            rffi.setintfield(self.bzs, 'c_avail_in', in_bufsize)
+        with OutBuffer(self.bzs) as out:
+            with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
 
-            while True:
-                bzerror = BZ2_bzCompress(self.bzs, BZ_RUN)
-                if bzerror != BZ_RUN_OK:
-                    _catch_bz2_error(self.space, bzerror)
+                for i in range(datasize):
+                    in_buf[i] = data[i]
 
-                if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
-                    break
-                elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
-                    out.prepare_next_chunk()
+                self.bzs.c_next_in = in_buf
+                rffi.setintfield(self.bzs, 'c_avail_in', in_bufsize)
 
-            res = out.make_result_string()
-            return self.space.wrap(res)
-        finally:
-            lltype.free(in_buf, flavor='raw')
-            out.free()
+                while True:
+                    bzerror = BZ2_bzCompress(self.bzs, BZ_RUN)
+                    if bzerror != BZ_RUN_OK:
+                        _catch_bz2_error(self.space, bzerror)
+
+                    if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
+                        break
+                    elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
+
+                res = out.make_result_string()
+                return self.space.wrap(res)
 
     compress.unwrap_spec = ['self', 'bufferstr']
     
@@ -566,9 +569,8 @@
             raise OperationError(self.space.w_ValueError,
                 self.space.wrap("this object was already flushed"))
         self.running = False
-        
-        out = OutBuffer(self.bzs)
-        try:
+
+        with OutBuffer(self.bzs) as out:
             while True:
                 bzerror = BZ2_bzCompress(self.bzs, BZ_FINISH)
                 if bzerror == BZ_STREAM_END:
@@ -581,8 +583,6 @@
 
             res = out.make_result_string()
             return self.space.wrap(res)
-        finally:
-            out.free()
     flush.unwrap_spec = ['self']
 
 W_BZ2Compressor.typedef = TypeDef("BZ2Compressor",
@@ -641,38 +641,37 @@
         if not self.running:
             raise OperationError(self.space.w_EOFError,
                 self.space.wrap("end of stream was already found"))
-        
+
         in_bufsize = len(data)
-        in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-        for i in range(in_bufsize):
-            in_buf[i] = data[i]
 
-        out = OutBuffer(self.bzs)
-        try:
+        with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
+            for i in range(in_bufsize):
+                in_buf[i] = data[i]
             self.bzs.c_next_in = in_buf
             rffi.setintfield(self.bzs, 'c_avail_in', in_bufsize)
 
-            while True:
-                bzerror = BZ2_bzDecompress(self.bzs)
-                if bzerror == BZ_STREAM_END:
-                    if rffi.getintfield(self.bzs, 'c_avail_in') != 0:
-                        unused = [self.bzs.c_next_in[i] for i in range(rffi.getintfield(self.bzs, 'c_avail_in'))]
-                        self.unused_data = "".join(unused)
-                    self.running = False
-                    break
-                if bzerror != BZ_OK:
-                    _catch_bz2_error(self.space, bzerror)
-
-                if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
-                    break
-                elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
-                    out.prepare_next_chunk()
+            with OutBuffer(self.bzs) as out:
+                while True:
+                    bzerror = BZ2_bzDecompress(self.bzs)
+                    if bzerror == BZ_STREAM_END:
+                        if rffi.getintfield(self.bzs, 'c_avail_in') != 0:
+                            unused = [self.bzs.c_next_in[i]
+                                      for i in range(
+                                          rffi.getintfield(self.bzs,
+                                                           'c_avail_in'))]
+                            self.unused_data = "".join(unused)
+                        self.running = False
+                        break
+                    if bzerror != BZ_OK:
+                        _catch_bz2_error(self.space, bzerror)
+
+                    if rffi.getintfield(self.bzs, 'c_avail_in') == 0:
+                        break
+                    elif rffi.getintfield(self.bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
 
-            res = out.make_result_string()
-            return self.space.wrap(res)
-        finally:
-            lltype.free(in_buf, flavor='raw')
-            out.free()
+                res = out.make_result_string()
+                return self.space.wrap(res)
 
     decompress.unwrap_spec = ['self', 'bufferstr']
 
@@ -695,43 +694,39 @@
     if compresslevel < 1 or compresslevel > 9:
         raise OperationError(space.w_ValueError,
             space.wrap("compresslevel must be between 1 and 9"))
-            
-    bzs = lltype.malloc(bz_stream.TO, flavor='raw', zero=True)
-    in_bufsize = len(data)
-    # conforming to bz2 manual, this is large enough to fit compressed
-    # data in one shot. We will check it later anyway.
-    out = OutBuffer(bzs, in_bufsize + (in_bufsize / 100 + 1) + 600)
-
-    in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-    for i in range(in_bufsize):
-        in_buf[i] = data[i]
-
-    try:
-        bzs.c_next_in = in_buf
-        rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
 
-        bzerror = BZ2_bzCompressInit(bzs, compresslevel, 0, 0)
-        if bzerror != BZ_OK:
-            _catch_bz2_error(space, bzerror)
+    with lltype.scoped_alloc(bz_stream.TO, zero=True) as bzs:
+        in_bufsize = len(data)
+
+        with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
+            for i in range(in_bufsize):
+                in_buf[i] = data[i]
+            bzs.c_next_in = in_buf
+            rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
+
+            # conforming to bz2 manual, this is large enough to fit compressed
+            # data in one shot. We will check it later anyway.
+            with OutBuffer(bzs,
+                           in_bufsize + (in_bufsize / 100 + 1) + 600) as out:
+
+                bzerror = BZ2_bzCompressInit(bzs, compresslevel, 0, 0)
+                if bzerror != BZ_OK:
+                    _catch_bz2_error(space, bzerror)
 
-        while True:
-            bzerror = BZ2_bzCompress(bzs, BZ_FINISH)
-            if bzerror == BZ_STREAM_END:
-                break
-            elif bzerror != BZ_FINISH_OK:
+                while True:
+                    bzerror = BZ2_bzCompress(bzs, BZ_FINISH)
+                    if bzerror == BZ_STREAM_END:
+                        break
+                    elif bzerror != BZ_FINISH_OK:
+                        BZ2_bzCompressEnd(bzs)
+                        _catch_bz2_error(space, bzerror)
+
+                    if rffi.getintfield(bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
+
+                res = out.make_result_string()
                 BZ2_bzCompressEnd(bzs)
-                _catch_bz2_error(space, bzerror)
-            
-            if rffi.getintfield(bzs, 'c_avail_out') == 0:
-                out.prepare_next_chunk()
-
-        res = out.make_result_string()
-        BZ2_bzCompressEnd(bzs)
-        return space.wrap(res)
-    finally:
-        lltype.free(bzs, flavor='raw')
-        lltype.free(in_buf, flavor='raw')
-        out.free()
+                return space.wrap(res)
 compress.unwrap_spec = [ObjSpace, 'bufferstr', int]
 
 def decompress(space, data):
@@ -744,40 +739,34 @@
     if in_bufsize == 0:
         return space.wrap("")
 
-    bzs = lltype.malloc(bz_stream.TO, flavor='raw', zero=True)
-    in_buf = lltype.malloc(rffi.CCHARP.TO, in_bufsize, flavor='raw')
-    for i in range(in_bufsize):
-        in_buf[i] = data[i]
-
-    out = OutBuffer(bzs)
-    try:
-        bzs.c_next_in = in_buf
-        rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
-    
-        bzerror = BZ2_bzDecompressInit(bzs, 0, 0)
-        if bzerror != BZ_OK:
-            _catch_bz2_error(space, bzerror)
-        
-        while True:
-            bzerror = BZ2_bzDecompress(bzs)
-            if bzerror == BZ_STREAM_END:
-                break
-            if bzerror != BZ_OK:
-                BZ2_bzDecompressEnd(bzs)
-                _catch_bz2_error(space, bzerror)
-        
-            if rffi.getintfield(bzs, 'c_avail_in') == 0:
+    with lltype.scoped_alloc(bz_stream.TO, zero=True) as bzs:
+        with lltype.scoped_alloc(rffi.CCHARP.TO, in_bufsize) as in_buf:
+            for i in range(in_bufsize):
+                in_buf[i] = data[i]
+            bzs.c_next_in = in_buf
+            rffi.setintfield(bzs, 'c_avail_in', in_bufsize)
+
+            with OutBuffer(bzs) as out:
+                bzerror = BZ2_bzDecompressInit(bzs, 0, 0)
+                if bzerror != BZ_OK:
+                    _catch_bz2_error(space, bzerror)
+
+                while True:
+                    bzerror = BZ2_bzDecompress(bzs)
+                    if bzerror == BZ_STREAM_END:
+                        break
+                    if bzerror != BZ_OK:
+                        BZ2_bzDecompressEnd(bzs)
+                    _catch_bz2_error(space, bzerror)
+
+                    if rffi.getintfield(bzs, 'c_avail_in') == 0:
+                        BZ2_bzDecompressEnd(bzs)
+                        raise OperationError(space.w_ValueError, space.wrap(
+                            "couldn't find end of stream"))
+                    elif rffi.getintfield(bzs, 'c_avail_out') == 0:
+                        out.prepare_next_chunk()
+
+                res = out.make_result_string()
                 BZ2_bzDecompressEnd(bzs)
-                raise OperationError(space.w_ValueError,
-                                     space.wrap("couldn't find end of stream"))
-            elif rffi.getintfield(bzs, 'c_avail_out') == 0:
-                out.prepare_next_chunk()
-
-        res = out.make_result_string()
-        BZ2_bzDecompressEnd(bzs)
-        return space.wrap(res)
-    finally:
-        lltype.free(bzs, flavor='raw')
-        lltype.free(in_buf, flavor='raw')
-        out.free()
+                return space.wrap(res)
 decompress.unwrap_spec = [ObjSpace, 'bufferstr']

Modified: pypy/branch/jit-unroll-loops/pypy/module/cpyext/classobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/cpyext/classobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/cpyext/classobject.py	Sat Oct 30 16:41:17 2010
@@ -15,16 +15,20 @@
     class is the class of new object.  The dict parameter will be used as the
     object's __dict__; if NULL, a new dictionary will be created for the
     instance."""
-    if not PyClass_Check(space, w_class):
+    if not isinstance(w_class, W_ClassObject):
         return PyErr_BadInternalCall(space)
-    return W_InstanceObject(space, w_class, w_dict)
+    w_result = w_class.instantiate(space)
+    if w_dict is not None:
+        w_result.setdict(space, w_dict)
+    return w_result
 
 @cpython_api([PyObject, PyObject], PyObject, error=CANNOT_FAIL)
 def _PyInstance_Lookup(space, w_instance, w_name):
+    name = space.str_w(w_name)
     assert isinstance(w_instance, W_InstanceObject)
-    w_result = space.finditem(w_instance.w_dict, w_name)
+    w_result = w_instance.getdictvalue(space, name)
     if w_result is not None:
         return w_result
-    return w_instance.w_class.lookup(space, w_name)
+    return w_instance.w_class.lookup(space, name)
 
 

Modified: pypy/branch/jit-unroll-loops/pypy/module/cpyext/test/test_cpyext.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/cpyext/test/test_cpyext.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/cpyext/test/test_cpyext.py	Sat Oct 30 16:41:17 2010
@@ -16,6 +16,7 @@
 from pypy.module.cpyext.pyobject import Py_DecRef, InvalidPointerException
 from pypy.translator.goal import autopath
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 
 @api.cpython_api([], api.PyObject)
 def PyPy_Crash1(space):
@@ -78,7 +79,6 @@
         self.frozen_refcounts[w_obj] = obj.c_ob_refcnt
     #state.print_refcounts()
     self.frozen_ll2callocations = set(ll2ctypes.ALLOCATED.values())
-    lltype.start_tracking_allocations()
 
 class LeakCheckingTest(object):
     def check_and_print_leaks(self):
@@ -126,17 +126,8 @@
         for w_obj in lost_objects_w:
             print >>sys.stderr, "Lost object %r" % (w_obj, )
             leaking = True
-        for llvalue in set(ll2ctypes.ALLOCATED.values()) - self.frozen_ll2callocations:
-            if getattr(llvalue, "_traceback", None): # this means that the allocation should be tracked
-                leaking = True
-                print >>sys.stderr, "Did not deallocate %r (ll2ctypes)" % (llvalue, )
-                print >>sys.stderr, "\t" + "\n\t".join(llvalue._traceback.splitlines())
-        for llvalue in lltype.ALLOCATED.keys():
-            leaking = True
-            print >>sys.stderr, "Did not deallocate %r (llvalue)" % (llvalue, )
-            print >>sys.stderr, "\t" + "\n\t".join(llvalue._traceback.splitlines())
-
-        lltype.stop_tracking_allocations()
+        # the actual low-level leak checking is done by pypy.tool.leakfinder,
+        # enabled automatically by pypy.conftest.
         return leaking
 
 class AppTestCpythonExtensionBase(LeakCheckingTest):

Modified: pypy/branch/jit-unroll-loops/pypy/module/cpyext/test/test_unicodeobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/cpyext/test/test_unicodeobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/cpyext/test/test_unicodeobject.py	Sat Oct 30 16:41:17 2010
@@ -177,13 +177,14 @@
             encoded_charp = rffi.str2charp(encoded)
             strict_charp = rffi.str2charp("strict")
             if endian is not None:
-                pendian = lltype.malloc(rffi.INTP.TO, 1, flavor='raw')
                 if endian < 0:
-                    pendian[0] = -1
+                    value = -1
                 elif endian > 0:
-                    pendian[0] = 1
+                    value = 1
                 else:
-                    pendian[0] = 0
+                    value = 0
+                pendian = lltype.malloc(rffi.INTP.TO, 1, flavor='raw')
+                pendian[0] = rffi.cast(rffi.INT, value)
             else:
                 pendian = None
 

Modified: pypy/branch/jit-unroll-loops/pypy/module/gc/interp_gc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/gc/interp_gc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/gc/interp_gc.py	Sat Oct 30 16:41:17 2010
@@ -5,6 +5,11 @@
 
 def collect(space):
     "Run a full collection."
+    # First clear the method cache.  See test_gc for an example of why.
+    if space.config.objspace.std.withmethodcache:
+        from pypy.objspace.std.typeobject import MethodCache
+        cache = space.fromcache(MethodCache)
+        cache.clear()
     rgc.collect()
     return space.wrap(0)
     

Modified: pypy/branch/jit-unroll-loops/pypy/module/gc/referents.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/gc/referents.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/gc/referents.py	Sat Oct 30 16:41:17 2010
@@ -15,8 +15,16 @@
 
 def try_cast_gcref_to_w_root(gcref):
     w_obj = rgc.try_cast_gcref_to_instance(W_Root, gcref)
-    if not we_are_translated() and not hasattr(w_obj, 'typedef'):
-        w_obj = None
+    # Ignore the instances of W_Root that are not really valid as Python
+    # objects.  There is e.g. WeakrefLifeline in module/_weakref that
+    # inherits from W_Root for internal reasons.  Such instances don't
+    # have a typedef at all (or have a null typedef after translation).
+    if not we_are_translated():
+        if not hasattr(w_obj, 'typedef'):
+            return None
+    else:
+        if w_obj is None or not w_obj.typedef:
+            return None
     return w_obj
 
 def wrap(space, gcref):

Modified: pypy/branch/jit-unroll-loops/pypy/module/gc/test/test_gc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/gc/test/test_gc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/gc/test/test_gc.py	Sat Oct 30 16:41:17 2010
@@ -103,3 +103,22 @@
         import gc
         gc.dump_heap_stats(self.fname)
 
+
+class AppTestGcMethodCache(object):
+    def setup_class(cls):
+        cls.space = gettestobjspace(**{"objspace.std.withmethodcache": True})
+
+    def test_clear_method_cache(self):
+        import gc, weakref
+        rlist = []
+        def f():
+            class C(object):
+                def f(self):
+                    pass
+            C().f()    # Fill the method cache
+            rlist.append(weakref.ref(C))
+        for i in range(5):
+            f()
+        gc.collect()    # the classes C should all go away here
+        for r in rlist:
+            assert r() is None

Modified: pypy/branch/jit-unroll-loops/pypy/module/imp/importing.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/imp/importing.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/imp/importing.py	Sat Oct 30 16:41:17 2010
@@ -122,7 +122,7 @@
                     n = len(ctxt_name_prefix_parts)-level+1
                     assert n>=0
                     ctxt_name_prefix_parts = ctxt_name_prefix_parts[:n]
-                if ctxt_w_path is None: # plain module
+                if ctxt_name_prefix_parts and ctxt_w_path is None: # plain module
                     ctxt_name_prefix_parts.pop()
                 if ctxt_name_prefix_parts:
                     rel_modulename = '.'.join(ctxt_name_prefix_parts)

Modified: pypy/branch/jit-unroll-loops/pypy/module/imp/test/test_import.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/imp/test/test_import.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/imp/test/test_import.py	Sat Oct 30 16:41:17 2010
@@ -360,6 +360,12 @@
         """.rstrip()
         raises(ValueError, imp)
 
+    def test_future_relative_import_error_when_in_non_package2(self):
+        exec """def imp():
+                    from .. import inpackage
+        """.rstrip()
+        raises(ValueError, imp)
+
     def test_relative_import_with___name__(self):
         import sys
         mydict = {'__name__': 'sys.foo'}
@@ -784,6 +790,7 @@
     extrapath = udir.ensure("pythonpath", dir=1) 
     extrapath.join("urllib.py").write("print 42\n")
     old = os.environ.get('PYTHONPATH', None)
+    oldlang = os.environ.pop('LANG', None)
     try: 
         os.environ['PYTHONPATH'] = str(extrapath)
         output = py.process.cmdexec('''"%s" "%s" -c "import urllib"''' % 
@@ -792,6 +799,8 @@
     finally: 
         if old: 
             os.environ['PYTHONPATH'] = old 
+        if oldlang:
+            os.environ['LANG'] = oldlang
 
 class AppTestImportHooks(object):
     def test_meta_path(self):

Modified: pypy/branch/jit-unroll-loops/pypy/module/parser/test/test_parser.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/parser/test/test_parser.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/parser/test/test_parser.py	Sat Oct 30 16:41:17 2010
@@ -10,6 +10,9 @@
         cls.w_m = space.appexec([], """():
     import parser
     return parser""")
+        cls.w_symbol = space.appexec([], """():
+    import symbol
+    return symbol""")
 
 
 class AppTestParser(ParserModuleTest):
@@ -36,7 +39,7 @@
             seq = getattr(s, meth)()
             assert isinstance(seq, tp)
             assert len(seq) == 4
-            assert seq[0] == 286
+            assert seq[0] == self.symbol.file_input
             assert len(seq[2]) == 2
             assert len(seq[3]) == 2
             assert seq[2][0] == 4

Modified: pypy/branch/jit-unroll-loops/pypy/module/pypyjit/__init__.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/pypyjit/__init__.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/pypyjit/__init__.py	Sat Oct 30 16:41:17 2010
@@ -6,6 +6,7 @@
 
     interpleveldefs = {
         'set_param':    'interp_jit.set_param',
+        'residual_call': 'interp_jit.residual_call',
     }
 
     def setup_after_space_initialization(self):

Modified: pypy/branch/jit-unroll-loops/pypy/module/pypyjit/interp_jit.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/pypyjit/interp_jit.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/pypyjit/interp_jit.py	Sat Oct 30 16:41:17 2010
@@ -5,10 +5,10 @@
 
 from pypy.tool.pairtype import extendabletype
 from pypy.rlib.rarithmetic import r_uint, intmask
-from pypy.rlib.jit import JitDriver, hint, we_are_jitted
+from pypy.rlib.jit import JitDriver, hint, we_are_jitted, dont_look_inside
 import pypy.interpreter.pyopcode   # for side-effects
 from pypy.interpreter.error import OperationError, operationerrfmt
-from pypy.interpreter.gateway import ObjSpace, Arguments
+from pypy.interpreter.gateway import ObjSpace, Arguments, W_Root
 from pypy.interpreter.pycode import PyCode, CO_GENERATOR
 from pypy.interpreter.pyframe import PyFrame
 from pypy.interpreter.pyopcode import ExitFrame
@@ -131,3 +131,10 @@
                                   "no JIT parameter '%s'", key)
 
 set_param.unwrap_spec = [ObjSpace, Arguments]
+
+ at dont_look_inside
+def residual_call(space, w_callable, args):
+    '''For testing.  Invokes callable(...), but without letting
+    the JIT follow the call.'''
+    return space.call_args(w_callable, args)
+residual_call.unwrap_spec = [ObjSpace, W_Root, Arguments]

Modified: pypy/branch/jit-unroll-loops/pypy/module/pypyjit/policy.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/pypyjit/policy.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/pypyjit/policy.py	Sat Oct 30 16:41:17 2010
@@ -11,7 +11,7 @@
         if '.' in modname:
             modname, _ = modname.split('.', 1)
         if modname in ['pypyjit', 'signal', 'micronumpy', 'math', 'exceptions',
-                       'imp', 'sys', 'array']:
+                       'imp', 'sys', 'array', '_ffi']:
             return True
         return False
 

Modified: pypy/branch/jit-unroll-loops/pypy/module/pypyjit/test/test_pypy_c.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/pypyjit/test/test_pypy_c.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/pypyjit/test/test_pypy_c.py	Sat Oct 30 16:41:17 2010
@@ -79,8 +79,11 @@
 
 
 class PyPyCJITTests(object):
-    def run_source(self, source, expected_max_ops, *testcases):
+    def run_source(self, source, expected_max_ops, *testcases, **kwds):
         assert isinstance(expected_max_ops, int)
+        threshold = kwds.pop('threshold', 3)
+        if kwds:
+            raise TypeError, 'Unsupported keyword arguments: %s' % kwds.keys()
         source = py.code.Source(source)
         filepath = self.tmpdir.join('case%d.py' % self.counter)
         logfilepath = filepath.new(ext='.log')
@@ -92,7 +95,7 @@
             import sys
             try: # make the file runnable by CPython
                 import pypyjit
-                pypyjit.set_param(threshold=3)
+                pypyjit.set_param(threshold=%d)
             except ImportError:
                 pass
 
@@ -102,7 +105,7 @@
                 print >> sys.stderr, 'got:', repr(result)
                 assert result == expected
                 assert type(result) is type(expected)
-        """)
+        """ % threshold)
         for testcase in testcases * 2:
             print >> f, "check(%r, %r)" % testcase
         print >> f, "print 'OK :-)'"
@@ -116,6 +119,8 @@
         result = child_stdout.read()
         child_stdout.close()
         assert result
+        if result.strip().startswith('SKIP:'):
+            py.test.skip(result.strip())
         assert result.splitlines()[-1].strip() == 'OK :-)'
         self.parse_loops(logfilepath)
         self.print_loops()
@@ -123,16 +128,18 @@
         if self.total_ops > expected_max_ops:
             assert 0, "too many operations: got %d, expected maximum %d" % (
                 self.total_ops, expected_max_ops)
+        return result
 
     def parse_loops(self, opslogfile):
-        from pypy.jit.metainterp.test.oparser import parse
+        from pypy.jit.tool.oparser import parse
         from pypy.tool import logparser
         assert opslogfile.check()
         log = logparser.parse_log_file(str(opslogfile))
         parts = logparser.extract_category(log, 'jit-log-opt-')
+        self.rawloops = [part for part in parts
+                         if not from_entry_bridge(part, parts)]
         # skip entry bridges, they can contain random things
-        self.loops = [parse(part, no_namespace=True) for part in parts
-                          if not from_entry_bridge(part, parts)]
+        self.loops = [parse(part, no_namespace=True) for part in self.rawloops]
         self.sliced_loops = [] # contains all bytecodes of all loops
         self.total_ops = 0
         for loop in self.loops:
@@ -156,12 +163,11 @@
         return [ops for ops in self.sliced_loops if ops.bytecode == name]
 
     def print_loops(self):
-        for loop in self.loops:
+        for rawloop in self.rawloops:
             print
             print '@' * 79
             print
-            for op in loop.operations:
-                print op
+            print rawloop.rstrip()
         print
         print '@' * 79
 
@@ -272,7 +278,7 @@
         assert len(ops) == 2
         assert not ops[0].get_opnames("call")
         assert not ops[0].get_opnames("new")
-        assert len(ops[0].get_opnames("guard")) <= 7
+        assert len(ops[0].get_opnames("guard")) <= 2
         assert not ops[1] # second LOOKUP_METHOD folded away
 
         ops = self.get_by_bytecode("CALL_METHOD")
@@ -283,12 +289,16 @@
             else:
                 assert not bytecode.get_opnames("call")
             assert not bytecode.get_opnames("new")
-            assert len(bytecode.get_opnames("guard")) <= 9
+            assert len(bytecode.get_opnames("guard")) <= 6
         assert len(ops[1]) < len(ops[0])
 
         ops = self.get_by_bytecode("LOAD_ATTR")
         assert len(ops) == 2
-        assert ops[0].get_opnames() == ["getfield_gc", "getarrayitem_gc",
+        # With mapdict, we get fast access to (so far) the 5 first
+        # attributes, which means it is done with only the following
+        # operations.  (For the other attributes there is additionally
+        # a getarrayitem_gc.)
+        assert ops[0].get_opnames() == ["getfield_gc",
                                         "guard_nonnull_class"]
         assert not ops[1] # second LOAD_ATTR folded away
 
@@ -317,8 +327,8 @@
         assert len(ops) == 2
         assert not ops[0].get_opnames("call")
         assert not ops[0].get_opnames("new")
-        assert len(ops[0].get_opnames("guard")) <= 7
-        assert len(ops[0].get_opnames("getfield")) < 6
+        assert len(ops[0].get_opnames("guard")) <= 2
+        assert len(ops[0].get_opnames("getfield")) <= 4
         assert not ops[1] # second LOOKUP_METHOD folded away
 
     def test_default_and_kw(self):
@@ -382,7 +392,7 @@
                     a.x = 2
                     i = i + a.x
                 return i
-        ''', 67,
+        ''', 69,
                    ([20], 20),
                    ([31], 32))
 
@@ -390,7 +400,7 @@
                 self.get_by_bytecode("CALL_FUNCTION"))
         assert not callA.get_opnames("call")
         assert not callA.get_opnames("new")
-        assert len(callA.get_opnames("guard")) <= 8
+        assert len(callA.get_opnames("guard")) <= 2
         assert not callisinstance1.get_opnames("call")
         assert not callisinstance1.get_opnames("new")
         assert len(callisinstance1.get_opnames("guard")) <= 2
@@ -554,17 +564,13 @@
 
     def test_blockstack_virtualizable(self):
         self.run_source('''
-        def g(k):
-            s = 0
-            for i in range(k, k+2):
-                s += 1
-            return s
+        from pypyjit import residual_call
 
         def main():
             i = 0
             while i < 100:
                 try:
-                    g(i)
+                    residual_call(len, [])
                 except:
                     pass
                 i += 1
@@ -606,16 +612,17 @@
         #     call that can raise is not exchanged into getarrayitem_gc
 
     def test_overflow_checking(self):
+        startvalue = sys.maxint - 2147483647
         self.run_source('''
         def main():
             def f(a,b):
                 if a < 0: return -1
                 return a-b
-            total = 0
+            total = %d
             for i in range(100000):
                 total += f(i, 5)
             return total
-        ''', 170, ([], 4999450000L))
+        ''' % startvalue, 170, ([], startvalue + 4999450000L))
 
     def test_boolrewrite_invers(self):
         for a, b, res, ops in (('2000', '2000', 20001000, 51),
@@ -742,6 +749,8 @@
                     '''%(op1, float(a)/4.0, float(b)/4.0, op2), 109, ([], res))
 
     def test_boolrewrite_ptr(self):
+        # XXX this test is way too imprecise in what it is actually testing
+        # it should count the number of guards instead
         compares = ('a == b', 'b == a', 'a != b', 'b != a', 'a == c', 'c != b')
         for e1 in compares:
             for e2 in compares:
@@ -765,7 +774,7 @@
                 print
                 print 'Test:', e1, e2, n, res
                 self.run_source('''
-                class tst:
+                class tst(object):
                     pass
                 def main():
                     a = tst()
@@ -781,24 +790,6 @@
                     return sa
                 '''%(e1, e2), n, ([], res))
 
-    def test_boolrewrite_ptr_single(self):
-        self.run_source('''
-            class tst:
-                pass
-            def main():
-                a = tst()
-                b = tst()
-                c = tst()
-                sa = 0
-                for i in range(1000):
-                    if a == b: sa += 1
-                    else: sa += 2
-                    if a != b: sa += 10000
-                    else: sa += 20000
-                    if i > 750: a = b
-                return sa
-            ''', 215, ([], 12481752))
-
     def test_array_sum(self):
         for tc, maxops in zip('bhilBHILfd', (38,) * 6 + (40, 40, 41, 38)):
             res = 19352859
@@ -847,7 +838,12 @@
             ''', 65, ([], 122880))
 
     def test_array_intimg(self):
-        for tc, maxops in zip('ilILd', (67, 67, 69, 69, 61)):
+        # XXX this test is way too imprecise in what it is actually testing
+        # it should count the number of guards instead
+        for tc, maxops in zip('ilILd', (67, 67, 70, 70, 61)):
+            print
+            print '='*65
+            print '='*20, 'running test for tc=%r' % (tc,), '='*20
             res = 73574560
             if tc in 'IL':
                 res = long(res)
@@ -1130,6 +1126,44 @@
             return sa
         ''', 88, ([], 1997001))
 
+    def test__ffi_call(self):
+        from pypy.rlib.test.test_libffi import get_libm_name
+        libm_name = get_libm_name(sys.platform)
+        out = self.run_source('''
+        def main():
+            try:
+                from _ffi import CDLL, types
+            except ImportError:
+                sys.stdout.write('SKIP: cannot import _ffi')
+                return 0
+
+            libm = CDLL('%(libm_name)s')
+            pow = libm.getfunc('pow', [types.double, types.double],
+                               types.double)
+            print pow.getaddr()
+            i = 0
+            res = 0
+            while i < 2000:
+                res += pow(2, 3)
+                i += 1
+            return res
+        ''' % locals(),
+                              76, ([], 8.0*2000), threshold=1000)
+        pow_addr = int(out.splitlines()[0])
+        ops = self.get_by_bytecode('CALL_FUNCTION')
+        assert len(ops) == 2 # we get two loops, because of specialization
+        call_function = ops[0]
+        last_ops = [op.getopname() for op in call_function[-5:]]
+        assert last_ops == ['force_token',
+                            'setfield_gc',
+                            'call_may_force',
+                            'guard_not_forced',
+                            'guard_no_exception']
+        call = call_function[-3]
+        assert call.getarg(0).value == pow_addr
+        assert call.getarg(1).value == 2.0
+        assert call.getarg(2).value == 3.0
+
     # test_circular
 
 class AppTestJIT(PyPyCJITTests):
@@ -1155,6 +1189,17 @@
         cls.pypy_c = option.pypy_c
 
 
+def test_interface_residual_call():
+    space = gettestobjspace(usemodules=['pypyjit'])
+    space.appexec([], """():
+        import pypyjit
+        def f(*args, **kwds):
+            return (args, kwds)
+        res = pypyjit.residual_call(f, 4, x=6)
+        assert res == ((4,), {'x': 6})
+    """)
+
+
 def has_info(pypy_c, option):
     g = os.popen('"%s" --info' % pypy_c, 'r')
     lines = g.readlines()

Modified: pypy/branch/jit-unroll-loops/pypy/module/signal/interp_signal.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/signal/interp_signal.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/signal/interp_signal.py	Sat Oct 30 16:41:17 2010
@@ -1,7 +1,6 @@
 from pypy.interpreter.error import OperationError
 from pypy.interpreter.baseobjspace import W_Root, ObjSpace
 from pypy.interpreter.executioncontext import AsyncAction, AbstractActionFlag
-from pypy.rlib.rarithmetic import LONG_BIT, intmask
 import signal as cpy_signal
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.translator.tool.cbuild import ExternalCompilationInfo
@@ -64,8 +63,8 @@
 class CheckSignalAction(AsyncAction):
     """An action that is automatically invoked when a signal is received."""
 
-    # The C-level signal handler sets the highest bit of pypysig_occurred:
-    bitmask = intmask(1 << (LONG_BIT-1))
+    # The C-level signal handler sets the bit 30 of pypysig_occurred:
+    bitmask = 1 << 30
 
     def __init__(self, space):
         AsyncAction.__init__(self, space)

Modified: pypy/branch/jit-unroll-loops/pypy/module/sys/__init__.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/sys/__init__.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/sys/__init__.py	Sat Oct 30 16:41:17 2010
@@ -7,13 +7,15 @@
     """Sys Builtin Module. """
     def __init__(self, space, w_name):
         """NOT_RPYTHON""" # because parent __init__ isn't
+        if space.config.translating:
+            del self.__class__.interpleveldefs['pypy_getudir']
         super(Module, self).__init__(space, w_name) 
         self.checkinterval = 100
         self.recursionlimit = 100
         self.w_default_encoder = None
         self.defaultencoding = "ascii"
         self.filesystemencoding = None
-        
+
     interpleveldefs = {
         '__name__'              : '(space.wrap("sys"))', 
         '__doc__'               : '(space.wrap("PyPy sys module"))', 
@@ -37,7 +39,7 @@
         'argv'                  : 'state.get(space).w_argv', 
         'warnoptions'           : 'state.get(space).w_warnoptions', 
         'builtin_module_names'  : 'state.w_None',
-        'pypy_getudir'          : 'state.pypy_getudir', 
+        'pypy_getudir'          : 'state.pypy_getudir',    # not translated
         'pypy_initial_path'     : 'state.pypy_initial_path',
 
         '_getframe'             : 'vm._getframe', 

Modified: pypy/branch/jit-unroll-loops/pypy/module/sys/state.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/sys/state.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/sys/state.py	Sat Oct 30 16:41:17 2010
@@ -95,15 +95,8 @@
 def getio(space):
     return space.fromcache(IOState)
 
-def _pypy_getudir(space):
-    """NOT_RPYTHON"""
+def pypy_getudir(space):
+    """NOT_RPYTHON
+    (should be removed from interpleveldefs before translation)"""
     from pypy.tool.udir import udir
     return space.wrap(str(udir))
-_pypy_getudir._annspecialcase_ = "override:ignore"
-
-# we need the indirection because this function will live in a dictionary with other 
-# RPYTHON functions and share call sites with them. Better it not be a special-case
-# directly. 
-def pypy_getudir(space):
-    return _pypy_getudir(space)
-

Modified: pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/_ctypes_test.c	Sat Oct 30 16:41:17 2010
@@ -533,3 +533,7 @@
 	return inp;
 }
 
+int my_unused_function(void)
+{
+    return 42;
+}

Modified: pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_funcptr.py	Sat Oct 30 16:41:17 2010
@@ -148,3 +148,7 @@
         # but it segfaults for some reason.
         if sys.platform == 'win32':
             assert f() == 0x12345678
+
+    def test_restype(self):
+        foo = lib.my_unused_function
+        assert foo.restype is c_int     # by default

Modified: pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_functions.py	Sat Oct 30 16:41:17 2010
@@ -29,6 +29,7 @@
 class RECT(Structure):
     _fields_ = [("left", c_int), ("top", c_int),
                 ("right", c_int), ("bottom", c_int)]
+
 class TestFunctions(BaseCTypesTestChecker):
 
     def test_mro(self):
@@ -392,6 +393,18 @@
         result = f("abcd", ord("b"))
         assert result == "bcd"
 
+    def test_caching_bug_1(self):
+        # the same test as test_call_some_args, with two extra lines
+        # in the middle that trigger caching in f._ptr, which then
+        # makes the last two lines fail
+        f = dll.my_strchr
+        f.argtypes = [c_char_p, c_int]
+        f.restype = c_char_p
+        result = f("abcd", ord("b"))
+        assert result == "bcd"
+        result = f("abcd", ord("b"), 42)
+        assert result == "bcd"
+
     def test_sf1651235(self):
         py.test.skip("we are less strict in checking callback parameters")
         # see http://www.python.org/sf/1651235

Modified: pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/test_lib_pypy/ctypes_tests/test_keepalive.py	Sat Oct 30 16:41:17 2010
@@ -10,7 +10,7 @@
     """
     def test_array_of_pointers(self):
         # tests array item assignements & pointer.contents = ...
-        A = POINTER(c_int) * 24
+        A = POINTER(c_long) * 24
         a = A()
         l = c_long(2)
         p = pointer(l)

Modified: pypy/branch/jit-unroll-loops/pypy/module/thread/ll_thread.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/module/thread/ll_thread.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/module/thread/ll_thread.py	Sat Oct 30 16:41:17 2010
@@ -111,7 +111,7 @@
             c_thread_releaselock(self._lock)
 
     def __del__(self):
-        lltype.free(self._lock, flavor='raw')
+        lltype.free(self._lock, flavor='raw', track_allocation=False)
 
 # ____________________________________________________________
 #
@@ -128,10 +128,13 @@
 null_ll_lock = lltype.nullptr(TLOCKP.TO)
 
 def allocate_ll_lock():
-    ll_lock = lltype.malloc(TLOCKP.TO, flavor='raw')
+    # track_allocation=False here; be careful to lltype.free() it.  The
+    # reason it is set to False is that we get it from all app-level
+    # lock objects, as well as from the GIL, which exists at shutdown.
+    ll_lock = lltype.malloc(TLOCKP.TO, flavor='raw', track_allocation=False)
     res = c_thread_lock_init(ll_lock)
     if res == -1:
-        lltype.free(ll_lock, flavor='raw')
+        lltype.free(ll_lock, flavor='raw', track_allocation=False)
         raise error("out of resources")
     return ll_lock
 

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/descroperation.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/descroperation.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/descroperation.py	Sat Oct 30 16:41:17 2010
@@ -51,7 +51,7 @@
     return space.is_w(w_typ1, w_typ2)
 
 
-class Object:
+class Object(object):
     def descr__getattribute__(space, w_obj, w_name):
         name = space.str_w(w_name)
         w_descr = space.lookup(w_obj, name)
@@ -64,9 +64,7 @@
                     w_type = space.type(w_obj)
                     return space.get_and_call_function(w_get, w_descr, w_obj,
                                                        w_type)
-            w_value = w_obj.getdictvalue_attr_is_in_class(space, name)
-        else:
-            w_value = w_obj.getdictvalue(space, name)
+        w_value = w_obj.getdictvalue(space, name)
         if w_value is not None:
             return w_value
         if w_descr is not None:
@@ -76,13 +74,11 @@
     def descr__setattr__(space, w_obj, w_name, w_value):
         name = space.str_w(w_name)
         w_descr = space.lookup(w_obj, name)
-        shadows_type = False
         if w_descr is not None:
             if space.is_data_descr(w_descr):
                 space.set(w_descr, w_obj, w_value)
                 return
-            shadows_type = True
-        if w_obj.setdictvalue(space, name, w_value, shadows_type):
+        if w_obj.setdictvalue(space, name, w_value):
             return
         raiseattrerror(space, w_obj, name, w_descr)
 
@@ -100,7 +96,7 @@
     def descr__init__(space, w_obj, __args__):
         pass
 
-class DescrOperation:
+class DescrOperation(object):
     _mixin_ = True
 
     def is_data_descr(space, w_obj):
@@ -545,7 +541,7 @@
 
 # what is the maximum value slices can get on CPython?
 # we need to stick to that value, because fake.py etc.
-class Temp:
+class Temp(object):
     def __getslice__(self, i, j):
         return j
 slice_max = Temp()[:]

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/flow/flowcontext.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/flow/flowcontext.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/flow/flowcontext.py	Sat Oct 30 16:41:17 2010
@@ -460,7 +460,22 @@
         else:
             v = self.peekvalue(oparg - 1)
         self.space.call_method(v, 'append', w)
-    
+
+    # `with` statement
+
+    def SETUP_WITH(self, offsettoend, next_instr):
+        # A simpler version than the 'real' 2.7 one:
+        # directly call manager.__enter__(), don't use special lookup functions
+        # which don't make sense on the RPython type system.
+        from pypy.interpreter.pyopcode import WithBlock
+        w_manager = self.peekvalue()
+        w_exit = self.space.getattr(w_manager, self.space.wrap("__exit__"))
+        self.settopvalue(w_exit)
+        w_result = self.space.call_method(w_manager, "__enter__")
+        block = WithBlock(self, next_instr + offsettoend)
+        self.append_block(block)
+        self.pushvalue(w_result)
+
     # XXX Unimplemented 2.7 opcodes ----------------
 
     # Set literals, set comprehensions
@@ -476,12 +491,6 @@
     def MAP_ADD(self, oparg, next_instr):
         raise NotImplementedError("MAP_ADD")
 
-    # `with` statement
-
-    def SETUP_WITH(self, oparg, next_instr):
-        raise NotImplementedError("SETUP_WITH")
-
-    
     def make_arguments(self, nargs):
         return ArgumentsForTranslation(self.space, self.peekvalues(nargs))
     def argument_factory(self, *args):
@@ -493,3 +502,12 @@
             raise operr
         return pyframe.PyFrame.handle_operation_error(self, ec, operr,
                                                       *args, **kwds)
+
+    def call_contextmanager_exit_function(self, w_func, w_typ, w_val, w_tb):
+        if w_typ is not self.space.w_None:
+            # The annotator won't allow to merge exception types with None.
+            # Replace it with an object which will break translation when used
+            # (except maybe with 'exc_typ is None')
+            w_typ = self.space.wrap(self.space)
+        return self.space.call_function(w_func, w_typ, w_val, w_tb)
+

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/flow/test/test_objspace.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/flow/test/test_objspace.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/flow/test/test_objspace.py	Sat Oct 30 16:41:17 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 import new
 import py
 from pypy.objspace.flow.model import Constant, Block, Link, Variable, traverse
@@ -828,6 +829,25 @@
         simplify_graph(graph)
         assert self.all_operations(graph) == {'getitem': 1}
 
+    def test_context_manager(self):
+        def f(c, x):
+            with x:
+                pass
+        graph = self.codetest(f)
+        # 2 method calls: x.__enter__() and x.__exit__(None, None, None)
+        assert self.all_operations(graph) == {'getattr': 2,
+                                              'simple_call': 2}
+        #
+        def g(): pass
+        def f(c, x):
+            with x:
+                g()
+        graph = self.codetest(f)
+        assert self.all_operations(graph) == {
+            'getattr': 2,     # __enter__ and __exit__
+            'simple_call': 4, # __enter__, g and 2 possible calls to __exit__
+            'is_true': 1}     # check the result of __exit__()
+
     def monkey_patch_code(self, code, stacksize, flags, codestring, names, varnames):
         c = code
         return new.code(c.co_argcount, c.co_nlocals, stacksize, flags,

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/callmethod.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/callmethod.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/callmethod.py	Sat Oct 30 16:41:17 2010
@@ -44,7 +44,7 @@
         else:
             typ = type(w_descr)
             if typ is function.Function or typ is function.FunctionWithFixedCode:
-                w_value = w_obj.getdictvalue_attr_is_in_class(space, name)
+                w_value = w_obj.getdictvalue(space, name)
                 if w_value is None:
                     # fast method path: a function object in the class,
                     # nothing in the instance
@@ -103,7 +103,7 @@
         w_descr = space.lookup(w_obj, methname)
         typ = type(w_descr)
         if typ is function.Function or typ is function.FunctionWithFixedCode:
-            w_value = w_obj.getdictvalue_attr_is_in_class(space, methname)
+            w_value = w_obj.getdictvalue(space, methname)
             if w_value is None:
                 # fast method path: a function object in the class,
                 # nothing in the instance

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/celldict.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/celldict.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/celldict.py	Sat Oct 30 16:41:17 2010
@@ -45,9 +45,9 @@
         if space.is_w(space.type(w_key), space.w_str):
             self.impl_setitem_str(self.space.str_w(w_key), w_value)
         else:
-            self._as_rdict().setitem(w_key, w_value)
+            self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
-    def impl_setitem_str(self, name, w_value, shadows_type=True):
+    def impl_setitem_str(self, name, w_value):
         self.getcell(name, True).w_value = w_value
 
     def impl_delitem(self, w_key):
@@ -66,7 +66,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
         
     def impl_length(self):
         # inefficient, but do we care?
@@ -85,7 +85,7 @@
         elif _is_sane_hash(space, w_lookup_type):
             return None
         else:
-            return self._as_rdict().getitem(w_lookup)
+            return self._as_rdict().impl_fallback_getitem(w_lookup)
 
     def impl_getitem_str(self, lookup):
         res = self.getcell(lookup, False)

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/dictmultiobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/dictmultiobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/dictmultiobject.py	Sat Oct 30 16:41:17 2010
@@ -49,14 +49,6 @@
         elif space.config.objspace.std.withdictmeasurement:
             assert w_type is None
             return MeasuringDictImplementation(space)
-        elif space.config.objspace.std.withsharingdict and instance:
-            from pypy.objspace.std.sharingdict import SharedDictImplementation
-            assert w_type is None
-            return SharedDictImplementation(space)
-        elif (space.config.objspace.std.withshadowtracking and instance and
-                classofinstance is not None):
-            assert w_type is None
-            return ShadowDetectingDictImplementation(space, classofinstance)
         elif instance or strdict or module:
             assert w_type is None
             return StrDictImplementation(space)
@@ -102,17 +94,17 @@
         else:
             return None
 
-    # _________________________________________________________________ 
+    # _________________________________________________________________
     # implementation methods
     def impl_getitem(self, w_key):
         #return w_value or None
         raise NotImplementedError("abstract base class")
 
-    def impl_getitem_str(self, w_key):
+    def impl_getitem_str(self, key):
         #return w_value or None
         raise NotImplementedError("abstract base class")
 
-    def impl_setitem_str(self,  key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         raise NotImplementedError("abstract base class")
 
     def impl_setitem(self,  w_key, w_value):
@@ -120,7 +112,7 @@
 
     def impl_delitem(self, w_key):
         raise NotImplementedError("abstract base class")
- 
+
     def impl_length(self):
         raise NotImplementedError("abstract base class")
 
@@ -165,12 +157,15 @@
         key = OPTIMIZED_BUILTINS[i]
         return self.impl_getitem_str(key)
 
-    # this method will only be seen whan a certain config option is used
-    def impl_shadows_anything(self):
-        return True
-
-    def impl_set_shadows_anything(self):
-        pass
+    def impl_popitem(self):
+        # default implementation
+        space = self.space
+        iterator = self.impl_iter()
+        w_key, w_value = iterator.next()
+        if w_key is None:
+            raise KeyError
+        self.impl_delitem(w_key)
+        return w_key, w_value
 
     # _________________________________________________________________
     # fallback implementation methods
@@ -178,7 +173,7 @@
     def impl_fallback_setitem(self, w_key, w_value):
         self.r_dict_content[w_key] = w_value
 
-    def impl_fallback_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_fallback_setitem_str(self, key, w_value):
         return self.impl_fallback_setitem(self.space.wrap(key), w_value)
 
     def impl_fallback_delitem(self, w_key):
@@ -211,18 +206,15 @@
         key = OPTIMIZED_BUILTINS[i]
         return self.impl_fallback_getitem_str(key)
 
-    def impl_fallback_shadows_anything(self):
-        return True
-
-    def impl_fallback_set_shadows_anything(self):
-        pass
+    def impl_fallback_popitem(self):
+        return self.r_dict_content.popitem()
 
 
 implementation_methods = [
     ("getitem", 1),
     ("getitem_str", 1),
     ("length", 0),
-    ("setitem_str", 3),
+    ("setitem_str", 2),
     ("setitem", 2),
     ("delitem", 1),
     ("iter", 0),
@@ -231,8 +223,7 @@
     ("keys", 0),
     ("clear", 0),
     ("get_builtin_indexed", 1),
-    ("shadows_anything", 0),
-    ("set_shadows_anything", 0),
+    ("popitem", 0),
 ]
 
 
@@ -310,9 +301,9 @@
         if space.is_w(space.type(w_key), space.w_str):
             self.impl_setitem_str(self.space.str_w(w_key), w_value)
         else:
-            self._as_rdict().setitem(w_key, w_value)
+            self._as_rdict().impl_fallback_setitem(w_key, w_value)
 
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         self.content[key] = w_value
 
     def impl_delitem(self, w_key):
@@ -324,7 +315,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
         
     def impl_length(self):
         return len(self.content)
@@ -344,7 +335,7 @@
         elif _is_sane_hash(space, w_lookup_type):
             return None
         else:
-            return self._as_rdict().getitem(w_key)
+            return self._as_rdict().impl_fallback_getitem(w_key)
 
     def impl_iter(self):
         return StrIteratorImplementation(self.space, self)
@@ -388,47 +379,12 @@
             return None, None
 
 
-class ShadowDetectingDictImplementation(StrDictImplementation):
-    def __init__(self, space, w_type):
-        StrDictImplementation.__init__(self, space)
-        self.w_type = w_type
-        self.original_version_tag = w_type.version_tag()
-        if self.original_version_tag is None:
-            self._shadows_anything = True
-        else:
-            self._shadows_anything = False
-
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
-        if shadows_type:
-            self._shadows_anything = True
-        StrDictImplementation.impl_setitem_str(
-            self, key, w_value, shadows_type)
-
-    def impl_setitem(self, w_key, w_value):
-        space = self.space
-        if space.is_w(space.type(w_key), space.w_str):
-            if not self._shadows_anything:
-                w_obj = self.w_type.lookup(space.str_w(w_key))
-                if w_obj is not None:
-                    self._shadows_anything = True
-            StrDictImplementation.impl_setitem_str(
-                self, self.space.str_w(w_key), w_value, False)
-        else:
-            self._as_rdict().setitem(w_key, w_value)
-
-    def impl_shadows_anything(self):
-        return (self._shadows_anything or 
-                self.w_type.version_tag() is not self.original_version_tag)
-
-    def impl_set_shadows_anything(self):
-        self._shadows_anything = True
-
 class WaryDictImplementation(StrDictImplementation):
     def __init__(self, space):
         StrDictImplementation.__init__(self, space)
         self.shadowed = [None] * len(BUILTIN_TO_INDEX)
 
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         i = BUILTIN_TO_INDEX.get(key, -1)
         if i != -1:
             self.shadowed[i] = w_value
@@ -446,7 +402,7 @@
         elif _is_sane_hash(space, w_key_type):
             raise KeyError
         else:
-            self._as_rdict().delitem(w_key)
+            self._as_rdict().impl_fallback_delitem(w_key)
 
     def impl_get_builtin_indexed(self, i):
         return self.shadowed[i]
@@ -558,7 +514,7 @@
         self.info.writes += 1
         self.content[w_key] = w_value
         self.info.maxcontents = max(self.info.maxcontents, len(self.content))
-    def impl_setitem_str(self, key, w_value, shadows_type=True):
+    def impl_setitem_str(self, key, w_value):
         self.info.setitem_strs += 1
         self.impl_setitem(self.space.wrap(key), w_value)
     def impl_delitem(self, w_key):
@@ -641,30 +597,50 @@
 init_signature = Signature(['seq_or_map'], None, 'kwargs')
 init_defaults = [None]
 
-def init__DictMulti(space, w_dict, __args__):
-    w_src, w_kwds = __args__.parse_obj(
-            None, 'dict',
-            init_signature, # signature
-            init_defaults)                           # default argument
-    if w_src is None:
-        pass
-    elif space.findattr(w_src, space.wrap("keys")) is None:
-        list_of_w_pairs = space.listview(w_src)
-        for w_pair in list_of_w_pairs:
+def update1(space, w_dict, w_data):
+    if space.findattr(w_data, space.wrap("keys")) is None:
+        # no 'keys' method, so we assume it is a sequence of pairs
+        for w_pair in space.listview(w_data):
             pair = space.fixedview(w_pair)
-            if len(pair)!=2:
+            if len(pair) != 2:
                 raise OperationError(space.w_ValueError,
-                             space.wrap("dict() takes a sequence of pairs"))
-            w_k, w_v = pair
-            w_dict.setitem(w_k, w_v)
+                             space.wrap("sequence of pairs expected"))
+            w_key, w_value = pair
+            w_dict.setitem(w_key, w_value)
     else:
-        if space.is_true(w_src):
-            from pypy.objspace.std.dicttype import update1
-            update1(space, w_dict, w_src)
+        if isinstance(w_data, W_DictMultiObject):    # optimization case only
+            update1_dict_dict(space, w_dict, w_data)
+        else:
+            # general case -- "for k in o.keys(): dict.__setitem__(d, k, o[k])"
+            w_keys = space.call_method(w_data, "keys")
+            for w_key in space.listview(w_keys):
+                w_value = space.getitem(w_data, w_key)
+                w_dict.setitem(w_key, w_value)
+
+def update1_dict_dict(space, w_dict, w_data):
+    iterator = w_data.iter()
+    while 1:
+        w_key, w_value = iterator.next()
+        if w_key is None:
+            break
+        w_dict.setitem(w_key, w_value)
+
+def init_or_update(space, w_dict, __args__, funcname):
+    w_src, w_kwds = __args__.parse_obj(
+            None, funcname,
+            init_signature, # signature
+            init_defaults)  # default argument
+    if w_src is not None:
+        update1(space, w_dict, w_src)
     if space.is_true(w_kwds):
-        from pypy.objspace.std.dicttype import update1
         update1(space, w_dict, w_kwds)
 
+def init__DictMulti(space, w_dict, __args__):
+    init_or_update(space, w_dict, __args__, 'dict')
+
+def dict_update__DictMulti(space, w_dict, __args__):
+    init_or_update(space, w_dict, __args__, 'dict.update')
+
 def getitem__DictMulti_ANY(space, w_dict, w_key):
     w_value = w_dict.getitem(w_key)
     if w_value is not None:
@@ -758,9 +734,8 @@
     return w_res
 
 def dict_copy__DictMulti(space, w_self):
-    from pypy.objspace.std.dicttype import update1
     w_new = W_DictMultiObject.allocate_and_init_instance(space)
-    update1(space, w_new, w_self)
+    update1_dict_dict(space, w_new, w_self)
     return w_new
 
 def dict_items__DictMulti(space, w_self):
@@ -791,6 +766,15 @@
     else:
         return w_default
 
+def dict_setdefault__DictMulti_ANY_ANY(space, w_dict, w_key, w_default):
+    # XXX should be more efficient, with only one dict lookup
+    w_value = w_dict.getitem(w_key)
+    if w_value is not None:
+        return w_value
+    else:
+        w_dict.setitem(w_key, w_default)
+        return w_default
+
 def dict_pop__DictMulti_ANY(space, w_dict, w_key, w_defaults):
     defaults = space.listview(w_defaults)
     len_defaults = len(defaults)
@@ -808,6 +792,14 @@
         w_dict.delitem(w_key)
         return w_item
 
+def dict_popitem__DictMulti(space, w_dict):
+    try:
+        w_key, w_value = w_dict.popitem()
+    except KeyError:
+        raise OperationError(space.w_KeyError,
+                             space.wrap("popitem(): dictionary is empty"))
+    return space.newtuple([w_key, w_value])
+
 app = gateway.applevel('''
     def dictrepr(currently_in_repr, d):
         # Now we only handle one implementation of dicts, this one.

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/dicttype.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/dicttype.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/dicttype.py	Sat Oct 30 16:41:17 2010
@@ -53,106 +53,21 @@
 # This can return when multimethods have been fixed
 #dict_str        = StdObjSpace.str
 
-# default application-level implementations for some operations
-# most of these (notably not popitem and update*) are overwritten
-# in dictmultiobject
-# gateway is imported in the stdtypedef module
-app = gateway.applevel('''
-
-    # in the following functions we use dict.__setitem__ instead of
-    # d[k]=...  because when a subclass of dict override __setitem__,
-    # CPython does not call it when doing builtin operations.  The
-    # same for other operations.
-
-    def update1(d, o):
-        if hasattr(o, 'keys'):
-            for k in o.keys():
-                dict.__setitem__(d, k, o[k])
-        else:
-            for k,v in o:
-                dict.__setitem__(d, k, v)
-
-    def update(d, *args, **kwargs):
-        len_args = len(args)
-        if len_args == 1:
-            update1(d, args[0])
-        elif len_args > 1:
-            raise TypeError("update takes at most 1 (non-keyword) argument")
-        if kwargs:
-            update1(d, kwargs)
-
-    def popitem(d):
-        for k in dict.iterkeys(d):
-            break
-        else:
-            raise KeyError("popitem(): dictionary is empty")
-        v = dict.__getitem__(d, k)
-        dict.__delitem__(d, k)
-        return k, v
-
-    def get(d, k, v=None):
-        if k in d:
-            return dict.__getitem__(d, k)
-        else:
-            return v
-
-    def setdefault(d, k, v=None):
-        if k in d:
-            return dict.__getitem__(d, k)
-        else:
-            dict.__setitem__(d, k, v)
-            return v
-
-    def pop(d, k, defaults):     # XXX defaults is actually *defaults
-        if len(defaults) > 1:
-            raise TypeError, "pop expected at most 2 arguments, got %d" % (
-                1 + len(defaults))
-        try:
-            v = dict.__getitem__(d, k)
-            dict.__delitem__(d, k)
-        except KeyError, e:
-            if defaults:
-                return defaults[0]
-            else:
-                raise e
-        return v
-
-    def iteritems(d):
-        return iter(dict.items(d))
-
-    def iterkeys(d):
-        return iter(dict.keys(d))
-
-    def itervalues(d):
-        return iter(dict.values(d))
-''', filename=__file__)
-
-dict_update__ANY             = app.interphook("update")
-dict_popitem__ANY            = app.interphook("popitem")
-dict_get__ANY_ANY_ANY        = app.interphook("get")
-dict_setdefault__ANY_ANY_ANY = app.interphook("setdefault")
-dict_pop__ANY_ANY            = app.interphook("pop")
-dict_iteritems__ANY          = app.interphook("iteritems")
-dict_iterkeys__ANY           = app.interphook("iterkeys")
-dict_itervalues__ANY         = app.interphook("itervalues")
-update1                      = app.interphook("update1")
-
 register_all(vars(), globals())
 
 @gateway.unwrap_spec(ObjSpace, W_Root, W_Root, W_Root)
 def descr_fromkeys(space, w_type, w_keys, w_fill=None):
+    from pypy.objspace.std.dictmultiobject import W_DictMultiObject
     if w_fill is None:
         w_fill = space.w_None
-    w_dict = space.call_function(w_type)
-    w_iter = space.iter(w_keys)
-    while True:
-        try:
-            w_key = space.next(w_iter)
-        except OperationError, e:
-            if not e.match(space, space.w_StopIteration):
-                raise
-            break
-        space.setitem(w_dict, w_key, w_fill)
+    if w_type is space.w_dict:
+        w_dict = W_DictMultiObject.allocate_and_init_instance(space, w_type)
+        for w_key in space.listview(w_keys):
+            w_dict.setitem(w_key, w_fill)
+    else:
+        w_dict = space.call_function(w_type)
+        for w_key in space.listview(w_keys):
+            space.setitem(w_dict, w_key, w_fill)
     return w_dict
 
 

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/fake.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/fake.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/fake.py	Sat Oct 30 16:41:17 2010
@@ -21,7 +21,6 @@
     #debug_print("faking obj %s" % x)
     ft = fake_type(type(x))
     return ft(space, x)
-fake_object._annspecialcase_ = "override:fake_object"
 
 import sys
 
@@ -47,7 +46,6 @@
         w_exc = space.wrap(exc)
         w_value = space.wrap(value)
     raise OperationError, OperationError(w_exc, w_value), tb
-wrap_exception._annspecialcase_ = "override:ignore"
 
 def fake_type(cpy_type):
     assert type(cpy_type) is type

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/frozensettype.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/frozensettype.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/frozensettype.py	Sat Oct 30 16:41:17 2010
@@ -39,13 +39,13 @@
 def descr__frozenset__new__(space, w_frozensettype,
                             w_iterable=gateway.NoneNotWrapped):
     from pypy.objspace.std.setobject import W_FrozensetObject
-    from pypy.objspace.std.setobject import _is_frozenset_exact
+    from pypy.objspace.std.setobject import make_setdata_from_w_iterable
     if (space.is_w(w_frozensettype, space.w_frozenset) and
-        _is_frozenset_exact(w_iterable)):
+        w_iterable is not None and type(w_iterable) is W_FrozensetObject):
         return w_iterable
     w_obj = space.allocate_instance(W_FrozensetObject, w_frozensettype)
-    W_FrozensetObject.__init__(w_obj, space, None)
-
+    data = make_setdata_from_w_iterable(space, w_iterable)
+    W_FrozensetObject.__init__(w_obj, space, data)
     return w_obj
 
 frozenset_typedef = StdTypeDef("frozenset",

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/inttype.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/inttype.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/inttype.py	Sat Oct 30 16:41:17 2010
@@ -1,7 +1,7 @@
 from pypy.interpreter import gateway
 from pypy.interpreter.error import OperationError
 from pypy.objspace.std.stdtypedef import StdTypeDef
-from pypy.objspace.std.strutil import (string_to_int, string_to_w_long,
+from pypy.objspace.std.strutil import (string_to_int, string_to_bigint,
                                        ParseStringError,
                                        ParseStringOverflowError)
 from pypy.rlib.rarithmetic import r_uint
@@ -45,10 +45,12 @@
 def retry_to_w_long(space, parser, base=0):
     parser.rewind()
     try:
-        return string_to_w_long(space, None, base=base, parser=parser)
+        bigint = string_to_bigint(None, base=base, parser=parser)
     except ParseStringError, e:
         raise OperationError(space.w_ValueError,
                              space.wrap(e.msg))
+    from pypy.objspace.std.longobject import W_LongObject
+    return W_LongObject(bigint)
 
 def descr__new__(space, w_inttype, w_x=0, w_base=gateway.NoneNotWrapped):
     from pypy.objspace.std.intobject import W_IntObject

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/listobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/listobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/listobject.py	Sat Oct 30 16:41:17 2010
@@ -283,7 +283,7 @@
         elif start >= 0:
             del items[start:start+delta]
         else:
-            assert delta==0
+            assert delta==0   # start<0 is only possible with slicelength==0
     elif len2 != slicelength:  # No resize for extended slices
         raise operationerrfmt(space.w_ValueError, "attempt to "
               "assign sequence of size %d to extended slice of size %d",

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/longtype.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/longtype.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/longtype.py	Sat Oct 30 16:41:17 2010
@@ -1,7 +1,7 @@
 from pypy.interpreter.error import OperationError
 from pypy.interpreter import gateway
 from pypy.objspace.std.stdtypedef import StdTypeDef
-from pypy.objspace.std.strutil import string_to_w_long, ParseStringError
+from pypy.objspace.std.strutil import string_to_bigint, ParseStringError
 
 def descr__new__(space, w_longtype, w_x=0, w_base=gateway.NoneNotWrapped):
     from pypy.objspace.std.longobject import W_LongObject
@@ -9,10 +9,10 @@
     if w_base is None:
         # check for easy cases
         if type(w_value) is W_LongObject:
-            pass
+            bigint = w_value.num
         elif space.is_true(space.isinstance(w_value, space.w_str)):
             try:
-                w_value = string_to_w_long(space, space.str_w(w_value))
+                bigint = string_to_bigint(space.str_w(w_value))
             except ParseStringError, e:
                 raise OperationError(space.w_ValueError,
                                      space.wrap(e.msg))
@@ -22,7 +22,7 @@
                     from pypy.objspace.std.ropeunicodeobject import unicode_to_decimal_w
                 else:
                     from pypy.objspace.std.unicodeobject import unicode_to_decimal_w
-                w_value = string_to_w_long(space, unicode_to_decimal_w(space, w_value))
+                bigint = string_to_bigint(unicode_to_decimal_w(space, w_value))
             except ParseStringError, e:
                 raise OperationError(space.w_ValueError,
                                      space.wrap(e.msg))
@@ -35,10 +35,11 @@
             if space.is_true(space.isinstance(w_obj, space.w_long)):
                 assert isinstance(w_obj, W_LongObject)  # XXX this could fail!
                 # XXX find a way to do that even if w_obj is not a W_LongObject
-                w_value = w_obj
+                bigint = w_obj.num
             elif space.is_true(space.isinstance(w_obj, space.w_int)):
+                from pypy.rlib.rbigint import rbigint
                 intval = space.int_w(w_obj)
-                w_value = W_LongObject.fromint(space, intval)
+                bigint = rbigint.fromint(intval)
             else:
                 raise OperationError(space.w_ValueError,
                                     space.wrap("value can't be converted to long"))
@@ -56,13 +57,13 @@
                                      space.wrap("long() can't convert non-string "
                                                 "with explicit base"))
         try:
-            w_value = string_to_w_long(space, s, base)
+            bigint = string_to_bigint(s, base)
         except ParseStringError, e:
             raise OperationError(space.w_ValueError,
                                  space.wrap(e.msg))
 
     w_obj = space.allocate_instance(W_LongObject, w_longtype)
-    W_LongObject.__init__(w_obj, w_value.num)
+    W_LongObject.__init__(w_obj, bigint)
     return w_obj
 
 # ____________________________________________________________

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/model.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/model.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/model.py	Sat Oct 30 16:41:17 2010
@@ -18,6 +18,7 @@
     "withsmallint"   : ["smallintobject.W_SmallIntObject"],
     "withstrslice"   : ["strsliceobject.W_StringSliceObject"],
     "withstrjoin"    : ["strjoinobject.W_StringJoinObject"],
+    "withstrbuf"     : ["strbufobject.W_StringBufferObject"],
     "withrope"       : ["ropeobject.W_RopeObject",
                         "ropeobject.W_RopeIterObject"],
     "withropeunicode": ["ropeunicodeobject.W_RopeUnicodeObject",
@@ -75,6 +76,7 @@
         from pypy.objspace.std import ropeunicodeobject
         from pypy.objspace.std import strsliceobject
         from pypy.objspace.std import strjoinobject
+        from pypy.objspace.std import strbufobject
         from pypy.objspace.std import typeobject
         from pypy.objspace.std import sliceobject
         from pypy.objspace.std import longobject
@@ -222,6 +224,13 @@
                 (unicodeobject.W_UnicodeObject,
                                        strjoinobject.delegate_join2unicode)
                 ]
+        elif config.objspace.std.withstrbuf:
+            self.typeorder[strbufobject.W_StringBufferObject] += [
+                (stringobject.W_StringObject,
+                                       strbufobject.delegate_buf2str),
+                (unicodeobject.W_UnicodeObject,
+                                       strbufobject.delegate_buf2unicode)
+                ]
         if config.objspace.std.withrangelist:
             self.typeorder[rangeobject.W_RangeListObject] += [
                 (listobject.W_ListObject,

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/objspace.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/objspace.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/objspace.py	Sat Oct 30 16:41:17 2010
@@ -23,6 +23,7 @@
 from pypy.objspace.std.listobject import W_ListObject
 from pypy.objspace.std.longobject import W_LongObject
 from pypy.objspace.std.noneobject import W_NoneObject
+from pypy.objspace.std.objectobject import W_ObjectObject
 from pypy.objspace.std.ropeobject import W_RopeObject
 from pypy.objspace.std.iterobject import W_SeqIterObject
 from pypy.objspace.std.setobject import W_SetObject, W_FrozensetObject
@@ -242,7 +243,6 @@
             w_result = getattr(self, 'w_' + x.__name__)
             return w_result
         return None
-    wrap_exception_cls._annspecialcase_ = "override:wrap_exception_cls"
 
     def unwrap(self, w_obj):
         if isinstance(w_obj, Wrappable):
@@ -318,9 +318,14 @@
             w_subtype = w_type.check_user_subclass(w_subtype)
             if cls.typedef.applevel_subclasses_base is not None:
                 cls = cls.typedef.applevel_subclasses_base
-            subcls = get_unique_interplevel_subclass(
-                    self.config, cls, w_subtype.hasdict, w_subtype.nslots != 0,
-                    w_subtype.needsdel, w_subtype.weakrefable)
+            if (self.config.objspace.std.withmapdict and cls is W_ObjectObject
+                    and not w_subtype.needsdel):
+                from pypy.objspace.std.mapdict import get_subclass_of_correct_size
+                subcls = get_subclass_of_correct_size(self, cls, w_subtype)
+            else:
+                subcls = get_unique_interplevel_subclass(
+                        self.config, cls, w_subtype.hasdict, w_subtype.nslots != 0,
+                        w_subtype.needsdel, w_subtype.weakrefable)
             instance = instantiate(subcls)
             assert isinstance(instance, cls)
             instance.user_setup(self, w_subtype)
@@ -432,7 +437,7 @@
             if is_data:
                 w_get = self.lookup(w_descr, "__get__")
             if w_get is None:
-                w_value = w_obj.getdictvalue_attr_is_in_class(self, name)
+                w_value = w_obj.getdictvalue(self, name)
                 if w_value is not None:
                     return w_value
                 if not is_data:
@@ -484,14 +489,12 @@
             return w_obj.getitem(w_key)
         return ObjSpace.finditem(self, w_obj, w_key)
 
-    def setitem_str(self, w_obj, key, w_value, shadows_type=True):
+    def setitem_str(self, w_obj, key, w_value):
         """ Same as setitem, but takes string instead of any wrapped object
-
-        XXX what shadows_type means???
         """
         if (isinstance(w_obj, W_DictMultiObject) and
                 not w_obj.user_overridden_class):
-            w_obj.setitem_str(key, w_value, shadows_type)
+            w_obj.setitem_str(key, w_value)
         else:
             self.setitem(w_obj, self.wrap(key), w_value)
 

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/proxyobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/proxyobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/proxyobject.py	Sat Oct 30 16:41:17 2010
@@ -43,7 +43,7 @@
                     raise
                 return None
         
-        def setdictvalue(self, space, attr, w_value, shadows_type=True):
+        def setdictvalue(self, space, attr, w_value):
             try:
                 space.call_function(self.w_controller, space.wrap('__setattr__'),
                    space.wrap(attr), w_value)

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/setobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/setobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/setobject.py	Sat Oct 30 16:41:17 2010
@@ -21,11 +21,10 @@
         return False
 
 
-    def __init__(w_self, space, setdata=None):
-        if setdata is None:
-            w_self.setdata = r_dict(space.eq_w, space.hash_w)
-        else:
-            w_self.setdata = setdata.copy()
+    def __init__(w_self, space, setdata):
+        """Initialize the set by taking ownership of 'setdata'."""
+        assert setdata is not None
+        w_self.setdata = setdata
 
     def __repr__(w_self):
         """representation for debugging purposes"""
@@ -33,6 +32,7 @@
         return "<%s(%s)>" % (w_self.__class__.__name__, ', '.join(reprlist))
 
     def _newobj(w_self, space, rdict_w=None):
+        """Make a new set or frozenset by taking ownership of 'rdict_w'."""
         #return space.call(space.type(w_self),W_SetIterObject(rdict_w))
         objtype = type(w_self)
         if objtype is W_SetObject:
@@ -55,10 +55,7 @@
 
 class W_FrozensetObject(W_BaseSetObject):
     from pypy.objspace.std.frozensettype import frozenset_typedef as typedef
-
-    def __init__(w_self, space, setdata):
-        W_BaseSetObject.__init__(w_self, space, setdata)
-        w_self.hash = -1
+    hash = 0
 
 registerimplementation(W_BaseSetObject)
 registerimplementation(W_SetObject)
@@ -109,8 +106,14 @@
 
 # some helper functions
 
+def newset(space):
+    return r_dict(space.eq_w, space.hash_w)
+
 def make_setdata_from_w_iterable(space, w_iterable=None):
-    data = r_dict(space.eq_w, space.hash_w)
+    """Return a new r_dict with the content of w_iterable."""
+    if isinstance(w_iterable, W_BaseSetObject):
+        return w_iterable.setdata.copy()
+    data = newset(space)
     if w_iterable is not None:
         for w_item in space.listview(w_iterable):
             data[w_item] = None
@@ -119,7 +122,7 @@
 def _initialize_set(space, w_obj, w_iterable=None):
     w_obj.setdata.clear()
     if w_iterable is not None:
-        w_obj.setdata.update(make_setdata_from_w_iterable(space, w_iterable))
+        w_obj.setdata = make_setdata_from_w_iterable(space, w_iterable)
 
 def _convert_set_to_frozenset(space, w_obj):
     if space.is_true(space.isinstance(w_obj, space.w_set)):
@@ -130,12 +133,6 @@
 
 # helper functions for set operation on dicts
 
-def _is_frozenset_exact(w_obj):
-    if (w_obj is not None) and (type(w_obj) is W_FrozensetObject):
-        return True
-    else:
-        return False
-
 def _is_eq(ld, rd):
     if len(ld) != len(rd):
         return False
@@ -144,66 +141,41 @@
             return False
     return True
 
-def _union_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    ld.update(rdict)
-    return ld, rdict
-
-def _difference_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    del_list_w = []
+def _difference_dict(space, ld, rd):
+    result = newset(space)
     for w_key in ld:
-        if w_key in rdict:
-            del_list_w.append(w_key)
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    return ld, rdict
-
-def _intersection_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
-    else:
-        ld = ldict.copy()
-    del_list_w = []
-    for w_key in ld:
-        if w_key not in rdict:
-            del_list_w.append(w_key)
-
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    return ld, rdict
-
+        if w_key not in rd:
+            result[w_key] = None
+    return result
 
-def _symmetric_difference_dict(ldict, rdict, isupdate):
-    if isupdate:
-        ld = ldict
+def _difference_dict_update(space, ld, rd):
+    if ld is rd:
+        ld.clear()     # for the case 'a.difference_update(a)'
     else:
-        ld = ldict.copy()
-    del_list_w = []
-    add_list_w = []
+        for w_key in rd:
+            try:
+                del ld[w_key]
+            except KeyError:
+                pass
+
+def _intersection_dict(space, ld, rd):
+    result = newset(space)
+    if len(ld) > len(rd):
+        ld, rd = rd, ld     # loop over the smaller dict
     for w_key in ld:
-        if w_key in rdict:
-            del_list_w.append(w_key)
+        if w_key in rd:
+            result[w_key] = None
+    return result
 
-    for w_key in rdict:
+def _symmetric_difference_dict(space, ld, rd):
+    result = newset(space)
+    for w_key in ld:
+        if w_key not in rd:
+            result[w_key] = None
+    for w_key in rd:
         if w_key not in ld:
-            add_list_w.append(w_key)
-
-    for w_key in del_list_w:
-        del ld[w_key]
-
-    for w_key in add_list_w:
-        ld[w_key] = None
-
-    return ld, rdict
+            result[w_key] = None
+    return result
 
 def _issubset_dict(ldict, rdict):
     if len(ldict) > len(rdict):
@@ -220,14 +192,13 @@
 def set_update__Set_BaseSet(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _union_dict(ld, rd, True)
-    return space.w_None
+    ld.update(rd)
 
 def set_update__Set_ANY(space, w_left, w_other):
     """Update a set with the union of itself and another."""
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _union_dict(ld, rd, True)
-    return space.w_None
+    ld = w_left.setdata
+    for w_item in space.listview(w_other):
+        ld[w_item] = None
 
 def inplace_or__Set_Set(space, w_left, w_other):
     set_update__Set_BaseSet(space, w_left, w_other)
@@ -241,25 +212,23 @@
     This has no effect if the element is already present.
     """
     w_left.setdata[w_other] = None
-    return space.w_None
 
 def set_copy__Set(space, w_set):
-    return w_set._newobj(space,w_set.setdata)
+    return w_set._newobj(space, w_set.setdata.copy())
 
 def frozenset_copy__Frozenset(space, w_left):
-    if _is_frozenset_exact(w_left):
+    if type(w_left) is W_FrozensetObject:
         return w_left
     else:
-        return set_copy__Set(space,w_left)
+        return set_copy__Set(space, w_left)
 
 def set_clear__Set(space, w_left):
     w_left.setdata.clear()
-    return space.w_None
 
 def set_difference__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _difference_dict(ld, rd, False)
+    new_ld = _difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 set_difference__Set_Frozenset = set_difference__Set_Set
@@ -272,7 +241,7 @@
 
 def set_difference__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _difference_dict(ld, rd, False)
+    new_ld = _difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 frozenset_difference__Frozenset_ANY = set_difference__Set_ANY
@@ -281,15 +250,17 @@
 def set_difference_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _difference_dict(ld, rd, True)
-    return space.w_None
+    _difference_dict_update(space, ld, rd)
 
 set_difference_update__Set_Frozenset = set_difference_update__Set_Set
 
 def set_difference_update__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _difference_dict(ld, rd, True)
-    return space.w_None
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        try:
+            del ld[w_key]
+        except KeyError:
+            pass
 
 def inplace_sub__Set_Set(space, w_left, w_other):
     set_difference_update__Set_Set(space, w_left, w_other)
@@ -467,7 +438,7 @@
 
 def hash__Frozenset(space, w_set):
     multi = r_uint(1822399083) + r_uint(1822399083) + 1
-    if w_set.hash != -1:
+    if w_set.hash != 0:
         return space.wrap(w_set.hash)
     hash = 1927868237
     hash *= (len(w_set.setdata) + 1)
@@ -476,7 +447,7 @@
         value = ((h ^ (h << 16) ^ 89869747)  * multi)
         hash = intmask(hash ^ value)
     hash = hash * 69069 + 907133923
-    if hash == -1:
+    if hash == 0:
         hash = 590923713
     hash = intmask(hash)
     w_set.hash = hash
@@ -484,29 +455,31 @@
     return space.wrap(hash)
 
 def set_pop__Set(space, w_left):
-    if len(w_left.setdata) == 0:
+    for w_key in w_left.setdata:
+        break
+    else:
         raise OperationError(space.w_KeyError,
                                 space.wrap('pop from an empty set'))
-    w_keys = w_left.setdata.keys()
-    w_value = w_keys[0]
-    del w_left.setdata[w_value]
-
-    return w_value
+    del w_left.setdata[w_key]
+    return w_key
 
 def set_intersection__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _intersection_dict(ld, rd, False)
-    return w_left._newobj(space,new_ld)
+    new_ld = _intersection_dict(space, ld, rd)
+    return w_left._newobj(space, new_ld)
 
 set_intersection__Set_Frozenset = set_intersection__Set_Set
 set_intersection__Frozenset_Frozenset = set_intersection__Set_Set
 set_intersection__Frozenset_Set = set_intersection__Set_Set
 
 def set_intersection__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _intersection_dict(ld, rd, False)
-    return w_left._newobj(space,new_ld)
+    result = newset(space)
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        if w_key in ld:
+            result[w_key] = None
+    return w_left._newobj(space, result)
 
 frozenset_intersection__Frozenset_ANY = set_intersection__Set_ANY
 
@@ -518,15 +491,18 @@
 def set_intersection_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _intersection_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _intersection_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 set_intersection_update__Set_Frozenset = set_intersection_update__Set_Set
 
 def set_intersection_update__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _intersection_dict(ld, rd, True)
-    return space.w_None
+    result = newset(space)
+    ld = w_left.setdata
+    for w_key in space.listview(w_other):
+        if w_key in ld:
+            result[w_key] = None
+    w_left.setdata = result
 
 def inplace_and__Set_Set(space, w_left, w_other):
     set_intersection_update__Set_Set(space, w_left, w_other)
@@ -537,7 +513,7 @@
 def set_symmetric_difference__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _symmetric_difference_dict(ld, rd, False)
+    new_ld = _symmetric_difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 set_symmetric_difference__Set_Frozenset = set_symmetric_difference__Set_Set
@@ -553,7 +529,7 @@
 
 def set_symmetric_difference__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _symmetric_difference_dict(ld, rd, False)
+    new_ld = _symmetric_difference_dict(space, ld, rd)
     return w_left._newobj(space, new_ld)
 
 frozenset_symmetric_difference__Frozenset_ANY = \
@@ -562,16 +538,16 @@
 def set_symmetric_difference_update__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _symmetric_difference_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _symmetric_difference_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 set_symmetric_difference_update__Set_Frozenset = \
                                     set_symmetric_difference_update__Set_Set
 
 def set_symmetric_difference_update__Set_ANY(space, w_left, w_other):
     ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _symmetric_difference_dict(ld, rd, True)
-    return space.w_None
+    new_ld = _symmetric_difference_dict(space, ld, rd)
+    w_left.setdata = new_ld
 
 def inplace_xor__Set_Set(space, w_left, w_other):
     set_symmetric_difference_update__Set_Set(space, w_left, w_other)
@@ -582,8 +558,9 @@
 def set_union__Set_Set(space, w_left, w_other):
     # optimization only (the general case works too)
     ld, rd = w_left.setdata, w_other.setdata
-    new_ld, rd = _union_dict(ld, rd, False)
-    return w_left._newobj(space, new_ld)
+    result = ld.copy()
+    result.update(rd)
+    return w_left._newobj(space, result)
 
 set_union__Set_Frozenset = set_union__Set_Set
 set_union__Frozenset_Set = set_union__Set_Set
@@ -595,9 +572,11 @@
 
 
 def set_union__Set_ANY(space, w_left, w_other):
-    ld, rd = w_left.setdata, make_setdata_from_w_iterable(space, w_other)
-    new_ld, rd = _union_dict(ld, rd, False)
-    return w_left._newobj(space, new_ld)
+    ld = w_left.setdata
+    result = ld.copy()
+    for w_key in space.listview(w_other):
+        result[w_key] = None
+    return w_left._newobj(space, result)
 
 frozenset_union__Frozenset_ANY = set_union__Set_ANY
 
@@ -629,15 +608,6 @@
             init_defaults)
     _initialize_set(space, w_set, w_iterable)
 
-def init__Frozenset(space, w_set, __args__):
-    w_iterable, = __args__.parse_obj(
-            None, 'set',
-            init_signature,
-            init_defaults)
-    if w_set.hash == -1:
-        _initialize_set(space, w_set, w_iterable)
-        hash__Frozenset(space, w_set)
-
 app = gateway.applevel("""
     def setrepr(currently_in_repr, s):
         'The app-level part of repr().'

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/settype.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/settype.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/settype.py	Sat Oct 30 16:41:17 2010
@@ -66,9 +66,9 @@
 register_all(vars(), globals())
 
 def descr__new__(space, w_settype, __args__):
-    from pypy.objspace.std.setobject import W_SetObject
+    from pypy.objspace.std.setobject import W_SetObject, newset
     w_obj = space.allocate_instance(W_SetObject, w_settype)
-    W_SetObject.__init__(w_obj, space, None)
+    W_SetObject.__init__(w_obj, space, newset(space))
     return w_obj
 
 set_typedef = StdTypeDef("set",

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/smallintobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/smallintobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/smallintobject.py	Sat Oct 30 16:41:17 2010
@@ -2,18 +2,13 @@
 Implementation of small ints, stored as odd-valued pointers in the
 translated PyPy.  To enable them, see inttype.py.
 """
-import types
-from pypy.interpreter.error import OperationError
 from pypy.objspace.std import intobject
 from pypy.objspace.std.model import registerimplementation, W_Object
 from pypy.objspace.std.register_all import register_all
-from pypy.objspace.std.multimethod import FailedToImplementArgs
 from pypy.objspace.std.noneobject import W_NoneObject
-from pypy.rlib.rarithmetic import ovfcheck, ovfcheck_lshift, LONG_BIT, r_uint
-from pypy.objspace.std.inttype import wrapint
-from pypy.objspace.std.intobject import W_IntObject, _impl_int_int_pow
+from pypy.objspace.std.intobject import W_IntObject
 from pypy.rlib.objectmodel import UnboxedValue
-from pypy.rlib.rbigint import rbigint
+from pypy.tool.sourcetools import func_with_new_name
 
 
 class W_SmallIntObject(W_Object, UnboxedValue):
@@ -46,9 +41,7 @@
             new_name = name.replace("Int", "SmallInt")
             # Copy the function, so the annotator specializes it for
             # W_SmallIntObject.
-            ns[new_name] = types.FunctionType(func.func_code, ns, new_name,
-                                              func.func_defaults,
-                                              func.func_closure)
+            ns[new_name] = func_with_new_name(func, new_name)
     ns["get_integer"] = ns["pos__SmallInt"] = ns["int__SmallInt"]
     ns["get_negint"] = ns["neg__SmallInt"]
 

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/stringobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/stringobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/stringobject.py	Sat Oct 30 16:41:17 2010
@@ -14,7 +14,7 @@
 from pypy.rlib.rstring import StringBuilder, string_repeat
 from pypy.interpreter.buffer import StringBuffer
 
-from pypy.objspace.std.stringtype import sliced, joined, wrapstr, wrapchar, \
+from pypy.objspace.std.stringtype import sliced, wrapstr, wrapchar, \
      stringendswith, stringstartswith, joined2
 
 from pypy.objspace.std.formatting import mod_format

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/stringtype.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/stringtype.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/stringtype.py	Sat Oct 30 16:41:17 2010
@@ -55,19 +55,14 @@
             return W_StringSliceObject(s, start, stop)
     return wrapstr(space, s[start:stop])
 
-def joined(space, strlist):
-    assert not space.config.objspace.std.withrope
-    if space.config.objspace.std.withstrjoin:
-        from pypy.objspace.std.strjoinobject import W_StringJoinObject
-        return W_StringJoinObject(strlist)
-    else:
-        return wrapstr(space, "".join(strlist))
-
 def joined2(space, str1, str2):
     assert not space.config.objspace.std.withrope
     if space.config.objspace.std.withstrjoin:
         from pypy.objspace.std.strjoinobject import W_StringJoinObject
         return W_StringJoinObject([str1, str2])
+    elif space.config.objspace.std.withstrbuf:
+        from pypy.objspace.std.strbufobject import joined2
+        return joined2(str1, str2)
     else:
         return wrapstr(space, str1 + str2)
 

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/strutil.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/strutil.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/strutil.py	Sat Oct 30 16:41:17 2010
@@ -4,6 +4,7 @@
 
 from pypy.rlib.rarithmetic import ovfcheck, break_up_float, parts_to_float,\
      INFINITY, NAN
+from pypy.rlib.rbigint import rbigint, parse_digit_string
 from pypy.interpreter.error import OperationError
 import math
 
@@ -91,9 +92,10 @@
             return -1
 
 def string_to_int(s, base=10):
-    """Utility to converts a string to an integer (or possibly a long).
+    """Utility to converts a string to an integer.
     If base is 0, the proper base is guessed based on the leading
     characters of 's'.  Raises ParseStringError in case of error.
+    Raises ParseStringOverflowError in case the result does not fit.
     """
     s = literal = strip_spaces(s)
     p = NumberStringParser(s, literal, base, 'int')
@@ -113,11 +115,9 @@
         except OverflowError:
             raise ParseStringOverflowError(p)
 
-def string_to_long(space, s, base=10, parser=None):
-    return string_to_w_long(space, s, base, parser).longval()
-
-def string_to_w_long(space, s, base=10, parser=None):
-    """As string_to_int(), but ignores an optional 'l' or 'L' suffix."""
+def string_to_bigint(s, base=10, parser=None):
+    """As string_to_int(), but ignores an optional 'l' or 'L' suffix
+    and returns an rbigint."""
     if parser is None:
         s = literal = strip_spaces(s)
         if (s.endswith('l') or s.endswith('L')) and base < 22:
@@ -126,18 +126,7 @@
         p = NumberStringParser(s, literal, base, 'long')
     else:
         p = parser
-    w_base = space.newlong(p.base)
-    w_result = space.newlong(0)
-    while True:
-        digit = p.next_digit()
-        if digit == -1:
-            if p.sign == -1:
-                w_result = space.neg(w_result)
-            # XXX grumble
-            from pypy.objspace.std.longobject import W_LongObject
-            assert isinstance(w_result, W_LongObject)
-            return w_result
-        w_result = space.add(space.mul(w_result,w_base), space.newlong(digit))
+    return parse_digit_string(p)
 
 # Tim's comment:
 # 57 bits are more than needed in any case.

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_dictmultiobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_dictmultiobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_dictmultiobject.py	Sat Oct 30 16:41:17 2010
@@ -4,7 +4,6 @@
      StrDictImplementation
 
 from pypy.objspace.std.celldict import ModuleDictImplementation
-from pypy.objspace.std.sharingdict import SharedDictImplementation
 from pypy.conftest import gettestobjspace
 
 
@@ -238,7 +237,17 @@
         it1 = d.popitem()
         assert len(d) == 0
         assert (it!=it1) and (it1==(1,2) or it1==(3,4))
-    
+        raises(KeyError, d.popitem)
+
+    def test_popitem_2(self):
+        class A(object):
+            pass
+        d = A().__dict__
+        d['x'] = 5
+        it1 = d.popitem()
+        assert it1 == ('x', 5)
+        raises(KeyError, d.popitem)
+
     def test_setdefault(self):
         d = {1:2, 3:4}
         dd = d.copy()
@@ -446,7 +455,9 @@
         d1 = {}
         d2 = D(a='foo')
         d1.update(d2)
-        assert d1['a'] == 42 # fails on CPython, d1['a'] == 'foo'
+        assert d1['a'] == 'foo'
+        # a bit of an obscure case: now (from r78295) we get the same result
+        # as CPython does
 
     def test_index_keyerror_unpacking(self):
         d = {}
@@ -510,32 +521,6 @@
         assert getattr(a, s) == 42
 
 
-class TestW_DictSharing(TestW_DictObject):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withsharingdict": True})
-
-class AppTest_DictSharing(AppTest_DictObject):
-    def setup_class(cls):
-        cls.space = gettestobjspace(**{"objspace.std.withsharingdict": True})
-
-    def test_values_does_not_share(self):
-        class A(object):
-            pass
-        a = A()
-        a.abc = 12
-        l = a.__dict__.values()
-        assert l == [12]
-        l[0] = 24
-        assert a.abc == 12
-
-    def test_items(self):
-        class A(object):
-            pass
-        a = A()
-        a.abc = 12
-        a.__dict__.items() == [("abc", 12)]
-
-
 class AppTestModuleDict(object):
     def setup_class(cls):
         cls.space = gettestobjspace(**{"objspace.std.withcelldict": True})
@@ -602,6 +587,15 @@
                 classofinstance=classofinstance,
                 from_strdict_shared=from_strdict_shared)
 
+    def finditem_str(self, w_dict, s):
+        return w_dict.getitem_str(s) # assume it's a multidict
+
+    def setitem_str(self, w_dict, s, w_value):
+        return w_dict.setitem_str(s, w_value) # assume it's a multidict
+
+    def delitem(self, w_dict, w_s):
+        return w_dict.delitem(w_s) # assume it's a multidict
+
     def allocate_instance(self, cls, type):
         return object.__new__(cls)
 
@@ -611,7 +605,7 @@
     w_StopIteration = StopIteration
     w_None = None
     StringObjectCls = FakeString
-    w_dict = None
+    w_dict = W_DictMultiObject
     iter = iter
     fixedview = list
     listview  = list
@@ -620,10 +614,8 @@
     class objspace:
         class std:
             withdictmeasurement = False
-            withsharingdict = False
             withsmalldicts = False
             withcelldict = False
-            withshadowtracking = False
         class opcodes:
             CALL_LIKELY_BUILTIN = False
 
@@ -687,6 +679,14 @@
         assert self.impl.length() == 0
         self.check_not_devolved()
 
+    def test_clear(self):
+        self.fill_impl()
+        assert self.impl.length() == 2
+        self.impl.clear()
+        assert self.impl.length() == 0
+        self.check_not_devolved()
+
+
     def test_keys(self):
         self.fill_impl()
         keys = self.impl.keys()
@@ -750,9 +750,6 @@
     string = "int"
     string2 = "isinstance"
 
-class TestSharedDictImplementation(BaseTestRDictImplementation):
-    ImplementionClass = SharedDictImplementation
-
 
 class BaseTestDevolvedDictImplementation(BaseTestRDictImplementation):
     def fill_impl(self):
@@ -774,8 +771,6 @@
     string = "int"
     string2 = "isinstance"
 
-class TestDevolvedSharedDictImplementation(BaseTestDevolvedDictImplementation):
-    ImplementionClass = SharedDictImplementation
 
 def test_module_uses_strdict():
     fakespace = FakeSpace()

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_longobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_longobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_longobject.py	Sat Oct 30 16:41:17 2010
@@ -41,56 +41,64 @@
 
 class AppTestLong:
     def test_add(self):
-        assert int(123L + 12443L) == 123 + 12443
-        assert -20 + 2 + 3L + True == -14L
+        x = 123L
+        assert int(x + 12443L) == 123 + 12443
+        x = -20
+        assert x + 2 + 3L + True == -14L
 
     def test_sub(self):
-        assert int(58543L - 12332L) == 58543 - 12332
-        assert 237123838281233L * 12 == 237123838281233L * 12L
+        x = 58543L
+        assert int(x - 12332L) == 58543 - 12332
+        x = 237123838281233L
+        assert x * 12 == x * 12L
 
     def test_mul(self):
-        assert 363L * 2 ** 40 == 363L << 40
+        x = 363L
+        assert x * 2 ** 40 == x << 40
 
     def test_truediv(self):
         exec "from __future__ import division; a = 31415926L / 10000000L"
         assert a == 3.1415926
 
     def test_floordiv(self):
-        a = 31415926L // 10000000L
+        x = 31415926L
+        a = x // 10000000L
         assert a == 3L
 
     def test_compare(self):
+        Z = 0
+        ZL = 0L
         for BIG in (1L, 1L << 62, 1L << 9999):
-            assert 0 == 0L
-            assert not (0 != 0L)
-            assert 0L == 0
-            assert not (0L != 0)
-            assert not (0 == BIG)
-            assert 0 != BIG
-            assert not (BIG == 0)
-            assert BIG != 0
-            assert not (0L == BIG)
-            assert 0L != BIG
-            assert 0 <= 0L
-            assert not (0 < 0L)
-            assert 0 <= BIG
-            assert 0 < BIG
-            assert not (BIG <= 0)
-            assert not (BIG < 0)
-            assert 0L <= 0L
-            assert not (0L < 0L)
-            assert 0L <= BIG
-            assert 0L < BIG
-            assert not (BIG <= 0L)
-            assert not (BIG < 0L)
-            assert not (0 <= -BIG)
-            assert not (0 < -BIG)
-            assert -BIG <= 0
-            assert -BIG < 0
-            assert not (0L <= -BIG)
-            assert not (0L < -BIG)
-            assert -BIG <= 0L
-            assert -BIG < 0L
+            assert Z == ZL
+            assert not (Z != ZL)
+            assert ZL == Z
+            assert not (ZL != Z)
+            assert not (Z == BIG)
+            assert Z != BIG
+            assert not (BIG == Z)
+            assert BIG != Z
+            assert not (ZL == BIG)
+            assert ZL != BIG
+            assert Z <= ZL
+            assert not (Z < ZL)
+            assert Z <= BIG
+            assert Z < BIG
+            assert not (BIG <= Z)
+            assert not (BIG < Z)
+            assert ZL <= ZL
+            assert not (ZL < ZL)
+            assert ZL <= BIG
+            assert ZL < BIG
+            assert not (BIG <= ZL)
+            assert not (BIG < ZL)
+            assert not (Z <= -BIG)
+            assert not (Z < -BIG)
+            assert -BIG <= Z
+            assert -BIG < Z
+            assert not (ZL <= -BIG)
+            assert not (ZL < -BIG)
+            assert -BIG <= ZL
+            assert -BIG < ZL
             #
             assert not (BIG <  int(BIG))
             assert     (BIG <= int(BIG))
@@ -137,7 +145,8 @@
     def test_conversion(self):
         class long2(long):
             pass
-        x = long2(1L<<100)
+        x = 1L
+        x = long2(x<<100)
         y = int(x)
         assert type(y) == long
         assert type(+long2(5)) is long
@@ -152,7 +161,8 @@
         assert type(long2(5) // 1) is long
 
     def test_pow(self):
-        assert pow(0L, 0L, 1L) == 0L
+        x = 0L
+        assert pow(x, 0L, 1L) == 0L
 
     def test_getnewargs(self):
         assert  0L .__getnewargs__() == (0L,)
@@ -193,13 +203,14 @@
         assert oct(01234567012345670L) == '01234567012345670L'
 
     def test_bits(self):
-        assert 0xAAAAAAAAL | 0x55555555L == 0xFFFFFFFFL
-        assert 0xAAAAAAAAL & 0x55555555L == 0x00000000L
-        assert 0xAAAAAAAAL ^ 0x55555555L == 0xFFFFFFFFL
-        assert -0xAAAAAAAAL | 0x55555555L == -0xAAAAAAA9L
-        assert 0xAAAAAAAAL | 0x555555555L == 0x5FFFFFFFFL
-        assert 0xAAAAAAAAL & 0x555555555L == 0x000000000L
-        assert 0xAAAAAAAAL ^ 0x555555555L == 0x5FFFFFFFFL
+        x = 0xAAAAAAAAL
+        assert x | 0x55555555L == 0xFFFFFFFFL
+        assert x & 0x55555555L == 0x00000000L
+        assert x ^ 0x55555555L == 0xFFFFFFFFL
+        assert -x | 0x55555555L == -0xAAAAAAA9L
+        assert x | 0x555555555L == 0x5FFFFFFFFL
+        assert x & 0x555555555L == 0x000000000L
+        assert x ^ 0x555555555L == 0x5FFFFFFFFL
 
     def test_hash(self):
         # ints have the same hash as equal longs
@@ -229,7 +240,8 @@
 
     def test_huge_longs(self):
         import operator
-        huge = 1L << 40000L
+        x = 1L
+        huge = x << 40000L
         raises(OverflowError, float, huge)
         raises(OverflowError, operator.truediv, huge, 3)
         raises(OverflowError, operator.truediv, huge, 3L)
@@ -242,3 +254,7 @@
         class myotherlong(long):
             pass
         assert long(myotherlong(21)) == 21L
+
+    def test_negative_zero(self):
+        x = eval("-0L")
+        assert x == 0L

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_setobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_setobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_setobject.py	Sat Oct 30 16:41:17 2010
@@ -10,6 +10,7 @@
 import py.test
 from pypy.objspace.std.setobject import W_SetObject, W_FrozensetObject
 from pypy.objspace.std.setobject import _initialize_set
+from pypy.objspace.std.setobject import newset, make_setdata_from_w_iterable
 from pypy.objspace.std.setobject import set_intersection__Set_Set
 from pypy.objspace.std.setobject import set_intersection__Set_ANY
 from pypy.objspace.std.setobject import eq__Set_Set
@@ -28,12 +29,12 @@
         self.false = self.space.w_False
 
     def test_and(self):
-        s = W_SetObject(self.space, None)
+        s = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, s, self.word)
-        t0 = W_SetObject(self.space, None)
+        t0 = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, t0, self.otherword)
-        t1 = W_FrozensetObject(self.space, None)
-        _initialize_set(self.space, t1, self.otherword)
+        t1 = W_FrozensetObject(self.space,
+                make_setdata_from_w_iterable(self.space, self.otherword))
         r0 = set_intersection__Set_Set(self.space, s, t0)
         r1 = set_intersection__Set_Set(self.space, s, t1)
         assert eq__Set_Set(self.space, r0, r1) == self.true
@@ -41,9 +42,9 @@
         assert eq__Set_Set(self.space, r0, sr) == self.true
 
     def test_compare(self):
-        s = W_SetObject(self.space, None)
+        s = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, s, self.word)
-        t = W_SetObject(self.space, None)
+        t = W_SetObject(self.space, newset(self.space))
         _initialize_set(self.space, t, self.word)
         assert self.space.eq_w(s,t)
         u = self.space.wrap(set('simsalabim'))
@@ -56,6 +57,11 @@
         b = a | set('abc')
         assert type(b) is subset
 
+    def test_union(self):
+        a = set([4, 5])
+        b = a.union([5, 7])
+        assert sorted(b) == [4, 5, 7]
+
     def test_compare(self):
         raises(TypeError, cmp, set('abc'), set('abd'))
         assert set('abc') != 'abc'

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_strutil.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_strutil.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_strutil.py	Sat Oct 30 16:41:17 2010
@@ -25,7 +25,7 @@
                  ]
         for s, expected in cases:
             assert string_to_int(s) == expected
-            assert string_to_w_long(space, s).longval() == expected
+            assert string_to_bigint(s).tolong() == expected
 
     def test_string_to_int_base(self):
         space = self.space        
@@ -119,17 +119,16 @@
             raises(ParseStringError, string_to_int, '+'+s, base)
             raises(ParseStringError, string_to_int, '-'+s, base)
 
-    def test_string_to_w_long(self):
-        space = self.space
-        assert string_to_w_long(space, '123L').longval() == 123
-        assert string_to_w_long(space, '123L  ').longval() == 123
-        raises(ParseStringError, string_to_w_long, space, 'L')
-        raises(ParseStringError, string_to_w_long, space, 'L  ')
-        assert string_to_w_long(space, '123L', 4).longval() == 27
-        assert string_to_w_long(space, '123L', 30).longval() == 27000 + 1800 + 90 + 21
-        assert string_to_w_long(space, '123L', 22).longval() == 10648 + 968 + 66 + 21
-        assert string_to_w_long(space, '123L', 21).longval() == 441 + 42 + 3
-        assert string_to_w_long(space, '1891234174197319').longval() == 1891234174197319
+    def test_string_to_bigint(self):
+        assert string_to_bigint('123L').tolong() == 123
+        assert string_to_bigint('123L  ').tolong() == 123
+        raises(ParseStringError, string_to_bigint, 'L')
+        raises(ParseStringError, string_to_bigint, 'L  ')
+        assert string_to_bigint('123L', 4).tolong() == 27
+        assert string_to_bigint('123L', 30).tolong() == 27000 + 1800 + 90 + 21
+        assert string_to_bigint('123L', 22).tolong() == 10648 + 968 + 66 + 21
+        assert string_to_bigint('123L', 21).tolong() == 441 + 42 + 3
+        assert string_to_bigint('1891234174197319').tolong() == 1891234174197319
 
     def test_string_to_float(self):
         def string_to_float(x):

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_typeobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_typeobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_typeobject.py	Sat Oct 30 16:41:17 2010
@@ -1085,4 +1085,12 @@
 
         assert b == 1
 
-        
+    def test_slots_with_method_in_class(self):
+        # this works in cpython...
+        class A(object):
+            __slots__ = ["f"]
+            def f(self, x):
+                return x + 1
+        a = A()
+        assert a.f(1) == 2
+

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_versionedtype.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_versionedtype.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/test/test_versionedtype.py	Sat Oct 30 16:41:17 2010
@@ -12,10 +12,12 @@
                 def f(self): pass
             class B(A):
                 pass
-            class metatype(type):
+            class X:
+                pass
+            class Y(object):
+                pass
+            class C(Y, X):
                 pass
-            class C(object):
-                __metaclass__ = metatype
             return A, B, C
         """)
         return space.unpackiterable(w_types)

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/std/typeobject.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/std/typeobject.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/std/typeobject.py	Sat Oct 30 16:41:17 2010
@@ -60,6 +60,15 @@
             self.hits = {}
             self.misses = {}
 
+    def clear(self):
+        None_None = (None, None)
+        for i in range(len(self.versions)):
+            self.versions[i] = None
+        for i in range(len(self.names)):
+            self.names[i] = None
+        for i in range(len(self.lookup_where)):
+            self.lookup_where[i] = None_None
+
 
 class W_TypeObject(W_Object):
     from pypy.objspace.std.typetype import type_typedef as typedef
@@ -75,7 +84,9 @@
                           'weakrefable',
                           'hasdict',
                           'nslots',
-                          'instancetypedef']
+                          'instancetypedef',
+                          'terminator',
+                          ]
 
     # for config.objspace.std.getattributeshortcut
     # (False is a conservative default, fixed during real usage)
@@ -102,20 +113,24 @@
 
         if overridetypedef is not None:
             setup_builtin_type(w_self)
-            custom_metaclass = False
         else:
             setup_user_defined_type(w_self)
-            custom_metaclass = not space.is_w(space.type(w_self), space.w_type)
         w_self.w_same_layout_as = get_parent_layout(w_self)
 
         if space.config.objspace.std.withtypeversion:
-            if custom_metaclass or not is_mro_purely_of_types(w_self.mro_w):
+            if not is_mro_purely_of_types(w_self.mro_w):
                 pass
             else:
                 # the _version_tag should change, whenever the content of
                 # dict_w of any of the types in the mro changes, or if the mro
                 # itself changes
                 w_self._version_tag = VersionTag()
+        if space.config.objspace.std.withmapdict:
+            from pypy.objspace.std.mapdict import DictTerminator, NoDictTerminator
+            if w_self.hasdict:
+                w_self.terminator = DictTerminator(space, w_self)
+            else:
+                w_self.terminator = NoDictTerminator(space, w_self)
 
     def mutated(w_self):
         space = w_self.space
@@ -551,11 +566,13 @@
                              space.wrap('__slots__ must be identifiers'))
     # create member
     slot_name = _mangle(slot_name, w_self.name)
-    # Force interning of slot names.
-    slot_name = space.str_w(space.new_interned_str(slot_name))
-    member = Member(w_self.nslots, slot_name, w_self)
-    w_self.dict_w[slot_name] = space.wrap(member)
-    w_self.nslots += 1
+    if slot_name not in w_self.dict_w:
+        # Force interning of slot names.
+        slot_name = space.str_w(space.new_interned_str(slot_name))
+        # in cpython it is ignored less, but we probably don't care
+        member = Member(w_self.nslots, slot_name, w_self)
+        w_self.dict_w[slot_name] = space.wrap(member)
+        w_self.nslots += 1
 
 def create_dict_slot(w_self):
     if not w_self.hasdict:

Modified: pypy/branch/jit-unroll-loops/pypy/objspace/test/test_descroperation.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/objspace/test/test_descroperation.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/objspace/test/test_descroperation.py	Sat Oct 30 16:41:17 2010
@@ -400,7 +400,7 @@
         class D(object):
             def __init__(self, a):
                 self.a = a
-        
+
         assert A(1) == B(1)
         assert B(1) == A(1)
         assert A(1) == C(1)
@@ -457,6 +457,20 @@
         else:
             assert False, "did not raise"
 
+    def test_attribute_error(self):
+        class classmethodonly(classmethod):
+            def __get__(self, instance, type):
+                if instance is not None:
+                    raise AttributeError("Must be called on a class, not an instance.")
+                return super(classmethodonly, self).__get__(instance, type)
+
+        class A(object):
+            @classmethodonly
+            def a(cls):
+                return 3
+
+        raises(AttributeError, lambda: A().a)
+
 
 class AppTestWithBuiltinShortcut(AppTest_Descroperation):
     OPTIONS = {'objspace.std.builtinshortcut': True}

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/_rsocket_rffi.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/_rsocket_rffi.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/_rsocket_rffi.py	Sat Oct 30 16:41:17 2010
@@ -324,10 +324,11 @@
                                ('sll_hatype', rffi.INT),
                                ('sll_addr', rffi.CFixedArray(rffi.CHAR, 8)),
                                ('sll_halen', rffi.INT)],
-                              )
+                              ifdef='AF_PACKET')
 
     CConfig.ifreq = platform.Struct('struct ifreq', [('ifr_ifindex', rffi.INT),
-                                 ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))])
+                                 ('ifr_name', rffi.CFixedArray(rffi.CHAR, 8))],
+                                    ifdef='AF_PACKET')
 
 if _WIN32:
     CConfig.WSAEVENT = platform.SimpleType('WSAEVENT', rffi.VOIDP)
@@ -532,8 +533,9 @@
     socketpair_t = rffi.CArray(socketfd_type)
     socketpair = external('socketpair', [rffi.INT, rffi.INT, rffi.INT,
                           lltype.Ptr(socketpair_t)], rffi.INT)
-    ioctl = external('ioctl', [socketfd_type, rffi.INT, lltype.Ptr(ifreq)],
-                     rffi.INT)
+    if ifreq is not None:
+        ioctl = external('ioctl', [socketfd_type, rffi.INT, lltype.Ptr(ifreq)],
+                         rffi.INT)
 
 if _WIN32:
     ioctlsocket = external('ioctlsocket',

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/jit.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/jit.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/jit.py	Sat Oct 30 16:41:17 2010
@@ -77,6 +77,12 @@
         return result
     return decorator
 
+def oopspec(spec):
+    def decorator(func):
+        func.oopspec = spec
+        return func
+    return decorator
+
 class Entry(ExtRegistryEntry):
     _about_ = hint
 
@@ -139,6 +145,24 @@
         return hop.inputconst(lltype.Signed, _we_are_jitted)
 
 
+def jit_debug(string, arg1=-sys.maxint-1, arg2=-sys.maxint-1,
+                      arg3=-sys.maxint-1, arg4=-sys.maxint-1):
+    """When JITted, cause an extra operation DEBUG_MERGE_POINT to appear in
+    the graphs.  Should not be left after debugging."""
+    keepalive_until_here(string) # otherwise the whole function call is removed
+jit_debug.oopspec = 'jit.debug(string, arg1, arg2, arg3, arg4)'
+
+def assert_green(value):
+    """Very strong assert: checks that 'value' is a green
+    (a JIT compile-time constant)."""
+    keepalive_until_here(value)
+assert_green._annspecialcase_ = 'specialize:argtype(0)'
+assert_green.oopspec = 'jit.assert_green(value)'
+
+class AssertGreenFailed(Exception):
+    pass
+
+
 ##def force_virtualizable(virtualizable):
 ##    pass
 
@@ -251,8 +275,9 @@
     several independent JITting interpreters in it.
     """
 
+    active = True          # if set to False, this JitDriver is ignored
     virtualizables = []
-    
+
     def __init__(self, greens=None, reds=None, virtualizables=None,
                  get_jitcell_at=None, set_jitcell_at=None,
                  get_printable_location=None, confirm_enter_jit=None,
@@ -267,7 +292,8 @@
             self.virtualizables = virtualizables
         for v in self.virtualizables:
             assert v in self.reds
-        self._alllivevars = dict.fromkeys(self.greens + self.reds)
+        self._alllivevars = dict.fromkeys(
+            [name for name in self.greens + self.reds if '.' not in name])
         self._make_extregistryentries()
         self.get_jitcell_at = get_jitcell_at
         self.set_jitcell_at = set_jitcell_at
@@ -356,10 +382,16 @@
 
     def compute_result_annotation(self, **kwds_s):
         from pypy.annotation import model as annmodel
+
+        if self.instance.__name__ == 'jit_merge_point':
+            if not self.annotate_hooks(**kwds_s):
+                return None      # wrong order, try again later
+
         driver = self.instance.im_self
         keys = kwds_s.keys()
         keys.sort()
-        expected = ['s_' + name for name in driver.greens + driver.reds]
+        expected = ['s_' + name for name in driver.greens + driver.reds
+                                if '.' not in name]
         expected.sort()
         if keys != expected:
             raise JitHintError("%s expects the following keyword "
@@ -383,30 +415,35 @@
                                    key[2:])
             cache[key] = s_value
 
-        if self.instance.__name__ == 'jit_merge_point':
-            self.annotate_hooks(**kwds_s)
-            
         return annmodel.s_None
 
     def annotate_hooks(self, **kwds_s):
         driver = self.instance.im_self
         s_jitcell = self.bookkeeper.valueoftype(BaseJitCell)
-        self.annotate_hook(driver.get_jitcell_at, driver.greens, **kwds_s)
-        self.annotate_hook(driver.set_jitcell_at, driver.greens, [s_jitcell],
-                           **kwds_s)
-        self.annotate_hook(driver.get_printable_location, driver.greens, **kwds_s)
+        h = self.annotate_hook
+        return (h(driver.get_jitcell_at, driver.greens, **kwds_s)
+            and h(driver.set_jitcell_at, driver.greens, [s_jitcell], **kwds_s)
+            and h(driver.get_printable_location, driver.greens, **kwds_s))
 
     def annotate_hook(self, func, variables, args_s=[], **kwds_s):
         if func is None:
-            return
+            return True
         bk = self.bookkeeper
         s_func = bk.immutablevalue(func)
         uniquekey = 'jitdriver.%s' % func.func_name
         args_s = args_s[:]
         for name in variables:
-            s_arg = kwds_s['s_' + name]
+            if '.' not in name:
+                s_arg = kwds_s['s_' + name]
+            else:
+                objname, fieldname = name.split('.')
+                s_instance = kwds_s['s_' + objname]
+                s_arg = s_instance.classdef.about_attribute(fieldname)
+                if s_arg is None:
+                    return False     # wrong order, try again later
             args_s.append(s_arg)
         bk.emulate_pbc_call(uniquekey, s_func, args_s)
+        return True
 
     def specialize_call(self, hop, **kwds_i):
         # XXX to be complete, this could also check that the concretetype
@@ -417,9 +454,42 @@
         greens_v = []
         reds_v = []
         for name in driver.greens:
-            i = kwds_i['i_' + name]
-            r_green = hop.args_r[i]
-            v_green = hop.inputarg(r_green, arg=i)
+            if '.' not in name:
+                i = kwds_i['i_' + name]
+                r_green = hop.args_r[i]
+                v_green = hop.inputarg(r_green, arg=i)
+            else:
+                if hop.rtyper.type_system.name == 'ootypesystem':
+                    py.test.skip("lltype only")
+                objname, fieldname = name.split('.')   # see test_green_field
+                assert objname in driver.reds
+                i = kwds_i['i_' + objname]
+                s_red = hop.args_s[i]
+                r_red = hop.args_r[i]
+                while True:
+                    try:
+                        mangled_name, r_field = r_red._get_field(fieldname)
+                        break
+                    except KeyError:
+                        pass
+                    assert r_red.rbase is not None, (
+                        "field %r not found in %r" % (name,
+                                                      r_red.lowleveltype.TO))
+                    r_red = r_red.rbase
+                GTYPE = r_red.lowleveltype.TO
+                assert GTYPE._immutable_field(mangled_name), (
+                    "field %r must be declared as immutable" % name)
+                if not hasattr(driver, 'll_greenfields'):
+                    driver.ll_greenfields = {}
+                driver.ll_greenfields[name] = GTYPE, mangled_name
+                #
+                v_red = hop.inputarg(r_red, arg=i)
+                c_llname = hop.inputconst(lltype.Void, mangled_name)
+                v_green = hop.genop('getfield', [v_red, c_llname],
+                                    resulttype = r_field)
+                s_green = s_red.classdef.about_attribute(fieldname)
+                assert s_green is not None
+                hop.rtyper.annotator.setbinding(v_green, s_green)
             greens_v.append(v_green)
         for name in driver.reds:
             i = kwds_i['i_' + name]

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/objectmodel.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/objectmodel.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/objectmodel.py	Sat Oct 30 16:41:17 2010
@@ -475,6 +475,10 @@
     def setdefault(self, key, default):
         return self._dict.setdefault(_r_dictkey(self, key), default)
 
+    def popitem(self):
+        dk, value = self._dict.popitem()
+        return dk.key, value
+
     def copy(self):
         result = r_dict(self.key_eq, self.key_hash)
         result.update(self)

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/rbigint.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/rbigint.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/rbigint.py	Sat Oct 30 16:41:17 2010
@@ -1596,11 +1596,16 @@
 
 # a few internal helpers
 
-DEC_PER_DIGIT = 1
-while int('9' * DEC_PER_DIGIT) < MASK:
-    DEC_PER_DIGIT += 1
-DEC_PER_DIGIT -= 1
-DEC_MAX = 10 ** DEC_PER_DIGIT
+def digits_max_for_base(base):
+    dec_per_digit = 1
+    while base ** dec_per_digit < MASK:
+        dec_per_digit += 1
+    dec_per_digit -= 1
+    return base ** dec_per_digit
+
+BASE_MAX = [0, 0] + [digits_max_for_base(_base) for _base in range(2, 37)]
+DEC_MAX = digits_max_for_base(10)
+assert DEC_MAX == BASE_MAX[10]
 
 def _decimalstr_to_bigint(s):
     # a string that has been already parsed to be decimal and valid,
@@ -1615,7 +1620,6 @@
         p += 1
 
     a = rbigint.fromint(0)
-    cnt = DEC_PER_DIGIT
     tens = 1
     dig = 0
     ord0 = ord('0')
@@ -1627,8 +1631,26 @@
             a = _muladd1(a, tens, dig)
             tens = 1
             dig = 0
-    if sign:
+    if sign and a.sign == 1:
         a.sign = -1
     return a
 
-
+def parse_digit_string(parser):
+    # helper for objspace.std.strutil
+    a = rbigint.fromint(0)
+    base = parser.base
+    digitmax = BASE_MAX[base]
+    tens, dig = 1, 0
+    while True:
+        digit = parser.next_digit()
+        if tens == digitmax or digit < 0:
+            a = _muladd1(a, tens, dig)
+            if digit < 0:
+                break
+            dig = digit
+            tens = base
+        else:
+            dig = dig * base + digit
+            tens *= base
+    a.sign *= parser.sign
+    return a

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/rgc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/rgc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/rgc.py	Sat Oct 30 16:41:17 2010
@@ -170,7 +170,14 @@
         return hop.genop('gc_set_max_heap_size', [v_nbytes],
                          resulttype=lltype.Void)
 
-def can_move(p):    # NB. must not be called with NULL pointers
+def can_move(p):
+    """Check if the GC object 'p' is at an address that can move.
+    Must not be called with None.  With non-moving GCs, it is always False.
+    With some moving GCs like the SemiSpace GC, it is always True.
+    With other moving GCs like the MiniMark GC, it can be True for some
+    time, then False for the same object, when we are sure that it won't
+    move any more.
+    """
     return True
 
 class CanMoveEntry(ExtRegistryEntry):

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/rmmap.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/rmmap.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/rmmap.py	Sat Oct 30 16:41:17 2010
@@ -50,7 +50,7 @@
     constant_names = ['MAP_SHARED', 'MAP_PRIVATE',
                       'PROT_READ', 'PROT_WRITE',
                       'MS_SYNC']
-    opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS',
+    opt_constant_names = ['MAP_ANON', 'MAP_ANONYMOUS', 'MAP_NORESERVE',
                           'PROT_EXEC',
                           'MAP_DENYWRITE', 'MAP_EXECUTABLE']
     for name in constant_names:

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/rsre/rsre_char.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/rsre/rsre_char.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/rsre/rsre_char.py	Sat Oct 30 16:41:17 2010
@@ -4,6 +4,7 @@
 import sys
 from pypy.rlib.rlocale import tolower, isalnum
 from pypy.rlib.unroll import unrolling_iterable
+from pypy.rlib import jit
 
 # Note: the unicode parts of this module require you to call
 # rsre_char.set_unicode_db() first, to select one of the modules
@@ -43,6 +44,7 @@
 # XXX can we import those safely from sre_constants?
 SRE_INFO_PREFIX = 1
 SRE_INFO_LITERAL = 2
+SRE_INFO_CHARSET = 4
 SRE_FLAG_LOCALE = 4 # honour system locale
 SRE_FLAG_UNICODE = 32 # use unicode locale
 OPCODE_INFO = 17
@@ -64,33 +66,27 @@
 
 #### Category helpers
 
-ascii_char_info = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 6, 2,
-2, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0,
-0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 25, 25, 25, 25, 25, 25, 25, 25,
-25, 25, 0, 0, 0, 0, 0, 0, 0, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
-24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 0, 0,
-0, 0, 16, 0, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
-24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 0, 0, 0, 0, 0 ]
-
+is_a_word = [(chr(i).isalnum() or chr(i) == '_') for i in range(256)]
 linebreak = ord("\n")
 underline = ord("_")
 
 def is_digit(code):
-    return code < 128 and (ascii_char_info[code] & 1 != 0)
+    return code <= 57 and code >= 48
 
 def is_uni_digit(code):
     assert unicodedb is not None
     return unicodedb.isdigit(code)
 
 def is_space(code):
-    return code < 128 and (ascii_char_info[code] & 2 != 0)
+    return code == 32 or (code <= 13 and code >= 9)
 
 def is_uni_space(code):
     assert unicodedb is not None
     return unicodedb.isspace(code)
 
 def is_word(code):
-    return code < 128 and (ascii_char_info[code] & 16 != 0)
+    assert code >= 0
+    return code < 256 and is_a_word[code]
 
 def is_uni_word(code):
     assert unicodedb is not None
@@ -142,6 +138,7 @@
 SET_OK = -1
 SET_NOT_OK = -2
 
+ at jit.unroll_safe
 def check_charset(pattern, ppos, char_code):
     """Checks whether a character matches set of arbitrary length.
     The set starts at pattern[ppos]."""

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/rsre/rsre_core.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/rsre/rsre_core.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/rsre/rsre_core.py	Sat Oct 30 16:41:17 2010
@@ -1,9 +1,11 @@
 import sys
-from pypy.rlib.debug import check_nonneg
+from pypy.rlib.debug import check_nonneg, make_sure_not_modified
 from pypy.rlib.unroll import unrolling_iterable
 from pypy.rlib.rsre import rsre_char
 from pypy.tool.sourcetools import func_with_new_name
 from pypy.rlib.objectmodel import we_are_translated
+from pypy.rlib import jit
+from pypy.rlib.rsre.rsre_jit import install_jitdriver, install_jitdriver_spec
 
 
 OPCODE_FAILURE            = 0
@@ -56,16 +58,19 @@
     _seen_specname[specname] = True
     # Install a copy of the function under the name '_spec_funcname' in each
     # concrete subclass
+    specialized_methods = []
     for prefix, concreteclass in [('str', StrMatchContext),
                                   ('uni', UnicodeMatchContext)]:
         newfunc = func_with_new_name(func, prefix + specname)
         assert not hasattr(concreteclass, specname)
         setattr(concreteclass, specname, newfunc)
+        specialized_methods.append(newfunc)
     # Return a dispatcher function, specialized on the exact type of 'ctx'
     def dispatch(ctx, *args):
         return getattr(ctx, specname)(*args)
     dispatch._annspecialcase_ = 'specialize:argtype(0)'
-    return dispatch
+    dispatch._specialized_methods_ = specialized_methods
+    return func_with_new_name(dispatch, specname)
 
 # ____________________________________________________________
 
@@ -75,6 +80,7 @@
 
 class AbstractMatchContext(object):
     """Abstract base class"""
+    _immutable_fields_ = ['pattern[*]', 'flags', 'end']
     match_start = 0
     match_end = 0
     match_marks = None
@@ -164,6 +170,8 @@
     def __init__(self, pattern, string, match_start, end, flags):
         AbstractMatchContext.__init__(self, pattern, match_start, end, flags)
         self._string = string
+        if not we_are_translated() and isinstance(string, unicode):
+            self.flags |= rsre_char.SRE_FLAG_UNICODE   # for rsre_re.py
 
     def str(self, index):
         check_nonneg(index)
@@ -238,8 +246,9 @@
         self.start_ptr = ptr
         self.start_marks = marks
 
+    @jit.unroll_safe
     def find_first_result(self, ctx):
-        ppos = self.ppos
+        ppos = jit.hint(self.ppos, promote=True)
         while ctx.pat(ppos):
             result = sre_match(ctx, ppos + 1, self.start_ptr, self.start_marks)
             ppos += ctx.pat(ppos)
@@ -250,6 +259,10 @@
     find_next_result = find_first_result
 
 class RepeatOneMatchResult(MatchResult):
+    install_jitdriver('RepeatOne',
+                      greens=['nextppos', 'ctx.pattern'],
+                      reds=['ptr', 'self', 'ctx'],
+                      debugprint=(1, 0))   # indices in 'greens'
 
     def __init__(self, nextppos, minptr, ptr, marks):
         self.nextppos = nextppos
@@ -259,8 +272,11 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
+        nextppos = self.nextppos
         while ptr >= self.minptr:
-            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
+            ctx.jitdriver_RepeatOne.jit_merge_point(
+                self=self, ptr=ptr, ctx=ctx, nextppos=nextppos)
+            result = sre_match(ctx, nextppos, ptr, self.start_marks)
             ptr -= 1
             if result is not None:
                 self.subresult = result
@@ -270,6 +286,10 @@
 
 
 class MinRepeatOneMatchResult(MatchResult):
+    install_jitdriver('MinRepeatOne',
+                      greens=['nextppos', 'ppos3', 'ctx.pattern'],
+                      reds=['ptr', 'self', 'ctx'],
+                      debugprint=(2, 0))   # indices in 'greens'
 
     def __init__(self, nextppos, ppos3, maxptr, ptr, marks):
         self.nextppos = nextppos
@@ -280,29 +300,32 @@
 
     def find_first_result(self, ctx):
         ptr = self.start_ptr
+        nextppos = self.nextppos
+        ppos3 = self.ppos3
         while ptr <= self.maxptr:
-            result = sre_match(ctx, self.nextppos, ptr, self.start_marks)
+            ctx.jitdriver_MinRepeatOne.jit_merge_point(
+                self=self, ptr=ptr, ctx=ctx, nextppos=nextppos, ppos3=ppos3)
+            result = sre_match(ctx, nextppos, ptr, self.start_marks)
             if result is not None:
                 self.subresult = result
                 self.start_ptr = ptr
                 return self
-            if not self.next_char_ok(ctx, ptr):
+            if not self.next_char_ok(ctx, ptr, ppos3):
                 break
             ptr += 1
 
     def find_next_result(self, ctx):
         ptr = self.start_ptr
-        if not self.next_char_ok(ctx, ptr):
+        if not self.next_char_ok(ctx, ptr, self.ppos3):
             return
         self.start_ptr = ptr + 1
         return self.find_first_result(ctx)
 
-    def next_char_ok(self, ctx, ptr):
+    def next_char_ok(self, ctx, ptr, ppos):
         if ptr == ctx.end:
             return False
-        ppos = self.ppos3
         op = ctx.pat(ppos)
-        for op1, (checkerfn, _) in unroll_char_checker:
+        for op1, checkerfn in unroll_char_checker:
             if op1 == op:
                 return checkerfn(ctx, ptr, ppos)
         raise Error("next_char_ok[%d]" % op)
@@ -325,41 +348,34 @@
         self.next = next     # chained list
 
 class MaxUntilMatchResult(AbstractUntilMatchResult):
+    install_jitdriver('MaxUntil',
+                      greens=['ppos', 'tailppos', 'match_more', 'ctx.pattern'],
+                      reds=['ptr', 'marks', 'self', 'ctx'],
+                      debugprint=(3, 0, 2))
 
     def find_first_result(self, ctx):
-        enum = sre_match(ctx, self.ppos + 3, self.cur_ptr, self.cur_marks)
-        return self.search_next(ctx, enum, resume=False)
+        return self.search_next(ctx, match_more=True)
 
     def find_next_result(self, ctx):
-        return self.search_next(ctx, None, resume=True)
+        return self.search_next(ctx, match_more=False)
 
-    def search_next(self, ctx, enum, resume):
+    def search_next(self, ctx, match_more):
         ppos = self.ppos
-        min = ctx.pat(ppos+1)
-        max = ctx.pat(ppos+2)
+        tailppos = self.tailppos
         ptr = self.cur_ptr
         marks = self.cur_marks
         while True:
-            while True:
-                if (enum is not None and
-                    (ptr != ctx.match_end or self.num_pending < min)):
-                    #               ^^^^^^^^^^ zero-width match protection
-                    # matched one more 'item'.  record it and continue.
-                    self.pending = Pending(ptr, marks, enum, self.pending)
-                    self.num_pending += 1
-                    ptr = ctx.match_end
-                    marks = ctx.match_marks
-                    break
-                # 'item' no longer matches.
-                if not resume and self.num_pending >= min:
-                    # try to match 'tail' if we have enough 'item'
-                    result = sre_match(ctx, self.tailppos, ptr, marks)
-                    if result is not None:
-                        self.subresult = result
-                        self.cur_ptr = ptr
-                        self.cur_marks = marks
-                        return self
-                resume = False
+            ctx.jitdriver_MaxUntil.jit_merge_point(
+                ppos=ppos, tailppos=tailppos, match_more=match_more,
+                ptr=ptr, marks=marks, self=self, ctx=ctx)
+            if match_more:
+                max = ctx.pat(ppos+2)
+                if max == 65535 or self.num_pending < max:
+                    # try to match one more 'item'
+                    enum = sre_match(ctx, ppos + 3, ptr, marks)
+                else:
+                    enum = None    # 'max' reached, no more matches
+            else:
                 p = self.pending
                 if p is None:
                     return
@@ -369,11 +385,27 @@
                 marks = p.marks
                 enum = p.enum.move_to_next_result(ctx)
             #
-            if max == 65535 or self.num_pending < max:
-                # try to match one more 'item'
-                enum = sre_match(ctx, ppos + 3, ptr, marks)
+            min = ctx.pat(ppos+1)
+            if (enum is not None and
+                (ptr != ctx.match_end or self.num_pending < min)):
+                #               ^^^^^^^^^^ zero-width match protection
+                # matched one more 'item'.  record it and continue.
+                self.pending = Pending(ptr, marks, enum, self.pending)
+                self.num_pending += 1
+                ptr = ctx.match_end
+                marks = ctx.match_marks
+                match_more = True
             else:
-                enum = None    # 'max' reached, no more matches
+                # 'item' no longer matches.
+                if self.num_pending >= min:
+                    # try to match 'tail' if we have enough 'item'
+                    result = sre_match(ctx, tailppos, ptr, marks)
+                    if result is not None:
+                        self.subresult = result
+                        self.cur_ptr = ptr
+                        self.cur_marks = marks
+                        return self
+                match_more = False
 
 class MinUntilMatchResult(AbstractUntilMatchResult):
 
@@ -384,6 +416,7 @@
         return self.search_next(ctx, resume=True)
 
     def search_next(self, ctx, resume):
+        # XXX missing jit support here
         ppos = self.ppos
         min = ctx.pat(ppos+1)
         max = ctx.pat(ppos+2)
@@ -429,6 +462,7 @@
 # ____________________________________________________________
 
 @specializectx
+ at jit.unroll_safe
 def sre_match(ctx, ppos, ptr, marks):
     """Returns either None or a MatchResult object.  Usually we only need
     the first result, but there is the case of REPEAT...UNTIL where we
@@ -437,6 +471,13 @@
     while True:
         op = ctx.pat(ppos)
         ppos += 1
+        make_sure_not_modified(ctx.pattern)
+
+        #jit.jit_debug("sre_match", op, ppos, ptr)
+        #
+        # When using the JIT, calls to sre_match() must always have a constant
+        # (green) argument for 'ppos'.  If not, the following assert fails.
+        jit.assert_green(op)
 
         if op == OPCODE_FAILURE:
             return
@@ -712,13 +753,23 @@
 @specializectx
 def find_repetition_end(ctx, ppos, ptr, maxcount):
     end = ctx.end
-    # adjust end
-    if maxcount != 65535:
+    if maxcount <= 1:
+        if maxcount == 1 and ptr < end:
+            # Relatively common case: maxcount == 1.  If we are not at the
+            # end of the string, it's done by a single direct check.
+            op = ctx.pat(ppos)
+            for op1, checkerfn in unroll_char_checker:
+                if op1 == op:
+                    if checkerfn(ctx, ptr, ppos):
+                        return ptr + 1
+        return ptr
+    elif maxcount != 65535:
+        # adjust end
         end1 = ptr + maxcount
         if end1 <= end:
             end = end1
     op = ctx.pat(ppos)
-    for op1, (_, fre) in unroll_char_checker:
+    for op1, fre in unroll_fre_checker:
         if op1 == op:
             return fre(ctx, ptr, end, ppos)
     raise Error("rsre.find_repetition_end[%d]" % op)
@@ -751,23 +802,60 @@
     if checkerfn == match_ANY_ALL:
         def fre(ctx, ptr, end, ppos):
             return end
+    elif checkerfn == match_IN:
+        install_jitdriver_spec('MatchIn', 
+                               greens=['ppos', 'ctx.pattern'],
+                               reds=['ptr', 'end', 'ctx'],
+                               debugprint=(1, 0))
+        @specializectx
+        def fre(ctx, ptr, end, ppos):
+            while True:
+                ctx.jitdriver_MatchIn.jit_merge_point(ctx=ctx, ptr=ptr,
+                                                      end=end, ppos=ppos)
+                if ptr < end and checkerfn(ctx, ptr, ppos):
+                    ptr += 1
+                else:
+                    return ptr
+    elif checkerfn == match_IN_IGNORE:
+        install_jitdriver_spec('MatchInIgnore', 
+                               greens=['ppos', 'ctx.pattern'],
+                               reds=['ptr', 'end', 'ctx'],
+                               debugprint=(1, 0))
+        @specializectx
+        def fre(ctx, ptr, end, ppos):
+            while True:
+                ctx.jitdriver_MatchInIgnore.jit_merge_point(ctx=ctx, ptr=ptr,
+                                                            end=end, ppos=ppos)
+                if ptr < end and checkerfn(ctx, ptr, ppos):
+                    ptr += 1
+                else:
+                    return ptr
     else:
+        # in the other cases, the fre() function is not JITted at all
+        # and is present as a residual call.
+        @specializectx
         def fre(ctx, ptr, end, ppos):
             while ptr < end and checkerfn(ctx, ptr, ppos):
                 ptr += 1
             return ptr
-    return checkerfn, fre
+    fre = func_with_new_name(fre, 'fre_' + checkerfn.__name__)
+    return fre
+
+unroll_char_checker = [
+    (OPCODE_ANY,                match_ANY),
+    (OPCODE_ANY_ALL,            match_ANY_ALL),
+    (OPCODE_IN,                 match_IN),
+    (OPCODE_IN_IGNORE,          match_IN_IGNORE),
+    (OPCODE_LITERAL,            match_LITERAL),
+    (OPCODE_LITERAL_IGNORE,     match_LITERAL_IGNORE),
+    (OPCODE_NOT_LITERAL,        match_NOT_LITERAL),
+    (OPCODE_NOT_LITERAL_IGNORE, match_NOT_LITERAL_IGNORE),
+    ]
+unroll_fre_checker = [(_op, _make_fre(_fn))
+                      for (_op, _fn) in unroll_char_checker]
 
-unroll_char_checker = unrolling_iterable([
-    (OPCODE_ANY,                _make_fre(match_ANY)),
-    (OPCODE_ANY_ALL,            _make_fre(match_ANY_ALL)),
-    (OPCODE_IN,                 _make_fre(match_IN)),
-    (OPCODE_IN_IGNORE,          _make_fre(match_IN_IGNORE)),
-    (OPCODE_LITERAL,            _make_fre(match_LITERAL)),
-    (OPCODE_LITERAL_IGNORE,     _make_fre(match_LITERAL_IGNORE)),
-    (OPCODE_NOT_LITERAL,        _make_fre(match_NOT_LITERAL)),
-    (OPCODE_NOT_LITERAL_IGNORE, _make_fre(match_NOT_LITERAL_IGNORE)),
-    ])
+unroll_char_checker = unrolling_iterable(unroll_char_checker)
+unroll_fre_checker  = unrolling_iterable(unroll_fre_checker)
 
 ##### At dispatch
 
@@ -873,74 +961,139 @@
     else:
         return None
 
+install_jitdriver('Match',
+                  greens=['ctx.pattern'], reds=['ctx'],
+                  debugprint=(0,))
+
 def match_context(ctx):
     ctx.original_pos = ctx.match_start
     if ctx.end < ctx.match_start:
         return False
+    ctx.jitdriver_Match.jit_merge_point(ctx=ctx)
     return sre_match(ctx, 0, ctx.match_start, None) is not None
 
 def search_context(ctx):
     ctx.original_pos = ctx.match_start
     if ctx.end < ctx.match_start:
         return False
-    if ctx.pat(0) == OPCODE_INFO:
-        if ctx.pat(2) & rsre_char.SRE_INFO_PREFIX and ctx.pat(5) > 1:
-            return fast_search(ctx)
-    return regular_search(ctx)
+    base = 0
+    charset = False
+    if ctx.pat(base) == OPCODE_INFO:
+        flags = ctx.pat(2)
+        if flags & rsre_char.SRE_INFO_PREFIX:
+            if ctx.pat(5) > 1:
+                return fast_search(ctx)
+        else:
+            charset = (flags & rsre_char.SRE_INFO_CHARSET)
+        base += 1 + ctx.pat(1)
+    if ctx.pat(base) == OPCODE_LITERAL:
+        return literal_search(ctx, base)
+    if charset:
+        return charset_search(ctx, base)
+    return regular_search(ctx, base)
+
+install_jitdriver('RegularSearch',
+                  greens=['base', 'ctx.pattern'],
+                  reds=['start', 'ctx'],
+                  debugprint=(1, 0))
 
-def regular_search(ctx):
+def regular_search(ctx, base):
     start = ctx.match_start
     while start <= ctx.end:
-        if sre_match(ctx, 0, start, None) is not None:
+        ctx.jitdriver_RegularSearch.jit_merge_point(ctx=ctx, start=start,
+                                                    base=base)
+        if sre_match(ctx, base, start, None) is not None:
             ctx.match_start = start
             return True
         start += 1
     return False
 
+install_jitdriver_spec("LiteralSearch",
+                       greens=['base', 'character', 'ctx.pattern'],
+                       reds=['start', 'ctx'],
+                       debugprint=(2, 0, 1))
+ at specializectx
+def literal_search(ctx, base):
+    # pattern starts with a literal character.  this is used
+    # for short prefixes, and if fast search is disabled
+    character = ctx.pat(base + 1)
+    base += 2
+    start = ctx.match_start
+    while start < ctx.end:
+        ctx.jitdriver_LiteralSearch.jit_merge_point(ctx=ctx, start=start,
+                                          base=base, character=character)
+        if ctx.str(start) == character:
+            if sre_match(ctx, base, start + 1, None) is not None:
+                ctx.match_start = start
+                return True
+        start += 1
+    return False
+
+install_jitdriver_spec("CharsetSearch",
+                       greens=['base', 'ctx.pattern'],
+                       reds=['start', 'ctx'],
+                       debugprint=(1, 0))
+ at specializectx
+def charset_search(ctx, base):
+    # pattern starts with a character from a known set
+    start = ctx.match_start
+    while start < ctx.end:
+        ctx.jitdriver_CharsetSearch.jit_merge_point(ctx=ctx, start=start,
+                                                    base=base)
+        if rsre_char.check_charset(ctx.pattern, 5, ctx.str(start)):
+            if sre_match(ctx, base, start, None) is not None:
+                ctx.match_start = start
+                return True
+        start += 1
+    return False
+
+install_jitdriver_spec('FastSearch',
+                       greens=['i', 'prefix_len', 'ctx.pattern'],
+                       reds=['string_position', 'ctx'],
+                       debugprint=(2, 0))
 @specializectx
 def fast_search(ctx):
     # skips forward in a string as fast as possible using information from
     # an optimization info block
     #  <1=skip> <2=flags> <3=min> <4=...>
     #        <5=length> <6=skip> <7=prefix data> 
-    flags = ctx.pat(2)
+    string_position = ctx.match_start
+    if string_position >= ctx.end:
+        return False
     prefix_len = ctx.pat(5)
     assert prefix_len >= 0
-    prefix_skip = ctx.pat(6)
-    assert prefix_skip >= 0
-    overlap_offset = 7 + prefix_len - 1
-    assert overlap_offset >= 0
-    pattern_offset = ctx.pat(1) + 1
-    ppos_start = pattern_offset + 2 * prefix_skip
-    assert ppos_start >= 0
     i = 0
-    string_position = ctx.match_start
-    end = ctx.end
-    while string_position < end:
-        while True:
-            char_ord = ctx.str(string_position)
-            if char_ord != ctx.pat(7 + i):
-                if i == 0:
-                    break
-                else:
-                    i = ctx.pat(overlap_offset + i)
-            else:
-                i += 1
-                if i == prefix_len:
-                    # found a potential match
-                    start = string_position + 1 - prefix_len
-                    assert start >= 0
-                    ptr = start + prefix_skip
-                    if flags & rsre_char.SRE_INFO_LITERAL:
-                        # matched all of pure literal pattern
-                        ctx.match_start = start
-                        ctx.match_end = ptr
-                        ctx.match_marks = None
-                        return True
-                    if sre_match(ctx, ppos_start, ptr, None) is not None:
-                        ctx.match_start = start
-                        return True
-                    i = ctx.pat(overlap_offset + i)
-                break
+    while True:
+        ctx.jitdriver_FastSearch.jit_merge_point(ctx=ctx,
+                string_position=string_position, i=i, prefix_len=prefix_len)
+        char_ord = ctx.str(string_position)
+        if char_ord != ctx.pat(7 + i):
+            if i > 0:
+                overlap_offset = prefix_len + (7 - 1)
+                i = ctx.pat(overlap_offset + i)
+                continue
+        else:
+            i += 1
+            if i == prefix_len:
+                # found a potential match
+                start = string_position + 1 - prefix_len
+                assert start >= 0
+                prefix_skip = ctx.pat(6)
+                ptr = start + prefix_skip
+                #flags = ctx.pat(2)
+                #if flags & rsre_char.SRE_INFO_LITERAL:
+                #    # matched all of pure literal pattern
+                #    ctx.match_start = start
+                #    ctx.match_end = ptr
+                #    ctx.match_marks = None
+                #    return True
+                pattern_offset = ctx.pat(1) + 1
+                ppos_start = pattern_offset + 2 * prefix_skip
+                if sre_match(ctx, ppos_start, ptr, None) is not None:
+                    ctx.match_start = start
+                    return True
+                overlap_offset = prefix_len + (7 - 1)
+                i = ctx.pat(overlap_offset + i)
         string_position += 1
-    return False
+        if string_position >= ctx.end:
+            return False

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/rsre/test/test_match.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/rsre/test/test_match.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/rsre/test/test_match.py	Sat Oct 30 16:41:17 2010
@@ -1,24 +1,49 @@
-import _sre, re, sre_compile
-from pypy.rlib.rsre import rsre_core
+import re
+from pypy.rlib.rsre import rsre_core, rsre_char
 
 
-def get_code(regexp, flags=0, allargs=False):
-    class GotIt(Exception):
-        pass
-    def my_compile(pattern, flags, code, *args):
-        print code
-        raise GotIt(code, flags, args)
-    saved = _sre.compile
-    try:
-        _sre.compile = my_compile
-        try:
-            sre_compile.compile(regexp, flags)
-        except GotIt, e:
-            pass
+def get_hacked_sre_compile(my_compile):
+    """Return a copy of the sre_compile module for which the _sre
+    module is a custom module that has _sre.compile == my_compile
+    and CODESIZE == rsre_char.CODESIZE.
+    """
+    import sre_compile, __builtin__, new
+    sre_hacked = new.module("_sre_hacked")
+    sre_hacked.compile = my_compile
+    sre_hacked.MAGIC = sre_compile.MAGIC
+    sre_hacked.CODESIZE = rsre_char.CODESIZE
+    sre_hacked.getlower = rsre_char.getlower
+    def my_import(name, *args):
+        if name == '_sre':
+            return sre_hacked
         else:
-            raise ValueError("did not reach _sre.compile()!")
+            return default_import(name, *args)
+    src = sre_compile.__file__
+    if src.lower().endswith('.pyc') or src.lower().endswith('.pyo'):
+        src = src[:-1]
+    mod = new.module("sre_compile_hacked")
+    default_import = __import__
+    try:
+        __builtin__.__import__ = my_import
+        execfile(src, mod.__dict__)
     finally:
-        _sre.compile = saved
+        __builtin__.__import__ = default_import
+    return mod
+
+class GotIt(Exception):
+    pass
+def my_compile(pattern, flags, code, *args):
+    print code
+    raise GotIt(code, flags, args)
+sre_compile_hacked = get_hacked_sre_compile(my_compile)
+
+def get_code(regexp, flags=0, allargs=False):
+    try:
+        sre_compile_hacked.compile(regexp, flags)
+    except GotIt, e:
+        pass
+    else:
+        raise ValueError("did not reach _sre.compile()!")
     if allargs:
         return e.args
     else:

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/rstring.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/rstring.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/rstring.py	Sat Oct 30 16:41:17 2010
@@ -54,6 +54,7 @@
         self.l = []
 
     def append(self, s):
+        assert isinstance(s, self._type)
         self.l.append(s)
 
     def append_slice(self, s, start, end):
@@ -63,11 +64,16 @@
     def append_multiple_char(self, c, times):
         self.l.append(c * times)
 
+    def getlength(self):
+        return len(self.build())
+
 class StringBuilder(AbstractStringBuilder):
+    _type = str
     def build(self):
         return "".join(self.l)
 
 class UnicodeBuilder(AbstractStringBuilder):
+    _type = unicode
     def build(self):
         return u''.join(self.l)
 
@@ -121,9 +127,12 @@
         assert s_times.nonneg
         return s_None
 
+    def method_getlength(self):
+        return SomeInteger(nonneg=True)
+
     def method_build(self):
         return SomeString()
-    
+
     def rtyper_makerepr(self, rtyper):
         return rtyper.type_system.rbuilder.stringbuilder_repr
 
@@ -146,6 +155,9 @@
         assert s_times.nonneg
         return s_None
 
+    def method_getlength(self):
+        return SomeInteger(nonneg=True)
+
     def method_build(self):
         return SomeUnicodeString()
     

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/test/test_jit.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/test/test_jit.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/test/test_jit.py	Sat Oct 30 16:41:17 2010
@@ -1,10 +1,17 @@
 import py
+from pypy.conftest import option
 from pypy.rlib.jit import hint, we_are_jitted, JitDriver, purefunction_promote
-from pypy.rlib.jit import JitHintError
+from pypy.rlib.jit import JitHintError, oopspec
 from pypy.translator.translator import TranslationContext, graphof
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.rpython.lltypesystem import lltype
 
+def test_oopspec():
+    @oopspec('foobar')
+    def fn():
+        pass
+    assert fn.oopspec == 'foobar'
+    
 class BaseTestJIT(BaseRtypingTest):
     def test_hint(self):
         def f():
@@ -104,6 +111,26 @@
             return n
         py.test.raises(JitHintError, self.gengraph, fn, [int])
 
+    def test_green_field(self):
+        def get_printable_location(xfoo):
+            return str(ord(xfoo))   # xfoo must be annotated as a character
+        myjitdriver = JitDriver(greens=['x.foo'], reds=['n', 'x'],
+                                get_printable_location=get_printable_location)
+        class A(object):
+            _immutable_fields_ = ['foo']
+        def fn(n):
+            x = A()
+            x.foo = chr(n)
+            while n > 0:
+                myjitdriver.can_enter_jit(x=x, n=n)
+                myjitdriver.jit_merge_point(x=x, n=n)
+                n -= 1
+            return n
+        t = self.gengraph(fn, [int])[0]
+        if option.view:
+            t.view()
+        # assert did not raise
+
 
 class TestJITLLtype(BaseTestJIT, LLRtypeMixin):
     pass

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rbigint.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rbigint.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rbigint.py	Sat Oct 30 16:41:17 2010
@@ -112,6 +112,23 @@
         assert rbigint.fromrarith_int(r_uint(2*sys.maxint+1)).eq(
             rbigint.fromlong(2*sys.maxint+1))
 
+    def test_fromdecimalstr(self):
+        x = rbigint.fromdecimalstr("12345678901234567890523897987")
+        assert x.tolong() == 12345678901234567890523897987L
+        assert x.tobool() is True
+        x = rbigint.fromdecimalstr("+12345678901234567890523897987")
+        assert x.tolong() == 12345678901234567890523897987L
+        assert x.tobool() is True
+        x = rbigint.fromdecimalstr("-12345678901234567890523897987")
+        assert x.tolong() == -12345678901234567890523897987L
+        assert x.tobool() is True
+        x = rbigint.fromdecimalstr("+0")
+        assert x.tolong() == 0
+        assert x.tobool() is False
+        x = rbigint.fromdecimalstr("-0")
+        assert x.tolong() == 0
+        assert x.tobool() is False
+
     def test_add(self):
         x = 123456789123456789000000L
         y = 123858582373821923936744221L
@@ -448,6 +465,35 @@
         assert (rbigint.fromlong(-9**50).ulonglongmask() ==
                 r_ulonglong(-9**50))
 
+    def test_parse_digit_string(self):
+        from pypy.rlib.rbigint import parse_digit_string
+        class Parser:
+            def __init__(self, base, sign, digits):
+                self.base = base
+                self.sign = sign
+                self.next_digit = iter(digits + [-1]).next
+        x = parse_digit_string(Parser(10, 1, [6]))
+        assert x.eq(rbigint.fromint(6))
+        x = parse_digit_string(Parser(10, 1, [6, 2, 3]))
+        assert x.eq(rbigint.fromint(623))
+        x = parse_digit_string(Parser(10, -1, [6, 2, 3]))
+        assert x.eq(rbigint.fromint(-623))
+        x = parse_digit_string(Parser(16, 1, [0xA, 0x4, 0xF]))
+        assert x.eq(rbigint.fromint(0xA4F))
+        num = 0
+        for i in range(36):
+            x = parse_digit_string(Parser(36, 1, range(i)))
+            assert x.eq(rbigint.fromlong(num))
+            num = num * 36 + i
+        x = parse_digit_string(Parser(16, -1, range(15,-1,-1)*99))
+        assert x.eq(rbigint.fromlong(long('-0x' + 'FEDCBA9876543210'*99, 16)))
+        assert x.tobool() is True
+        x = parse_digit_string(Parser(7, 1, [0, 0, 0]))
+        assert x.tobool() is False
+        x = parse_digit_string(Parser(7, -1, [0, 0, 0]))
+        assert x.tobool() is False
+
+
 BASE = 2 ** SHIFT
 
 class TestTranslatable(object):

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rdynload.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rdynload.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rdynload.py	Sat Oct 30 16:41:17 2010
@@ -1,15 +1,22 @@
 from pypy.rlib.rdynload import *
-from pypy.rlib.libffi import get_libc_name
+from pypy.rlib.clibffi import get_libc_name
 from pypy.rpython.lltypesystem import rffi, lltype
 import py
 
 class TestDLOperations:
     def test_dlopen(self):
-        py.test.raises(DLOpenError, "dlopen(rffi.str2charp('xxxxxxxxxxxx'))")
-        assert dlopen(rffi.str2charp(get_libc_name()))
+        s = rffi.str2charp('xxxxxxxxxxxx')
+        py.test.raises(DLOpenError, "dlopen(s)")
+        rffi.free_charp(s)
+        #
+        s = rffi.str2charp(get_libc_name())
+        assert dlopen(s)
+        rffi.free_charp(s)
 
     def test_dlsym(self):
-        lib = dlopen(rffi.str2charp(get_libc_name()))
+        s = rffi.str2charp(get_libc_name())
+        lib = dlopen(s)
+        rffi.free_charp(s)
         handle = rffi.cast(lltype.Ptr(lltype.FuncType([lltype.Signed],
                            lltype.Signed)), dlsym(lib, 'abs'))
         assert 1 == handle(1)

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rsocket.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rsocket.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rsocket.py	Sat Oct 30 16:41:17 2010
@@ -437,3 +437,31 @@
             foo = self.serv.accept()
         py.test.raises(SocketError, raise_error)
 
+def _test_cond_include(cond):
+    # Test that _rsocket_rffi is importable even on platforms where
+    # AF_PACKET or AF_NETLINK is not defined.
+    import re
+    from pypy.rlib import _rsocket_rffi
+    srcfile = _rsocket_rffi.__file__
+    if srcfile.lower().endswith('c') or srcfile.lower().endswith('o'):
+        srcfile = srcfile[:-1]      # .pyc => .py
+    assert srcfile.lower().endswith('.py')
+    sourcelines = open(srcfile, 'rb').read().splitlines()
+    found = False
+    for i, line in enumerate(sourcelines):
+        line2 = re.sub(r"(\s*COND_HEADER\s*=)",
+                      r"\1'#undef %s\\n'+" % cond,
+                      line)
+        if line2 != line:
+            found = True
+            sourcelines[i] = line2
+    assert found
+    d = {}
+    sourcelines.append('')
+    exec '\n'.join(sourcelines) in d
+
+def test_no_AF_PACKET():
+    _test_cond_include('AF_PACKET')
+
+def test_no_AF_NETLINK():
+    _test_cond_include('AF_NETLINK')

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rstring.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rstring.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rstring.py	Sat Oct 30 16:41:17 2010
@@ -29,6 +29,7 @@
     s = StringBuilder()
     s.append("a")
     s.append("abc")
+    assert s.getlength() == len('aabc')
     s.append("a")
     s.append_slice("abc", 1, 2)
     s.append_multiple_char('d', 4)
@@ -39,6 +40,7 @@
     s.append(u'a')
     s.append(u'abc')
     s.append_slice(u'abcdef', 1, 2)
+    assert s.getlength() == len('aabcb')
     s.append_multiple_char('d', 4)
     assert s.build() == 'aabcbdddd'
     assert isinstance(s.build(), unicode)

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/test/test_runicode.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/test/test_runicode.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/test/test_runicode.py	Sat Oct 30 16:41:17 2010
@@ -76,7 +76,7 @@
                 assert start == startingpos
                 assert stop == endingpos
                 return u"42424242", stop
-            return "", endingpos
+            return u"", endingpos
         decoder = self.getdecoder(encoding)
         if addstuff:
             s += "some rest in ascii"

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rzlib.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rzlib.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/test/test_rzlib.py	Sat Oct 30 16:41:17 2010
@@ -189,6 +189,8 @@
     assert unused3 == len('more_garbage')
     assert data3 == ''
 
+    rzlib.deflateEnd(stream)
+
 
 def test_decompress_max_length():
     """
@@ -205,6 +207,8 @@
     assert finished2 is True
     assert unused2 == 0
 
+    rzlib.deflateEnd(stream)
+
 
 def test_cornercases():
     """
@@ -215,6 +219,7 @@
     bytes += rzlib.compress(stream, "")
     bytes += rzlib.compress(stream, "", rzlib.Z_FINISH)
     assert zlib.decompress(bytes) == ""
+    rzlib.deflateEnd(stream)
 
     stream = rzlib.inflateInit()
     data, finished, unused = rzlib.decompress(stream, "")
@@ -228,3 +233,4 @@
         assert finished is False
         assert unused > 0
         buf = buf[-unused:]
+    rzlib.deflateEnd(stream)

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/llinterp.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/llinterp.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/llinterp.py	Sat Oct 30 16:41:17 2010
@@ -48,8 +48,7 @@
 
     current_interpreter = None
 
-    def __init__(self, typer, tracing=True, exc_data_ptr=None,
-                 malloc_check=True):
+    def __init__(self, typer, tracing=True, exc_data_ptr=None):
         self.bindings = {}
         self.typer = typer
         # 'heap' is module or object that provides malloc, etc for lltype ops
@@ -57,9 +56,7 @@
         self.exc_data_ptr = exc_data_ptr
         self.frame_stack = []
         self.tracer = None
-        self.malloc_check = malloc_check
         self.frame_class = LLFrame
-        self.mallocs = {}
         if tracing:
             self.tracer = Tracer()
 
@@ -163,24 +160,6 @@
             return self.exc_data_ptr
         return None
 
-    def remember_malloc(self, ptr, llframe):
-        # err....
-        self.mallocs[ptr._obj] = llframe
-
-    def remember_free(self, ptr):
-        try:
-            del self.mallocs[ptr._obj]
-        except KeyError:
-            self._rehash_mallocs()
-            del self.mallocs[ptr._obj]
-
-    def _rehash_mallocs(self):
-        # rehashing is needed because some objects' hash may change
-        # when being turned to 
-        items = self.mallocs.items()
-        self.mallocs = {}
-        self.mallocs.update(items)
-
     def _store_exception(self, exc):
         raise PleaseOverwriteStoreException("You just invoked ll2ctypes callback without overwriting _store_exception on llinterpreter")
 
@@ -726,23 +705,23 @@
     def op_malloc(self, obj, flags):
         flavor = flags['flavor']
         zero = flags.get('zero', False)
+        track_allocation = flags.get('track_allocation', True)
         if flavor == "stack":
             result = self.heap.malloc(obj, zero=zero, flavor='raw')
             self.alloca_objects.append(result)
             return result
-        ptr = self.heap.malloc(obj, zero=zero, flavor=flavor)
-        if flavor == 'raw' and self.llinterpreter.malloc_check:
-            self.llinterpreter.remember_malloc(ptr, self)
+        ptr = self.heap.malloc(obj, zero=zero, flavor=flavor,
+                               track_allocation=track_allocation)
         return ptr
 
     def op_malloc_varsize(self, obj, flags, size):
         flavor = flags['flavor']
         zero = flags.get('zero', False)
+        track_allocation = flags.get('track_allocation', True)
         assert flavor in ('gc', 'raw')
         try:
-            ptr = self.heap.malloc(obj, size, zero=zero, flavor=flavor)
-            if flavor == 'raw' and self.llinterpreter.malloc_check:
-                self.llinterpreter.remember_malloc(ptr, self)
+            ptr = self.heap.malloc(obj, size, zero=zero, flavor=flavor,
+                                   track_allocation=track_allocation)
             return ptr
         except MemoryError:
             self.make_llexception()
@@ -759,11 +738,10 @@
         zero = flags.get('zero', False)
         return self.heap.malloc_nonmovable(TYPE, size, zero=zero)
 
-    def op_free(self, obj, flavor):
-        assert isinstance(flavor, str)
-        if flavor == 'raw' and self.llinterpreter.malloc_check:
-            self.llinterpreter.remember_free(obj)
-        self.heap.free(obj, flavor=flavor)
+    def op_free(self, obj, flags):
+        assert flags['flavor'] == 'raw'
+        track_allocation = flags.get('track_allocation', True)
+        self.heap.free(obj, flavor='raw', track_allocation=track_allocation)
 
     def op_shrink_array(self, obj, smallersize):
         return self.heap.shrink_array(obj, smallersize)
@@ -1037,6 +1015,13 @@
     def op_stack_malloc(self, size): # mmh
         raise NotImplementedError("backend only")
 
+    def op_track_alloc_start(self, addr):
+        # we don't do tracking at this level
+        checkadr(addr)
+
+    def op_track_alloc_stop(self, addr):
+        checkadr(addr)
+
     # ____________________________________________________________
     # Overflow-detecting variants
 

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/ll2ctypes.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/ll2ctypes.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/ll2ctypes.py	Sat Oct 30 16:41:17 2010
@@ -69,7 +69,7 @@
                 PIECESIZE = 0x08000000
         PIECES = 10
         m = rmmap.mmap(-1, PIECES * PIECESIZE,
-                       rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS,
+                       rmmap.MAP_PRIVATE|rmmap.MAP_ANONYMOUS|rmmap.MAP_NORESERVE,
                        rmmap.PROT_READ|rmmap.PROT_WRITE)
         m.close = lambda : None    # leak instead of giving a spurious
                                    # error at CPython's shutdown
@@ -823,6 +823,8 @@
         except (ValueError, OverflowError):
             for tc in 'HIL':
                 if array(tc).itemsize == array('u').itemsize:
+                    import struct
+                    cobj &= 256 ** struct.calcsize(tc) - 1
                     llobj = array('u', array(tc, (cobj,)).tostring())[0]
                     break
             else:

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/llarena.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/llarena.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/llarena.py	Sat Oct 30 16:41:17 2010
@@ -440,7 +440,7 @@
                                    [rffi.INT],
                                    rffi.INT,
                                    sandboxsafe=True, _nowrapper=True)
-    _dev_zero = rffi.str2charp('/dev/zero')   # prebuilt
+    _dev_zero = rffi.str2charp_immortal('/dev/zero')   # prebuilt
 
     def clear_large_memory_chunk(baseaddr, size):
         # on some Unixy platforms, reading from /dev/zero is the fastest way

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/llmemory.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/llmemory.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/llmemory.py	Sat Oct 30 16:41:17 2010
@@ -105,11 +105,13 @@
         if (isinstance(self.TYPE, lltype.ContainerType)
             and self.TYPE._gckind == 'gc'):
             assert self.repeat == 1
-            p = lltype.malloc(self.TYPE, flavor='raw', zero=zero)
+            p = lltype.malloc(self.TYPE, flavor='raw', zero=zero,
+                              track_allocation=False)
             return cast_ptr_to_adr(p)
         else:
             T = lltype.FixedSizeArray(self.TYPE, self.repeat)
-            p = lltype.malloc(T, flavor='raw', zero=zero)
+            p = lltype.malloc(T, flavor='raw', zero=zero,
+                              track_allocation=False)
             array_adr = cast_ptr_to_adr(p)
             return array_adr + ArrayItemsOffset(T)
 
@@ -288,7 +290,8 @@
             count = 0
         p = lltype.malloc(parenttype or self.TYPE, count,
                           immortal = self.TYPE._gckind == 'raw',
-                          zero = zero)
+                          zero = zero,
+                          track_allocation = False)
         return cast_ptr_to_adr(p)
 
     def raw_memcopy(self, srcadr, dstadr):

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/lloperation.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/lloperation.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/lloperation.py	Sat Oct 30 16:41:17 2010
@@ -403,6 +403,8 @@
     'raw_load':             LLOp(sideeffects=False),
     'raw_store':            LLOp(),
     'stack_malloc':         LLOp(), # mmh
+    'track_alloc_start':    LLOp(),
+    'track_alloc_stop':     LLOp(),
     'adr_add':              LLOp(canfold=True),
     'adr_sub':              LLOp(canfold=True),
     'adr_delta':            LLOp(canfold=True),

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/lltype.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/lltype.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/lltype.py	Sat Oct 30 16:41:17 2010
@@ -1,7 +1,3 @@
-import StringIO
-import traceback
-import sys
-
 import py
 from pypy.rlib.rarithmetic import (r_int, r_uint, intmask, r_singlefloat,
                                    r_ulonglong, r_longlong, base_int,
@@ -10,25 +6,13 @@
 from pypy.tool.uid import Hashable
 from pypy.tool.tls import tlsobject
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 from types import NoneType
 from sys import maxint
 import weakref
 
 TLS = tlsobject()
 
-# Track allocations to detect memory leaks
-# Don't track 'gc' and immortal mallocs
-TRACK_ALLOCATIONS = False
-ALLOCATED = identity_dict()
-
-def start_tracking_allocations():
-    global TRACK_ALLOCATIONS
-    TRACK_ALLOCATIONS = True
-    ALLOCATED.clear()
-
-def stop_tracking_allocations():
-    global TRACK_ALLOCATIONS
-    TRACK_ALLOCATIONS = False
 
 class _uninitialized(object):
     def __init__(self, TYPE):
@@ -794,6 +778,8 @@
             return llmemory.cast_adr_to_ptr(value, TGT)
     elif TGT == llmemory.Address and isinstance(ORIG, Ptr):
         return llmemory.cast_ptr_to_adr(value)
+    elif TGT == Signed and isinstance(ORIG, Ptr) and ORIG.TO._gckind == 'raw':
+        return llmemory.cast_adr_to_int(llmemory.cast_ptr_to_adr(value), 'symbolic')
     raise TypeError("don't know how to cast from %r to %r" % (ORIG, TGT))
 
 
@@ -1380,41 +1366,21 @@
     __slots__ = ('_TYPE',
                  '_parent_type', '_parent_index', '_keepparent',
                  '_wrparent',
-                 '__weakref__', '_traceback',
-                 '__storage')
+                 '__weakref__',
+                 '_storage')
 
-    def __init__(self, TYPE, track_allocation=None):
+    def __init__(self, TYPE):
         self._wrparent = None
         self._TYPE = TYPE
         self._storage = True    # means "use default storage", as opposed to:
                                 #    None            - container was freed
                                 #     - using ctypes
                                 #                      (see ll2ctypes.py)
-        if track_allocation is not False and TRACK_ALLOCATIONS:
-            self._traceback = self._get_traceback()
-            ALLOCATED[self] = None
-        else:
-            self._traceback = None
-
-    def _get_traceback(self):
-        frame = sys._getframe().f_back.f_back.f_back.f_back
-        sio = StringIO.StringIO()
-        traceback.print_stack(frame, file=sio)
-        return sio.getvalue()
 
     def _free(self):
         self._check()   # no double-frees
         self._storage = None
 
-    def _storage_get(self):
-        return self.__storage
-
-    def _storage_set(self, value):
-        self.__storage = value
-        if value is not True and self in ALLOCATED:
-            del ALLOCATED[self]
-    _storage = property(_storage_get, _storage_set)
-
     def _was_freed(self):
         if self._storage is None:
             return True
@@ -1493,12 +1459,12 @@
 
     __slots__ = ('_hash_cache_', '_compilation_info')
 
-    def __new__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None, track_allocation=None):
+    def __new__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None):
         my_variety = _struct_variety(TYPE._names)
         return object.__new__(my_variety)
 
-    def __init__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None, track_allocation=None):
-        _parentable.__init__(self, TYPE, track_allocation)
+    def __init__(self, TYPE, n=None, initialization=None, parent=None, parentindex=None):
+        _parentable.__init__(self, TYPE)
         if n is not None and TYPE._arrayfld is None:
             raise TypeError("%r is not variable-sized" % (TYPE,))
         if n is None and TYPE._arrayfld is not None:
@@ -1506,8 +1472,7 @@
         first, FIRSTTYPE = TYPE._first_struct()
         for fld, typ in TYPE._flds.items():
             if fld == TYPE._arrayfld:
-                value = _array(typ, n, initialization=initialization, parent=self, parentindex=fld,
-                               track_allocation=track_allocation)
+                value = _array(typ, n, initialization=initialization, parent=self, parentindex=fld)
             else:
                 value = typ._allocate(initialization=initialization, parent=self, parentindex=fld)
             setattr(self, fld, value)
@@ -1568,12 +1533,12 @@
 
     __slots__ = ('items',)
 
-    def __init__(self, TYPE, n, initialization=None, parent=None, parentindex=None, track_allocation=None):
+    def __init__(self, TYPE, n, initialization=None, parent=None, parentindex=None):
         if not isinstance(n, int):
             raise TypeError, "array length must be an int"
         if n < 0:
             raise ValueError, "negative array length"
-        _parentable.__init__(self, TYPE, track_allocation)
+        _parentable.__init__(self, TYPE)
         try:
             myrange = range(n)
         except OverflowError:
@@ -1640,7 +1605,7 @@
     _cache = weakref.WeakKeyDictionary()  # parentarray -> {subarrays}
 
     def __init__(self, TYPE, parent, baseoffset_or_fieldname):
-        _parentable.__init__(self, TYPE, track_allocation=False)
+        _parentable.__init__(self, TYPE)
         self._setparentstructure(parent, baseoffset_or_fieldname)
         # Keep the parent array alive, we share the same allocation.
         # Don't do it if we are inside a GC object, though -- it's someone
@@ -1648,6 +1613,13 @@
         if typeOf(top_container(parent))._gckind == 'raw':
             self._keepparent = parent
 
+    def __str__(self):
+        parent = self._wrparent()
+        if parent is None:
+            return '_subarray at %s in already freed' % (self._parent_index,)
+        return '_subarray at %r in %s' % (self._parent_index,
+                                          parent._TYPE)
+
     def __repr__(self):
         parent = self._wrparent()
         if parent is None:
@@ -1861,8 +1833,9 @@
         return id(self.value)
 
 
-def malloc(T, n=None, flavor='gc', immortal=False, zero=False):
-    assert flavor != 'cpy'
+def malloc(T, n=None, flavor='gc', immortal=False, zero=False,
+           track_allocation=True):
+    assert flavor in ('gc', 'raw')
     if zero or immortal:
         initialization = 'example'
     elif flavor == 'raw':
@@ -1870,9 +1843,9 @@
     else:
         initialization = 'malloc'
     if isinstance(T, Struct):
-        o = _struct(T, n, initialization=initialization, track_allocation=flavor == "raw" and not immortal)
+        o = _struct(T, n, initialization=initialization)
     elif isinstance(T, Array):
-        o = _array(T, n, initialization=initialization, track_allocation=flavor == "raw" and not immortal)
+        o = _array(T, n, initialization=initialization)
     elif isinstance(T, OpaqueType):
         assert n is None
         o = _opaque(T, initialization=initialization)
@@ -1880,17 +1853,50 @@
         raise TypeError, "malloc for Structs and Arrays only"
     if T._gckind != 'gc' and not immortal and flavor.startswith('gc'):
         raise TypeError, "gc flavor malloc of a non-GC non-immortal structure"
+    if flavor == "raw" and not immortal and track_allocation:
+        leakfinder.remember_malloc(o, framedepth=2)
     solid = immortal or not flavor.startswith('gc') # immortal or non-gc case
     return _ptr(Ptr(T), o, solid)
 
-def free(p, flavor):
+def free(p, flavor, track_allocation=True):
     if flavor.startswith('gc'):
         raise TypeError, "gc flavor free"
     T = typeOf(p)
     if not isinstance(T, Ptr) or p._togckind() != 'raw':
         raise TypeError, "free(): only for pointers to non-gc containers"
+    if track_allocation:
+        leakfinder.remember_free(p._obj0)
     p._obj0._free()
 
+def _make_scoped_allocator(T):
+    class ScopedAlloc:
+        def __init__(self, n=None, zero=False):
+            if n is None:
+                self.buf = malloc(T, flavor='raw', zero=zero)
+            else:
+                self.buf = malloc(T, n, flavor='raw', zero=zero)
+
+        def __enter__(self):
+            return self.buf
+
+        def __exit__(self, *args):
+            free(self.buf, flavor='raw')
+
+    ScopedAlloc.__name__ = 'ScopedAlloc_%s' % (T,)
+    return ScopedAlloc
+_make_scoped_allocator._annspecialcase_ = 'specialize:memo'
+
+def scoped_alloc(T, n=None, zero=False):
+    """Returns a context manager which handles allocation and
+    deallocation of temporary memory. Use it in a with statement::
+
+        with scoped_alloc(Array(Signed), 1) as array:
+            ...use array...
+        ...it's freed now.
+    """
+    return _make_scoped_allocator(T)(n=n, zero=zero)
+scoped_alloc._annspecialcase_ = 'specialize:arg(0)'
+
 def functionptr(TYPE, name, **attrs):
     if not isinstance(TYPE, FuncType):
         raise TypeError, "functionptr() for FuncTypes only"

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rbuilder.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rbuilder.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rbuilder.py	Sat Oct 30 16:41:17 2010
@@ -100,6 +100,10 @@
         ll_builder.used = used
 
     @staticmethod
+    def ll_getlength(ll_builder):
+        return ll_builder.used
+
+    @staticmethod
     def ll_build(ll_builder):
         final_size = ll_builder.used
         assert final_size >= 0

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rclass.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rclass.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rclass.py	Sat Oct 30 16:41:17 2010
@@ -329,16 +329,33 @@
             fields['__class__'] = 'typeptr', get_type_repr(self.rtyper)
         else:
             # instance attributes
-            if llfields is None:
-                llfields = []
             attrs = self.classdef.attrs.items()
             attrs.sort()
+            myllfields = []
             for name, attrdef in attrs:
                 if not attrdef.readonly:
                     r = self.rtyper.getrepr(attrdef.s_value)
                     mangled_name = 'inst_' + name
                     fields[name] = mangled_name, r
-                    llfields.append((mangled_name, r.lowleveltype))
+                    myllfields.append((mangled_name, r.lowleveltype))
+
+            # Sort the instance attributes by decreasing "likely size",
+            # as reported by rffi.sizeof(), to minimize padding holes in C.
+            # Fields of the same size are sorted by name (by attrs.sort()
+            # above) just to minimize randomness.
+            def keysize((_, T)):
+                if T is lltype.Void:
+                    return None
+                from pypy.rpython.lltypesystem.rffi import sizeof
+                try:
+                    return -sizeof(T)
+                except StandardError:
+                    return None
+            myllfields.sort(key = keysize)
+            if llfields is None:
+                llfields = myllfields
+            else:
+                llfields = llfields + myllfields
 
             self.rbase = getinstancerepr(self.rtyper, self.classdef.basedef,
                                          self.gcflavor)
@@ -403,7 +420,7 @@
         return cast_pointer(self.lowleveltype, result)
 
     def create_instance(self):
-        return malloc(self.object_type, flavor=self.gcflavor)
+        return malloc(self.object_type, flavor=self.gcflavor, immortal=True)
 
     def initialize_prebuilt_data(self, value, classdef, result):
         if self.classdef is not None:

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rdict.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rdict.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rdict.py	Sat Oct 30 16:41:17 2010
@@ -306,6 +306,13 @@
         hop.exception_cannot_occur()
         return hop.gendirectcall(ll_clear, v_dict)
 
+    def rtype_method_popitem(self, hop):
+        v_dict, = hop.inputargs(self)
+        r_tuple = hop.r_result
+        cTUPLE = hop.inputconst(lltype.Void, r_tuple.lowleveltype)
+        hop.exception_is_here()
+        return hop.gendirectcall(ll_popitem, cTUPLE, v_dict)
+
 class __extend__(pairtype(DictRepr, rmodel.Repr)): 
 
     def rtype_getitem((r_dict, r_key), hop):
@@ -465,6 +472,10 @@
     i = ll_dict_lookup(d, key, d.keyhash(key))
     if not d.entries.valid(i):
         raise KeyError
+    _ll_dict_del(d, i)
+ll_dict_delitem.oopspec = 'dict.delitem(d, key)'
+
+def _ll_dict_del(d, i):
     d.entries.mark_deleted(i)
     d.num_items -= 1
     # clear the key and the value if they are GC pointers
@@ -481,7 +492,6 @@
     num_entries = len(d.entries)
     if num_entries > DICT_INITSIZE and d.num_items < num_entries / 4:
         ll_dict_resize(d)
-ll_dict_delitem.oopspec = 'dict.delitem(d, key)'
 
 def ll_dict_resize(d):
     old_entries = d.entries
@@ -810,3 +820,26 @@
     i = ll_dict_lookup(d, key, d.keyhash(key))
     return d.entries.valid(i)
 ll_contains.oopspec = 'dict.contains(d, key)'
+
+POPITEMINDEX = lltype.Struct('PopItemIndex', ('nextindex', lltype.Signed))
+global_popitem_index = lltype.malloc(POPITEMINDEX, zero=True, immortal=True)
+
+def ll_popitem(ELEM, dic):
+    entries = dic.entries
+    dmask = len(entries) - 1
+    base = global_popitem_index.nextindex
+    counter = 0
+    while counter <= dmask:
+        i = (base + counter) & dmask
+        counter += 1
+        if entries.valid(i):
+            break
+    else:
+        raise KeyError
+    global_popitem_index.nextindex += counter
+    entry = entries[i]
+    r = lltype.malloc(ELEM.TO)
+    r.item0 = recast(ELEM.TO.item0, entry.key)
+    r.item1 = recast(ELEM.TO.item1, entry.value)
+    _ll_dict_del(dic, i)
+    return r

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rffi.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rffi.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/rffi.py	Sat Oct 30 16:41:17 2010
@@ -607,6 +607,15 @@
         return array
     str2charp._annenforceargs_ = [strtype]
 
+    def str2charp_immortal(s):
+        "NOT_RPYTHON"
+        array = lltype.malloc(TYPEP.TO, len(s) + 1, flavor='raw',
+                              immortal=True)
+        for i in range(len(s)):
+            array[i] = s[i]
+        array[len(s)] = lastchar
+        return array
+
     def free_charp(cp):
         lltype.free(cp, flavor='raw')
 
@@ -644,10 +653,18 @@
         """
         Either free a non-moving buffer or keep the original storage alive.
         """
-        if rgc.can_move(data):
+        # We cannot rely on rgc.can_move(data) here, because its result
+        # might have changed since get_nonmovingbuffer().  Instead we check
+        # if 'buf' points inside 'data'.  This is only possible if we
+        # followed the 2nd case in get_nonmovingbuffer(); in the first case,
+        # 'buf' points to its own raw-malloced memory.
+        data = llstrtype(data)
+        data_start = cast_ptr_to_adr(data) + \
+            offsetof(STRTYPE, 'chars') + itemoffsetof(STRTYPE.chars, 0)
+        followed_2nd_path = (buf == cast(TYPEP, data_start))
+        keepalive_until_here(data)
+        if not followed_2nd_path:
             lltype.free(buf, flavor='raw')
-        else:
-            keepalive_until_here(data)
 
     # int -> (char*, str)
     def alloc_buffer(count):
@@ -717,19 +734,19 @@
         l = [cp[i] for i in range(size)]
         return emptystr.join(l)
 
-    return (str2charp, free_charp, charp2str,
+    return (str2charp, str2charp_immortal, free_charp, charp2str,
             get_nonmovingbuffer, free_nonmovingbuffer,
             alloc_buffer, str_from_buffer, keep_buffer_alive_until_here,
             charp2strn, charpsize2str,
             )
 
-(str2charp, free_charp, charp2str,
+(str2charp, str2charp_immortal, free_charp, charp2str,
  get_nonmovingbuffer, free_nonmovingbuffer,
  alloc_buffer, str_from_buffer, keep_buffer_alive_until_here,
  charp2strn, charpsize2str,
  ) = make_string_mappings(str)
 
-(unicode2wcharp, free_wcharp, wcharp2unicode,
+(unicode2wcharp, unicode2wcharp_immortal, free_wcharp, wcharp2unicode,
  get_nonmoving_unicodebuffer, free_nonmoving_unicodebuffer,
  alloc_unicodebuffer, unicode_from_buffer, keep_unicodebuffer_alive_until_here,
  wcharp2unicoden, wcharpsize2unicode,
@@ -916,3 +933,11 @@
     """
     return cast(lltype.Signed, getattr(pdst, fieldname))
 getintfield._annspecialcase_ = 'specialize:ll_and_arg(1)'
+
+class scoped_str2charp:
+    def __init__(self, value):
+        self.buf = str2charp(value)
+    def __enter__(self):
+        return self.buf
+    def __exit__(self, *args):
+        free_charp(self.buf)

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_ll2ctypes.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_ll2ctypes.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_ll2ctypes.py	Sat Oct 30 16:41:17 2010
@@ -765,6 +765,7 @@
         assert abs(float(b[1]) - 1.1) < 1E-6
         assert isinstance(b[2], rffi.r_singlefloat)
         assert abs(float(b[2]) - 2.2) < 1E-6
+        lltype.free(a, flavor='raw')
 
     def test_different_signatures(self):
         if sys.platform=='win32':
@@ -879,6 +880,7 @@
         qsort(rffi.cast(rffi.VOIDP, a), 5, rffi.sizeof(rffi.INT), compare)
         for i in range(5):
             assert a[i] == i + 1
+        lltype.free(a, flavor='raw')
 
     def test_array_type_bug(self):
         A = lltype.Array(lltype.Signed)

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_llmemory.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_llmemory.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_llmemory.py	Sat Oct 30 16:41:17 2010
@@ -324,12 +324,14 @@
     p_t = lltype.malloc(T)
     assert p_t.s == lltype.nullptr(S)
     # raw malloc does not
-    p_raw_t = lltype.malloc(T, flavor="raw")
-    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.s")
+    U = lltype.Struct("U", ('x', lltype.Signed))
+    p_raw_t = lltype.malloc(U, flavor="raw")
+    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.x")
+    lltype.free(p_raw_t, flavor="raw")
     # this sort of raw_malloc too
-    p_raw_t = cast_adr_to_ptr(raw_malloc(sizeof(T)), lltype.Ptr(T))
-    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.s")
-    
+    p_raw_t = cast_adr_to_ptr(raw_malloc(sizeof(U)), lltype.Ptr(U))
+    py.test.raises(lltype.UninitializedMemoryAccess, "p_raw_t.x")
+
 
 def test_raw_malloc_signed_bunch():
     adr = raw_malloc(sizeof(lltype.Signed) * 50)
@@ -601,7 +603,8 @@
     a = lltype.malloc(A, flavor='raw')
     src = cast_ptr_to_adr(a) + itemoffsetof(A, 0)
     raw_memclear(src, sizeof(lltype.Signed) * 0)
-    
+    lltype.free(a, flavor="raw")
+
 def test_nonneg():
     S1 = lltype.GcStruct('S1', ('x', lltype.Float))
     A1 = lltype.GcArray(lltype.Float)

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_lltype.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_lltype.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_lltype.py	Sat Oct 30 16:41:17 2010
@@ -1,7 +1,9 @@
+from __future__ import with_statement
 import py
 from pypy.rpython.lltypesystem.lltype import *
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.tool.identity_dict import identity_dict
+from pypy.tool import leakfinder
 
 def isweak(p, T):
     try:
@@ -804,22 +806,20 @@
 
 
 class TestTrackAllocation:
-    def setup_method(self, func):
-        start_tracking_allocations()
-
-    def teardown_method(self, func):
-        assert not lltype.ALLOCATED, "Memory was not correctly freed"
-        stop_tracking_allocations()
+    def test_automatic_tracking(self):
+        # calls to start_tracking_allocations/stop_tracking_allocations
+        # should occur automatically from pypy/conftest.py.  Check that.
+        assert leakfinder.TRACK_ALLOCATIONS
 
     def test_track_allocation(self):
         """A malloc'd buffer fills the ALLOCATED dictionary"""
-        assert lltype.TRACK_ALLOCATIONS
-        assert not lltype.ALLOCATED
+        assert leakfinder.TRACK_ALLOCATIONS
+        assert not leakfinder.ALLOCATED
         buf = malloc(Array(Signed), 1, flavor="raw")
-        assert len(lltype.ALLOCATED) == 1
-        assert lltype.ALLOCATED.keys() == [buf._obj]
+        assert len(leakfinder.ALLOCATED) == 1
+        assert leakfinder.ALLOCATED.keys() == [buf._obj]
         free(buf, flavor="raw")
-        assert not lltype.ALLOCATED
+        assert not leakfinder.ALLOCATED
 
     def test_str_from_buffer(self):
         """gc-managed memory does not need to be freed"""
@@ -828,16 +828,28 @@
         for i in range(size): raw_buf[i] = 'a'
         rstr = rffi.str_from_buffer(raw_buf, gc_buf, size, size)
         rffi.keep_buffer_alive_until_here(raw_buf, gc_buf)
-        assert not lltype.ALLOCATED
+        assert not leakfinder.ALLOCATED
 
     def test_leak_traceback(self):
         """Test info stored for allocated items"""
         buf = malloc(Array(Signed), 1, flavor="raw")
-        traceback = lltype.ALLOCATED.keys()[0]._traceback
+        traceback = leakfinder.ALLOCATED.values()[0]
         lines = traceback.splitlines()
         assert 'malloc(' in lines[-1] and 'flavor="raw")' in lines[-1]
 
-        # XXX The traceback should not be too long
+        # The traceback should not be too long
         print traceback
 
         free(buf, flavor="raw")
+
+    def test_no_tracking(self):
+        p1 = malloc(Array(Signed), 1, flavor='raw', track_allocation=False)
+        p2 = malloc(Array(Signed), 1, flavor='raw', track_allocation=False)
+        free(p2, flavor='raw', track_allocation=False)
+        # p1 is not freed
+
+    def test_scoped_allocator(self):
+        with scoped_alloc(Array(Signed), 1) as array:
+            array[0] = -42
+            x = array[0]
+        assert x == -42

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_rffi.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_rffi.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/lltypesystem/test/test_rffi.py	Sat Oct 30 16:41:17 2010
@@ -9,7 +9,7 @@
 from pypy.rpython.lltypesystem.rstr import STR
 from pypy.rpython.lltypesystem import lltype
 from pypy.tool.udir import udir
-from pypy.rpython.test.test_llinterp import interpret, MallocMismatch
+from pypy.rpython.test.test_llinterp import interpret
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.annotation.annrpython import RPythonAnnotator
 from pypy.rpython.rtyper import RPythonTyper
@@ -787,3 +787,10 @@
     mixann.getgraph(f2, [], s_None)
     mixann.finish()
 
+def test_force_cast_unichar():
+    x = cast(lltype.UniChar, -1)
+    assert isinstance(x, unicode)
+    if sys.maxunicode == 65535:
+        assert cast(LONG, x) == 65535
+    else:
+        assert cast(LONG, cast(INT, x)) == -1

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/base.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/base.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/base.py	Sat Oct 30 16:41:17 2010
@@ -20,12 +20,15 @@
     prebuilt_gc_objects_are_static_roots = True
     object_minimal_size = 0
 
-    def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE):
+    def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE,
+                 translated_to_c=True):
         self.gcheaderbuilder = GCHeaderBuilder(self.HDR)
         self.AddressStack = get_address_stack(chunk_size)
         self.AddressDeque = get_address_deque(chunk_size)
         self.AddressDict = AddressDict
         self.config = config
+        assert isinstance(translated_to_c, bool)
+        self.translated_to_c = translated_to_c
 
     def setup(self):
         # all runtime mutable values' setup should happen here
@@ -79,7 +82,7 @@
     def set_root_walker(self, root_walker):
         self.root_walker = root_walker
 
-    def write_barrier(self, addr_struct):
+    def write_barrier(self, newvalue, addr_struct):
         pass
 
     def statistics(self, index):

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/generation.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/generation.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/generation.py	Sat Oct 30 16:41:17 2010
@@ -5,7 +5,6 @@
 from pypy.rpython.memory.gc.base import read_from_env
 from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage
 from pypy.rpython.lltypesystem import lltype, llmemory, llarena
-from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE
 from pypy.rlib.objectmodel import free_non_gc_object
 from pypy.rlib.debug import ll_assert
 from pypy.rlib.debug import debug_print, debug_start, debug_stop
@@ -49,15 +48,17 @@
 
     nursery_hash_base = -1
 
-    def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE,
+    def __init__(self, config,
                  nursery_size=32*WORD,
                  min_nursery_size=32*WORD,
                  auto_nursery_size=False,
                  space_size=1024*WORD,
-                 max_space_size=sys.maxint//2+1):
-        SemiSpaceGC.__init__(self, config, chunk_size = chunk_size,
+                 max_space_size=sys.maxint//2+1,
+                 **kwds):
+        SemiSpaceGC.__init__(self, config,
                              space_size = space_size,
-                             max_space_size = max_space_size)
+                             max_space_size = max_space_size,
+                             **kwds)
         assert min_nursery_size <= nursery_size <= space_size // 2
         self.initial_nursery_size = nursery_size
         self.auto_nursery_size = auto_nursery_size
@@ -157,6 +158,14 @@
                   "odd-valued (i.e. tagged) pointer unexpected here")
         return self.nursery <= addr < self.nursery_top
 
+    def appears_to_be_in_nursery(self, addr):
+        # same as is_in_nursery(), but may return True accidentally if
+        # 'addr' is a tagged pointer with just the wrong value.
+        if not self.translated_to_c:
+            if not self.is_valid_gc_object(addr):
+                return False
+        return self.nursery <= addr < self.nursery_top
+
     def malloc_fixedsize_clear(self, typeid, size, can_collect,
                                has_finalizer=False, contains_weakptr=False):
         if (has_finalizer or not can_collect or
@@ -326,7 +335,7 @@
         addr = pointer.address[0]
         newaddr = self.copy(addr)
         pointer.address[0] = newaddr
-        self.write_into_last_generation_obj(obj)
+        self.write_into_last_generation_obj(obj, newaddr)
 
     # ____________________________________________________________
     # Implementation of nursery-only collections
@@ -457,9 +466,9 @@
     #  "if addr_struct.int0 & JIT_WB_IF_FLAG: remember_young_pointer()")
     JIT_WB_IF_FLAG = GCFLAG_NO_YOUNG_PTRS
 
-    def write_barrier(self, addr_struct):
-        if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS:
-            self.remember_young_pointer(addr_struct)
+    def write_barrier(self, newvalue, addr_struct):
+         if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS:
+            self.remember_young_pointer(addr_struct, newvalue)
 
     def _setup_wb(self):
         DEBUG = self.DEBUG
@@ -470,23 +479,33 @@
         # For x86, there is also an extra requirement: when the JIT calls
         # remember_young_pointer(), it assumes that it will not touch the SSE
         # registers, so it does not save and restore them (that's a *hack*!).
-        def remember_young_pointer(addr_struct):
+        def remember_young_pointer(addr_struct, addr):
             #llop.debug_print(lltype.Void, "\tremember_young_pointer",
             #                 addr_struct, "<-", addr)
             if DEBUG:
                 ll_assert(not self.is_in_nursery(addr_struct),
                           "nursery object with GCFLAG_NO_YOUNG_PTRS")
-            self.old_objects_pointing_to_young.append(addr_struct)
-            self.header(addr_struct).tid &= ~GCFLAG_NO_YOUNG_PTRS
-            self.write_into_last_generation_obj(addr_struct)
+            #
+            # What is important in this function is that it *must*
+            # clear the flag GCFLAG_NO_YOUNG_PTRS from 'addr_struct'
+            # if 'addr' is in the nursery.  It is ok if, accidentally,
+            # it also clears the flag in some more rare cases, like
+            # 'addr' being a tagged pointer whose value happens to be
+            # a large integer that fools is_in_nursery().
+            if self.appears_to_be_in_nursery(addr):
+                self.old_objects_pointing_to_young.append(addr_struct)
+                self.header(addr_struct).tid &= ~GCFLAG_NO_YOUNG_PTRS
+            self.write_into_last_generation_obj(addr_struct, addr)
         remember_young_pointer._dont_inline_ = True
         self.remember_young_pointer = remember_young_pointer
 
-    def write_into_last_generation_obj(self, addr_struct):
+    def write_into_last_generation_obj(self, addr_struct, addr):
         objhdr = self.header(addr_struct)
         if objhdr.tid & GCFLAG_NO_HEAP_PTRS:
-            objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS
-            self.last_generation_root_objects.append(addr_struct)
+            if (self.is_valid_gc_object(addr) and
+                    not self.is_last_generation(addr)):
+                objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS
+                self.last_generation_root_objects.append(addr_struct)
     write_into_last_generation_obj._always_inline_ = True
 
     def assume_young_pointers(self, addr_struct):

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/markcompact.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/markcompact.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/markcompact.py	Sat Oct 30 16:41:17 2010
@@ -2,7 +2,6 @@
 from pypy.rpython.memory.gc.base import MovingGCBase, read_from_env
 from pypy.rlib.debug import ll_assert, have_debug_prints
 from pypy.rlib.debug import debug_print, debug_start, debug_stop
-from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE
 from pypy.rpython.memory.support import get_address_stack, get_address_deque
 from pypy.rpython.memory.support import AddressDict
 from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage
@@ -86,9 +85,9 @@
     free = NULL
     next_collect_after = -1
 
-    def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, space_size=4096,
-                 min_next_collect_after=128):
-        MovingGCBase.__init__(self, config, chunk_size)
+    def __init__(self, config, space_size=4096,
+                 min_next_collect_after=128, **kwds):
+        MovingGCBase.__init__(self, config, **kwds)
         self.space_size = space_size
         self.min_next_collect_after = min_next_collect_after
 

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/marksweep.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/marksweep.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/marksweep.py	Sat Oct 30 16:41:17 2010
@@ -1,7 +1,6 @@
 from pypy.rpython.lltypesystem.llmemory import raw_malloc, raw_free
 from pypy.rpython.lltypesystem.llmemory import raw_memcopy, raw_memclear
 from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage
-from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE
 from pypy.rpython.memory.support import get_address_stack
 from pypy.rpython.memory.gcheader import GCHeaderBuilder
 from pypy.rpython.lltypesystem import lltype, llmemory, rffi, llgroup
@@ -48,9 +47,9 @@
     # translating to a real backend.
     TRANSLATION_PARAMS = {'start_heap_size': 8*1024*1024} # XXX adjust
 
-    def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, start_heap_size=4096):
+    def __init__(self, config, start_heap_size=4096, **kwds):
         self.param_start_heap_size = start_heap_size
-        GCBase.__init__(self, config, chunk_size)
+        GCBase.__init__(self, config, **kwds)
 
     def setup(self):
         GCBase.setup(self)
@@ -714,8 +713,8 @@
     _alloc_flavor_ = "raw"
     COLLECT_EVERY = 2000
 
-    def __init__(self, chunk_size=DEFAULT_CHUNK_SIZE, start_heap_size=4096):
-        MarkSweepGC.__init__(self, chunk_size, start_heap_size)
+    def __init__(self, config, **kwds):
+        MarkSweepGC.__init__(self, config, **kwds)
         self.count_mallocs = 0
 
     def maybe_collect(self):

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/minimark.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/minimark.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/minimark.py	Sat Oct 30 16:41:17 2010
@@ -4,7 +4,6 @@
 from pypy.rpython.lltypesystem.llmemory import raw_malloc_usage
 from pypy.rpython.memory.gc.base import GCBase, MovingGCBase
 from pypy.rpython.memory.gc import minimarkpage, base, generation
-from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE
 from pypy.rlib.rarithmetic import ovfcheck, LONG_BIT, intmask, r_uint
 from pypy.rlib.rarithmetic import LONG_BIT_SHIFT
 from pypy.rlib.debug import ll_assert, debug_print, debug_start, debug_stop
@@ -140,7 +139,7 @@
         "large_object_gcptrs": 8250*WORD,
         }
 
-    def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE,
+    def __init__(self, config,
                  read_from_env=False,
                  nursery_size=32*WORD,
                  page_size=16*WORD,
@@ -150,8 +149,9 @@
                  card_page_indices=0,
                  large_object=8*WORD,
                  large_object_gcptrs=10*WORD,
-                 ArenaCollectionClass=None):
-        MovingGCBase.__init__(self, config, chunk_size)
+                 ArenaCollectionClass=None,
+                 **kwds):
+        MovingGCBase.__init__(self, config, **kwds)
         assert small_request_threshold % WORD == 0
         self.read_from_env = read_from_env
         self.nursery_size = nursery_size
@@ -636,6 +636,14 @@
                   "odd-valued (i.e. tagged) pointer unexpected here")
         return self.nursery <= addr < self.nursery_top
 
+    def appears_to_be_in_nursery(self, addr):
+        # same as is_in_nursery(), but may return True accidentally if
+        # 'addr' is a tagged pointer with just the wrong value.
+        if not self.translated_to_c:
+            if not self.is_valid_gc_object(addr):
+                return False
+        return self.nursery <= addr < self.nursery_top
+
     def is_forwarded(self, obj):
         """Returns True if the nursery obj is marked as forwarded.
         Implemented a bit obscurely by checking an unrelated flag
@@ -726,16 +734,16 @@
     def JIT_max_size_of_young_obj(cls):
         return cls.TRANSLATION_PARAMS['large_object']
 
-    def write_barrier(self, addr_struct):
+    def write_barrier(self, newvalue, addr_struct):
         if self.header(addr_struct).tid & GCFLAG_NO_YOUNG_PTRS:
-            self.remember_young_pointer(addr_struct)
+            self.remember_young_pointer(addr_struct, newvalue)
 
-    def write_barrier_from_array(self, addr_array, index):
+    def write_barrier_from_array(self, newvalue, addr_array, index):
         if self.header(addr_array).tid & GCFLAG_NO_YOUNG_PTRS:
             if self.card_page_indices > 0:     # <- constant-folded
                 self.remember_young_pointer_from_array(addr_array, index)
             else:
-                self.remember_young_pointer(addr_array)
+                self.remember_young_pointer(addr_array, newvalue)
 
     def _init_writebarrier_logic(self):
         DEBUG = self.DEBUG
@@ -746,27 +754,28 @@
         # For x86, there is also an extra requirement: when the JIT calls
         # remember_young_pointer(), it assumes that it will not touch the SSE
         # registers, so it does not save and restore them (that's a *hack*!).
-        def remember_young_pointer(addr_struct):
+        def remember_young_pointer(addr_struct, newvalue):
             # 'addr_struct' is the address of the object in which we write.
+            # 'newvalue' is the address that we are going to write in there.
             if DEBUG:
                 ll_assert(not self.is_in_nursery(addr_struct),
                           "nursery object with GCFLAG_NO_YOUNG_PTRS")
             #
-            # We assume that what we are writing is a pointer to the nursery
-            # (and don't care for the fact that this new pointer may not
-            # actually point to the nursery, which seems ok).  What we need is
+            # If it seems that what we are writing is a pointer to the nursery
+            # (as checked with appears_to_be_in_nursery()), then we need
             # to remove the flag GCFLAG_NO_YOUNG_PTRS and add the old object
             # to the list 'old_objects_pointing_to_young'.  We know that
             # 'addr_struct' cannot be in the nursery, because nursery objects
             # never have the flag GCFLAG_NO_YOUNG_PTRS to start with.
-            self.old_objects_pointing_to_young.append(addr_struct)
             objhdr = self.header(addr_struct)
-            objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS
+            if self.appears_to_be_in_nursery(newvalue):
+                self.old_objects_pointing_to_young.append(addr_struct)
+                objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS
             #
             # Second part: if 'addr_struct' is actually a prebuilt GC
             # object and it's the first time we see a write to it, we
             # add it to the list 'prebuilt_root_objects'.  Note that we
-            # do it even in the (rare?) case of 'addr' being another
+            # do it even in the (rare?) case of 'addr' being NULL or another
             # prebuilt object, to simplify code.
             if objhdr.tid & GCFLAG_NO_HEAP_PTRS:
                 objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS
@@ -780,16 +789,24 @@
 
 
     def _init_writebarrier_with_card_marker(self):
+        DEBUG = self.DEBUG
         def remember_young_pointer_from_array(addr_array, index):
             # 'addr_array' is the address of the object in which we write,
             # which must have an array part;  'index' is the index of the
             # item that is (or contains) the pointer that we write.
+            if DEBUG:
+                ll_assert(not self.is_in_nursery(addr_array),
+                          "nursery array with GCFLAG_NO_YOUNG_PTRS")
             objhdr = self.header(addr_array)
             if objhdr.tid & GCFLAG_HAS_CARDS == 0:
                 #
-                # no cards, use default logic.  The 'nocard_logic()' is just
-                # 'remember_young_pointer()', but forced to be inlined here.
-                nocard_logic(addr_array)
+                # no cards, use default logic.  Mostly copied from above.
+                self.old_objects_pointing_to_young.append(addr_array)
+                objhdr = self.header(addr_array)
+                objhdr.tid &= ~GCFLAG_NO_YOUNG_PTRS
+                if objhdr.tid & GCFLAG_NO_HEAP_PTRS:
+                    objhdr.tid &= ~GCFLAG_NO_HEAP_PTRS
+                    self.prebuilt_root_objects.append(addr_array)
                 return
             #
             # 'addr_array' is a raw_malloc'ed array with card markers
@@ -807,17 +824,15 @@
                 return
             #
             # We set the flag (even if the newly written address does not
-            # actually point to the nursery -- like remember_young_pointer()).
+            # actually point to the nursery, which seems to be ok -- actually
+            # it seems more important that remember_young_pointer_from_array()
+            # does not take 3 arguments).
             addr_byte.char[0] = chr(byte | bitmask)
             #
             if objhdr.tid & GCFLAG_CARDS_SET == 0:
                 self.old_objects_with_cards_set.append(addr_array)
                 objhdr.tid |= GCFLAG_CARDS_SET
 
-        nocard_logic = func_with_new_name(self.remember_young_pointer,
-                                          'remember_young_pointer_nocard')
-        del nocard_logic._dont_inline_
-        nocard_logic._always_inline_ = True
         remember_young_pointer_from_array._dont_inline_ = True
         self.remember_young_pointer_from_array = (
             remember_young_pointer_from_array)

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/minimarkpage.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/minimarkpage.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/minimarkpage.py	Sat Oct 30 16:41:17 2010
@@ -100,11 +100,14 @@
         # allocation of the given size.
         length = small_request_threshold / WORD + 1
         self.page_for_size = lltype.malloc(rffi.CArray(PAGE_PTR), length,
-                                           flavor='raw', zero=True)
+                                           flavor='raw', zero=True,
+                                           immortal=True)
         self.full_page_for_size = lltype.malloc(rffi.CArray(PAGE_PTR), length,
-                                                flavor='raw', zero=True)
+                                                flavor='raw', zero=True,
+                                                immortal=True)
         self.nblocks_for_size = lltype.malloc(rffi.CArray(lltype.Signed),
-                                              length, flavor='raw')
+                                              length, flavor='raw',
+                                              immortal=True)
         self.hdrsize = llmemory.raw_malloc_usage(llmemory.sizeof(PAGE_HEADER))
         assert page_size > self.hdrsize
         self.nblocks_for_size[0] = 0    # unused
@@ -114,11 +117,13 @@
         self.max_pages_per_arena = arena_size // page_size
         self.arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR),
                                           self.max_pages_per_arena,
-                                          flavor='raw', zero=True)
+                                          flavor='raw', zero=True,
+                                          immortal=True)
         # this is used in mass_free() only
         self.old_arenas_lists = lltype.malloc(rffi.CArray(ARENA_PTR),
                                               self.max_pages_per_arena,
-                                              flavor='raw', zero=True)
+                                              flavor='raw', zero=True,
+                                              immortal=True)
         #
         # the arena currently consumed; it must have at least one page
         # available, or be NULL.  The arena object that we point to is
@@ -281,7 +286,7 @@
         npages = (arena_end - firstpage) // self.page_size
         #
         # Allocate an ARENA object and initialize it
-        arena = lltype.malloc(ARENA, flavor='raw')
+        arena = lltype.malloc(ARENA, flavor='raw', track_allocation=False)
         arena.base = arena_base
         arena.nfreepages = 0        # they are all uninitialized pages
         arena.totalpages = npages
@@ -332,7 +337,7 @@
                     #
                     # The whole arena is empty.  Free it.
                     llarena.arena_free(arena.base)
-                    lltype.free(arena, flavor='raw')
+                    lltype.free(arena, flavor='raw', track_allocation=False)
                     #
                 else:
                     # Insert 'arena' in the correct arenas_lists[n]

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/semispace.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/semispace.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/semispace.py	Sat Oct 30 16:41:17 2010
@@ -1,7 +1,6 @@
 from pypy.rpython.lltypesystem.llmemory import raw_malloc, raw_free
 from pypy.rpython.lltypesystem.llmemory import raw_memcopy, raw_memclear
 from pypy.rpython.lltypesystem.llmemory import NULL, raw_malloc_usage
-from pypy.rpython.memory.support import DEFAULT_CHUNK_SIZE
 from pypy.rpython.memory.support import get_address_stack, get_address_deque
 from pypy.rpython.memory.support import AddressDict
 from pypy.rpython.lltypesystem import lltype, llmemory, llarena, rffi, llgroup
@@ -59,11 +58,11 @@
     # translating to a real backend.
     TRANSLATION_PARAMS = {'space_size': 8*1024*1024} # XXX adjust
 
-    def __init__(self, config, chunk_size=DEFAULT_CHUNK_SIZE, space_size=4096,
-                 max_space_size=sys.maxint//2+1):
+    def __init__(self, config, space_size=4096, max_space_size=sys.maxint//2+1,
+                 **kwds):
         self.param_space_size = space_size
         self.param_max_space_size = max_space_size
-        MovingGCBase.__init__(self, config, chunk_size)
+        MovingGCBase.__init__(self, config, **kwds)
 
     def setup(self):
         #self.total_collection_time = 0.0

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/test/test_direct.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/test/test_direct.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gc/test/test_direct.py	Sat Oct 30 16:41:17 2010
@@ -70,6 +70,7 @@
         GC_PARAMS = self.GC_PARAMS.copy()
         if hasattr(meth, 'GC_PARAMS'):
             GC_PARAMS.update(meth.GC_PARAMS)
+        GC_PARAMS['translated_to_c'] = False
         self.gc = self.GCClass(config, **GC_PARAMS)
         self.gc.DEBUG = True
         self.rootwalker = DirectRootWalker(self)
@@ -86,17 +87,19 @@
 
     def write(self, p, fieldname, newvalue):
         if self.gc.needs_write_barrier:
+            newaddr = llmemory.cast_ptr_to_adr(newvalue)
             addr_struct = llmemory.cast_ptr_to_adr(p)
-            self.gc.write_barrier(addr_struct)
+            self.gc.write_barrier(newaddr, addr_struct)
         setattr(p, fieldname, newvalue)
 
     def writearray(self, p, index, newvalue):
         if self.gc.needs_write_barrier:
+            newaddr = llmemory.cast_ptr_to_adr(newvalue)
             addr_struct = llmemory.cast_ptr_to_adr(p)
             if hasattr(self.gc, 'write_barrier_from_array'):
-                self.gc.write_barrier_from_array(addr_struct, index)
+                self.gc.write_barrier_from_array(newaddr, addr_struct, index)
             else:
-                self.gc.write_barrier(addr_struct)
+                self.gc.write_barrier(newaddr, addr_struct)
         p[index] = newvalue
 
     def malloc(self, TYPE, n=None):

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/asmgcroot.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/asmgcroot.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/asmgcroot.py	Sat Oct 30 16:41:17 2010
@@ -75,7 +75,8 @@
             key = (TYPE, num)
             if key not in sradict:
                 CONTAINER = lltype.FixedSizeArray(TYPE, 1)
-                p = lltype.malloc(CONTAINER, flavor='raw', zero=True)
+                p = lltype.malloc(CONTAINER, flavor='raw', zero=True,
+                                  immortal=True)
                 sradict[key] = Constant(p, lltype.Ptr(CONTAINER))
             sra.append(sradict[key])
         #

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/framework.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/framework.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/framework.py	Sat Oct 30 16:41:17 2010
@@ -426,6 +426,7 @@
         if GCClass.needs_write_barrier:
             self.write_barrier_ptr = getfn(GCClass.write_barrier.im_func,
                                            [s_gc,
+                                            annmodel.SomeAddress(),
                                             annmodel.SomeAddress()],
                                            annmodel.s_None,
                                            inline=True)
@@ -434,13 +435,15 @@
                 # func should not be a bound method, but a real function
                 assert isinstance(func, types.FunctionType)
                 self.write_barrier_failing_case_ptr = getfn(func,
-                                               [annmodel.SomeAddress()],
+                                               [annmodel.SomeAddress(),
+                                                annmodel.SomeAddress()],
                                                annmodel.s_None)
             func = getattr(GCClass, 'write_barrier_from_array', None)
             if func is not None:
                 self.write_barrier_from_array_ptr = getfn(func.im_func,
                                            [s_gc,
                                             annmodel.SomeAddress(),
+                                            annmodel.SomeAddress(),
                                             annmodel.SomeInteger()],
                                            annmodel.s_None,
                                            inline=True)
@@ -1021,6 +1024,8 @@
             and not isinstance(v_newvalue, Constant)
             and v_struct.concretetype.TO._gckind == "gc"
             and hop.spaceop not in self.clean_sets):
+            v_newvalue = hop.genop("cast_ptr_to_adr", [v_newvalue],
+                                   resulttype = llmemory.Address)
             v_structaddr = hop.genop("cast_ptr_to_adr", [v_struct],
                                      resulttype = llmemory.Address)
             if (self.write_barrier_from_array_ptr is not None and
@@ -1030,12 +1035,14 @@
                 assert v_index.concretetype == lltype.Signed
                 hop.genop("direct_call", [self.write_barrier_from_array_ptr,
                                           self.c_const_gc,
+                                          v_newvalue,
                                           v_structaddr,
                                           v_index])
             else:
                 self.write_barrier_calls += 1
                 hop.genop("direct_call", [self.write_barrier_ptr,
                                           self.c_const_gc,
+                                          v_newvalue,
                                           v_structaddr])
         hop.rename('bare_' + opname)
 

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/transform.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/transform.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gctransform/transform.py	Sat Oct 30 16:41:17 2010
@@ -430,7 +430,8 @@
         return self.parenttransformer.gct_malloc_varsize(hop)
     
     def gct_free(self, hop):
-        flavor = hop.spaceop.args[1].value
+        flags = hop.spaceop.args[1].value
+        flavor = flags['flavor']
         assert flavor == 'raw'
         return self.parenttransformer.gct_free(hop)
 
@@ -532,6 +533,8 @@
                           resulttype=llmemory.Address)
         if flags.get('zero'):
             hop.genop("raw_memclear", [v_raw, c_size])
+        if flags.get('track_allocation', True):
+            hop.genop("track_alloc_start", [v_raw])
         return v_raw
 
     def gct_fv_stack_malloc(self, hop, flags, TYPE, c_size):
@@ -602,15 +605,20 @@
                                [self.raw_malloc_varsize_ptr, v_length,
                                 c_const_size, c_item_size, c_offset_to_length],
                                resulttype=llmemory.Address)
+        if flags.get('track_allocation', True):
+            hop.genop("track_alloc_start", [v_raw])
         return v_raw
 
     def gct_free(self, hop):
         op = hop.spaceop
-        flavor = op.args[1].value
+        flags = op.args[1].value
+        flavor = flags['flavor']
         v = op.args[0]
         assert flavor != 'cpy', "cannot free CPython objects directly"
         if flavor == 'raw':
             v = hop.genop("cast_ptr_to_adr", [v], resulttype=llmemory.Address)
+            if flags.get('track_allocation', True):
+                hop.genop("track_alloc_stop", [v])
             hop.genop('raw_free', [v])
         else:
             assert False, "%s has no support for free with flavor %r" % (self, flavor)           

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/gcwrapper.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/gcwrapper.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/gcwrapper.py	Sat Oct 30 16:41:17 2010
@@ -9,7 +9,10 @@
     def __init__(self, llinterp, flowgraphs, gc_class, GC_PARAMS={}):
         translator = llinterp.typer.annotator.translator
         config = translator.config.translation
-        self.gc = gc_class(config, chunk_size = 10, **GC_PARAMS)
+        self.gc = gc_class(config,
+                           chunk_size      = 10,
+                           translated_to_c = False,
+                           **GC_PARAMS)
         self.gc.set_root_walker(LLInterpRootWalker(self))
         self.gc.DEBUG = True
         self.llinterp = llinterp
@@ -39,7 +42,8 @@
     #
     # Interface for the llinterp
     #
-    def malloc(self, TYPE, n=None, flavor='gc', zero=False):
+    def malloc(self, TYPE, n=None, flavor='gc', zero=False,
+               track_allocation=True):
         if flavor == 'gc':
             typeid = self.get_type_id(TYPE)
             addr = self.gc.malloc(typeid, n, zero=zero)
@@ -48,7 +52,8 @@
                 gctypelayout.zero_gc_pointers(result)
             return result
         else:
-            return lltype.malloc(TYPE, n, flavor=flavor, zero=zero)
+            return lltype.malloc(TYPE, n, flavor=flavor, zero=zero,
+                                 track_allocation=track_allocation)
 
     def malloc_nonmovable(self, TYPE, n=None, zero=False):
         typeid = self.get_type_id(TYPE)
@@ -66,9 +71,10 @@
             return self.gc.shrink_array(addr, smallersize)
         return False
 
-    def free(self, TYPE, flavor='gc'):
+    def free(self, TYPE, flavor='gc', track_allocation=True):
         assert flavor != 'gc'
-        return lltype.free(TYPE, flavor=flavor)
+        return lltype.free(TYPE, flavor=flavor,
+                           track_allocation=track_allocation)
 
     def setfield(self, obj, fieldname, fieldvalue):
         STRUCT = lltype.typeOf(obj).TO
@@ -94,6 +100,7 @@
                         assert (type(index) is int    # <- fast path
                                 or lltype.typeOf(index) == lltype.Signed)
                         self.gc.write_barrier_from_array(
+                            llmemory.cast_ptr_to_adr(newvalue),
                             llmemory.cast_ptr_to_adr(toplevelcontainer),
                             index)
                         wb = False
@@ -101,6 +108,7 @@
             #
             if wb:
                 self.gc.write_barrier(
+                    llmemory.cast_ptr_to_adr(newvalue),
                     llmemory.cast_ptr_to_adr(toplevelcontainer))
         llheap.setinterior(toplevelcontainer, inneraddr, INNERTYPE, newvalue)
 

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/support.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/support.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/support.py	Sat Oct 30 16:41:17 2010
@@ -30,7 +30,8 @@
                 # we zero-initialize the chunks to make the translation
                 # backends happy, but we don't need to do it at run-time.
                 zero = not we_are_translated()
-                return lltype.malloc(CHUNK, flavor="raw", zero=zero)
+                return lltype.malloc(CHUNK, flavor="raw", zero=zero,
+                                     track_allocation=False)
                 
             result = self.free_list
             self.free_list = result.next
@@ -44,7 +45,7 @@
                 # Don't cache the old chunks but free them immediately.
                 # Helps debugging, and avoids that old chunks full of
                 # addresses left behind by a test end up in genc...
-                lltype.free(chunk, flavor="raw")
+                lltype.free(chunk, flavor="raw", track_allocation=False)
 
     unused_chunks = FreeList()
     cache[chunk_size] = unused_chunks, null_chunk

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/test/test_gc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/test/test_gc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/test/test_gc.py	Sat Oct 30 16:41:17 2010
@@ -368,6 +368,14 @@
         res = self.interpret(f, [4, 42])
         assert res == 12
 
+    def test_print_leak(self):
+        def f(n):
+            for i in range(n):
+                print i
+            return 42
+        res = self.interpret(f, [10])
+        assert res == 42
+
     def test_weakref_across_minor_collection(self):
         import weakref
         class A:

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/memory/test/test_transformed_gc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/memory/test/test_transformed_gc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/memory/test/test_transformed_gc.py	Sat Oct 30 16:41:17 2010
@@ -906,7 +906,8 @@
     gcname = "marksweep"
     class gcpolicy(gc.FrameworkGcPolicy):
         class transformerclass(framework.FrameworkGCTransformer):
-            GC_PARAMS = {'start_heap_size': 1024*WORD }
+            GC_PARAMS = {'start_heap_size': 1024*WORD,
+                         'translated_to_c': False}
             root_stack_depth = 200
 
 
@@ -1144,7 +1145,8 @@
     class gcpolicy(gc.FrameworkGcPolicy):
         class transformerclass(framework.FrameworkGCTransformer):
             from pypy.rpython.memory.gc.marksweep import PrintingMarkSweepGC as GCClass
-            GC_PARAMS = {'start_heap_size': 1024*WORD }
+            GC_PARAMS = {'start_heap_size': 1024*WORD,
+                         'translated_to_c': False}
             root_stack_depth = 200
 
 class TestSemiSpaceGC(GenericMovingGCTests):
@@ -1154,7 +1156,8 @@
     class gcpolicy(gc.FrameworkGcPolicy):
         class transformerclass(framework.FrameworkGCTransformer):
             from pypy.rpython.memory.gc.semispace import SemiSpaceGC as GCClass
-            GC_PARAMS = {'space_size': 512*WORD}
+            GC_PARAMS = {'space_size': 512*WORD,
+                         'translated_to_c': False}
             root_stack_depth = 200
 
 class TestMarkCompactGC(GenericMovingGCTests):
@@ -1163,7 +1166,8 @@
     class gcpolicy(gc.FrameworkGcPolicy):
         class transformerclass(framework.FrameworkGCTransformer):
             from pypy.rpython.memory.gc.markcompact import MarkCompactGC as GCClass
-            GC_PARAMS = {'space_size': 4096*WORD}
+            GC_PARAMS = {'space_size': 4096*WORD,
+                         'translated_to_c': False}
             root_stack_depth = 200
 
 class TestGenerationGC(GenericMovingGCTests):
@@ -1175,7 +1179,8 @@
             from pypy.rpython.memory.gc.generation import GenerationGC as \
                                                           GCClass
             GC_PARAMS = {'space_size': 512*WORD,
-                         'nursery_size': 32*WORD}
+                         'nursery_size': 32*WORD,
+                         'translated_to_c': False}
             root_stack_depth = 200
 
     def define_weakref_across_minor_collection(cls):
@@ -1372,7 +1377,8 @@
                 GenerationGC._teardown(self)
                 
             GC_PARAMS = {'space_size': 512*WORD,
-                         'nursery_size': 128*WORD}
+                         'nursery_size': 128*WORD,
+                         'translated_to_c': False}
             root_stack_depth = 200
 
     def define_working_nursery(cls):
@@ -1404,7 +1410,8 @@
             from pypy.rpython.memory.gc.hybrid import HybridGC as GCClass
             GC_PARAMS = {'space_size': 512*WORD,
                          'nursery_size': 32*WORD,
-                         'large_object': 8*WORD}
+                         'large_object': 8*WORD,
+                         'translated_to_c': False}
             root_stack_depth = 200
 
     def define_ref_from_rawmalloced_to_regular(cls):
@@ -1477,6 +1484,7 @@
                          'large_object': 8*WORD,
                          'large_object_gcptrs': 10*WORD,
                          'card_page_indices': 4,
+                         'translated_to_c': False,
                          }
             root_stack_depth = 200
 
@@ -1585,7 +1593,8 @@
     gcname = "marksweep"
     class gcpolicy(gc.FrameworkGcPolicy):
         class transformerclass(framework.FrameworkGCTransformer):
-            GC_PARAMS = {'start_heap_size': 1024*WORD }
+            GC_PARAMS = {'start_heap_size': 1024*WORD,
+                         'translated_to_c': False}
             root_stack_depth = 200
 
 class TestHybridTaggedPointerGC(TaggedPointerGCTests):
@@ -1596,7 +1605,8 @@
             from pypy.rpython.memory.gc.generation import GenerationGC as \
                                                           GCClass
             GC_PARAMS = {'space_size': 512*WORD,
-                         'nursery_size': 32*WORD}
+                         'nursery_size': 32*WORD,
+                         'translated_to_c': False}
             root_stack_depth = 200
 
 class TestMarkCompactTaggedpointerGC(TaggedPointerGCTests):
@@ -1605,5 +1615,6 @@
     class gcpolicy(gc.FrameworkGcPolicy):
         class transformerclass(framework.FrameworkGCTransformer):
             from pypy.rpython.memory.gc.markcompact import MarkCompactGC as GCClass
-            GC_PARAMS = {'space_size': 4096*WORD}
+            GC_PARAMS = {'space_size': 4096*WORD,
+                         'translated_to_c': False}
             root_stack_depth = 200

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/module/ll_time.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/module/ll_time.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/module/ll_time.py	Sat Oct 30 16:41:17 2010
@@ -108,7 +108,7 @@
 
                 errcode = -1
                 if self.GETTIMEOFDAY_NO_TZ:
-                    errcode = g_gettimeofday(t)
+                    errcode = c_gettimeofday(t)
                 else:
                     errcode = c_gettimeofday(t, void)
 

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/rbuilder.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/rbuilder.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/rbuilder.py	Sat Oct 30 16:41:17 2010
@@ -36,8 +36,12 @@
         hop.exception_cannot_occur()
         return hop.gendirectcall(self.ll_append_multiple_char, *vlist)
 
+    def rtype_method_getlength(self, hop):
+        vlist = hop.inputargs(self)
+        hop.exception_cannot_occur()
+        return hop.gendirectcall(self.ll_getlength, *vlist)
+
     def rtype_method_build(self, hop):
         vlist = hop.inputargs(self)
         hop.exception_cannot_occur()
         return hop.gendirectcall(self.ll_build, *vlist)
-

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/rbuiltin.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/rbuiltin.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/rbuiltin.py	Sat Oct 30 16:41:17 2010
@@ -345,17 +345,22 @@
 BUILTIN_TYPER[object.__init__] = rtype_object__init__
 # annotation of low-level types
 
-def rtype_malloc(hop, i_flavor=None, i_zero=None):
+def rtype_malloc(hop, i_flavor=None, i_zero=None, i_track_allocation=None):
     assert hop.args_s[0].is_constant()
     vlist = [hop.inputarg(lltype.Void, arg=0)]
     opname = 'malloc'
-    v_flavor, v_zero = parse_kwds(hop, (i_flavor, lltype.Void), (i_zero, None))
+    v_flavor, v_zero, v_track_allocation = parse_kwds(hop,
+        (i_flavor, lltype.Void),
+        (i_zero, None),
+        (i_track_allocation, None))
 
     flags = {'flavor': 'gc'}
     if v_flavor is not None:
         flags['flavor'] = v_flavor.value
     if i_zero is not None:
         flags['zero'] = v_zero.value
+    if i_track_allocation is not None:
+        flags['track_allocation'] = v_track_allocation.value
     vlist.append(hop.inputconst(lltype.Void, flags))
         
     if hop.nb_args == 2:
@@ -366,10 +371,19 @@
     hop.exception_is_here()
     return hop.genop(opname, vlist, resulttype = hop.r_result.lowleveltype)
 
-def rtype_free(hop, i_flavor):
-    assert i_flavor == 1
+def rtype_free(hop, i_flavor, i_track_allocation=None):
+    vlist = [hop.inputarg(hop.args_r[0], arg=0)]
+    v_flavor, v_track_allocation = parse_kwds(hop,
+        (i_flavor, lltype.Void),
+        (i_track_allocation, None))
+    #
+    assert v_flavor is not None and v_flavor.value == 'raw'
+    flags = {'flavor': 'raw'}
+    if i_track_allocation is not None:
+        flags['track_allocation'] = v_track_allocation.value
+    vlist.append(hop.inputconst(lltype.Void, flags))
+    #
     hop.exception_cannot_occur()
-    vlist = hop.inputargs(hop.args_r[0], lltype.Void)
     hop.genop('free', vlist)
 
 def rtype_const_result(hop):
@@ -584,8 +598,9 @@
     vinst, = hop.inputargs(hop.args_r[0])
     flavor = hop.args_r[0].gcflavor
     assert flavor != 'gc'
-    cflavor = hop.inputconst(lltype.Void, flavor)
-    return hop.genop('free', [vinst, cflavor])
+    flags = {'flavor': flavor}
+    cflags = hop.inputconst(lltype.Void, flags)
+    return hop.genop('free', [vinst, cflags])
     
 BUILTIN_TYPER[objectmodel.free_non_gc_object] = rtype_free_non_gc_object
 

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/rpbc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/rpbc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/rpbc.py	Sat Oct 30 16:41:17 2010
@@ -256,6 +256,8 @@
     def convert_const(self, value):
         if isinstance(value, types.MethodType) and value.im_self is None:
             value = value.im_func   # unbound method -> bare function
+        elif isinstance(value, staticmethod):
+            value = value.__get__(42) # hackish, get the function wrapped by staticmethod
         if self.lowleveltype is Void:
             return None
         if value is None:

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/rtyper.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/rtyper.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/rtyper.py	Sat Oct 30 16:41:17 2010
@@ -421,7 +421,7 @@
             assert noexclink.exitcase is None
             if pos == "removed":
                 # the exception cannot actually occur at all.
-                # See for example rspecialcase.rtype_call_specialcase().
+                # This is set by calling exception_cannot_occur().
                 # We just remove all exception links.
                 block.exitswitch = None
                 block.exits = block.exits[:1]
@@ -1019,7 +1019,7 @@
 from pypy.rpython import rint, rbool, rfloat
 from pypy.rpython import rrange
 from pypy.rpython import rstr, rdict, rlist
-from pypy.rpython import rclass, rbuiltin, rpbc, rspecialcase
+from pypy.rpython import rclass, rbuiltin, rpbc
 from pypy.rpython import rexternalobj
 from pypy.rpython import rptr
 from pypy.rpython import rgeneric

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/test/test_llinterp.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/test/test_llinterp.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/test/test_llinterp.py	Sat Oct 30 16:41:17 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 import py
 import sys
 from pypy.rpython.lltypesystem.lltype import typeOf, pyobjectptr, Ptr,\
@@ -5,6 +6,7 @@
 from pypy.rpython.lltypesystem.lloperation import llop
 from pypy.rpython.llinterp import LLInterpreter, LLException, log
 from pypy.rpython.rmodel import inputconst
+from pypy.rpython.annlowlevel import hlstr
 from pypy.translator.translator import TranslationContext, graphof
 from pypy.rpython.rint import signed_repr
 from pypy.rpython.lltypesystem import rstr, lltype
@@ -12,13 +14,11 @@
 from pypy.annotation.model import lltype_to_annotation
 from pypy.rlib.rarithmetic import r_uint, ovfcheck
 from pypy.rpython.ootypesystem import ootype
+from pypy.tool import leakfinder
 from pypy import conftest
 
 # switch on logging of interp to show more info on failing tests
 
-class MallocMismatch(Exception):
-    pass
-
 def setup_module(mod):
     mod.logstate = py.log._getstate()
     py.log.setconsumer("llinterp", py.log.STDOUT)
@@ -72,7 +72,7 @@
 
 def get_interpreter(func, values, view='auto', viewbefore='auto', policy=None,
                     someobjects=False, type_system="lltype", backendopt=False,
-                    config=None, malloc_check=True, **extraconfigopts):
+                    config=None, **extraconfigopts):
     extra_key = [(key, value) for key, value in extraconfigopts.iteritems()]
     extra_key.sort()
     extra_key = tuple(extra_key)
@@ -97,7 +97,7 @@
                                    viewbefore, policy, type_system=type_system,
                                    backendopt=backendopt, config=config,
                                    **extraconfigopts)
-        interp = LLInterpreter(typer, malloc_check=malloc_check)
+        interp = LLInterpreter(typer)
         _tcache[key] = (t, interp, graph)
         # keep the cache small 
         _lastinterpreted.append(key) 
@@ -115,10 +115,17 @@
     interp, graph = get_interpreter(func, values, view, viewbefore, policy,
                                     someobjects, type_system=type_system,
                                     backendopt=backendopt, config=config,
-                                    malloc_check=malloc_check, **kwargs)
-    result = interp.eval_graph(graph, values)
-    if malloc_check and interp.mallocs:
-        raise MallocMismatch(interp.mallocs)
+                                    **kwargs)
+    if not malloc_check:
+        result = interp.eval_graph(graph, values)
+    else:
+        prev = leakfinder.start_tracking_allocations()
+        try:
+            result = interp.eval_graph(graph, values)
+        finally:
+            leaks = leakfinder.stop_tracking_allocations(False, prev)
+        if leaks:
+            raise leakfinder.MallocMismatch(leaks)
     return result
 
 def interpret_raises(exc, func, values, view='auto', viewbefore='auto',
@@ -418,6 +425,7 @@
             assert result
 
 def test_stack_malloc():
+    py.test.skip("stack-flavored mallocs no longer supported")
     class A(object):
         pass
     def f():
@@ -430,6 +438,7 @@
     assert result == 1
 
 def test_invalid_stack_access():
+    py.test.skip("stack-flavored mallocs no longer supported")
     class A(object):
         pass
     globala = A()
@@ -605,7 +614,7 @@
         if x:
             free(t, flavor='raw')
     interpret(f, [1])
-    py.test.raises(MallocMismatch, "interpret(f, [0])")
+    py.test.raises(leakfinder.MallocMismatch, "interpret(f, [0])")
     
     def f():
         t1 = malloc(T, flavor='raw')
@@ -615,3 +624,37 @@
 
     interpret(f, [])
 
+def test_context_manager():
+    state = []
+    class C:
+        def __enter__(self):
+            state.append('acquire')
+            return self
+        def __exit__(self, *args):
+            if args[1] is not None:
+                state.append('raised')
+            state.append('release')
+    def f():
+        try:
+            with C() as c:
+                state.append('use')
+                raise ValueError
+        except ValueError:
+            pass
+        return ', '.join(state)
+    res = interpret(f, [])
+    assert hlstr(res) == 'acquire, use, raised, release'
+
+
+def test_scoped_allocator():
+    from pypy.rpython.lltypesystem.lltype import scoped_alloc, Array, Signed
+    T = Array(Signed)
+    
+    def f():
+        x = 0
+        with scoped_alloc(T, 1) as array:
+            array[0] = -42
+            x = array[0]
+        assert x == -42
+
+    res = interpret(f, [])

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/test/test_nongc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/test/test_nongc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/test/test_nongc.py	Sat Oct 30 16:41:17 2010
@@ -79,7 +79,7 @@
     py.test.raises(TypeError,rtyper.specialize) # results in an invalid cast
 
 def test_isinstance():
-    class A:
+    class A(object):
         _alloc_flavor_ = "raw"
     class B(A):
         pass
@@ -95,7 +95,24 @@
             o = B()
         else:
             o = C()
-        return 100*isinstance(o, A)+10*isinstance(o, B)+1*isinstance(o ,C)
+        res = 100*isinstance(o, A) + 10*isinstance(o, B) + 1*isinstance(o, C)
+        if i == 0:
+            pass
+        elif i == 1:
+            assert isinstance(o, A)
+            free_non_gc_object(o)
+        elif i == 2:
+            assert isinstance(o, B)
+            free_non_gc_object(o)
+        else:
+            assert isinstance(o, C)
+            free_non_gc_object(o)
+        return res
+
+    assert f(1) == 100
+    assert f(2) == 110
+    assert f(3) == 111
+    assert f(0) == 0
 
     a = RPythonAnnotator()
     #does not raise:
@@ -131,10 +148,14 @@
             d = b
         elif i == 2:
             e = c
-        return (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
+        res =  (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
                 0x0008*(a is e) | 0x0010*(b is c) | 0x0020*(b is d) |
                 0x0040*(b is e) | 0x0080*(c is d) | 0x0100*(c is e) |
                 0x0200*(d is e))
+        free_non_gc_object(a)
+        free_non_gc_object(b)
+        free_non_gc_object(c)
+        return res
     a = RPythonAnnotator()
     #does not raise:
     s = a.build_types(f, [int])
@@ -169,10 +190,13 @@
             d = b
         elif i == 2:
             e = c
-        return (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
+        res =  (0x0001*(a is b) | 0x0002*(a is c) | 0x0004*(a is d) |
                 0x0008*(a is e) | 0x0010*(b is c) | 0x0020*(b is d) |
                 0x0040*(b is e) | 0x0080*(c is d) | 0x0100*(c is e) |
                 0x0200*(d is e))
+        free_non_gc_object(a)
+        free_non_gc_object(b)
+        return res
     a = RPythonAnnotator()
     #does not raise:
     s = a.build_types(f, [int])

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rbuilder.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rbuilder.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rbuilder.py	Sat Oct 30 16:41:17 2010
@@ -1,4 +1,4 @@
-
+import py
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.rpython.lltypesystem.rbuilder import *
 from pypy.rpython.annlowlevel import llstr, hlstr
@@ -55,8 +55,29 @@
         assert res == 'aabcabcdefbuuuu'
         assert isinstance(res, unicode)
 
+    def test_string_getlength(self):
+        def func():
+            s = StringBuilder()
+            s.append("a")
+            s.append("abc")
+            return s.getlength()
+        res = self.interpret(func, [])
+        assert res == 4
+
+    def test_unicode_getlength(self):
+        def func():
+            s = UnicodeBuilder()
+            s.append(u"a")
+            s.append(u"abc")
+            return s.getlength()
+        res = self.interpret(func, [])
+        assert res == 4
+
 class TestLLtype(BaseTestStringBuilder, LLRtypeMixin):
     pass
 
 class TestOOtype(BaseTestStringBuilder, OORtypeMixin):
-    pass
+    def test_string_getlength(self):
+        py.test.skip("getlength(): not implemented on ootype")
+    def test_unicode_getlength(self):
+        py.test.skip("getlength(): not implemented on ootype")

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rclass.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rclass.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rclass.py	Sat Oct 30 16:41:17 2010
@@ -3,7 +3,7 @@
 from pypy.translator.translator import TranslationContext, graphof
 from pypy.rpython.lltypesystem.lltype import *
 from pypy.rpython.ootypesystem import ootype
-from pypy.rlib.rarithmetic import intmask
+from pypy.rlib.rarithmetic import intmask, r_longlong
 from pypy.rpython.test.tool import BaseRtypingTest, LLRtypeMixin, OORtypeMixin
 from pypy.objspace.flow.model import summary
 
@@ -319,6 +319,17 @@
         res = self.interpret(f, [])
         assert res == 42
 
+    def test_staticmethod2(self):
+        class A(object):
+            f = staticmethod(lambda x, y: x*y)
+        class B(A):
+            f = staticmethod(lambda x, y: x+y)
+        def f():
+            b = B()
+            return b.f(6, 7)
+        res = self.interpret(f, [])
+        assert res == 13
+
     def test_is(self):
         class A: pass
         class B(A): pass
@@ -1001,6 +1012,40 @@
         res = self.interpret(f, [5])
         assert res == 0
 
+    def test_order_of_fields(self):
+        class A(object):
+            pass
+        def f(n):
+            a = A()
+            a.as_int = n
+            a.as_char = chr(n)
+            a.as_unichar = unichr(n)
+            a.as_double = n + 0.5
+            a.as_bool = bool(n)
+            a.as_void = None
+            a.as_longlong = r_longlong(n)
+            a.as_reference = A()
+            return a
+
+        res = self.interpret(f, [5])
+        names = list(typeOf(res).TO._names)
+        i = names.index('inst_as_int')
+        c = names.index('inst_as_char')
+        u = names.index('inst_as_unichar')
+        d = names.index('inst_as_double')
+        b = names.index('inst_as_bool')
+        v = names.index('inst_as_void')
+        l = names.index('inst_as_longlong')
+        r = names.index('inst_as_reference')
+        assert v == 1      # void fields are first
+        assert sorted([c, b]) == [7, 8]
+        if sys.maxint == 2147483647:
+            assert sorted([u, i, r]) == [4, 5, 6]        # 32-bit types
+            assert sorted([d, l]) == [2, 3]              # 64-bit types
+        else:
+            assert sorted([u]) == [6]                    # 32-bit types
+            assert sorted([i, r, d, l]) == [2, 3, 4, 5]  # 64-bit types
+
 
 class TestOOtype(BaseTestRclass, OORtypeMixin):
 

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rdict.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rdict.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rdict.py	Sat Oct 30 16:41:17 2010
@@ -682,6 +682,26 @@
         # if it does not crash, we are fine. It crashes if you forget the hash field.
         self.interpret(func, [])
 
+    def test_dict_popitem(self):
+        def func():
+            d = {}
+            d[5] = 2
+            d[6] = 3
+            k1, v1 = d.popitem()
+            assert len(d) == 1
+            k2, v2 = d.popitem()
+            try:
+                d.popitem()
+            except KeyError:
+                pass
+            else:
+                assert 0, "should have raised KeyError"
+            assert len(d) == 0
+            return k1*1000 + v1*100 + k2*10 + v2
+
+        res = self.interpret(func, [])
+        assert res in [5263, 6352]
+
     # ____________________________________________________________
 
     def test_opt_nullkeymarker(self):

Modified: pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rptr.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rptr.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rpython/test/test_rptr.py	Sat Oct 30 16:41:17 2010
@@ -212,10 +212,31 @@
 
     S = Struct('S', ('x', Signed))
     def fn(n):
-        p = malloc(S, flavor='whatever')
+        p = malloc(S, flavor='raw')
         p.x = n
         result = p.x
-        free(p, flavor='whatever')
+        free(p, flavor='raw')
+        return n
+
+    res = interpret(fn, [23])
+    assert res == 23
+
+    S = Struct('S', ('x', Signed))
+    def fn(n):
+        p = malloc(S, flavor='raw', track_allocation=False)
+        p.x = n
+        result = p.x
+        return n
+
+    res = interpret(fn, [23])
+    assert res == 23
+
+    S = Struct('S', ('x', Signed))
+    def fn(n):
+        p = malloc(S, flavor='raw', track_allocation=False)
+        p.x = n
+        result = p.x
+        free(p, flavor='raw', track_allocation=False)
         return n
 
     res = interpret(fn, [23])

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/funcgen.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/funcgen.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/funcgen.py	Sat Oct 30 16:41:17 2010
@@ -427,7 +427,7 @@
         r = self.expr(op.result)
         return 'OP_CALL_ARGS((%s), %s);' % (', '.join(args), r)
 
-    def generic_call(self, FUNC, fnexpr, args_v, v_result):
+    def generic_call(self, FUNC, fnexpr, args_v, v_result, targets=None):
         args = []
         assert len(args_v) == len(FUNC.TO.ARGS)
         for v, ARGTYPE in zip(args_v, FUNC.TO.ARGS):
@@ -444,17 +444,26 @@
             # skip assignment of 'void' return value
             r = self.expr(v_result)
             line = '%s = %s' % (r, line)
+        if targets:
+            for graph in targets:
+                if getattr(graph, 'inhibit_tail_call', False):
+                    line += '\nPYPY_INHIBIT_TAIL_CALL();'
+                    break
         return line
 
     def OP_DIRECT_CALL(self, op):
         fn = op.args[0]
+        try:
+            targets = [fn.value._obj.graph]
+        except AttributeError:
+            targets = None
         return self.generic_call(fn.concretetype, self.expr(fn),
-                                 op.args[1:], op.result)
+                                 op.args[1:], op.result, targets)
 
     def OP_INDIRECT_CALL(self, op):
         fn = op.args[0]
         return self.generic_call(fn.concretetype, self.expr(fn),
-                                 op.args[1:-1], op.result)
+                                 op.args[1:-1], op.result, op.args[-1].value)
 
     def OP_ADR_CALL(self, op):
         ARGTYPES = [v.concretetype for v in op.args[1:]]

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/instruction.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/instruction.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/instruction.py	Sat Oct 30 16:41:17 2010
@@ -82,6 +82,11 @@
     def all_sources_of(self, localvar):
         return [localvar]
 
+class InsnCondJump(Insn):     # only for debugging; not used internally
+    _args_ = ['label']
+    def __init__(self, label):
+        self.label = label
+
 class Label(Insn):
     _args_ = ['label', 'lineno']
     def __init__(self, label, lineno):
@@ -170,9 +175,12 @@
         self.delta = -7     # use an odd value as marker
 
 class InsnStop(Insn):
-    pass
+    _args_ = ['reason']
+    def __init__(self, reason='?'):
+        self.reason = reason
 
 class InsnRet(InsnStop):
+    _args_ = []
     framesize = 0
     def __init__(self, registers):
         self.registers = registers

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/elf/track5.s
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/elf/track5.s	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/elf/track5.s	Sat Oct 30 16:41:17 2010
@@ -44,7 +44,7 @@
 	addl	%eax, %ebx
 	jmp	.L1221
 .L1227:
-	call	RPyAbort
+	;;call	RPyAbort
 	cmpl	12(%esi), %ebx
 	jb	.L1229
 	addl	$20, %esp

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/test_trackgcroot.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/test_trackgcroot.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/test/test_trackgcroot.py	Sat Oct 30 16:41:17 2010
@@ -109,7 +109,7 @@
  
 def test_computegcmaptable():
     tests = []
-    for format in ('elf', 'darwin', 'msvc', 'elf64'):
+    for format in ('elf', 'elf64', 'darwin', 'darwin64', 'msvc'):
         for path in this_dir.join(format).listdir("track*.s"):
             n = path.purebasename[5:]
             try:

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/trackgcroot.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/trackgcroot.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/gcc/trackgcroot.py	Sat Oct 30 16:41:17 2010
@@ -6,7 +6,7 @@
 from pypy.translator.c.gcc.instruction import InsnFunctionStart, InsnStop
 from pypy.translator.c.gcc.instruction import InsnSetLocal, InsnCopyLocal
 from pypy.translator.c.gcc.instruction import InsnPrologue, InsnEpilogue
-from pypy.translator.c.gcc.instruction import InsnGCROOT
+from pypy.translator.c.gcc.instruction import InsnGCROOT, InsnCondJump
 from pypy.translator.c.gcc.instruction import InsnStackAdjust
 from pypy.translator.c.gcc.instruction import InsnCannotFollowEsp
 from pypy.translator.c.gcc.instruction import LocalVar, somenewvalue
@@ -46,6 +46,7 @@
         self.findlabels()
         self.parse_instructions()
         try:
+            self.trim_unreachable_instructions()
             self.find_noncollecting_calls()
             if not self.list_collecting_call_insns():
                 return []
@@ -122,19 +123,36 @@
                 assert label not in self.labels, "duplicate label: %s" % label
                 self.labels[label] = Label(label, lineno)
 
+    def trim_unreachable_instructions(self):
+        reached = set([self.insns[0]])
+        prevlen = 0
+        while len(reached) > prevlen:
+            prevlen = len(reached)
+            for insn in self.insns:
+                if insn not in reached:
+                    for previnsn in insn.previous_insns:
+                        if previnsn in reached:
+                            # this instruction is reachable too
+                            reached.add(insn)
+                            break
+        # now kill all unreachable instructions
+        i = 0
+        while i < len(self.insns):
+            if self.insns[i] in reached:
+                i += 1
+            else:
+                del self.insns[i]
+
     def find_noncollecting_calls(self):
-        cannot_collect = self.CANNOT_COLLECT.copy()
+        cannot_collect = {}
         for line in self.lines:
             match = self.r_gcnocollect_marker.search(line)
             if match:
                 name = match.group(1)
                 cannot_collect[name] = True
         #
-        if self.format in ('darwin', 'mingw32', 'msvc'):
-            self.cannot_collect = dict.fromkeys(
-                ['_' + name for name in cannot_collect])
-        else:
-            self.cannot_collect = cannot_collect
+        self.cannot_collect = dict.fromkeys(
+            [self.function_names_prefix + name for name in cannot_collect])
 
     def append_instruction(self, insn):
         # Add the instruction to the list, and link it to the previous one.
@@ -410,7 +428,8 @@
         return result
     # ____________________________________________________________
 
-    CANNOT_COLLECT = {    # some of the most used functions that cannot collect
+    BASE_FUNCTIONS_NOT_RETURNING = {
+        'abort': None,
         'pypy_debug_catch_fatal_exception': None,
         'RPyAbort': None,
         'RPyAssertFailed': None,
@@ -427,6 +446,7 @@
     IGNORE_OPS_WITH_PREFIXES = dict.fromkeys([
         'cmp', 'test', 'set', 'sahf', 'lahf', 'cltd', 'cld', 'std',
         'rep', 'movs', 'lods', 'stos', 'scas', 'cwtl', 'cwde', 'prefetch',
+        'pslld', 
         # floating-point operations cannot produce GC pointers
         'f',
         'cvt', 'ucomi', 'comi', 'subs', 'subp' , 'adds', 'addp', 'xorp',
@@ -438,7 +458,7 @@
         'bswap', 'bt', 'rdtsc',
         'punpck', 'pshufd', 
         # zero-extending moves should not produce GC pointers
-        'movz',
+        'movz', 
         ])
 
     visit_movb = visit_nop
@@ -644,7 +664,7 @@
                 if label != '0':
                     self.register_jump_to(label)
                 tablelin += 1
-            return InsnStop()
+            return InsnStop("jump table")
         if self.r_unaryinsn_star.match(line):
             # that looks like an indirect tail-call.
             # tail-calls are equivalent to RET for us
@@ -658,7 +678,7 @@
             assert not target.startswith('.')
             # tail-calls are equivalent to RET for us
             return InsnRet(self.CALLEE_SAVE_REGISTERS)
-        return InsnStop()
+        return InsnStop("jump")
     
     def register_jump_to(self, label):
         if not isinstance(self.insns[-1], InsnStop):
@@ -682,7 +702,7 @@
         else:
             label = match.group(1)
         self.register_jump_to(label)
-        return []
+        return [InsnCondJump(label)]
 
     visit_jmpl = visit_jmp
     visit_je = conditional_jump
@@ -754,7 +774,7 @@
                     target, = sources
 
         if target in self.FUNCTIONS_NOT_RETURNING:
-            return [InsnStop(), InsnCannotFollowEsp()]
+            return [InsnStop(target)]
         if self.format == 'mingw32' and target == '__alloca':
             # in functions with large stack requirements, windows
             # needs a call to _alloca(), to turn reserved pages
@@ -885,7 +905,7 @@
             # statically known pointer to a register
 
             # %eax -> %rax
-            new_line = re.sub(r"%e(ax|bx|cx|dx|di|si)$", r"%r\1", line)
+            new_line = re.sub(r"%e(ax|bx|cx|dx|di|si|bp)$", r"%r\1", line)
             # %r10d -> %r10
             new_line = re.sub(r"%r(\d+)d$", r"%r\1", new_line)
             return func(self, new_line)
@@ -951,6 +971,7 @@
 
 class ElfFunctionGcRootTracker32(FunctionGcRootTracker32):
     format = 'elf'
+    function_names_prefix = ''
 
     ESP     = '%esp'
     EBP     = '%ebp'
@@ -984,13 +1005,14 @@
     r_bottom_marker = re.compile(r"\t/[*] GC_STACK_BOTTOM [*]/")
 
     FUNCTIONS_NOT_RETURNING = {
-        'abort': None,
         '_exit': None,
         '__assert_fail': None,
         '___assert_rtn': None,
         'L___assert_rtn$stub': None,
         'L___eprintf$stub': None,
         }
+    for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING:
+        FUNCTIONS_NOT_RETURNING[_name] = None
 
     def __init__(self, lines, filetag=0):
         match = self.r_functionstart.match(lines[0])
@@ -1010,6 +1032,8 @@
 
 class ElfFunctionGcRootTracker64(FunctionGcRootTracker64):
     format = 'elf64'
+    function_names_prefix = ''
+
     ESP = '%rsp'
     EBP = '%rbp'
     EAX = '%rax'
@@ -1042,13 +1066,14 @@
     r_bottom_marker = re.compile(r"\t/[*] GC_STACK_BOTTOM [*]/")
 
     FUNCTIONS_NOT_RETURNING = {
-        'abort': None,
         '_exit': None,
         '__assert_fail': None,
         '___assert_rtn': None,
         'L___assert_rtn$stub': None,
         'L___eprintf$stub': None,
         }
+    for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING:
+        FUNCTIONS_NOT_RETURNING[_name] = None
 
     def __init__(self, lines, filetag=0):
         match = self.r_functionstart.match(lines[0])
@@ -1066,8 +1091,9 @@
 
 ElfFunctionGcRootTracker64.init_regexp()
 
-class DarwinFunctionGcRootTracker(ElfFunctionGcRootTracker32):
+class DarwinFunctionGcRootTracker32(ElfFunctionGcRootTracker32):
     format = 'darwin'
+    function_names_prefix = '_'
 
     r_functionstart = re.compile(r"_(\w+):\s*$")
     OFFSET_LABELS   = 0
@@ -1077,17 +1103,36 @@
         funcname = '_' + match.group(1)
         FunctionGcRootTracker32.__init__(self, funcname, lines, filetag)
 
-class Mingw32FunctionGcRootTracker(DarwinFunctionGcRootTracker):
+class DarwinFunctionGcRootTracker64(ElfFunctionGcRootTracker64):
+    format = 'darwin64'
+    function_names_prefix = '_'
+
+    LABEL = ElfFunctionGcRootTracker32.LABEL
+    r_jmptable_item = re.compile(r"\t.(?:long|quad)\t"+LABEL+"(-\"?[A-Za-z0-9$]+\"?)?\s*$")
+
+    r_functionstart = re.compile(r"_(\w+):\s*$")
+    OFFSET_LABELS   = 0
+
+    def __init__(self, lines, filetag=0):
+        match = self.r_functionstart.match(lines[0])
+        funcname = '_' + match.group(1)
+        FunctionGcRootTracker64.__init__(self, funcname, lines, filetag)
+
+class Mingw32FunctionGcRootTracker(DarwinFunctionGcRootTracker32):
     format = 'mingw32'
+    function_names_prefix = '_'
 
     FUNCTIONS_NOT_RETURNING = {
-        '_abort': None,
         '_exit': None,
         '__assert': None,
         }
+    for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING:
+        FUNCTIONS_NOT_RETURNING['_' + _name] = None
 
 class MsvcFunctionGcRootTracker(FunctionGcRootTracker32):
     format = 'msvc'
+    function_names_prefix = '_'
+
     ESP = 'esp'
     EBP = 'ebp'
     EAX = 'eax'
@@ -1127,7 +1172,6 @@
     r_bottom_marker = re.compile(r"; .+\tpypy_asm_stack_bottom\(\);")
 
     FUNCTIONS_NOT_RETURNING = {
-        '_abort': None,
         '__exit': None,
         '__assert': None,
         '__wassert': None,
@@ -1136,6 +1180,8 @@
         'DWORD PTR __imp__abort': None,
         'DWORD PTR __imp___wassert': None,
         }
+    for _name in FunctionGcRootTracker.BASE_FUNCTIONS_NOT_RETURNING:
+        FUNCTIONS_NOT_RETURNING['_' + _name] = None
 
     @classmethod
     def init_regexp(cls):
@@ -1343,7 +1389,7 @@
 
 class DarwinAssemblerParser(AssemblerParser):
     format = "darwin"
-    FunctionGcRootTracker = DarwinFunctionGcRootTracker
+    FunctionGcRootTracker = DarwinFunctionGcRootTracker32
 
     r_textstart = re.compile(r"\t.text\s*$")
 
@@ -1389,6 +1435,10 @@
         return super(DarwinAssemblerParser, self).process_function(
             lines, entrypoint, filename)
 
+class DarwinAssemblerParser64(DarwinAssemblerParser):
+    format = "darwin64"
+    FunctionGcRootTracker = DarwinFunctionGcRootTracker64
+
 class Mingw32AssemblerParser(DarwinAssemblerParser):
     format = "mingw32"
     FunctionGcRootTracker = Mingw32FunctionGcRootTracker
@@ -1512,6 +1562,7 @@
     'elf': ElfAssemblerParser,
     'elf64': ElfAssemblerParser64,
     'darwin': DarwinAssemblerParser,
+    'darwin64': DarwinAssemblerParser64,
     'mingw32': Mingw32AssemblerParser,
     'msvc': MsvcAssemblerParser,
     }
@@ -1543,15 +1594,13 @@
         assert self.seen_main
 
         def _globalname(name, disp=""):
-            if self.format in ('darwin', 'mingw32', 'msvc'):
-                name = '_' + name
-            return name
+            return tracker_cls.function_names_prefix + name
 
         def _variant(**kwargs):
             txt = kwargs[self.format]
             print >> output, "\t%s" % txt
 
-        if self.format == 'elf64':
+        if self.format in ('elf64', 'darwin64'):
             word_decl = '.quad'
         else:
             word_decl = '.long'
@@ -1604,10 +1653,11 @@
                }
             }
             """
-        elif self.format == 'elf64':
+        elif self.format in ('elf64', 'darwin64'):
             print >> output, "\t.text"
             print >> output, "\t.globl %s" % _globalname('pypy_asm_stackwalk')
-            print >> output, "\t.type pypy_asm_stackwalk, @function"
+            _variant(elf64='.type pypy_asm_stackwalk, @function',
+                     darwin64='')
             print >> output, "%s:" % _globalname('pypy_asm_stackwalk')
 
             print >> output, """\
@@ -1652,8 +1702,9 @@
             /* the return value is the one of the 'call' above, */
             /* because %rax (and possibly %rdx) are unmodified  */
             ret
-            .size pypy_asm_stackwalk, .-pypy_asm_stackwalk
             """
+            _variant(elf64='.size pypy_asm_stackwalk, .-pypy_asm_stackwalk',
+                     darwin64='')
         else:
             print >> output, "\t.text"
             print >> output, "\t.globl %s" % _globalname('pypy_asm_stackwalk')
@@ -1780,6 +1831,7 @@
             _variant(elf='.section\t.rodata',
                      elf64='.section\t.rodata',
                      darwin='.const',
+                     darwin64='.const',
                      mingw32='')
 
             print >> output, """\
@@ -1857,7 +1909,10 @@
     shuffle = False
     output_raw_table = False
     if sys.platform == 'darwin':
-        format = 'darwin'
+        if sys.maxint > 2147483647:
+            format = 'darwin64'
+        else:
+            format = 'darwin'
     elif sys.platform == 'win32':
         format = 'mingw32'
     else:

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/src/g_include.h
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/src/g_include.h	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/src/g_include.h	Sat Oct 30 16:41:17 2010
@@ -53,6 +53,7 @@
 #  include "src/rtyper.h"
 #  include "src/debug_print.h"
 #  include "src/debug_traceback.h"
+#  include "src/debug_alloc.h"
 #ifndef AVR
 #  include "src/ll_os.h"
 #  include "src/ll_strtod.h"

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/src/main.h
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/src/main.h	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/src/main.h	Sat Oct 30 16:41:17 2010
@@ -53,10 +53,16 @@
     }
 
     exitcode = STANDALONE_ENTRY_POINT(list);
+
+#ifdef RPY_ASSERT
+    pypy_debug_alloc_results();
+#endif
+
     if (RPyExceptionOccurred()) {
         /* print the RPython traceback */
         pypy_debug_catch_fatal_exception();
     }
+
     return exitcode;
 
  memory_out:

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/src/signals.h
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/src/signals.h	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/src/signals.h	Sat Oct 30 16:41:17 2010
@@ -6,20 +6,6 @@
 
 #include 
 
-#ifndef LONG_MAX
-#if SIZEOF_LONG == 4
-#define LONG_MAX 0X7FFFFFFFL
-#elif SIZEOF_LONG == 8
-#define LONG_MAX 0X7FFFFFFFFFFFFFFFL
-#else
-#error "could not set LONG_MAX in pyport.h"
-#endif
-#endif
-
-#ifndef LONG_MIN
-#define LONG_MIN (-LONG_MAX-1)
-#endif
-
 #include 
 
 #ifdef MS_WINDOWS
@@ -28,10 +14,6 @@
 
 #include 
 
-#ifndef SIG_ERR
-#define SIG_ERR ((PyOS_sighandler_t)(-1))
-#endif
-
 #if defined(PYOS_OS2) && !defined(PYCC_GCC)
 #define NSIG 12
 #include 
@@ -65,11 +47,11 @@
 /* utility to poll for signals that arrived */
 int pypysig_poll(void);   /* => signum or -1 */
 
-/* When a signal is received, the high bit of pypysig_occurred is set.
-   After all signals are processed by pypysig_poll(), the high bit is
+/* When a signal is received, the bit 30 of pypysig_occurred is set.
+   After all signals are processed by pypysig_poll(), the bit 30 is
    cleared again.  The variable is exposed and RPython code is free to
    use the other bits in any way. */
-#define PENDING_SIGNAL_BIT   (LONG_MIN)   /* high bit */
+#define PENDING_SIGNAL_BIT   (1 << 30)
 /* This is a struct for the JIT. See interp_signal.py. */
 struct pypysig_long_struct {
     long value;

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/src/stack.h
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/src/stack.h	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/src/stack.h	Sat Oct 30 16:41:17 2010
@@ -33,6 +33,12 @@
 		&& LL_stack_too_big_slowpath());
 }
 
+#ifdef __GNUC__
+#  define PYPY_INHIBIT_TAIL_CALL()   asm("/* inhibit_tail_call */")
+#else
+#  define PYPY_INHIBIT_TAIL_CALL()   /* add hints for other compilers here */
+#endif
+
 
 #ifndef PYPY_NOT_MAIN_FILE
 #include 

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_genc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_genc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_genc.py	Sat Oct 30 16:41:17 2010
@@ -426,6 +426,7 @@
     if py.test.config.option.view:
         t.view()
     assert ' BarStruct ' in t.driver.cbuilder.c_source_filename.read()
+    free(foo, flavor="raw")
 
 def test_recursive_llhelper():
     from pypy.rpython.annlowlevel import llhelper
@@ -473,7 +474,27 @@
         return f(s)
     a_f = A(f, "f")
     a_g = A(g, "g")
-    t = lltype.malloc(STRUCT, flavor="raw")
+    t = lltype.malloc(STRUCT, flavor="raw", immortal=True)
     t.bar = llhelper(FTPTR, a_f.make_func())
     fn = compile(chooser, [bool])
     assert fn(True)
+
+def test_inhibit_tail_call():
+    from pypy.rpython.lltypesystem import lltype
+    def foobar_fn(n):
+        return 42
+    foobar_fn._dont_inline_ = True
+    def main(n):
+        return foobar_fn(n)
+    #
+    t = Translation(main, [int], backend="c")
+    t.rtype()
+    t.context._graphof(foobar_fn).inhibit_tail_call = True
+    t.source_c()
+    lines = t.driver.cbuilder.c_source_filename.readlines()
+    for i, line in enumerate(lines):
+        if '= pypy_g_foobar_fn' in line:
+            break
+    else:
+        assert 0, "the call was not found in the C source"
+    assert 'PYPY_INHIBIT_TAIL_CALL();' in lines[i+1]

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_lltyped.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_lltyped.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_lltyped.py	Sat Oct 30 16:41:17 2010
@@ -401,6 +401,7 @@
             for i in range(n):
                 p = malloc(S, flavor='raw', zero=True)
                 if p.x != 0 or p.y != 0:
+                    free(p, flavor='raw')
                     return -1
                 p.x = i
                 p.y = i
@@ -418,14 +419,16 @@
         def f(n):
             for length in range(n-1, -1, -1):
                 p = malloc(S, length, flavor='raw', zero=True)
-                if p.x != 0:
-                    return -1
-                p.x = n
-                for j in range(length):
-                    if p.y[j] != 0:
-                        return -3
-                    p.y[j] = n^j
-                free(p, flavor='raw')
+                try:
+                    if p.x != 0:
+                        return -1
+                    p.x = n
+                    for j in range(length):
+                        if p.y[j] != 0:
+                            return -3
+                        p.y[j] = n^j
+                finally:
+                    free(p, flavor='raw')
             return 42
 
         fn = self.getcompiled(f, [int])
@@ -655,7 +658,7 @@
     def test_prebuilt_ll2ctypes_array(self):
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         A = rffi.CArray(Char)
-        a = malloc(A, 6, flavor='raw')
+        a = malloc(A, 6, flavor='raw', immortal=True)
         a[0] = 'a'
         a[1] = 'b'
         a[2] = 'c'
@@ -676,7 +679,7 @@
     def test_ll2ctypes_array_from_c(self):
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         A = rffi.CArray(Char)
-        a = malloc(A, 6, flavor='raw')
+        a = malloc(A, 6, flavor='raw', immortal=True)
         a[0] = 'a'
         a[1] = 'b'
         a[2] = 'c'

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_newgc.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_newgc.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_newgc.py	Sat Oct 30 16:41:17 2010
@@ -624,13 +624,13 @@
         os.unlink(self.filename)
 
     def define_callback_with_collect(cls):
-        from pypy.rlib.libffi import ffi_type_pointer, cast_type_to_ffitype,\
+        from pypy.rlib.clibffi import ffi_type_pointer, cast_type_to_ffitype,\
              CDLL, ffi_type_void, CallbackFuncPtr, ffi_type_sint
         from pypy.rpython.lltypesystem import rffi, ll2ctypes
         import gc
         ffi_size_t = cast_type_to_ffitype(rffi.SIZE_T)
 
-        from pypy.rlib.libffi import get_libc_name
+        from pypy.rlib.clibffi import get_libc_name
 
         def callback(ll_args, ll_res, stuff):
             gc.collect()

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_standalone.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_standalone.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_standalone.py	Sat Oct 30 16:41:17 2010
@@ -16,11 +16,16 @@
 class StandaloneTests(object):
     config = None
 
-    def compile(self, entry_point, debug=True, shared=False):
+    def compile(self, entry_point, debug=True, shared=False,
+                stackcheck=False):
         t = TranslationContext(self.config)
         t.buildannotator().build_types(entry_point, [s_list_of_strings])
         t.buildrtyper().specialize()
 
+        if stackcheck:
+            from pypy.translator.transform import insert_ll_stackcheck
+            insert_ll_stackcheck(t)
+
         t.config.translation.shared = shared
 
         cbuilder = CStandaloneBuilder(t, entry_point, t.config)
@@ -630,6 +635,22 @@
             else:
                 os.environ['CC'] = old_cc
 
+    def test_inhibit_tail_call(self):
+        # the point is to check that the f()->f() recursion stops
+        from pypy.rlib.rstackovf import StackOverflow
+        def f(n):
+            if n <= 0:
+                return 42
+            return f(n+1)
+        def entry_point(argv):
+            try:
+                return f(1)
+            except StackOverflow:
+                print 'hi!'
+                return 0
+        t, cbuilder = self.compile(entry_point, stackcheck=True)
+        out = cbuilder.cmdexec("")
+        assert out.strip() == "hi!"
 
 class TestMaemo(TestStandalone):
     def setup_class(cls):

Modified: pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_typed.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_typed.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/c/test/test_typed.py	Sat Oct 30 16:41:17 2010
@@ -1,3 +1,4 @@
+from __future__ import with_statement
 import autopath
 import sys
 import math
@@ -823,3 +824,41 @@
             while int(x + frac) >= -sys.maxint-1:
                 x -= 1
             assert f(x + frac) == -666
+
+    def test_context_manager(self):
+        state = []
+        class C:
+            def __init__(self, name):
+                self.name = name
+            def __enter__(self):
+                state.append('acquire')
+                return self
+            def __exit__(self, typ, value, tb):
+                if typ is not None:
+                    if value is None:
+                        raise RuntimeError('test failed')
+                    state.append('raised')
+                else:
+                    if value is not None:
+                        raise RuntimeError('test failed')
+                state.append('release')
+
+        def func(n):
+            del state[:]
+            try:
+                with C('hello') as c:
+                    state.append(c.name)
+                    if n == 1:
+                        raise ValueError
+                    elif n == 2:
+                        raise TypeError
+            except (ValueError, TypeError):
+                pass
+            return ', '.join(state)
+        f = self.getcompiled(func, [int])
+        res = f(0)
+        assert res == 'acquire, hello, release'
+        res = f(1)
+        assert res == 'acquire, hello, raised, release'
+        res = f(2)
+        assert res == 'acquire, hello, raised, release'

Modified: pypy/branch/jit-unroll-loops/pypy/translator/driver.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/driver.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/driver.py	Sat Oct 30 16:41:17 2010
@@ -426,6 +426,22 @@
                                   [OOTYPE],
                                   "JIT compiler generation")
 
+    def task_jittest_lltype(self):
+        """ Run with the JIT on top of the llgraph backend
+        """
+        # parent process loop: spawn a child, wait for the child to finish,
+        # print a message, and restart
+        from pypy.translator.goal import unixcheckpoint
+        unixcheckpoint.restartable_point(auto='run')
+        # load the module pypy/jit/tl/jittest.py, which you can hack at
+        # and restart without needing to restart the whole translation process
+        from pypy.jit.tl import jittest
+        jittest.jittest(self)
+    #
+    task_jittest_lltype = taskdef(task_jittest_lltype,
+                                  [RTYPE],
+                                  "test of the JIT on the llgraph backend")
+
     def task_backendopt_lltype(self):
         """ Run all backend optimizations - lltype version
         """
@@ -433,7 +449,8 @@
         backend_optimizations(self.translator)
     #
     task_backendopt_lltype = taskdef(task_backendopt_lltype,
-                                     [RTYPE, '??pyjitpl_lltype'],
+                                     [RTYPE, '??pyjitpl_lltype',
+                                             '??jittest_lltype'],
                                      "lltype back-end optimisations")
     BACKENDOPT = 'backendopt_lltype'
 

Modified: pypy/branch/jit-unroll-loops/pypy/translator/goal/ann_override.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/goal/ann_override.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/goal/ann_override.py	Sat Oct 30 16:41:17 2010
@@ -27,21 +27,6 @@
         pol.pypytypes = {}
         pol.single_space = single_space
 
-    #def override__wrap_exception_cls(pol, space, x):
-    #    import pypy.objspace.std.typeobject as typeobject
-    #    clsdef = getbookkeeper().getuniqueclassdef(typeobject.W_TypeObject)
-    #    return annmodel.SomeInstance(clsdef, can_be_None=True)
-    #
-    #def override__fake_object(pol, space, x):
-    #    from pypy.interpreter import typedef
-    #    clsdef = getbookkeeper().getuniqueclassdef(typedef.W_Root)
-    #    return annmodel.SomeInstance(clsdef)    
-    #
-    #def override__cpy_compile(pol, self, source, filename, mode, flags):
-    #    from pypy.interpreter import pycode
-    #    clsdef = getbookkeeper().getuniqueclassdef(pycode.PyCode)
-    #    return annmodel.SomeInstance(clsdef)    
-
     def specialize__wrap(pol,  funcdesc, args_s):
         from pypy.interpreter.baseobjspace import Wrappable
         from pypy.annotation.classdef import ClassDef

Modified: pypy/branch/jit-unroll-loops/pypy/translator/goal/app_main.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/goal/app_main.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/goal/app_main.py	Sat Oct 30 16:41:17 2010
@@ -255,7 +255,7 @@
             break
         elif arg == '-u':
             unbuffered = True
-        elif arg == '-O':
+        elif arg == '-O' or arg == '-OO':
             pass
         elif arg == '--version' or arg == '-V':
             print "Python", sys.version
@@ -385,9 +385,12 @@
                 python_startup = os.getenv('PYTHONSTARTUP')
                 if python_startup:
                     try:
-                        startup = open(python_startup).read()
-                    except IOError:
-                        pass
+                        f = open(python_startup)
+                        startup = f.read()
+                        f.close()
+                    except IOError, e:
+                        print >> sys.stderr, "Could not open PYTHONSTARTUP"
+                        print >> sys.stderr, "IOError:", e
                     else:
                         def run_it():
                             co_python_startup = compile(startup,

Modified: pypy/branch/jit-unroll-loops/pypy/translator/goal/translate.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/goal/translate.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/goal/translate.py	Sat Oct 30 16:41:17 2010
@@ -27,6 +27,7 @@
         ("annotate", "do type inference", "-a --annotate", ""),
         ("rtype", "do rtyping", "-t --rtype", ""),
         ("pyjitpl", "JIT generation step", "--pyjitpl", ""),
+        ("jittest", "JIT test with llgraph backend", "--jittest", ""),
         ("backendopt", "do backend optimizations", "--backendopt", ""),
         ("source", "create source", "-s --source", ""),
         ("compile", "compile", "-c --compile", " (default goal)"),

Modified: pypy/branch/jit-unroll-loops/pypy/translator/jvm/test/test_class.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/jvm/test/test_class.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/jvm/test/test_class.py	Sat Oct 30 16:41:17 2010
@@ -1,6 +1,6 @@
 import py
 from pypy.translator.jvm.test.runtest import JvmTest
-from pypy.translator.oosupport.test_template.class_ import BaseTestClass, BaseTestSpecialcase
+from pypy.translator.oosupport.test_template.class_ import BaseTestClass
 
 class TestJvmClass(JvmTest, BaseTestClass):    
     def test_overridden_classattr_as_defaults(self):
@@ -26,6 +26,3 @@
             
     def test_specialize_methods(self):
         py.test.skip('ABSTRACT METHOD FIX: RE-TEST AFTER MERGE')
-
-class TestJvmSpecialCase(JvmTest, BaseTestSpecialcase):
-    pass

Modified: pypy/branch/jit-unroll-loops/pypy/translator/oosupport/test_template/class_.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/oosupport/test_template/class_.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/oosupport/test_template/class_.py	Sat Oct 30 16:41:17 2010
@@ -1,6 +1,5 @@
 import py
 from pypy.rpython.test import test_rclass
-from pypy.rpython.test.test_rspecialcase import BaseTestRspecialcase
 
 class BaseTestClass(test_rclass.TestOOtype):
     def test_abstract_method(self):
@@ -66,6 +65,3 @@
 
     def test_cast_object_mix_null(self):
         py.test.skip('cannot return ootype.NULL from translated functions')
-
-class BaseTestSpecialcase(BaseTestRspecialcase):
-    pass

Modified: pypy/branch/jit-unroll-loops/pypy/translator/platform/darwin.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/platform/darwin.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/platform/darwin.py	Sat Oct 30 16:41:17 2010
@@ -14,7 +14,10 @@
     
     def __init__(self, cc=None):
         if cc is None:
-            cc = 'gcc'
+            try:
+                cc = os.environ['CC']
+            except KeyError:
+                cc = 'gcc'
         self.cc = cc
 
     def _args_for_shared(self, args):

Modified: pypy/branch/jit-unroll-loops/pypy/translator/transform.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/translator/transform.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/translator/transform.py	Sat Oct 30 16:41:17 2010
@@ -221,15 +221,19 @@
     stack_check_ptr_const = Constant(stack_check_ptr, lltype.typeOf(stack_check_ptr))
     edges = set()
     insert_in = set()
+    block2graph = {}
     for caller in translator.graphs:
         for block, callee in find_calls_from(translator, caller):
             if getattr(getattr(callee, 'func', None),
                        'insert_stack_check_here', False):
                 insert_in.add(callee.startblock)
+                block2graph[callee.startblock] = callee
                 continue
             if block is not caller.startblock:
                 edges.add((caller.startblock, block))
+                block2graph[caller.startblock] = caller
             edges.add((block, callee.startblock))
+            block2graph[block] = caller
 
     edgelist = [Edge(block1, block2) for (block1, block2) in edges]
     edgedict = make_edge_dict(edgelist)
@@ -241,6 +245,10 @@
         v.concretetype = lltype.Void
         unwind_op = SpaceOperation('direct_call', [stack_check_ptr_const], v)
         block.operations.insert(0, unwind_op)
+        # prevents cycles of tail calls from occurring -- such cycles would
+        # not consume any stack, so would turn into potentially infinite loops
+        graph = block2graph[block]
+        graph.inhibit_tail_call = True
     return len(insert_in)
 
 


From hakanardo at codespeak.net  Sat Oct 30 16:53:10 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 16:53:10 +0200 (CEST)
Subject: [pypy-svn] r78531 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030145310.BFC9D282BF2@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 16:53:09 2010
New Revision: 78531

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
Log:
(hakanardo, cfbolz, arigo) fixed merge

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	Sat Oct 30 16:53:09 2010
@@ -11,7 +11,7 @@
 from pypy.jit.metainterp.jitprof import EmptyProfiler
 from pypy.jit.metainterp import executor, compile, resume, history
 from pypy.jit.metainterp.resoperation import rop, opname, ResOperation
-from pypy.jit.metainterp.test.oparser import pure_parse
+from pypy.jit.tool.oparser import pure_parse
 
 ##class FakeFrame(object):
 ##    parent_resumedata_snapshot = None


From hakanardo at codespeak.net  Sat Oct 30 16:53:26 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 16:53:26 +0200 (CEST)
Subject: [pypy-svn] r78532 - pypy/branch/jit-unroll-loops/pypy/jit/tool
Message-ID: <20101030145326.8852A282BF4@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 16:53:25 2010
New Revision: 78532

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/tool/oparser.py
Log:
(hakanardo, cfbolz, arigo) fixed merge

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tool/oparser.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tool/oparser.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tool/oparser.py	Sat Oct 30 16:53:25 2010
@@ -31,6 +31,9 @@
     def getopnum(self):
         return self.OPNUM
 
+    def clone(self):
+        return ESCAPE_OP(self.OPNUM, self.getarglist()[:], self.result, self.getdescr())
+
 class ExtendedTreeLoop(TreeLoop):
 
     def getboxes(self):


From cfbolz at codespeak.net  Sat Oct 30 16:59:02 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 16:59:02 +0200 (CEST)
Subject: [pypy-svn] r78533 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: . optimizeopt test
Message-ID: <20101030145902.69068282B90@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 16:58:59 2010
New Revision: 78533

Added:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeutil.py
      - copied, changed from r78527, pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefindnode.py
Removed:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizefindnode.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/specnode.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefindnode.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_specnode.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/viewnode.py
Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/history.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py
Log:
kill perfect specialization and specnodes


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py	Sat Oct 30 16:58:59 2010
@@ -11,7 +11,6 @@
 from pypy.jit.metainterp.history import AbstractFailDescr, BoxInt
 from pypy.jit.metainterp.history import BoxPtr, BoxObj, BoxFloat, Const
 from pypy.jit.metainterp import history
-from pypy.jit.metainterp.specnode import NotSpecNode, more_general_specnodes
 from pypy.jit.metainterp.typesystem import llhelper, oohelper
 from pypy.jit.metainterp.optimizeutil import InvalidLoop
 from pypy.jit.codewriter import heaptracker
@@ -42,7 +41,6 @@
 
 def make_loop_token(nb_args, jitdriver_sd):
     loop_token = LoopToken()
-    loop_token.specnodes = [prebuiltNotSpecNode] * nb_args
     loop_token.outermost_jitdriver_sd = jitdriver_sd
     return loop_token
 
@@ -82,8 +80,6 @@
         return old_loop_token
 
     if loop.preamble.operations:
-        loop.token.specnodes = [prebuiltNotSpecNode] * len(loop.inputargs) # FIXME
-        loop.preamble.token.specnodes = [prebuiltNotSpecNode] * len(loop.preamble.inputargs) # FIXME
         send_loop_to_backend(metainterp_sd, loop, "loop")
         send_loop_to_backend(metainterp_sd, loop.preamble, "entry bridge")
         insert_loop_token(old_loop_tokens, loop.preamble.token)
@@ -100,13 +96,8 @@
     # The following algo means "as late as possible, but before another
     # loop token that would be more general and so completely mask off
     # the new loop_token".
-    for i in range(len(old_loop_tokens)):
-        if more_general_specnodes(old_loop_tokens[i].specnodes,
-                                  loop_token.specnodes):
-            old_loop_tokens.insert(i, loop_token)
-            break
-    else:
-        old_loop_tokens.append(loop_token)
+    # XXX do we still need a list?
+    old_loop_tokens.append(loop_token)
 
 def send_loop_to_backend(metainterp_sd, loop, type):
     globaldata = metainterp_sd.globaldata
@@ -189,13 +180,10 @@
         raise metainterp_sd.ExitFrameWithExceptionRef(cpu, value)
 
 
-prebuiltNotSpecNode = NotSpecNode()
-
 class TerminatingLoopToken(LoopToken):
     terminating = True
 
     def __init__(self, nargs, finishdescr):
-        self.specnodes = [prebuiltNotSpecNode]*nargs
         self.finishdescr = finishdescr
 
 def make_done_loop_tokens():

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/history.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/history.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/history.py	Sat Oct 30 16:58:59 2010
@@ -728,7 +728,6 @@
     """
     terminating = False # see TerminatingLoopToken in compile.py
     outermost_jitdriver_sd = None
-    # specnodes = ...
     # and more data specified by the backend when the loop is compiled
     number = 0
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize.py	Sat Oct 30 16:58:59 2010
@@ -2,9 +2,7 @@
 
 # ____________________________________________________________
 
-from pypy.jit.metainterp.optimizefindnode import PerfectSpecializationFinder
 from pypy.jit.metainterp.optimizeopt import optimize_loop_1
-from pypy.jit.metainterp.specnode import equals_specnodes
 
 def optimize_loop(metainterp_sd, old_loop_tokens, loop):
     debug_start("jit-optimize")
@@ -16,19 +14,14 @@
 def _optimize_loop(metainterp_sd, old_loop_tokens, loop):
     cpu = metainterp_sd.cpu
     metainterp_sd.logger_noopt.log_loop(loop.inputargs, loop.operations)
-    finder = PerfectSpecializationFinder(cpu)
-    finder.find_nodes_loop(loop)
-    for old_loop_token in old_loop_tokens:
-        if equals_specnodes(old_loop_token.specnodes, loop.token.specnodes):
-            return old_loop_token
+    # XXX do we really still need a list?
+    if old_loop_tokens:
+        return old_loop_tokens[0]
     optimize_loop_1(metainterp_sd, loop)
     return None
 
 # ____________________________________________________________
 
-from pypy.jit.metainterp.optimizefindnode import BridgeSpecializationFinder
-from pypy.jit.metainterp.optimizeopt import optimize_bridge_1
-
 def optimize_bridge(metainterp_sd, old_loop_tokens, bridge):
     debug_start("jit-optimize")
     try:
@@ -37,15 +30,10 @@
         debug_stop("jit-optimize")
 
 def _optimize_bridge(metainterp_sd, old_loop_tokens, bridge):
-    cpu = metainterp_sd.cpu    
+    cpu = metainterp_sd.cpu
     metainterp_sd.logger_noopt.log_loop(bridge.inputargs, bridge.operations)
-    finder = BridgeSpecializationFinder(cpu)
-    finder.find_nodes_bridge(bridge)
-    for old_loop_token in old_loop_tokens:
-        if finder.bridge_matches(old_loop_token.specnodes):
-            bridge.operations[-1].setdescr(old_loop_token)   # patch jump target
-            optimize_bridge_1(metainterp_sd, bridge)
-            return old_loop_token
+    if old_loop_tokens:
+        return old_loop_tokens[0]
     return None
 
 # ____________________________________________________________

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	Sat Oct 30 16:58:59 2010
@@ -8,10 +8,7 @@
 from pypy.jit.metainterp.optimizeopt.unroll import OptUnroll
 
 def optimize_loop_1(metainterp_sd, loop, not_a_bridge=True):
-    """Optimize loop.operations to make it match the input of loop.specnodes
-    and to remove internal overheadish operations.  Note that loop.specnodes
-    must be applicable to the loop; you will probably get an AssertionError
-    if not.
+    """Optimize loop.operations to remove internal overheadish operations. 
     """
     optimizations = [OptIntBounds(),
                      OptRewrite(),
@@ -26,7 +23,5 @@
     optimizer.propagate_all_forward()
 
 def optimize_bridge_1(metainterp_sd, bridge):
-    """The same, but for a bridge.  The only difference is that we don't
-    expect 'specnodes' on the bridge.
-    """
+    """The same, but for a bridge. """
     optimize_loop_1(metainterp_sd, bridge, False)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/rewrite.py	Sat Oct 30 16:58:59 2010
@@ -207,9 +207,6 @@
         assert isinstance(expectedclassbox, Const)
         realclassbox = value.get_constant_class(self.optimizer.cpu)
         if realclassbox is not None:
-            # the following assert should always be true for now,
-            # because invalid loops that would fail it are detected
-            # earlier, in optimizefindnode.py.
             if realclassbox.same_constant(expectedclassbox):
                 return
             raise InvalidLoop

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Sat Oct 30 16:58:59 2010
@@ -1,9 +1,4 @@
 from pypy.jit.metainterp.history import Const, ConstInt
-from pypy.jit.metainterp.specnode import SpecNode, NotSpecNode, ConstantSpecNode
-from pypy.jit.metainterp.specnode import AbstractVirtualStructSpecNode
-from pypy.jit.metainterp.specnode import VirtualInstanceSpecNode
-from pypy.jit.metainterp.specnode import VirtualArraySpecNode
-from pypy.jit.metainterp.specnode import VirtualStructSpecNode
 from pypy.jit.metainterp.resoperation import rop, ResOperation
 from pypy.jit.metainterp.optimizeutil import _findall, sort_descrs
 from pypy.jit.metainterp.optimizeutil import descrlist_dict
@@ -228,79 +223,9 @@
             boxes.append(self.box)
 
 
-class __extend__(SpecNode):
-    def setup_virtual_node(self, optimizer, box, newinputargs):
-        raise NotImplementedError
-    def teardown_virtual_node(self, optimizer, value, newexitargs):
-        raise NotImplementedError
-
-class __extend__(NotSpecNode):
-    def setup_virtual_node(self, optimizer, box, newinputargs):
-        newinputargs.append(box)
-    def teardown_virtual_node(self, optimizer, value, newexitargs):
-        newexitargs.append(value.force_box())
-
-class __extend__(ConstantSpecNode):
-    def setup_virtual_node(self, optimizer, box, newinputargs):
-        optimizer.make_constant(box, self.constbox)
-    def teardown_virtual_node(self, optimizer, value, newexitargs):
-        pass
-
-class __extend__(AbstractVirtualStructSpecNode):
-    def setup_virtual_node(self, optimizer, box, newinputargs):
-        vvalue = self._setup_virtual_node_1(optimizer, box)
-        for ofs, subspecnode in self.fields:
-            subbox = optimizer.new_box(ofs)
-            subspecnode.setup_virtual_node(optimizer, subbox, newinputargs)
-            vvaluefield = optimizer.getvalue(subbox)
-            vvalue.setfield(ofs, vvaluefield)
-    def _setup_virtual_node_1(self, optimizer, box):
-        raise NotImplementedError
-    def teardown_virtual_node(self, optimizer, value, newexitargs):
-        assert value.is_virtual()
-        for ofs, subspecnode in self.fields:
-            subvalue = value.getfield(ofs, optimizer.new_const(ofs))
-            subspecnode.teardown_virtual_node(optimizer, subvalue, newexitargs)
-
-class __extend__(VirtualInstanceSpecNode):
-    def _setup_virtual_node_1(self, optimizer, box):
-        return optimizer.make_virtual(self.known_class, box)
-
-class __extend__(VirtualStructSpecNode):
-    def _setup_virtual_node_1(self, optimizer, box):
-        return optimizer.make_vstruct(self.typedescr, box)
-
-class __extend__(VirtualArraySpecNode):
-    def setup_virtual_node(self, optimizer, box, newinputargs):
-        vvalue = optimizer.make_varray(self.arraydescr, len(self.items), box)
-        for index in range(len(self.items)):
-            subbox = optimizer.new_box_item(self.arraydescr)
-            subspecnode = self.items[index]
-            subspecnode.setup_virtual_node(optimizer, subbox, newinputargs)
-            vvalueitem = optimizer.getvalue(subbox)
-            vvalue.setitem(index, vvalueitem)
-    def teardown_virtual_node(self, optimizer, value, newexitargs):
-        assert value.is_virtual()
-        for index in range(len(self.items)):
-            subvalue = value.getitem(index)
-            subspecnode = self.items[index]
-            subspecnode.teardown_virtual_node(optimizer, subvalue, newexitargs)
-
 class OptVirtualize(optimizer.Optimization):
     "Virtualize objects until they escape."
 
-    def setup(self, not_a_bridge):
-        if not not_a_bridge:
-            return
-
-        inputargs = self.optimizer.loop.inputargs
-        specnodes = self.optimizer.loop.token.specnodes
-        assert len(inputargs) == len(specnodes)
-        newinputargs = []
-        for i in range(len(inputargs)):
-            specnodes[i].setup_virtual_node(self, inputargs[i], newinputargs)
-        self.optimizer.loop.inputargs = newinputargs
-
     def make_virtual(self, known_class, box, source_op=None):
         vvalue = VirtualValue(self.optimizer, known_class, box, source_op)
         self.make_equal_to(box, vvalue)
@@ -316,21 +241,6 @@
         self.make_equal_to(box, vvalue)
         return vvalue
 
-    def optimize_JUMP(self, op):
-        self.emit_operation(op) # FIXME
-        return
-        orgop = self.optimizer.loop.operations[-1]
-        exitargs = []
-        target_loop_token = orgop.getdescr()
-        assert isinstance(target_loop_token, LoopToken)
-        specnodes = target_loop_token.specnodes
-        assert op.numargs() == len(specnodes)
-        for i in range(len(specnodes)):
-            value = self.getvalue(op.getarg(i))
-            specnodes[i].teardown_virtual_node(self, value, exitargs)
-        op = op.copy_and_change(op.getopnum(), args=exitargs[:])
-        self.emit_operation(op)
-
     def optimize_VIRTUAL_REF(self, op):
         indexbox = op.getarg(1)
         #
@@ -377,7 +287,6 @@
     def optimize_GETFIELD_GC(self, op):
         value = self.getvalue(op.getarg(0))
         if value.is_virtual():
-            # optimizefindnode should ensure that fieldvalue is found
             assert isinstance(value, AbstractVirtualValue)
             fieldvalue = value.getfield(op.getdescr(), None)
             if fieldvalue is None:

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/pyjitpl.py	Sat Oct 30 16:58:59 2010
@@ -1746,8 +1746,7 @@
     def designate_target_loop(self, gmp):
         loop_token = gmp.target_loop_token
         num_green_args = self.jitdriver_sd.num_green_args
-        residual_args = self.get_residual_args(loop_token.specnodes,
-                                               gmp.argboxes[num_green_args:])
+        residual_args = gmp.argboxes[num_green_args:]
         history.set_future_values(self.cpu, residual_args)
         return loop_token
 
@@ -1854,16 +1853,6 @@
         if target_loop_token is not loop_tokens[0]:
             compile.giveup()
 
-    def get_residual_args(self, specnodes, args):
-        if specnodes is None:     # it is None only for tests
-            return args
-        assert len(specnodes) == len(args)
-        expanded_args = []
-        for i in range(len(specnodes)):
-            specnode = specnodes[i]
-            specnode.extract_runtime_data(self.cpu, args[i], expanded_args)
-        return expanded_args
-
     @specialize.arg(1)
     def initialize_original_boxes(self, jitdriver_sd, *args):
         original_boxes = []

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py	Sat Oct 30 16:58:59 2010
@@ -1,32 +1,29 @@
 from pypy.jit.metainterp.history import LoopToken, ConstInt, History, Stats
 from pypy.jit.metainterp.history import BoxInt, INT
-from pypy.jit.metainterp.specnode import NotSpecNode, ConstantSpecNode
 from pypy.jit.metainterp.compile import insert_loop_token, compile_new_loop
 from pypy.jit.metainterp.compile import ResumeGuardDescr
 from pypy.jit.metainterp.compile import ResumeGuardCountersInt
 from pypy.jit.metainterp.compile import compile_tmp_callback
 from pypy.jit.metainterp import optimize_nopspec, jitprof, typesystem, compile
-from pypy.jit.metainterp.test.test_optimizefindnode import LLtypeMixin
+from pypy.jit.metainterp.test.test_optimizeutil import LLtypeMixin
 from pypy.jit.tool.oparser import parse
 
 
 def test_insert_loop_token():
+    # XXX this test is a bit useless now that there are no specnodes
     lst = []
     #
     tok1 = LoopToken()
-    tok1.specnodes = [NotSpecNode()]
     insert_loop_token(lst, tok1)
     assert lst == [tok1]
     #
     tok2 = LoopToken()
-    tok2.specnodes = [ConstantSpecNode(ConstInt(8))]
     insert_loop_token(lst, tok2)
-    assert lst == [tok2, tok1]
+    assert lst == [tok1, tok2]
     #
     tok3 = LoopToken()
-    tok3.specnodes = [ConstantSpecNode(ConstInt(-13))]
     insert_loop_token(lst, tok3)
-    assert lst == [tok2, tok3, tok1]
+    assert lst == [tok1, tok2, tok3]
 
 
 class FakeCPU:

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	Sat Oct 30 16:58:59 2010
@@ -1,9 +1,8 @@
 import py
 from pypy.rlib.objectmodel import instantiate
-from pypy.jit.metainterp.test.test_optimizefindnode import (LLtypeMixin,
-                                                            #OOtypeMixin,
-                                                            BaseTest)
-from pypy.jit.metainterp.optimizefindnode import PerfectSpecializationFinder
+from pypy.jit.metainterp.test.test_optimizeutil import (LLtypeMixin,
+                                                        #OOtypeMixin,
+                                                        BaseTest)
 import pypy.jit.metainterp.optimizeopt.optimizer as optimizeopt
 import pypy.jit.metainterp.optimizeopt.virtualize as virtualize
 from pypy.jit.metainterp.optimizeutil import InvalidLoop
@@ -244,9 +243,8 @@
         assert equaloplists(optimized.operations,
                             expected.operations, False, remap)
 
-    def optimize_loop(self, ops, spectext, optops, checkspecnodes=True):
+    def optimize_loop(self, ops, spectext, optops):
         loop = self.parse(ops)
-        loop.token.specnodes = self.unpack_specnodes(spectext)
         #
         self.loop = loop
         metainterp_sd = FakeMetaInterpStaticData(self.cpu)
@@ -858,7 +856,7 @@
         """
         py.test.skip("XXX")
         self.optimize_loop(ops, 'Not, Virtual(node_vtable, floatdescr=Not)',
-                           expected, checkspecnodes=False)
+                           expected)
 
     def test_virtual_2(self):
         ops = """
@@ -913,7 +911,7 @@
         self.optimize_loop(ops, '''Virtual(node_vtable),
                                    Virtual(node_vtable),
                                    Not''',
-                           expected, checkspecnodes=False)
+                           expected)
         #
         # to be complete, we also check the no-opt case where most comparisons
         # are not removed.  The exact set of comparisons removed depends on

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 16:58:59 2010
@@ -1,9 +1,8 @@
 import py
 from pypy.rlib.objectmodel import instantiate
-from pypy.jit.metainterp.test.test_optimizefindnode import (LLtypeMixin,
-                                                            #OOtypeMixin,
-                                                            BaseTest)
-from pypy.jit.metainterp.optimizefindnode import PerfectSpecializationFinder
+from pypy.jit.metainterp.test.test_optimizeutil import (LLtypeMixin,
+                                                        #OOtypeMixin,
+                                                        BaseTest)
 import pypy.jit.metainterp.optimizeopt.optimizer as optimizeopt
 import pypy.jit.metainterp.optimizeopt.virtualize as virtualize
 from pypy.jit.metainterp.optimizeopt import optimize_loop_1
@@ -67,7 +66,6 @@
 
     def optimize_loop(self, ops, spectext, optops, expected_preamble=None):
         loop = self.parse(ops)
-        loop.token.specnodes = self.unpack_specnodes(spectext)
         #
         self.loop = loop
         loop.preamble = TreeLoop('preamble')

Copied: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeutil.py (from r78527, pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefindnode.py)
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefindnode.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeutil.py	Sat Oct 30 16:58:59 2010
@@ -8,17 +8,10 @@
 from pypy.jit.metainterp.history import (BoxInt, BoxPtr, ConstInt, ConstPtr,
                                          Const, TreeLoop, BoxObj,
                                          ConstObj, AbstractDescr)
-from pypy.jit.metainterp.optimizefindnode import PerfectSpecializationFinder
-from pypy.jit.metainterp.optimizefindnode import BridgeSpecializationFinder
 from pypy.jit.metainterp.optimizeutil import sort_descrs, InvalidLoop
-from pypy.jit.metainterp.specnode import NotSpecNode, prebuiltNotSpecNode
-from pypy.jit.metainterp.specnode import VirtualInstanceSpecNode
-from pypy.jit.metainterp.specnode import VirtualArraySpecNode
-from pypy.jit.metainterp.specnode import VirtualStructSpecNode
-from pypy.jit.metainterp.specnode import ConstantSpecNode
 from pypy.jit.codewriter.effectinfo import EffectInfo
 from pypy.jit.codewriter.heaptracker import register_known_gctype, adr2int
-from pypy.jit.metainterp.test.oparser import parse
+from pypy.jit.tool.oparser import parse
 
 def test_sort_descrs():
     class PseudoDescr(AbstractDescr):
@@ -245,956 +238,5 @@
                      boxkinds=boxkinds,
                      invent_fail_descr=self.invent_fail_descr)
 
-    def unpack_specnodes(self, text):
-        #
-        def constclass(cls_vtable):
-            if self.type_system == 'lltype':
-                return ConstInt(adr2int(llmemory.cast_ptr_to_adr(cls_vtable)))
-            else:
-                return ConstObj(ootype.cast_to_object(cls_vtable))
-        def constant(value):
-            if isinstance(lltype.typeOf(value), lltype.Ptr):
-                return ConstPtr(value)
-            elif isinstance(ootype.typeOf(value), ootype.OOType):
-                return ConstObj(ootype.cast_to_object(value))
-            else:
-                return ConstInt(value)
-
-        def parsefields(kwds_fields):
-            fields = []
-            for key, value in kwds_fields.items():
-                fields.append((self.namespace[key], value))
-            fields.sort(key = lambda (x, _): x.sort_key())
-            return fields
-        def makeConstant(value):
-            return ConstantSpecNode(constant(value))
-        def makeVirtual(cls_vtable, **kwds_fields):
-            fields = parsefields(kwds_fields)
-            return VirtualInstanceSpecNode(constclass(cls_vtable), fields)
-        def makeVirtualArray(arraydescr, *items):
-            return VirtualArraySpecNode(arraydescr, items)
-        def makeVirtualStruct(typedescr, **kwds_fields):
-            fields = parsefields(kwds_fields)
-            return VirtualStructSpecNode(typedescr, fields)
-        #
-        context = {'Not': prebuiltNotSpecNode,
-                   'Constant': makeConstant,
-                   'Virtual': makeVirtual,
-                   'VArray': makeVirtualArray,
-                   'VStruct': makeVirtualStruct}
-        lst = eval('[' + text + ']', self.namespace, context)
-        return lst
-
-    def check_specnodes(self, specnodes, text):
-        lst = self.unpack_specnodes(text)
-        assert len(specnodes) == len(lst)
-        for x, y in zip(specnodes, lst):
-            assert x.equals(y, ge=False)
-        return True
-
 # ____________________________________________________________
 
-class BaseTestOptimizeFindNode(BaseTest):
-
-    def find_nodes(self, ops, spectext, boxkinds=None):
-        assert boxkinds is None or isinstance(boxkinds, dict)
-        loop = self.parse(ops, boxkinds=boxkinds)
-        perfect_specialization_finder = PerfectSpecializationFinder(self.cpu)
-        perfect_specialization_finder.find_nodes_loop(loop)
-        self.check_specnodes(loop.token.specnodes, spectext)
-        return (loop.getboxes(), perfect_specialization_finder.getnode)
-
-    def test_find_nodes_simple(self):
-        ops = """
-        [i]
-        i0 = int_sub(i, 1)
-        guard_value(i0, 0) [i0]
-        jump(i0)
-        """
-        boxes, getnode = self.find_nodes(ops, 'Not')
-        assert getnode(boxes.i).fromstart
-        assert not getnode(boxes.i0).fromstart
-
-    def test_find_nodes_non_escape(self):
-        ops = """
-        [p0]
-        p1 = getfield_gc(p0, descr=nextdescr)
-        i0 = getfield_gc(p1, descr=valuedescr)
-        i1 = int_sub(i0, 1)
-        p2 = getfield_gc(p0, descr=nextdescr)
-        setfield_gc(p2, i1, descr=valuedescr)
-        p3 = new_with_vtable(ConstClass(node_vtable))
-        jump(p3)
-        """
-        boxes, getnode = self.find_nodes(ops,
-                                         'Virtual(node_vtable, nextdescr=Not)')
-        assert not getnode(boxes.p0).escaped
-        assert getnode(boxes.p1).escaped
-        assert getnode(boxes.p2).escaped
-        assert getnode(boxes.p0).fromstart
-        assert getnode(boxes.p1).fromstart
-        assert getnode(boxes.p2).fromstart
-
-    def test_find_nodes_escape(self):
-        ops = """
-        [p0]
-        p1 = getfield_gc(p0, descr=nextdescr)
-        p2 = getfield_gc(p1, descr=nextdescr)
-        i0 = getfield_gc(p2, descr=valuedescr)
-        i1 = int_sub(i0, 1)
-        escape(p1)
-        p3 = getfield_gc(p0, descr=nextdescr)
-        setfield_gc(p3, i1, descr=valuedescr)
-        p4 = getfield_gc(p1, descr=nextdescr)
-        setfield_gc(p4, i1, descr=valuedescr)
-        p5 = new_with_vtable(ConstClass(node_vtable))
-        jump(p5)
-        """
-        boxes, getnode = self.find_nodes(ops,
-                                         'Virtual(node_vtable, nextdescr=Not)')
-        assert not getnode(boxes.p0).escaped
-        assert getnode(boxes.p1).escaped
-        assert getnode(boxes.p2).escaped    # forced by p1
-        assert getnode(boxes.p3).escaped    # forced because p3 == p1
-        assert getnode(boxes.p4).escaped    # forced by p1
-        assert getnode(boxes.p0).fromstart
-        assert getnode(boxes.p1).fromstart
-        assert getnode(boxes.p2).fromstart
-        assert getnode(boxes.p3).fromstart
-        assert not getnode(boxes.p4).fromstart
-
-    def test_find_nodes_new_1(self):
-        ops = """
-        [p1]
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        jump(p2)
-        """
-        boxes, getnode = self.find_nodes(ops, 'Virtual(node_vtable)')
-
-        boxp1 = getnode(boxes.p1)
-        boxp2 = getnode(boxes.p2)
-        assert not boxp1.escaped
-        assert not boxp2.escaped
-
-        assert not boxp1.origfields
-        assert not boxp1.curfields
-        assert not boxp2.origfields
-        assert not boxp2.curfields
-
-        assert boxp1.fromstart
-        assert not boxp2.fromstart
-
-        assert boxp1.knownclsbox is None
-        assert boxp2.knownclsbox.getaddr() == self.node_vtable_adr
-
-    def test_find_nodes_new_2(self):
-        ops = """
-        [i1, p1]
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        p3 = new_with_vtable(ConstClass(node_vtable2))
-        setfield_gc(p2, p3, descr=nextdescr)
-        setfield_gc(p3, i1, descr=valuedescr)
-        jump(i1, p2)
-        """
-        self.find_nodes(ops,
-            '''Not,
-               Virtual(node_vtable,
-                       nextdescr=Virtual(node_vtable2,
-                                         valuedescr=Not))''')
-
-    def test_find_nodes_new_3(self):
-        ops = """
-        [sum, p1]
-        guard_class(p1, ConstClass(node_vtable)) []
-        i1 = getfield_gc(p1, descr=valuedescr)
-        i2 = int_sub(i1, 1)
-        sum2 = int_add(sum, i1)
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p2, i2, descr=valuedescr)
-        p3 = new_with_vtable(ConstClass(node_vtable2))
-        setfield_gc(p2, p3, descr=nextdescr)
-        jump(sum2, p2)
-        """
-        boxes, getnode = self.find_nodes(
-            ops,
-            '''Not,
-               Virtual(node_vtable,
-                       valuedescr=Not,
-                       nextdescr=Virtual(node_vtable2))''',
-            boxkinds={'sum': BoxInt, 'sum2': BoxInt})
-        assert getnode(boxes.sum) is not getnode(boxes.sum2)
-        assert getnode(boxes.p1) is not getnode(boxes.p2)
-
-        boxp1 = getnode(boxes.p1)
-        boxp2 = getnode(boxes.p2)
-        boxp3 = getnode(boxes.p3)
-        assert not boxp1.escaped
-        assert not boxp2.escaped
-        assert not boxp3.escaped
-
-        assert not boxp1.curfields
-        assert boxp1.origfields[self.valuedescr] is getnode(boxes.i1)
-        assert not boxp2.origfields
-        assert boxp2.curfields[self.nextdescr] is boxp3
-
-        assert boxp1.fromstart
-        assert not boxp2.fromstart
-        assert not boxp3.fromstart
-
-        assert boxp2.knownclsbox.getaddr() == self.node_vtable_adr
-        assert boxp3.knownclsbox.getaddr() == self.node_vtable_adr2
-
-    def test_find_nodes_new_aliasing_0(self):
-        ops = """
-        [p1, p2]
-        p3 = new_with_vtable(ConstClass(node_vtable))
-        jump(p3, p3)
-        """
-        # both p1 and p2 must be NotSpecNodes; it's not possible to pass
-        # the same Virtual both in p1 and p2 (at least so far).
-        self.find_nodes(ops, 'Not, Not')
-
-    def test_find_nodes_new_aliasing_1(self):
-        ops = """
-        [sum, p1]
-        guard_class(p1, ConstClass(node_vtable)) []
-        p3 = getfield_gc(p1, descr=nextdescr)
-        guard_class(p3, ConstClass(node_vtable)) []
-        i1 = getfield_gc(p1, descr=valuedescr)
-        i2 = int_sub(i1, 1)
-        sum2 = int_add(sum, i1)
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p2, i2, descr=valuedescr)
-        setfield_gc(p2, p2, descr=nextdescr)
-        jump(sum2, p2)
-        """
-        # the issue is the cycle "p2->p2", which cannot be represented
-        # with SpecNodes so far
-        self.find_nodes(ops, 'Not, Not',
-                        boxkinds={'sum': BoxInt, 'sum2': BoxInt})
-
-    def test_find_nodes_new_aliasing_2(self):
-        ops = """
-        [p1, p2]
-        escape(p2)
-        p3 = new_with_vtable(ConstClass(node_vtable))
-        jump(p3, p3)
-        """
-        # both p1 and p2 must be NotSpecNodes; it's not possible to pass
-        # in p1 a Virtual and not in p2, as they both come from the same p3.
-        self.find_nodes(ops, 'Not, Not')
-
-    def test_find_nodes_new_mismatch(self):
-        ops = """
-        [p1]
-        guard_class(p1, ConstClass(node_vtable)) []
-        p2 = new_with_vtable(ConstClass(node_vtable2))
-        jump(p2)
-        """
-        # this is not a valid loop at all, because of the mismatch
-        # between the produced and the consumed class.
-        py.test.raises(InvalidLoop, self.find_nodes, ops, None)
-
-    def test_find_nodes_new_aliasing_mismatch(self):
-        ops = """
-        [p0, p1]
-        guard_class(p0, ConstClass(node_vtable)) []
-        guard_class(p1, ConstClass(node_vtable2)) []
-        p2 = new_with_vtable(ConstClass(node_vtable2))
-        jump(p2, p2)
-        """
-        # this is also not really a valid loop, but it's not detected
-        # because p2 is passed more than once in the jump().
-        self.find_nodes(ops, 'Not, Not')
-
-    def test_find_nodes_new_escapes(self):
-        ops = """
-        [p0]
-        escape(p0)
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        jump(p1)
-        """
-        self.find_nodes(ops, 'Not')
-
-    def test_find_nodes_new_unused(self):
-        ops = """
-        [p0]
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        p3 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, p2, descr=nextdescr)
-        setfield_gc(p2, p3, descr=nextdescr)
-        jump(p1)
-        """
-        self.find_nodes(ops, '''
-            Virtual(node_vtable,
-                    nextdescr=Virtual(node_vtable,
-                                      nextdescr=Virtual(node_vtable)))''')
-
-    def test_find_nodes_ptr_eq(self):
-        ops = """
-        [p3, p4, p2]
-        p0 = new_with_vtable(ConstClass(node_vtable))
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        guard_nonnull(p0) []
-        i3 = ptr_ne(p0, NULL)
-        guard_true(i3) []
-        i4 = ptr_eq(p0, NULL)
-        guard_false(i4) []
-        i5 = ptr_ne(NULL, p0)
-        guard_true(i5) []
-        i6 = ptr_eq(NULL, p0)
-        guard_false(i6) []
-        i7 = ptr_ne(p0, p1)
-        guard_true(i7) []
-        i8 = ptr_eq(p0, p1)
-        guard_false(i8) []
-        i9 = ptr_ne(p0, p2)
-        guard_true(i9) []
-        i10 = ptr_eq(p0, p2)
-        guard_false(i10) []
-        i11 = ptr_ne(p2, p1)
-        guard_true(i11) []
-        i12 = ptr_eq(p2, p1)
-        guard_false(i12) []
-        jump(p0, p1, p2)
-        """
-        self.find_nodes(ops, '''Virtual(node_vtable),
-                                Virtual(node_vtable),
-                                Not''')
-
-    def test_find_nodes_call(self):
-        ops = """
-        [i0, p2]
-        p0 = new_with_vtable(ConstClass(node_vtable))
-        i1 = call_pure(i0, p0)     # forces p0 to not be virtual
-        jump(i1, p0)
-        """
-        self.find_nodes(ops, 'Not, Not')
-
-    def test_find_nodes_default_field(self):
-        ops = """
-        [p0]
-        i0 = getfield_gc(p0, descr=valuedescr)
-        guard_value(i0, 5) []
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        # the field 'value' has its default value of 0
-        jump(p1)
-        """
-        # The answer must contain the 'value' field, because otherwise
-        # we might get incorrect results: when tracing, i0 was 5.
-        self.find_nodes(ops, 'Virtual(node_vtable, valuedescr=Not)')
-
-    def test_find_nodes_nonvirtual_guard_class(self):
-        ops = """
-        [p1]
-        guard_class(p1, ConstClass(node_vtable)) [p1]
-        jump(p1)
-        """
-        self.find_nodes(ops, 'Not')
-
-    def test_find_nodes_p12_simple(self):
-        ops = """
-        [p1]
-        i3 = getfield_gc(p1, descr=valuedescr)
-        escape(i3)
-        jump(p1)
-        """
-        self.find_nodes(ops, 'Not')
-
-    def test_find_nodes_p123_simple(self):
-        ops = """
-        [i1, p2, p3]
-        i3 = getfield_gc(p3, descr=valuedescr)
-        escape(i3)
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, i1, descr=valuedescr)
-        jump(i1, p1, p2)
-        """
-        # We cannot track virtuals that survive for more than two iterations.
-        self.find_nodes(ops, 'Not, Not, Not')
-
-    def test_find_nodes_p1234_simple(self):
-        ops = """
-        [i1, p2, p3, p4]
-        i4 = getfield_gc(p4, descr=valuedescr)
-        escape(i4)
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, i1, descr=valuedescr)
-        jump(i1, p1, p2, p3)
-        """
-        # We cannot track virtuals that survive for more than two iterations.
-        self.find_nodes(ops, 'Not, Not, Not, Not')
-
-    def test_find_nodes_p123_guard_class(self):
-        ops = """
-        [i1, p2, p3]
-        guard_class(p3, ConstClass(node_vtable)) [i1, p2, p3]
-        i3 = getfield_gc(p3, descr=valuedescr)
-        escape(i3)
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, i1, descr=valuedescr)
-        jump(i1, p1, p2)
-        """
-        # We cannot track virtuals that survive for more than two iterations.
-        self.find_nodes(ops, 'Not, Not, Not')
-
-    def test_find_nodes_p123_rec(self):
-        ops = """
-        [i1, p2, p0d]
-        p3 = getfield_gc(p0d, descr=nextdescr)
-        i3 = getfield_gc(p3, descr=valuedescr)
-        escape(i3)
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, i1, descr=valuedescr)
-        p0c = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p0c, p2, descr=nextdescr)
-        jump(i1, p1, p0c)
-        """
-        # We cannot track virtuals that survive for more than two iterations.
-        self.find_nodes(ops, '''Not,
-                                Not,
-                                Virtual(node_vtable, nextdescr=Not)''')
-
-    def test_find_nodes_setfield_bug(self):
-        ops = """
-        [p1, p2]
-        escape(p1)
-        setfield_gc(p1, p2, descr=nextdescr)
-        p3 = new_with_vtable(ConstClass(node_vtable))
-        jump(p1, p3)
-        """
-        self.find_nodes(ops, 'Not, Not')
-
-    def test_find_nodes_array_virtual_1(self):
-        ops = """
-        [i1, p2]
-        i2 = getarrayitem_gc(p2, 1, descr=arraydescr)
-        escape(i2)
-        p3 = new_array(3, descr=arraydescr)
-        setarrayitem_gc(p3, 1, i1, descr=arraydescr)
-        jump(i1, p3)
-        """
-        self.find_nodes(ops, 'Not, VArray(arraydescr, Not, Not, Not)')
-
-    def test_find_nodes_array_virtual_2(self):
-        ops = """
-        [i1, p2]
-        i2 = arraylen_gc(p2, descr=arraydescr)
-        escape(i2)
-        p3 = new_array(3, descr=arraydescr)
-        setarrayitem_gc(p3, 1, i1, descr=arraydescr)
-        jump(i1, p3)
-        """
-        self.find_nodes(ops, 'Not, VArray(arraydescr, Not, Not, Not)')
-
-    def test_find_nodes_array_virtual_3(self):
-        ops = """
-        [pvalue1, p2]
-        pvalue2 = new_with_vtable(ConstClass(node_vtable2))
-        ps2 = getarrayitem_gc(p2, 1, descr=arraydescr)
-        setfield_gc(ps2, pvalue2, descr=nextdescr)
-        ps3 = getarrayitem_gc(p2, 1, descr=arraydescr)
-        pvalue3 = getfield_gc(ps3, descr=nextdescr)
-        ps1 = new_with_vtable(ConstClass(node_vtable))
-        p3 = new_array(3, descr=arraydescr)
-        setarrayitem_gc(p3, 1, ps1, descr=arraydescr)
-        jump(pvalue3, p3)
-        """
-        self.find_nodes(ops, 'Virtual(node_vtable2), VArray(arraydescr, Not, Virtual(node_vtable), Not)')
-
-    def test_find_nodes_array_virtual_empty(self):
-        ops = """
-        [i1, p2]
-        p3 = new_array(3, descr=arraydescr)
-        jump(i1, p3)
-        """
-        self.find_nodes(ops, 'Not, VArray(arraydescr, Not, Not, Not)')
-
-    def test_find_nodes_array_nonvirtual_1(self):
-        ops = """
-        [i1, p2]
-        i2 = getarrayitem_gc(p2, i1, descr=arraydescr)
-        escape(i2)
-        p3 = new_array(4, descr=arraydescr)
-        setarrayitem_gc(p3, i1, i2, descr=arraydescr)
-        jump(i1, p3)
-        """
-        # Does not work because of the variable index, 'i1'.
-        self.find_nodes(ops, 'Not, Not')
-
-    def test_find_nodes_array_forced_1(self):
-        ops = """
-        [p1, i1]
-        p2 = new_array(1, descr=arraydescr)
-        setarrayitem_gc(p2, 0, p1, descr=arraydescr)
-        p3 = getarrayitem_gc(p2, i1, descr=arraydescr)
-        p4 = new_with_vtable(ConstClass(node_vtable))
-        jump(p4, i1)
-        """
-        # escapes because getarrayitem_gc uses a non-constant index
-        self.find_nodes(ops, 'Not, Not')
-
-    def test_find_nodes_arrayitem_forced(self):
-        ops = """
-        [p1]
-        p2 = new_array(1, descr=arraydescr)
-        escape(p2)
-        p4 = new_with_vtable(ConstClass(node_vtable))
-        setarrayitem_gc(p2, 0, p4, descr=arraydescr)
-        jump(p4)
-        """
-        self.find_nodes(ops, 'Not')
-
-    def test_find_nodes_struct_virtual_1(self):
-        ops = """
-        [i1, p2]
-        i2 = getfield_gc(p2, descr=adescr)
-        escape(i2)
-        p3 = new(descr=ssize)
-        setfield_gc(p3, i1, descr=adescr)
-        jump(i1, p3)
-        """
-        self.find_nodes(ops, 'Not, VStruct(ssize, adescr=Not)')
-
-    def test_find_nodes_struct_nonvirtual_1(self):
-        ops = """
-        [i1, p2]
-        i2 = getfield_gc(p2, descr=adescr)
-        escape(p2)
-        p3 = new(descr=ssize)
-        setfield_gc(p3, i1, descr=adescr)
-        jump(i1, p3)
-        """
-        self.find_nodes(ops, 'Not, Not')
-
-    def test_find_nodes_guard_value_constant(self):
-        ops = """
-        [p1]
-        guard_value(p1, ConstPtr(myptr)) []
-        jump(ConstPtr(myptr))
-        """
-        self.find_nodes(ops, 'Constant(myptr)')
-
-    def test_find_nodes_guard_value_constant_mismatch(self):
-        ops = """
-        [p1]
-        guard_value(p1, ConstPtr(myptr2)) []
-        jump(ConstPtr(myptr))
-        """
-        py.test.raises(InvalidLoop, self.find_nodes, ops, None)
-
-    def test_find_nodes_guard_value_escaping_constant(self):
-        ops = """
-        [p1]
-        escape(p1)
-        guard_value(p1, ConstPtr(myptr)) []
-        jump(ConstPtr(myptr))
-        """
-        self.find_nodes(ops, 'Constant(myptr)')
-
-    def test_find_nodes_guard_value_same_as_constant(self):
-        ops = """
-        [p1]
-        guard_value(p1, ConstPtr(myptr)) []
-        p2 = same_as(ConstPtr(myptr))
-        jump(p2)
-        """
-        self.find_nodes(ops, 'Constant(myptr)')
-
-    def test_find_nodes_store_into_loop_constant_1(self):
-        ops = """
-        [i0, p1, p4]
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, p2, descr=nextdescr)
-        jump(i0, p1, p2)
-        """
-        self.find_nodes(ops, 'Not, Not, Not')
-
-    def test_find_nodes_store_into_loop_constant_2(self):
-        ops = """
-        [i0, p4, p1]
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, p2, descr=nextdescr)
-        jump(i0, p2, p1)
-        """
-        self.find_nodes(ops, 'Not, Not, Not')
-
-    def test_find_nodes_store_into_loop_constant_3(self):
-        ops = """
-        [i0, p1]
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, p2, descr=nextdescr)
-        call(i0)
-        jump(i0, p1)
-        """
-        self.find_nodes(ops, 'Not, Not')
-
-    def test_find_nodes_arithmetic_propagation_bug_0(self):
-        ops = """
-        [p1]
-        i1 = getarrayitem_gc(p1, 0, descr=arraydescr)
-        escape(i1)
-        i2 = int_add(0, 1)
-        p2 = new_array(i2, descr=arraydescr)
-        i3 = escape()
-        setarrayitem_gc(p2, 0, i3, descr=arraydescr)
-        jump(p2)
-        """
-        self.find_nodes(ops, 'VArray(arraydescr, Not)')
-
-    def test_find_nodes_arithmetic_propagation_bug_1(self):
-        ops = """
-        [p1]
-        i1 = getarrayitem_gc(p1, 0, descr=arraydescr)
-        escape(i1)
-        i2 = same_as(1)
-        p2 = new_array(i2, descr=arraydescr)
-        setarrayitem_gc(p2, 0, 5)
-        jump(p2)
-        """
-        self.find_nodes(ops, 'VArray(arraydescr, Not)')
-
-    def test_find_nodes_arithmetic_propagation_bug_2(self):
-        ops = """
-        [p1]
-        i0 = int_sub(17, 17)
-        i1 = getarrayitem_gc(p1, i0, descr=arraydescr)
-        escape(i1)
-        i2 = int_add(0, 1)
-        p2 = new_array(i2, descr=arraydescr)
-        i3 = escape()
-        setarrayitem_gc(p2, i0, i3, descr=arraydescr)
-        jump(p2)
-        """
-        self.find_nodes(ops, 'VArray(arraydescr, Not)')
-
-    def test_find_nodes_arithmetic_propagation_bug_3(self):
-        ops = """
-        [p1]
-        i1 = getarrayitem_gc(p1, 0, descr=arraydescr)
-        escape(i1)
-        p3 = new_array(1, descr=arraydescr)
-        i2 = arraylen_gc(p3, descr=arraydescr)
-        p2 = new_array(i2, descr=arraydescr)
-        i3 = escape()
-        setarrayitem_gc(p2, 0, i3, descr=arraydescr)
-        jump(p2)
-        """
-        self.find_nodes(ops, 'VArray(arraydescr, Not)')
-
-    def test_find_nodes_bug_1(self):
-        ops = """
-        [p12]
-        guard_nonnull(p12) []
-        guard_class(p12, ConstClass(node_vtable)) []
-        guard_class(p12, ConstClass(node_vtable)) []
-        i22 = getfield_gc_pure(p12, descr=valuedescr)
-        escape(i22)
-        guard_nonnull(p12) []
-        guard_class(p12, ConstClass(node_vtable)) []
-        guard_class(p12, ConstClass(node_vtable)) []
-        i29 = getfield_gc_pure(p12, descr=valuedescr)
-        i31 = int_add_ovf(i29, 1)
-        guard_no_overflow() []
-        p33 = new_with_vtable(ConstClass(node_vtable))      # NODE
-        setfield_gc(p33, i31, descr=valuedescr)
-        #
-        p35 = new_array(1, descr=arraydescr3)               # Array(NODE)
-        setarrayitem_gc(p35, 0, p33, descr=arraydescr3)
-        p38 = new_with_vtable(ConstClass(u_vtable))         # U
-        setfield_gc(p38, p35, descr=onedescr)
-        guard_nonnull(p38) []
-        guard_nonnull(p38) []
-        guard_class(p38, ConstClass(u_vtable)) []
-        p42 = getfield_gc(p38, descr=onedescr)              # Array(NODE)
-        i43 = arraylen_gc(p42, descr=arraydescr3)
-        i45 = int_sub(i43, 0)
-        p46 = new(descr=tsize)                              # T
-        setfield_gc(p46, i45, descr=cdescr)
-        p47 = new_array(i45, descr=arraydescr3)             # Array(NODE)
-        setfield_gc(p46, p47, descr=ddescr)
-        i48 = int_lt(0, i43)
-        guard_true(i48) []
-        p49 = getarrayitem_gc(p42, 0, descr=arraydescr3)    # NODE
-        p50 = getfield_gc(p46, descr=ddescr)                # Array(NODE)
-        setarrayitem_gc(p50, 0, p49, descr=arraydescr3)
-        i52 = int_lt(1, i43)
-        guard_false(i52) []
-        i53 = getfield_gc(p46, descr=cdescr)
-        i55 = int_ne(i53, 1)
-        guard_false(i55) []
-        p56 = getfield_gc(p46, descr=ddescr)                # Array(NODE)
-        p58 = getarrayitem_gc(p56, 0, descr=arraydescr3)    # NODE
-        guard_nonnull(p38) []
-        jump(p58)
-        """
-        self.find_nodes(ops, 'Virtual(node_vtable, valuedescr=Not)')
-
-    # ------------------------------
-    # Bridge tests
-
-    def find_bridge(self, ops, inputspectext, outputspectext, boxkinds=None,
-                    mismatch=False):
-        assert boxkinds is None or isinstance(boxkinds, dict)
-        inputspecnodes = self.unpack_specnodes(inputspectext)
-        outputspecnodes = self.unpack_specnodes(outputspectext)
-        bridge = self.parse(ops, boxkinds=boxkinds)
-        bridge_specialization_finder = BridgeSpecializationFinder(self.cpu)
-        bridge_specialization_finder.find_nodes_bridge(bridge, inputspecnodes)
-        matches = bridge_specialization_finder.bridge_matches(outputspecnodes)
-        if mismatch:
-            assert not matches
-        else:
-            assert matches
-
-    def test_bridge_simple(self):
-        ops = """
-        [i0]
-        i1 = int_add(i0, 1)
-        jump(i1)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-        self.find_bridge(ops, 'Not', 'Virtual(node_vtable)', mismatch=True)
-
-    def test_bridge_simple_known_class(self):
-        ops = """
-        [p0]
-        setfield_gc(p0, 123, descr=valuedescr)
-        jump(p0)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-
-    def test_bridge_simple_constant(self):
-        ops = """
-        []
-        jump(ConstPtr(myptr))
-        """
-        self.find_bridge(ops, '', 'Not')
-        self.find_bridge(ops, '', 'Constant(myptr)')
-        self.find_bridge(ops, '', 'Constant(myptr2)', mismatch=True)
-
-    def test_bridge_simple_constant_mismatch(self):
-        ops = """
-        [p0]
-        jump(p0)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-        self.find_bridge(ops, 'Not', 'Constant(myptr)', mismatch=True)
-
-    def test_bridge_simple_virtual_1(self):
-        ops = """
-        [i0]
-        p0 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p0, i0, descr=valuedescr)
-        jump(p0)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-        self.find_bridge(ops, 'Not', 'Virtual(node_vtable, valuedescr=Not)')
-        self.find_bridge(ops, 'Not',
-                         '''Virtual(node_vtable,
-                                    valuedescr=Not,
-                                    nextdescr=Not)''')
-        #
-        self.find_bridge(ops, 'Not', 'Virtual(node_vtable)',
-                         mismatch=True)   # missing valuedescr
-        self.find_bridge(ops, 'Not', 'Virtual(node_vtable, nextdescr=Not)',
-                         mismatch=True)   # missing valuedescr
-        self.find_bridge(ops, 'Not', 'Virtual(node_vtable2, valuedescr=Not)',
-                         mismatch=True)   # bad class
-
-    def test_bridge_simple_virtual_struct(self):
-        ops = """
-        [i0]
-        p0 = new(descr=ssize)
-        setfield_gc(p0, i0, descr=adescr)
-        jump(p0)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-        self.find_bridge(ops, 'Not', 'VStruct(ssize, adescr=Not)')
-
-    def test_bridge_simple_virtual_struct_non_unique(self):
-        ops = """
-        [i0]
-        p0 = new(descr=ssize)
-        setfield_gc(p0, i0, descr=adescr)
-        jump(p0, p0)
-        """
-        self.find_bridge(ops, 'Not', 'Not, Not')
-        self.find_bridge(ops, 'Not', 'VStruct(ssize), VStruct(ssize)',
-                         mismatch=True)
-
-
-    def test_bridge_simple_virtual_2(self):
-        ops = """
-        [p0]
-        setfield_gc(p0, 123, descr=valuedescr)
-        jump(p0)
-        """
-        self.find_bridge(ops, 'Virtual(node_vtable)', 'Not')
-        self.find_bridge(ops, 'Virtual(node_vtable)',
-                              'Virtual(node_vtable, valuedescr=Not)')
-        self.find_bridge(ops, 'Virtual(node_vtable, valuedescr=Not)',
-                              'Virtual(node_vtable, valuedescr=Not)')
-        self.find_bridge(ops, 'Virtual(node_vtable, valuedescr=Not)',
-                            '''Virtual(node_vtable,
-                                       valuedescr=Not,
-                                       nextdescr=Not)''')
-        self.find_bridge(ops, '''Virtual(node_vtable,
-                                         valuedescr=Not,
-                                         nextdescr=Not)''',
-                              '''Virtual(node_vtable,
-                                         valuedescr=Not,
-                                         nextdescr=Not)''')
-        #
-        self.find_bridge(ops, 'Virtual(node_vtable)', 'Virtual(node_vtable)',
-                         mismatch=True)    # because of missing valuedescr
-        self.find_bridge(ops, 'Virtual(node_vtable)',
-                         'Virtual(node_vtable2, valuedescr=Not)',
-                         mismatch=True)    # bad class
-
-    def test_bridge_virtual_mismatch_1(self):
-        ops = """
-        [i0]
-        p0 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p0, i0, descr=valuedescr)
-        jump(p0, p0)
-        """
-        self.find_bridge(ops, 'Not', 'Not, Not')
-        #
-        self.find_bridge(ops, 'Not',
-                         '''Virtual(node_vtable, valuedescr=Not),
-                            Virtual(node_vtable, valuedescr=Not)''',
-                         mismatch=True)    # duplicate p0
-
-    def test_bridge_guard_class(self):
-        ops = """
-        [p1]
-        p2 = getfield_gc(p1, descr=nextdescr)
-        guard_class(p2, ConstClass(node_vtable)) []
-        jump(p2)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-        self.find_bridge(ops, 'Virtual(node_vtable2, nextdescr=Not)', 'Not')
-        self.find_bridge(ops,
-            '''Virtual(node_vtable,
-                       nextdescr=Virtual(node_vtable,
-                                         nextdescr=Not))''',
-            '''Virtual(node_vtable,
-                       nextdescr=Not)''')
-        #
-        self.find_bridge(ops, 'Not', 'Virtual(node_vtable)',
-                         mismatch=True)
-
-    def test_bridge_unused(self):
-        ops = """
-        []
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        p3 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, p2, descr=nextdescr)
-        setfield_gc(p2, p3, descr=nextdescr)
-        jump(p1)
-        """
-        self.find_bridge(ops, '',
-            '''Not''')
-        self.find_bridge(ops, '',
-            '''Virtual(node_vtable,
-                       nextdescr=Not)''')
-        self.find_bridge(ops, '',
-            '''Virtual(node_vtable,
-                       nextdescr=Virtual(node_vtable,
-                                         nextdescr=Not))''')
-        self.find_bridge(ops, '',
-            '''Virtual(node_vtable,
-                       nextdescr=Virtual(node_vtable,
-                                         nextdescr=Virtual(node_vtable)))''')
-        self.find_bridge(ops, '',
-            '''Virtual(node_vtable,
-                       nextdescr=Virtual(node_vtable,
-                                         nextdescr=Virtual(node_vtable,
-                                                           nextdescr=Not)))''')
-
-    def test_bridge_to_finish(self):
-        ops = """
-        [i1]
-        i2 = int_add(i1, 5)
-        finish(i2)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-
-    def test_bridge_virtual_to_finish(self):
-        ops = """
-        [i1]
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, i1, descr=valuedescr)
-        finish(p1)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-        self.find_bridge(ops, 'Not',
-                         'Virtual(node_vtable, valuedescr=Not)',
-                         mismatch=True)
-
-    def test_bridge_array_virtual_1(self):
-        ops = """
-        [i1]
-        p1 = new_array(3, descr=arraydescr)
-        setarrayitem_gc(p1, 0, i1, descr=arraydescr)
-        jump(p1)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-        self.find_bridge(ops, 'Not', 'VArray(arraydescr, Not, Not, Not)')
-
-    def test_bridge_array_virtual_size_mismatch(self):
-        ops = """
-        [i1]
-        p1 = new_array(5, descr=arraydescr)
-        setarrayitem_gc(p1, 0, i1, descr=arraydescr)
-        jump(p1)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-        self.find_bridge(ops, 'Not', 'VArray(arraydescr, Not, Not, Not)',
-                         mismatch=True)
-
-    def test_bridge_array_virtual_2(self):
-        ops = """
-        [i1]
-        p1 = new_array(3, descr=arraydescr)
-        setarrayitem_gc(p1, 0, i1, descr=arraydescr)
-        escape(p1)
-        jump(p1)
-        """
-        self.find_bridge(ops, 'Not', 'Not')
-        self.find_bridge(ops, 'Not', 'VArray(arraydescr, Not, Not, Not)',
-                         mismatch=True)
-
-    def test_bridge_nested_structs(self):
-        ops = """
-        []
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, p2, descr=nextdescr)
-        jump(p1)
-        """
-        self.find_bridge(ops, '', 'Not')
-        self.find_bridge(ops, '', 'Virtual(node_vtable, nextdescr=Not)')
-        self.find_bridge(ops, '',
-                   'Virtual(node_vtable, nextdescr=Virtual(node_vtable))')
-        self.find_bridge(ops, '',
-                   'Virtual(node_vtable, nextdescr=Virtual(node_vtable2))',
-                   mismatch=True)
-
-
-class TestLLtype(BaseTestOptimizeFindNode, LLtypeMixin):
-    pass
-
-##class TestOOtype(BaseTestOptimizeFindNode, OOtypeMixin):
-##    def test_find_nodes_instanceof(self):
-##        ops = """
-##        [i0]
-##        p0 = new_with_vtable(ConstClass(node_vtable))
-##        i1 = instanceof(p0, descr=nodesize)
-##        jump(i1)
-##        """
-##        boxes, getnode = self.find_nodes(ops, 'Not')
-##        assert not getnode(boxes.p0).escaped

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py	Sat Oct 30 16:58:59 2010
@@ -5,7 +5,7 @@
 from pypy.jit.metainterp.resume import *
 from pypy.jit.metainterp.history import BoxInt, BoxPtr, ConstInt
 from pypy.jit.metainterp.history import ConstPtr, ConstFloat
-from pypy.jit.metainterp.test.test_optimizefindnode import LLtypeMixin
+from pypy.jit.metainterp.test.test_optimizeutil import LLtypeMixin
 from pypy.jit.metainterp import executor
 from pypy.jit.codewriter import heaptracker
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py	Sat Oct 30 16:58:59 2010
@@ -11,7 +11,7 @@
 from pypy.rpython.rclass import FieldListAccessor
 from pypy.jit.metainterp.warmspot import get_stats, get_translator
 from pypy.jit.metainterp import history
-from pypy.jit.metainterp.test.test_optimizefindnode import LLtypeMixin
+from pypy.jit.metainterp.test.test_optimizeutil import LLtypeMixin
 
 def promote_virtualizable(*args):
     pass


From hakanardo at codespeak.net  Sat Oct 30 17:11:33 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 17:11:33 +0200 (CEST)
Subject: [pypy-svn] r78534 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030151133.14442282B9D@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 17:11:31 2010
New Revision: 78534

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 17:11:31 2010
@@ -947,11 +947,15 @@
         i1 = ptr_eq(p2, NULL)
         jump(i1)
         """
-        expected = """
+        preamble = """
         [i0]
-        jump(1)
+        jump()
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        []
+        jump()
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
 
     def test_virtual_constant_isnonnull(self):
@@ -963,9 +967,13 @@
         i1 = ptr_eq(p2, NULL)
         jump(i1)
         """
-        expected = """
+        preamble = """
         [i0]
-        jump(0)
+        jump()
+        """
+        expected = """
+        []
+        jump()
         """
         self.optimize_loop(ops, 'Not', expected)
 
@@ -1001,8 +1009,22 @@
         setfield_gc(p1, i1, descr=valuedescr)
         jump(i, p1)
         """
-        expected = ops
-        self.optimize_loop(ops, 'Not, Not', expected)
+        preamble = """
+        [i, p0]
+        i0 = getfield_gc(p0, descr=valuedescr)
+        escape(p0)
+        i1 = int_add(i0, i)
+        jump(i, i1)
+        """
+        expected = """
+        [i, i1]
+        p1 = new_with_vtable(ConstClass(node_vtable))
+        setfield_gc(p1, i1, descr=valuedescr)
+        escape(p1)
+        i2 = int_add(i1, i)
+        jump(i, i2)
+        """
+        self.optimize_loop(ops, 'Not, Not', expected, preamble)
 
     def test_nonvirtual_later(self):
         ops = """
@@ -1079,8 +1101,12 @@
         i1 = getfield_gc(ConstPtr(myptr), descr=valuedescr)
         jump(i1)
         """
-        expected = ops
-        self.optimize_loop(ops, 'Not', expected)
+        preamble = ops
+        expected = """
+        [i]
+        jump(i)
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_varray_1(self):
         ops = """
@@ -1107,11 +1133,15 @@
         i2 = getarrayitem_gc(p1, 1, descr=arraydescr)
         jump(i2)
         """
-        expected = """
+        preamble = """
         [i1]
-        jump(0)
+        jump()
         """
-        self.optimize_loop(ops, 'Not', expected)
+        expected = """
+        []
+        jump()
+        """
+        self.optimize_loop(ops, 'Not', expected, preamble)
 
     def test_varray_float(self):
         ops = """
@@ -1176,13 +1206,21 @@
         setarrayitem_gc(p2, 0, 20, descr=arraydescr)
         jump(i0, p2)
         """
-        expected = """
-        [i0, i1, i2]
+        preamble = """
+        [i0, p1]
+        i1 = getarrayitem_gc(p1, 0, descr=arraydescr)
+        i2 = getarrayitem_gc(p1, 1, descr=arraydescr)
         i3 = int_sub(i1, i2)
         guard_value(i3, 15) []
-        jump(i0, 20, i0)
+        jump(i0)
+        """
+        expected = """
+        [i0]
+        i3 = int_sub(20, i0)
+        guard_value(i3, 15) []
+        jump(5)
         """
-        self.optimize_loop(ops, 'Not, VArray(arraydescr, Not, Not)', expected)
+        self.optimize_loop(ops, 'Not, Not', expected, preamble)
 
     def test_p123_array(self):
         ops = """


From hakanardo at codespeak.net  Sat Oct 30 17:23:16 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 17:23:16 +0200 (CEST)
Subject: [pypy-svn] r78535 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030152316.37B3C282BF4@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 17:23:14 2010
New Revision: 78535

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 17:23:14 2010
@@ -1231,8 +1231,22 @@
         setarrayitem_gc(p1, 0, i1, descr=arraydescr)
         jump(i1, p1, p2)
         """
+        preamble = """
+        [i1, p2, p3]
+        i3 = getarrayitem_gc(p3, 0, descr=arraydescr)
+        escape(i3)
+        jump(i1, p2)
+        """
+        expected = """
+        [i1, p2]
+        i3 = getarrayitem_gc(p2, 0, descr=arraydescr)
+        escape(i3)
+        p1 = new_array(1, descr=arraydescr)
+        setarrayitem_gc(p1, 0, i1, descr=arraydescr)
+        jump(i1, p1)
+        """
         # We cannot track virtuals that survive for more than two iterations.
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
 
     def test_varray_forced_1(self):
         ops = """
@@ -1265,12 +1279,18 @@
         setfield_gc(p3, i1, descr=adescr)
         jump(i1, p3)
         """
-        expected = """
-        [i1, i2]
+        preamble = """
+        [i1, p2]
+        i2 = getfield_gc(p2, descr=adescr)
         escape(i2)
-        jump(i1, i1)
+        jump(i1)
+        """
+        expected = """
+        [i1]
+        escape(i1)
+        jump(i1)
         """
-        self.optimize_loop(ops, 'Not, VStruct(ssize, adescr=Not)', expected)
+        self.optimize_loop(ops, 'Not, Not', expected, preamble)
 
     def test_p123_vstruct(self):
         ops = """
@@ -1281,8 +1301,22 @@
         setfield_gc(p1, i1, descr=adescr)
         jump(i1, p1, p2)
         """
+        preamble = """
+        [i1, p2, p3]
+        i3 = getfield_gc(p3, descr=adescr)
+        escape(i3)
+        jump(i1, p2)
+        """
+        expected = """
+        [i1, p2]
+        i3 = getfield_gc(p2, descr=adescr)
+        escape(i3)
+        p1 = new(descr=ssize)
+        setfield_gc(p1, i1, descr=adescr)
+        jump(i1, p1)
+        """
         # We cannot track virtuals that survive for more than two iterations.
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
 
     def test_duplicate_getfield_1(self):
         ops = """


From cfbolz at codespeak.net  Sat Oct 30 17:28:19 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 17:28:19 +0200 (CEST)
Subject: [pypy-svn] r78536 - pypy/branch/jit-unroll-loops/pypy/jit/metainterp
Message-ID: <20101030152819.1CCBA282BE3@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 17:28:17 2010
New Revision: 78536

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py
Log:
(arigo, cfbolz) kill findnode here


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py	Sat Oct 30 17:28:17 2010
@@ -1,8 +1,8 @@
 
 from pypy.rlib.debug import debug_start, debug_stop
 from pypy.jit.metainterp.optimizeopt import optimize_loop_1, optimize_bridge_1
-from pypy.jit.metainterp.optimizefindnode import PerfectSpecializationFinder
-from pypy.jit.metainterp.optimizefindnode import BridgeSpecializationFinder
+
+# XXX this file should probably disable unrolling instead
 
 def optimize_loop(metainterp_sd, old_loop_tokens, loop):
     debug_start("jit-optimize")
@@ -14,11 +14,6 @@
 def _optimize_loop(metainterp_sd, old_loop_tokens, loop):
     cpu = metainterp_sd.cpu
     metainterp_sd.logger_noopt.log_loop(loop.inputargs, loop.operations)
-    # XXX the following lines are probably still needed, to discard invalid
-    # loops. bit silly to run a full perfect specialization and throw the
-    # result away.
-    finder = PerfectSpecializationFinder(cpu)
-    finder.find_nodes_loop(loop, False)
     if old_loop_tokens:
         return old_loop_tokens[0]
     optimize_loop_1(metainterp_sd, loop)
@@ -34,9 +29,6 @@
 def _optimize_bridge(metainterp_sd, old_loop_tokens, bridge):
     cpu = metainterp_sd.cpu    
     metainterp_sd.logger_noopt.log_loop(bridge.inputargs, bridge.operations)
-    # XXX same comment as above applies
-    finder = BridgeSpecializationFinder(cpu)
-    finder.find_nodes_bridge(bridge)
     if old_loop_tokens:
         old_loop_token = old_loop_tokens[0]
         bridge.operations[-1].setdescr(old_loop_token)   # patch jump target


From hakanardo at codespeak.net  Sat Oct 30 17:32:19 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 17:32:19 +0200 (CEST)
Subject: [pypy-svn] r78537 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030153219.B805D282BAD@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 17:32:17 2010
New Revision: 78537

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 17:32:17 2010
@@ -1569,7 +1569,7 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2, p3)
         """
-        expected = """
+        preamble = """
         [p1, i1, i2, p3]
         #
         i3 = getarrayitem_gc_pure(p3, 1, descr=arraydescr)
@@ -1579,9 +1579,20 @@
         #
         setfield_gc(p1, i2, descr=valuedescr)
         setfield_gc(p1, i4, descr=nextdescr)
-        jump(p1, i1, i2, p3)
+        jump(p1, i1, i2, p3, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        expected = """
+        [p1, i1, i2, p3, i3]
+        #
+        i4 = getarrayitem_gc(p3, i3, descr=arraydescr)
+        i5 = int_add(i3, i4)
+        setarrayitem_gc(p3, 0, i5, descr=arraydescr)
+        #
+        setfield_gc(p1, i2, descr=valuedescr)
+        setfield_gc(p1, i4, descr=nextdescr)
+        jump(p1, i1, i2, p3, i3)
+        """
+        self.optimize_loop(ops, 'Not, Not, Not, Not', expected, preamble)
 
     def test_duplicate_setfield_5(self):
         ops = """


From david at codespeak.net  Sat Oct 30 17:53:49 2010
From: david at codespeak.net (david at codespeak.net)
Date: Sat, 30 Oct 2010 17:53:49 +0200 (CEST)
Subject: [pypy-svn] r78538 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101030155349.CA711282BAD@codespeak.net>

Author: david
Date: Sat Oct 30 17:53:48 2010
New Revision: 78538

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
Log:
Implement uint_floordiv and some helper functions to generate operations

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/arch.py	Sat Oct 30 17:53:48 2010
@@ -1,14 +1,18 @@
+from pypy.rlib.rarithmetic import r_uint
 from pypy.rpython.lltypesystem import lltype
 
 FUNC_ALIGN=8
 WORD=4
 
 arm_int_div_sign = lltype.Ptr(lltype.FuncType([lltype.Signed, lltype.Signed], lltype.Signed))
-arm_int_mod_sign = arm_int_div_sign
-
 def arm_int_div(a, b):
     return int(a/float(b))
 
+arm_uint_div_sign = lltype.Ptr(lltype.FuncType([lltype.Unsigned, lltype.Unsigned], lltype.Unsigned))
+def arm_uint_div(a, b):
+    return r_uint(a)/r_uint(b)
+
+arm_int_mod_sign = arm_int_div_sign
 def arm_int_mod(a, b):
     sign = 1
     if a < 0:
@@ -18,3 +22,4 @@
         b = -1 * b
     res = a % b
     return sign * res
+

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Sat Oct 30 17:53:48 2010
@@ -1,7 +1,7 @@
+from pypy.jit.backend.arm import arch
 from pypy.jit.backend.arm import conditions as cond
 from pypy.jit.backend.arm import registers as reg
-from pypy.jit.backend.arm.arch import (WORD, FUNC_ALIGN, arm_int_div,
-                                        arm_int_div_sign, arm_int_mod_sign, arm_int_mod)
+from pypy.jit.backend.arm.arch import (WORD, FUNC_ALIGN)
 from pypy.jit.backend.arm.instruction_builder import define_instructions
 
 from pypy.rlib.rmmap import alloc, PTR
@@ -9,6 +9,20 @@
 from pypy.rpython.lltypesystem import lltype, rffi
 from pypy.jit.metainterp.history import ConstInt, BoxInt, Box, BasicFailDescr
 
+def binary_helper_call(name):
+    signature = getattr(arch, 'arm_%s_sign' % name)
+    function = getattr(arch, 'arm_%s' % name)
+    def f(self, cond=cond.AL):
+        """Generates a call to a helper function, takes its
+        arguments in r0 and r1, result is placed in r0"""
+        self.PUSH(range(2, 12), cond=cond)
+        addr = rffi.cast(lltype.Signed, llhelper(signature, function))
+        self.gen_load_int(reg.r2.value, addr, cond=cond)
+        self.gen_load_int(reg.lr.value, self.curraddr()+self.size_of_gen_load_int+WORD, cond=cond)
+        self.MOV_rr(reg.pc.value, reg.r2.value, cond=cond)
+        self.LDM(reg.sp.value, range(2, 12), w=1, cond=cond) # XXX Replace with POP instr. someday
+    return f
+
 class AbstractARMv7Builder(object):
     def _init(self, data, map_size):
         self._data = data
@@ -43,25 +57,9 @@
     def BKPT(self, cond=cond.AL):
         self.write32(cond << 28 | 0x1200070)
 
-    def DIV(self, cond=cond.AL):
-        """Generates a call to a helper function used for division, takes its
-        arguments in r0 and r1, result is placed in r0"""
-        self.PUSH(range(2, 12), cond=cond)
-        div_addr = rffi.cast(lltype.Signed, llhelper(arm_int_div_sign, arm_int_div))
-        self.gen_load_int(reg.r2.value, div_addr, cond=cond)
-        self.gen_load_int(reg.lr.value, self.curraddr()+self.size_of_gen_load_int+WORD, cond=cond)
-        self.MOV_rr(reg.pc.value, reg.r2.value, cond=cond)
-        self.LDM(reg.sp.value, range(2, 12), w=1, cond=cond) # XXX Replace with POP instr. someday
-
-    def MOD(self, cond=cond.AL):
-        """Generate a call to a helper function used for modulo, takes its
-        arguments in r0 and r1, result is placed in r0"""
-        self.PUSH(range(2, 12), cond=cond)
-        mod_addr = rffi.cast(lltype.Signed, llhelper(arm_int_mod_sign, arm_int_mod))
-        self.gen_load_int(reg.r2.value, mod_addr, cond=cond)
-        self.gen_load_int(reg.lr.value, self.curraddr()+self.size_of_gen_load_int+WORD, cond=cond)
-        self.MOV_rr(reg.pc.value, reg.r2.value, cond=cond)
-        self.LDM(reg.sp.value, range(2, 12), w=1, cond=cond) # XXX Replace with POP instr. someday
+    DIV = binary_helper_call('int_div')
+    MOD = binary_helper_call('int_mod')
+    UDIV = binary_helper_call('uint_div')
 
     def _encode_reg_list(self, instr, regs):
         for reg in regs:
@@ -121,7 +119,7 @@
         self.checks = True
         self.n_data=0
 
-    _space_for_jump = 10 * WORD
+    _space_for_jump = 9 * WORD
     def writechar(self, char):
         if self.checks and not self._pos < self._size - self._space_for_jump:
             self.checks = False
@@ -133,13 +131,13 @@
     def _add_more_mem(self):
         new_mem = alloc(self._size)
         new_mem_addr = rffi.cast(lltype.Signed, new_mem)
-        self.PUSH([reg.r0.value, reg.ip.value])
-        self.gen_load_int(reg.r0.value, new_mem_addr)
-        self.MOV_rr(reg.pc.value, reg.r0.value)
+        self.PUSH([reg.ip.value, reg.lr.value])
+        self.gen_load_int(reg.lr.value, new_mem_addr)
+        self.MOV_rr(reg.pc.value, reg.lr.value)
         self._dump_trace('data%d.asm' % self.n_data)
-        self.n_data+=1
+        self.n_data += 1
         self._data = new_mem
         self._pos = 0
-        self.LDM(reg.sp.value, [reg.r0.value, reg.ip.value], w=1) # XXX Replace with POP instr. someday
+        self.LDM(reg.sp.value, [reg.ip.value, reg.lr.value], w=1) # XXX Replace with POP instr. someday
 
 define_instructions(AbstractARMv7Builder)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instruction_builder.py	Sat Oct 30 17:53:48 2010
@@ -175,6 +175,8 @@
 
     return f
 
+def define_long_mult_div_instructions(name, table):
+    pass
 def imm_operation(rt, rn, imm):
     return ((rn & 0xFF) << 16
     | (rt & 0xFF) << 12

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	Sat Oct 30 17:53:48 2010
@@ -39,6 +39,19 @@
         return fcond
     return f
 
+def gen_emit_op_by_helper_call(opname):
+    def f(self, op, regalloc, fcond):
+        arg1 = regalloc.make_sure_var_in_reg(op.getarg(0), selected_reg=r.r0)
+        arg2 = regalloc.make_sure_var_in_reg(op.getarg(1), selected_reg=r.r1)
+        assert arg1 == r.r0
+        assert arg2 == r.r1
+        res = regalloc.try_allocate_reg(op.result)
+        getattr(self.mc, opname)(fcond)
+        self.mc.MOV_rr(res.value, r.r0.value, cond=fcond)
+        regalloc.possibly_free_vars_for_op(op)
+        return fcond
+    return f
+
 class IntOpAsslember(object):
     _mixin_ = True
 
@@ -104,28 +117,6 @@
         regalloc.possibly_free_var(reg2)
         return fcond
 
-    def emit_op_int_floordiv(self, op, regalloc, fcond):
-        arg1 = regalloc.make_sure_var_in_reg(op.getarg(0), selected_reg=r.r0)
-        arg2 = regalloc.make_sure_var_in_reg(op.getarg(1), selected_reg=r.r1)
-        assert arg1 == r.r0
-        assert arg2 == r.r1
-        res = regalloc.try_allocate_reg(op.result)
-        self.mc.DIV(fcond)
-        self.mc.MOV_rr(res.value, r.r0.value, cond=fcond)
-        regalloc.possibly_free_vars_for_op(op)
-        return fcond
-
-    def emit_op_int_mod(self, op, regalloc, fcond):
-        res = regalloc.force_allocate_reg(op.result)
-        arg1 = regalloc.make_sure_var_in_reg(op.getarg(0), selected_reg=r.r0)
-        arg2 = regalloc.make_sure_var_in_reg(op.getarg(1), selected_reg=r.r1)
-        assert arg1 == r.r0
-        assert arg2 == r.r1
-        self.mc.MOD(fcond)
-        self.mc.MOV_rr(res.value, r.r0.value, cond=fcond)
-        regalloc.possibly_free_vars_for_op(op)
-        return fcond
-
     def _put_in_reg(self, box, regalloc):
         if isinstance(box, ConstInt):
             t = Box()
@@ -135,6 +126,10 @@
             reg = regalloc.try_allocate_reg(box)
         return reg
 
+    emit_op_int_floordiv = gen_emit_op_by_helper_call('DIV')
+    emit_op_int_mod = gen_emit_op_by_helper_call('MOD')
+    emit_op_uint_floordiv = gen_emit_op_by_helper_call('UDIV')
+
     emit_op_int_and = gen_emit_op_ri('AND')
     emit_op_int_or = gen_emit_op_ri('ORR')
     emit_op_int_xor = gen_emit_op_ri('EOR')


From hakanardo at codespeak.net  Sat Oct 30 17:58:35 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 17:58:35 +0200 (CEST)
Subject: [pypy-svn] r78539 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030155835.EAF50282C00@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 17:58:34 2010
New Revision: 78539

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 17:58:34 2010
@@ -1635,7 +1635,22 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', ops)
+        preamble = """
+        [p1, i1, i2, i3]
+        setfield_gc(p1, i1, descr=valuedescr)
+        guard_true(i3) []
+        i4 = int_neg(i2)
+        setfield_gc(p1, i2, descr=valuedescr)
+        jump(p1, i1, i2, i4)
+        """
+        expected = """
+        [p1, i1, i2, i4]
+        setfield_gc(p1, i1, descr=valuedescr)
+        guard_true(i4) []
+        setfield_gc(p1, i2, descr=valuedescr)
+        jump(p1, i1, i2, 1)
+        """
+        self.optimize_loop(ops, 'Not, Not, Not, Not', expected, preamble)
 
     def test_duplicate_setfield_residual_guard_2(self):
         # the difference with the previous test is that the field value is
@@ -1649,14 +1664,20 @@
         setfield_gc(p1, NULL, descr=nextdescr)
         jump(p1, i2, i4)
         """
-        expected = """
+        preamble = """
         [p1, i2, i3]
         guard_true(i3) [p1]
         i4 = int_neg(i2)
         setfield_gc(p1, NULL, descr=nextdescr)
         jump(p1, i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        expected = """
+        [p1, i2, i4]
+        guard_true(i4) [p1]
+        setfield_gc(p1, NULL, descr=nextdescr)
+        jump(p1, i2, 1)
+        """
+        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
 
     def test_duplicate_setfield_residual_guard_3(self):
         ops = """
@@ -1669,13 +1690,19 @@
         setfield_gc(p1, NULL, descr=nextdescr)
         jump(p1, i2, i4)
         """
-        expected = """
+        preamble = """
         [p1, i2, i3]
         guard_true(i3) [i2, p1]
         i4 = int_neg(i2)
         setfield_gc(p1, NULL, descr=nextdescr)
         jump(p1, i2, i4)
         """
+        expected = """
+        [p1, i2, i4]
+        guard_true(i4) [i2, p1]
+        setfield_gc(p1, NULL, descr=nextdescr)
+        jump(p1, i2, 1)
+        """
         self.optimize_loop(ops, 'Not, Not, Not', expected)
 
     def test_duplicate_setfield_residual_guard_4(self):
@@ -1690,7 +1717,16 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', ops)
+        preamble = ops
+        expected = """
+        [p1, i1, i2, i4]
+        setfield_gc(p1, i1, descr=valuedescr)
+        i5 = int_eq(i4, 5)
+        guard_true(i5) []
+        setfield_gc(p1, i2, descr=valuedescr)
+        jump(p1, i1, i2, 5)
+        """
+        self.optimize_loop(ops, 'Not, Not, Not, Not', expected, preamble)
 
     def test_duplicate_setfield_aliasing(self):
         # a case where aliasing issues (and not enough cleverness) mean


From arigo at codespeak.net  Sat Oct 30 18:03:10 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 18:03:10 +0200 (CEST)
Subject: [pypy-svn] r78540 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030160310.71671282C01@codespeak.net>

Author: arigo
Date: Sat Oct 30 18:03:08 2010
New Revision: 78540

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
Log:
Fix.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	Sat Oct 30 18:03:08 2010
@@ -265,7 +265,7 @@
                          OptHeap(),
                          ]
         optimizer = Optimizer(metainterp_sd, loop, optimizations,
-                              virtuals=True)
+                              not_a_bridge=True)
         optimizer.propagate_all_forward()
         #
         expected = self.parse(optops)


From cfbolz at codespeak.net  Sat Oct 30 18:06:52 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 18:06:52 +0200 (CEST)
Subject: [pypy-svn] r78541 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: . test
Message-ID: <20101030160652.6FB3D282C03@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 18:06:50 2010
New Revision: 78541

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(arigo, cfbolz): turn make_fail_descr into a setup_method and sanitize it a bit.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py	Sat Oct 30 18:06:50 2010
@@ -299,6 +299,7 @@
         storage = self.storage
         # make sure that nobody attached resume data to this guard yet
         assert storage.rd_numb is None
+        assert storage.rd_snapshot is not None # is that true?
         numb, liveboxes_from_env, v = self.memo.number(values,
                                                        storage.rd_snapshot)
         self.liveboxes_from_env = liveboxes_from_env

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 18:06:50 2010
@@ -66,6 +66,9 @@
 
     def optimize_loop(self, ops, spectext, optops, expected_preamble=None):
         loop = self.parse(ops)
+        expected = self.parse(optops)
+        if expected_preamble:
+            expected_preamble = self.parse(expected_preamble)
         #
         self.loop = loop
         loop.preamble = TreeLoop('preamble')
@@ -75,7 +78,6 @@
             metainterp_sd.virtualref_info = self.vrefinfo
         optimize_loop_1(metainterp_sd, loop)
         #
-        expected = self.parse(optops)
 
         print
         print loop.preamble.inputargs
@@ -87,13 +89,36 @@
         
         self.assert_equal(loop, expected)
         if expected_preamble:
-            expected = self.parse(expected_preamble)
-            self.assert_equal(loop.preamble, expected)
+            self.assert_equal(loop.preamble, expected_preamble)
 
         return loop
 
 class OptimizeOptTest(BaseTestOptimizeOpt):
 
+    def setup_method(self, meth=None):
+        class FailDescr(compile.ResumeGuardDescr):
+            oparse = None
+            def _oparser_uses_descr_of_guard(self, oparse, fail_args):
+                # typically called twice, before and after optimization
+                if self.oparse is None:
+                    self.rd_frame_info_list = resume.FrameInfo(None,
+                                                                 "code", 11)
+                    self.rd_snapshot = resume.Snapshot(None, fail_args)
+                self.oparse = oparse
+            def _clone_if_mutable(self):
+                assert self is fdescr
+                return fdescr2
+        #
+        fdescr = instantiate(FailDescr)
+        self.namespace['fdescr'] = fdescr
+        fdescr2 = instantiate(FailDescr)
+        self.namespace['fdescr2'] = fdescr2
+
+    def teardown_method(self, meth):
+        self.namespace.pop('fdescr', None)
+        self.namespace.pop('fdescr2', None)
+
+
     def test_simple(self):
         ops = """
         []
@@ -2079,7 +2104,6 @@
         self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected)
 
     def test_merge_guard_nonnull_guard_class(self):
-        self.make_fail_descr()
         ops = """
         [p1, i0, i1, i2, p2]
         guard_nonnull(p1, descr=fdescr) [i0]
@@ -2087,17 +2111,22 @@
         guard_class(p1, ConstClass(node_vtable)) [i1]
         jump(p2, i0, i1, i3, p2)
         """
-        expected = """
+        preamble = """
         [p1, i0, i1, i2, p2]
         guard_nonnull_class(p1, ConstClass(node_vtable), descr=fdescr) [i0]
         i3 = int_add(i1, i2)
-        jump(p2, i0, i1, i3, p2)
+        jump(p2, i0, i1, i3)
         """
-        self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected)
+        expected = """
+        [p2, i0, i1, i2]
+        guard_nonnull_class(p2, ConstClass(node_vtable), descr=fdescr2) [i0]
+        i3 = int_add(i1, i2)
+        jump(p2, i0, i1, i3)
+        """
+        self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected, preamble)
         self.check_expanded_fail_descr("i0", rop.GUARD_NONNULL_CLASS)
 
     def test_merge_guard_nonnull_guard_value(self):
-        self.make_fail_descr()
         ops = """
         [p1, i0, i1, i2, p2]
         guard_nonnull(p1, descr=fdescr) [i0]
@@ -2115,7 +2144,6 @@
         self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
 
     def test_merge_guard_nonnull_guard_class_guard_value(self):
-        self.make_fail_descr()
         ops = """
         [p1, i0, i1, i2, p2]
         guard_nonnull(p1, descr=fdescr) [i0]
@@ -2333,23 +2361,6 @@
 
     # ----------
 
-    def make_fail_descr(self):
-        class FailDescr(compile.ResumeGuardDescr):
-            oparse = None
-            def _oparser_uses_descr_of_guard(self, oparse, fail_args):
-                # typically called twice, before and after optimization
-                if self.oparse is None:
-                    fdescr.rd_frame_info_list = resume.FrameInfo(None,
-                                                                 "code", 11)
-                    fdescr.rd_snapshot = resume.Snapshot(None, fail_args)
-                self.oparse = oparse
-        #
-        fdescr = instantiate(FailDescr)
-        self.namespace['fdescr'] = fdescr
-
-    def teardown_method(self, meth):
-        self.namespace.pop('fdescr', None)
-
     def _verify_fail_args(self, boxes, oparse, text):
         import re
         r = re.compile(r"\bwhere\s+(\w+)\s+is a\s+(\w+)")
@@ -2458,7 +2469,6 @@
         self._verify_fail_args(boxes, fdescr.oparse, expectedtext)
 
     def test_expand_fail_1(self):
-        self.make_fail_descr()
         ops = """
         [i1, i3]
         # first rename i3 into i4
@@ -2479,7 +2489,6 @@
         self.check_expanded_fail_descr('15, i3', rop.GUARD_TRUE)
 
     def test_expand_fail_2(self):
-        self.make_fail_descr()
         ops = """
         [i1, i2]
         p1 = new_with_vtable(ConstClass(node_vtable))
@@ -2499,7 +2508,6 @@
             ''', rop.GUARD_TRUE)
 
     def test_expand_fail_3(self):
-        self.make_fail_descr()
         ops = """
         [i1, i2, i3, p3]
         p1 = new_with_vtable(ConstClass(node_vtable))
@@ -2525,7 +2533,7 @@
     def test_expand_fail_4(self):
         for arg in ['p1', 'i2,p1', 'p1,p2', 'p2,p1',
                     'i2,p1,p2', 'i2,p2,p1']:
-            self.make_fail_descr()
+            self.setup_method() # humpf
             ops = """
             [i1, i2, i3]
             p1 = new_with_vtable(ConstClass(node_vtable))
@@ -2550,7 +2558,6 @@
                                            rop.GUARD_TRUE)
 
     def test_expand_fail_5(self):
-        self.make_fail_descr()
         ops = """
         [i1, i2, i3, i4]
         p1 = new_with_vtable(ConstClass(node_vtable))
@@ -2574,7 +2581,6 @@
             ''', rop.GUARD_TRUE)
 
     def test_expand_fail_6(self):
-        self.make_fail_descr()
         ops = """
         [p0, i0, i1]
         guard_true(i0, descr=fdescr) [p0]
@@ -2594,7 +2600,6 @@
             ''', rop.GUARD_TRUE)
 
     def test_expand_fail_varray(self):
-        self.make_fail_descr()
         ops = """
         [i1]
         p1 = new_array(3, descr=arraydescr)
@@ -2615,7 +2620,6 @@
             ''', rop.GUARD_TRUE)
 
     def test_expand_fail_vstruct(self):
-        self.make_fail_descr()
         ops = """
         [i1, p1]
         p2 = new(descr=ssize)
@@ -2637,7 +2641,6 @@
             ''', rop.GUARD_TRUE)
 
     def test_expand_fail_v_all_1(self):
-        self.make_fail_descr()
         ops = """
         [i1, p1a, i2]
         p6s = getarrayitem_gc(p1a, 0, descr=arraydescr2)
@@ -2678,7 +2681,6 @@
             ''', rop.GUARD_TRUE)
 
     def test_expand_fail_lazy_setfield_1(self):
-        self.make_fail_descr()
         ops = """
         [p1, i2, i3]
         p2 = new_with_vtable(ConstClass(node_vtable))
@@ -2704,7 +2706,6 @@
             ''', rop.GUARD_TRUE)
 
     def test_expand_fail_lazy_setfield_2(self):
-        self.make_fail_descr()
         ops = """
         [i2, i3]
         p2 = new_with_vtable(ConstClass(node_vtable))
@@ -3022,7 +3023,6 @@
         self.optimize_loop(ops, 'Not, Not', expected)
 
     def test_vref_virtual_2(self):
-        self.make_fail_descr()
         ops = """
         [p0, i1]
         #
@@ -3069,7 +3069,6 @@
             ''', rop.GUARD_NOT_FORCED)
 
     def test_vref_virtual_and_lazy_setfield(self):
-        self.make_fail_descr()
         ops = """
         [p0, i1]
         #
@@ -3108,7 +3107,6 @@
             ''', rop.GUARD_NO_EXCEPTION)
 
     def test_vref_virtual_after_finish(self):
-        self.make_fail_descr()
         ops = """
         [i1]
         p1 = new_with_vtable(ConstClass(node_vtable))
@@ -3136,7 +3134,6 @@
         self.optimize_loop(ops, 'Not', expected)
 
     def test_vref_nonvirtual_and_lazy_setfield(self):
-        self.make_fail_descr()
         ops = """
         [i1, p1]
         p2 = virtual_ref(p1, 23)


From hakanardo at codespeak.net  Sat Oct 30 18:11:57 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 18:11:57 +0200 (CEST)
Subject: [pypy-svn] r78542 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030161157.DAC59282C06@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 18:11:56 2010
New Revision: 78542

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 18:11:56 2010
@@ -1926,6 +1926,9 @@
         [p1, i2, i3, p4]
         guard_true(i3) [p1, p4]
         i4 = int_neg(i2)
+        p2 = new_with_vtable(ConstClass(node_vtable))
+        setfield_gc(p2, p4, descr=nextdescr)
+        setfield_gc(p1, p2, descr=nextdescr)
         jump(p1, i2, i4, p4)
         """
         expected = """
@@ -2004,6 +2007,19 @@
         setfield_gc(p1a, p3a, descr=otherdescr)
         jump(p1a)
         """
+        preamble = """
+        [p1]
+        guard_nonnull(p1) []
+        guard_class(p1, ConstClass(node_vtable2)) []
+        p2 = getfield_gc(p1, descr=nextdescr)
+        guard_class(p2, ConstClass(node_vtable)) []
+        p3 = getfield_gc(p1, descr=otherdescr)
+        guard_class(p3, ConstClass(node_vtable)) []
+        p3a = new_with_vtable(ConstClass(node_vtable))
+        escape(p3a)
+        setfield_gc(p3, p2, descr=otherdescr)
+        jump(p2a, p3a)
+        """
         expected = """
         [p2, p3]
         guard_class(p2, ConstClass(node_vtable)) []


From cfbolz at codespeak.net  Sat Oct 30 18:21:49 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 18:21:49 +0200 (CEST)
Subject: [pypy-svn] r78543 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030162149.6AF19282BFE@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 18:21:45 2010
New Revision: 78543

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(arigo, cfbolz): remove spectext everywhere in test_optimizeopt


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 18:21:45 2010
@@ -64,7 +64,7 @@
         assert equaloplists(optimized.operations,
                             expected.operations, False, remap)
 
-    def optimize_loop(self, ops, spectext, optops, expected_preamble=None):
+    def optimize_loop(self, ops, optops, expected_preamble=None):
         loop = self.parse(ops)
         expected = self.parse(optops)
         if expected_preamble:
@@ -128,7 +128,7 @@
         escape(f)
         jump()
         """
-        self.optimize_loop(ops, '', ops)
+        self.optimize_loop(ops, ops)
 
     def test_constant_propagate(self):
         ops = """
@@ -145,7 +145,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constant_propagate_ovf(self):
         ops = """
@@ -163,7 +163,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constfold_all(self):
         from pypy.jit.backend.llgraph.llimpl import TYPES     # xxx fish
@@ -197,7 +197,7 @@
             escape(%d)
             jump()
             """ % expected_value
-            self.optimize_loop(ops, '', expected)
+            self.optimize_loop(ops, expected)
 
     # ----------
 
@@ -217,7 +217,7 @@
         [p0]
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected, expected_preamble=preamble)
+        self.optimize_loop(ops, expected, expected_preamble=preamble)
 
     def test_remove_guard_class_2(self):
         ops = """
@@ -233,7 +233,7 @@
         escape(p0)
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_guard_class_constant(self):
         ops = """
@@ -246,7 +246,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constant_boolrewrite_lt(self):
         ops = """
@@ -267,7 +267,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected, expected_preamble=preamble)
+        self.optimize_loop(ops, expected, expected_preamble=preamble)
 
     def test_constant_boolrewrite_gt(self):
         ops = """
@@ -288,7 +288,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected, expected_preamble=preamble)
+        self.optimize_loop(ops, expected, expected_preamble=preamble)
 
     def test_constant_boolrewrite_reflex(self):
         ops = """
@@ -309,7 +309,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected, expected_preamble=preamble)
+        self.optimize_loop(ops, expected, expected_preamble=preamble)
 
     def test_constant_boolrewrite_reflex_invers(self):
         ops = """
@@ -330,7 +330,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected, expected_preamble=preamble)
+        self.optimize_loop(ops, expected, expected_preamble=preamble)
 
     def test_remove_consecutive_guard_value_constfold(self):
         ops = """
@@ -350,7 +350,7 @@
         escape(3)
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_guard_value_if_constant(self):
         ops = """
@@ -362,7 +362,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_ooisnull_oononnull_1(self):
         ops = """
@@ -380,7 +380,7 @@
         [p0]
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_int_is_true_1(self):
         ops = """
@@ -401,7 +401,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_int_is_true_is_zero(self):
         py.test.skip("XXX implement me")
@@ -419,7 +419,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_ooisnull_oononnull_2(self):
         ops = """
@@ -437,7 +437,7 @@
         [p0]
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_ooisnull_on_null_ptr_1(self):
         ops = """
@@ -453,7 +453,7 @@
         guard_isnull(p0) []
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_ooisnull_oononnull_via_virtual(self):
         ops = """
@@ -474,7 +474,7 @@
         [p0]
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_oois_1(self):
         ops = """
@@ -499,7 +499,7 @@
         [p0]
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_nonnull_1(self):
         ops = """
@@ -521,7 +521,7 @@
         setfield_gc(p0, 5, descr=valuedescr)
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_const_guard_value(self):
         ops = """
@@ -534,7 +534,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constptr_guard_value(self):
         ops = """
@@ -543,7 +543,7 @@
         guard_value(p1, ConstPtr(myptr)) []
         jump()
         """
-        self.optimize_loop(ops, '', ops)
+        self.optimize_loop(ops, ops)
 
     def test_guard_value_to_guard_true(self):
         ops = """
@@ -562,7 +562,7 @@
         [i]
         jump(i)
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_guard_value_to_guard_false(self):
         ops = """
@@ -581,7 +581,7 @@
         [i]
         jump(i)
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_guard_value_on_nonbool(self):
         ops = """
@@ -600,7 +600,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_int_is_true_of_bool(self):
         ops = """
@@ -621,7 +621,7 @@
         [i0, i1]
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
 
 
@@ -650,7 +650,7 @@
         jump(i1, p3)
         """
         # We cannot track virtuals that survive for more than two iterations.
-        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_p123_nested(self):
         ops = """
@@ -683,7 +683,7 @@
         setfield_gc(p4, p1sub, descr=nextdescr)
         jump(i1, p4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_p123_anti_nested(self):
         ops = """
@@ -719,7 +719,7 @@
         setfield_gc(p1, p3sub, descr=nextdescr)
         jump(i1, p1, p3sub)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     # ----------
 
@@ -743,7 +743,7 @@
         i3 = call(i2, descr=nonwritedescr)
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected, expected)
+        self.optimize_loop(ops, expected, expected)
 
     # ----------
 
@@ -772,7 +772,7 @@
         [i1]
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
 
     # ----------
@@ -797,7 +797,7 @@
         i1 = int_add(i2, i)
         jump(i, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_virtual_float(self):
         ops = """
@@ -819,7 +819,7 @@
         f1 = float_add(f2, f)
         jump(f, f1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_virtual_oois(self):
         ops = """
@@ -867,7 +867,7 @@
         guard_true(i11) []
         jump(p0, p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected, expected2)
+        self.optimize_loop(ops, expected, expected2)
 
     def test_virtual_default_field(self):
         ops = """
@@ -888,7 +888,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_virtual_3(self):
         ops = """
@@ -904,7 +904,7 @@
         i1 = int_add(i, 1)
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_virtual_4(self):
         ops = """
@@ -931,7 +931,7 @@
         i3 = int_add(i0, i1)
         jump(i3, i2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_virtual_5(self):
         ops = """
@@ -961,7 +961,7 @@
         i3 = int_add(i0, i1)
         jump(i3, i2, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_virtual_constant_isnull(self):
         ops = """
@@ -980,7 +980,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
 
     def test_virtual_constant_isnonnull(self):
@@ -1000,7 +1000,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_nonvirtual_1(self):
         ops = """
@@ -1022,7 +1022,7 @@
         escape(p1)
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_nonvirtual_2(self):
         ops = """
@@ -1049,7 +1049,7 @@
         i2 = int_add(i1, i)
         jump(i, i2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_nonvirtual_later(self):
         ops = """
@@ -1071,7 +1071,7 @@
         i3 = int_add(i, i2)
         jump(i3)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_nonvirtual_dont_write_null_fields_on_force(self):
         ops = """
@@ -1091,7 +1091,7 @@
         i2 = getfield_gc(p1, descr=valuedescr)
         jump(i2)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_getfield_gc_pure_1(self):
         ops = """
@@ -1105,7 +1105,7 @@
         [i]
         jump(i)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_getfield_gc_pure_2(self):
         ops = """
@@ -1118,7 +1118,7 @@
         jump()
         """
         self.node.value = 5
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_getfield_gc_nonpure_2(self):
         ops = """
@@ -1131,7 +1131,7 @@
         [i]
         jump(i)
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_varray_1(self):
         ops = """
@@ -1148,7 +1148,7 @@
         [i1]
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_varray_alloc_and_set(self):
         ops = """
@@ -1166,7 +1166,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, 'Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_varray_float(self):
         ops = """
@@ -1183,7 +1183,7 @@
         [f1]
         jump(f1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_array_non_optimized(self):
         ops = """
@@ -1199,7 +1199,7 @@
         p1 = new_array(i1, descr=arraydescr)
         jump(i1, p1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_nonvirtual_array_dont_write_null_fields_on_force(self):
         ops = """
@@ -1217,7 +1217,7 @@
         escape(p1)
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_varray_2(self):
         ops = """
@@ -1245,7 +1245,7 @@
         guard_value(i3, 15) []
         jump(5)
         """
-        self.optimize_loop(ops, 'Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_p123_array(self):
         ops = """
@@ -1271,7 +1271,7 @@
         jump(i1, p1)
         """
         # We cannot track virtuals that survive for more than two iterations.
-        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_varray_forced_1(self):
         ops = """
@@ -1293,7 +1293,7 @@
         escape(i2)
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vstruct_1(self):
         ops = """
@@ -1315,7 +1315,7 @@
         escape(i1)
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_p123_vstruct(self):
         ops = """
@@ -1341,7 +1341,7 @@
         jump(i1, p1)
         """
         # We cannot track virtuals that survive for more than two iterations.
-        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_duplicate_getfield_1(self):
         ops = """
@@ -1366,7 +1366,7 @@
         escape(i2)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_getfield_after_setfield(self):
         ops = """
@@ -1382,7 +1382,7 @@
         escape(i1)
         jump(p1, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_setfield_of_different_type_does_not_clear(self):
         ops = """
@@ -1400,7 +1400,7 @@
         escape(i1)
         jump(p1, p2, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_setfield_of_same_type_clears(self):
         ops = """
@@ -1411,7 +1411,7 @@
         escape(i3)
         jump(p1, p2, i1, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_getfield_mergepoint_has_no_side_effects(self):
         ops = """
@@ -1431,7 +1431,7 @@
         escape(i1)
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getfield_ovf_op_does_not_clear(self):
         ops = """
@@ -1453,7 +1453,7 @@
         escape(i1)
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getfield_setarrayitem_does_not_clear(self):
         ops = """
@@ -1473,7 +1473,7 @@
         escape(i1)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getfield_constant(self):
         ops = """
@@ -1491,7 +1491,7 @@
         escape(i1)
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getfield_guard_value_const(self):
         ops = """
@@ -1510,7 +1510,7 @@
         escape(i1)
         jump()
         """
-        self.optimize_loop(ops, 'Constant(myptr)', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getfield_sideeffects_1(self):
         ops = """
@@ -1522,7 +1522,7 @@
         escape(i2)
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_getfield_sideeffects_2(self):
         ops = """
@@ -1533,7 +1533,7 @@
         escape(i2)
         jump(p1, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_setfield_1(self):
         ops = """
@@ -1547,7 +1547,7 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_2(self):
         ops = """
@@ -1564,7 +1564,7 @@
         escape(i1)
         jump(p1, i1, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_3(self):
         ops = """
@@ -1577,7 +1577,7 @@
         """
         # potential aliasing of p1 and p2 means that we cannot kill the
         # the setfield_gc
-        self.optimize_loop(ops, 'Not, Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_setfield_4(self):
         ops = """
@@ -1617,7 +1617,7 @@
         setfield_gc(p1, i4, descr=nextdescr)
         jump(p1, i1, i2, p3, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_duplicate_setfield_5(self):
         ops = """
@@ -1639,7 +1639,7 @@
         escape(i1)
         jump(p0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_sideeffects_1(self):
         ops = """
@@ -1649,7 +1649,7 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_setfield_residual_guard_1(self):
         ops = """
@@ -1675,7 +1675,7 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2, 1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_duplicate_setfield_residual_guard_2(self):
         # the difference with the previous test is that the field value is
@@ -1702,7 +1702,7 @@
         setfield_gc(p1, NULL, descr=nextdescr)
         jump(p1, i2, 1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_duplicate_setfield_residual_guard_3(self):
         ops = """
@@ -1728,7 +1728,7 @@
         setfield_gc(p1, NULL, descr=nextdescr)
         jump(p1, i2, 1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_residual_guard_4(self):
         # test that the setfield_gc does not end up between int_eq and
@@ -1751,7 +1751,7 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2, 5)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_duplicate_setfield_aliasing(self):
         # a case where aliasing issues (and not enough cleverness) mean
@@ -1763,7 +1763,7 @@
         setfield_gc(p1, i3, descr=valuedescr)
         jump(p1, p2, i1, i2, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_setfield_guard_value_const(self):
         ops = """
@@ -1778,7 +1778,7 @@
         setfield_gc(ConstPtr(myptr), i2, descr=valuedescr)
         jump(i1, i2)
         """
-        self.optimize_loop(ops, 'Constant(myptr), Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getarrayitem_1(self):
         ops = """
@@ -1803,7 +1803,7 @@
         escape(p3)
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getarrayitem_after_setarrayitem_1(self):
         ops = """
@@ -1819,7 +1819,7 @@
         escape(p2)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getarrayitem_after_setarrayitem_2(self):
         ops = """
@@ -1841,7 +1841,7 @@
         escape(p3)
         jump(p1, p2, p3, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getarrayitem_after_setarrayitem_3(self):
         ops = """
@@ -1868,7 +1868,7 @@
         escape(p4)
         jump(p1, p2, p3, p4, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_getarrayitem_pure_does_not_invalidate(self):
         ops = """
@@ -1889,7 +1889,7 @@
         escape(p3)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getarrayitem_after_setarrayitem_two_arrays(self):
         ops = """
@@ -1910,7 +1910,7 @@
         escape(p4)
         jump(p1, p2, p3, p4, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_virtual(self):
         ops = """
@@ -1939,7 +1939,7 @@
         setfield_gc(p1, p2, descr=nextdescr)
         jump(p1, i2, 1, p4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bug_1(self):
         ops = """
@@ -1961,7 +1961,8 @@
         p3 = escape()
         jump(i0, p3)
         """
-        self.optimize_loop(ops, 'Not, Virtual(node_vtable, nextdescr=Not)',
+        xxx # was Not, Virtual(node_vtable, nextdescr=Not)
+        self.optimize_loop(ops,
                            expected)
 
     def test_bug_2(self):
@@ -1984,7 +1985,8 @@
         p3 = escape()
         jump(i0, p3)
         """
-        self.optimize_loop(ops, 'Not, VArray(arraydescr2, Not)',
+        xxx # was Not, VArray(arraydescr2, Not)
+        self.optimize_loop(ops,
                            expected)
 
     def test_bug_3(self):
@@ -2030,7 +2032,7 @@
         p2a = new_with_vtable(ConstClass(node_vtable))
         jump(p2a, p3a)
         """
-        self.optimize_loop(ops, 'Virtual(node_vtable2, nextdescr=Not, otherdescr=Not)', expected)
+        self.optimize_loop(ops, expected) # XXX Virtual(node_vtable2, nextdescr=Not, otherdescr=Not)
 
     def test_bug_3bis(self):
         ops = """
@@ -2062,7 +2064,7 @@
         escape(p3a)
         jump(p2a, p3a)
         """
-        self.optimize_loop(ops, 'Virtual(node_vtable2, nextdescr=Not, otherdescr=Not)', expected)
+        self.optimize_loop(ops, expected) # XXX was Virtual(node_vtable2, nextdescr=Not, otherdescr=Not)
 
     def test_invalid_loop_1(self):
         ops = """
@@ -2117,7 +2119,7 @@
         i3 = int_add(i1, i2)
         jump(p2, i0, i1, i3, p2)
         """
-        self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected)
+        self.optimize_loop(ops, expected)
 
     def test_merge_guard_nonnull_guard_class(self):
         ops = """
@@ -2139,7 +2141,7 @@
         i3 = int_add(i1, i2)
         jump(p2, i0, i1, i3)
         """
-        self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected, preamble)
+        self.optimize_loop(ops, expected, preamble)
         self.check_expanded_fail_descr("i0", rop.GUARD_NONNULL_CLASS)
 
     def test_merge_guard_nonnull_guard_value(self):
@@ -2156,7 +2158,7 @@
         i3 = int_add(i1, i2)
         jump(p2, i0, i1, i3, p2)
         """
-        self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
 
     def test_merge_guard_nonnull_guard_class_guard_value(self):
@@ -2176,7 +2178,7 @@
         i4 = int_sub(i3, 1)
         jump(p2, i0, i1, i4, p2)
         """
-        self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
 
     def test_guard_class_oois(self):
@@ -2192,7 +2194,7 @@
         guard_class(p1, ConstClass(node_vtable2)) []
         jump(p1)
         """
-        self.optimize_loop(ops, "Not", expected)
+        self.optimize_loop(ops, expected)
 
     def test_oois_of_itself(self):
         ops = """
@@ -2210,7 +2212,7 @@
         p1 = getfield_gc(p0, descr=nextdescr)
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_duplicate_pure_op(self):
         ops = """
@@ -2240,7 +2242,7 @@
         guard_true(i1) []
         jump(p1, p2)
         """
-        self.optimize_loop(ops, "Not, Not", expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_duplicate_pure_op_with_descr(self):
         ops = """
@@ -2260,7 +2262,7 @@
         guard_true(i1) []
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_duplicate_pure_op_ovf(self):
         ops = """
@@ -2287,7 +2289,7 @@
         escape(i3)
         jump(i1)
         """
-        self.optimize_loop(ops, "Not", expected)
+        self.optimize_loop(ops, expected)
 
     def test_int_and_or_with_zero(self):
         ops = """
@@ -2302,7 +2304,7 @@
         [i0, i1]
         jump(i1, i0)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_fold_partially_constant_ops(self):
         ops = """
@@ -2314,7 +2316,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
         ops = """
         [i0]
@@ -2325,7 +2327,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
         ops = """
         [i0]
@@ -2336,7 +2338,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_fold_partially_constant_ops_ovf(self):
         ops = """
@@ -2349,7 +2351,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
         ops = """
         [i0]
@@ -2361,7 +2363,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
         ops = """
         [i0]
@@ -2373,7 +2375,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     # ----------
 
@@ -2501,7 +2503,7 @@
         guard_true(i1, descr=fdescr) [i3]
         jump(1, i3)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('15, i3', rop.GUARD_TRUE)
 
     def test_expand_fail_2(self):
@@ -2518,7 +2520,7 @@
         guard_true(i1, descr=fdescr) [i2]
         jump(1, i2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''ptr
             where ptr is a node_vtable, valuedescr=i2
             ''', rop.GUARD_TRUE)
@@ -2540,7 +2542,7 @@
         guard_true(i1, descr=fdescr) [i3, i2, p3]
         jump(i2, 1, i3, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''i3, p1
             where p1 is a node_vtable, valuedescr=1, nextdescr=p2
             where p2 is a node_vtable, valuedescr=i2, nextdescr=p3
@@ -2567,7 +2569,7 @@
             guard_true(i1, descr=fdescr) [i3, i2]
             jump(1, i2, i3)
             """
-            self.optimize_loop(ops % arg, 'Not, Not, Not', expected)
+            self.optimize_loop(ops % arg, expected)
             self.check_expanded_fail_descr('''i3, i3, %s
                 where p1 is a node_vtable, valuedescr=i2, nextdescr=p2
                 where p2 is a node_vtable, valuedescr=i2''' % arg,
@@ -2590,7 +2592,7 @@
         guard_true(i1, descr=fdescr) [i3, i4, i2]
         jump(i2, 1, i3, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''i3, i4, p1, p2
             where p1 is a node_vtable, valuedescr=i4, nextdescr=p2
             where p2 is a node_vtable, valuedescr=i2, nextdescr=p1
@@ -2609,6 +2611,7 @@
         guard_true(i0, descr=fdescr) [i1b]
         jump(i1, i1, i1)
         """
+        xxx
         self.optimize_loop(ops, '''Virtual(node_vtable, valuedescr=Not),
                                    Not, Not''', expected)
         self.check_expanded_fail_descr('''p0
@@ -2630,7 +2633,7 @@
         guard_true(i1, descr=fdescr) [i1]
         jump(1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''p1
             where p1 is a varray arraydescr: 25, i1
             ''', rop.GUARD_TRUE)
@@ -2651,7 +2654,7 @@
         guard_true(i1, descr=fdescr) [i1, p1]
         jump(1, p1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''p2
             where p2 is a vstruct ssize, adescr=i1, bdescr=p1
             ''', rop.GUARD_TRUE)
@@ -2680,6 +2683,7 @@
         guard_true(i1, descr=fdescr) [ia, iv, i2]
         jump(1, 1, i2, NULL, i2)
         """
+        xxx
         self.optimize_loop(ops, '''
             Not,
             VArray(arraydescr2,
@@ -2714,7 +2718,7 @@
         setfield_gc(p1, NULL, descr=nextdescr)
         jump(p1, i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.loop.inputargs[0].value = self.nodebox.value
         self.check_expanded_fail_descr('''
             p1.nextdescr = p2
@@ -2739,7 +2743,7 @@
         setfield_gc(ConstPtr(myptr), NULL, descr=nextdescr)
         jump(i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''
             ConstPtr(myptr).nextdescr = p2
             where p2 is a node_vtable, valuedescr=i2
@@ -2766,7 +2770,7 @@
         escape(i1)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidate_some_caches(self):
         ops = """
@@ -2794,7 +2798,7 @@
         escape(i2)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidate_arrays(self):
         ops = """
@@ -2821,7 +2825,7 @@
         escape(p4)
         jump(p1, p2, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidate_some_arrays(self):
         ops = """
@@ -2856,7 +2860,7 @@
         escape(i4)
         jump(p1, p2, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidates_some_read_caches_1(self):
         ops = """
@@ -2876,7 +2880,7 @@
         setfield_gc(p2, i3, descr=adescr)
         jump(p1, i1, p2, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidates_some_read_caches_2(self):
         ops = """
@@ -2896,7 +2900,7 @@
         setfield_gc(p2, i3, descr=adescr)
         jump(p1, i1, p2, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidates_some_read_caches_3(self):
         ops = """
@@ -2908,7 +2912,7 @@
         setfield_gc(p2, i3, descr=adescr)
         jump(p1, i1, p2, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_call_assembler_invalidates_caches(self):
         ops = '''
@@ -2918,7 +2922,7 @@
         setfield_gc(p1, i3, descr=valuedescr)
         jump(p1, i3)
         '''
-        self.optimize_loop(ops, 'Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_call_pure_invalidates_caches(self):
         # CALL_PURE should still force the setfield_gc() to occur before it
@@ -2936,7 +2940,7 @@
         setfield_gc(p1, i3, descr=valuedescr)
         jump(p1, i3)
         '''
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_call_pure_constant_folding(self):
         # CALL_PURE is not marked as is_always_pure(), because it is wrong
@@ -2959,7 +2963,7 @@
         i4 = call(123456, 4, i0, 6, descr=plaincalldescr)
         jump(i0, 42, i4)
         '''
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vref_nonvirtual_nonescape(self):
         ops = """
@@ -2973,7 +2977,7 @@
         i0 = force_token()
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vref_nonvirtual_escape(self):
         ops = """
@@ -2996,7 +3000,7 @@
         """
         # XXX we should optimize a bit more the case of a nonvirtual.
         # in theory it is enough to just do 'p2 = p1'.
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vref_virtual_1(self):
         ops = """
@@ -3036,7 +3040,7 @@
         setfield_gc(p2, -3, descr=virtualtokendescr)
         jump(p0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vref_virtual_2(self):
         ops = """
@@ -3078,7 +3082,7 @@
         """
         # the point of this test is that 'i1' should show up in the fail_args
         # of 'guard_not_forced', because it was stored in the virtual 'p1b'.
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''p2, p1
             where p1 is a node_vtable, nextdescr=p1b
             where p1b is a node_vtable, valuedescr=i1
@@ -3109,7 +3113,7 @@
         setfield_gc(p0, NULL, descr=refdescr)
         jump(p0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         # the fail_args contain [i3, i1, p0]:
         #  - i3 is from the virtual expansion of p2
         #  - i1 is from the virtual expansion of p1
@@ -3147,7 +3151,7 @@
         guard_not_forced() []
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vref_nonvirtual_and_lazy_setfield(self):
         ops = """
@@ -3172,7 +3176,7 @@
         guard_not_forced() [i1]
         jump(i1, p1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_arraycopy_1(self):
         ops = '''
@@ -3189,7 +3193,7 @@
         []
         jump()
         '''
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_arraycopy_2(self):
         ops = '''
@@ -3206,7 +3210,7 @@
         []
         jump()
         '''
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_arraycopy_not_virtual(self):
         ops = '''
@@ -3226,7 +3230,7 @@
         escape(p2)
         jump()
         '''
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_arraycopy_no_elem(self):
         """ this was actually observed in the wild
@@ -3241,7 +3245,7 @@
         [p1]
         jump(p1)
         '''
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt(self):
         ops = """
@@ -3258,7 +3262,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_noguard(self):
         ops = """
@@ -3273,7 +3277,7 @@
         i2 = int_lt(i0, 5)
         jump(i2)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_noopt(self):
         ops = """
@@ -3292,7 +3296,7 @@
         guard_true(i2) []
         jump(4)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_rev(self):
         ops = """
@@ -3309,7 +3313,7 @@
         guard_false(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_tripple(self):
         ops = """
@@ -3328,7 +3332,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_add(self):
         ops = """
@@ -3347,7 +3351,7 @@
         i2 = int_add(i0, 10)
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_add_before(self):
         ops = """
@@ -3366,7 +3370,7 @@
         guard_true(i3) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_add_ovf(self):
         ops = """
@@ -3386,7 +3390,7 @@
         i2 = int_add(i0, 10)
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_add_ovf_before(self):
         ops = """
@@ -3407,7 +3411,7 @@
         guard_true(i3) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_sub(self):
         ops = """
@@ -3426,7 +3430,7 @@
         i2 = int_sub(i0, 10)
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_sub_before(self):
         ops = """
@@ -3445,7 +3449,7 @@
         guard_true(i3) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_ltle(self):
         ops = """
@@ -3462,7 +3466,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lelt(self):
         ops = """
@@ -3479,7 +3483,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_gt(self):
         ops = """
@@ -3496,7 +3500,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_gtge(self):
         ops = """
@@ -3513,7 +3517,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_gegt(self):
         ops = """
@@ -3530,7 +3534,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_ovf(self):
         ops = """
@@ -3552,7 +3556,7 @@
         i3 = int_add(i0, 1)
         jump(i3)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_arraylen(self):
         ops = """
@@ -3572,7 +3576,7 @@
         setarrayitem_gc(p0, 0, p1)
         jump(i0, p0)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_strlen(self):
         ops = """
@@ -3588,7 +3592,7 @@
         i0 = strlen(p0)
         jump(p0)
         """
-        self.optimize_strunicode_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_addsub_const(self):
         ops = """
@@ -3605,7 +3609,7 @@
         i4 = int_mul(i0, i1)
         jump(i4)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_addsub_int(self):
         ops = """
@@ -3622,7 +3626,7 @@
         i4 = int_add(i0, i1)
         jump(i4, i10)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_addsub_int2(self):
         ops = """
@@ -3639,7 +3643,7 @@
         i4 = int_add(i0, i1)
         jump(i4, i10)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_framestackdepth_overhead(self):
         ops = """
@@ -3665,7 +3669,7 @@
         setfield_gc(p0, i1, descr=valuedescr)
         jump(p0, i22, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_setgetfield_raw(self):
         ops = """
@@ -3684,7 +3688,7 @@
         setfield_raw(p7, i33, descr=nextdescr)        
         jump(p4, p7, i30, i33)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_pure(self):
         ops = """
@@ -3701,7 +3705,7 @@
 
         """
         self.node.value = 5
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_addsub_ovf(self):
         ops = """
@@ -3719,7 +3723,7 @@
         i2 = int_sub(i1, 5)
         jump(i2)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_subadd_ovf(self):
         ops = """
@@ -3737,7 +3741,7 @@
         i2 = int_add(i1, 5)
         jump(i2)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_and(self):
         ops = """
@@ -3782,7 +3786,7 @@
         guard_true(i15) []
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_subsub_ovf(self):
         ops = """
@@ -3806,7 +3810,7 @@
         i3 = int_sub(1, i0)
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_eq(self):
         ops = """
@@ -3831,7 +3835,7 @@
         guard_true(i3) []
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_eq_const(self):
         ops = """
@@ -3851,7 +3855,7 @@
         escape(10)
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_eq_const_not(self):
         ops = """
@@ -3869,7 +3873,7 @@
         jump(i2)
 
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_ne_const(self):
         ops = """
@@ -3886,7 +3890,7 @@
         jump(10)
 
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_ne_const_not(self):
         ops = """
@@ -3903,7 +3907,7 @@
         i2 = int_add(i0, 3)
         jump(i2)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_ltne(self):
         ops = """
@@ -3920,7 +3924,7 @@
         guard_true(i2) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lege_const(self):
         ops = """
@@ -3941,7 +3945,7 @@
         jump(10)
 
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_mul_ovf(self):
         ops = """
@@ -3973,7 +3977,7 @@
         guard_true(i8) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_mul_ovf_before(self):
         ops = """
@@ -4002,7 +4006,7 @@
         guard_true(i5) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_sub_ovf_before(self):
         ops = """
@@ -4031,15 +4035,14 @@
         guard_true(i5) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     # ----------
-    def optimize_strunicode_loop(self, ops, spectext, optops):
+    def optimize_strunicode_loop(self, ops, optops):
         # check with the arguments passed in
-        self.optimize_loop(ops, spectext, optops)
+        self.optimize_loop(ops, optops)
         # check with replacing 'str' with 'unicode' everywhere
         self.optimize_loop(ops.replace('str','unicode').replace('s"', 'u"'),
-                           spectext,
                            optops.replace('str','unicode').replace('s"', 'u"'))
 
     def test_newstr_1(self):
@@ -4054,7 +4057,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_strunicode_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_newstr_2(self):
         ops = """
@@ -4070,7 +4073,7 @@
         [i0, i1]
         jump(i1, i0)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_1(self):
         ops = """
@@ -4091,7 +4094,7 @@
         copystrcontent(p2, p3, 0, i4, i5)
         jump(p2, p3)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_vstr2_str(self):
         ops = """
@@ -4114,7 +4117,7 @@
         copystrcontent(p2, p3, 0, 2, i4)
         jump(i1, i0, p3)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_str_vstr2(self):
         ops = """
@@ -4138,7 +4141,7 @@
         i6 = int_add(i5, 1)      # will be killed by the backend
         jump(i1, i0, p3)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_str_str_str(self):
         ops = """
@@ -4165,7 +4168,7 @@
         copystrcontent(p3, p5, 0, i12b, i3b)
         jump(p2, p3, p5)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_str_cstr1(self):
         ops = """
@@ -4184,7 +4187,7 @@
         i5 = int_add(i4, 1)      # will be killed by the backend
         jump(p3)
         """
-        self.optimize_strunicode_loop(ops, 'Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_consts(self):
         ops = """
@@ -4200,7 +4203,7 @@
         escape(s"abcde")
         jump()
         """
-        self.optimize_strunicode_loop(ops, '', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_1(self):
         ops = """
@@ -4215,7 +4218,7 @@
         copystrcontent(p1, p2, i1, 0, i3)
         jump(p2, i1, i2)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_2(self):
         ops = """
@@ -4229,7 +4232,7 @@
         copystrcontent(p1, p2, 0, 0, i2)
         jump(p2, i2)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_3(self):
         ops = """
@@ -4247,7 +4250,7 @@
         copystrcontent(p1, p3, i6, 0, i5)
         jump(p3, i1, i2, i3, i4)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_getitem1(self):
         ops = """
@@ -4265,7 +4268,7 @@
         escape(i4)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_plain(self):
         ops = """
@@ -4283,7 +4286,7 @@
         escape(i4)
         jump(i3, i4)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_concat(self):
         ops = """
@@ -4304,10 +4307,10 @@
         copystrcontent(p2, p4, 0, i3, i4b)
         jump(p4, i1, i2, p2)
         """
-        self.optimize_strunicode_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     # ----------
-    def optimize_strunicode_loop_extradescrs(self, ops, spectext, optops):
+    def optimize_strunicode_loop_extradescrs(self, ops, optops):
         from pypy.jit.metainterp.optimizeopt import string
         def my_callinfo_for_oopspec(oopspecindex):
             calldescrtype = type(LLtypeMixin.strequaldescr)
@@ -4322,7 +4325,7 @@
         saved = string.callinfo_for_oopspec
         try:
             string.callinfo_for_oopspec = my_callinfo_for_oopspec
-            self.optimize_strunicode_loop(ops, spectext, optops)
+            self.optimize_strunicode_loop(ops, optops)
         finally:
             string.callinfo_for_oopspec = saved
 
@@ -4333,7 +4336,7 @@
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', ops)
+        self.optimize_strunicode_loop_extradescrs(ops, ops)
 
     def test_str_equal_noop2(self):
         ops = """
@@ -4358,7 +4361,7 @@
         escape(i0)
         jump(p1, p2, p3)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not',
+        self.optimize_strunicode_loop_extradescrs(ops,
                                                   expected)
 
     def test_str_equal_slice1(self):
@@ -4376,7 +4379,7 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+        self.optimize_strunicode_loop_extradescrs(ops,
                                                   expected)
 
     def test_str_equal_slice2(self):
@@ -4394,7 +4397,7 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+        self.optimize_strunicode_loop_extradescrs(ops,
                                                   expected)
 
     def test_str_equal_slice3(self):
@@ -4414,7 +4417,7 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+        self.optimize_strunicode_loop_extradescrs(ops,
                                                   expected)
 
     def test_str_equal_slice4(self):
@@ -4432,7 +4435,7 @@
         escape(i0)
         jump(p1, i1, i2)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not',
+        self.optimize_strunicode_loop_extradescrs(ops,
                                                   expected)
 
     def test_str_equal_slice5(self):
@@ -4452,7 +4455,7 @@
         escape(i0)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not, Not, Not',
+        self.optimize_strunicode_loop_extradescrs(ops,
                                                   expected)
 
     def test_str_equal_none1(self):
@@ -4468,7 +4471,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_none2(self):
         ops = """
@@ -4483,7 +4486,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull1(self):
         ops = """
@@ -4500,7 +4503,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull2(self):
         ops = """
@@ -4518,7 +4521,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull3(self):
         ops = """
@@ -4535,7 +4538,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull4(self):
         ops = """
@@ -4560,7 +4563,7 @@
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars0(self):
         ops = """
@@ -4575,7 +4578,7 @@
         escape(1)
         jump(i1)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars1(self):
         ops = """
@@ -4592,7 +4595,7 @@
         escape(i0)
         jump(i1)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars2(self):
         ops = """
@@ -4613,7 +4616,7 @@
         escape(i0)
         jump(i1, i2)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not, Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars3(self):
         ops = """
@@ -4628,7 +4631,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_lengthmismatch1(self):
         ops = """
@@ -4644,7 +4647,7 @@
         escape(0)
         jump(i1)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str2unicode_constant(self):
         ops = """
@@ -4658,7 +4661,7 @@
         escape(u"xy")
         jump()
         """
-        self.optimize_strunicode_loop_extradescrs(ops, '', expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str2unicode_nonconstant(self):
         ops = """
@@ -4667,7 +4670,7 @@
         escape(p1)
         jump(p1)
         """
-        self.optimize_strunicode_loop_extradescrs(ops, 'Not', ops)
+        self.optimize_strunicode_loop_extradescrs(ops, ops)
         # more generally, supporting non-constant but virtual cases is
         # not obvious, because of the exception UnicodeDecodeError that
         # can be raised by ll_str2unicode()
@@ -4686,7 +4689,7 @@
 ##        [i0]
 ##        jump(1)
 ##        """
-##        self.optimize_loop(ops, 'Not', expected)
+##        self.optimize_loop(ops, expected)
 
 ##    def test_instanceof_guard_class(self):
 ##        ops = """
@@ -4700,4 +4703,4 @@
 ##        guard_class(p0, ConstClass(node_vtable)) []
 ##        jump(1, p0)
 ##        """
-##        self.optimize_loop(ops, 'Not, Not', expected)
+##        self.optimize_loop(ops, expected)


From cfbolz at codespeak.net  Sat Oct 30 18:23:42 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 18:23:42 +0200 (CEST)
Subject: [pypy-svn] r78544 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030162342.D2732282C00@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 18:23:40 2010
New Revision: 78544

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
those just work


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 18:23:40 2010
@@ -1961,9 +1961,7 @@
         p3 = escape()
         jump(i0, p3)
         """
-        xxx # was Not, Virtual(node_vtable, nextdescr=Not)
-        self.optimize_loop(ops,
-                           expected)
+        self.optimize_loop(ops, expected)
 
     def test_bug_2(self):
         ops = """
@@ -1985,9 +1983,7 @@
         p3 = escape()
         jump(i0, p3)
         """
-        xxx # was Not, VArray(arraydescr2, Not)
-        self.optimize_loop(ops,
-                           expected)
+        self.optimize_loop(ops, expected)
 
     def test_bug_3(self):
         ops = """


From cfbolz at codespeak.net  Sat Oct 30 18:26:58 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 18:26:58 +0200 (CEST)
Subject: [pypy-svn] r78545 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030162658.0A1BC36E0CB@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 18:26:57 2010
New Revision: 78545

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py
Log:
fix import


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_resume.py	Sat Oct 30 18:26:57 2010
@@ -1,6 +1,7 @@
 import py
 from pypy.rpython.lltypesystem import lltype, llmemory, rffi
-from pypy.jit.metainterp.optimizeopt.virtualize import VirtualValue, OptValue, VArrayValue
+from pypy.jit.metainterp.optimizeopt.optimizer import OptValue
+from pypy.jit.metainterp.optimizeopt.virtualize import VirtualValue, VArrayValue
 from pypy.jit.metainterp.optimizeopt.virtualize import VStructValue
 from pypy.jit.metainterp.resume import *
 from pypy.jit.metainterp.history import BoxInt, BoxPtr, ConstInt


From arigo at codespeak.net  Sat Oct 30 18:32:08 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 18:32:08 +0200 (CEST)
Subject: [pypy-svn] r78546 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101030163208.7FF04282C27@codespeak.net>

Author: arigo
Date: Sat Oct 30 18:32:05 2010
New Revision: 78546

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
Log:
Remove the 'spectext' argument from test_optimizebasic too.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Sat Oct 30 18:32:05 2010
@@ -1,4 +1,4 @@
-from pypy.jit.metainterp.history import Const, ConstInt
+from pypy.jit.metainterp.history import Const, ConstInt, BoxInt
 from pypy.jit.metainterp.resoperation import rop, ResOperation
 from pypy.jit.metainterp.optimizeutil import _findall, sort_descrs
 from pypy.jit.metainterp.optimizeutil import descrlist_dict

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	Sat Oct 30 18:32:05 2010
@@ -243,7 +243,7 @@
         assert equaloplists(optimized.operations,
                             expected.operations, False, remap)
 
-    def optimize_loop(self, ops, spectext, optops):
+    def optimize_loop(self, ops, optops):
         loop = self.parse(ops)
         #
         self.loop = loop
@@ -285,7 +285,7 @@
         guard_value(i0, 0) [i0]
         jump(1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constant_propagate(self):
         ops = """
@@ -302,7 +302,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constant_propagate_ovf(self):
         ops = """
@@ -320,7 +320,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constfold_all(self):
         from pypy.jit.backend.llgraph.llimpl import TYPES     # xxx fish
@@ -354,7 +354,7 @@
             escape(%d)
             jump()
             """ % expected_value
-            self.optimize_loop(ops, '', expected)
+            self.optimize_loop(ops, expected)
 
     # ----------
 
@@ -370,7 +370,7 @@
         guard_class(p0, ConstClass(node_vtable)) []
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_guard_class_2(self):
         ops = """
@@ -386,7 +386,7 @@
         escape(p0)
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_guard_class_constant(self):
         ops = """
@@ -399,7 +399,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constant_boolrewrite_lt(self):
         ops = """
@@ -416,7 +416,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constant_boolrewrite_gt(self):
         ops = """
@@ -433,7 +433,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constant_boolrewrite_reflex(self):
         ops = """
@@ -450,7 +450,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constant_boolrewrite_reflex_invers(self):
         ops = """
@@ -467,7 +467,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_consecutive_guard_value_constfold(self):
         ops = """
@@ -487,7 +487,7 @@
         escape(3)
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_guard_value_if_constant(self):
         ops = """
@@ -514,7 +514,7 @@
         guard_class(p0, ConstClass(node_vtable)) []
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_int_is_true_1(self):
         ops = """
@@ -531,7 +531,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_int_is_true_is_zero(self):
         py.test.skip("XXX implement me")
@@ -549,7 +549,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_ooisnull_oononnull_2(self):
         ops = """
@@ -563,7 +563,7 @@
         guard_nonnull(p0) []
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_ooisnull_on_null_ptr_1(self):
         ops = """
@@ -579,7 +579,7 @@
         guard_isnull(p0) []
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_ooisnull_oononnull_via_virtual(self):
         ops = """
@@ -596,7 +596,7 @@
         guard_nonnull(p0) []
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_oois_1(self):
         ops = """
@@ -617,7 +617,7 @@
         guard_class(p0, ConstClass(node_vtable)) []
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_nonnull_1(self):
         ops = """
@@ -639,7 +639,7 @@
         setfield_gc(p0, 5, descr=valuedescr)
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_const_guard_value(self):
         ops = """
@@ -652,7 +652,7 @@
         []
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_constptr_guard_value(self):
         ops = """
@@ -661,7 +661,7 @@
         guard_value(p1, ConstPtr(myptr)) []
         jump()
         """
-        self.optimize_loop(ops, '', ops)
+        self.optimize_loop(ops, ops)
 
     def test_guard_value_to_guard_true(self):
         ops = """
@@ -676,7 +676,7 @@
         guard_true(i1) [i]
         jump(i)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_guard_value_to_guard_false(self):
         ops = """
@@ -691,7 +691,7 @@
         guard_false(i1) [i]
         jump(i)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_guard_value_on_nonbool(self):
         ops = """
@@ -706,7 +706,7 @@
         guard_value(i1, 0) [i]
         jump(-3)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_int_is_true_of_bool(self):
         ops = """
@@ -723,7 +723,7 @@
         guard_false(i2) [i0, i1]
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
 
 
@@ -738,7 +738,7 @@
         jump(i1, p1, p2)
         """
         # We cannot track virtuals that survive for more than two iterations.
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_p123_nested(self):
         ops = """
@@ -754,7 +754,7 @@
         """
         # The same as test_p123_simple, but with a virtual containing another
         # virtual.
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_p123_anti_nested(self):
         ops = """
@@ -770,7 +770,7 @@
         """
         # The same as test_p123_simple, but in the end the "old" p2 contains
         # a "young" virtual p2sub.  Make sure it is all forced.
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     # ----------
 
@@ -794,7 +794,7 @@
         i3 = call(i2, descr=nonwritedescr)
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     # ----------
 
@@ -819,7 +819,7 @@
         guard_value(i2, 1) []
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
 
     # ----------
@@ -927,7 +927,7 @@
         guard_true(i11) []
         jump(p0, p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected2)
+        self.optimize_loop(ops, expected2)
 
     def test_virtual_default_field(self):
         ops = """
@@ -963,7 +963,7 @@
         i1 = int_add(i, 1)
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_virtual_4(self):
         ops = """
@@ -1027,7 +1027,7 @@
         [i0]
         jump(1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
 
     def test_virtual_constant_isnonnull(self):
@@ -1043,7 +1043,7 @@
         [i0]
         jump(0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_nonvirtual_1(self):
         ops = """
@@ -1065,7 +1065,7 @@
         escape(p1)
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_nonvirtual_2(self):
         ops = """
@@ -1078,7 +1078,7 @@
         jump(i, p1)
         """
         expected = ops
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_nonvirtual_later(self):
         ops = """
@@ -1100,7 +1100,7 @@
         i3 = int_add(i, i2)
         jump(i3)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_nonvirtual_dont_write_null_fields_on_force(self):
         ops = """
@@ -1120,7 +1120,7 @@
         i2 = getfield_gc(p1, descr=valuedescr)
         jump(i2)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_getfield_gc_pure_1(self):
         ops = """
@@ -1134,7 +1134,7 @@
         [i]
         jump(i)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_getfield_gc_pure_2(self):
         ops = """
@@ -1147,7 +1147,7 @@
         jump(5)
         """
         self.node.value = 5
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_getfield_gc_nonpure_2(self):
         ops = """
@@ -1156,7 +1156,7 @@
         jump(i1)
         """
         expected = ops
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_varray_1(self):
         ops = """
@@ -1173,7 +1173,7 @@
         [i1]
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_varray_alloc_and_set(self):
         ops = """
@@ -1187,7 +1187,7 @@
         [i1]
         jump(0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_varray_float(self):
         ops = """
@@ -1204,7 +1204,7 @@
         [f1]
         jump(f1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_array_non_optimized(self):
         ops = """
@@ -1220,7 +1220,7 @@
         p1 = new_array(i1, descr=arraydescr)
         jump(i1, p1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_nonvirtual_array_dont_write_null_fields_on_force(self):
         ops = """
@@ -1238,7 +1238,7 @@
         escape(p1)
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_varray_2(self):
         ops = """
@@ -1271,7 +1271,7 @@
         jump(i1, p1, p2)
         """
         # We cannot track virtuals that survive for more than two iterations.
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_varray_forced_1(self):
         ops = """
@@ -1293,7 +1293,7 @@
         escape(i2)
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vstruct_1(self):
         ops = """
@@ -1322,7 +1322,7 @@
         jump(i1, p1, p2)
         """
         # We cannot track virtuals that survive for more than two iterations.
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_getfield_1(self):
         ops = """
@@ -1347,7 +1347,7 @@
         escape(i2)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_getfield_after_setfield(self):
         ops = """
@@ -1363,7 +1363,7 @@
         escape(i1)
         jump(p1, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_setfield_of_different_type_does_not_clear(self):
         ops = """
@@ -1381,7 +1381,7 @@
         escape(i1)
         jump(p1, p2, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_setfield_of_same_type_clears(self):
         ops = """
@@ -1392,7 +1392,7 @@
         escape(i3)
         jump(p1, p2, i1, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_getfield_mergepoint_has_no_side_effects(self):
         ops = """
@@ -1412,7 +1412,7 @@
         escape(i1)
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getfield_ovf_op_does_not_clear(self):
         ops = """
@@ -1434,7 +1434,7 @@
         escape(i1)
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getfield_setarrayitem_does_not_clear(self):
         ops = """
@@ -1454,7 +1454,7 @@
         escape(i1)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getfield_constant(self):
         ops = """
@@ -1472,7 +1472,7 @@
         escape(i1)
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getfield_guard_value_const(self):
         ops = """
@@ -1504,7 +1504,7 @@
         escape(i2)
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_getfield_sideeffects_2(self):
         ops = """
@@ -1515,7 +1515,7 @@
         escape(i2)
         jump(p1, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_setfield_1(self):
         ops = """
@@ -1529,7 +1529,7 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_2(self):
         ops = """
@@ -1546,7 +1546,7 @@
         escape(i1)
         jump(p1, i1, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_3(self):
         ops = """
@@ -1559,7 +1559,7 @@
         """
         # potential aliasing of p1 and p2 means that we cannot kill the
         # the setfield_gc
-        self.optimize_loop(ops, 'Not, Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_setfield_4(self):
         ops = """
@@ -1588,7 +1588,7 @@
         setfield_gc(p1, i4, descr=nextdescr)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_5(self):
         ops = """
@@ -1610,7 +1610,7 @@
         escape(i1)
         jump(p0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_sideeffects_1(self):
         ops = """
@@ -1620,7 +1620,7 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_setfield_residual_guard_1(self):
         ops = """
@@ -1631,7 +1631,7 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_setfield_residual_guard_2(self):
         # the difference with the previous test is that the field value is
@@ -1652,7 +1652,7 @@
         setfield_gc(p1, NULL, descr=nextdescr)
         jump(p1, i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_residual_guard_3(self):
         ops = """
@@ -1672,7 +1672,7 @@
         setfield_gc(p1, NULL, descr=nextdescr)
         jump(p1, i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_setfield_residual_guard_4(self):
         # test that the setfield_gc does not end up between int_eq and
@@ -1686,7 +1686,7 @@
         setfield_gc(p1, i2, descr=valuedescr)
         jump(p1, i1, i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_setfield_aliasing(self):
         # a case where aliasing issues (and not enough cleverness) mean
@@ -1698,7 +1698,7 @@
         setfield_gc(p1, i3, descr=valuedescr)
         jump(p1, p2, i1, i2, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_duplicate_setfield_guard_value_const(self):
         ops = """
@@ -1739,7 +1739,7 @@
         escape(p3)
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getarrayitem_after_setarrayitem_1(self):
         ops = """
@@ -1755,7 +1755,7 @@
         escape(p2)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getarrayitem_after_setarrayitem_2(self):
         ops = """
@@ -1777,7 +1777,7 @@
         escape(p3)
         jump(p1, p2, p3, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getarrayitem_after_setarrayitem_3(self):
         ops = """
@@ -1804,7 +1804,7 @@
         escape(p4)
         jump(p1, p2, p3, p4, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_getarrayitem_pure_does_not_invalidate(self):
         ops = """
@@ -1825,7 +1825,7 @@
         escape(p3)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_duplicate_getarrayitem_after_setarrayitem_two_arrays(self):
         ops = """
@@ -1846,7 +1846,7 @@
         escape(p4)
         jump(p1, p2, p3, p4, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bug_1(self):
         ops = """
@@ -2017,7 +2017,7 @@
         i3 = int_add(i1, i2)
         jump(p2, i0, i1, i3, p2)
         """
-        self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected)
+        self.optimize_loop(ops, expected)
 
     def test_merge_guard_nonnull_guard_class(self):
         self.make_fail_descr()
@@ -2034,7 +2034,7 @@
         i3 = int_add(i1, i2)
         jump(p2, i0, i1, i3, p2)
         """
-        self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr("i0", rop.GUARD_NONNULL_CLASS)
 
     def test_merge_guard_nonnull_guard_value(self):
@@ -2052,7 +2052,7 @@
         i3 = int_add(i1, i2)
         jump(p2, i0, i1, i3, p2)
         """
-        self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
 
     def test_merge_guard_nonnull_guard_class_guard_value(self):
@@ -2073,7 +2073,7 @@
         i4 = int_sub(i3, 1)
         jump(p2, i0, i1, i4, p2)
         """
-        self.optimize_loop(ops, "Not, Not, Not, Not, Not", expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
 
     def test_guard_class_oois(self):
@@ -2089,7 +2089,7 @@
         guard_class(p1, ConstClass(node_vtable2)) []
         jump(p1)
         """
-        self.optimize_loop(ops, "Not", expected)
+        self.optimize_loop(ops, expected)
 
     def test_oois_of_itself(self):
         ops = """
@@ -2107,7 +2107,7 @@
         p1 = getfield_gc(p0, descr=nextdescr)
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_duplicate_pure_op(self):
         ops = """
@@ -2137,7 +2137,7 @@
         guard_true(i1) []
         jump(p1, p2)
         """
-        self.optimize_loop(ops, "Not, Not", expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_duplicate_pure_op_with_descr(self):
         ops = """
@@ -2157,7 +2157,7 @@
         guard_true(i1) []
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_remove_duplicate_pure_op_ovf(self):
         ops = """
@@ -2184,7 +2184,7 @@
         escape(i3)
         jump(i1)
         """
-        self.optimize_loop(ops, "Not", expected)
+        self.optimize_loop(ops, expected)
 
     def test_int_and_or_with_zero(self):
         ops = """
@@ -2199,7 +2199,7 @@
         [i0, i1]
         jump(i1, i0)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_fold_partially_constant_ops(self):
         ops = """
@@ -2211,7 +2211,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
         ops = """
         [i0]
@@ -2222,7 +2222,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
         ops = """
         [i0]
@@ -2233,7 +2233,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_fold_partially_constant_ops_ovf(self):
         ops = """
@@ -2246,7 +2246,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
         ops = """
         [i0]
@@ -2258,7 +2258,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
         ops = """
         [i0]
@@ -2270,7 +2270,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     # ----------
 
@@ -2416,7 +2416,7 @@
         guard_true(i1, descr=fdescr) [i3]
         jump(1, i3)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('15, i3', rop.GUARD_TRUE)
 
     def test_expand_fail_2(self):
@@ -2434,7 +2434,7 @@
         guard_true(i1, descr=fdescr) [i2]
         jump(1, i2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''ptr
             where ptr is a node_vtable, valuedescr=i2
             ''', rop.GUARD_TRUE)
@@ -2457,7 +2457,7 @@
         guard_true(i1, descr=fdescr) [i3, i2, p3]
         jump(i2, 1, i3, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''i3, p1
             where p1 is a node_vtable, valuedescr=1, nextdescr=p2
             where p2 is a node_vtable, valuedescr=i2, nextdescr=p3
@@ -2484,7 +2484,7 @@
             guard_true(i1, descr=fdescr) [i3, i2]
             jump(1, i2, i3)
             """
-            self.optimize_loop(ops % arg, 'Not, Not, Not', expected)
+            self.optimize_loop(ops % arg, expected)
             self.check_expanded_fail_descr('''i3, i3, %s
                 where p1 is a node_vtable, valuedescr=i2, nextdescr=p2
                 where p2 is a node_vtable, valuedescr=i2''' % arg,
@@ -2508,7 +2508,7 @@
         guard_true(i1, descr=fdescr) [i3, i4, i2]
         jump(i2, 1, i3, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''i3, i4, p1, p2
             where p1 is a node_vtable, valuedescr=i4, nextdescr=p2
             where p2 is a node_vtable, valuedescr=i2, nextdescr=p1
@@ -2551,7 +2551,7 @@
         guard_true(i1, descr=fdescr) [i1]
         jump(1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''p1
             where p1 is a varray arraydescr: 25, i1
             ''', rop.GUARD_TRUE)
@@ -2573,7 +2573,7 @@
         guard_true(i1, descr=fdescr) [i1, p1]
         jump(1, p1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''p2
             where p2 is a vstruct ssize, adescr=i1, bdescr=p1
             ''', rop.GUARD_TRUE)
@@ -2639,7 +2639,7 @@
         setfield_gc(p1, NULL, descr=nextdescr)
         jump(p1, i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.loop.inputargs[0].value = self.nodebox.value
         self.check_expanded_fail_descr('''
             p1.nextdescr = p2
@@ -2665,7 +2665,7 @@
         setfield_gc(ConstPtr(myptr), NULL, descr=nextdescr)
         jump(i2, i4)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''
             ConstPtr(myptr).nextdescr = p2
             where p2 is a node_vtable, valuedescr=i2
@@ -2692,7 +2692,7 @@
         escape(i1)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidate_some_caches(self):
         ops = """
@@ -2720,7 +2720,7 @@
         escape(i2)
         jump(p1, p2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidate_arrays(self):
         ops = """
@@ -2747,7 +2747,7 @@
         escape(p4)
         jump(p1, p2, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidate_some_arrays(self):
         ops = """
@@ -2782,7 +2782,7 @@
         escape(i4)
         jump(p1, p2, i1)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidates_some_read_caches_1(self):
         ops = """
@@ -2802,7 +2802,7 @@
         setfield_gc(p2, i3, descr=adescr)
         jump(p1, i1, p2, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidates_some_read_caches_2(self):
         ops = """
@@ -2822,7 +2822,7 @@
         setfield_gc(p2, i3, descr=adescr)
         jump(p1, i1, p2, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_residual_call_invalidates_some_read_caches_3(self):
         ops = """
@@ -2834,7 +2834,7 @@
         setfield_gc(p2, i3, descr=adescr)
         jump(p1, i1, p2, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_call_assembler_invalidates_caches(self):
         ops = '''
@@ -2844,7 +2844,7 @@
         setfield_gc(p1, i3, descr=valuedescr)
         jump(p1, i3)
         '''
-        self.optimize_loop(ops, 'Not, Not', ops)
+        self.optimize_loop(ops, ops)
 
     def test_call_pure_invalidates_caches(self):
         # CALL_PURE should still force the setfield_gc() to occur before it
@@ -2862,7 +2862,7 @@
         setfield_gc(p1, i3, descr=valuedescr)
         jump(p1, i3)
         '''
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_call_pure_constant_folding(self):
         # CALL_PURE is not marked as is_always_pure(), because it is wrong
@@ -2885,7 +2885,7 @@
         i4 = call(123456, 4, i0, 6, descr=plaincalldescr)
         jump(i0, 42, i4)
         '''
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vref_nonvirtual_nonescape(self):
         ops = """
@@ -2899,7 +2899,7 @@
         i0 = force_token()
         jump(p1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vref_nonvirtual_escape(self):
         ops = """
@@ -2922,7 +2922,7 @@
         """
         # XXX we should optimize a bit more the case of a nonvirtual.
         # in theory it is enough to just do 'p2 = p1'.
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vref_virtual_1(self):
         ops = """
@@ -2962,7 +2962,7 @@
         setfield_gc(p2, -3, descr=virtualtokendescr)
         jump(p0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vref_virtual_2(self):
         self.make_fail_descr()
@@ -3005,7 +3005,7 @@
         """
         # the point of this test is that 'i1' should show up in the fail_args
         # of 'guard_not_forced', because it was stored in the virtual 'p1b'.
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         self.check_expanded_fail_descr('''p2, p1
             where p1 is a node_vtable, nextdescr=p1b
             where p1b is a node_vtable, valuedescr=i1
@@ -3037,7 +3037,7 @@
         setfield_gc(p0, NULL, descr=refdescr)
         jump(p0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
         # the fail_args contain [i3, i1, p0]:
         #  - i3 is from the virtual expansion of p2
         #  - i1 is from the virtual expansion of p1
@@ -3076,7 +3076,7 @@
         guard_not_forced() []
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_vref_nonvirtual_and_lazy_setfield(self):
         self.make_fail_descr()
@@ -3102,7 +3102,7 @@
         guard_not_forced() [i1]
         jump(i1, p1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_arraycopy_1(self):
         ops = '''
@@ -3119,7 +3119,7 @@
         [i0]
         jump(1)
         '''
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_arraycopy_2(self):
         ops = '''
@@ -3136,7 +3136,7 @@
         [i0]
         jump(3)
         '''
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_arraycopy_not_virtual(self):
         ops = '''
@@ -3154,7 +3154,7 @@
         setarrayitem_gc(p2, 2, 10, descr=arraydescr)
         jump(p2)
         '''
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_arraycopy_no_elem(self):
         """ this was actually observed in the wild
@@ -3169,7 +3169,7 @@
         [p1]
         jump(p1)
         '''
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt(self):
         ops = """
@@ -3186,7 +3186,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_noguard(self):
         ops = """
@@ -3201,7 +3201,7 @@
         i2 = int_lt(i0, 5)
         jump(i2)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_noopt(self):
         ops = """
@@ -3220,7 +3220,7 @@
         guard_true(i2) []
         jump(4)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_rev(self):
         ops = """
@@ -3237,7 +3237,7 @@
         guard_false(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_tripple(self):
         ops = """
@@ -3256,7 +3256,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_add(self):
         ops = """
@@ -3275,7 +3275,7 @@
         i2 = int_add(i0, 10)
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_add_before(self):
         ops = """
@@ -3294,7 +3294,7 @@
         guard_true(i3) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_add_ovf(self):
         ops = """
@@ -3314,7 +3314,7 @@
         i2 = int_add(i0, 10)
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_add_ovf_before(self):
         ops = """
@@ -3335,7 +3335,7 @@
         guard_true(i3) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_sub(self):
         ops = """
@@ -3354,7 +3354,7 @@
         i2 = int_sub(i0, 10)
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lt_sub_before(self):
         ops = """
@@ -3373,7 +3373,7 @@
         guard_true(i3) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_ltle(self):
         ops = """
@@ -3390,7 +3390,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lelt(self):
         ops = """
@@ -3407,7 +3407,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_gt(self):
         ops = """
@@ -3424,7 +3424,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_gtge(self):
         ops = """
@@ -3441,7 +3441,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_gegt(self):
         ops = """
@@ -3458,7 +3458,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_ovf(self):
         ops = """
@@ -3480,7 +3480,7 @@
         i3 = int_add(i0, 1)
         jump(i3)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_arraylen(self):
         ops = """
@@ -3500,7 +3500,7 @@
         setarrayitem_gc(p0, 0, p1)
         jump(i0, p0)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_strlen(self):
         ops = """
@@ -3516,7 +3516,7 @@
         i0 = strlen(p0)
         jump(p0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_addsub_const(self):
         ops = """
@@ -3533,7 +3533,7 @@
         i4 = int_mul(i0, i1)
         jump(i4)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_addsub_int(self):
         ops = """
@@ -3550,7 +3550,7 @@
         i4 = int_add(i0, i1)
         jump(i4, i10)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_addsub_int2(self):
         ops = """
@@ -3567,7 +3567,7 @@
         i4 = int_add(i0, i1)
         jump(i4, i10)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_framestackdepth_overhead(self):
         ops = """
@@ -3597,7 +3597,7 @@
         setfield_gc(p0, i1, descr=valuedescr)
         jump(p0, i22)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_addsub_ovf(self):
         ops = """
@@ -3615,7 +3615,7 @@
         i2 = int_sub(i1, 5)
         jump(i2)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_subadd_ovf(self):
         ops = """
@@ -3633,7 +3633,7 @@
         i2 = int_add(i1, 5)
         jump(i2)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_and(self):
         ops = """
@@ -3678,7 +3678,7 @@
         guard_true(i15) []
         jump(i1)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_subsub_ovf(self):
         ops = """
@@ -3702,7 +3702,7 @@
         i3 = int_sub(1, i0)
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_eq(self):
         ops = """
@@ -3723,7 +3723,7 @@
         guard_true(i3) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_eq_const(self):
         ops = """
@@ -3740,7 +3740,7 @@
         jump(10)
 
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_eq_const_not(self):
         ops = """
@@ -3758,7 +3758,7 @@
         jump(i2)
 
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_ne_const(self):
         ops = """
@@ -3775,7 +3775,7 @@
         jump(10)
 
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_ne_const_not(self):
         ops = """
@@ -3792,7 +3792,7 @@
         i2 = int_add(i0, 3)
         jump(i2)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_ltne(self):
         ops = """
@@ -3809,7 +3809,7 @@
         guard_true(i2) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_bound_lege_const(self):
         ops = """
@@ -3830,7 +3830,7 @@
         jump(10)
 
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_mul_ovf(self):
         ops = """
@@ -3862,7 +3862,7 @@
         guard_true(i8) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_mul_ovf_before(self):
         ops = """
@@ -3891,7 +3891,7 @@
         guard_true(i5) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_sub_ovf_before(self):
         ops = """
@@ -3920,7 +3920,7 @@
         guard_true(i5) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_newstr_1(self):
         ops = """
@@ -3934,7 +3934,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_newstr_2(self):
         ops = """
@@ -3950,7 +3950,7 @@
         [i0, i1]
         jump(i1, i0)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_concat_1(self):
         ops = """
@@ -3971,7 +3971,7 @@
         copystrcontent(p2, p3, 0, i4, i5)
         jump(p2, p3)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_concat_vstr2_str(self):
         ops = """
@@ -3994,7 +3994,7 @@
         copystrcontent(p2, p3, 0, 2, i4)
         jump(i1, i0, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_concat_str_vstr2(self):
         ops = """
@@ -4018,7 +4018,7 @@
         i6 = int_add(i5, 1)      # will be killed by the backend
         jump(i1, i0, p3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_concat_str_str_str(self):
         ops = """
@@ -4045,7 +4045,7 @@
         copystrcontent(p3, p5, 0, i12b, i3b)
         jump(p2, p3, p5)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_concat_str_cstr1(self):
         ops = """
@@ -4064,7 +4064,7 @@
         i5 = int_add(i4, 1)      # will be killed by the backend
         jump(p3)
         """
-        self.optimize_loop(ops, 'Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_concat_consts(self):
         ops = """
@@ -4080,7 +4080,7 @@
         escape("abcde")
         jump()
         """
-        self.optimize_loop(ops, '', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_slice_1(self):
         ops = """
@@ -4095,7 +4095,7 @@
         copystrcontent(p1, p2, i1, 0, i3)
         jump(p2, i1, i2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_slice_2(self):
         ops = """
@@ -4109,7 +4109,7 @@
         copystrcontent(p1, p2, 0, 0, i2)
         jump(p2, i2)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_slice_3(self):
         ops = """
@@ -4127,7 +4127,7 @@
         copystrcontent(p1, p3, i6, 0, i5)
         jump(p3, i1, i2, i3, i4)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_slice_getitem1(self):
         ops = """
@@ -4145,7 +4145,7 @@
         escape(i4)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_slice_plain(self):
         ops = """
@@ -4163,7 +4163,7 @@
         escape(i4)
         jump(i3, i4)
         """
-        self.optimize_loop(ops, 'Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     def test_str_slice_concat(self):
         ops = """
@@ -4184,10 +4184,10 @@
         copystrcontent(p2, p4, 0, i3, i4b)
         jump(p4, i1, i2, p2)
         """
-        self.optimize_loop(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop(ops, expected)
 
     # ----------
-    def optimize_loop_extradescrs(self, ops, spectext, optops):
+    def optimize_loop_extradescrs(self, ops, optops):
         from pypy.jit.metainterp.optimizeopt import string
         def my_callinfo_for_oopspec(oopspecindex):
             calldescrtype = type(LLtypeMixin.strequaldescr)
@@ -4202,7 +4202,7 @@
         saved = string.callinfo_for_oopspec
         try:
             string.callinfo_for_oopspec = my_callinfo_for_oopspec
-            self.optimize_loop(ops, spectext, optops)
+            self.optimize_loop(ops, optops)
         finally:
             string.callinfo_for_oopspec = saved
 
@@ -4213,7 +4213,7 @@
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', ops)
+        self.optimize_loop_extradescrs(ops, ops)
 
     def test_str_equal_noop2(self):
         ops = """
@@ -4238,7 +4238,7 @@
         escape(i0)
         jump(p1, p2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_slice1(self):
         ops = """
@@ -4255,7 +4255,7 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_slice2(self):
         ops = """
@@ -4272,7 +4272,7 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_slice3(self):
         ops = """
@@ -4291,7 +4291,7 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_slice4(self):
         ops = """
@@ -4308,7 +4308,7 @@
         escape(i0)
         jump(p1, i1, i2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_slice5(self):
         ops = """
@@ -4327,7 +4327,7 @@
         escape(i0)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not, Not, Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_none1(self):
         ops = """
@@ -4342,7 +4342,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_none2(self):
         ops = """
@@ -4357,7 +4357,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull1(self):
         ops = """
@@ -4374,7 +4374,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull2(self):
         ops = """
@@ -4392,7 +4392,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull3(self):
         ops = """
@@ -4409,7 +4409,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull4(self):
         ops = """
@@ -4434,7 +4434,7 @@
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars0(self):
         ops = """
@@ -4449,7 +4449,7 @@
         escape(1)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars1(self):
         ops = """
@@ -4466,7 +4466,7 @@
         escape(i0)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars2(self):
         ops = """
@@ -4487,7 +4487,7 @@
         escape(i0)
         jump(i1, i2)
         """
-        self.optimize_loop_extradescrs(ops, 'Not, Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars3(self):
         ops = """
@@ -4502,7 +4502,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     def test_str_equal_lengthmismatch1(self):
         ops = """
@@ -4518,7 +4518,7 @@
         escape(0)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, 'Not', expected)
+        self.optimize_loop_extradescrs(ops, expected)
 
     # XXX unicode operations
     # XXX str2unicode
@@ -4537,7 +4537,7 @@
 ##        [i0]
 ##        jump(1)
 ##        """
-##        self.optimize_loop(ops, 'Not', expected)
+##        self.optimize_loop(ops, expected)
 
 ##    def test_instanceof_guard_class(self):
 ##        ops = """
@@ -4551,4 +4551,4 @@
 ##        guard_class(p0, ConstClass(node_vtable)) []
 ##        jump(1, p0)
 ##        """
-##        self.optimize_loop(ops, 'Not, Not', expected)
+##        self.optimize_loop(ops, expected)


From cfbolz at codespeak.net  Sat Oct 30 18:32:13 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 18:32:13 +0200 (CEST)
Subject: [pypy-svn] r78547 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030163213.894AD282C28@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 18:32:11 2010
New Revision: 78547

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fix a test


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 18:32:11 2010
@@ -2109,13 +2109,19 @@
         guard_value(p1, ConstPtr(myptr)) [i1]
         jump(p2, i0, i1, i3, p2)
         """
-        expected = """
+        preamble = """
         [p1, i0, i1, i2, p2]
         guard_value(p1, ConstPtr(myptr)) [i0]
         i3 = int_add(i1, i2)
-        jump(p2, i0, i1, i3, p2)
+        jump(p2, i0, i1, i3)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [p2, i0, i1, i2]
+        guard_value(p2, ConstPtr(myptr)) [i0]
+        i3 = int_add(i1, i2)
+        jump(ConstPtr(myptr), i0, i1, i3)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_merge_guard_nonnull_guard_class(self):
         ops = """


From cfbolz at codespeak.net  Sat Oct 30 18:33:42 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 18:33:42 +0200 (CEST)
Subject: [pypy-svn] r78548 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030163342.737D4282BF2@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 18:33:39 2010
New Revision: 78548

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
next


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 18:33:39 2010
@@ -2154,13 +2154,19 @@
         guard_value(p1, ConstPtr(myptr)) [i1]
         jump(p2, i0, i1, i3, p2)
         """
-        expected = """
+        preamble = """
         [p1, i0, i1, i2, p2]
         guard_value(p1, ConstPtr(myptr), descr=fdescr) [i0]
         i3 = int_add(i1, i2)
-        jump(p2, i0, i1, i3, p2)
+        jump(p2, i0, i1, i3)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [p2, i0, i1, i2]
+        guard_value(p2, ConstPtr(myptr), descr=fdescr2) [i0]
+        i3 = int_add(i1, i2)
+        jump(ConstPtr(myptr), i0, i1, i3)
+        """
+        self.optimize_loop(ops, expected, preamble)
         self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
 
     def test_merge_guard_nonnull_guard_class_guard_value(self):


From cfbolz at codespeak.net  Sat Oct 30 18:35:06 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 18:35:06 +0200 (CEST)
Subject: [pypy-svn] r78549 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030163506.D66CE282C28@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 18:35:04 2010
New Revision: 78549

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
everything repeats


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 18:35:04 2010
@@ -2179,14 +2179,21 @@
         guard_value(p1, ConstPtr(myptr)) [i1]
         jump(p2, i0, i1, i4, p2)
         """
-        expected = """
+        preamble = """
         [p1, i0, i1, i2, p2]
         guard_value(p1, ConstPtr(myptr), descr=fdescr) [i0]
         i3 = int_add(i1, i2)
         i4 = int_sub(i3, 1)
-        jump(p2, i0, i1, i4, p2)
+        jump(p2, i0, i1, i4)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [p2, i0, i1, i2]
+        guard_value(p2, ConstPtr(myptr), descr=fdescr2) [i0]
+        i3 = int_add(i1, i2)
+        i4 = int_sub(i3, 1)
+        jump(ConstPtr(myptr), i0, i1, i4)
+        """
+        self.optimize_loop(ops, expected, preamble)
         self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
 
     def test_guard_class_oois(self):


From cfbolz at codespeak.net  Sat Oct 30 18:36:45 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 18:36:45 +0200 (CEST)
Subject: [pypy-svn] r78550 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030163645.D7692282C2B@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 18:36:42 2010
New Revision: 78550

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
everything  repeats


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 18:36:42 2010
@@ -2204,12 +2204,16 @@
         guard_true(i) []
         jump(p1)
         """
-        expected = """
+        preamble = """
         [p1]
         guard_class(p1, ConstClass(node_vtable2)) []
         jump(p1)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [p1]
+        jump(p1)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_oois_of_itself(self):
         ops = """


From cfbolz at codespeak.net  Sat Oct 30 18:38:12 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 18:38:12 +0200 (CEST)
Subject: [pypy-svn] r78551 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030163812.3455A36E3FB@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 18:38:09 2010
New Revision: 78551

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
nothing is left


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 18:38:09 2010
@@ -2226,12 +2226,16 @@
         guard_false(i2) []
         jump(p0)
         """
-        expected = """
+        preamble = """
         [p0]
         p1 = getfield_gc(p0, descr=nextdescr)
         jump(p0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [p0]
+        jump(p0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_remove_duplicate_pure_op(self):
         ops = """


From arigo at codespeak.net  Sat Oct 30 18:45:14 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 18:45:14 +0200 (CEST)
Subject: [pypy-svn] r78552 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030164514.D2D08282C2C@codespeak.net>

Author: arigo
Date: Sat Oct 30 18:45:05 2010
New Revision: 78552

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeutil.py
Log:
Manually merge the changes done to test_optimizefindnode
and test_optimizeopt in trunk.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	Sat Oct 30 18:45:05 2010
@@ -806,10 +806,10 @@
         guard_value(i2, 1) []
         i3 = call_loopinvariant(1, i1, descr=nonwritedescr)
         guard_no_exception() []
-        guard_value(i2, 1) []
+        guard_value(i3, 1) []
         i4 = call_loopinvariant(1, i1, descr=nonwritedescr)
         guard_no_exception() []
-        guard_value(i2, 1) []
+        guard_value(i4, 1) []
         jump(i1)
         """
         expected = """
@@ -3516,7 +3516,7 @@
         i0 = strlen(p0)
         jump(p0)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_addsub_const(self):
         ops = """
@@ -3922,6 +3922,14 @@
         """
         self.optimize_loop(ops, expected)
 
+    # ----------
+    def optimize_strunicode_loop(self, ops, optops):
+        # check with the arguments passed in
+        self.optimize_loop(ops, optops)
+        # check with replacing 'str' with 'unicode' everywhere
+        self.optimize_loop(ops.replace('str','unicode').replace('s"', 'u"'),
+                           optops.replace('str','unicode').replace('s"', 'u"'))
+
     def test_newstr_1(self):
         ops = """
         [i0]
@@ -3934,7 +3942,7 @@
         [i0]
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_newstr_2(self):
         ops = """
@@ -3950,7 +3958,7 @@
         [i0, i1]
         jump(i1, i0)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_1(self):
         ops = """
@@ -3971,7 +3979,7 @@
         copystrcontent(p2, p3, 0, i4, i5)
         jump(p2, p3)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_vstr2_str(self):
         ops = """
@@ -3994,7 +4002,7 @@
         copystrcontent(p2, p3, 0, 2, i4)
         jump(i1, i0, p3)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_str_vstr2(self):
         ops = """
@@ -4018,7 +4026,7 @@
         i6 = int_add(i5, 1)      # will be killed by the backend
         jump(i1, i0, p3)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_str_str_str(self):
         ops = """
@@ -4045,12 +4053,12 @@
         copystrcontent(p3, p5, 0, i12b, i3b)
         jump(p2, p3, p5)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_str_cstr1(self):
         ops = """
         [p2]
-        p3 = call(0, p2, "x", descr=strconcatdescr)
+        p3 = call(0, p2, s"x", descr=strconcatdescr)
         jump(p3)
         """
         expected = """
@@ -4064,28 +4072,28 @@
         i5 = int_add(i4, 1)      # will be killed by the backend
         jump(p3)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_concat_consts(self):
         ops = """
         []
-        p1 = same_as("ab")
-        p2 = same_as("cde")
+        p1 = same_as(s"ab")
+        p2 = same_as(s"cde")
         p3 = call(0, p1, p2, descr=strconcatdescr)
         escape(p3)
         jump()
         """
         expected = """
         []
-        escape("abcde")
+        escape(s"abcde")
         jump()
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_1(self):
         ops = """
         [p1, i1, i2]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
         jump(p2, i1, i2)
         """
         expected = """
@@ -4095,12 +4103,12 @@
         copystrcontent(p1, p2, i1, 0, i3)
         jump(p2, i1, i2)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_2(self):
         ops = """
         [p1, i2]
-        p2 = call(0, p1, 0, i2, descr=slicedescr)
+        p2 = call(0, p1, 0, i2, descr=strslicedescr)
         jump(p2, i2)
         """
         expected = """
@@ -4109,13 +4117,13 @@
         copystrcontent(p1, p2, 0, 0, i2)
         jump(p2, i2)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_3(self):
         ops = """
         [p1, i1, i2, i3, i4]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
-        p3 = call(0, p2, i3, i4, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
+        p3 = call(0, p2, i3, i4, descr=strslicedescr)
         jump(p3, i1, i2, i3, i4)
         """
         expected = """
@@ -4127,12 +4135,12 @@
         copystrcontent(p1, p3, i6, 0, i5)
         jump(p3, i1, i2, i3, i4)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_getitem1(self):
         ops = """
         [p1, i1, i2, i3]
-        p2 = call(0, p1, i1, i2, descr=slicedescr)
+        p2 = call(0, p1, i1, i2, descr=strslicedescr)
         i4 = strgetitem(p2, i3)
         escape(i4)
         jump(p1, i1, i2, i3)
@@ -4145,7 +4153,7 @@
         escape(i4)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_plain(self):
         ops = """
@@ -4153,7 +4161,7 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i3)
         strsetitem(p1, 1, i4)
-        p2 = call(0, p1, 1, 2, descr=slicedescr)
+        p2 = call(0, p1, 1, 2, descr=strslicedescr)
         i5 = strgetitem(p2, 0)
         escape(i5)
         jump(i3, i4)
@@ -4163,12 +4171,12 @@
         escape(i4)
         jump(i3, i4)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     def test_str_slice_concat(self):
         ops = """
         [p1, i1, i2, p2]
-        p3 = call(0, p1, i1, i2, descr=slicedescr)
+        p3 = call(0, p1, i1, i2, descr=strslicedescr)
         p4 = call(0, p3, p2, descr=strconcatdescr)
         jump(p4, i1, i2, p2)
         """
@@ -4184,10 +4192,10 @@
         copystrcontent(p2, p4, 0, i3, i4b)
         jump(p4, i1, i2, p2)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected)
 
     # ----------
-    def optimize_loop_extradescrs(self, ops, optops):
+    def optimize_strunicode_loop_extradescrs(self, ops, optops):
         from pypy.jit.metainterp.optimizeopt import string
         def my_callinfo_for_oopspec(oopspecindex):
             calldescrtype = type(LLtypeMixin.strequaldescr)
@@ -4202,7 +4210,7 @@
         saved = string.callinfo_for_oopspec
         try:
             string.callinfo_for_oopspec = my_callinfo_for_oopspec
-            self.optimize_loop(ops, optops)
+            self.optimize_strunicode_loop(ops, optops)
         finally:
             string.callinfo_for_oopspec = saved
 
@@ -4213,7 +4221,7 @@
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_loop_extradescrs(ops, ops)
+        self.optimize_strunicode_loop_extradescrs(ops, ops)
 
     def test_str_equal_noop2(self):
         ops = """
@@ -4238,12 +4246,12 @@
         escape(i0)
         jump(p1, p2, p3)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_slice1(self):
         ops = """
         [p1, i1, i2, p3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p4, p3, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4255,12 +4263,12 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_slice2(self):
         ops = """
         [p1, i1, i2, p3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p3, p4, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4272,13 +4280,13 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_slice3(self):
         ops = """
         [p1, i1, i2, p3]
         guard_nonnull(p3) []
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         i0 = call(0, p3, p4, descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2, p3)
@@ -4291,13 +4299,13 @@
         escape(i0)
         jump(p1, i1, i2, p3)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_slice4(self):
         ops = """
         [p1, i1, i2]
-        p3 = call(0, p1, i1, i2, descr=slicedescr)
-        i0 = call(0, p3, "x", descr=strequaldescr)
+        p3 = call(0, p1, i1, i2, descr=strslicedescr)
+        i0 = call(0, p3, s"x", descr=strequaldescr)
         escape(i0)
         jump(p1, i1, i2)
         """
@@ -4308,12 +4316,12 @@
         escape(i0)
         jump(p1, i1, i2)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_slice5(self):
         ops = """
         [p1, i1, i2, i3]
-        p4 = call(0, p1, i1, i2, descr=slicedescr)
+        p4 = call(0, p1, i1, i2, descr=strslicedescr)
         p5 = newstr(1)
         strsetitem(p5, 0, i3)
         i0 = call(0, p5, p4, descr=strequaldescr)
@@ -4327,7 +4335,7 @@
         escape(i0)
         jump(p1, i1, i2, i3)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_none1(self):
         ops = """
@@ -4342,7 +4350,7 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_none2(self):
         ops = """
@@ -4357,30 +4365,30 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull1(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "hello world", descr=strequaldescr)
+        i0 = call(0, p1, s"hello world", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
         expected = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "hello world", descr=streq_nonnull_descr)
+        i0 = call(0, p1, s"hello world", descr=streq_nonnull_descr)
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull2(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "", descr=strequaldescr)
+        i0 = call(0, p1, s"", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4392,13 +4400,13 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull3(self):
         ops = """
         [p1]
         guard_nonnull(p1) []
-        i0 = call(0, p1, "x", descr=strequaldescr)
+        i0 = call(0, p1, s"x", descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4409,13 +4417,13 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_nonnull4(self):
         ops = """
         [p1, p2]
         p4 = call(0, p1, p2, descr=strconcatdescr)
-        i0 = call(0, "hello world", p4, descr=strequaldescr)
+        i0 = call(0, s"hello world", p4, descr=strequaldescr)
         escape(i0)
         jump(p1, p2)
         """
@@ -4430,17 +4438,17 @@
         i5 = strlen(p2)
         i6 = int_add(i4, i5)      # will be killed by the backend
         copystrcontent(p2, p4, 0, i4, i5)
-        i0 = call(0, "hello world", p4, descr=streq_nonnull_descr)
+        i0 = call(0, s"hello world", p4, descr=streq_nonnull_descr)
         escape(i0)
         jump(p1, p2)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars0(self):
         ops = """
         [i1]
         p1 = newstr(0)
-        i0 = call(0, p1, "", descr=strequaldescr)
+        i0 = call(0, p1, s"", descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4449,14 +4457,14 @@
         escape(1)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars1(self):
         ops = """
         [i1]
         p1 = newstr(1)
         strsetitem(p1, 0, i1)
-        i0 = call(0, p1, "x", descr=strequaldescr)
+        i0 = call(0, p1, s"x", descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4466,7 +4474,7 @@
         escape(i0)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars2(self):
         ops = """
@@ -4474,7 +4482,7 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i1)
         strsetitem(p1, 1, i2)
-        i0 = call(0, p1, "xy", descr=strequaldescr)
+        i0 = call(0, p1, s"xy", descr=strequaldescr)
         escape(i0)
         jump(i1, i2)
         """
@@ -4483,16 +4491,16 @@
         p1 = newstr(2)
         strsetitem(p1, 0, i1)
         strsetitem(p1, 1, i2)
-        i0 = call(0, p1, "xy", descr=streq_lengthok_descr)
+        i0 = call(0, p1, s"xy", descr=streq_lengthok_descr)
         escape(i0)
         jump(i1, i2)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_chars3(self):
         ops = """
         [p1]
-        i0 = call(0, "x", p1, descr=strequaldescr)
+        i0 = call(0, s"x", p1, descr=strequaldescr)
         escape(i0)
         jump(p1)
         """
@@ -4502,14 +4510,14 @@
         escape(i0)
         jump(p1)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
     def test_str_equal_lengthmismatch1(self):
         ops = """
         [i1]
         p1 = newstr(1)
         strsetitem(p1, 0, i1)
-        i0 = call(0, "xy", p1, descr=strequaldescr)
+        i0 = call(0, s"xy", p1, descr=strequaldescr)
         escape(i0)
         jump(i1)
         """
@@ -4518,10 +4526,33 @@
         escape(0)
         jump(i1)
         """
-        self.optimize_loop_extradescrs(ops, expected)
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
 
-    # XXX unicode operations
-    # XXX str2unicode
+    def test_str2unicode_constant(self):
+        ops = """
+        []
+        p0 = call(0, "xy", descr=s2u_descr)      # string -> unicode
+        escape(p0)
+        jump()
+        """
+        expected = """
+        []
+        escape(u"xy")
+        jump()
+        """
+        self.optimize_strunicode_loop_extradescrs(ops, expected)
+
+    def test_str2unicode_nonconstant(self):
+        ops = """
+        [p0]
+        p1 = call(0, p0, descr=s2u_descr)      # string -> unicode
+        escape(p1)
+        jump(p1)
+        """
+        self.optimize_strunicode_loop_extradescrs(ops, ops)
+        # more generally, supporting non-constant but virtual cases is
+        # not obvious, because of the exception UnicodeDecodeError that
+        # can be raised by ll_str2unicode()
 
 
 ##class TestOOtype(BaseTestOptimizeBasic, OOtypeMixin):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeutil.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeutil.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeutil.py	Sat Oct 30 18:45:05 2010
@@ -110,33 +110,32 @@
                             EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE))
     arraycopydescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
                  EffectInfo([], [], [], oopspecindex=EffectInfo.OS_ARRAYCOPY))
-    strconcatdescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_CONCAT))
-    slicedescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_SLICE))
-    strequaldescr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR_EQUAL))
-    streq_slice_checknull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_CHECKNULL))
-    streq_slice_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_NONNULL))
-    streq_slice_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_SLICE_CHAR))
-    streq_nonnull_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_NONNULL))
-    streq_nonnull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_NONNULL_CHAR))
-    streq_checknull_char_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_CHECKNULL_CHAR))
-    streq_lengthok_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
-                 EffectInfo([], [], [],
-                     oopspecindex=EffectInfo.OS_STREQ_LENGTHOK))
+
+    for _name, _os in [
+        ('strconcatdescr',               'OS_STR_CONCAT'),
+        ('strslicedescr',                'OS_STR_SLICE'),
+        ('strequaldescr',                'OS_STR_EQUAL'),
+        ('streq_slice_checknull_descr',  'OS_STREQ_SLICE_CHECKNULL'),
+        ('streq_slice_nonnull_descr',    'OS_STREQ_SLICE_NONNULL'),
+        ('streq_slice_char_descr',       'OS_STREQ_SLICE_CHAR'),
+        ('streq_nonnull_descr',          'OS_STREQ_NONNULL'),
+        ('streq_nonnull_char_descr',     'OS_STREQ_NONNULL_CHAR'),
+        ('streq_checknull_char_descr',   'OS_STREQ_CHECKNULL_CHAR'),
+        ('streq_lengthok_descr',         'OS_STREQ_LENGTHOK'),
+        ]:
+        _oopspecindex = getattr(EffectInfo, _os)
+        locals()[_name] = \
+            cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=_oopspecindex))
+        #
+        _oopspecindex = getattr(EffectInfo, _os.replace('STR', 'UNI'))
+        locals()[_name.replace('str', 'unicode')] = \
+            cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=_oopspecindex))
+
+    s2u_descr = cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT,
+                EffectInfo([], [], [], oopspecindex=EffectInfo.OS_STR2UNICODE))
+    #
 
     class LoopToken(AbstractDescr):
         pass


From hakanardo at codespeak.net  Sat Oct 30 18:56:00 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 18:56:00 +0200 (CEST)
Subject: [pypy-svn] r78553 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030165600.488F136E3FF@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 18:55:58 2010
New Revision: 78553

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(armin, cfbolz, hakanardo) fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 18:55:58 2010
@@ -2007,28 +2007,32 @@
         """
         preamble = """
         [p1]
-        guard_nonnull(p1) []
-        guard_class(p1, ConstClass(node_vtable2)) []
+        guard_nonnull_class(p1, ConstClass(node_vtable2)) []
         p2 = getfield_gc(p1, descr=nextdescr)
         guard_class(p2, ConstClass(node_vtable)) []
         p3 = getfield_gc(p1, descr=otherdescr)
         guard_class(p3, ConstClass(node_vtable)) []
+        setfield_gc(p3, p2, descr=otherdescr)
         p3a = new_with_vtable(ConstClass(node_vtable))
         escape(p3a)
-        setfield_gc(p3, p2, descr=otherdescr)
-        jump(p2a, p3a)
+        jump(p3a)
         """
         expected = """
-        [p2, p3]
-        guard_class(p2, ConstClass(node_vtable)) []
-        guard_class(p3, ConstClass(node_vtable)) []
-        setfield_gc(p3, p2, descr=otherdescr)
-        p3a = new_with_vtable(ConstClass(node_vtable))
-        escape(p3a)
-        p2a = new_with_vtable(ConstClass(node_vtable))
-        jump(p2a, p3a)
+        [p3a]
+        # p1=p1a(next=p2a, other=p3a), p2()
+        # p2 = getfield_gc(p1, descr=nextdescr) # p2a
+        # p3 = getfield_gc(p1, descr=otherdescr)# p3a
+        # setfield_gc(p3, p2, descr=otherdescr) # p3a.other = p2a
+        # p1a = new_with_vtable(ConstClass(node_vtable2))
+        # p2a = new_with_vtable(ConstClass(node_vtable))
+        p2 = new_with_vtable(ConstClass(node_vtable))
+        setfield_gc(p3a, p2, descr=otherdescr) # p3a.other = p2a
+        p3anew = new_with_vtable(ConstClass(node_vtable))
+        escape(p3anew)
+        jump(p3anew)
         """
-        self.optimize_loop(ops, expected) # XXX Virtual(node_vtable2, nextdescr=Not, otherdescr=Not)
+        #self.optimize_loop(ops, expected) # XXX Virtual(node_vtable2, nextdescr=Not, otherdescr=Not)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bug_3bis(self):
         ops = """


From hakanardo at codespeak.net  Sat Oct 30 19:03:28 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 19:03:28 +0200 (CEST)
Subject: [pypy-svn] r78554 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030170328.2747C282C2E@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 19:03:26 2010
New Revision: 78554

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(armin, hakanardo) fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 19:03:26 2010
@@ -2054,17 +2054,31 @@
         setfield_gc(p1a, p3a, descr=otherdescr)
         jump(p1a)
         """
-        expected = """
-        [p2, p3]
+        preamble = """
+        [p1]
+        guard_nonnull_class(p1, ConstClass(node_vtable2)) []
+        p2 = getfield_gc(p1, descr=nextdescr)
         guard_class(p2, ConstClass(node_vtable)) []
+        p3 = getfield_gc(p1, descr=otherdescr)
         guard_class(p3, ConstClass(node_vtable)) []
+        # p1a = new_with_vtable(ConstClass(node_vtable2))
+        p2a = new_with_vtable(ConstClass(node_vtable))
+        setfield_gc(p3, p2a, descr=otherdescr)
+        p3a = new_with_vtable(ConstClass(node_vtable))
+        escape(p3a)
+        # setfield_gc(p1a, p2a, descr=nextdescr)
+        # setfield_gc(p1a, p3a, descr=otherdescr)
+        jump(p2a, p3a)
+        """
+        expected = """
+        [p2, p3]
         p2a = new_with_vtable(ConstClass(node_vtable))
         setfield_gc(p3, p2a, descr=otherdescr)
         p3a = new_with_vtable(ConstClass(node_vtable))
         escape(p3a)
         jump(p2a, p3a)
         """
-        self.optimize_loop(ops, expected) # XXX was Virtual(node_vtable2, nextdescr=Not, otherdescr=Not)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_invalid_loop_1(self):
         ops = """


From cfbolz at codespeak.net  Sat Oct 30 19:07:33 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 19:07:33 +0200 (CEST)
Subject: [pypy-svn] r78555 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030170733.1B4E936E3BC@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 19:07:31 2010
New Revision: 78555

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
kill tests that are about expanding the fail args. they have nearly nothing to
do with unrolling and still exist in optimizebasic


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 19:07:31 2010
@@ -2527,179 +2527,6 @@
         boxes = reader.consume_boxes()
         self._verify_fail_args(boxes, fdescr.oparse, expectedtext)
 
-    def test_expand_fail_1(self):
-        ops = """
-        [i1, i3]
-        # first rename i3 into i4
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, i3, descr=valuedescr)
-        i4 = getfield_gc(p1, descr=valuedescr)
-        #
-        i2 = int_add(10, 5)
-        guard_true(i1, descr=fdescr) [i2, i4]
-        jump(i1, i4)
-        """
-        expected = """
-        [i1, i3]
-        guard_true(i1, descr=fdescr) [i3]
-        jump(1, i3)
-        """
-        self.optimize_loop(ops, expected)
-        self.check_expanded_fail_descr('15, i3', rop.GUARD_TRUE)
-
-    def test_expand_fail_2(self):
-        ops = """
-        [i1, i2]
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, i2, descr=valuedescr)
-        setfield_gc(p1, p1, descr=nextdescr)
-        guard_true(i1, descr=fdescr) [p1]
-        jump(i1, i2)
-        """
-        expected = """
-        [i1, i2]
-        guard_true(i1, descr=fdescr) [i2]
-        jump(1, i2)
-        """
-        self.optimize_loop(ops, expected)
-        self.check_expanded_fail_descr('''ptr
-            where ptr is a node_vtable, valuedescr=i2
-            ''', rop.GUARD_TRUE)
-
-    def test_expand_fail_3(self):
-        ops = """
-        [i1, i2, i3, p3]
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, 1, descr=valuedescr)
-        setfield_gc(p1, p2, descr=nextdescr)
-        setfield_gc(p2, i2, descr=valuedescr)
-        setfield_gc(p2, p3, descr=nextdescr)
-        guard_true(i1, descr=fdescr) [i3, p1]
-        jump(i2, i1, i3, p3)
-        """
-        expected = """
-        [i1, i2, i3, p3]
-        guard_true(i1, descr=fdescr) [i3, i2, p3]
-        jump(i2, 1, i3, p3)
-        """
-        self.optimize_loop(ops, expected)
-        self.check_expanded_fail_descr('''i3, p1
-            where p1 is a node_vtable, valuedescr=1, nextdescr=p2
-            where p2 is a node_vtable, valuedescr=i2, nextdescr=p3
-            ''', rop.GUARD_TRUE)
-
-    def test_expand_fail_4(self):
-        for arg in ['p1', 'i2,p1', 'p1,p2', 'p2,p1',
-                    'i2,p1,p2', 'i2,p2,p1']:
-            self.setup_method() # humpf
-            ops = """
-            [i1, i2, i3]
-            p1 = new_with_vtable(ConstClass(node_vtable))
-            setfield_gc(p1, i3, descr=valuedescr)
-            i4 = getfield_gc(p1, descr=valuedescr)   # copy of i3
-            p2 = new_with_vtable(ConstClass(node_vtable))
-            setfield_gc(p1, i2, descr=valuedescr)
-            setfield_gc(p1, p2, descr=nextdescr)
-            setfield_gc(p2, i2, descr=valuedescr)
-            guard_true(i1, descr=fdescr) [i4, i3, %s]
-            jump(i1, i2, i3)
-            """
-            expected = """
-            [i1, i2, i3]
-            guard_true(i1, descr=fdescr) [i3, i2]
-            jump(1, i2, i3)
-            """
-            self.optimize_loop(ops % arg, expected)
-            self.check_expanded_fail_descr('''i3, i3, %s
-                where p1 is a node_vtable, valuedescr=i2, nextdescr=p2
-                where p2 is a node_vtable, valuedescr=i2''' % arg,
-                                           rop.GUARD_TRUE)
-
-    def test_expand_fail_5(self):
-        ops = """
-        [i1, i2, i3, i4]
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, i4, descr=valuedescr)
-        setfield_gc(p1, p2, descr=nextdescr)
-        setfield_gc(p2, i2, descr=valuedescr)
-        setfield_gc(p2, p1, descr=nextdescr)      # a cycle
-        guard_true(i1, descr=fdescr) [i3, i4, p1, p2]
-        jump(i2, i1, i3, i4)
-        """
-        expected = """
-        [i1, i2, i3, i4]
-        guard_true(i1, descr=fdescr) [i3, i4, i2]
-        jump(i2, 1, i3, i4)
-        """
-        self.optimize_loop(ops, expected)
-        self.check_expanded_fail_descr('''i3, i4, p1, p2
-            where p1 is a node_vtable, valuedescr=i4, nextdescr=p2
-            where p2 is a node_vtable, valuedescr=i2, nextdescr=p1
-            ''', rop.GUARD_TRUE)
-
-    def test_expand_fail_6(self):
-        ops = """
-        [p0, i0, i1]
-        guard_true(i0, descr=fdescr) [p0]
-        p1 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p1, i1, descr=valuedescr)
-        jump(p1, i1, i1)
-        """
-        expected = """
-        [i1b, i0, i1]
-        guard_true(i0, descr=fdescr) [i1b]
-        jump(i1, i1, i1)
-        """
-        xxx
-        self.optimize_loop(ops, '''Virtual(node_vtable, valuedescr=Not),
-                                   Not, Not''', expected)
-        self.check_expanded_fail_descr('''p0
-            where p0 is a node_vtable, valuedescr=i1b
-            ''', rop.GUARD_TRUE)
-
-    def test_expand_fail_varray(self):
-        ops = """
-        [i1]
-        p1 = new_array(3, descr=arraydescr)
-        setarrayitem_gc(p1, 1, i1, descr=arraydescr)
-        setarrayitem_gc(p1, 0, 25, descr=arraydescr)
-        guard_true(i1, descr=fdescr) [p1]
-        i2 = getarrayitem_gc(p1, 1, descr=arraydescr)
-        jump(i2)
-        """
-        expected = """
-        [i1]
-        guard_true(i1, descr=fdescr) [i1]
-        jump(1)
-        """
-        self.optimize_loop(ops, expected)
-        self.check_expanded_fail_descr('''p1
-            where p1 is a varray arraydescr: 25, i1
-            ''', rop.GUARD_TRUE)
-
-    def test_expand_fail_vstruct(self):
-        ops = """
-        [i1, p1]
-        p2 = new(descr=ssize)
-        setfield_gc(p2, i1, descr=adescr)
-        setfield_gc(p2, p1, descr=bdescr)
-        guard_true(i1, descr=fdescr) [p2]
-        i3 = getfield_gc(p2, descr=adescr)
-        p3 = getfield_gc(p2, descr=bdescr)
-        jump(i3, p3)
-        """
-        expected = """
-        [i1, p1]
-        guard_true(i1, descr=fdescr) [i1, p1]
-        jump(1, p1)
-        """
-        self.optimize_loop(ops, expected)
-        self.check_expanded_fail_descr('''p2
-            where p2 is a vstruct ssize, adescr=i1, bdescr=p1
-            ''', rop.GUARD_TRUE)
-
     def test_expand_fail_v_all_1(self):
         ops = """
         [i1, p1a, i2]


From arigo at codespeak.net  Sat Oct 30 19:07:57 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 19:07:57 +0200 (CEST)
Subject: [pypy-svn] r78556 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030170757.B60F1282C2E@codespeak.net>

Author: arigo
Date: Sat Oct 30 19:07:55 2010
New Revision: 78556

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
Improve the printing done by equaloplists(). 


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	Sat Oct 30 19:07:55 2010
@@ -129,7 +129,8 @@
 
 # ____________________________________________________________
 
-def equaloplists(oplist1, oplist2, strict_fail_args=True, remap={}):
+def equaloplists(oplist1, oplist2, strict_fail_args=True, remap={},
+                 text_right=None):
     # try to use the full width of the terminal to display the list
     # unfortunately, does not work with the default capture method of py.test
     # (which is fd), you you need to use either -s or --capture=sys, else you
@@ -137,7 +138,8 @@
     totwidth = py.io.get_terminal_width()
     width = totwidth / 2 - 1
     print ' Comparing lists '.center(totwidth, '-')
-    print '%s| %s' % ('optimized'.center(width), 'expected'.center(width))
+    text_right = text_right or 'expected'
+    print '%s| %s' % ('optimized'.center(width), text_right.center(width))
     for op1, op2 in zip(oplist1, oplist2):
         txt1 = str(op1)
         txt2 = str(op2)
@@ -167,7 +169,7 @@
                 fail_args2 = set([remap.get(y, y) for y in op2.getfailargs()])
                 assert fail_args1 == fail_args2
     assert len(oplist1) == len(oplist2)
-    print '-'*57
+    print '-'*totwidth
     return True
 
 def test_equaloplists():

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 19:07:55 2010
@@ -55,14 +55,14 @@
         descr.rd_snapshot = resume.Snapshot(None, _sortboxes(fail_args))
         return descr
 
-    def assert_equal(self, optimized, expected):
+    def assert_equal(self, optimized, expected, text_right=None):
         assert len(optimized.inputargs) == len(expected.inputargs)
         remap = {}
         for box1, box2 in zip(optimized.inputargs, expected.inputargs):
             assert box1.__class__ == box2.__class__
             remap[box2] = box1
         assert equaloplists(optimized.operations,
-                            expected.operations, False, remap)
+                            expected.operations, False, remap, text_right)
 
     def optimize_loop(self, ops, optops, expected_preamble=None):
         loop = self.parse(ops)
@@ -89,7 +89,8 @@
         
         self.assert_equal(loop, expected)
         if expected_preamble:
-            self.assert_equal(loop.preamble, expected_preamble)
+            self.assert_equal(loop.preamble, expected_preamble,
+                              text_right='expected preamble')
 
         return loop
 


From hakanardo at codespeak.net  Sat Oct 30 19:10:08 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 19:10:08 +0200 (CEST)
Subject: [pypy-svn] r78557 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030171008.D351636E46C@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 19:10:07 2010
New Revision: 78557

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(armin, cfbolz, hakanardo) fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 19:10:07 2010
@@ -2090,10 +2090,9 @@
         jump(p2)
         """
         py.test.raises(InvalidLoop, self.optimize_loop,
-                       ops, 'Virtual(node_vtable)', None)
+                       ops, ops)
 
     def test_invalid_loop_2(self):
-        py.test.skip("this would fail if we had Fixed again in the specnodes")
         ops = """
         [p1]
         guard_class(p1, ConstClass(node_vtable2)) []
@@ -2103,7 +2102,7 @@
         jump(p2)
         """
         py.test.raises(InvalidLoop, self.optimize_loop,
-                       ops, '...', None)
+                       ops, ops)
 
     def test_invalid_loop_3(self):
         ops = """
@@ -2116,9 +2115,8 @@
         setfield_gc(p3, p4, descr=nextdescr)
         jump(p3)
         """
-        py.test.raises(InvalidLoop, self.optimize_loop, ops,
-                       'Virtual(node_vtable, nextdescr=Virtual(node_vtable))',
-                       None)
+        py.test.raises(InvalidLoop, self.optimize_loop, ops, ops)
+
 
     def test_merge_guard_class_guard_value(self):
         ops = """


From hakanardo at codespeak.net  Sat Oct 30 19:14:28 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 19:14:28 +0200 (CEST)
Subject: [pypy-svn] r78558 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030171428.C2BC1282C30@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 19:14:27 2010
New Revision: 78558

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(armin, cfbolz, hakanardo) fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 19:14:27 2010
@@ -2271,7 +2271,7 @@
         guard_true(i2) []
         jump(p1, p2)
         """
-        expected = """
+        preamble = """
         [p1, p2]
         i1 = ptr_eq(p1, p2)
         i3 = int_add(i1, 1)
@@ -2282,7 +2282,13 @@
         guard_true(i1) []
         jump(p1, p2)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [p1, p2]
+        escape(2)
+        escape(2)
+        jump(p1, p2)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_remove_duplicate_pure_op_with_descr(self):
         ops = """


From cfbolz at codespeak.net  Sat Oct 30 19:15:29 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 19:15:29 +0200 (CEST)
Subject: [pypy-svn] r78559 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030171529.1424136E476@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 19:15:27 2010
New Revision: 78559

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
further improvement possible


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 19:15:27 2010
@@ -2821,6 +2821,7 @@
         # time.  Check that it is either constant-folded (and replaced by
         # the result of the call, recorded as the first arg), or turned into
         # a regular CALL.
+        # XXX can this test be improved with unrolling?
         ops = '''
         [i0, i1, i2]
         escape(i1)
@@ -2829,14 +2830,21 @@
         i4 = call_pure(43, 123456, 4, i0, 6, descr=plaincalldescr)
         jump(i0, i3, i4)
         '''
-        expected = '''
+        preamble = '''
         [i0, i1, i2]
         escape(i1)
         escape(i2)
         i4 = call(123456, 4, i0, 6, descr=plaincalldescr)
-        jump(i0, 42, i4)
+        jump(i0, i4)
         '''
-        self.optimize_loop(ops, expected)
+        expected = '''
+        [i0, i2]
+        escape(42)
+        escape(i2)
+        i4 = call(123456, 4, i0, 6, descr=plaincalldescr)
+        jump(i0, i4)
+        '''
+        self.optimize_loop(ops, expected, preamble)
 
     def test_vref_nonvirtual_nonescape(self):
         ops = """


From hakanardo at codespeak.net  Sat Oct 30 19:19:00 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 19:19:00 +0200 (CEST)
Subject: [pypy-svn] r78560 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030171900.ED228282C33@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 19:18:59 2010
New Revision: 78560

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 19:18:59 2010
@@ -2301,14 +2301,18 @@
         guard_true(i3) []
         jump(p1)
         """
-        expected = """
+        preamble = """
         [p1]
         i0 = arraylen_gc(p1, descr=arraydescr)
         i1 = int_gt(i0, 0)
         guard_true(i1) []
         jump(p1)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [p1]
+        jump(p1)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_remove_duplicate_pure_op_ovf(self):
         ops = """
@@ -2325,7 +2329,7 @@
         escape(i4)
         jump(i1)
         """
-        expected = """
+        preamble = """
         [i1]
         i3 = int_add_ovf(i1, 1)
         guard_no_overflow() []
@@ -2333,9 +2337,15 @@
         guard_true(i3b) []
         escape(i3)
         escape(i3)
-        jump(i1)
+        jump(i1, i3)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i1, i3]
+        escape(i3)
+        escape(i3)
+        jump(i1, i3)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_int_and_or_with_zero(self):
         ops = """


From cfbolz at codespeak.net  Sat Oct 30 19:20:06 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 19:20:06 +0200 (CEST)
Subject: [pypy-svn] r78561 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030172006.1FE8136E477@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 19:20:04 2010
New Revision: 78561

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fix


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 19:20:04 2010
@@ -3147,13 +3147,18 @@
         guard_true(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_lt(i0, 4)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lt_noguard(self):
         ops = """


From cfbolz at codespeak.net  Sat Oct 30 19:20:45 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sat, 30 Oct 2010 19:20:45 +0200 (CEST)
Subject: [pypy-svn] r78562 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030172045.0B502282C35@codespeak.net>

Author: cfbolz
Date: Sat Oct 30 19:20:43 2010
New Revision: 78562

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
kill some tests. their equivalent in test_optimizebasic should be fixed


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 19:20:43 2010
@@ -2542,97 +2542,6 @@
         boxes = reader.consume_boxes()
         self._verify_fail_args(boxes, fdescr.oparse, expectedtext)
 
-    def test_expand_fail_v_all_1(self):
-        ops = """
-        [i1, p1a, i2]
-        p6s = getarrayitem_gc(p1a, 0, descr=arraydescr2)
-        p7v = getfield_gc(p6s, descr=bdescr)
-        p5s = new(descr=ssize)
-        setfield_gc(p5s, i2, descr=adescr)
-        setfield_gc(p5s, p7v, descr=bdescr)
-        setarrayitem_gc(p1a, 1, p5s, descr=arraydescr2)
-        guard_true(i1, descr=fdescr) [p1a]
-        p2s = new(descr=ssize)
-        p3v = new_with_vtable(ConstClass(node_vtable))
-        p4a = new_array(2, descr=arraydescr2)
-        setfield_gc(p2s, i1, descr=adescr)
-        setfield_gc(p2s, p3v, descr=bdescr)
-        setfield_gc(p3v, i2, descr=valuedescr)
-        setarrayitem_gc(p4a, 0, p2s, descr=arraydescr2)
-        jump(i1, p4a, i2)
-        """
-        expected = """
-        [i1, ia, iv, pnull, i2]
-        guard_true(i1, descr=fdescr) [ia, iv, i2]
-        jump(1, 1, i2, NULL, i2)
-        """
-        xxx
-        self.optimize_loop(ops, '''
-            Not,
-            VArray(arraydescr2,
-                   VStruct(ssize,
-                           adescr=Not,
-                           bdescr=Virtual(node_vtable,
-                                          valuedescr=Not)),
-                   Not),
-            Not''', expected)
-        self.check_expanded_fail_descr('''p1a
-            where p1a is a varray arraydescr2: p6s, p5s
-            where p6s is a vstruct ssize, adescr=ia, bdescr=p7v
-            where p5s is a vstruct ssize, adescr=i2, bdescr=p7v
-            where p7v is a node_vtable, valuedescr=iv
-            ''', rop.GUARD_TRUE)
-
-    def test_expand_fail_lazy_setfield_1(self):
-        ops = """
-        [p1, i2, i3]
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p2, i2, descr=valuedescr)
-        setfield_gc(p1, p2, descr=nextdescr)
-        guard_true(i3, descr=fdescr) []
-        i4 = int_neg(i2)
-        setfield_gc(p1, NULL, descr=nextdescr)
-        jump(p1, i2, i4)
-        """
-        expected = """
-        [p1, i2, i3]
-        guard_true(i3, descr=fdescr) [p1, i2]
-        i4 = int_neg(i2)
-        setfield_gc(p1, NULL, descr=nextdescr)
-        jump(p1, i2, i4)
-        """
-        self.optimize_loop(ops, expected)
-        self.loop.inputargs[0].value = self.nodebox.value
-        self.check_expanded_fail_descr('''
-            p1.nextdescr = p2
-            where p2 is a node_vtable, valuedescr=i2
-            ''', rop.GUARD_TRUE)
-
-    def test_expand_fail_lazy_setfield_2(self):
-        ops = """
-        [i2, i3]
-        p2 = new_with_vtable(ConstClass(node_vtable))
-        setfield_gc(p2, i2, descr=valuedescr)
-        setfield_gc(ConstPtr(myptr), p2, descr=nextdescr)
-        guard_true(i3, descr=fdescr) []
-        i4 = int_neg(i2)
-        setfield_gc(ConstPtr(myptr), NULL, descr=nextdescr)
-        jump(i2, i4)
-        """
-        expected = """
-        [i2, i3]
-        guard_true(i3, descr=fdescr) [i2]
-        i4 = int_neg(i2)
-        setfield_gc(ConstPtr(myptr), NULL, descr=nextdescr)
-        jump(i2, i4)
-        """
-        self.optimize_loop(ops, expected)
-        self.check_expanded_fail_descr('''
-            ConstPtr(myptr).nextdescr = p2
-            where p2 is a node_vtable, valuedescr=i2
-            ''', rop.GUARD_TRUE)
-
-
 class TestLLtype(OptimizeOptTest, LLtypeMixin):
 
     def test_residual_call_does_not_invalidate_caches(self):


From arigo at codespeak.net  Sat Oct 30 19:24:25 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 19:24:25 +0200 (CEST)
Subject: [pypy-svn] r78563 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030172425.DE591282BE7@codespeak.net>

Author: arigo
Date: Sat Oct 30 19:24:24 2010
New Revision: 78563

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefficall.py
Log:
Fix test_optimizefficall.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	Sat Oct 30 19:24:24 2010
@@ -226,7 +226,7 @@
     _kind2count = {history.INT: 1, history.REF: 2, history.FLOAT: 3}
     return sorted(boxes, key=lambda box: _kind2count[box.type])
 
-class BaseTestOptimizeBasic(BaseTest):
+class BaseTestBasic(BaseTest):
 
     def invent_fail_descr(self, fail_args):
         if fail_args is None:
@@ -259,12 +259,14 @@
                                                      OptVirtualize,
                                                      OptString,
                                                      OptHeap,
+                                                     OptFfiCall,
                                                      Optimizer)
         optimizations = [OptIntBounds(),
                          OptRewrite(),
                          OptVirtualize(),
                          OptString(),
                          OptHeap(),
+                         OptFfiCall(),
                          ]
         optimizer = Optimizer(metainterp_sd, loop, optimizations,
                               not_a_bridge=True)
@@ -274,6 +276,9 @@
         print '\n'.join([str(o) for o in loop.operations])
         self.assert_equal(loop, expected)
 
+
+class BaseTestOptimizeBasic(BaseTestBasic):
+
     def test_simple(self):
         ops = """
         [i]

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefficall.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefficall.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizefficall.py	Sat Oct 30 19:24:24 2010
@@ -2,7 +2,8 @@
 from pypy.rlib.libffi import Func, types
 from pypy.jit.metainterp.history import AbstractDescr
 from pypy.jit.codewriter.effectinfo import EffectInfo
-from pypy.jit.metainterp.test.test_optimizeopt import BaseTestOptimizeOpt, LLtypeMixin
+from pypy.jit.metainterp.test.test_optimizebasic import BaseTestBasic
+from pypy.jit.metainterp.test.test_optimizebasic import LLtypeMixin
 
 class MyCallDescr(AbstractDescr):
     """
@@ -30,7 +31,7 @@
         return id(self)
 
 
-class TestFfiCall(BaseTestOptimizeOpt, LLtypeMixin):
+class TestFfiCall(BaseTestBasic, LLtypeMixin):
 
     class namespace:
         cpu = LLtypeMixin.cpu
@@ -45,13 +46,15 @@
                              argtypes=[types.sint, types.double],
                              restype=types.sint)
         #
-        def calldescr(cpu, FUNC, oopspecindex):
-            einfo = EffectInfo([], [], [], oopspecindex=oopspecindex)
+        def calldescr(cpu, FUNC, oopspecindex, extraeffect=None):
+            einfo = EffectInfo([], [], [], oopspecindex=oopspecindex,
+                               extraeffect=extraeffect)
             return cpu.calldescrof(FUNC, FUNC.ARGS, FUNC.RESULT, einfo)
         #
         libffi_prepare =  calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_PREPARE)
         libffi_push_arg = calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_PUSH_ARG)
-        libffi_call =     calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_CALL)
+        libffi_call =     calldescr(cpu, FUNC, EffectInfo.OS_LIBFFI_CALL,
+                                 EffectInfo.EF_FORCES_VIRTUAL_OR_VIRTUALIZABLE)
     
     namespace = namespace.__dict__
 
@@ -77,7 +80,7 @@
         guard_no_exception() []
         jump(i3, f1)
         """
-        loop = self.optimize_loop(ops, 'Not, Not', expected)
+        loop = self.optimize_loop(ops, expected)
 
     def test_ffi_call_nonconst(self):
         ops = """
@@ -91,7 +94,7 @@
         jump(i3, f1, p2)
         """
         expected = ops
-        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+        loop = self.optimize_loop(ops, expected)
 
     def test_handle_virtualizables(self):
         # this test needs an explanation to understand what goes on: see the
@@ -117,7 +120,7 @@
         guard_no_exception() [p2]
         jump(i3, f1, p2)
         """
-        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+        loop = self.optimize_loop(ops, expected)
 
     # ----------------------------------------------------------------------
     # in pratice, the situations described in these tests should never happen,
@@ -136,7 +139,7 @@
         jump(i3, f1)
         """
         expected = ops
-        loop = self.optimize_loop(ops, 'Not, Not', expected)
+        loop = self.optimize_loop(ops, expected)
 
     def test_rollback_multiple_calls(self):
         ops = """
@@ -159,7 +162,7 @@
         jump(i3, i4, f1)
         """
         expected = ops
-        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+        loop = self.optimize_loop(ops, expected)
 
     def test_rollback_multiple_prepare(self):
         ops = """
@@ -182,7 +185,7 @@
         jump(i3, i4, f1)
         """
         expected = ops
-        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+        loop = self.optimize_loop(ops, expected)
 
     def test_optimize_nested_call(self):
         ops = """
@@ -220,7 +223,7 @@
         guard_no_exception() []
         jump(i3, i4, f1)
         """
-        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+        loop = self.optimize_loop(ops, expected)
 
     def test_rollback_force_token(self):
         ops = """
@@ -237,4 +240,4 @@
         jump(i3, f1, p2)
         """
         expected = ops
-        loop = self.optimize_loop(ops, 'Not, Not, Not', expected)
+        loop = self.optimize_loop(ops, expected)


From arigo at codespeak.net  Sat Oct 30 19:29:58 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 19:29:58 +0200 (CEST)
Subject: [pypy-svn] r78564 - pypy/branch/jit-unroll-loops/pypy/rlib
Message-ID: <20101030172958.EA1C0282C37@codespeak.net>

Author: arigo
Date: Sat Oct 30 19:29:57 2010
New Revision: 78564

Modified:
   pypy/branch/jit-unroll-loops/pypy/rlib/jit.py
Log:
Re-change the default optimizer to be OPTIMIZER_FULL.


Modified: pypy/branch/jit-unroll-loops/pypy/rlib/jit.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/jit.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/jit.py	Sat Oct 30 19:29:57 2010
@@ -260,8 +260,8 @@
               'trace_eagerness': 200,
               'trace_limit': 10000,
               'inlining': False,
-              #'optimizer': OPTIMIZER_FULL,
-              'optimizer': OPTIMIZER_NO_PERFECTSPEC,
+              'optimizer': OPTIMIZER_FULL,
+              #'optimizer': OPTIMIZER_NO_PERFECTSPEC,
               'debug' : DEBUG_STEPS,
               }
 unroll_parameters = unrolling_iterable(PARAMETERS.keys())


From arigo at codespeak.net  Sat Oct 30 19:34:45 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sat, 30 Oct 2010 19:34:45 +0200 (CEST)
Subject: [pypy-svn] r78565 - pypy/branch/jit-unroll-loops/pypy/jit/metainterp
Message-ID: <20101030173445.D727B282C37@codespeak.net>

Author: arigo
Date: Sat Oct 30 19:34:44 2010
New Revision: 78565

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize.py
Log:
Fix?  This file is really the same as optimize_nopspec right now.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize.py	Sat Oct 30 19:34:44 2010
@@ -2,7 +2,7 @@
 
 # ____________________________________________________________
 
-from pypy.jit.metainterp.optimizeopt import optimize_loop_1
+from pypy.jit.metainterp.optimizeopt import optimize_loop_1, optimize_bridge_1
 
 def optimize_loop(metainterp_sd, old_loop_tokens, loop):
     debug_start("jit-optimize")
@@ -33,6 +33,9 @@
     cpu = metainterp_sd.cpu
     metainterp_sd.logger_noopt.log_loop(bridge.inputargs, bridge.operations)
     if old_loop_tokens:
+        old_loop_token = old_loop_tokens[0]
+        bridge.operations[-1].setdescr(old_loop_token)   # patch jump target
+        optimize_bridge_1(metainterp_sd, bridge)
         return old_loop_tokens[0]
     return None
 


From hakanardo at codespeak.net  Sat Oct 30 19:47:09 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sat, 30 Oct 2010 19:47:09 +0200 (CEST)
Subject: [pypy-svn] r78566 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101030174709.50EF0282BEA@codespeak.net>

Author: hakanardo
Date: Sat Oct 30 19:47:07 2010
New Revision: 78566

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sat Oct 30 19:47:07 2010
@@ -3082,7 +3082,7 @@
         i2 = int_lt(i0, 5)
         jump(i2)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_loop(ops, expected, expected)
 
     def test_bound_lt_noopt(self):
         ops = """
@@ -3093,15 +3093,19 @@
         guard_true(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_lt(i0, 4)
         guard_false(i1) []
         i2 = int_lt(i0, 5)
         guard_true(i2) []
-        jump(4)
+        jump()
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        []
+        jump()
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lt_rev(self):
         ops = """
@@ -3112,13 +3116,17 @@
         guard_true(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_lt(i0, 4)
         guard_false(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lt_tripple(self):
         ops = """
@@ -3131,13 +3139,17 @@
         guard_true(i3) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_lt(i0, 0)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lt_add(self):
         ops = """
@@ -3149,14 +3161,18 @@
         guard_true(i3) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_lt(i0, 4)
         guard_true(i1) []
         i2 = int_add(i0, 10)
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lt_add_before(self):
         ops = """
@@ -3168,14 +3184,18 @@
         guard_true(i1) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i2 = int_add(i0, 10)
         i3 = int_lt(i2, 15)
         guard_true(i3) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lt_add_ovf(self):
         ops = """
@@ -3188,14 +3208,18 @@
         guard_true(i3) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_lt(i0, 4)
         guard_true(i1) []
         i2 = int_add(i0, 10)
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lt_add_ovf_before(self):
         ops = """


From afa at codespeak.net  Sat Oct 30 20:32:02 2010
From: afa at codespeak.net (afa at codespeak.net)
Date: Sat, 30 Oct 2010 20:32:02 +0200 (CEST)
Subject: [pypy-svn] r78567 - in
	pypy/branch/fast-forward/pypy/module/_hashlib: . test
Message-ID: <20101030183202.BC8BE282BAD@codespeak.net>

Author: afa
Date: Sat Oct 30 20:32:01 2010
New Revision: 78567

Modified:
   pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py
   pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py
Log:
Note to self: a function definition in a loop is probably an error!
all functions capture the same variale, and so get the
last value of the list!


Modified: pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py	Sat Oct 30 20:32:01 2010
@@ -124,10 +124,13 @@
     return space.wrap(w_hash)
 
 # shortcut functions
-for name in algorithms:
-    newname = 'new_%s' % (name,)
+def make_new_hash(name):
     @func_renamer(newname)
     @unwrap_spec(ObjSpace, str)
     def new_hash(space, string=''):
         return new(space, name, string)
-    globals()[newname] = new_hash
+    return new_hash
+
+for name in algorithms:
+    newname = 'new_%s' % (name,)
+    globals()[newname] = make_new_hash(name)

Modified: pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/test/test_hashlib.py	Sat Oct 30 20:32:01 2010
@@ -57,6 +57,10 @@
             assert c_digest    == digest
             assert c_hexdigest == hexdigest
 
+    def test_shortcut(self):
+        import hashlib
+        assert repr(hashlib.md5()).startswith("

Author: afa
Date: Sat Oct 30 20:36:22 2010
New Revision: 78568

Modified:
   pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py
Log:
Rename each function with a different name,
thanks Alex!


Modified: pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py
==============================================================================
--- pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py	(original)
+++ pypy/branch/fast-forward/pypy/module/_hashlib/interp_hashlib.py	Sat Oct 30 20:36:22 2010
@@ -124,8 +124,8 @@
     return space.wrap(w_hash)
 
 # shortcut functions
-def make_new_hash(name):
-    @func_renamer(newname)
+def make_new_hash(name, funcname):
+    @func_renamer(funcname)
     @unwrap_spec(ObjSpace, str)
     def new_hash(space, string=''):
         return new(space, name, string)
@@ -133,4 +133,4 @@
 
 for name in algorithms:
     newname = 'new_%s' % (name,)
-    globals()[newname] = make_new_hash(name)
+    globals()[newname] = make_new_hash(name, newname)


From david at codespeak.net  Sat Oct 30 21:20:54 2010
From: david at codespeak.net (david at codespeak.net)
Date: Sat, 30 Oct 2010 21:20:54 +0200 (CEST)
Subject: [pypy-svn] r78569 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101030192054.B8811282BE7@codespeak.net>

Author: david
Date: Sat Oct 30 21:20:51 2010
New Revision: 78569

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
Log:
Improve and correct jumps from one codeblock to the next one allocated to fit isntructions

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Sat Oct 30 21:20:51 2010
@@ -15,6 +15,7 @@
     def f(self, cond=cond.AL):
         """Generates a call to a helper function, takes its
         arguments in r0 and r1, result is placed in r0"""
+        self.ensure_can_fit(self.size_of_gen_load_int*2+3*WORD)
         self.PUSH(range(2, 12), cond=cond)
         addr = rffi.cast(lltype.Signed, llhelper(signature, function))
         self.gen_load_int(reg.r2.value, addr, cond=cond)
@@ -35,6 +36,9 @@
             f.write(self._data[i])
         f.close()
 
+    def ensure_can_fit(self, n):
+        raise NotImplentedError
+
     def PUSH(self, regs, cond=cond.AL):
         assert reg.sp not in regs
         instr = self._encode_reg_list(cond << 28 | 0x92D << 16, regs)
@@ -109,6 +113,7 @@
         data = rffi.cast(PTR, start)
         self._init(data, map_size)
 
+
 class ARMv7Builder(AbstractARMv7Builder):
 
     def __init__(self):
@@ -119,7 +124,7 @@
         self.checks = True
         self.n_data=0
 
-    _space_for_jump = 9 * WORD
+    _space_for_jump = 2 * WORD
     def writechar(self, char):
         if self.checks and not self._pos < self._size - self._space_for_jump:
             self.checks = False
@@ -131,13 +136,17 @@
     def _add_more_mem(self):
         new_mem = alloc(self._size)
         new_mem_addr = rffi.cast(lltype.Signed, new_mem)
-        self.PUSH([reg.ip.value, reg.lr.value])
-        self.gen_load_int(reg.lr.value, new_mem_addr)
-        self.MOV_rr(reg.pc.value, reg.lr.value)
+        self.LDR_ri(reg.pc.value, reg.pc.value, -4)
+        self.write32(new_mem_addr)
         self._dump_trace('data%d.asm' % self.n_data)
         self.n_data += 1
         self._data = new_mem
         self._pos = 0
-        self.LDM(reg.sp.value, [reg.ip.value, reg.lr.value], w=1) # XXX Replace with POP instr. someday
+
+    def ensure_can_fit(self, n):
+        """ensure after this call there is enough space for n instructions
+        in a contiguous memory chunk"""
+        if not self._pos + n + self._space_for_jump < self._size:
+            self._add_more_mem()
 
 define_instructions(AbstractARMv7Builder)


From david at codespeak.net  Sat Oct 30 23:02:31 2010
From: david at codespeak.net (david at codespeak.net)
Date: Sat, 30 Oct 2010 23:02:31 +0200 (CEST)
Subject: [pypy-svn] r78570 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101030210231.A99E4282B90@codespeak.net>

Author: david
Date: Sat Oct 30 23:02:28 2010
New Revision: 78570

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/conditions.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
Log:
Implement int and uint comparison operations

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/codebuilder.py	Sat Oct 30 23:02:28 2010
@@ -16,12 +16,12 @@
         """Generates a call to a helper function, takes its
         arguments in r0 and r1, result is placed in r0"""
         self.ensure_can_fit(self.size_of_gen_load_int*2+3*WORD)
-        self.PUSH(range(2, 12), cond=cond)
+        self.PUSH(range(2, 4), cond=cond)
         addr = rffi.cast(lltype.Signed, llhelper(signature, function))
         self.gen_load_int(reg.r2.value, addr, cond=cond)
         self.gen_load_int(reg.lr.value, self.curraddr()+self.size_of_gen_load_int+WORD, cond=cond)
         self.MOV_rr(reg.pc.value, reg.r2.value, cond=cond)
-        self.LDM(reg.sp.value, range(2, 12), w=1, cond=cond) # XXX Replace with POP instr. someday
+        self.LDM(reg.sp.value, range(2, 4), w=1, cond=cond) # XXX Replace with POP instr. someday
     return f
 
 class AbstractARMv7Builder(object):
@@ -49,15 +49,6 @@
         instr = self._encode_reg_list(instr, regs)
         self.write32(instr)
 
-    def CMP(self, rn, imm, cond=cond.AL):
-        if 0 <= imm <= 255:
-            self.write32(cond << 28
-                        | 0x35 << 20
-                        | (rn & 0xFF) <<  16
-                        | (imm & 0xFFF))
-        else:
-            raise NotImplentedError
-
     def BKPT(self, cond=cond.AL):
         self.write32(cond << 28 | 0x1200070)
 

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/conditions.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/conditions.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/conditions.py	Sat Oct 30 23:02:28 2010
@@ -13,3 +13,8 @@
 GT = 0xC
 LE = 0xD
 AL = 0xE
+
+opposites = [NE, EQ, CC, CS, PL, MI, VC, VS, LS, HI, LT, GE, LE, GT, AL]
+def get_opposite_of(operation):
+    return opposites[operation]
+

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	Sat Oct 30 23:02:28 2010
@@ -52,6 +52,32 @@
         return fcond
     return f
 
+def gen_emit_cmp_op(condition, inverse=False):
+    def f(self, op, regalloc, fcond):
+        assert fcond == c.AL
+        if not inverse:
+            arg0 = op.getarg(0)
+            arg1 = op.getarg(1)
+        else:
+            arg0 = op.getarg(1)
+            arg1 = op.getarg(0)
+        res = regalloc.try_allocate_reg(op.result)
+        # XXX consider swapping argumentes if arg0 is const
+        if self._check_imm_arg(arg1) and not isinstance(arg0, ConstInt):
+            reg = regalloc.try_allocate_reg(arg0)
+            self.mc.CMP_ri(reg.value, imm=arg1.getint(), cond=fcond)
+        else:
+            reg = self._put_in_reg(arg0, regalloc)
+            reg2 = self._put_in_reg(arg1, regalloc)
+            self.mc.CMP_rr(reg.value, reg2.value)
+            regalloc.possibly_free_var(reg2)
+
+        inv = c.get_opposite_of(condition)
+        self.mc.MOV_ri(res.value, 1, cond=condition)
+        self.mc.MOV_ri(res.value, 0, cond=inv)
+        return condition
+    return f
+
 class IntOpAsslember(object):
     _mixin_ = True
 
@@ -137,8 +163,21 @@
     emit_op_int_rshift = gen_emit_op_ri('ASR', imm_size=0x1F, commutative=False)
     emit_op_uint_rshift = gen_emit_op_ri('LSR', imm_size=0x1F, commutative=False)
 
+    emit_op_int_lt = gen_emit_cmp_op(c.LT)
+    emit_op_int_le = gen_emit_cmp_op(c.LE)
+    emit_op_int_eq = gen_emit_cmp_op(c.EQ)
+    emit_op_int_ne = gen_emit_cmp_op(c.NE)
+    emit_op_int_gt = gen_emit_cmp_op(c.GT)
+    emit_op_int_ge = gen_emit_cmp_op(c.GE)
 
-    def _check_imm_arg(self, arg, size):
+    emit_op_uint_le = gen_emit_cmp_op(c.LS)
+    emit_op_uint_gt = gen_emit_cmp_op(c.HI)
+
+    emit_op_uint_lt = gen_emit_cmp_op(c.HI, inverse=True)
+    emit_op_uint_ge = gen_emit_cmp_op(c.LS, inverse=True)
+
+
+    def _check_imm_arg(self, arg, size=0xFF):
         #XXX check ranges for different operations
         return isinstance(arg, ConstInt) and arg.getint() <= size and arg.getint() > 0
 
@@ -155,8 +194,10 @@
         descr._arm_guard_cond = fcond
 
     def emit_op_guard_true(self, op, regalloc, fcond):
-        assert fcond == c.GT
-        self._emit_guard(op, regalloc, fcond)
+        assert fcond == c.LE
+        cond = c.get_opposite_of(fcond)
+        assert cond == c.GT
+        self._emit_guard(op, regalloc, cond)
         return c.AL
 
     def emit_op_guard_false(self, op, regalloc, fcond):
@@ -183,17 +224,5 @@
         return fcond
 
     def emit_op_finish(self, op, regalloc, fcond):
-        self._gen_path_to_exit_path(op, op.getarglist(), regalloc, fcond)
+        self._gen_path_to_exit_path(op, op.getarglist(), regalloc, c.AL)
         return fcond
-
-    def emit_op_int_le(self, op, regalloc, fcond):
-        reg = regalloc.try_allocate_reg(op.getarg(0))
-        assert isinstance(op.getarg(1), ConstInt)
-        self.mc.CMP(reg.value, op.getarg(1).getint())
-        return c.GT
-
-    def emit_op_int_eq(self, op, regalloc, fcond):
-        reg = regalloc.try_allocate_reg(op.getarg(0))
-        assert isinstance(op.getarg(1), ConstInt)
-        self.mc.CMP(reg.value, op.getarg(1).getint())
-        return c.EQ


From david at codespeak.net  Sat Oct 30 23:10:11 2010
From: david at codespeak.net (david at codespeak.net)
Date: Sat, 30 Oct 2010 23:10:11 +0200 (CEST)
Subject: [pypy-svn] r78571 -
	pypy/branch/arm-backend/pypy/jit/backend/arm/test
Message-ID: <20101030211011.56AD436E0C1@codespeak.net>

Author: david
Date: Sat Oct 30 23:10:09 2010
New Revision: 78571

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
Log:
Update tests after removing CMP method

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_assembler.py	Sat Oct 30 23:10:09 2010
@@ -69,7 +69,7 @@
     def test_cmp(self):
         self.a.gen_func_prolog()
         self.a.mc.gen_load_int(r.r1.value, 22)
-        self.a.mc.CMP(r.r1.value, 123)
+        self.a.mc.CMP_ri(r.r1.value, 123)
         self.a.mc.MOV_ri(r.r0.value, 1, c.LE)
         self.a.mc.MOV_ri(r.r0.value, 0, c.GT)
         self.a.gen_func_epilog()
@@ -78,7 +78,7 @@
     def test_int_le_false(self):
         self.a.gen_func_prolog()
         self.a.mc.gen_load_int(r.r1.value, 2222)
-        self.a.mc.CMP(r.r1.value, 123)
+        self.a.mc.CMP_ri(r.r1.value, 123)
         self.a.mc.MOV_ri(r.r0.value, 1, c.LE)
         self.a.mc.MOV_ri(r.r0.value, 0, c.GT)
         self.a.gen_func_epilog()
@@ -88,7 +88,7 @@
         self.a.gen_func_prolog()
         self.a.mc.MOV_ri(r.r1.value, 1)
         loop_head = self.a.mc.curraddr()
-        self.a.mc.CMP(r.r1.value, 0) # z=0, z=1
+        self.a.mc.CMP_ri(r.r1.value, 0) # z=0, z=1
         self.a.mc.MOV_ri(r.r1.value, 0, cond=c.NE)
         self.a.mc.MOV_ri(r.r1.value, 7, cond=c.EQ)
         self.a.mc.gen_load_int(r.r4.value, loop_head, cond=c.NE)
@@ -102,7 +102,7 @@
         self.a.mc.MOV_ri(r.r1.value, 1)
         loop_head = self.a.mc.curraddr()
         self.a.mc.ADD_ri(r.r1.value, r.r1.value, 1)
-        self.a.mc.CMP(r.r1.value, 9)
+        self.a.mc.CMP_ri(r.r1.value, 9)
         self.a.mc.gen_load_int(r.r4.value, loop_head, cond=c.NE)
         self.a.mc.MOV_rr(r.pc.value, r.r4.value, cond=c.NE)
         self.a.mc.MOV_rr(r.r0.value, r.r1.value)

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/test/test_instr_codebuilder.py	Sat Oct 30 23:10:09 2010
@@ -113,7 +113,7 @@
         self.assert_equal('SUB r3, r7, #4095')
 
     def test_cmp_ri(self):
-        self.cb.CMP(r.r3.value, 123)
+        self.cb.CMP_ri(r.r3.value, 123)
         self.assert_equal('CMP r3, #123')
 
     def test_mcr(self):


From arigo at codespeak.net  Sun Oct 31 10:54:36 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 10:54:36 +0100 (CET)
Subject: [pypy-svn] r78580 - pypy/trunk/lib-python/modified-2.5.2/test
Message-ID: <20101031095436.77C96282B90@codespeak.net>

Author: arigo
Date: Sun Oct 31 10:54:33 2010
New Revision: 78580

Modified:
   pypy/trunk/lib-python/modified-2.5.2/test/mapping_tests.py
Log:
Mark this as an implementation detail.  This is a copy of test_dict. :-(


Modified: pypy/trunk/lib-python/modified-2.5.2/test/mapping_tests.py
==============================================================================
--- pypy/trunk/lib-python/modified-2.5.2/test/mapping_tests.py	(original)
+++ pypy/trunk/lib-python/modified-2.5.2/test/mapping_tests.py	Sun Oct 31 10:54:33 2010
@@ -1,6 +1,7 @@
 # tests common to dict and UserDict
 import unittest
 import UserDict
+from test import test_support
 
 
 class BasicTestMappingProtocol(unittest.TestCase):
@@ -525,7 +526,8 @@
                     self.assertEqual(va, int(ka))
                     kb, vb = tb = b.popitem()
                     self.assertEqual(vb, int(kb))
-                    self.assert_(not(copymode < 0 and ta != tb))
+                    if test_support.check_impl_detail():
+                        self.assert_(not(copymode < 0 and ta != tb))
                 self.assert_(not a)
                 self.assert_(not b)
 


From arigo at codespeak.net  Sun Oct 31 10:58:13 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 10:58:13 +0100 (CET)
Subject: [pypy-svn] r78581 - pypy/branch/smalllong
Message-ID: <20101031095813.DCEC7282BDA@codespeak.net>

Author: arigo
Date: Sun Oct 31 10:58:12 2010
New Revision: 78581

Added:
   pypy/branch/smalllong/
      - copied from r78580, pypy/trunk/
Log:
Implement "small longs", i.e. longs implemented as an r_longlong or
as two r_longs (undecided so far).



From hakanardo at codespeak.net  Sun Oct 31 11:11:57 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 11:11:57 +0100 (CET)
Subject: [pypy-svn] r78582 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101031101157.5EFBB282BDD@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 11:11:55 2010
New Revision: 78582

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed bound tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Sun Oct 31 11:11:55 2010
@@ -78,7 +78,7 @@
                     args.append(self.getvalue(arg).force_box())
                 newop.initarglist(args + inputargs[len(args):])
 
-            #print 'P: ', str(newop)
+            print 'P: ', str(newop)
             self.emit_operation(newop)
 
         jmp = self.optimizer.newoperations[-1]

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sun Oct 31 11:11:55 2010
@@ -3232,7 +3232,7 @@
         guard_true(i1) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i2 = int_add_ovf(i0, 10)
         guard_no_overflow() []
@@ -3240,7 +3240,12 @@
         guard_true(i3) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        i2 = int_add(i0, 10)
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lt_sub(self):
         ops = """
@@ -3252,14 +3257,18 @@
         guard_true(i3) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_lt(i0, 4)
         guard_true(i1) []
         i2 = int_sub(i0, 10)
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lt_sub_before(self):
         ops = """
@@ -3271,14 +3280,18 @@
         guard_true(i1) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i2 = int_sub(i0, 10)
         i3 = int_lt(i2, -5)
         guard_true(i3) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_ltle(self):
         ops = """
@@ -3289,13 +3302,17 @@
         guard_true(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_lt(i0, 4)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lelt(self):
         ops = """
@@ -3306,13 +3323,17 @@
         guard_true(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_le(i0, 4)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_gt(self):
         ops = """
@@ -3323,13 +3344,17 @@
         guard_true(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_gt(i0, 5)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_gtge(self):
         ops = """
@@ -3340,13 +3365,17 @@
         guard_true(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_gt(i0, 5)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_gegt(self):
         ops = """
@@ -3357,13 +3386,17 @@
         guard_true(i2) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_ge(i0, 5)
         guard_true(i1) []
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_ovf(self):
         ops = """
@@ -3376,7 +3409,7 @@
         guard_no_overflow() []
         jump(i3)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_ge(i0, 0)
         guard_true(i1) []
@@ -3385,7 +3418,14 @@
         i3 = int_add(i0, 1)
         jump(i3)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        i2 = int_lt(i0, 10)
+        guard_true(i2) []
+        i3 = int_add(i0, 1)
+        jump(i3)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_arraylen(self):
         ops = """
@@ -3712,14 +3752,7 @@
         i2 = int_add(i0, 3)
         jump(i2)
         """
-        expected = """
-        [i0]
-        i1 = int_ne(i0, 7)
-        guard_false(i1) []
-        jump(10)
-
-        """
-        self.optimize_loop(ops, expected)
+        py.test.raises(InvalidLoop, self.optimize_loop, ops, ops)
 
     def test_bound_ne_const_not(self):
         ops = """
@@ -3747,13 +3780,17 @@
         guard_true(i2) []
         jump(i0, i1)
         """
-        expected = """
+        preamble = """
         [i0, i1]
         i2 = int_lt(i0, 7)
         guard_true(i2) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0, i1]
+        jump(i0, i1)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_lege_const(self):
         ops = """
@@ -3765,16 +3802,7 @@
         i3 = int_add(i0, 3)
         jump(i3)
         """
-        expected = """
-        [i0]
-        i1 = int_ge(i0, 7)
-        guard_true(i1) []
-        i2 = int_le(i0, 7)
-        guard_true(i2) []
-        jump(10)
-
-        """
-        self.optimize_loop(ops, expected)
+        py.test.raises(InvalidLoop, self.optimize_loop, ops, ops)
 
     def test_mul_ovf(self):
         ops = """


From hakanardo at codespeak.net  Sun Oct 31 11:18:50 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 11:18:50 +0100 (CET)
Subject: [pypy-svn] r78583 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101031101850.61F625080B@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 11:18:48 2010
New Revision: 78583

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(armin, cfbolz, hakanardo) Dont invalidate cached on set*_raw

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py	Sun Oct 31 11:18:48 2010
@@ -107,7 +107,9 @@
             return
         opnum = op.getopnum()
         if (opnum == rop.SETFIELD_GC or
+            opnum == rop.SETFIELD_RAW or
             opnum == rop.SETARRAYITEM_GC or
+            opnum == rop.SETARRAYITEM_RAW or
             opnum == rop.DEBUG_MERGE_POINT):
             return
         assert opnum != rop.CALL_PURE

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Sun Oct 31 11:18:48 2010
@@ -78,7 +78,7 @@
                     args.append(self.getvalue(arg).force_box())
                 newop.initarglist(args + inputargs[len(args):])
 
-            print 'P: ', str(newop)
+            #print 'P: ', str(newop)
             self.emit_operation(newop)
 
         jmp = self.optimizer.newoperations[-1]

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sun Oct 31 11:18:48 2010
@@ -3552,12 +3552,14 @@
         jump(p4, p7, i30)
         """
         expected = """
-        [p4, p7, i30, i2]
-        i33 = int_add(i2, i30)
-        setfield_raw(p7, i33, descr=nextdescr)        
-        jump(p4, p7, i30, i33)
+        [p4, p7, i30]
+        i1 = getfield_raw(p7, descr=nextdescr)
+        i2 = int_add(i1, i30)
+        setfield_raw(p7, 7, descr=nextdescr)
+        setfield_raw(p7, i2, descr=nextdescr)
+        jump(p4, p7, i30)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_loop(ops, expected, ops)
 
     def test_pure(self):
         ops = """


From hakanardo at codespeak.net  Sun Oct 31 11:29:03 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 11:29:03 +0100 (CET)
Subject: [pypy-svn] r78584 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031102903.086F6282B90@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 11:29:02 2010
New Revision: 78584

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(cfbolz, hakanardo) Test that caches are not invalidated on setarrayitem_raw

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sun Oct 31 11:29:02 2010
@@ -3544,6 +3544,7 @@
         ops = """
         [p4, p7, i30]
         p16 = getfield_gc(p4, descr=valuedescr)
+        p17 = getarrayitem_gc(p4, 1, descr=arraydescr)        
         guard_value(p16, ConstPtr(myptr), descr=) []
         i1 = getfield_raw(p7, descr=nextdescr)
         i2 = int_add(i1, i30)
@@ -3561,6 +3562,28 @@
         """
         self.optimize_loop(ops, expected, ops)
 
+    def test_setgetarrayitem_raw(self):
+        ops = """
+        [p4, p7, i30]
+        p16 = getfield_gc(p4, descr=valuedescr)
+        guard_value(p16, ConstPtr(myptr), descr=) []
+        p17 = getarrayitem_gc(p4, 1, descr=arraydescr)
+        i1 = getarrayitem_raw(p7, 1, descr=arraydescr)
+        i2 = int_add(i1, i30)
+        setarrayitem_raw(p7, 1, 7, descr=arraydescr)
+        setarrayitem_raw(p7, 1, i2, descr=arraydescr)
+        jump(p4, p7, i30)
+        """
+        expected = """
+        [p4, p7, i30]
+        i1 = getarrayitem_raw(p7, 1, descr=arraydescr)
+        i2 = int_add(i1, i30)
+        setarrayitem_raw(p7, 1, 7, descr=arraydescr)
+        setarrayitem_raw(p7, 1, i2, descr=arraydescr)
+        jump(p4, p7, i30)
+        """
+        self.optimize_loop(ops, expected, ops)
+
     def test_pure(self):
         ops = """
         [p42]
@@ -3672,7 +3695,7 @@
         guard_true(i4) []
         jump(i0)
         """
-        expected = """
+        preamble = """
         [i0]
         i1 = int_sub_ovf(1, i0)
         guard_no_overflow() []
@@ -3681,7 +3704,11 @@
         i3 = int_sub(1, i0)
         jump(i0)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0]
+        jump(i0)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_bound_eq(self):
         ops = """


From arigo at codespeak.net  Sun Oct 31 11:30:02 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 11:30:02 +0100 (CET)
Subject: [pypy-svn] r78585 - in pypy/branch/smalllong/pypy: config
	objspace/std objspace/std/test
Message-ID: <20101031103002.56848282B90@codespeak.net>

Author: arigo
Date: Sun Oct 31 11:30:00 2010
New Revision: 78585

Added:
   pypy/branch/smalllong/pypy/objspace/std/smalllongobject.py
   pypy/branch/smalllong/pypy/objspace/std/test/test_smalllongobject.py
Modified:
   pypy/branch/smalllong/pypy/config/pypyoption.py
   pypy/branch/smalllong/pypy/objspace/std/longobject.py
   pypy/branch/smalllong/pypy/objspace/std/longtype.py
   pypy/branch/smalllong/pypy/objspace/std/objspace.py
   pypy/branch/smalllong/pypy/objspace/std/strutil.py
Log:
In-progress.


Modified: pypy/branch/smalllong/pypy/config/pypyoption.py
==============================================================================
--- pypy/branch/smalllong/pypy/config/pypyoption.py	(original)
+++ pypy/branch/smalllong/pypy/config/pypyoption.py	Sun Oct 31 11:30:00 2010
@@ -194,6 +194,9 @@
         IntOption("prebuiltintto", "highest integer which is prebuilt",
                   default=100, cmdline="--prebuiltintto"),
 
+        BoolOption("withsmalllong", "use a version of 'long' in a C long long",
+                   default=False),
+
         BoolOption("withstrjoin", "use strings optimized for addition",
                    default=False),
 

Modified: pypy/branch/smalllong/pypy/objspace/std/longobject.py
==============================================================================
--- pypy/branch/smalllong/pypy/objspace/std/longobject.py	(original)
+++ pypy/branch/smalllong/pypy/objspace/std/longobject.py	Sun Oct 31 11:30:00 2010
@@ -60,6 +60,12 @@
 
 registerimplementation(W_LongObject)
 
+def newbigint(space, w_longtype, bigint):
+    w_obj = space.allocate_instance(W_LongObject, w_longtype)
+    W_LongObject.__init__(w_obj, bigint)
+    return w_obj
+
+
 # bool-to-long
 def delegate_Bool2Long(space, w_bool):
     return W_LongObject(rbigint.frombool(space.is_true(w_bool)))

Modified: pypy/branch/smalllong/pypy/objspace/std/longtype.py
==============================================================================
--- pypy/branch/smalllong/pypy/objspace/std/longtype.py	(original)
+++ pypy/branch/smalllong/pypy/objspace/std/longtype.py	Sun Oct 31 11:30:00 2010
@@ -4,45 +4,48 @@
 from pypy.objspace.std.strutil import string_to_bigint, ParseStringError
 
 def descr__new__(space, w_longtype, w_x=0, w_base=gateway.NoneNotWrapped):
-    from pypy.objspace.std.longobject import W_LongObject
+    from pypy.objspace.std.longobject import W_LongObject, newbigint
+    from pypy.rlib.rbigint import rbigint
+    if space.config.objspace.std.withsmalllong:
+        from pypy.objspace.std.smalllongobject import W_SmallLongObject
+    else:
+        W_SmallLongObject = None
     w_value = w_x     # 'x' is the keyword argument name in CPython
     if w_base is None:
         # check for easy cases
-        if type(w_value) is W_LongObject:
-            bigint = w_value.num
+        if (W_SmallLongObject and type(w_value) is W_SmallLongObject
+            and space.is_w(w_longtype, space.w_long)):
+            return w_value
+        elif type(w_value) is W_LongObject:
+            return newbigint(space, w_longtype, w_value.num)
         elif space.is_true(space.isinstance(w_value, space.w_str)):
-            try:
-                bigint = string_to_bigint(space.str_w(w_value))
-            except ParseStringError, e:
-                raise OperationError(space.w_ValueError,
-                                     space.wrap(e.msg))
+            return string_to_w_long(space, w_longtype, space.str_w(w_value))
         elif space.is_true(space.isinstance(w_value, space.w_unicode)):
-            try:
-                if space.config.objspace.std.withropeunicode:
-                    from pypy.objspace.std.ropeunicodeobject import unicode_to_decimal_w
-                else:
-                    from pypy.objspace.std.unicodeobject import unicode_to_decimal_w
-                bigint = string_to_bigint(unicode_to_decimal_w(space, w_value))
-            except ParseStringError, e:
-                raise OperationError(space.w_ValueError,
-                                     space.wrap(e.msg))
+            if space.config.objspace.std.withropeunicode:
+                from pypy.objspace.std.ropeunicodeobject import unicode_to_decimal_w
+            else:
+                from pypy.objspace.std.unicodeobject import unicode_to_decimal_w
+            return string_to_w_long(space, w_longtype,
+                                    unicode_to_decimal_w(space, w_value))
         else:
             # otherwise, use the __long__() method
             w_obj = space.long(w_value)
             # 'long(x)' should return whatever x.__long__() returned
             if space.is_w(w_longtype, space.w_long):
                 return w_obj
-            if space.is_true(space.isinstance(w_obj, space.w_long)):
-                assert isinstance(w_obj, W_LongObject)  # XXX this could fail!
-                # XXX find a way to do that even if w_obj is not a W_LongObject
+            # the following is all for the 'subclass_of_long(x)' case
+            if W_SmallLongObject and isinstance(w_obj, W_SmallLongObject):
+                bigint = w_obj.as_bigint()
+            elif isinstance(w_obj, W_LongObject):
                 bigint = w_obj.num
             elif space.is_true(space.isinstance(w_obj, space.w_int)):
                 from pypy.rlib.rbigint import rbigint
-                intval = space.int_w(w_obj)
-                bigint = rbigint.fromint(intval)
+                bigint = rbigint.fromint(space.int_w(w_obj))
             else:
                 raise OperationError(space.w_ValueError,
-                                    space.wrap("value can't be converted to long"))
+                                space.wrap("value can't be converted to long"))
+            return newbigint(space, w_longtype, bigint)
+    #
     else:
         base = space.int_w(w_base)
 
@@ -56,15 +59,25 @@
                 raise OperationError(space.w_TypeError,
                                      space.wrap("long() can't convert non-string "
                                                 "with explicit base"))
+        return string_to_w_long(space, w_longtype, s, base)
+
+
+def string_to_w_long(space, w_longtype, s, base=10):
+    try:
+        bigint = string_to_bigint(s, base)
+    except ParseStringError, e:
+        raise OperationError(space.w_ValueError,
+                             space.wrap(e.msg))
+    if (space.config.objspace.std.withsmalllong
+        and space.is_w(w_longtype, space.w_long)):
         try:
-            bigint = string_to_bigint(s, base)
-        except ParseStringError, e:
-            raise OperationError(space.w_ValueError,
-                                 space.wrap(e.msg))
-
-    w_obj = space.allocate_instance(W_LongObject, w_longtype)
-    W_LongObject.__init__(w_obj, bigint)
-    return w_obj
+            longlong = bigint.tolonglong()
+        except OverflowError:
+            pass
+        else:
+            from pypy.objspace.std.smalllongobject import W_SmallLongObject
+            return W_SmallLongObject(longlong)
+    return newbigint(space, w_longtype, bigint)
 
 # ____________________________________________________________
 

Modified: pypy/branch/smalllong/pypy/objspace/std/objspace.py
==============================================================================
--- pypy/branch/smalllong/pypy/objspace/std/objspace.py	(original)
+++ pypy/branch/smalllong/pypy/objspace/std/objspace.py	Sun Oct 31 11:30:00 2010
@@ -176,6 +176,7 @@
             #print 'wrapping', x, '->', w_result
             return w_result
         if isinstance(x, base_int):
+            xxxxxxxxx
             return W_LongObject.fromrarith_int(x)
 
         # _____ below here is where the annotator should not get _____
@@ -198,6 +199,16 @@
         # The following cases are even stranger.
         # Really really only for tests.
         if type(x) is long:
+            if self.config.objspace.std.withsmalllong:
+                from pypy.rlib.rarithmetic import r_longlong
+                try:
+                    rx = r_longlong(x)
+                except OverflowError:
+                    pass
+                else:
+                    from pypy.objspace.std.smalllongobject import \
+                                                   W_SmallLongObject
+                    return W_SmallLongObject(rx)
             return W_LongObject.fromlong(x)
         if isinstance(x, slice):
             return W_SliceObject(self.wrap(x.start),
@@ -261,6 +272,9 @@
         return W_ComplexObject(realval, imagval)
 
     def newlong(self, val): # val is an int
+        if self.config.objspace.std.withsmalllong:
+            from pypy.objspace.std.smalllongobject import W_SmallLongObject
+            return W_SmallLongObject.fromint(self, val)
         return W_LongObject.fromint(self, val)
 
     def newtuple(self, list_w):

Added: pypy/branch/smalllong/pypy/objspace/std/smalllongobject.py
==============================================================================
--- (empty file)
+++ pypy/branch/smalllong/pypy/objspace/std/smalllongobject.py	Sun Oct 31 11:30:00 2010
@@ -0,0 +1,19 @@
+"""
+Implementation of 'small' longs, stored as a C 'long long' value.
+Useful for 32-bit applications manipulating 64-bit values.
+"""
+from pypy.objspace.std.model import registerimplementation, W_Object
+from pypy.rlib.rarithmetic import r_longlong
+
+
+class W_SmallLongObject(W_Object):
+    from pypy.objspace.std.longtype import long_typedef as typedef
+
+    def __init__(w_self, value):
+        assert isinstance(value, r_longlong)
+        w_self._longlong = value
+
+    def as_bigint(w_self):
+        xxx
+
+registerimplementation(W_SmallLongObject)

Modified: pypy/branch/smalllong/pypy/objspace/std/strutil.py
==============================================================================
--- pypy/branch/smalllong/pypy/objspace/std/strutil.py	(original)
+++ pypy/branch/smalllong/pypy/objspace/std/strutil.py	Sun Oct 31 11:30:00 2010
@@ -3,7 +3,7 @@
 """
 
 from pypy.rlib.rarithmetic import ovfcheck, break_up_float, parts_to_float,\
-     INFINITY, NAN
+     INFINITY, NAN, r_longlong
 from pypy.rlib.rbigint import rbigint, parse_digit_string
 from pypy.interpreter.error import OperationError
 import math

Added: pypy/branch/smalllong/pypy/objspace/std/test/test_smalllongobject.py
==============================================================================
--- (empty file)
+++ pypy/branch/smalllong/pypy/objspace/std/test/test_smalllongobject.py	Sun Oct 31 11:30:00 2010
@@ -0,0 +1,12 @@
+from pypy.objspace.std.test import test_longobject
+from pypy.conftest import gettestobjspace
+
+
+class AppTestSmallLong(test_longobject.AppTestLong):
+    def setup_class(cls):
+        cls.space = gettestobjspace(**{"objspace.std.withsmalllong": True})
+
+    def test_sl_simple(self):
+        import __pypy__
+        s = __pypy__.internal_repr(5L)
+        assert 'SmallLong' in s


From hakanardo at codespeak.net  Sun Oct 31 11:41:16 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 11:41:16 +0100 (CET)
Subject: [pypy-svn] r78586 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031104116.4ED4A282B9D@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 11:41:14 2010
New Revision: 78586

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sun Oct 31 11:41:14 2010
@@ -3851,7 +3851,7 @@
         guard_true(i8) []
         jump(i0, i1)
         """
-        expected = """
+        preamble = """
         [i0, i1]
         i2 = int_and(i0, 255)
         i3 = int_lt(i1, 5)
@@ -3863,7 +3863,11 @@
         guard_true(i8) []
         jump(i0, i1)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [i0, i1]
+        jump(i0, i1)
+        """
+        self.optimize_loop(ops, expected, preamble)
 
     def test_mul_ovf_before(self):
         ops = """
@@ -3880,7 +3884,7 @@
         guard_false(i6) []
         jump(i0, i1)
         """
-        expected = """
+        preamble = """
         [i0, i1]
         i2 = int_and(i0, 255)
         i22 = int_add(i2, 1)
@@ -3890,9 +3894,18 @@
         guard_true(i4) []
         i5 = int_gt(i3, 2)
         guard_true(i5) []
-        jump(i0, i1)
+        jump(i0, i1, i22)
+        """
+        expected = """
+        [i0, i1, i22]
+        i3 = int_mul(i22, i1)
+        i4 = int_lt(i3, 10)
+        guard_true(i4) []
+        i5 = int_gt(i3, 2)
+        guard_true(i5) []
+        jump(i0, i1, i22)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_loop(ops, expected, preamble)
 
     def test_sub_ovf_before(self):
         ops = """
@@ -3910,7 +3923,7 @@
         guard_false(i7) []
         jump(i0, i1)
         """
-        expected = """
+        preamble = """
         [i0, i1]
         i2 = int_and(i0, 255)
         i3 = int_sub_ovf(i2, i1)
@@ -3919,9 +3932,18 @@
         guard_true(i4) []
         i5 = int_ge(i3, 2)
         guard_true(i5) []
-        jump(i0, i1)
+        jump(i0, i1, i2)
+        """
+        expected = """
+        [i0, i1, i2]
+        i3 = int_sub(i2, i1)
+        i4 = int_le(i3, 10)
+        guard_true(i4) []
+        i5 = int_ge(i3, 2)
+        guard_true(i5) []
+        jump(i0, i1, i2)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_loop(ops, expected, preamble)
 
     # ----------
     def optimize_strunicode_loop(self, ops, optops):


From cfbolz at codespeak.net  Sun Oct 31 11:43:28 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 11:43:28 +0100 (CET)
Subject: [pypy-svn] r78587 -
	pypy/branch/mapdict-without-jit/pypy/objspace/std
Message-ID: <20101031104328.E21D95080B@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 11:43:27 2010
New Revision: 78587

Modified:
   pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
Log:
the jit shouldn't look into this function



Modified: pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py
==============================================================================
--- pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	(original)
+++ pypy/branch/mapdict-without-jit/pypy/objspace/std/mapdict.py	Sun Oct 31 11:43:27 2010
@@ -48,6 +48,7 @@
         else:
             return self._index(selector)
 
+    @jit.dont_look_inside
     def _index_cache(self, selector):
         space = self.space
         cache = space.fromcache(IndexCache)


From arigo at codespeak.net  Sun Oct 31 12:03:06 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 12:03:06 +0100 (CET)
Subject: [pypy-svn] r78588 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101031110306.F2420282BDA@codespeak.net>

Author: arigo
Date: Sun Oct 31 12:03:03 2010
New Revision: 78588

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
Fix VStringConcatValue: it cannot easily pass the 'lengthbox'
around the unrolled loop, so we reset it to None and (re-)compute
it lazily.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py	Sun Oct 31 12:03:03 2010
@@ -180,12 +180,22 @@
 class VStringConcatValue(VAbstractStringValue):
     """The concatenation of two other strings."""
 
-    def setup(self, left, right, lengthbox):
+    lengthbox = None     # or the computed length
+
+    def setup(self, left, right):
         self.left = left
         self.right = right
-        self.lengthbox = lengthbox
 
-    def getstrlen(self, _, mode):
+    def getstrlen(self, newoperations, mode):
+        if self.lengthbox is None:
+            len1box = self.left.getstrlen(newoperations, mode)
+            if len1box is None:
+                return None
+            len2box = self.right.getstrlen(newoperations, mode)
+            if len2box is None:
+                return None
+            self.lengthbox = _int_add(newoperations, len1box, len2box)
+            # ^^^ may still be None, if newoperations is None
         return self.lengthbox
 
     @specialize.arg(1)
@@ -223,6 +233,7 @@
         if self.box is None:
             self.left.enum_forced_boxes(boxes, already_seen)
             self.right.enum_forced_boxes(boxes, already_seen)
+            self.lengthbox = None
         else:
             boxes.append(self.box)
 
@@ -322,6 +333,8 @@
             return ConstInt(box1.value + box2.value)
     elif isinstance(box2, ConstInt) and box2.value == 0:
         return box1
+    if newoperations is None:
+        return None
     resbox = BoxInt()
     newoperations.append(ResOperation(rop.INT_ADD, [box1, box2], resbox))
     return resbox
@@ -481,11 +494,8 @@
         vleft.ensure_nonnull()
         vright.ensure_nonnull()
         newoperations = self.optimizer.newoperations
-        len1box = vleft.getstrlen(newoperations, mode)
-        len2box = vright.getstrlen(newoperations, mode)
-        lengthbox = _int_add(newoperations, len1box, len2box)
         value = self.make_vstring_concat(op.result, op, mode)
-        value.setup(vleft, vright, lengthbox)
+        value.setup(vleft, vright)
         return True
 
     def opt_call_stroruni_STR_SLICE(self, op, mode):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sun Oct 31 12:03:03 2010
@@ -3990,7 +3990,7 @@
         jump(p2, p3)
         """
         expected = """
-        [p1, p2]
+        [p2, p1]
         i1 = strlen(p1)
         i2 = strlen(p2)
         i3 = int_add(i1, i2)
@@ -4000,7 +4000,7 @@
         i5 = strlen(p2)
         i6 = int_add(i4, i5)      # will be killed by the backend
         copystrcontent(p2, p3, 0, i4, i5)
-        jump(p2, p3)
+        jump(p3, p2)
         """
         self.optimize_strunicode_loop(ops, expected)
 


From cfbolz at codespeak.net  Sun Oct 31 12:09:29 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 12:09:29 +0100 (CET)
Subject: [pypy-svn] r78589 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031110929.3E9D9282BDA@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 12:09:26 2010
New Revision: 78589

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
this is needed for the tests to make sense


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sun Oct 31 12:09:26 2010
@@ -3946,12 +3946,13 @@
         self.optimize_loop(ops, expected, preamble)
 
     # ----------
-    def optimize_strunicode_loop(self, ops, optops):
+    def optimize_strunicode_loop(self, ops, optops, preamble):
         # check with the arguments passed in
-        self.optimize_loop(ops, optops)
+        self.optimize_loop(ops, optops, preamble)
         # check with replacing 'str' with 'unicode' everywhere
-        self.optimize_loop(ops.replace('str','unicode').replace('s"', 'u"'),
-                           optops.replace('str','unicode').replace('s"', 'u"'))
+        def r(s):
+            return r.replace('str','unicode').replace('s"', 'u"')
+        self.optimize_loop(r(ops), r(optops), r(preamble))
 
     def test_newstr_1(self):
         ops = """


From cfbolz at codespeak.net  Sun Oct 31 12:11:03 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 12:11:03 +0100 (CET)
Subject: [pypy-svn] r78590 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101031111103.9BBB8282BDD@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 12:11:01 2010
New Revision: 78590

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
Log:
remove the now useless argument to setup


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	Sun Oct 31 12:11:01 2010
@@ -7,7 +7,7 @@
 from pypy.jit.metainterp.optimizeopt.string import OptString
 from pypy.jit.metainterp.optimizeopt.unroll import OptUnroll
 
-def optimize_loop_1(metainterp_sd, loop, not_a_bridge=True):
+def optimize_loop_1(metainterp_sd, loop, unroll=True):
     """Optimize loop.operations to remove internal overheadish operations. 
     """
     optimizations = [OptIntBounds(),
@@ -17,9 +17,9 @@
                      OptHeap(),
                      OptFfiCall(),
                     ]
-    if not_a_bridge:
+    if unroll:
         optimizations.insert(0, OptUnroll())
-    optimizer = Optimizer(metainterp_sd, loop, optimizations, not_a_bridge)
+    optimizer = Optimizer(metainterp_sd, loop, optimizations)
     optimizer.propagate_all_forward()
 
 def optimize_bridge_1(metainterp_sd, bridge):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/intbounds.py	Sun Oct 31 12:11:01 2010
@@ -9,7 +9,7 @@
     """Keeps track of the bounds placed on integers by the guards and
        remove redundant guards"""
 
-    def setup(self, virtuals):
+    def setup(self):
         self.posponedop = None
         self.nextop = None
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	Sun Oct 31 12:11:01 2010
@@ -190,7 +190,7 @@
         op = ResOperation(opnum, args, result)
         self.optimizer.pure_operations[self.optimizer.make_args_key(op)] = op
 
-    def setup(self, not_a_bridge):
+    def setup(self):
         pass
 
     def force_at_end_of_preamble(self):
@@ -199,7 +199,7 @@
 
 class Optimizer(Optimization):
 
-    def __init__(self, metainterp_sd, loop, optimizations=None, not_a_bridge=True):
+    def __init__(self, metainterp_sd, loop, optimizations=None):
         self.metainterp_sd = metainterp_sd
         self.cpu = metainterp_sd.cpu
         self.loop = loop
@@ -220,7 +220,7 @@
             optimizations[-1].next_optimization = self
             for o in optimizations:
                 o.optimizer = self
-                o.setup(not_a_bridge)
+                o.setup()
         else:
             self.first_optimization = self
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Sun Oct 31 12:11:01 2010
@@ -8,8 +8,7 @@
     become the preamble or entry bridge (don't think there is a
     distinction anymore)"""
     
-    def setup(self, not_a_bridge):
-        assert not_a_bridge
+    def setup(self):
         self.cloned_operations = []
         for op in self.optimizer.loop.operations:
             self.cloned_operations.append(op.clone())
@@ -20,12 +19,13 @@
         if op.getopnum() == rop.JUMP:
             self.force_at_end_of_preamble()
             loop = self.optimizer.loop
+            assert op.getdescr() is loop.token
             loop.preamble.operations = self.optimizer.newoperations
             self.optimizer.newoperations = []
             jump_args = op.getarglist()
             op.initarglist([])
-            # Exceptions not caught in one itteration should not propagate to the next
-            self.optimizer.exception_might_have_happened = False 
+            # Exceptions not caught in one iteration should not propagate to the next
+            self.optimizer.exception_might_have_happened = False
             inputargs = self.inline(self.cloned_operations,
                                     loop.inputargs, jump_args)
             loop.inputargs = inputargs
@@ -60,6 +60,8 @@
                 if not isinstance(a, Const):
                     inputargs.append(a)
 
+        # this loop is equivalent to the main optimization loop in
+        # Optimizer.propagate_all_forward
         for newop in loop_operations:
             newop.initarglist([self.inline_arg(a) for a in newop.getarglist()])
             if newop.result:


From david at codespeak.net  Sun Oct 31 13:01:36 2010
From: david at codespeak.net (david at codespeak.net)
Date: Sun, 31 Oct 2010 13:01:36 +0100 (CET)
Subject: [pypy-svn] r78592 - in
	pypy/branch/arm-backend/pypy/jit/backend/arm: . helper
Message-ID: <20101031120136.4FB96282B90@codespeak.net>

Author: david
Date: Sun Oct 31 13:01:34 2010
New Revision: 78592

Added:
   pypy/branch/arm-backend/pypy/jit/backend/arm/helper/
   pypy/branch/arm-backend/pypy/jit/backend/arm/helper/__init__.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/helper/assembler.py
Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
   pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
Log:
Implement unary int operations and perform some refactoring on opassembler

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Sun Oct 31 13:01:34 2010
@@ -12,12 +12,14 @@
 from pypy.rpython.lltypesystem import lltype, rffi, llmemory
 from pypy.jit.backend.arm.opassembler import (GuardOpAssembler,
                                                 IntOpAsslember,
-                                                OpAssembler)
+                                                OpAssembler,
+                                                UnaryIntOpAssembler)
 # XXX Move to llsupport
 from pypy.jit.backend.x86.support import values_array
 
 
-class AssemblerARM(GuardOpAssembler, IntOpAsslember, OpAssembler):
+class AssemblerARM(GuardOpAssembler, IntOpAsslember,
+                    OpAssembler, UnaryIntOpAssembler):
 
     def __init__(self, cpu, failargs_limit=1000):
         self.mc = ARMv7Builder()
@@ -182,7 +184,6 @@
             opnum = op.getopnum()
             fcond = self.operations[opnum](self, op, regalloc, fcond)
         self.gen_func_epilog()
-        print inputargs, operations
         if self._debug_asm:
             self._dump_trace('loop.asm')
         print 'Done assembling'
@@ -210,6 +211,10 @@
     def _dump_trace(self, name):
         self.mc._dump_trace(name)
 
+    def _check_imm_arg(self, arg, size=0xFF):
+        #XXX check ranges for different operations
+        return isinstance(arg, ConstInt) and arg.getint() <= size and arg.getint() > 0
+
     def patch_trace(self, faildescr, bridge_addr):
         # XXX make sure there is enough space at patch target
         fcond = faildescr._arm_guard_cond

Added: pypy/branch/arm-backend/pypy/jit/backend/arm/helper/__init__.py
==============================================================================

Added: pypy/branch/arm-backend/pypy/jit/backend/arm/helper/assembler.py
==============================================================================
--- (empty file)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/helper/assembler.py	Sun Oct 31 13:01:34 2010
@@ -0,0 +1,82 @@
+from pypy.jit.backend.arm import conditions as c
+from pypy.jit.backend.arm import registers as r
+from pypy.jit.metainterp.history import ConstInt, BoxInt, Box
+
+def gen_emit_op_unary_cmp(true_cond, false_cond):
+    def f(self, op, regalloc, fcond):
+        arg = op.getarg(0)
+        reg = self._put_in_reg(arg, regalloc)
+        res = regalloc.try_allocate_reg(op.result)
+        self.mc.CMP_ri(reg.value, 0)
+        self.mc.MOV_ri(res.value, 1, true_cond)
+        self.mc.MOV_ri(res.value, 0, false_cond)
+        regalloc.possibly_free_var(reg)
+        regalloc.possibly_free_var(res)
+        return fcond
+    return f
+
+def gen_emit_op_ri(opname, imm_size=0xFF, commutative=True):
+    def f(self, op, regalloc, fcond):
+        ri_op = getattr(self.mc, '%s_ri' % opname)
+        rr_op = getattr(self.mc, '%s_rr' % opname)
+
+        arg0 = op.getarg(0)
+        arg1 = op.getarg(1)
+        res = regalloc.try_allocate_reg(op.result)
+        if (commutative
+                and self._check_imm_arg(arg0, imm_size)
+                and not isinstance(arg1, ConstInt)):
+            reg = regalloc.try_allocate_reg(arg1)
+            ri_op(res.value, reg.value, imm=arg0.getint(), cond=fcond)
+        elif self._check_imm_arg(arg1, imm_size) and not isinstance(arg0, ConstInt):
+            reg = regalloc.try_allocate_reg(arg0)
+            ri_op(res.value, reg.value, imm=arg1.getint(), cond=fcond)
+        else:
+            reg = self._put_in_reg(arg0, regalloc)
+            reg2 = self._put_in_reg(arg1, regalloc)
+            rr_op(res.value, reg.value, reg2.value)
+            regalloc.possibly_free_var(reg2)
+
+        regalloc.possibly_free_var(res)
+        regalloc.possibly_free_var(reg)
+        return fcond
+    return f
+
+def gen_emit_op_by_helper_call(opname):
+    def f(self, op, regalloc, fcond):
+        arg1 = regalloc.make_sure_var_in_reg(op.getarg(0), selected_reg=r.r0)
+        arg2 = regalloc.make_sure_var_in_reg(op.getarg(1), selected_reg=r.r1)
+        assert arg1 == r.r0
+        assert arg2 == r.r1
+        res = regalloc.try_allocate_reg(op.result)
+        getattr(self.mc, opname)(fcond)
+        self.mc.MOV_rr(res.value, r.r0.value, cond=fcond)
+        regalloc.possibly_free_vars_for_op(op)
+        return fcond
+    return f
+
+def gen_emit_cmp_op(condition, inverse=False):
+    def f(self, op, regalloc, fcond):
+        assert fcond == c.AL
+        if not inverse:
+            arg0 = op.getarg(0)
+            arg1 = op.getarg(1)
+        else:
+            arg0 = op.getarg(1)
+            arg1 = op.getarg(0)
+        res = regalloc.try_allocate_reg(op.result)
+        # XXX consider swapping argumentes if arg0 is const
+        if self._check_imm_arg(arg1) and not isinstance(arg0, ConstInt):
+            reg = regalloc.try_allocate_reg(arg0)
+            self.mc.CMP_ri(reg.value, imm=arg1.getint(), cond=fcond)
+        else:
+            reg = self._put_in_reg(arg0, regalloc)
+            reg2 = self._put_in_reg(arg1, regalloc)
+            self.mc.CMP_rr(reg.value, reg2.value)
+            regalloc.possibly_free_var(reg2)
+
+        inv = c.get_opposite_of(condition)
+        self.mc.MOV_ri(res.value, 1, cond=condition)
+        self.mc.MOV_ri(res.value, 0, cond=inv)
+        return condition
+    return f

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/instructions.py	Sun Oct 31 13:01:34 2010
@@ -32,7 +32,8 @@
     #'RRX_ri': {'op1':0x1A, 'op2':0, 'op3':0x3, 'op2cond':'0', 'result':False, 'base':True},
     'ROR_ri': {'op1':0x1A, 'op2':0x0, 'op3':0x3, 'op2cond':'!0', 'result':True, 'base':False},
     #BIC
-    #MVN
+    'MVN_rr': {'op1':0x1E, 'op2':0x0, 'op3':0x0, 'result':True, 'base':False},
+
 }
 
 data_proc_reg_shift_reg = {

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/opassembler.py	Sun Oct 31 13:01:34 2010
@@ -3,6 +3,10 @@
 from pypy.jit.backend.arm import registers as r
 from pypy.jit.backend.arm.arch import (WORD, FUNC_ALIGN, arm_int_div,
                                         arm_int_div_sign, arm_int_mod_sign, arm_int_mod)
+
+from pypy.jit.backend.arm.helper.assembler import (gen_emit_op_by_helper_call,
+                                                    gen_emit_op_unary_cmp,
+                                                    gen_emit_op_ri, gen_emit_cmp_op)
 from pypy.jit.backend.arm.codebuilder import ARMv7Builder, ARMv7InMemoryBuilder
 from pypy.jit.backend.arm.regalloc import ARMRegisterManager
 from pypy.jit.backend.llsupport.regalloc import compute_vars_longevity
@@ -12,95 +16,31 @@
 from pypy.rpython.annlowlevel import llhelper
 from pypy.rpython.lltypesystem import lltype, rffi, llmemory
 
-def gen_emit_op_ri(opname, imm_size=0xFF, commutative=True):
-    def f(self, op, regalloc, fcond):
-        ri_op = getattr(self.mc, '%s_ri' % opname)
-        rr_op = getattr(self.mc, '%s_rr' % opname)
-
-        arg0 = op.getarg(0)
-        arg1 = op.getarg(1)
-        res = regalloc.try_allocate_reg(op.result)
-        if (commutative
-                and self._check_imm_arg(arg0, imm_size)
-                and not isinstance(arg1, ConstInt)):
-            reg = regalloc.try_allocate_reg(arg1)
-            ri_op(res.value, reg.value, imm=arg0.getint(), cond=fcond)
-        elif self._check_imm_arg(arg1, imm_size) and not isinstance(arg0, ConstInt):
-            reg = regalloc.try_allocate_reg(arg0)
-            ri_op(res.value, reg.value, imm=arg1.getint(), cond=fcond)
-        else:
-            reg = self._put_in_reg(arg0, regalloc)
-            reg2 = self._put_in_reg(arg1, regalloc)
-            rr_op(res.value, reg.value, reg2.value)
-            regalloc.possibly_free_var(reg2)
-
-        regalloc.possibly_free_var(res)
-        regalloc.possibly_free_var(reg)
-        return fcond
-    return f
-
-def gen_emit_op_by_helper_call(opname):
-    def f(self, op, regalloc, fcond):
-        arg1 = regalloc.make_sure_var_in_reg(op.getarg(0), selected_reg=r.r0)
-        arg2 = regalloc.make_sure_var_in_reg(op.getarg(1), selected_reg=r.r1)
-        assert arg1 == r.r0
-        assert arg2 == r.r1
-        res = regalloc.try_allocate_reg(op.result)
-        getattr(self.mc, opname)(fcond)
-        self.mc.MOV_rr(res.value, r.r0.value, cond=fcond)
-        regalloc.possibly_free_vars_for_op(op)
-        return fcond
-    return f
-
-def gen_emit_cmp_op(condition, inverse=False):
-    def f(self, op, regalloc, fcond):
-        assert fcond == c.AL
-        if not inverse:
-            arg0 = op.getarg(0)
-            arg1 = op.getarg(1)
-        else:
-            arg0 = op.getarg(1)
-            arg1 = op.getarg(0)
-        res = regalloc.try_allocate_reg(op.result)
-        # XXX consider swapping argumentes if arg0 is const
-        if self._check_imm_arg(arg1) and not isinstance(arg0, ConstInt):
-            reg = regalloc.try_allocate_reg(arg0)
-            self.mc.CMP_ri(reg.value, imm=arg1.getint(), cond=fcond)
-        else:
-            reg = self._put_in_reg(arg0, regalloc)
-            reg2 = self._put_in_reg(arg1, regalloc)
-            self.mc.CMP_rr(reg.value, reg2.value)
-            regalloc.possibly_free_var(reg2)
-
-        inv = c.get_opposite_of(condition)
-        self.mc.MOV_ri(res.value, 1, cond=condition)
-        self.mc.MOV_ri(res.value, 0, cond=inv)
-        return condition
-    return f
-
 class IntOpAsslember(object):
     _mixin_ = True
 
-    # XXX support constants larger than imm
-
     def emit_op_int_add(self, op, regalloc, fcond):
         # assuming only one argument is constant
         res = regalloc.try_allocate_reg(op.result)
-        if isinstance(op.getarg(0), ConstInt) or isinstance(op.getarg(1), ConstInt):
-            if isinstance(op.getarg(1), ConstInt):
-                reg = regalloc.try_allocate_reg(op.getarg(0))
-                arg1 = op.getarg(1)
-            elif isinstance(op.getarg(0), ConstInt):
-                reg = regalloc.try_allocate_reg(op.getarg(1))
-                arg1 = op.getarg(0)
+        a0 = op.getarg(0)
+        a1 = op.getarg(1)
+        imm_a0 = isinstance(a0, ConstInt) and (a0.getint() <= 0xFF or -1 * a0.getint() <= 0xFF)
+        imm_a1 = isinstance(a1, ConstInt) and (a1.getint() <= 0xFF or -1 * a1.getint() <= 0xFF)
+        if imm_a0 or imm_a1:
+            if imm_a1:
+                reg = regalloc.try_allocate_reg(a0)
+                arg1 = a1
+            elif imm_a0:
+                reg = regalloc.try_allocate_reg(a1)
+                arg1 = a0
             value = arg1.getint()
             if value < 0:
                 self.mc.SUB_ri(res.value, reg.value, -1 * value)
             else:
                 self.mc.ADD_ri(res.value, reg.value, value)
         else:
-            r1 = regalloc.try_allocate_reg(op.getarg(0))
-            r2 = regalloc.try_allocate_reg(op.getarg(1))
+            r1 = regalloc.try_allocate_reg(a0)
+            r2 = regalloc.try_allocate_reg(a1)
             self.mc.ADD_rr(res.value, r1.value, r2.value)
 
         regalloc.possibly_free_vars_for_op(op)
@@ -109,17 +49,21 @@
     def emit_op_int_sub(self, op, regalloc, fcond):
         # assuming only one argument is constant
         res = regalloc.try_allocate_reg(op.result)
-        if isinstance(op.getarg(0), ConstInt) or isinstance(op.getarg(1), ConstInt):
-            if isinstance(op.getarg(1), ConstInt):
-                reg = regalloc.try_allocate_reg(op.getarg(0))
-                value = op.getarg(1).getint()
+        a0 = op.getarg(0)
+        a1 = op.getarg(1)
+        imm_a0 = isinstance(a0, ConstInt) and (a0.getint() <= 0xFF or -1 * a0.getint() <= 0xFF)
+        imm_a1 = isinstance(a1, ConstInt) and (a1.getint() <= 0xFF or -1 * a1.getint() <= 0xFF)
+        if imm_a0 or imm_a1:
+            if imm_a1:
+                reg = self._put_in_reg(a0, regalloc)
+                value = a1.getint()
                 if value < 0:
                     self.mc.ADD_ri(res.value, reg.value, -1 * value)
                 else:
                     self.mc.SUB_ri(res.value, reg.value, value)
-            elif isinstance(op.getarg(0), ConstInt):
-                reg = regalloc.try_allocate_reg(op.getarg(1))
-                value = op.getarg(0).getint()
+            elif imm_a0:
+                reg = self._put_in_reg(a1, regalloc)
+                value = a0.getint()
                 if value < 0:
                     self.mc.ADD_ri(res.value, reg.value, -1 * value)
                     self.mc.MVN_rr(res.value, res.value)
@@ -127,8 +71,8 @@
                     # reverse substract ftw
                     self.mc.RSB_ri(res.value, reg.value, value)
         else:
-            r1 = regalloc.try_allocate_reg(op.getarg(0))
-            r2 = regalloc.try_allocate_reg(op.getarg(1))
+            r1 = regalloc.try_allocate_reg(a0)
+            r2 = regalloc.try_allocate_reg(a1)
             self.mc.SUB_rr(res.value, r1.value, r2.value)
 
         regalloc.possibly_free_vars_for_op(op)
@@ -177,10 +121,32 @@
     emit_op_uint_ge = gen_emit_cmp_op(c.LS, inverse=True)
 
 
-    def _check_imm_arg(self, arg, size=0xFF):
-        #XXX check ranges for different operations
-        return isinstance(arg, ConstInt) and arg.getint() <= size and arg.getint() > 0
 
+class UnaryIntOpAssembler(object):
+    emit_op_int_is_true = gen_emit_op_unary_cmp(c.NE, c.EQ)
+    emit_op_int_is_zero = gen_emit_op_unary_cmp(c.EQ, c.NE)
+
+    def emit_op_int_invert(self, op, regalloc, fcond):
+        arg = op.getarg(0)
+        reg = self._put_in_reg(arg, regalloc)
+        res = regalloc.try_allocate_reg(op.result)
+
+        self.mc.MVN_rr(res.value, reg.value)
+        regalloc.possibly_free_var(reg)
+        regalloc.possibly_free_var(res)
+        return fcond
+
+    #XXX check for a better way of doing this
+    def emit_op_int_neg(self, op, regalloc, fcond):
+            arg = op.getarg(0)
+            reg = self._put_in_reg(arg, regalloc)
+            reg2 = self._put_in_reg(ConstInt(-1), regalloc)
+            res = regalloc.try_allocate_reg(op.result)
+            self.mc.MUL(res.value, reg.value, reg2.value)
+            regalloc.possibly_free_var(reg)
+            regalloc.possibly_free_var(reg2)
+            regalloc.possibly_free_var(res)
+            return fcond
 
 class GuardOpAssembler(object):
     _mixin_ = True


From cfbolz at codespeak.net  Sun Oct 31 13:09:28 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 13:09:28 +0100 (CET)
Subject: [pypy-svn] r78593 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: . test
Message-ID: <20101031120928.2B779282B90@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 13:09:26 2010
New Revision: 78593

Added:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/nounroll_optimize.py
      - copied, changed from r78588, pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py
      - copied unchanged from r78588, pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nopspec.py
Removed:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nopspec.py
Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py
Log:
make optimize_nopspec not use unrolling


Copied: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/nounroll_optimize.py (from r78588, pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py)
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimize_nopspec.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/nounroll_optimize.py	Sun Oct 31 13:09:26 2010
@@ -2,8 +2,6 @@
 from pypy.rlib.debug import debug_start, debug_stop
 from pypy.jit.metainterp.optimizeopt import optimize_loop_1, optimize_bridge_1
 
-# XXX this file should probably disable unrolling instead
-
 def optimize_loop(metainterp_sd, old_loop_tokens, loop):
     debug_start("jit-optimize")
     try:
@@ -16,7 +14,7 @@
     metainterp_sd.logger_noopt.log_loop(loop.inputargs, loop.operations)
     if old_loop_tokens:
         return old_loop_tokens[0]
-    optimize_loop_1(metainterp_sd, loop)
+    optimize_loop_1(metainterp_sd, loop, False)
     return None
 
 def optimize_bridge(metainterp_sd, old_loop_tokens, bridge):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py	Sun Oct 31 13:09:26 2010
@@ -4,7 +4,7 @@
 from pypy.jit.metainterp.compile import ResumeGuardDescr
 from pypy.jit.metainterp.compile import ResumeGuardCountersInt
 from pypy.jit.metainterp.compile import compile_tmp_callback
-from pypy.jit.metainterp import optimize_nopspec, jitprof, typesystem, compile
+from pypy.jit.metainterp import nounroll_optimize, jitprof, typesystem, compile
 from pypy.jit.metainterp.test.test_optimizeutil import LLtypeMixin
 from pypy.jit.tool.oparser import parse
 
@@ -38,7 +38,7 @@
         pass
 
 class FakeState:
-    optimize_loop = staticmethod(optimize_nopspec.optimize_loop)
+    optimize_loop = staticmethod(nounroll_optimize.optimize_loop)
     debug_level = 0
 
     def attach_unoptimized_bridge_from_interp(*args):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py	Sun Oct 31 13:09:26 2010
@@ -199,9 +199,9 @@
             self.optimize_loop = simple_optimize.optimize_loop
             self.optimize_bridge = simple_optimize.optimize_bridge
         elif optimizer == OPTIMIZER_NO_PERFECTSPEC:
-            from pypy.jit.metainterp import optimize_nopspec
-            self.optimize_loop = optimize_nopspec.optimize_loop
-            self.optimize_bridge = optimize_nopspec.optimize_bridge
+            from pypy.jit.metainterp import nounroll_optimize
+            self.optimize_loop = nounroll_optimize.optimize_loop
+            self.optimize_bridge = nounroll_optimize.optimize_bridge
         elif optimizer == OPTIMIZER_FULL:
             from pypy.jit.metainterp import optimize
             self.optimize_loop = optimize.optimize_loop


From david at codespeak.net  Sun Oct 31 13:22:53 2010
From: david at codespeak.net (david at codespeak.net)
Date: Sun, 31 Oct 2010 13:22:53 +0100 (CET)
Subject: [pypy-svn] r78594 - pypy/branch/arm-backend/pypy/jit/backend/arm
Message-ID: <20101031122253.45E7836E0C1@codespeak.net>

Author: david
Date: Sun Oct 31 13:22:47 2010
New Revision: 78594

Modified:
   pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
Log:
Improve trace exit path

Modified: pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py
==============================================================================
--- pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	(original)
+++ pypy/branch/arm-backend/pypy/jit/backend/arm/assembler.py	Sun Oct 31 13:22:47 2010
@@ -100,9 +100,10 @@
         self.mc.MOV_rr(r.r0.value, r.lr.value)  # move mem block address, to r0 to pass as
                                     # parameter to next procedure call
         self.mc.MOV_rr(r.r1.value, r.sp.value)  # pass the current stack pointer as second param
-        self.mc.gen_load_int(r.r2.value, rffi.cast(lltype.Signed, decode_registers_addr))
-        self.mc.gen_load_int(r.lr.value, self.mc.curraddr()+self.mc.size_of_gen_load_int+WORD)
-        self.mc.MOV_rr(r.pc.value, r.r2.value)
+
+        self.mc.ADD_ri(r.lr.value, r.pc.value, 4)
+        self.mc.LDR_ri(r.pc.value, r.pc.value, -4)
+        self.mc.write32(rffi.cast(lltype.Signed, decode_registers_addr))
         self.mc.MOV_rr(r.ip.value, r.r0.value)
         self.mc.LDM(r.sp.value, range(12), w=1) # XXX Replace with POP instr. someday
 


From hakanardo at codespeak.net  Sun Oct 31 13:25:56 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 13:25:56 +0100 (CET)
Subject: [pypy-svn] r78595 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101031122556.E1A21282BE3@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 13:25:51 2010
New Revision: 78595

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
(hakanardo, arigo) workaround that dissables string optimisation of the preamble

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/__init__.py	Sun Oct 31 13:25:51 2010
@@ -10,15 +10,18 @@
 def optimize_loop_1(metainterp_sd, loop, unroll=True):
     """Optimize loop.operations to remove internal overheadish operations. 
     """
+    opt_str = OptString()
     optimizations = [OptIntBounds(),
                      OptRewrite(),
                      OptVirtualize(),
-                     OptString(),
+                     opt_str,
                      OptHeap(),
                      OptFfiCall(),
                     ]
     if unroll:
         optimizations.insert(0, OptUnroll())
+        opt_str.enabled = False # FIXME: Workaround to disable string optimisation
+                                # during preamble but to keep it during the loop
     optimizer = Optimizer(metainterp_sd, loop, optimizations)
     optimizer.propagate_all_forward()
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/string.py	Sun Oct 31 13:25:51 2010
@@ -162,7 +162,7 @@
             for value in self._chars:
                 value.get_args_for_fail(modifier)
 
-    def enum_forced_boxes(self, boxes, already_seen):
+    def FIXME_enum_forced_boxes(self, boxes, already_seen):
         key = self.get_key_box()
         if key in already_seen:
             return
@@ -225,7 +225,7 @@
             self.left.get_args_for_fail(modifier)
             self.right.get_args_for_fail(modifier)
 
-    def enum_forced_boxes(self, boxes, already_seen):
+    def FIXME_enum_forced_boxes(self, boxes, already_seen):
         key = self.get_key_box()
         if key in already_seen:
             return
@@ -283,7 +283,7 @@
             self.vstart.get_args_for_fail(modifier)
             self.vlength.get_args_for_fail(modifier)
 
-    def enum_forced_boxes(self, boxes, already_seen):
+    def FIXME_enum_forced_boxes(self, boxes, already_seen):
         key = self.get_key_box()
         if key in already_seen:
             return
@@ -365,7 +365,8 @@
 
 class OptString(optimizer.Optimization):
     "Handling of strings and unicodes."
-
+    enabled = True
+    
     def make_vstring_plain(self, box, source_op, mode):
         vvalue = VStringPlainValue(self.optimizer, box, source_op, mode)
         self.make_equal_to(box, vvalue)
@@ -643,6 +644,10 @@
         self.optimizer.newoperations.append(op)
 
     def propagate_forward(self, op):
+        if not self.enabled:
+            self.emit_operation(op)
+            return
+            
         opnum = op.getopnum()
         for value, func in optimize_ops:
             if opnum == value:
@@ -651,6 +656,9 @@
         else:
             self.emit_operation(op)
 
+    def force_at_end_of_preamble(self):
+        self.enabled = True
+
 optimize_ops = _findall(OptString, 'optimize_')
 
 def _findall_call_oopspec():

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sun Oct 31 13:25:51 2010
@@ -3946,12 +3946,14 @@
         self.optimize_loop(ops, expected, preamble)
 
     # ----------
-    def optimize_strunicode_loop(self, ops, optops, preamble):
+    def optimize_strunicode_loop(self, ops, optops, preamble=None):
+        if not preamble:
+            preamble = ops # FIXME: Force proper testing of preamble
         # check with the arguments passed in
         self.optimize_loop(ops, optops, preamble)
         # check with replacing 'str' with 'unicode' everywhere
         def r(s):
-            return r.replace('str','unicode').replace('s"', 'u"')
+            return s.replace('str','unicode').replace('s"', 'u"')
         self.optimize_loop(r(ops), r(optops), r(preamble))
 
     def test_newstr_1(self):
@@ -3991,7 +3993,7 @@
         jump(p2, p3)
         """
         expected = """
-        [p2, p1]
+        [p1, p2]
         i1 = strlen(p1)
         i2 = strlen(p2)
         i3 = int_add(i1, i2)
@@ -4001,7 +4003,7 @@
         i5 = strlen(p2)
         i6 = int_add(i4, i5)      # will be killed by the backend
         copystrcontent(p2, p3, 0, i4, i5)
-        jump(p3, p2)
+        jump(p2, p3)
         """
         self.optimize_strunicode_loop(ops, expected)
 
@@ -4107,12 +4109,18 @@
         escape(p3)
         jump()
         """
+        preamble = """
+        []
+        p3 = call(0, s"ab", s"cde", descr=strconcatdescr)
+        escape(p3)
+        jump()
+        """
         expected = """
         []
         escape(s"abcde")
         jump()
         """
-        self.optimize_strunicode_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected, preamble)
 
     def test_str_slice_1(self):
         ops = """
@@ -4219,7 +4227,7 @@
         self.optimize_strunicode_loop(ops, expected)
 
     # ----------
-    def optimize_strunicode_loop_extradescrs(self, ops, optops):
+    def optimize_strunicode_loop_extradescrs(self, ops, optops, preamble=None):
         from pypy.jit.metainterp.optimizeopt import string
         def my_callinfo_for_oopspec(oopspecindex):
             calldescrtype = type(LLtypeMixin.strequaldescr)
@@ -4234,7 +4242,7 @@
         saved = string.callinfo_for_oopspec
         try:
             string.callinfo_for_oopspec = my_callinfo_for_oopspec
-            self.optimize_strunicode_loop(ops, optops)
+            self.optimize_strunicode_loop(ops, optops, preamble)
         finally:
             string.callinfo_for_oopspec = saved
 
@@ -4320,14 +4328,13 @@
         """
         expected = """
         [p1, i1, i2, p3]
-        guard_nonnull(p3) []
         i4 = int_sub(i2, i1)
         i0 = call(0, p1, i1, i4, p3, descr=streq_slice_nonnull_descr)
         escape(i0)
         jump(p1, i1, i2, p3)
         """
         self.optimize_strunicode_loop_extradescrs(ops,
-                                                  expected)
+                                                  expected, ops)
 
     def test_str_equal_slice4(self):
         ops = """
@@ -4407,7 +4414,6 @@
         """
         expected = """
         [p1]
-        guard_nonnull(p1) []
         i0 = call(0, p1, s"hello world", descr=streq_nonnull_descr)
         escape(i0)
         jump(p1)
@@ -4424,7 +4430,6 @@
         """
         expected = """
         [p1]
-        guard_nonnull(p1) []
         i1 = strlen(p1)
         i0 = int_eq(i1, 0)
         escape(i0)
@@ -4442,7 +4447,6 @@
         """
         expected = """
         [p1]
-        guard_nonnull(p1) []
         i0 = call(0, p1, 120, descr=streq_nonnull_char_descr)
         escape(i0)
         jump(p1)


From hakanardo at codespeak.net  Sun Oct 31 13:35:42 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 13:35:42 +0100 (CET)
Subject: [pypy-svn] r78596 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101031123542.33CC0282BDA@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 13:35:36 2010
New Revision: 78596

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
Log:
(hakanardo, arigo) saner propagation of force_at_end_of_preamble

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/heap.py	Sun Oct 31 13:35:36 2010
@@ -151,7 +151,6 @@
 
     def force_at_end_of_preamble(self):
         self.force_all_lazy_setfields()
-        self.next_optimization.force_at_end_of_preamble()
         
     def force_lazy_setfield(self, descr, before_guard=False):
         try:

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	Sun Oct 31 13:35:36 2010
@@ -194,8 +194,7 @@
         pass
 
     def force_at_end_of_preamble(self):
-        if self.next_optimization:
-            self.next_optimization.force_at_end_of_preamble()
+        pass
 
 class Optimizer(Optimization):
 
@@ -222,7 +221,14 @@
                 o.optimizer = self
                 o.setup()
         else:
+            optimizations = []
             self.first_optimization = self
+            
+        self.optimizations  = optimizations 
+
+    def force_at_end_of_preamble(self):
+        for o in self.optimizations:
+            o.force_at_end_of_preamble()
 
     def forget_numberings(self, virtualbox):
         self.metainterp_sd.profiler.count(jitprof.OPT_FORCINGS)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Sun Oct 31 13:35:36 2010
@@ -17,7 +17,7 @@
     def propagate_forward(self, op):
 
         if op.getopnum() == rop.JUMP:
-            self.force_at_end_of_preamble()
+            self.optimizer.force_at_end_of_preamble()
             loop = self.optimizer.loop
             assert op.getdescr() is loop.token
             loop.preamble.operations = self.optimizer.newoperations


From hakanardo at codespeak.net  Sun Oct 31 13:37:55 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 13:37:55 +0100 (CET)
Subject: [pypy-svn] r78597 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031123755.2725F282BDA@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 13:37:52 2010
New Revision: 78597

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
fixed test

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sun Oct 31 13:37:52 2010
@@ -3461,7 +3461,7 @@
         i0 = strlen(p0)
         jump(p0)
         """
-        self.optimize_strunicode_loop(ops, expected)
+        self.optimize_strunicode_loop(ops, expected, expected)
 
     def test_addsub_const(self):
         ops = """


From arigo at codespeak.net  Sun Oct 31 13:46:06 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 13:46:06 +0100 (CET)
Subject: [pypy-svn] r78598 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031124606.A8696282BE8@codespeak.net>

Author: arigo
Date: Sun Oct 31 13:46:01 2010
New Revision: 78598

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
Log:
Fix.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizebasic.py	Sun Oct 31 13:46:01 2010
@@ -268,8 +268,7 @@
                          OptHeap(),
                          OptFfiCall(),
                          ]
-        optimizer = Optimizer(metainterp_sd, loop, optimizations,
-                              not_a_bridge=True)
+        optimizer = Optimizer(metainterp_sd, loop, optimizations)
         optimizer.propagate_all_forward()
         #
         expected = self.parse(optops)


From cfbolz at codespeak.net  Sun Oct 31 14:12:59 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 14:12:59 +0100 (CET)
Subject: [pypy-svn] r78599 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: . test
Message-ID: <20101031131259.85EC6282BDA@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 14:12:58 2010
New Revision: 78599

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py
Log:
(hakanardo, cfbolz, arigo): when not using unrolling, always register every loop
as an entry bridge. this is safe, because there is no specialization any more.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py	Sun Oct 31 14:12:58 2010
@@ -79,7 +79,7 @@
         metainterp.staticdata.log("reusing old loop")
         return old_loop_token
 
-    if loop.preamble.operations:
+    if loop.preamble.operations is not None:
         send_loop_to_backend(metainterp_sd, loop, "loop")
         send_loop_to_backend(metainterp_sd, loop.preamble, "entry bridge")
         insert_loop_token(old_loop_tokens, loop.preamble.token)
@@ -89,6 +89,8 @@
     else:
         send_loop_to_backend(metainterp_sd, loop, "loop")
         insert_loop_token(old_loop_tokens, loop_token)
+        jitdriver_sd.warmstate.attach_unoptimized_bridge_from_interp(
+            greenkey, loop.token)
         return loop_token
 
 def insert_loop_token(old_loop_tokens, loop_token):

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py	Sun Oct 31 14:12:58 2010
@@ -20,8 +20,11 @@
     def check_jumps(self, maxcount):
         pass
 
+
 class TestLLtype(LoopNoPSpecTest, LLJitMixin):
-    pass
+    def check_tree_loop_count(self, count):
+        # we get one less entry bridge
+        return LLJitMixin.check_tree_loop_count(self, count - 1)
 
 class TestOOtype(LoopNoPSpecTest, OOJitMixin):
     pass


From cfbolz at codespeak.net  Sun Oct 31 14:17:54 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 14:17:54 +0100 (CET)
Subject: [pypy-svn] r78600 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031131754.B61A7282BE3@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 14:17:53 2010
New Revision: 78600

Added:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send_simple.py
      - copied unchanged from r78598, pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_dummy.py
Removed:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_dummy.py
Log:
rename to reflect more what it does



From cfbolz at codespeak.net  Sun Oct 31 14:23:19 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 14:23:19 +0100 (CET)
Subject: [pypy-svn] r78601 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031132319.87744282BE8@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 14:23:18 2010
New Revision: 78601

Added:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_unroll.py
      - copied unchanged from r78598, pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_spec.py
Removed:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_spec.py
Log:
another name fix



From cfbolz at codespeak.net  Sun Oct 31 14:23:28 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 14:23:28 +0100 (CET)
Subject: [pypy-svn] r78602 - pypy/extradoc/talk/pycon2011
Message-ID: <20101031132328.DC6DC5080B@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 14:23:27 2010
New Revision: 78602

Modified:
   pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
Log:
small fixes, an XXX


Modified: pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
==============================================================================
--- pypy/extradoc/talk/pycon2011/pypy-optimizations.txt	(original)
+++ pypy/extradoc/talk/pycon2011/pypy-optimizations.txt	Sun Oct 31 14:23:27 2010
@@ -10,14 +10,14 @@
 Classification: Discuss in depth
 Abstract:
 
-The talk would contain a list of features that are hard to optimize in Python,
+The talk will contain a list of features that are hard to optimize in Python,
 why and how we're fixing it in PyPy. This also might be used as a rough guide
 what features in Python are optimizable right now and what are not.
 
 The list of mentioned features is not exhaustive, however we will try to
 focus at least on the following:
 
-* Dynamic language - In Python code we have no known types, like a static
+* Dynamic language - In Python code we have no known types, like a statically typed
   language.  Even operations like "a + b" can do anything, unless we know more
   about the code.
 
@@ -27,10 +27,10 @@
 
 * Object model - All user defined Python objects have a dictionary which stores
   their attributes, as does every type.  When Python does an attribute lookup
-  this requires 2 dictionary lookups.
+  this requires at least two dictionary lookups.
 
-  In PyPy we use an approach similar to one used by v8 with hidden classes
-  (except more pypy specific) called shared dictionaries.
+  In PyPy we use an approach similar to the one used by V8 with hidden classes
+  (except more PyPy specific) called map dictionaries. XXX type versions?
 
 * FFI calls - calling C from Python is costly and hard to optimize. In PyPy we
   decided to go via ctypes, this part will explain how are we going to handle


From cfbolz at codespeak.net  Sun Oct 31 14:43:52 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 14:43:52 +0100 (CET)
Subject: [pypy-svn] r78603 - in pypy/branch/jit-unroll-loops/pypy:
	jit/metainterp jit/metainterp/test jit/tl jit/tl/tla rlib
Message-ID: <20101031134352.8B2DB282BE3@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 14:43:50 2010
New Revision: 78603

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py
   pypy/branch/jit-unroll-loops/pypy/jit/tl/tla/test_tla.py
   pypy/branch/jit-unroll-loops/pypy/rlib/jit.py
Log:
rename OPTIMIZER_NO_PERFECTSPEC to OPTIMIZER_NO_UNROLL


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py	Sun Oct 31 14:43:50 2010
@@ -1,12 +1,12 @@
 
 from pypy.jit.metainterp.test import test_loop, test_send
 from pypy.jit.metainterp.warmspot import ll_meta_interp
-from pypy.rlib.jit import OPTIMIZER_NO_PERFECTSPEC
+from pypy.rlib.jit import OPTIMIZER_NO_UNROLL
 from pypy.jit.metainterp.test.test_basic import LLJitMixin, OOJitMixin
 
 class LoopNoPSpecTest(test_send.SendTests):
     def meta_interp(self, func, args, **kwds):
-        return ll_meta_interp(func, args, optimizer=OPTIMIZER_NO_PERFECTSPEC,
+        return ll_meta_interp(func, args, optimizer=OPTIMIZER_NO_UNROLL,
                               CPUClass=self.CPUClass, 
                               type_system=self.type_system,
                               **kwds)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/warmstate.py	Sun Oct 31 14:43:50 2010
@@ -8,7 +8,7 @@
 from pypy.rlib.nonconst import NonConstant
 from pypy.rlib.unroll import unrolling_iterable
 from pypy.rlib.jit import (PARAMETERS, OPTIMIZER_SIMPLE, OPTIMIZER_FULL,
-                           OPTIMIZER_NO_PERFECTSPEC)
+                           OPTIMIZER_NO_UNROLL)
 from pypy.rlib.jit import DEBUG_PROFILE
 from pypy.rlib.jit import BaseJitCell
 from pypy.rlib.debug import debug_start, debug_stop, debug_print
@@ -198,7 +198,7 @@
             from pypy.jit.metainterp import simple_optimize
             self.optimize_loop = simple_optimize.optimize_loop
             self.optimize_bridge = simple_optimize.optimize_bridge
-        elif optimizer == OPTIMIZER_NO_PERFECTSPEC:
+        elif optimizer == OPTIMIZER_NO_UNROLL:
             from pypy.jit.metainterp import nounroll_optimize
             self.optimize_loop = nounroll_optimize.optimize_loop
             self.optimize_bridge = nounroll_optimize.optimize_bridge

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/pypyjit_child.py	Sun Oct 31 14:43:50 2010
@@ -2,7 +2,7 @@
 from pypy.rpython.lltypesystem import lltype
 from pypy.jit.metainterp import warmspot
 from pypy.module.pypyjit.policy import PyPyJitPolicy
-from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_NO_PERFECTSPEC
+from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_NO_UNROLL
 
 
 def run_child(glob, loc):
@@ -35,5 +35,5 @@
     warmspot.jittify_and_run(interp, graph, [], policy=policy,
                              listops=True, CPUClass=CPUClass,
                              backendopt=True, inline=False,
-                             optimizer=OPTIMIZER_NO_PERFECTSPEC)
+                             optimizer=OPTIMIZER_NO_UNROLL)
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/tl/tla/test_tla.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tl/tla/test_tla.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tl/tla/test_tla.py	Sun Oct 31 14:43:50 2010
@@ -156,7 +156,7 @@
 # ____________________________________________________________ 
 
 from pypy.jit.metainterp.test.test_basic import LLJitMixin
-from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_NO_PERFECTSPEC
+from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_NO_UNROLL
 
 class TestLLtype(LLJitMixin):
     def test_loop(self):
@@ -178,5 +178,5 @@
             assert isinstance(w_result, tla.W_IntObject)
             return w_result.intvalue
         res = self.meta_interp(interp_w, [42], listops=True,
-                               optimizer=OPTIMIZER_NO_PERFECTSPEC)
+                               optimizer=OPTIMIZER_NO_UNROLL)
         assert res == 0

Modified: pypy/branch/jit-unroll-loops/pypy/rlib/jit.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/rlib/jit.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/rlib/jit.py	Sun Oct 31 14:43:50 2010
@@ -248,7 +248,7 @@
     """Inconsistency in the JIT hints."""
 
 OPTIMIZER_SIMPLE = 0
-OPTIMIZER_NO_PERFECTSPEC = 1
+OPTIMIZER_NO_UNROLL = 1
 OPTIMIZER_FULL = 2
 
 DEBUG_OFF = 0
@@ -261,7 +261,7 @@
               'trace_limit': 10000,
               'inlining': False,
               'optimizer': OPTIMIZER_FULL,
-              #'optimizer': OPTIMIZER_NO_PERFECTSPEC,
+              #'optimizer': OPTIMIZER_NO_UNROLL,
               'debug' : DEBUG_STEPS,
               }
 unroll_parameters = unrolling_iterable(PARAMETERS.keys())


From arigo at codespeak.net  Sun Oct 31 14:47:13 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 14:47:13 +0100 (CET)
Subject: [pypy-svn] r78604 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101031134713.0652D5080C@codespeak.net>

Author: arigo
Date: Sun Oct 31 14:47:12 2010
New Revision: 78604

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
Log:
simplification.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/unroll.py	Sun Oct 31 14:47:12 2010
@@ -116,17 +116,7 @@
     def inline_snapshot(self, snapshot):
         if snapshot in self.snapshot_map:
             return self.snapshot_map[snapshot]
-        boxes = []
-        for a in snapshot.boxes:
-            if isinstance(a, Const):
-                boxes.append(a)
-            else:
-                boxes.append(self.inline_arg(a))
-        new_snapshot = Snapshot(self.inline_snapshot(snapshot.prev), boxes[:])
+        boxes = [self.inline_arg(a) for a in snapshot.boxes]
+        new_snapshot = Snapshot(self.inline_snapshot(snapshot.prev), boxes)
         self.snapshot_map[snapshot] = new_snapshot
         return new_snapshot
-    
-
-        
-        
-        


From arigo at codespeak.net  Sun Oct 31 14:47:26 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 14:47:26 +0100 (CET)
Subject: [pypy-svn] r78605 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt
Message-ID: <20101031134726.B9574282BE3@codespeak.net>

Author: arigo
Date: Sun Oct 31 14:47:25 2010
New Revision: 78605

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
Log:
Add a missing case in __repr__.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/virtualize.py	Sun Oct 31 14:47:25 2010
@@ -147,6 +147,8 @@
 
     def __repr__(self):
         cls_name = self.known_class.value.adr.ptr._obj._TYPE._name
+        if self._fields is None:
+            return '' % (cls_name,)
         field_names = [field.name for field in self._fields]
         return "" % (cls_name, field_names)
 


From cfbolz at codespeak.net  Sun Oct 31 14:48:05 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 14:48:05 +0100 (CET)
Subject: [pypy-svn] r78606 - pypy/extradoc/sprintinfo/ddorf2010
Message-ID: <20101031134805.A0E345080E@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 14:48:04 2010
New Revision: 78606

Added:
   pypy/extradoc/sprintinfo/ddorf2010/report.rst
Log:
draft of a sprint report


Added: pypy/extradoc/sprintinfo/ddorf2010/report.rst
==============================================================================
--- (empty file)
+++ pypy/extradoc/sprintinfo/ddorf2010/report.rst	Sun Oct 31 14:48:04 2010
@@ -0,0 +1,62 @@
+D?sseldorf Sprint Report 2010
+==============================
+
+This years installment of the yearly PyPy D?sseldorf Sprint is drawing to a
+close. As usual, we worked in the seminar room of the `programming language
+group`_ at the University of D?sseldorf. The sprint was different from previous
+ones in that we had fewer people than usual and many actually live in
+D?sseldorf all the time. 
+
+David spent the sprint working on the `arm-backend`_ branch, which is adding an
+ARM backend to the JIT. With the help of Armin he added support for bridges in
+the JIT and generally implemented missing operations, mostly for handling integers.
+
+.. _`arm-backend`: http://codespeak.net/svn/pypy/branch/arm-backend/
+
+Ronny and Anto worked the whole week trying to come up with a scheme for
+importing PyPy's SVN history into a mercurial repository without loosing too
+much information. This is a non-trivial task, because PyPy's history is gnarly.
+We are nearly at revision 79000 and when we started using it, Subversion was at
+version 0.1. All possible and impossible ways to mangle and mistreat a
+Subversion repository have been applied to PyPy's repo, so most of the
+importing tools just give up. Ronny and Anto came up with a new plan and new
+helper scripts every day, only to then discover another corner case that they
+hadn't thought of. Now they might actually have a final plan (but they said
+that every day, so who knows?).
+
+Carl Friedrich and Lukas started working in earnest on memory benchmarks to
+understand the memory behaviour of Python code better. They have now
+implemented a generic memory benchmark runner and a simple analysis that walks
+all objectsf and collects size information about them. They also added some
+benchmarks that were proposed in the comments of the recent `call for
+benchmarks`_. As soon as some results from that work are there, we will post
+about them.
+
+.. _`call for benchmarks`: http://morepypy.blogspot.com/2010/08/call-for-benchmarks.html
+
+There were also some minor tasks performed during the sprint. Armin implemented
+the ``_bisect`` module and the ``dict.popitem`` method in RPython. Armin and
+Carl Friedrich made the new memory-saving mapdict implementation more suitable
+to use without the JIT (blog post should come about that too, at some point).
+They also made classes with custom metaclasses a lot faster when the JIT is
+used.
+
+The last three days of the sprint were spent working on H?kan's
+`jit-unroll-loops`_ branch.  The branch is meant to move loop invariants out of
+the loop, using techniques very similar to what is described in the recent post
+on `escape analysis across loop boundaries`_ (see? it will soon stop being
+science-fiction). Some of the ideas of this approach also come from LuaJIT_
+which also uses very aggressive loop invariant code motion in its optimizers.
+Moving loop invariants outside of the loop is very useful, because many of the
+lookups that Python programs do in loops are loop invariants. An example is if
+you call a function in a loop: The global lookup can often be done only once. 
+
+This branch fundamentally changes some of the core assumptions of the JIT, so
+it is a huge amount of work to make it fit with all the other parts and to
+adapt all tests. That work is now nearly done, some failing tests remain. The
+next steps are to fix them and then do additional tests with the translated
+executable and look at the benchmarks.
+
+.. _`jit-unroll-loops`: http://codespeak.net/svn/pypy/branch/jit-unroll-loops/
+.. _`escape analysis across loop boundaries`: http://morepypy.blogspot.com/2010/09/using-escape-analysis-across-loop.html
+.. _LuaJIT: http://luajit.org/


From arigo at codespeak.net  Sun Oct 31 14:48:17 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 14:48:17 +0100 (CET)
Subject: [pypy-svn] r78607 - pypy/branch/jit-unroll-loops/pypy/jit/metainterp
Message-ID: <20101031134817.28141282BE9@codespeak.net>

Author: arigo
Date: Sun Oct 31 14:48:15 2010
New Revision: 78607

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py
Log:
Make "snapshot" a local var, allowing us to inspect it with pdb
even after the function cleared it.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/resume.py	Sun Oct 31 14:48:15 2010
@@ -299,9 +299,9 @@
         storage = self.storage
         # make sure that nobody attached resume data to this guard yet
         assert storage.rd_numb is None
-        assert storage.rd_snapshot is not None # is that true?
-        numb, liveboxes_from_env, v = self.memo.number(values,
-                                                       storage.rd_snapshot)
+        snapshot = storage.rd_snapshot
+        assert snapshot is not None # is that true?
+        numb, liveboxes_from_env, v = self.memo.number(values, snapshot)
         self.liveboxes_from_env = liveboxes_from_env
         self.liveboxes = {}
         storage.rd_numb = numb


From arigo at codespeak.net  Sun Oct 31 14:51:31 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 14:51:31 +0100 (CET)
Subject: [pypy-svn] r78608 - in
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp: optimizeopt test
Message-ID: <20101031135131.8E0FB5080F@codespeak.net>

Author: arigo
Date: Sun Oct 31 14:51:29 2010
New Revision: 78608

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
Log:
Mostly fix the tests by disabling them, which makes sense right now
because _verify_fail_args() would need to map from old to new box names
to inspect the loop produced by unroll.py.

Fix another piece of data that must not be propagated from the preamble
to the main loop.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/optimizeopt/optimizer.py	Sun Oct 31 14:51:29 2010
@@ -227,6 +227,7 @@
         self.optimizations  = optimizations 
 
     def force_at_end_of_preamble(self):
+        self.resumedata_memo = resume.ResumeDataLoopMemo(self.metainterp_sd)
         for o in self.optimizations:
             o.force_at_end_of_preamble()
 

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_optimizeopt.py	Sun Oct 31 14:51:29 2010
@@ -100,15 +100,27 @@
         class FailDescr(compile.ResumeGuardDescr):
             oparse = None
             def _oparser_uses_descr_of_guard(self, oparse, fail_args):
-                # typically called twice, before and after optimization
-                if self.oparse is None:
-                    self.rd_frame_info_list = resume.FrameInfo(None,
-                                                                 "code", 11)
-                    self.rd_snapshot = resume.Snapshot(None, fail_args)
+                # typically called 3 times: once when parsing 'ops',
+                # once when parsing 'preamble', once when parsing 'expected'.
                 self.oparse = oparse
+                self.rd_frame_info_list, self.rd_snapshot = snapshot(fail_args)
             def _clone_if_mutable(self):
                 assert self is fdescr
                 return fdescr2
+            def __repr__(self):
+                if self is fdescr:
+                    return 'fdescr'
+                if self is fdescr2:
+                    return 'fdescr2'
+                return compile.ResumeGuardDescr.__repr__(self)
+        #
+        def snapshot(fail_args, got=[]):
+            if not got:    # only the first time, i.e. when parsing 'ops'
+                rd_frame_info_list = resume.FrameInfo(None, "code", 11)
+                rd_snapshot = resume.Snapshot(None, fail_args)
+                got.append(rd_frame_info_list)
+                got.append(rd_snapshot)
+            return got
         #
         fdescr = instantiate(FailDescr)
         self.namespace['fdescr'] = fdescr
@@ -2161,7 +2173,7 @@
         jump(p2, i0, i1, i3)
         """
         self.optimize_loop(ops, expected, preamble)
-        self.check_expanded_fail_descr("i0", rop.GUARD_NONNULL_CLASS)
+        #self.check_expanded_fail_descr("i0", rop.GUARD_NONNULL_CLASS)
 
     def test_merge_guard_nonnull_guard_value(self):
         ops = """
@@ -2184,7 +2196,7 @@
         jump(ConstPtr(myptr), i0, i1, i3)
         """
         self.optimize_loop(ops, expected, preamble)
-        self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
+        #self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
 
     def test_merge_guard_nonnull_guard_class_guard_value(self):
         ops = """
@@ -2211,7 +2223,7 @@
         jump(ConstPtr(myptr), i0, i1, i4)
         """
         self.optimize_loop(ops, expected, preamble)
-        self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
+        #self.check_expanded_fail_descr("i0", rop.GUARD_VALUE)
 
     def test_guard_class_oois(self):
         ops = """
@@ -2435,113 +2447,6 @@
 
     # ----------
 
-    def _verify_fail_args(self, boxes, oparse, text):
-        import re
-        r = re.compile(r"\bwhere\s+(\w+)\s+is a\s+(\w+)")
-        parts = list(r.finditer(text))
-        ends = [match.start() for match in parts] + [len(text)]
-        #
-        virtuals = {}
-        for match, end in zip(parts, ends[1:]):
-            pvar = match.group(1)
-            fieldstext = text[match.end():end]
-            if match.group(2) == 'varray':
-                arrayname, fieldstext = fieldstext.split(':', 1)
-                tag = ('varray', self.namespace[arrayname.strip()])
-            elif match.group(2) == 'vstruct':
-                if ',' in fieldstext:
-                    structname, fieldstext = fieldstext.split(',', 1)
-                else:
-                    structname, fieldstext = fieldstext, ''
-                tag = ('vstruct', self.namespace[structname.strip()])
-            else:
-                tag = ('virtual', self.namespace[match.group(2)])
-            virtuals[pvar] = (tag, None, fieldstext)
-        #
-        r2 = re.compile(r"([\w\d()]+)[.](\w+)\s*=\s*([\w\d()]+)")
-        pendingfields = []
-        for match in r2.finditer(text):
-            pvar = match.group(1)
-            pfieldname = match.group(2)
-            pfieldvar = match.group(3)
-            pendingfields.append((pvar, pfieldname, pfieldvar))
-        #
-        def _variables_equal(box, varname, strict):
-            if varname not in virtuals:
-                if strict:
-                    assert box == oparse.getvar(varname)
-                else:
-                    assert box.value == oparse.getvar(varname).value
-            else:
-                tag, resolved, fieldstext = virtuals[varname]
-                if tag[0] == 'virtual':
-                    assert self.get_class_of_box(box) == tag[1]
-                elif tag[0] == 'varray':
-                    pass    # xxx check arraydescr
-                elif tag[0] == 'vstruct':
-                    pass    # xxx check typedescr
-                else:
-                    assert 0
-                if resolved is not None:
-                    assert resolved.value == box.value
-                else:
-                    virtuals[varname] = tag, box, fieldstext
-        #
-        basetext = text.splitlines()[0]
-        varnames = [s.strip() for s in basetext.split(',')]
-        if varnames == ['']:
-            varnames = []
-        assert len(boxes) == len(varnames)
-        for box, varname in zip(boxes, varnames):
-            _variables_equal(box, varname, strict=True)
-        for pvar, pfieldname, pfieldvar in pendingfields:
-            box = oparse.getvar(pvar)
-            fielddescr = self.namespace[pfieldname.strip()]
-            fieldbox = executor.execute(self.cpu, None,
-                                        rop.GETFIELD_GC,
-                                        fielddescr,
-                                        box)
-            _variables_equal(fieldbox, pfieldvar, strict=True)
-        #
-        for match in parts:
-            pvar = match.group(1)
-            tag, resolved, fieldstext = virtuals[pvar]
-            assert resolved is not None
-            index = 0
-            for fieldtext in fieldstext.split(','):
-                fieldtext = fieldtext.strip()
-                if not fieldtext:
-                    continue
-                if tag[0] in ('virtual', 'vstruct'):
-                    fieldname, fieldvalue = fieldtext.split('=')
-                    fielddescr = self.namespace[fieldname.strip()]
-                    fieldbox = executor.execute(self.cpu, None,
-                                                rop.GETFIELD_GC,
-                                                fielddescr,
-                                                resolved)
-                elif tag[0] == 'varray':
-                    fieldvalue = fieldtext
-                    fieldbox = executor.execute(self.cpu, None,
-                                                rop.GETARRAYITEM_GC,
-                                                tag[1],
-                                                resolved, ConstInt(index))
-                else:
-                    assert 0
-                _variables_equal(fieldbox, fieldvalue.strip(), strict=False)
-                index += 1
-
-    def check_expanded_fail_descr(self, expectedtext, guard_opnum):
-        from pypy.jit.metainterp.test.test_resume import ResumeDataFakeReader
-        from pypy.jit.metainterp.test.test_resume import MyMetaInterp
-        guard_op, = [op for op in self.loop.operations if op.is_guard()]
-        fail_args = guard_op.getfailargs()
-        fdescr = guard_op.getdescr()
-        assert fdescr.guard_opnum == guard_opnum
-        reader = ResumeDataFakeReader(fdescr, fail_args,
-                                      MyMetaInterp(self.cpu))
-        boxes = reader.consume_boxes()
-        self._verify_fail_args(boxes, fdescr.oparse, expectedtext)
-
 class TestLLtype(OptimizeOptTest, LLtypeMixin):
 
     def test_residual_call_does_not_invalidate_caches(self):
@@ -2765,6 +2670,8 @@
         '''
         self.optimize_loop(ops, expected, preamble)
 
+    # ----------
+
     def test_vref_nonvirtual_nonescape(self):
         ops = """
         [p1]
@@ -2777,7 +2684,7 @@
         i0 = force_token()
         jump(p1)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_loop(ops, expected, expected)
 
     def test_vref_nonvirtual_escape(self):
         ops = """
@@ -2800,7 +2707,7 @@
         """
         # XXX we should optimize a bit more the case of a nonvirtual.
         # in theory it is enough to just do 'p2 = p1'.
-        self.optimize_loop(ops, expected)
+        self.optimize_loop(ops, expected, expected)
 
     def test_vref_virtual_1(self):
         ops = """
@@ -2840,7 +2747,7 @@
         setfield_gc(p2, -3, descr=virtualtokendescr)
         jump(p0, i1)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_loop(ops, expected, expected)
 
     def test_vref_virtual_2(self):
         ops = """
@@ -2869,7 +2776,7 @@
         setfield_gc(p0, p2, descr=nextdescr)
         #
         call_may_force(i1, descr=mayforcevirtdescr)
-        guard_not_forced(descr=fdescr) [p2, i1]
+        guard_not_forced(descr=fdescr2) [p2, i1]
         #
         setfield_gc(p0, NULL, descr=nextdescr)
         p1 = new_with_vtable(ConstClass(node_vtable))
@@ -2883,10 +2790,10 @@
         # the point of this test is that 'i1' should show up in the fail_args
         # of 'guard_not_forced', because it was stored in the virtual 'p1b'.
         self.optimize_loop(ops, expected)
-        self.check_expanded_fail_descr('''p2, p1
-            where p1 is a node_vtable, nextdescr=p1b
-            where p1b is a node_vtable, valuedescr=i1
-            ''', rop.GUARD_NOT_FORCED)
+        #self.check_expanded_fail_descr('''p2, p1
+        #    where p1 is a node_vtable, nextdescr=p1b
+        #    where p1b is a node_vtable, valuedescr=i1
+        #    ''', rop.GUARD_NOT_FORCED)
 
     def test_vref_virtual_and_lazy_setfield(self):
         ops = """
@@ -2905,7 +2812,7 @@
         setfield_gc(p0, NULL, descr=refdescr)
         jump(p0, i1)
         """
-        expected = """
+        preamble = """
         [p0, i1]
         i3 = force_token()
         call(i1, descr=nonwritedescr)
@@ -2913,18 +2820,26 @@
         setfield_gc(p0, NULL, descr=refdescr)
         jump(p0, i1)
         """
-        self.optimize_loop(ops, expected)
+        expected = """
+        [p0, i1]
+        i3 = force_token()
+        call(i1, descr=nonwritedescr)
+        guard_no_exception(descr=fdescr2) [i3, i1, p0]
+        setfield_gc(p0, NULL, descr=refdescr)
+        jump(p0, i1)
+        """
+        self.optimize_loop(ops, expected, preamble)
         # the fail_args contain [i3, i1, p0]:
         #  - i3 is from the virtual expansion of p2
         #  - i1 is from the virtual expansion of p1
         #  - p0 is from the extra pendingfields
-        self.loop.inputargs[0].value = self.nodeobjvalue
-        self.check_expanded_fail_descr('''p2, p1
-            p0.refdescr = p2
-            where p2 is a jit_virtual_ref_vtable, virtualtokendescr=i3, virtualrefindexdescr=2
-            where p1 is a node_vtable, nextdescr=p1b
-            where p1b is a node_vtable, valuedescr=i1
-            ''', rop.GUARD_NO_EXCEPTION)
+        #self.loop.inputargs[0].value = self.nodeobjvalue
+        #self.check_expanded_fail_descr('''p2, p1
+        #    p0.refdescr = p2
+        #    where p2 is a jit_virtual_ref_vtable, virtualtokendescr=i3, virtualrefindexdescr=2
+        #    where p1 is a node_vtable, nextdescr=p1b
+        #    where p1b is a node_vtable, valuedescr=i1
+        #    ''', rop.GUARD_NO_EXCEPTION)
 
     def test_vref_virtual_after_finish(self):
         ops = """
@@ -2951,7 +2866,7 @@
         guard_not_forced() []
         jump(i1)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_loop(ops, expected, expected)
 
     def test_vref_nonvirtual_and_lazy_setfield(self):
         ops = """
@@ -2976,7 +2891,9 @@
         guard_not_forced() [i1]
         jump(i1, p1)
         """
-        self.optimize_loop(ops, expected)
+        self.optimize_loop(ops, expected, expected)
+
+    # ----------
 
     def test_arraycopy_1(self):
         ops = '''


From hakanardo at codespeak.net  Sun Oct 31 14:58:21 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 14:58:21 +0100 (CET)
Subject: [pypy-svn] r78609 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031135821.32131282B9E@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 14:58:19 2010
New Revision: 78609

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop.py
Log:
fixed test

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop.py	Sun Oct 31 14:58:19 2010
@@ -397,8 +397,12 @@
         res = self.meta_interp(f, [100, 5], policy=StopAtXPolicy(externfn))
         assert res == expected
 
-        self.check_loop_count(2)
-        self.check_tree_loop_count(2)   # 1 loop, 1 bridge from interp
+        if self.optimizer == OPTIMIZER_FULL:
+            self.check_loop_count(2)
+            self.check_tree_loop_count(2)   # 1 loop, 1 bridge from interp
+        else:
+            self.check_loop_count(2)
+            self.check_tree_loop_count(1)   # 1 loop, 1 bridge from interp
 
     def test_example(self):
         myjitdriver = JitDriver(greens = ['i'],


From hakanardo at codespeak.net  Sun Oct 31 15:00:10 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 15:00:10 +0100 (CET)
Subject: [pypy-svn] r78610 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031140010.81B9B36E0C1@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 15:00:03 2010
New Revision: 78610

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop.py
Log:
fixed comment

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop.py	Sun Oct 31 15:00:03 2010
@@ -402,7 +402,7 @@
             self.check_tree_loop_count(2)   # 1 loop, 1 bridge from interp
         else:
             self.check_loop_count(2)
-            self.check_tree_loop_count(1)   # 1 loop, 1 bridge from interp
+            self.check_tree_loop_count(1)   # 1 loop, callable from the interp
 
     def test_example(self):
         myjitdriver = JitDriver(greens = ['i'],


From arigo at codespeak.net  Sun Oct 31 15:00:46 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 15:00:46 +0100 (CET)
Subject: [pypy-svn] r78611 - pypy/branch/jit-unroll-loops/pypy/jit/metainterp
Message-ID: <20101031140046.666E6282B9E@codespeak.net>

Author: arigo
Date: Sun Oct 31 15:00:44 2010
New Revision: 78611

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py
Log:
Stick the name "Preample Loop" on the preamble loops.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/compile.py	Sun Oct 31 15:00:44 2010
@@ -35,9 +35,9 @@
             extraloops = [loop]
         metainterp_sd.stats.view(errmsg=errmsg, extraloops=extraloops)
 
-def create_empty_loop(metainterp):
+def create_empty_loop(metainterp, name_prefix=''):
     name = metainterp.staticdata.stats.name_for_new_loop()
-    return TreeLoop(name)
+    return TreeLoop(name_prefix + name)
 
 def make_loop_token(nb_args, jitdriver_sd):
     loop_token = LoopToken()
@@ -65,7 +65,7 @@
     loop.token = loop_token
     loop.operations[-1].setdescr(loop_token)     # patch the target of the JUMP
 
-    loop.preamble = create_empty_loop(metainterp)
+    loop.preamble = create_empty_loop(metainterp, 'Preamble ')
     loop.preamble.greenkey = greenkey
     loop.preamble.inputargs = loop.inputargs
     loop.preamble.token = make_loop_token(len(loop.inputargs), jitdriver_sd)


From cfbolz at codespeak.net  Sun Oct 31 15:11:26 2010
From: cfbolz at codespeak.net (cfbolz at codespeak.net)
Date: Sun, 31 Oct 2010 15:11:26 +0100 (CET)
Subject: [pypy-svn] r78612 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031141126.036A636E0C3@codespeak.net>

Author: cfbolz
Date: Sun Oct 31 15:11:25 2010
New Revision: 78612

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py
Log:
revert these numbers to how they were on trunk


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_compile.py	Sun Oct 31 15:11:25 2010
@@ -87,11 +87,11 @@
     loop_tokens = []
     loop_token = compile_new_loop(metainterp, loop_tokens, [], 0)
     assert loop_tokens == [loop_token]
-    assert loop_token.number == 2
-    assert staticdata.globaldata.loopnumbering == 3
+    assert loop_token.number == 1
+    assert staticdata.globaldata.loopnumbering == 2
     #
-    assert len(cpu.seen) == 2
-    assert cpu.seen[1][2] == loop_token
+    assert len(cpu.seen) == 1
+    assert cpu.seen[0][2] == loop_token
     #
     del cpu.seen[:]
     metainterp = FakeMetaInterp()
@@ -105,7 +105,7 @@
     assert loop_token_2 is loop_token
     assert loop_tokens == [loop_token]
     assert len(cpu.seen) == 0
-    assert staticdata.globaldata.loopnumbering == 3
+    assert staticdata.globaldata.loopnumbering == 2
 
 
 def test_resume_guard_counters():


From arigo at codespeak.net  Sun Oct 31 15:14:07 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 15:14:07 +0100 (CET)
Subject: [pypy-svn] r78613 - pypy/extradoc/sprintinfo/ddorf2010
Message-ID: <20101031141407.DFCEB282BE3@codespeak.net>

Author: arigo
Date: Sun Oct 31 15:14:06 2010
New Revision: 78613

Modified:
   pypy/extradoc/sprintinfo/ddorf2010/report.rst
Log:
Typos.


Modified: pypy/extradoc/sprintinfo/ddorf2010/report.rst
==============================================================================
--- pypy/extradoc/sprintinfo/ddorf2010/report.rst	(original)
+++ pypy/extradoc/sprintinfo/ddorf2010/report.rst	Sun Oct 31 15:14:06 2010
@@ -9,7 +9,7 @@
 
 David spent the sprint working on the `arm-backend`_ branch, which is adding an
 ARM backend to the JIT. With the help of Armin he added support for bridges in
-the JIT and generally implemented missing operations, mostly for handling integers.
+the JIT and generally implemented missing operations, mostly for handling integers so far.
 
 .. _`arm-backend`: http://codespeak.net/svn/pypy/branch/arm-backend/
 
@@ -27,7 +27,7 @@
 Carl Friedrich and Lukas started working in earnest on memory benchmarks to
 understand the memory behaviour of Python code better. They have now
 implemented a generic memory benchmark runner and a simple analysis that walks
-all objectsf and collects size information about them. They also added some
+all objects and collects size information about them. They also added some
 benchmarks that were proposed in the comments of the recent `call for
 benchmarks`_. As soon as some results from that work are there, we will post
 about them.


From hakanardo at codespeak.net  Sun Oct 31 15:14:11 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 15:14:11 +0100 (CET)
Subject: [pypy-svn] r78614 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031141411.38CBF282BF1@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 15:14:09 2010
New Revision: 78614

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send_simple.py
Log:
fixed test

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	Sun Oct 31 15:14:09 2010
@@ -2,10 +2,11 @@
 from pypy.rlib.jit import JitDriver, hint, purefunction
 from pypy.jit.codewriter.policy import StopAtXPolicy
 from pypy.jit.metainterp.test.test_basic import LLJitMixin, OOJitMixin
-
+from pypy.rlib.jit import OPTIMIZER_FULL, OPTIMIZER_SIMPLE
 
 class SendTests:
-
+    optimizer=OPTIMIZER_FULL
+    
     def test_green_send(self):
         myjitdriver = JitDriver(greens = ['i'], reds = ['counter'])
         lst = ["123", "45"]
@@ -164,8 +165,12 @@
         for j in range(69, 75):
             res = self.meta_interp(f, [j], policy=policy)
             assert res == 42
-            self.check_enter_count(5)
-            self.check_loop_count(5)
+            if self.optimizer == OPTIMIZER_SIMPLE:
+                self.check_enter_count(3)
+                self.check_loop_count(3)
+            else:
+                self.check_enter_count(5)
+                self.check_loop_count(5)
 
     def test_oosend_guard_failure(self):
         myjitdriver = JitDriver(greens = [], reds = ['x', 'y', 'w'])

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send_simple.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send_simple.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send_simple.py	Sun Oct 31 15:14:09 2010
@@ -6,8 +6,9 @@
 from pypy.jit.metainterp.test.test_basic import LLJitMixin, OOJitMixin
 
 class LoopDummyTest(test_send.SendTests):
+    optimizer=OPTIMIZER_SIMPLE
     def meta_interp(self, func, args, **kwds):
-        return ll_meta_interp(func, args, optimizer=OPTIMIZER_SIMPLE,
+        return ll_meta_interp(func, args, optimizer=self.optimizer,
                               CPUClass=self.CPUClass, 
                               type_system=self.type_system,
                               **kwds)


From hakanardo at codespeak.net  Sun Oct 31 15:20:44 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 15:20:44 +0100 (CET)
Subject: [pypy-svn] r78615 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031142044.1E022282BDA@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 15:20:42 2010
New Revision: 78615

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
Log:
fixed test

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	Sun Oct 31 15:20:42 2010
@@ -353,7 +353,10 @@
         assert res == f(198)
         # we get two TreeLoops: an initial one, and one entering from
         # the interpreter
-        self.check_tree_loop_count(2)
+        if self.optimizer == OPTIMIZER_SIMPLE:
+            self.check_tree_loop_count(1)
+        else:
+            self.check_tree_loop_count(2)
 
     def test_two_behaviors(self):
         py.test.skip("XXX fix me!!!!!!! problem in optimize.py")


From arigo at codespeak.net  Sun Oct 31 15:28:03 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 15:28:03 +0100 (CET)
Subject: [pypy-svn] r78616 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031142803.230FC36E0CB@codespeak.net>

Author: arigo
Date: Sun Oct 31 15:28:01 2010
New Revision: 78616

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py
Log:
fix this test.

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_virtualizable.py	Sun Oct 31 15:28:01 2010
@@ -197,7 +197,8 @@
             return xy.inst_x
         res = self.meta_interp(f, [20])
         assert res == 134
-        self.check_loops(getfield_gc=1, setfield_gc=1, everywhere=True)
+        self.check_loops(getfield_gc=0, setfield_gc=1)
+        self.check_loops(getfield_gc=1, setfield_gc=2, everywhere=True)
 
     # ------------------------------
 


From arigo at codespeak.net  Sun Oct 31 15:39:17 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 15:39:17 +0100 (CET)
Subject: [pypy-svn] r78617 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031143917.B7EF1282BAD@codespeak.net>

Author: arigo
Date: Sun Oct 31 15:39:16 2010
New Revision: 78617

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py
Log:
Raah.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_recursive.py	Sun Oct 31 15:39:16 2010
@@ -575,7 +575,7 @@
                 result += f('-c-----------l-', i+100)
         self.meta_interp(g, [10], backendopt=True)
         self.check_aborted_count(1)
-        self.check_history(call_assembler=1, call=0)
+        self.check_loops(call_assembler=1, call=0)
         self.check_tree_loop_count(3)
 
     def test_directly_call_assembler(self):


From hakanardo at codespeak.net  Sun Oct 31 15:39:53 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 15:39:53 +0100 (CET)
Subject: [pypy-svn] r78618 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031143953.5799036E0CD@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 15:39:51 2010
New Revision: 78618

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	Sun Oct 31 15:39:51 2010
@@ -409,7 +409,10 @@
         # we expect 1 loop, 1 entry bridge, and 1 bridge going from the
         # loop back to the start of the entry bridge
         self.check_loop_count(2)        # 1 loop + 1 bridge
-        self.check_tree_loop_count(2)   # 1 loop + 1 entry bridge  (argh)
+        if self.optimizer == OPTIMIZER_SIMPLE:
+            self.check_tree_loop_count(1)   # 1 loop 
+        else:
+            self.check_tree_loop_count(2)   # 1 loop + 1 entry bridge  (argh)
         self.check_aborted_count(0)
 
     def test_three_cases(self):
@@ -430,7 +433,10 @@
             return node.x
         res = self.meta_interp(f, [55])
         assert res == f(55)
-        self.check_tree_loop_count(2)
+        if self.optimizer == OPTIMIZER_SIMPLE:
+            self.check_tree_loop_count(1)
+        else:
+            self.check_tree_loop_count(2)
 
     def test_three_classes(self):
         class Base:
@@ -460,7 +466,10 @@
             return n
         res = self.meta_interp(f, [55], policy=StopAtXPolicy(extern))
         assert res == f(55)
-        self.check_tree_loop_count(2)
+        if self.optimizer == OPTIMIZER_SIMPLE:
+            self.check_tree_loop_count(1)
+        else:
+            self.check_tree_loop_count(2)
 
     def test_bug1(self):
         myjitdriver = JitDriver(greens = [], reds = ['n', 'node'])


From hakanardo at codespeak.net  Sun Oct 31 15:43:50 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 15:43:50 +0100 (CET)
Subject: [pypy-svn] r78619 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031144350.EC0ED282BAD@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 15:43:49 2010
New Revision: 78619

Added:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send_nounroll.py   (props changed)
      - copied unchanged from r78618, pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py
Removed:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_nounroll.py
Log:
rename


From hakanardo at codespeak.net  Sun Oct 31 15:52:39 2010
From: hakanardo at codespeak.net (hakanardo at codespeak.net)
Date: Sun, 31 Oct 2010 15:52:39 +0100 (CET)
Subject: [pypy-svn] r78620 -
	pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test
Message-ID: <20101031145239.E18BD36E0CB@codespeak.net>

Author: hakanardo
Date: Sun Oct 31 15:52:38 2010
New Revision: 78620

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send_nounroll.py
Log:
fixed tests

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send.py	Sun Oct 31 15:52:38 2010
@@ -165,7 +165,7 @@
         for j in range(69, 75):
             res = self.meta_interp(f, [j], policy=policy)
             assert res == 42
-            if self.optimizer == OPTIMIZER_SIMPLE:
+            if self.optimizer != OPTIMIZER_FULL:
                 self.check_enter_count(3)
                 self.check_loop_count(3)
             else:
@@ -353,7 +353,7 @@
         assert res == f(198)
         # we get two TreeLoops: an initial one, and one entering from
         # the interpreter
-        if self.optimizer == OPTIMIZER_SIMPLE:
+        if self.optimizer != OPTIMIZER_FULL:
             self.check_tree_loop_count(1)
         else:
             self.check_tree_loop_count(2)
@@ -409,7 +409,7 @@
         # we expect 1 loop, 1 entry bridge, and 1 bridge going from the
         # loop back to the start of the entry bridge
         self.check_loop_count(2)        # 1 loop + 1 bridge
-        if self.optimizer == OPTIMIZER_SIMPLE:
+        if self.optimizer != OPTIMIZER_FULL:
             self.check_tree_loop_count(1)   # 1 loop 
         else:
             self.check_tree_loop_count(2)   # 1 loop + 1 entry bridge  (argh)
@@ -433,7 +433,7 @@
             return node.x
         res = self.meta_interp(f, [55])
         assert res == f(55)
-        if self.optimizer == OPTIMIZER_SIMPLE:
+        if self.optimizer != OPTIMIZER_FULL:
             self.check_tree_loop_count(1)
         else:
             self.check_tree_loop_count(2)
@@ -466,7 +466,7 @@
             return n
         res = self.meta_interp(f, [55], policy=StopAtXPolicy(extern))
         assert res == f(55)
-        if self.optimizer == OPTIMIZER_SIMPLE:
+        if self.optimizer != OPTIMIZER_FULL:
             self.check_tree_loop_count(1)
         else:
             self.check_tree_loop_count(2)

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send_nounroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send_nounroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_send_nounroll.py	Sun Oct 31 15:52:38 2010
@@ -5,8 +5,10 @@
 from pypy.jit.metainterp.test.test_basic import LLJitMixin, OOJitMixin
 
 class LoopNoPSpecTest(test_send.SendTests):
+    optimizer=OPTIMIZER_NO_UNROLL
+    
     def meta_interp(self, func, args, **kwds):
-        return ll_meta_interp(func, args, optimizer=OPTIMIZER_NO_UNROLL,
+        return ll_meta_interp(func, args, optimizer=self.optimizer,
                               CPUClass=self.CPUClass, 
                               type_system=self.type_system,
                               **kwds)
@@ -22,9 +24,7 @@
 
 
 class TestLLtype(LoopNoPSpecTest, LLJitMixin):
-    def check_tree_loop_count(self, count):
-        # we get one less entry bridge
-        return LLJitMixin.check_tree_loop_count(self, count - 1)
+    pass
 
 class TestOOtype(LoopNoPSpecTest, OOJitMixin):
     pass


From arigo at codespeak.net  Sun Oct 31 15:52:58 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 15:52:58 +0100 (CET)
Subject: [pypy-svn] r78621 -
	pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test
Message-ID: <20101031145258.94810282B90@codespeak.net>

Author: arigo
Date: Sun Oct 31 15:52:57 2010
New Revision: 78621

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_jump.py
Log:
(hakanardo, arigo)  Kill this test.  The problem was actually that
the dstvalues list contains duplicates.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_jump.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_jump.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_jump.py	Sun Oct 31 15:52:57 2010
@@ -142,22 +142,3 @@
                              ('push', s12),
                              ('mov', ebx, s12),
                              ('pop', ebx)]
-def test_crash():
-    assembler = MockAssembler()
-    dstvalues = [-56, -64, -72, -80, -88, 15, 14, 15, 13, 12, 15, -96, 
-                 -104, -112, 10, 9, 8, 7, 6, 3, -120, 2, 1]
-    srcvalues = [-56, -64, -72, -80, -88, -224, -216, -224, -248, -232,
-                 -224, -96, -104, -112, -184, -192, -264, 2, -312, -344,
-                 10, 8, 14]
-    print len(dstvalues), len(srcvalues)
-    class MyLocation(AssemblerLocation):
-        def __init__(self, value):
-            self.value = value
-        def location_code(self):
-            return 'b'
-        
-    src = [MyLocation(i) for i in srcvalues]
-    dst = [MyLocation(i) for i in dstvalues]
-        
-    remap_frame_layout(assembler, src, dst, '?')
-    assert assembler.ops == []


From arigo at codespeak.net  Sun Oct 31 15:56:43 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 15:56:43 +0100 (CET)
Subject: [pypy-svn] r78622 -
	pypy/branch/jit-unroll-loops/pypy/jit/backend/x86
Message-ID: <20101031145643.9B57236E35D@codespeak.net>

Author: arigo
Date: Sun Oct 31 15:56:42 2010
New Revision: 78622

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/jump.py
Log:
Add an assert that complains explicitly.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/jump.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/jump.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/jump.py	Sun Oct 31 15:56:42 2010
@@ -7,7 +7,9 @@
     srccount = {}    # maps dst_locations to how many times the same
                      # location appears in src_locations
     for dst in dst_locations:
-        srccount[dst._getregkey()] = 0
+        key = dst._getregkey()
+        assert key not in srccount, "duplicate value in dst_locations!"
+        srccount[key] = 0
     for i in range(len(dst_locations)):
         src = src_locations[i]
         if isinstance(src, ImmedLoc):


From arigo at codespeak.net  Sun Oct 31 16:02:35 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 16:02:35 +0100 (CET)
Subject: [pypy-svn] r78623 - in pypy/branch/jit-unroll-loops/pypy/jit:
	backend/x86/test metainterp/test
Message-ID: <20101031150235.2A463282BD6@codespeak.net>

Author: arigo
Date: Sun Oct 31 16:02:33 2010
New Revision: 78623

Added:
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_loop_unroll.py
      - copied, changed from r78619, pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_loop_spec.py
Removed:
   pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_loop_spec.py
Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_unroll.py
Log:
Rename.


Copied: pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_loop_unroll.py (from r78619, pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_loop_spec.py)
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_loop_spec.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/backend/x86/test/test_loop_unroll.py	Sun Oct 31 16:02:33 2010
@@ -1,8 +1,8 @@
 import py
 from pypy.jit.backend.x86.test.test_basic import Jit386Mixin
-from pypy.jit.metainterp.test import test_loop_spec
+from pypy.jit.metainterp.test import test_loop_unroll
 
-class TestLoopSpec(Jit386Mixin, test_loop_spec.LoopSpecTest):
+class TestLoopSpec(Jit386Mixin, test_loop_unroll.LoopUnrollTest):
     # for the individual tests see
     # ====> ../../../metainterp/test/test_loop.py
     pass

Modified: pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_unroll.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_unroll.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/metainterp/test/test_loop_unroll.py	Sun Oct 31 16:02:33 2010
@@ -3,7 +3,7 @@
 from pypy.jit.metainterp.test import test_loop
 from pypy.jit.metainterp.test.test_basic import LLJitMixin, OOJitMixin
 
-class LoopSpecTest(test_loop.LoopTest):
+class LoopUnrollTest(test_loop.LoopTest):
     optimizer = OPTIMIZER_FULL
     automatic_promotion_result = {
         'int_add' : 3, 'int_gt' : 1, 'guard_false' : 1, 'jump' : 1, 
@@ -11,8 +11,8 @@
 
     # ====> test_loop.py
 
-class TestLLtype(LoopSpecTest, LLJitMixin):
+class TestLLtype(LoopUnrollTest, LLJitMixin):
     pass
 
-class TestOOtype(LoopSpecTest, OOJitMixin):
+class TestOOtype(LoopUnrollTest, OOJitMixin):
     pass


From arigo at codespeak.net  Sun Oct 31 16:47:20 2010
From: arigo at codespeak.net (arigo at codespeak.net)
Date: Sun, 31 Oct 2010 16:47:20 +0100 (CET)
Subject: [pypy-svn] r78625 - pypy/branch/jit-unroll-loops/pypy/jit/tool/test
Message-ID: <20101031154720.60B56282BDA@codespeak.net>

Author: arigo
Date: Sun Oct 31 16:47:18 2010
New Revision: 78625

Modified:
   pypy/branch/jit-unroll-loops/pypy/jit/tool/test/test_jitoutput.py
Log:
Fix.


Modified: pypy/branch/jit-unroll-loops/pypy/jit/tool/test/test_jitoutput.py
==============================================================================
--- pypy/branch/jit-unroll-loops/pypy/jit/tool/test/test_jitoutput.py	(original)
+++ pypy/branch/jit-unroll-loops/pypy/jit/tool/test/test_jitoutput.py	Sun Oct 31 16:47:18 2010
@@ -33,13 +33,13 @@
     assert info.tracing_no == 1
     assert info.asm_no == 1
     assert info.blackhole_no == 1
-    assert info.backend_no == 1
+    assert info.backend_no == 2
     assert info.ops.total == 2
     assert info.recorded_ops.total == 2
     assert info.recorded_ops.calls == 0
     assert info.guards == 1
-    assert info.opt_ops == 6
-    assert info.opt_guards == 1
+    assert info.opt_ops == 11
+    assert info.opt_guards == 2
     assert info.forcings == 0
 
 DATA = '''Tracing:         1       0.006992


From fijal at codespeak.net  Sun Oct 31 16:55:54 2010
From: fijal at codespeak.net (fijal at codespeak.net)
Date: Sun, 31 Oct 2010 16:55:54 +0100 (CET)
Subject: [pypy-svn] r78626 - pypy/extradoc/talk/pycon2011
Message-ID: <20101031155554.C3F82282BDB@codespeak.net>

Author: fijal
Date: Sun Oct 31 16:55:53 2010
New Revision: 78626

Modified:
   pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
Log:
oops. I think this it the final version, was laying down in my wc


Modified: pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
==============================================================================
--- pypy/extradoc/talk/pycon2011/pypy-optimizations.txt	(original)
+++ pypy/extradoc/talk/pycon2011/pypy-optimizations.txt	Sun Oct 31 16:55:53 2010
@@ -4,27 +4,35 @@
 PyPy is a virtual machine for Python, featuring an advanced just in time
 compiler, which can deliver exceptional performance.  This talk is going to be
 a deep dive into what exactly makes Python such a hard language to optimize,
-how PyPy is organized, and what optimizations our JIT can do for your code.
+how PyPy is organized, and what optimizations our JIT can do
+(and what it can't do) for your code.
+
+This talk will also contain a brief part about status of the PyPy project,
+it's goals and achievements.
 
 Session type: Talk
 Classification: Discuss in depth
 Abstract:
 
-The talk will contain a list of features that are hard to optimize in Python,
-why and how we're fixing it in PyPy. This also might be used as a rough guide
-what features in Python are optimizable right now and what are not.
+The talk will explain a bit in details how a python interpreter
+internally and why some operations are costly. We'll go through several
+python features, how the work, why they're slow in CPython and how we're
+fixing it.
 
 The list of mentioned features is not exhaustive, however we will try to
 focus at least on the following:
 
 * Dynamic language - In Python code we have no known types, like a statically typed
   language.  Even operations like "a + b" can do anything, unless we know more
-  about the code.
+  about the code, especially associated types.
 
 * Frame introspection - Frame objects need to be allocated for every function
   call, and all local variables are stored on the frame, and must be accessible
   from further down the call stack.
 
+  PyPy uses a novel approach called "virtualizables" which makes it possible
+  to not allocate frames at all in most common cases.
+
 * Object model - All user defined Python objects have a dictionary which stores
   their attributes, as does every type.  When Python does an attribute lookup
   this requires at least two dictionary lookups.
@@ -36,8 +44,8 @@
   decided to go via ctypes, this part will explain how are we going to handle
   ctypes calls to be fast.
 
-* `array` module - users of the CPython array module probably know it can save
-  them quiet a bit of memory, however it's also slower than using a list, due
+* `array` module - users of the CPython's array module probably know it can save
+  them quite a bit of memory, however it's also slower than using a list, due
   to the overhead of boxing and unboxing on every operations. Here we will tie
   everything together and describe how the ``array`` module is much faster with
   PyPy's JIT, combining our optimizations to: unbox values, remove the


From agaynor at codespeak.net  Sun Oct 31 17:04:44 2010
From: agaynor at codespeak.net (agaynor at codespeak.net)
Date: Sun, 31 Oct 2010 17:04:44 +0100 (CET)
Subject: [pypy-svn] r78627 - pypy/extradoc/talk/pycon2011
Message-ID: <20101031160444.94ACB282BE3@codespeak.net>

Author: agaynor
Date: Sun Oct 31 17:04:42 2010
New Revision: 78627

Modified:
   pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
Log:
Random grammar and typo fixes.


Modified: pypy/extradoc/talk/pycon2011/pypy-optimizations.txt
==============================================================================
--- pypy/extradoc/talk/pycon2011/pypy-optimizations.txt	(original)
+++ pypy/extradoc/talk/pycon2011/pypy-optimizations.txt	Sun Oct 31 17:04:42 2010
@@ -14,17 +14,16 @@
 Classification: Discuss in depth
 Abstract:
 
-The talk will explain a bit in details how a python interpreter
-internally and why some operations are costly. We'll go through several
-python features, how the work, why they're slow in CPython and how we're
-fixing it.
+The talk will explain a bit in details how a python interpreter works
+internally and why some operations are costly. We'll go through several python
+features, how they work, why they're slow in CPython and how we're fixing it.
 
 The list of mentioned features is not exhaustive, however we will try to
 focus at least on the following:
 
-* Dynamic language - In Python code we have no known types, like a statically typed
-  language.  Even operations like "a + b" can do anything, unless we know more
-  about the code, especially associated types.
+* Dynamic language - In Python code we have no known types, like a statically
+  typed language.  Even operations like "a + b" can do anything, unless we know
+  more about the code, especially associated types.
 
 * Frame introspection - Frame objects need to be allocated for every function
   call, and all local variables are stored on the frame, and must be accessible
@@ -41,10 +40,10 @@
   (except more PyPy specific) called map dictionaries. XXX type versions?
 
 * FFI calls - calling C from Python is costly and hard to optimize. In PyPy we
-  decided to go via ctypes, this part will explain how are we going to handle
-  ctypes calls to be fast.
+  expse C APIs to Python code via ctypes, this part will explain how are we
+  going to handle ctypes calls to be fast.
 
-* `array` module - users of the CPython's array module probably know it can save
+* `array` module - users of CPython's array module probably know it can save
   them quite a bit of memory, however it's also slower than using a list, due
   to the overhead of boxing and unboxing on every operations. Here we will tie
   everything together and describe how the ``array`` module is much faster with